Internet Marketing Podcast

Search Off the Record takes you behind the scenes of Google Search and its inner workings! In each episode, the folks from the Search Relations team will give you background info on the decision-making behind launches, feature prioritization in Search Console, and the projects Google Search teams are working on. They will share fun stories from the many conferences they attend as well as from their day-to-day working life at Google. They will also dive into the currently trending conversations in the SEO community at large. Have a listen!

May 29, 2025

In this episode of Search Off the Record, Martin and
Gary from the Google Search Relations team take a deep dive into
how Googlebot and web crawling work—past, present, and future.
Through their humorous and thoughtful conversation, they explore
how crawling evolved from the early days of the internet, when
scripts could index a chunk of the web from a single homepage, to
the more complex and considerate systems used today. They discuss
the basics of what a crawler is, how tools like cURL or Wget
relate, and how policies like robots.txt ensure crawlers play nice
with web infrastructure.

 

The conversation also covers Google’s internal shift
to unified infrastructure for all crawling needs, highlighting how
different teams moved from separate crawlers to a shared system
that enforces consistent policies. They explain why some fetches
bypass robots.txt (like user-initiated actions) and the rising
impact of automated traffic from new products and AI agents. With a
nod to initiatives like Common Crawl, the episode ends with a look
at the road ahead, acknowledging growing internet congestion but
remaining optimistic about the web’s capacity to adapt.

Resources:

Episode transcript →
https://goo.gle/sotr092-transcript 

 

Listen to more Search Off the Record → https://goo.gle/sotr-yt

Subscribe to Google Search Channel → https://goo.gle/SearchCentral

 

Search Off the Record is a podcast series that takes
you behind the scenes of Google Search with the Search Relations
team.

 

#SOTRpodcast #SEO #SearchOfTheRecord

 

Speakers: Martin Splitt, Gary Illyes

Products Mentioned: Googlebotl,
Gemma,
Google AI

Search Off the Record takes you behind the scenes of Google Search and its inner workings! In each episode, the folks from the Search Relations team will give you background info on the decision-making behind launches, feature prioritization in Search Console, and the projects Google Search teams are working on. They will share fun stories from the many conferences they attend as well as from their day-to-day working life at Google. They will also dive into the currently trending conversations in the SEO community at large. Have a listen!

In this episode of Search Off the Record, Gary Illyes
and Martin Splitt from the Google Search team dive deep into the
foundations of how the web works—specifically HTTP, TCP, UDP, and
newer technologies like QUIC and HTTP/3. The two reflect on how
even experienced web professionals often overlook or forget the
mechanics behind these core protocols, sharing insights through
technical discussion, playful banter, and analogies ranging from
messenger pigeons to teapots. The conversation spans key concepts
like packet transmission, connection handshakes, and the importance
of status codes such as 404, 204, and even 418 (“I’m a
teapot”).

Throughout the conversation, they connect these
protocols back to real-world implications for site owners,
developers, and SEOs—like why Search Console might report network
errors, and how browser or server behavior is influenced by
low-level transport decisions. With a mix of humor and expertise,
Gary and Martin aim to demystify a crucial part of the internet’s
infrastructure and remind listeners of the layered complexity that
makes modern web experiences possible.

Resources:

Episode transcript →https://goo.gle/sotr091-transcript 
 

Listen to more Search Off the Record → https://goo.gle/sotr-yt

Subscribe to Google Search Channel → https://goo.gle/SearchCentral

Search Off the Record is a podcast series that takes
you behind the scenes of Google Search with the Search Relations
team.

#SOTRpodcast #SEO #Http

Speakers: Lizzi Sassman, John Mueller, Martin Splitt,
Gary Illyes

Products Mentioned: Search Console – General

Search Off the Record takes you behind the scenes of Google Search and its inner workings! In each episode, the folks from the Search Relations team will give you background info on the decision-making behind launches, feature prioritization in Search Console, and the projects Google Search teams are working on. They will share fun stories from the many conferences they attend as well as from their day-to-day working life at Google. They will also dive into the currently trending conversations in the SEO community at large. Have a listen!