May 29, 2025
In this episode of Search Off the Record, Martin and
Gary from the Google Search Relations team take a deep dive into
how Googlebot and web crawling work—past, present, and future.
Through their humorous and thoughtful conversation, they explore
how crawling evolved from the early days of the internet, when
scripts could index a chunk of the web from a single homepage, to
the more complex and considerate systems used today. They discuss
the basics of what a crawler is, how tools like cURL or Wget
relate, and how policies like robots.txt ensure crawlers play nice
with web infrastructure.
The conversation also covers Google’s internal shift
to unified infrastructure for all crawling needs, highlighting how
different teams moved from separate crawlers to a shared system
that enforces consistent policies. They explain why some fetches
bypass robots.txt (like user-initiated actions) and the rising
impact of automated traffic from new products and AI agents. With a
nod to initiatives like Common Crawl, the episode ends with a look
at the road ahead, acknowledging growing internet congestion but
remaining optimistic about the web’s capacity to adapt.
Resources:
Episode transcript →
https://goo.gle/sotr092-transcript
Listen to more Search Off the Record → https://goo.gle/sotr-yt
Subscribe to Google Search Channel → https://goo.gle/SearchCentral
Search Off the Record is a podcast series that takes
you behind the scenes of Google Search with the Search Relations
team.
#SOTRpodcast #SEO #SearchOfTheRecord
Speakers: Martin Splitt, Gary Illyes
Products Mentioned: Googlebotl,
Gemma,
Google AI