The experience of crawling the web from a full scale crawler's point of view

brightonSEO April 2025

In this talk, Guillaume will present the history of the technology behind one of the top web crawler in the world (second SEO crawler in the world, while the company is still modest), and what it means to crawl the web like a search engine while not being a search engine.

Share this talk: