In this talk, Guillaume will present the history of the technology behind one of the top web crawler in the world (second SEO crawler in the world, while the company is still modest), and what it means to crawl the web like a search engine while not being a search engine.
The experience of crawling the web from a full scale crawler's point of view
brightonSEO April 2025
