Managing Crawling & Indexing

In just one day you’ll build results-oriented SEO plans based on search engine behaviour and hard facts.

Managing Crawling & Indexing Training Course

Who should attend this course?

If you want a better understanding of how search engines visit, interpret and index your site, this course is for you. It’s designed for anyone who understands the basic principles of SEO and wants to feel empowered to control how search engines interact with a site. It’s also a great course for junior and intermediate digital marketing teams agency-side who are looking to improve their technical SEO understanding.

Course Content

During this jam-packed training we’ll look at:

Crawling your site & controlling what search engines see

  • How Google crawls, assesses & indexes your pages
  • Crawl budget – what is it, and is it something you should worry about?
  • How to crawl your site and start spotting problems or opportunities
  • The different types of crawlers we have, and the advantages of each
  • Answers to any questions you have on setting up the SEO Spider to crawl your site
  • How to assess your content’s accessibility to search engine bots and spot common problems
  • How to use a crawl of your site, Search Console data and other sources to find the true size of your site
  • Search Console’s Crawl Stats report – what does it tell us, and what can changes mean?
  • How to find crawl errors, and what to do with them
  • Spotting crawl traps
  • The tools, tactics and tags we can use to make crawling our site as efficient as possible
  • Using XML sitemaps to help content discovery
  • Using the robots.txt file to control which URLs can be crawled
  • Using nofollow to stop links from being followed
  • How to audit your site architecture and internal link structure to optimise the way your site is crawled
  • Crawling Javascript-based websites (and why it’s not as scary as you might think)
  • How to mimic the way Google ‘sees’ and understands your content through page rendering
  • An introduction to log file analysis – how to get the data, and what you can discover
  • How using a combination of Search Console’s Coverage report, log file analysis and a crawl of your site gives you a 360-degree view of how a search engine is interacting with your content
  • Controlling indexing

Using the noindex directive to prevent indexing

  • How rel=canonical tags work, the potential issues and when you should use them
  • Mobile-first indexing
  • How to manage duplication and encourage Google to focus on your most important pages
  • Using the URL Inspection tool in Search Console to understand what Google thinks of any page on your site

Search Console’s Index Coverage report

  • A dive into the Index Coverage report – what should we look at, what do the different reports mean, and which should we be concerned by?
  • Seeing which pages from your XML sitemap have been indexed, and using multiple XML sitemaps to help get granular on what’s working
  • Error pages – what they classifications mean and how to show Google you’ve fixed problems
  • Valid with warning pages – what causes these warnings, and when should I worry about them?
  • Excluded pages – A run through the 15 different reasons Google can choose not to index a page, & when further analysis is warranted
  • A deeper look at why Google might see your pages, but choose not to index them even if we want them to
  • What the crawling and indexing of your site can tell us about potential content quality issues

What to expect

This detailed, fast-paced and highly engaging training course is for anyone looking to control what pages on your site are crawled and which can be indexed.

We’ll cover the common crawling & indexing tools and tactics we have at our disposal plus how to use Search Console to uncover what has been indexing on your site, and what issues Google sees.

We’ll also jump into next-level tactics such as crawling Javascript sites, using log files & finding content issues.

You’ll learn how to get actionable findings and generate better results from the mountain of data that’s available to SEOs today. This course aims to give you a starting point in the world of technical SEO – a great building block for any SEO’s foundational knowledge.

Ultimately, you’ll leave with a better understanding of the tools and options at your disposal to manage how search engines visit your site and dictate which pages they consider for indexing. And you’ll know how to find issues search engines have found with your site and the principles of building a solid site structure.

You’ll also learn how to use your crawl, index coverage and log file data to identify improvements for your SEO roadmap and help create a higher-quality, leaner & more efficient site. All in a friendly environment where you can ask any questions you can think of.

If you register for one of our training courses you don’t only get a day of in-depth and practical training from some of the industry’s most respected professionals, you also get a ticket to brightonSEO, the world’s biggest specialist SEO event.

13 Sep, 2023


Early Bird
Charlie Williams

Charlie Williams

Screaming Frog | SEO & Content Strategy Consultant

Charlie is a trainer for the team at Screaming Frog, the agency famous for the SEO Spider and Log File Analyser. He’s a regular industry speaker who loves to share ideas on content design & strategy, technical SEO, and creative user experiences. A search veteran with over a dozen years in the industry, Charlie is also an independent consultant. His SEO & content strategy consultancy, Chopped Digital, is dedicated to helping create better sites and smarter SEOs. Fascinated by the challenges of building an online business, you can find Charlie talking content, UX, SEO & food at the bar, or likely cooking something up in the kitchen..

Don't just take it from us

What people say about our courses

Why train with brightonSEO

If a bonus day at the seaside isn't enough reason for you to book one of our training sessions, here are some other things that make our training great:


Experts, not trainers

We won’t waste your time sitting you in front of full-time trainers who haven’t got their hands dirty for god knows how long; we get experts from different fields of the industry on board to give you the information they know works, not just the theory of what’s supposed to.


Specific, detailed and unique

Most training providers are looking for a course they can run month after month, week after week, so they end up being general and catch-all. Not our courses … they’re more detailed and in-depth than you’ll find anywhere else.


You drive the agenda

We don’t drive the agenda, you do. Before each training course, we ask what you want to be taught, and we actually listen. All the workshops we run have been selected because they’re the most commonly requested training subjects useful to you.


Group sizes

The group sizes are ideal, more than a handful to offer a bit of diversity and substance to the day, but small enough that the training can remain hands-on and personal, making sure all questions are directly answered.


The perfect duration

The day runs from 10am-4.30pm to keep it punchy. Any longer than that and it starts to drag, any shorter and you’re not able to properly sink your teeth into what you’re covering. Don’t worry, you’ll also get plenty of breaks to network.


Proper food & drink

There’s a couple of coffee breaks, morning and afternoon, and we also provide a proper lunch. That’s right, we’re talking real food with beer and wine rather than soggy sandwiches or mush slopped from a tray.

Get all the latest news and offers

Join our mailing list



c/o SiteVisibility, Plus X Innovation Hub, Lewes Road Brighton, BN2 4GL, United Kingdom
Follow us
© 2023 brightonSEO.
Designed & Developed by
Lemon Hive
Powered by
Lemon Hive
RoughAgenda Ltd. Company number: 07917764