Managing Crawling & Indexing Training Course

In just one day you’ll build results-oriented SEO plans based on search engine behaviour and hard facts.

Managing crawling and indexing
beginnertechnicalFundamentalsSpecialistHands On

Upcoming Courses

April 19, 2023


£1020 £680 - Early Bird

Book Now

Who should attend this course?

If you want a better understanding of how search engines visit, interpret and index your site, this course is for you. It’s designed for anyone who understands the basic principles of SEO and wants to feel empowered to control how search engines interact with a site. It’s also a great course for junior and intermediate digital marketing teams agency-side who are looking to improve their technical SEO understanding.

Course Content

During this jam-packed training we’ll look at:

Crawling your site & controlling what search engines see

  • How Google crawls, assesses & indexes your pages
  • Crawl budget – what is it, and is it something you should worry about?
  • How to crawl your site and start spotting problems or opportunities
  • The different types of crawlers we have, and the advantages of each
  • Answers to any questions you have on setting up the SEO Spider to crawl your site
  • How to assess your content’s accessibility to search engine bots and spot common problems
  • How to use a crawl of your site, Search Console data and other sources to find the true size of your site
  • Search Console’s Crawl Stats report – what does it tell us, and what can changes mean?
  • How to find crawl errors, and what to do with them
  • Spotting crawl traps
  • The tools, tactics and tags we can use to make crawling our site as efficient as possible
  • Using XML sitemaps to help content discovery
  • Using the robots.txt file to control which URLs can be crawled
  • Using nofollow to stop links from being followed
  • How to audit your site architecture and internal link structure to optimise the way your site is crawled
  • Crawling Javascript-based websites (and why it’s not as scary as you might think)
  • How to mimic the way Google ‘sees’ and understands your content through page rendering
  • An introduction to log file analysis – how to get the data, and what you can discover
  • How using a combination of Search Console’s Coverage report, log file analysis and a crawl of your site gives you a 360-degree view of how a search engine is interacting with your content
  • Controlling indexing

Using the noindex directive to prevent indexing

  • How rel=canonical tags work, the potential issues and when you should use them
  • Mobile-first indexing
  • How to manage duplication and encourage Google to focus on your most important pages
  • Using the URL Inspection tool in Search Console to understand what Google thinks of any page on your site

Search Console’s Index Coverage report

  • A dive into the Index Coverage report – what should we look at, what do the different reports mean, and which should we be concerned by?
  • Seeing which pages from your XML sitemap have been indexed, and using multiple XML sitemaps to help get granular on what’s working
  • Error pages – what they classifications mean and how to show Google you’ve fixed problems
  • Valid with warning pages – what causes these warnings, and when should I worry about them?
  • Excluded pages – A run through the 15 different reasons Google can choose not to index a page, & when further analysis is warranted
  • A deeper look at why Google might see your pages, but choose not to index them even if we want them to
  • What the crawling and indexing of your site can tell us about potential content quality issues

Meet Your Trainer

Charlie Williams

Charlie Williams | Screaming Frog

​​Charlie is a trainer for the team at Screaming Frog, the agency famous for the SEO Spider and Log File Analyser. He’s a regular industry speaker who loves to share ideas on content design & strategy, technical SEO, and creative user experiences.

A search veteran with over a dozen years in the industry, Charlie is also an independent consultant. His SEO & content strategy consultancy, Chopped Digital, is dedicated to helping create better sites and smarter SEOs.

Fascinated by the challenges of building an online business, you can find Charlie talking content, UX, SEO & food at the bar, or likely cooking something up in the kitchen.

What to expect

This detailed, fast-paced and highly engaging training course is for anyone looking to control what pages on your site are crawled and which can be indexed.

We’ll cover the common crawling & indexing tools and tactics we have at our disposal plus how to use Search Console to uncover what has been indexing on your site, and what issues Google sees.

We’ll also jump into next-level tactics such as crawling Javascript sites, using log files & finding content issues.

You’ll learn how to get actionable findings and generate better results from the mountain of data that’s available to SEOs today. This course aims to give you a starting point in the world of technical SEO – a great building block for any SEO’s foundational knowledge.

Ultimately, you’ll leave with a better understanding of the tools and options at your disposal to manage how search engines visit your site and dictate which pages they consider for indexing. And you’ll know how to find issues search engines have found with your site and the principles of building a solid site structure.

You’ll also learn how to use your crawl, index coverage and log file data to identify improvements for your SEO roadmap and help create a higher-quality, leaner & more efficient site. All in a friendly environment where you can ask any questions you can think of.

If you register for one of our training courses you don’t only get a day of in-depth and practical training from some of the industry’s most respected professionals, you also get a ticket to brightonSEO, the world’s biggest specialist SEO event.

“I would heartily recommend this course to anyone in the eCommerce field who wants to get to maximize their SEO potential. (Ecommerce Live Online)”

Jon Sharman
SEO Specialist, Life Science Group

“BrightonSEO's Local SEO Training with Greg Gifford is the perfect course to get you up to speed with all things local search, from review signals to link building, Google My Business to content creation, and much, much more. ”

Stephanie Newton
Content and Community Manager, BrightLocal

“Duncan’s CRO training was fantastic. He demonstrated the conversion optimisation process in a methodical and engaging way, using relevant case studies and activities to bring it to life. I definitely left with a reinforced appreciation of how important CRO is for us and also the confidence to strengthen our own optimisation programme.”

Sarah Wilhelmi
Head of SEO,

“BrightonSEO is THE ultimate event for anyone wanting to immerse themselves in talks from the leaders in both Search, Content and Influencer Marketing. The training sessions the day before are even more personal, supercharging you up with 1-2-1 coaching and getting under the bonnet with an expert.”

John Sadeghipoor
Global Head of Digital Marketing, FitFlop

“We attend every Brighton SEO (it's like our Christmas!)”

Laura Slingo
Digital Campaign Manager, Seeker Digital

“I've gained invaluable learnings and taken away actionable tactics and ideas for strategy formulation which I've been able to act upon almost immediately. The informal yet structured classroom style of delivery provided an excellent inclusive and engaging environment.”

Dawn Anderson
Director, Move It Marketing

“So much background information and explanations provided together with clear detailed plans for implementing what was covered.”

Mike Glaire
Director, Edinburgh Training and Conference Venue

“I had a long wishlist of things I wanted to be able to do, It was clear from the start that the trainer was a real specialist who knows the topic inside out.”

Rob Watson
Head of Marketing, Glendinning Group

“I absolutely loved the training I just attended and thought it was completely worth my time and my company's money. My head is about to explode with new information right now.”

Juliet Perrachon-Bazire d’Arguesse
Co Founder, Bébé Voyage

“I thought you’d like to know how your sessions are genuinely benefitting people and that your instructors are going above and beyond.”

Matt Allfrey
SEO Manager, iCrossing UK

Why Train with BrightonSEO

Experts, not Trainers

We won’t waste your time sitting you in front of full time trainers who haven’t got their hands dirty for god knows how long; we get experts from different fields of the industry on board to give you the information they know works, not just the theory of what’s supposed to.

Specific, Detailed and Unique

Most training providers are looking for a course they can run month after month, week after week, so they end up being general and catch-all. Not our courses … they’re more detailed and in-depth than you’ll find anywhere else.

You Drive the Agenda

We don’t drive the agenda, you do. Before each training course, we ask what you want to be taught, and we actually listen. All the workshops we run have been selected because they’re the most commonly requested training subjects useful to you.

Group Sizes

The group sizes are ideal, more than a handful to offer a bit of diversity and substance to the day, but small enough that the training can remain hands-on and personal, making sure all questions are directly answered.

The Perfect Duration

The day runs from 9.30-4.00 to keep it punchy. Any longer than that and it starts to drag, any shorter and you’re not able to properly sink your teeth into what you’re covering. Don’t worry, you’ll also get plenty of breaks to network.

Proper Food & Drink

There’s a couple of coffee breaks, morning and afternoon, and we also provide a proper lunch. That’s right, we’re talking real food with beer and wine rather than soggy sandwiches or mush slopped from a tray.

Money Back

There’s simply no worse feeling than putting aside a day out of the office to go to a training course, handing over your cash and a course failing to live up to your expectations.

We do all we possibly can to make our courses the best you’ve attended but if you ever feel we didn’t live up to your expectations, we’ll give you a total refund, no questions asked.

Sign up to the mailing list for more info about the event
How we use the data in this form.