Managing Crawling & Indexing

Who should attend this course?

If you want a better understanding of how search engines visit, interpret and index your site, this course is for you. It’s designed for anyone who understands the basic principles of SEO and wants to feel empowered to control how search engines interact with a site. It’s also a great course for junior and intermediate digital marketing teams agency-side who are looking to improve their technical SEO understanding.

Course Content

During this jam-packed training we’ll look at:

Crawling your site & controlling what search engines see

• How Google crawls, assesses & indexes your pages
• Crawl budget – what is it, and is it something you should worry about?
• How to crawl your site and start spotting problems or opportunities
• The different types of crawlers we have, and the advantages of each
• Answers to any questions you have on setting up the SEO Spider to crawl your site
• How to assess your content’s accessibility to search engine bots and spot common problems
• How to use a crawl of your site, Search Console data and other sources to find the true size of your site
• Search Console’s Crawl Stats report – what does it tell us, and what can changes mean?
• How to find crawl errors, and what to do with them
• Spotting crawl traps
• The tools, tactics and tags we can use to make crawling our site as efficient as possible
• Using XML sitemaps to help content discovery
• Using the robots.txt file to control which URLs can be crawled
• Using nofollow to stop links from being followed
• How to audit your site architecture and internal link structure to optimise the way your site is crawled
• Crawling Javascript-based websites (and why it’s not as scary as you might think)
• How to mimic the way Google ‘sees’ and understands your content through page rendering
• An introduction to log file analysis – how to get the data, and what you can discover
• How using a combination of Search Console’s Coverage report, log file analysis and a crawl of your site gives you a 360-degree view of how a search engine is interacting with your content
• Controlling indexing

Using the noindex directive to prevent indexing

• How rel=canonical tags work, the potential issues and when you should use them
• Mobile-first indexing
• How to manage duplication and encourage Google to focus on your most important pages
• Using the URL Inspection tool in Search Console to understand what Google thinks of any page on your site

Search Console’s Index Coverage report

• A dive into the Index Coverage report – what should we look at, what do the different reports mean, and which should we be concerned by?
• Seeing which pages from your XML sitemap have been indexed, and using multiple XML sitemaps to help get granular on what’s working
• Error pages – what they classifications mean and how to show Google you’ve fixed problems
• Valid with warning pages – what causes these warnings, and when should I worry about them?
• Excluded pages – A run through the 15 different reasons Google can choose not to index a page, & when further analysis is warranted
• A deeper look at why Google might see your pages, but choose not to index them even if we want them to
• What the crawling and indexing of your site can tell us about potential content quality issues

What to expect

This detailed, fast-paced and highly engaging training course is for anyone looking to control what pages on your site are crawled and which can be indexed.

We’ll cover the common crawling & indexing tools and tactics we have at our disposal plus how to use Search Console to uncover what has been indexing on your site, and what issues Google sees.

We’ll also jump into next-level tactics such as crawling Javascript sites, using log files & finding content issues.

You’ll learn how to get actionable findings and generate better results from the mountain of data that’s available to SEOs today. This course aims to give you a starting point in the world of technical SEO – a great building block for any SEO’s foundational knowledge.

Ultimately, you’ll leave with a better understanding of the tools and options at your disposal to manage how search engines visit your site and dictate which pages they consider for indexing. And you’ll know how to find issues search engines have found with your site and the principles of building a solid site structure.

You’ll also learn how to use your crawl, index coverage and log file data to identify improvements for your SEO roadmap and help create a higher-quality, leaner & more efficient site. All in a friendly environment where you can ask any questions you can think of.

If you register for one of our training courses you don’t only get a day of in-depth and practical training from some of the industry’s most respected professionals, you also get a ticket to brightonSEO, the world’s biggest specialist SEO event.