Crawl Rate Optimisation

Help search engines crawl your site more easily and effectively through crawl optimisation
ENQUIREBOOK CONSULTATION

How We Can Help You With Crawl Optimisation


Smart Crawl Optimisation Analysis and Strategy


Search engines naturally seek to be efficient in their crawling strategies when traversing the web and are thought to visit web pages which either change more frequently or are more important to search engine users than other less important pages.  Search engine crawlers also visit servers (hosting websites) based upon how much capacity the server can tolerate in simultaneous connections (hits from bots).  This is to ensure the server is not damaged, and also the websites hosted by the server.

On larger sites we find there are often issues whereby search engines are not either finding or visiting pages frequently because there are blockages, there are issues with content or structural quality on the site, or there are issues with the site being slow.

Our team have huge experience in analysing and interpreting crawl frequency, and then providing recommendations to open up blockages, reduce duplication of content and therefore combining signals for search engine crawlers to consolidate strength, and for speed optimisation.  It doesn’t necessarily follow you will rank higher in search engines as a result of crawl optimisation, but it certainly follows that combining otherwise competing signals so the strength is focused on non-duplicate URLs (web pages) cannot be negative.  It also follows that unblocking areas which are rich in content and which search engines cannot access because of issues with web page directives added in website development, can also not be negative.  It also follows that speeding up a website or website sections so that search engine crawlers can get round more URLs (web pages) from their scheduled ‘crawling list’ when they visit, can also not be negative.

These are the areas we focus on with crawl rate optimisation analysis and strategy development.  We use a range of tools to analyse and identify where and when search engine crawlers are visiting and redirect them to more appropriate URLs.  We help search engines understand which parts of the site we consider important and where they should focus their crawling activity.

Crawl optimisation via speed improvements

If you have a slow website, Googlebot or other search engine crawlers arriving with their scheduled ‘shopping list’ of website pages to visit may not get round all the URLs on the list.  They may also not have time to spend on pages which are also important and could be discovered on these visits.  This could impact your site’s indexing.  By understanding where search engines are visiting and how frequently, identifying issues with slow loading pages, and how to improve these, we can help you get maximum effectiveness from crawling.

To Index or Not To Index

There may be some areas of your site which actually are not very useful to be visited directly from search engine results pages.  That is not to say that they do not add some type of value to visitors when they are on the site (for example, for some aspects of navigation).  There may also be some pages on your site which you don’t feel are good enough quality to be a direct landing page for visitors from search engine users.  Search engines may also agree with this.  These types of pages might not be good enough to be indexed and search engines may choose to not include them in their results pages.  However, low quality pages when crawled could be dragging down your site down in search engine results so for now it might be best to choose to noindex these pages.  We’ll help you identify these pages and advise regarding whether to try to improve these pages or noindex them so that search engines put them to one side.

We study crawling a lot and share our knowledge with the industry

We’ve been a little bit obsessed with web search engine crawling for quite a few years.  We’ve shared our knowledge within the industry and spoken at conferences both in the UK and internationally (as far and wide as the United States and Australia).  We’ve contributed to industry publications around the subject of crawl frequency, URL scheduling and how these can impact website migrations on larger sites.  So, we know a thing or two about this topic.  We’re well placed to help you get the best when search engine bots come to call.

Duplicate, near duplicate and similar content

There’s a lot of confusion in the SEO world when it comes to duplicate, near duplicate and similar content.  There are even some SEOs who advise clients to remove all aspects of duplication on their website.  This is often a mistake as duplication is often tied more to the search engine query than the content on the page.  Does the content on more than one page meet the query behind the informational need in the same intent and context?  If so, regardless of what is on the page, this could be considered a case where duplication occurs.  Exact duplicate content is filtered out in search engines before they even get indexed anyway, because search engines are thought to have a ‘dup-checker’ in their search engine ‘boiler-room’ (anatomy).  This merely checks using a ‘content checksum’ (a kind of binary calculation) when when returned says whether two pages (or more) are exactly the same.  If these two (or more) pages are exactly the same, all but one of these will be dropped and one will be chosen to be shown in search engine results.  However, the web page chosen by the search engine may not always be the one which we would prefer the search engine to show as brands and website owners.  Crawl optimisation can also help with this as we can signal to search engines which of the URLs should be chosen as the ‘canonical’ (the chosen one).

SEO Services


Explore more of our digital services

Penalty recovery

Penalty Recovery

Backlink analysis

Backlink Analysis

Crawl rate optimisation

Crawl Rate Optimisation

More Reasons To Choose Us As Crawl Optimisation Partner


Get maximum value from search engine crawling

Identify the best places for Googlebot and other crawlers to visit and emphasise these to them

Speed up your site

Speeding up your site as part of crawl optimisation can never be a bad thing

Send better quality signals

We’ll help you identify the weak areas in your site and help you put together a strategy to either improve or exclude this content for better quality signals when search engines come to call.

Free Initial Crawl Consultation

If you’re wondering what could be possible for your brand, we’ll carry out an initial mobile analysis of your site and let you know what growth you could achieve realistically and what mobile success would look like.

Call us on:

0161 212 7348

Ready to start winning with mobile?

Call us on:

0161 212 7348

Ready to Start?

Receive free marketing tips

8 + 9 =