What is the crawl budget or tracking budget? |

This time at we want to talk to you about an element that has a fundamental impact on SEO, but that is sometimes not given the necessary attention: Google’s crawl budget. The crawl budget (or crawl budget) is the time that Google assigns to crawl our site. Depending on whether this time is greater or less, Google will be able to crawl more parts of our website and, as a consequence of this, it will be able to index more or fewer pages as it considers them relevant or not for the user… We have already begun to deduce the importance and transcendence that This factor will have for the online visibility of a website and for its SEO strategy.

Why do they call it a labyrinth? There are no turns, no corners, no nothing, it just goes on and on… Or maybe not… maybe I’m taking it for granted… (Labyrinth, 1986)

Factors that influence the crawl budget

As usual, Google hasn’t specifically stated what exact factors drive the crawl budget.but thanks to several tests and analysis of the Seos we can venture some:

  • upload speed: The importance that Google gives to the loading time of a website is already well known. Impatient users want websites that look and load fast. The majority use of mobile phones has led to this obsession with creating WPO-optimized sites and, without a doubt, loading speed is already a positioning factor for Google. In addition, the less time it takes for the Google bot to crawl your website, the higher the crawl budget (in general) it will grant you.
  • web architecture: A good architecture of the web in directories, folders, internal linking, URL structure, that is, a logical order in the structure and hierarchy of the web, will favor Googlebot to track a web more easily and therefore see as a positive factor for the crawl budget.
  • The robots.txt: how the robots.txt file is defined will set the tone for how Googlebot tracks us, which parts of the web it will go through and which ones it won’t.
  • The web authority (PA): It is not declared but it seems obvious that the more authority a website has, Google will give it more affection, which translates into a higher tracking budget, since they are usually trusted websites with many organic visits.
  • The update: Google likes websites that are alive, that continually update their content, which means that they remain operational. It is another factor that almost certainly improves the crawl budget.
See also  Configure robots.txt for Magento |

Crawl Demand + Crawl Limit = Crawl Budget

The Crawl Demand These are the factors that are directly related to the desire that Google has to track your website. In this case, the authority of the site and the update frequency of the already indexed content are what define this crawl demand.

The Crawl Limit It has to do with the url limit that Googlebot establishes to track your website and it has to do with the service capacity of the servers. A high crawl frequency can bring down a server that does not have much dedicated load and also allows Google to save on crawl resources.

How to optimize the crawl budget to improve the SEO of a website?

After the factors that count for the tracking budget, we already have an idea that an SEO will have to get down to work to optimize what is needed in order to improve this element. As always, this optimization has to have an objective and be within an SEO strategy, it is not enough to improve left and right because we will lose focus. Here we will see, for example, that a higher tracking budget does not always mean better positioning or better reading of the web. Sometimes, less is more.

Let’s imagine that we already have a website or that it is given to us by what little we can do at first to improve the architecture. It also has an authority that, although we can improve over time, it is not something that we can force quickly. So where do we direct our efforts, where do we start?

  • Analyze the crawl budget: The first thing is to see how Googlebot is visiting us (its desktop and mobile versions). The easiest and fastest way is to go to Search Console to the “Crawl > Crawl statistics” section. Although it is a very important field to review the daily tracking budget in relation to the loading speed, it does not offer us too detailed information. In any case, it helps us to observe major changes that may give us clues that something good or bad is happening in Google’s relationship with our site.

A much more powerful analysis is offered by the access logs. If we get a good package of logs for a limited period (note, the logs are the records of all the entries to the web and therefore in very large webs there are thousands and thousands of data that take up a lot of space), we can analyze them with tools of the kind , a Screaming Frog plugin. With the logs we can know exactly where the Google bots go. This information is pure gold to know if you are crawling what you want or need, or instead the bots are wasting your crawl budget on broken or irrelevant pages. This analysis will lead us to make the most precise optimizations.

See also  Geolocation of Images and its Importance for SEO -

  • Optimize the robots.txt: If we have done a good analysis and we discover that Googlebot is repeatedly wasting part of its crawl budget on directories or pages that do not interest us, it is time to include directives in the robots.txt that block this crawl. It can also help us for exactly the opposite, and that is that in the analysis we realize that it is not going through this or that route and maybe we have it blocked by mistake in the robots.txt. Some directives in robots.txt to limit the crawl frequency are not applicable for Google and will ignore them, such as the Crawl delay directive. With this guideline you can stipulate the speed at which a user-agent track your website by applying a time: Crawl delay: 30 (for instance); on the other hand, a very useful guideline if we want some trackers not to bring down our website. But as we have indicated, Google ignores it.
  • Search Console settings: There is a way to limit Google crawling through Search Console, although it is not highly recommended, it is possible that in some cases a high crawl can cause problems with server speed. Once in the account, in the upper right corner we can go to “Site settings”, once there, the “Crawl frequency” field will appear and two options: Allow Google to optimize for my site (recommended) and Limit Google’s maximum crawl rate.
  • Response codes: Log analysis can also tell us if Googlebot spends part of the crawl budget crawling redirected pages (301, 302, etc). This is not bad, but it always slows down and wastes resources. A site with few redirects will be more fluid for bots. The 404 is already another topic. If we notice that we have a lot of broken internal links that are throwing 404s, then we should try to either remove them or redirect them to the appropriate page; if not, Googlebot will waste part of the crawl budget on 404s that don’t add anything.
  • internal link: here we enter the perhaps more technical and tricky topic because how to prevent Google from following internal links that are necessary for the proper functioning of the web at the user level (U/X), but that we are not interested in at the SEO level (Menus, footer, Sides)? The solution proposed by the SEO masters. This advanced technique is only recommended in very exceptional cases, since it is a way of hiding from Google parts of your website that the user does see, so it is to be expected that Google is working to understand java code better and better and ends up , sooner rather than later, for detecting these links and who knows whether to penalize this technique.
See also  What is Business Manager and what is it used for? -

Why is the crawl budget important for a SME website?

It seems that the Crawl Budget is very important for the largest sites and, indeed, it is. Knowing how and where Google navigates on a huge site helps reduce efforts and fine-tune the strategy, but what about SMEs or more modest websites? Also. First, because it can involve an analysis that differentiates you from the competition and you can reach much more accurate or precise conclusions. Second, because adjusting and increasing the tracking budget is an unavoidable objective within the SEO strategy if we want Google to index and position what really matters. So from our point of view at , the legs of SEO would look like this:

  • Indexing Audit
  • Crawl budget optimization
  • On page optimization
  • WPO Optimization
  • link building

Of course, all these legs of the table are supporting an SEO strategy with well-defined objectives that will start from an online market study (competition, niches, keyword research).

We hope that the importance and what is the Google crawl budget or crawl budget has been made clear. We encourage you to leave any comments. In addition, you can count on our if you need it.

Loading Facebook Comments ...
Loading Disqus Comments ...