One of the most common problems that I usually find at the beginning of any audit for eCommerce is the content indexing problem in search engines. (When talking about search engines, I am going to refer to Google since in Spain it is the most used search engine by far)
Day after day the technology to create an online store is more accessible, to the point of being able to pay less than €30 per month and in a few clicks start selling online through platforms such as Shopify, Wix and among other CMS. It’s fantastic, but for SEOs and Google the ecosystem begins to get complicated, due to the about saturation of content on the internet, which should be tracked.
As a consequence this
Today, working on the basic onpage SEO of Meta Titles and Meta Descriptions is no longer enough to position yourself in search results. This is where the importance of technical SEO stands out for facilitate the crawling and indexing process
To be more specific in improving the crawl budget. (Don’t be scared “the parablex”, Below I will explain what it is and how its optimization can help us so that your online store is visible on Google).
Let’s see what it consists of 🔎
What is the crawl budget
The causes of existence of crawl budget are the crawlersthe “robots” (Googlebot) that crawl the internet to find content to be crawled and indexed in search results.
to understand what is the crawl budget it is essential to understand how Google works.
This uses web trackers, “crawlers” which are surfing the Internet following all links found. In order to collect all the information on the pages you have visited in order to save it on their servers. From there, they organize all this information based on some parameters (algorithms) and store it in their index to later show it to users.
If you want to know more about how it works
Googlebot is not just made up of a crawler as it uses multiple crawlers to find all kinds of information on the web. This fact must be taken into account since each of these crawlers usually focuses on a different type of information: images, video, mobile, etc.
Google informs us here about each of them and their function:
The most accurate definition of the Crawl Budget refers to number of pages that Google crawls on a website within a certain period of timewhich can be translated to the budget of time and resources that Google allocates to crawl and index a website.
Why Google allocates crawl budget to websites
Athough it does not seems, Google does not have unlimited resources to track the internet and distributes these to track the billions of pages on the internet. So they need a way prioritize your resources for tracking as efficient as possible by playing with the opportunity cost of allocating X resources to Y or Z.
Assigning a crawl budget to each website helps them to execute this action efficiently so as not to waste their server resources on low value websites (spam websites).
How they assign the crawl budget to websites
It is based on 2 factors: crawl limit and crawl demand.
- Crawl limit: What tracking capacity does a web page have? It is determined by the requests that can your server answer.
- Crawl demand: Which URLs are most worth (re)crawling, based on their popularity and how often they are updated.
Why is the crawl budget important for SEO?
Now you understand the problem, where the objective is to establish a framework to find a solution.
For us to understand each other, Google will only allocate resources to your eCommerce if you make his job easier, otherwise your competition will be waiting for their turn to be tracked and take a piece of the tracking budget pie for your theme.
When is it advisable to work the crawl budget
How can we read in the own you have to worry if your online store exceeds 10,000 URLsotherwise it should have no problem crawling your entire site.
How many pages does my eCommerce have?
To check the number of crawled pages of your store Go to Google Search Console>Coverage, there you can find the number of URLs Google has crawled.
In the example above I analyze an eCommerce: you can see how Google has tracked a total of about 11,966 pagesso I will have to worry about optimizing and prioritizing the pages to crawl and index.
How to know the crawl budget of an online store
The crawl budget as such it is a concept. Therefore, to measure it, we will use the parameter of number of pages crawled per day, cWith the objective of determining the number of pages that are tracked by the bots of our online store with respect to the total amount of it.
The easy way to see how many pages google crawls a day is to go to Google Search Console>Settings>Crawl Statistics.
The data to take into account will be the crawl requests by URL in order to know the URLs that it has tracked per day. In the event that the number of URLs requested during a month is below the total number of pages on the site, we may diagnose a crawl budget problem, andbecause Google bot is not having enough crawl budget to discover our entire web page.
If you want to know more about this panel here I leave the
How to audit my crawl budget
To know the “Health” of the Budget of an online store, it will be enough to know the number of URLs that Googlebot tracks during a period of time, as we have seen in the previous section.
On the other hand there are other indications to audit the Crawl Budget.
Discovered: currently unindexed
This is one of the reasons why a URL has not been indexed in Google, which can be found in the section on Google Search Console>Coverage>Excluded.
How well does he explain to us the : “Discovered: currently not indexed. Google has found the page, but has not yet crawled it; probably because you’ve determined that doing so would overload the website. Therefore, you have had to defer the crawl. For this reason, the date of the most recent crawl does not appear in the report.”
In case you find a considerable number of pages that you want to index excluded for this reason, you should start working on their optimization so that they are indexed.
But don’t worry now it will explain the different ways to optimize the crawl budget 👇
How to increase my crawl budget
To improve the crawl budget of your online store, you can do it in two ways:
- Reduce the number of URLs to crawl: This is the simplest, because it depends 100% on us.
- Get Google bot to allocate more crawl time for you: In this case it will be difficult to demonstrate the action correlation so that it shows that due to X actions Googlebot has not allocated more tracking time.
We see the different actions that you can carry out ordered by their ease of implementation and impact on the crawl budget according to my experience:
Robots.txt
The Robots.txt is your best ally in this field, since it is the document that will allow you reduce the number of pages to crawl. We’ll just tell Google which pages we don’t want it to crawl.For example, eCommerce management pages such as My Account, Cart, Whislist and all the pages that do not cover a search intent as such.
upload speed
Make it easy to track your eCommerce by optimizing your server and a lighter website.
301 redirect chains
The fewer steps Google has to take to find a page in code 200 (correct response codes from a server) the better, andto which each jump from URL to URL there is an unnecessary consumption of resources for the precious servers of Google and this does not interest us.
Sitemaps.xml
Prioritize Correctly crawling and indexing your business pages through the correct use of the Sitemap.xml document
Index tag control
Optimize and prioritize the pages that should be indexed. Take the opportunity to block by robots.txt those crawlable pages that are in
Thin Content
Don’t waste your crawl budget on pages that do not cover any search intent.
duplicate content
Obviously let’s not make Google crawl the same URLs twice, because they have duplicate content between them.
Content posting frequency
Remember that Google’s function is to discover new content on the internet so that it is indexed in search results. If you post frequently value content and optimized for user searches, Google will reward you by allocating more crawl budget.
Authority
increase your authority in the sector through strategies of and PR actions in other media to be an eCommerce with notoriety in your business vertical.
conclusion
If you still need more ideas on how to optimize the crawl budget I recommend that you be and the updated documentation offered by the search engine itself, especially the section on
If you have any questions about it, do not hesitate to write it in the comments, I will be happy to help you! Until the next article 😉
A hug!
Stay informed of the most relevant news on our news channel
Facebook Comments
Disqus