To ensure that your site ranks well in search engine results pages (SERPs), you’ll need to make it easy for search engine bots to explore your most important pages. have a file robots.txt in WordPress it will help direct those bots to the pages you want them to index (and skip the rest).
In this article, we will cover:
- what is a file robots.txtand why it is important
- where the file is located robots.txt in WordPress.
- How to create a file robots.txt.
- What rules to include in your file robots.txt of WordPress.
- How to test the file robots.txt And send it to .
By the end of our discussion, you will have everything you need to set up a file robots.txt perfect for your WordPress website. Let us begin!
What is a WordPress robots.txt file (and why do you need it)
The file robots.txt WordPress default is pretty basic, but you can easily override it.
When you create a new website, search engines send a to ‘crawl’ it and create a containing. This way they will know which pages to display when someone searches for related keywords. At a basic level, it’s that simple.
The problem is that modern websites contain Many more elements apart from pages. For example, WordPress allows you to install plugins, which often come with their own directories. However, you don’t want it to show up in search engine results, as it’s not relevant content.
What it does is provide a set of instructions for search engine robots. He tells them, “Hey, you can look in here, but don’t go in those rooms over there!” This file can be as detailed as you like, and it’s fairly easy to create, even if you’re not an expert in the technical aspects.
In practice, search engines will crawl your website, even if you don’t have an archive. robots.txt configured. However, not creating it is inefficient. Without this file, you are letting the robots index all of your content, and they are so thorough that they could end up showing parts of your website that you don’t want other people to have access to.
Most importantly, without a file robots.txt, you will have many robots on your website. This can negatively affect its performance. Even if the contact is insignificant, communication is something that should always be among your top priorities. After all, there are few things that people hate slow websites more than (and that includes us!).
Where is the WordPress robots.txt file located?
When you create a WordPress website, a file is automatically configured robots.txt virtual located in the main folder of your server. For example, if your site is located at yourwebsite.comyou should be able to visit the address yourwebsite.com/robots.txtand see a file like this:
User-agent: * Disallow: /wp-admin/ Disallow: /wp-includes/
This is an example of a file robots.txt very basic. To put it in clear Spanish, the part just after User-agent: declare which robots the rules below apply to. An asterisk means that the rules are universal and apply to all robots. In this case, the file tells those bots that they can’t go to your directories wp-admin Y wp-includes. That makes some sense, since those two folders contain a lot of sensitive files.
However, you may want to add more rules to your own file. Before you can do that, you’ll need to understand that this is a file virtual. Usually the location of the file robots.txt of WordPress is inside your directory rootwhich is often called public_html either www (or bear the name of your website):
However, the file robots.txt that WordPress configures by default is not accessible at all, from any directory. It works, but if you want to make changes to it, you’ll need to create your own file and upload it to your folder root as a replacement.
We will cover several ways to create a new robots.txt for WordPress in a moment. For now though, let’s talk about how to determine which rules yours should include.
What rules to include in the WordPress robots.txt file
In the previous section, we saw an example of a file robots.txt generated by WordPress. It only included two short rules, but most websites set up more than that. Let’s take a look at two files robots.txt different and let’s talk about what each one does differently.
Here is our first example of robots.txt from WordPress:
User-agent: * Allow: / # Disallowed Sub-Directories Disallow: /checkout/ Disallow: /images/ Disallow: /forum/
this is a file robots.txt generic for a website with a forum. Search engines often index every thread within a forum. However, depending on what your forum is for, you may want to opt it out. This way, Google won’t index hundreds of threads about users making small talk. You can also set rules that indicate specific subforums to avoid, and allow search engines to crawl the rest.
You will also see a line that says Allow: / at the top of the file. That line tells the bots that they can crawl all the pages on your website, apart from the exceptions you set below. Similarly, you’ll notice that we indicate these rules to be universal (with an asterisk), as does the file robots.txt WordPress host.
Now let’s look at another example of robots.txt from WordPress:
User-agent: * Disallow: /wp-admin/ Disallow: /wp-includes/ User-agent: Bingbot Disallow: /
In this file, we configure the same rules that WordPress does by default. However, we also added a new set of rules that prevent it from crawling our website. Bingbot, as you can imagine, is the name of that robot.
You can be quite specific about which search engine robots have access to your website and which ones do not. In practice, of course, Bingbot is pretty benign (even if it’s not as cool as Googlebot). Nevertheless, there are some malicious bots out there.
The bad news is that they don’t always follow the instructions in your file. robots.txt (They are rebels, after all). It’s worth noting that while most bots will follow the instructions you give in this file, they won’t. will force To do it. You’re just asking nicely.
If you read up on the subject, you will find plenty of suggestions on what to allow and what to block on your WordPress website. However, in our experience, it is better to have few rules. Below is an example of what we recommend your first file be robots.txt:
User-Agent: * Allow: /wp-content/uploads/ Disallow: /wp-content/plugins/
WordPress traditionally likes to block access to directories wp-admin Y wp-includes. However, that is no longer considered good practice. Also, if you add metadata to your images for SEO purposes, there is no point in banning bots from crawling that information. Instead, the two rules above cover what most basic sites will require.
However, what you include in your file robots.txt It will depend on your site and your specific needs. So feel free to do some more research!
As a summary, we leave you a summary of the syntax of the robots.txt:
User-agent: bot-name Directive: directive-content Sitemap: http://www.domain.com/sitemap.xml
Directives that can be added:
- Disallow: to indicate that you do not want the bot to crawl the indicated files or directories.
- Allow: to indicate that bots should crawl this directory.
- Crawl-delay: the wait that the bots must do between each request within the site.
- no index: special for Google, to indicate which files you don’t want indexed.
Wildcards that can be combined:
- *: to indicate that the policy applies to all bots.
- $: This symbol allows you to put characters and make the bots not index documents or files with these characters.
- #: comments that will be ignored by bots.
How to Create a Robots.txt File in WordPress (3 Methods)
Once you have decided what to include in your file robots.txt, all you have to do is create it. you can edit robots.txt in WordPress using a plugin or manually. In this section, we’ll show you how to use two popular plugins to get the job done, and walk through how to create and upload the file yourself. Let’s go!
1. Use Yoast SEO
Yoast SEO needs no introduction. It is the most popular SEO plugin for WordPress, and it allows you to optimize your posts and pages to make better use of your keywords. On top of that, it also provides you with help when it comes to increasing the readability of your content, which means more people will be able to enjoy it.
We are personally fans of Yoast SEO because of its ease of use. Which also applies to creating a file robots.txt. Once you install and activate the plugin, go to the tab SEO > Tools on your desktop, and look for the option that says file editor:
Clicking on that link will take you to a new page, where you can edit your without leaving your desktop. There is also a useful button with the text Create robots.txt filewhich does exactly what it says:
When you click this button, the tab will show a new editor where you can modify your file robots.txt directly. Please note that Yoast SEO sets its own default rules, which override your file robots.txt existing virtual.
Every time you add or remove rules, remember to click the button Save Changes in robots.txtto be applied:
That was pretty easy! Now let’s see how another popular plugin does the same thing.
2. With the All in One SEO Pack plugin
is another well-known name when it comes to WordPress SEO. It includes most of the features that Yoast SEO offers, but some people prefer it because it is a more lightweight plugin. As to robots.txtcreating the file with this plugin is just as simple.
Once you have , go to the page All in One SEO > Utility Manager on your desk. Inside, you will find an option called Robots.txtwith a visible button for Activate just below. Click on it:
Now, a new tab will appear Robots.txt in your menu All in One SEO. If you click it, you’ll see options to add new rules to your file, save the changes you make, or delete it entirely:
Please note that you cannot make changes to your file robots.txt directly with this plugin. The file itself is grayed out, unlike Yoast SEO, which allows you to type whatever you want:
In any case, adding new rules is simple, so don’t let that little inconvenience put you off. More importantly, All in One SEO Pack also includes a feature that can help you block “bad” bots, which you can access from your tab. All in One SEO:
That’s all you have to do if you choose this…
Facebook Comments
Disqus