A Meta Robots tag is a snippet of HTML code that provides instructions to search engines on how to crawl or index pages on a website. This gives you a higher level of control over how specific pages are indexed and displayed in the SERPs.
The sEO is the art of controlling the way your site links to search engines
Usually this means getting as much visibility as possible, but there are times when you may want to exclude a page from the SERPs or restrict how search engines use it.
To do this, you need to give instructions to the search engines on how you want a specific page to be treated. A tag called Meta Robots tag is the perfect way to do this.
So:
- What is the meta robots tag and why is it important?
- What are the different types of robots guidelines?
- How do you implement this attribute on a website?
These are all questions to which I provide clear and precise solutions in this mini-guide.
Let’s go!
Chapter 1: What are the basics to know about the meta robots tag?
In this chapter, I will simply focus on the basic notions of the meta robots tag tag and the main reasons why it is important.
1.1 What is a meta robots tag?
To put it in simple terms, this meta tag tag tells search engines what actions they can (or cannot) take on a certain page
The major search engines will respect the commands included in this tag.
IMPORTANT: This tag has no site-wide effect. It can contain different values on different pages of the same website.
1.2 Basic syntax
The basic syntax of the meta robots tag is quite simple:
<meta name=”robots” content=”directives“>
Example:
To use it, add the attribute to the <head> section of each web page to provide instructions to crawlers
Unlike the robots.txt file which provides site-wide instructions, the meta robots attribute only applies to the page on which it appears.
IMPORTANT: THEcorrect use of the syntax when constructing the tag is extremely important as using incorrect syntax can negatively impact your site’s presence and ranking on the SERPs.
1.3. Why is the use of the Meta Robots tag important?
This attribute is used to control the access of crawlers to a page and to prevent search engines from indexing pages that you do not want to be indexed.
This avoids wasting time on crawling unimportant pages of your site and thus increase the efficiency of the crawling. It also allows you to prevent pages from showing up in the SERPS and can help limit the problems of duplicate content.
In general, you won’t need to use the robots meta tag. If you want to maximize your web traffic and pagerank, you will generally let the search engines do their job unimpeded
But there may be specific cases where it will make sense to restrict what they should do.
For example:
- Limited privacy
If a page is not really intended for the general public, the meta robots tag will reduce the chances that people will find it. It won’t be invisible and anyone with the link can still see it, but it won’t show up in the SERPS.
- A page is not ready for the public
Let’s say a new site feature is in beta and you want only your employees or certain customers to see it
You can launch the page and use a meta robots attribute to keep search traffic away until it’s ready. However, be sure to review the tag when you want to take the page live!
- Spam links
There’s no easier way to hurt your site’s ranking than to link to websites that contain spam. But what if such a site actually contains a great resource that you want to share with your visitors?
The use of a meta robots tag tag can tell search engines to not follow the linklink, thus protecting your page’s ranking.
- Image protection
For most businesses, it’s best to let your images (watermarked, attributed) spread all over the net. But if you want to present an image without seeing it used elsewhere, the robots attribute can tell Google not toindex.
Now that you know the basics about this tag, let’s see the different directives and tips to configure it on a website.
Chapter 2: What are the main Robots directives and how to implement them on a website?
In this section, we’ll look at different ways to apply robots directives at the page level to tell search engine robots rules about how they should treat the pages on your site.
Then, I’ll give you some tips for using this tag and techniques for implementing it on your site
2.1. What are the different types of robots directives?
Robots directives directives are sections of code that give instructions to web crawlers on how the content of a page should be parsed or indexed
There are a number of directives you can use to communicate with search engines about how to handle the pages on your site
Here are some examples and how each one works.
2.1.1. All
This directive tells the crawlers that there are no limitations on crawling and indexing. It works in a similar way to the Index and directives Follow.
2.1.2 Index
The index directive indicates that crawlers are allowed to index the page. This is taken into account by default. You do not necessarily need to add it to a page for it to be indexed.
2.1.3. Noindex
If you use the above code in the header section of a web page, you are effectively telling search engines not to index the page’s content and not to display it in Google search results.
If the page was already indexed, this directive will tell the crawler to remove the page from the index.
2.1.4. Follow
When you use this directive on a page, you tell the search engines to follow all the links on the page and also pass on the authority of the links.
2.1.5. Nofollow
If you use the code below in the head section of a web page, you are basically telling search engines not to follow any links (both internal and external links) on that page and not to pass on Google’s pagerank through those links.
In short, it prevents search engines from following links on a page and passing on any equity.
2.1.6. Noindex, Nofollow
Note that you can combine multiple directives into one code and say multiple things at once. For example, the code below tells search engines not to index the content of the page and at the same time not to follow any links on the page.
2.1.7 Noindex, Follow
Similarly, the code below prohibits search engines from indexing the content of the page, but requires the tracking of all links on that page.
2.1.8. None
This directive is equivalent to ”noindex, nofollow”. It basically tells search engines that the page should be ignored.
2.1.9. Noarchive
This directive prevents search engines from displaying the cached link in search results
It can also be used to prevent competitors from scraping your content, which is especially useful on e-commerce sites where prices are frequently updated.
2.1.10. Nocache
This directive is the same as the Noarchive directive, but used by Internet Explorer and Firefox only.
2.1.11. Nosnippet
This directive tells search engines not to display the extended description (i.e., meta-description) of the page in search results.
Nosnippet is useful if you want to have more control over what users should see on the site’s content, which could lead to an increase in CTR based on the search query.
2.1.12. Notranslate
This directive prohibits search engines from suggesting a translation of the page in SERPs. This can be useful when you may not want your page content to be translated, as sometimes machine translation is not accurate.
You can also specify sections of your page not to be translated using a class:
<...>
2.1.13. Noimageindex
This directive tells search engines not to index images on the page. This is useful if you don’t want others to republish your images as if they were the owners, especially if those images are copyrighted
2.1.14. Unavailable_after
This directive tells search engines not to display the page in SERPs after a specified date or time
This is useful for pages that are relevant to a particular time period, such as event registration pages or promotional landing pages (e.g. Black Friday)
The content will be de-indexed once the event is over and you will no longer see the page appear.
For pages with a limited window of useful time, it is more practical to use ” unavailable_after“than to place a noindex on the page at a later date.
In short, think of it as a noindex tag with a timer.
2.1.15. Noodp
This directive instructs search engines not to use Open Directory Project metadata for the titles or snippets displayed for this page. However, this one has recently deprecated.
2.1.16. Max-snippet
This directive helps you set the maximum number of characters that Google should display in the SERPs for a page. In the example below, the number of characters will be limited to 150.
2.1.17. Max-video-preview
This will configure a maximum number of seconds for a video clip preview. In the example below, Google will display a 10 second preview
2.1.18. Max-image-preview
This tells Google the size of the image it displays for a page on the SERP. There are three possible values:
- None – No image snippet will be displayed;
- Standard – The default image preview will be used;
- Large – The largest possible preview can be displayed.
Now, depending on the requirements, you can use these guidelines in the robots meta tag of your website.
2.2. usage tips
Many site owners without web design experience may want to tinker with meta tags for SEO purposes
You don’t need to be an html guru to edit your meta robot tag, but you do need to use the values correctly.
Here are three things to keep in mind:
- You can use multiple values (directives) you can use multiple values (directives): Combining them gives you more nuanced control and you can group them all into one robot tag. Just separate these different values with commas
Example
- Uppercase or lowercase does not matter:
Example is the same as
- Spaces do not matter: Only commas commas are needed to separate multiple values. Spaces will make it easier for you to read, but you can also ignore them
Example
2.3. Conflicting tag values (directives)
It is possible that your tag contains conflicting values
For example:
{code type = HTML} {/ code}
The tag ” None “tag tells search engines to not to follow your linksbut the tag ” Follow “tag tells them that they should follow those links. This tag says one thing and its opposite at the same time.
You should always check your Meta tags when you enter them to avoid this kind of situation. However, if it does happen, it’s not disastrous, it just brings out unpredictable results.
Most search engines have never given a public answer on how they handle a conflict like this
Nevertheless, Google has, and it still relies on the the most restrictive in the example above, Google will ignore “Follow”
Here’s another example:
Most search engines will choose the “ nosnippet “, as it is the most restrictive.
Yandex has also explained its policy and goes with the parameter by default setting
Also, if your robots.txt file file conflicts with your robots meta tagtag, there is no definitive answer as to how the two are resolved by most search engines.
Now, let’s look at techniques for implementing the meta robots tag on your website.
2.4. How to implement a meta robots tag on a website?
There are two ways to implement a robots meta tag, and they both have the same effect on crawlers
2.4.1. tag placed in the section.
One of the most common ways to use the meta robots tag is to place it as a meta tag in the head section of your page like this:
{code type = HTML}
..
…{/code}
This is always just a single line of code, and should be inserted into the html header section of the page
If you use WordPress.org, there are plugins that allow you to tinker with your meta tags tags alone or as part of your as part of your SEO strategy without directly modifying the code
Meta robots tags do not affect the appearance of your website to your visitors.What they do depends on the values you put in them.
In addition, by placing the value ” robots “in the ” Name “attribute, you specify that all crawlers must follow the value “ noindex “in the ” Content ”
In many cases, you may want to refer to specific crawlers using their user-agent name (user-agent) in the “Content” attribute Name ”
In this case, you will need to replace the value “ robots “with theuser-agent when you want to follow a specific directive
For example
As mentioned above, a Meta tag can contain several directives in a robots meta tag
Similarly, a page can contain various Meta Robot tags if you want to target different user agents
2.4.2. X-Robots-Tag in the HTTP header
Just like the use of the meta robots tagtag, the robots directives directives can be applied as the x-Robots tag tag in the HTTP header of any given URL
That is, any directive used in the meta tag can also be applied to the HTTP header tag
For example:
Or if you want to reference a specific user agent in the X-Robots-Tag, here is an example:
2.5. Tools for adding Meta Robots tags to a website
If you want to add this attribute to your site, you can copy the tags presented in this article and paste them into the HTML code of your site
In addition, there are tools you can use to generate custom instructions and to automatically add tags to web pages generated by content management systems.
Chapter 3: What are the best SEO practices for robots tags?
A robots.txt file file will only tell the crawler which areas of the website it can crawl. However, it will not tell the crawler whether to index or not.
To help you, you can use robot tags to guide crawlers on indexing and many other functions
There are two types of robot tags: The meta Robots tag tag and the x-robots tag.
3.1. Meta Robots beacon
I already talked about this tag above. Let’s go to the X-robots tag.
3.2. X-Robots tag
The meta Robots tag is similar to the x-robots tag, because it allows you to control crawling and indexing only at the page level
The only difference is that the x-robots tag tag is defined in thehTTP header header of a page to control the crawling and indexing of an entire page or specific elements of it
This X-Robots tag is mainly used to control, crawl and index pages non-HTML pages
Example of X-Robots tag :
x-robots-tag: noindex, nofollow, nosippet
As shown in the example below, the x-robots tag tag uses the same set of directives as the Meta Robots tag. To implement it, you need to have access to the .htaccess, .php or server file to modify the headers.
3.3. best SEO practices for Robots tags
- Do not use the Meta Robots tag and x-robots on the same page, as one of them will become redundant.
- If you don’t want the pages to be indexed, but want the link equity to be passed on to the linked pages, you should use the Meta Robots tag with directives like noindex, follow. This is the best technique to control indexing instead of blocking with the robots.txt file.
- You don’t need to add index or directives follow directives on every page of your website to index it. They are considered as by default.
- In case your pages are indexed, avoid blocking them in the robots.txt file and using Meta Robots simultaneously. Because, to take into account the Meta Robots tag, the crawlers must explore the page, but the blocking the robots.txt file file will not allow them to do so.
- Use the X-robots tag to control the crawling of non-HTML files such as: images, PDFs, flash files, videos, etc.
Chapter 4: Meta Robots vs. Robots.txt tags: Which is better for SEO?
This ended up being a bit of a trick question, as both are important for your site’s SEO.
Since the meta robots tags tags and robots.txt files files aren’t really interchangeable, you’ll need to use both to provide the correct parameters to web crawlers
As the Search Engine Journal says, “The robots.txt files files tell crawlers about the entire site, while the meta robots tags tags focus on a specific page [on a Web site]. “
For example, if you want to de-index one of your web pages from the Google SERPs, it’s best to use a “Noindex“meta tag rather than a robots.txt directive
However, if you want to prevent entire sections of your website from appearing on the SERPs, using a robots.txt file would be the best option.
Chapter 5: Other Questions About Meta Robots Tags
Here are some frequently asked questions about the Meta Robots tag.
5.1. What is the robots tag in SEO?
The Meta robots tag is a piece of code in the section of your web page that gives you the power to decide which pages you want to hide from search engine spiders and which pages you want them to follow and index.
5.2, Where is the robots meta tag placed?
The robots tag is a piece of HTML code found inside the section of a web page. Its role is to control the way this page is crawled and indexed by search engines.
5.3. What are the advanced settings for meta robots?
These settings allow you to define the meta robots of a page or a publication:
- No Image Index you forbid search engines to crawl images on this page or this article
- No Archive no Archive: Prevent search engines from displaying a cached copy of this page
- No Snippet prevents search engines from displaying a snippet of this page in search results and prevents them from caching the page.
By default, this field is empty, indicating that the default settings are respected. So if your default site is set to index/follow (or if you have not set any specific or restrictive rules), then this should be equivalent to “index/follow”.
5.4. Can I forget about the Meta Robots tags on my website?
If all this is too much, you can safely omit the robots tags. You don’t have to include them, even to say ” All “or ” Index ”
All search engines, by default, will index your page and crawl your links unless you tell them not to. The use of meta tags is completely optional.
However, if you run a complex website or want to keep certain content less visible, the meta robots tag can be a great way to do so.
While it’s not complete proof, it is respected by search engines and is a way to control how they use your content.
5.5. Does the Robots meta tag add any additional SEO benefit?
Not at all. The addition of a meta robots tag tag does not give you an “advantage” or additional benefit over not having added it
It has no effect on search rankings, so there is no need to focus on using the meta robots tag to improve a web page’s search rankings.
5.6. Are search engines required to follow the guideline given in the Meta Robots tag?
No.
Search engines are not required to follow the guidelines given in the meta robots tag, as these guidelines are given by you as a guideline, and it is up to the search engines to decide whether or not to honor this request
That said, in 99% of cases, search engines will honor your request and carry out the instructions provided in the meta robots tag.
5.7. Does adding a Meta Robots tag to a web page prevent crawling?
No, adding a Meta Robots tag to a page does not prevent or prohibit search engine spiders from crawling that specific page. Search engine spiders will sometimes crawl that URL to see if the Meta tag has changed.
In fact, it all depends on the guidelines you have specified in the code for this tag.
5.8. Is it a good idea to change the Meta Robots tag too often on a section of your website?
No, it is generally not a good idea to change the Meta Robots tag frequently on specific sections of your website
The use of the meta robots tag should have a clear purpose and you should not keep switching from one preference to another, without a clear purpose in mind.
5.9. Which is better between the meta robots tag and robots.txt?
The robots.txt are best for banning an entire section of a site, such as a category, while a meta robot tag tag is more effective for banning single files and pages. But you can use both at the same time.
5.10. What is a meta noindex tag?
A meta robots noindex tag allows you to prohibit search engines from indexing a page. Excluding a page from the index means that the page can no longer be displayed in the SERPs.
In summary
If you want to control how web crawlers index your site, the meta robots tag is a good mechanism
It is easy to understand, easy to implement and has a significant impact on how search engines index your website.
When used properly, this tag will ensure that search engines treat your site exactly as you want them to. However, using it incorrectly can lead to inconvenience, as search engines may forget about you.
Therefore, just make sure to use it when necessary and with great care.
See you soon!
