NoIndex

The Noindex tag is a code that is added to web pages to indicate to search engines that they should not be indexed, even if they are crawled. In this way, all pages on which the “noindex” directive is placed will not be integrated into the search engine index, and therefore will not be displayed in the SERPs.

The digital world is driven by performance. Indeed, all sites want to have their pages indexed to get more traffic, a good return on investment and a good visibility.

At the same time, other owners want to prevent search engines from ranking certain pages for various reasons, but don’t necessarily know how to do it. And if you are reading this article, you are probably in the case, which is where the noindex tag comes in.

This directive that allows website owners or developers to prohibit search engines to not display certain pages of their website in search results is increasingly used nowadays.

But to apply it, it is first important to know the necessary know-how, as well as the criteria to follow.

So,

• What is Noindex?
• How to apply it on a page or a website?
• What are the pages on which noindex can be applied?

These are some of the issues that we will find solutions to in the rest of this article. Let’s follow together!

Chapter 1: Noindex: What is it?

In this chapter, I will introduce you to what a noindex tag means, and then we will see the following series of elements:

• How to apply the noindex tag on a page?
• Which pages to add this tag to?
• How to use noindex and nofollow?

1.1. what is noindex?

Noindex is a tag that tells search engines such as Google, Bing or Yahoo that they should not include a given page in the search results. This tag is often used in the head section of HTML or on response headers

If search engines find the noindex tag in the HTML code of a web page, it will not be included in the indexing and will not be displayed to users in the SERPs after a query.

As you can already see, the opposite of noindex is “index” which clearly states indexing. Thanks to noindex, you have the power to control the indexing of the different pages of your website without making a big effort.

Therefore, it represents an essential tool for SEO. Google always obeys a noindex directive, while it considers the index as a simple recommendation that you have made for your pages.

Also to be continued: how to check if your site is indexed in Google.

1.2. how to insert noindex ?

To exclude a web page from the indexing of search engines, simply insert Noindex in the metadata of the page concerned. Indeed, it is in this meta tag that are indicated the instructions that Google or other search engines must follow for a website.

However, to prevent search engines from inserting a page into their index you must add the following tag in the header of the HTML code:

Rather than boycotting all crawlers, you can decide to apply this meta tag to a particular search engine

For example, if you want to prohibit the indexing of a certain page to Googlebot only, you can replace the value of the “name” attribute with “Googlebot”, it gives this:

As for Yahoo, their crawler name is “slurp”. However, in the procedures for good SEO, it is not important to tag the robots individually.

You need to insert the noindex directive from a field in the HTTP response header. To do this, you need to add the following code to your HTTP response header:

X- Robots-Tag: noindex

1.3. What pages does the noindex directive apply to?

Here are the pages on which you can apply the noindex directive, so the pages that potentially do not need to be indexed in Google.

1.3.1. Author archives on blogs with only one author

When you are a single author on your blog, you probably have pages that will have a 90% similarity to the main page of your blog. These pages are not useful for Google and can sometimes be held as duplicate content

The solution to avoid these types of content is to completely disable the author archive.

You also have the choice to enable or disable it with the Yoast SEO tool. However, if for any reason you would like to keep this archive on your site, but without it appearing in search results, you can use Noindex.

You can also do it with the Yoast seo tool. So here is how to noindex single author pages with this tool:

• Log in to your wordpress site: Once you are logged in, you will go to the dashboard;
• Click on user: You will see the button “users”, in the left menu ;

Source: Yoast

• Head to all users;

Source: Yoast

• Choose from the user overview, the right user profile to edit;
• Get to the page where you want to edit the profile, select the box that says not to use the author’s archives in search results. This will tell Google that it should not index the page of this author.

Source: Yoast

1.3.2. Some forms of custom publications

Sometimes a web programmer or a plugin adds custom page types to the site, which you might not want to be indexed. For example, some e-commerce sites may add specifications such as dimensions and weight as a custom post type.

Source: Road to Blogging

This kind of content is considered low quality content. You will find that these pages are of no use to either users or Google, and therefore should not be included in the search results pages.

1.3.2.1. Thank you page

The only purpose of the thank you page is to thank the customer or the subscribers to your newsletter.

https://i.pinimg.com/originals/e7/4f/0c/e74f0c0edd1b8c0c81d4210ce8388d40.jpg

These are often pages with low content, with possibilities of additional sales and social diffusion, but which do not provide any value for Internet users who take Google to find good answers to their queries. Therefore, these pages should not appear in the Google SERPs

1.3.2.2. Administrator or login pages

The majority of login pages should be excluded from Google. But these pages often appear in Google. I advise you to keep yours out of Google’s index by adding a noindex tag.

Source : Behance

The ones that should make the exception are the pages connecting to community services such as Dropbox or similar platforms.

Ask yourself why you would search for a login page on Google, when you are not involved with the site in question. If there’s no chance you’re doing such searches, it’s safe to say that Google doesn’t need to index these login pages.

On the other hand, if you use WordPress, you are safe, since the CMS cannot automatically index the login page of the site itself.

1.3.2.3. Internal search results

Search results are practically the last pages that Google would want to direct its users to. If you want to disrupt the search experience, you should link to other search pages and not to an actual results page.

On the other hand, links on search results pages still have great value. Therefore, all these links should be followed. The robots meta configuration for following these links is:.

1.4. Lists of content types to set on noindex with the Yoast SEO tool

Here is a list of content types on which you can add a noindex tag with Yoast SEO tool.

1.4.1. Individual articles, pages, media or taxonomy items.

The advanced settings area in the yoast SEO meta field of the content item edit screen provides administrators and editors with the ability to set custom directive information such as noindex and nofollow as well as other advanced directive settings.

To noindex an individual article, set the “Allow search engines to display this… in search results” setting to “No“.

Source: yoast

The presentation may be different, as the above example is taken from a default post type.

1.4.2. Noindex of an individual news article

The yoast seo new plugin gives you the possibility to tell Google news, that it should not index a given news article. You can find this option in the plugins section of the meta area in “Google news”.

Source : yoast

1.4.3. Noindex of post types, categories, tags or other taxonomies

If you want to not index a group of items, set up this feature by following the steps below. By default, 404 pages and search pages are set to noindex.

• Login to your WordPress website: Once logged in, you will be on your dashboard;
• Select “SEO”: on the left side of the screen you will see a menu, on it click on SEO ;
• Click on “Search Appearance”;

Source: Yoast

• Choose the tab that represents the type of content, taxonomy or archive type you want to remove from search results;

Source: Yoast

• Activate the “Show XYZ in search results” button: To index the content, set the button to “yes”, otherwise set it to “no”;

Source : Yoast

• Select “Save changes”;

1.4.4. Noindex of an entire site

It is also possible to prevent the entire site from being indexed, but it is up to the search engines to honor this request. Here’s how to noindex an entire site on Yoast SEO:

• Log into your wordpress site: when you are logged in, you will land on your dashboard;
• Select “settings”: On your left, you will find a menu, this is where you will find the settings;

Source: yoast

• Select the “read” option;

Source: yoast

• Then, mark the box “discourage search engines from indexing this site” ;

Source: yoast

• Finally, select “save changes”.

1.4. When to use “noindex” and “nofollow” separately?

When you add a “nofollow” tag to a web page, it prevents search engines from crawling the links on that page. This also means that any ranking authority that the pages have in the SERPs will not be transferred to the pages that this tag links to.

This way, any page with a “nofollow” directive will find that its links have been ignored by Google, as well as by other search engines.

You should know that you can add a “noindex” directive alone or with a “nofollow” directive. Or vice versa, i.e. add the nofollow directive alone

But do you know when to really use “noindex” and “nofollow” separately?

Indeed, add only the noindex tag when you would not like a search engine like Google to index your page, but you accept that it follows the links included on it. This gives ranking authority to the other pages your page links to.

Paidlanding pages are a good illustration of this fact. Indeed, it is not necessary for search engines to index landing pages for which users must pay, but it is necessary for the pages to which they are linked to benefit from their authority.

You should only add a nofollow tag, if you want a search engine not to follow the links on your web page, but to index the page. There aren’t many examples of when an entire page can be marked with a “nofollow” tag without being accompanied by a “noindex” tag.

When thinking about what you should do for a given page, the question is whether to add the “noindex” tag with or without the “nofollow” tag.

1.5. When to use noindex and nofollow in combination?

It is recommended to use both the nofollow and noindex tag, when you would not like Google or other search engines to index a page in the search results and you would also not like them to follow the links on that page.

Thank you pages are the perfect illustration of this fact. You don’t want search engines to be able to index your thank you page or to follow the link to your offers and start indexing the content of those offers.

Here are the steps to add together or separately a nofollow and noindex tag.

1.5.1. Step 1: Copy one of the following tags

In the case of noindex use separately and we had already seen a little above:

In the case of nofollow:

When nofollow and noindex are combined it looks like this:

1.5.2. Step 2: Add the tag to the part of the page’s HTML code

This is to add the tag to the header of the web page. All you have to do is manually paste the tag into the code of your web page. It may seem complicated at first, but don’t worry, because it’s very simple. Here’s how to do it:

First, you need to open the source code of the page you’d like to deindex. After that, paste the entire tag within a new line in the section of the page’s HTML code or the page’s header. The screenshots captured above will help you in the process.

The tag means the beginning of your header :

Source: hubspot

Here is the meta tag for “noindex” and “nofollow” pasted into the header:

Source: hubspot

The tag indicates the end of the header:

Source: hubspot

And that’s it, it’s done. This tag tells the search engines to turn around by excluding the page from all search results.

1.6. Difference between the noindex and disallow directive in a robots.txt file

When you use the disallow directive in a robots.txt file, you are telling search engines not to crawl the areas that are defined in this file.

This directive is used, for example, for files such as those containing images in large databases to save the precious Budget crawl of search engines.

On the other hand, you should not use this directive if you do not want any content to be indexed. Even if the disallow command prohibits the exploration of a specific page, it can still be displayed in the index, if backlinks from other websites point to it.

That’s why you should never combine noindex and disallow on the same page. Because robots usually access the robots.txt file first. When they index a website, these crawlers notice the disallow directive first and therefore give up crawling the subpages that are included.

As a result, the robots fail to detect any noindex directives on these pages and can still include the subpages in the index, if they are linked by backlinks.

If you do not want a particular page to be included in the index of search engines, you should therefore only use noindex directives.

Read also how to create backlinks.

Now you know what a noindex directive is, and how it works. Now let’s see the best SEO practices for noindex directives

Chapter 2: 03 SEO best practices for noindex directives.

In this chapter, I will show you the 03 best SEO practices for noindex commands. But before I get into the steps, I’ll show you the reasons why noindex can improve SEO as well as how to use it to increase visibility in search results.

2.1. Why can noindex improve SEO?

At this point, I can clearly imagine that you are wondering how a command that is meant to remove pages from the search engine index can help you optimize your performance on search engines.

Here’s the thing: there are pages that you just don’t want to show up in the search engine SERPs. Do you think a user would want to come across any page on your site? Is it really useful for them? Will they be able to find answers to their queries?

You might think that at least they will land on your site and that some of these people might find what they are looking for, or even become customers.

This reasoning is partly true. But it’s not as easy as it sounds. In fact, what happens if any page on your website ranks higher than the one that best fits the user’s need?

Indeed, relevant pages can be in the index, but not in the SERPs, since Google’s algorithm often tries to avoid placing several pages from the same website in its results. While the less relevant archive pages may finally occupy its place

By noindexing archive pages, tag pages or identical pages, you can prevent these problems from occurring. And better rank your site for the best pages involved.

2.2. how to use NoIndex to increase visibility in search results?

From a case study by Harrison Jones, we can see how using the noindex tag can generate more referral traffic when used properly.

Here’s how Harrison’s experiment went: He noindexed taxonomy pages (as well as category and tag pages) on a domain. Taken in isolation, this was a bad decision, which caused traffic to drop by 20%.

However, after improving the layout (pagination) using rel=next and rel=prev tags, the number of visitors skyrocketed.

On a second site, he tried to improve the pagination without using the noindex directive. This did not result in any noticeable change.

On two other sites, he improved the layout and noindexed the taxonomy pages. This approach has paid off. Traffic increased by 30% on one site and 20% on the second.

Therefore, on its own, the noindex tag can affect the visibility of your website in search results, but if combined with an optimized layout it really increases your visibility in the SERPs.

2.3. The 03 best practices for SEO applied to noindex directives

2.3.1. Do not use noindex on important pages

Accidentally including the noindex tag on an important page can have the effect of removing this page from the search engine indexes and stop receiving any organic traffic.

For example, when you launch a new version of your website, but you forget the “noindex” tags, which were intended to prevent search engines from indexing new variants of the page before it was ready.

Thus, the new version of the website may systematically stop generating traffic on search engines.

2.3.2. Be aware that “noindex” is now treated as “nofollow”

Site owners often make use of meta tags, as well as response headers to tell search engines not to perform indexing of current pages, but to continue crawling page links:

This is generally applied to paginated list pages. As an example, “noindex, follow” can be used on a blog’s archives, so that the archive pages do not appear in search results.

This while allowing search engines to crawl, index, and rank the blog posts themselves

However, this approach may not work as desired, as Google has stated that their system ultimately treats a “noindex, follow” directive as a “noindex, nofollow”. In other words, they will stop link crawling all pages that have a “noindex” tag.

This situation may prevent the indexing of the links’ destination pages or decrease their pagerank, or authority. This decreases their rank in relation to relevant keywords.

2.3.3. Do not use “noindex” rules in Robots.txt files

Google advises against this use of robots.txt files to set “noindex” directives and removed the code that supported these rules in September 2019.

Although they never provided official support, search engines have respected the “noindex” directives contained in robots.txt rules for some time.

Since robots.txt files containing wildcards can be used on multiple pages at once without having to modify the pages themselves, this approach was still favored by many webmasters.

Summary

If you simply don’t want a specific page to be indexed by search engines, you now know what to do. You just have to use the noindex tag on the response header of the concerned pages and it’s done.

In this article, I showed you how to properly use noindex on a page. After that, I had to present the different types of pages that should not appear in search engines and that should be noindexed however.

Noindex and nofollow are two tags that differ from each other, but complement each other. That’s why I took the time to describe to you under which condition they can be combined or used separately.

The noindex tag although it has the particularity of ordering search engines not to index pages can also participate in one way or another in the SEO of your site, the practices leading to such results have also been discussed in this article.

We are at the end of our article. I look forward to your comments.