Noindex in robots.txt Stops Working in September

What Google announced yesterday

Google announced yesterday that from September 1 it will stop following a handful of rules in robots.txt files that it never officially supported.

The one that matters is noindex. Some sites have used a noindex line in robots.txt to keep pages out of Google's results, and Google has quietly honoured it.

From September, it will not. Pages kept out of search only by that line may start appearing in results, including pages that were never meant to be public.

Two months is plenty of time to replace it properly. The work is usually small, once somebody knows the line is there.

What robots.txt does, and does not do

Robots.txt is a short text file at the root of a website that tells search engine crawlers which addresses they should not fetch.

It controls crawling, which is Google visiting a page. It was never designed to control indexing, which is Google listing a page in its results. The two are easy to confuse and behave differently.

On Monday, Google proposed that the robots.txt rules become a formal internet standard, and released the software it uses to interpret them. Retiring the unofficial rules is part of the same tidying.

Where a noindex line is likely to be hiding

• Staging and test copies, kept out of search by a line instead of a password.
• Thank-you pages shown after a form, hidden so they never appear in results.
• Search and filter pages generated by a shop or listings plugin.
• Old campaign pages meant only for visitors arriving from ads.
• Client or member areas protected only by an address nobody was told.
• Sites tuned by a search consultant some years ago.

The first is the one with real consequences. A test copy of your site appearing in results competes with the live one, shows unfinished pages, and sometimes carries old prices or a form that still works.

Checking your own file

Every site's robots.txt can be read by anyone. Type your domain into a browser followed by /robots.txt and the file appears as plain text.

Look for any line beginning Noindex, followed by a path. If there are none, this change does not affect that site.

Check every copy and subdomain separately, since each has its own file. A staging copy at a different address is the most likely place to find one.

Lines beginning Crawl-delay are a different matter. Google has never followed them, but some other search engines do, so leave them alone unless whoever maintains the site says otherwise.

Why a Disallow line is not the replacement

The obvious substitute is a Disallow line, the standard robots.txt rule that tells crawlers not to fetch a path. It does something different.

Disallow stops Google reading the page, not listing it. If other pages link to it, Google can still show the address in results, usually with no description, because it was never allowed to look.

Worse, a page blocked by Disallow cannot be read, so Google never sees a noindex instruction placed on the page itself. The two methods cancel each other out.

To take a page out of results, Google must be allowed to fetch it and find the instruction there.

A worked example

Two years ago, a family law practice with four lawyers had its website redesigned. The new site was built on a test address and kept out of search with a noindex line in that copy's robots.txt.

The test copy was never deleted. It still held the previous fee schedule, two lawyers who had since left, and a contact form that delivered to a former assistant's inbox.

Nobody at the practice knew it existed. From September, it could have started appearing in search results beside the real site.

The live site's own robots.txt held two more noindex lines, for a thank-you page and the registration page of a long-finished seminar.

The test copy was deleted outright, since nobody needed it. The thank-you page was given a robots meta tag, and the seminar page was removed so that it returns the standard not found error.

The robots meta tag

For ordinary pages, the supported method is a robots meta tag: a single line in the page's head section, the part visitors never see, with the value noindex.

Google reads it each time it fetches the page, and drops the page from its results after the next visit. Most SEO plugins add it through a setting on each page.

The tag only works if the page can be fetched. Check that robots.txt does not also block it, for the reason above.

Files that are not pages

A PDF, an image or a spreadsheet has no head section to put a tag in.

For those, the same instruction can be sent as an X-Robots-Tag header: a line the server adds when it delivers the file, set in the server's configuration rather than in the file itself.

This is the method for old price lists, brochures and internal documents that were uploaded to the site and found their way into results. It needs somebody with access to the server's settings.

Passwords, deletion and the removals tool

Anything that should never be public, such as a staging copy or a client area, belongs behind a password. A password keeps out people as well as search engines, which a robots rule never did.

A page that no longer serves any purpose can simply be removed. Once it returns a not found or gone response, Google drops it after visiting again.

Search Console's removal tool hides a page from results quickly, but only temporarily. It is for emergencies, while one of the permanent methods takes effect.

The counter-case

Most small business sites have never had a noindex line in robots.txt, and for them this announcement changes nothing.

Where a line exists, the page behind it may not matter much. An old thank-you page appearing in results is untidy rather than harmful.

But the cases that do matter, test copies and private areas, are exactly the ones nobody remembers, which is why the file is worth five minutes before September.

Check the file, and replace any noindex line with a method Google supports.

Before September

1. Open your robots.txt in a browser.
2. Look for noindex lines and note each path.
3. Check test copies and subdomains too.
4. Delete or password-protect anything not meant to be public.
5. Add a robots meta tag to pages that stay.
6. Use the header for PDFs and documents.
7. Remove the old lines once replaced.

Step three is the one most likely to find something serious. The live site is usually looked after; the forgotten copy from the last redesign is not.

How Google reads the pages it is allowed to fetch is covered in Googlebot now sees pages like current Chrome.

Frequently asked questions

West Coast Media Solutions Inc. provides web design, web development, hosting, digital marketing, and business consulting to organisations across Canada, drawing on more than twenty-five years in the field.