Use robots.txt to control crawling, noindex to control indexing, and canonical tags to tell search engines which of several similar URLs is the main one. They are not interchangeable. robots.txt stops a page from being fetched but not from being indexed if other pages link to it. noindex removes a page from search, but only if crawlers can fetch the page to see the tag. A canonical is a strong hint, not an instruction, and does nothing to stop crawling. Picking the wrong one is behind many indexing problems.
The three tools compared
| robots.txt Disallow | noindex | rel=”canonical” | |
|---|---|---|---|
| Controls | Crawling | Indexing | Which duplicate is shown |
| Crawler must fetch the page? | No, it is blocked before fetching | Yes, to see the tag | Yes |
| Can the URL still appear in search? | Yes, without a snippet, if linked from elsewhere | No, once processed | Usually the canonical appears instead |
| Strength | Directive for compliant crawlers | Directive | Hint; Google may choose another URL |
| Passes signals to another URL? | No | No | Yes, consolidates to the canonical |
| Works for PDFs and images? | Yes | Yes, with the X-Robots-Tag header | Yes, with a Link header |
Which one to use
| Situation | Use |
|---|---|
| Internal search results, filter and sort parameters you never want crawled | robots.txt Disallow |
| Thank-you pages, login pages, thin tag pages that must exist for users | noindex |
| The same product reachable through several URLs | canonical to the main URL |
| Printer-friendly or tracking-parameter versions of a page | canonical |
| A page permanently moved | 301 redirect, not any of the three |
| A page deleted with no replacement | 404 or 410 status |
| Staging and development sites | Password protection; noindex and robots.txt as backups |
| Private or confidential content | Authentication. None of the three protects content. |
| Already indexed pages you want removed | noindex first; block in robots.txt only after they drop out |
The mistakes that cause most problems
Blocking a noindexed page in robots.txt
If robots.txt blocks a page, crawlers never see its noindex tag, so the page can stay indexed. To remove pages, allow crawling until the noindex has been processed.
Using robots.txt to hide a page
A disallowed URL can still be indexed if other sites link to it, and it shows in results without a description. robots.txt is also public, so it advertises the paths you are trying to hide.
Canonical to a different page
Canonicalising page 2 of a list to page 1, or every product variant to the category, tells Google to ignore content that is not a duplicate. Canonicals should point only to true equivalents.
Conflicting signals
A page with a canonical to itself, listed in the sitemap, but carrying noindex, sends mixed messages. Keep sitemaps limited to indexable, canonical URLs.
Staging noindex going live
Environment settings that add noindex on staging sometimes reach production. Test for it on every deployment; see catching SEO regressions before deployment.
AI crawlers
AI search crawlers from OpenAI, Anthropic and Perplexity say they follow robots.txt. Their handling of noindex is less documented, and some user-triggered fetchers, such as Perplexity-User, state that they generally do not follow robots.txt. Content that must not be read by any system belongs behind a login.
Frequently asked questions
Is noindex in robots.txt supported?
No. Google stopped supporting noindex rules inside robots.txt in 2019. Use the meta tag or the X-Robots-Tag header.
How fast does noindex take effect?
After the page is recrawled. For urgent cases, Search Console’s Removals tool hides a URL for about six months while the noindex is processed.
Should every page have a canonical tag?
Yes. A self-referencing canonical on every indexable page protects against parameter and tracking-code duplicates.
Sources
- Google Search Central, “Introduction to robots.txt” and “Block Search indexing with noindex”
- Google Search Central, “How to specify a canonical URL”
- Google Search Central blog, “A note on unsupported rules in robots.txt”, July 2019
- Perplexity, “Perplexity crawlers”