Does a sitemap guarantee indexing?

No. It helps discovery. Indexing still depends on the page being crawlable, canonical and worth including.

Should every page be in the sitemap?

Only canonical, indexable pages. Redirects, noindex pages and duplicates should be left out.

Where should robots.txt live?

At the root of the domain, at /robots.txt. Crawlers do not look anywhere else.

Sitemaps and robots.txt: getting crawling right

2 min read · updated 2026-08-22

These two small files decide what a crawler is allowed to fetch and what it is invited to fetch. They are trivial to write and remarkably easy to get wrong, and a single mistaken line in either can keep a perfectly good site out of the index for months.

Audit crawling and indexing

Different jobs, often confused

robots.txt controls access: it tells crawlers which paths not to request. sitemap.xml controls discovery: it lists the URLs you want found and, optionally, when each last changed. Blocking a URL in robots.txt does not remove it from the index, and listing a URL in a sitemap does not guarantee it is indexed.

noindex versus disallow

To keep a page out of search results, allow the crawl and serve a noindex tag. If you disallow the path instead, the crawler never fetches the page, never sees the tag, and the URL can still appear in results with no description. Use disallow for infinite parameter space and private endpoints, noindex for pages that exist but should not rank.

Disallow — search endpoints, faceted parameter explosions, internal APIs.
Noindex — thin utility pages, duplicate print views, internal dashboards.
Canonical — near-duplicates that should consolidate into one URL.

A sitemap that stays correct

Generate the sitemap from the same source that renders the pages. A hand-maintained file drifts the moment someone publishes an article, and a drifting sitemap teaches the crawler to trust it less. Include only canonical, indexable URLs, and give lastmod a real content timestamp or leave it out entirely.

Verifying instead of assuming

Fetch both files as a crawler would and read the output. Then check that a sample of URLs in the sitemap returns 200, carries a self-referencing canonical and is not blocked by any rule. Verification takes five minutes and catches the whole class of invisible indexing failures.

Frequently asked

Does a sitemap guarantee indexing?: No. It helps discovery. Indexing still depends on the page being crawlable, canonical and worth including.
Should every page be in the sitemap?: Only canonical, indexable pages. Redirects, noindex pages and duplicates should be left out.
Where should robots.txt live?: At the root of the domain, at /robots.txt. Crawlers do not look anywhere else.