Robots.txt and XML Sitemap: How to Get Your Pages Indexed Faster

Getting indexed starts with discovery. Two plain-text files tell search engines where to look and what to skip: robots.txt and your XML sitemap. They sound technical, but each can be created in a minute — and mistakes in either can quietly keep your best pages out of Google.

Crawling vs indexing: the key difference

Crawling is when a bot fetches your page. Indexing is when the search engine stores and may show it. Robots.txt influences crawling only. A sitemap is a hint about which URLs you consider important. Neither guarantees a page will be indexed — quality, duplication and the page's own directives decide that.

What robots.txt does

Robots.txt lives at the root of your domain (https://example.com/robots.txt) and gives crawl instructions to bots. A simple, safe example:

User-agent: *
Disallow: /admin/
Disallow: /cart/
Allow: /

Sitemap: https://example.com/sitemap.xml

Build one without memorising syntax using the Robots.txt Generator, which lets you set rules for major crawlers and download the file.

Robots.txt mistakes that cost traffic

• Disallow: / left on after launch. This blocks the whole site. It is the most damaging single line you can leave from a staging environment.
• Blocking CSS and JavaScript. Google needs them to render pages; blocking them can hurt how your pages are understood.
• Using robots.txt to hide a page from search. A blocked page can still be indexed from external links. Use noindex instead, and keep the page crawlable.
• Wrong location. The file must be at the domain root, not in a subfolder.

What an XML sitemap does

An XML sitemap lists the URLs you want search engines to find, optionally with a last-modified date. It is especially helpful for new sites, large sites, and pages with few internal links.

Sitemap rules of thumb

• Include only canonical, indexable URLs that return 200 — no redirects, 404s, noindex or blocked pages.
• Keep each file under 50,000 URLs and 50 MB uncompressed; split larger sites with a sitemap index.
• Use accurate lastmod dates. Google ignores priority and changefreq.
• Use the same hostname and protocol (https, with or without www) as your canonical URLs.

Create one from a list of URLs with the XML Sitemap Generator, then run it through the Sitemap Validator to catch errors before you submit.

How to submit your sitemap

1. Upload sitemap.xml to your site root.
2. Add Sitemap: https://example.com/sitemap.xml to robots.txt.
3. In Google Search Console, open Sitemaps, enter the sitemap URL and click Submit.
4. Repeat in Bing Webmaster Tools if you want Bing coverage.

Why is my page not indexed? Common statuses

To check a handful of URLs quickly, use the Google Index Checker, and test redirects with the Redirect Checker. Then use URL Inspection → Request indexing in Search Console for your most important pages.

Five ways to get indexed faster

• Link new pages from your homepage or a strong hub page.
• Keep the sitemap current and free of dead URLs.
• Publish unique, useful content — Google indexes selectively.
• Fix slow or error-prone servers so crawling is not throttled.
• Earn a few quality external links to new pages.

Part of a bigger technical review? See our free SEO audit checklist.

Robots.txt vs noindex vs canonical

These three are often confused, but they solve different problems:

Never combine a robots.txt block with a noindex tag on the same page: if the page is blocked, Google cannot crawl it to see the noindex instruction.

Do you need to worry about crawl budget?

For most small and medium sites, no. Crawl budget matters mainly for very large sites with hundreds of thousands of URLs or heavy parameter-generated duplicates. The practical advice is the same for everyone: keep the site fast, avoid endless URL variations, fix redirect chains and keep the sitemap clean so crawlers spend their time on pages you care about.