Generate a clean robots.txt file for Google & SEO in seconds.

What is a robots.txt file?

A robots.txt file is a plain text file that lives at the root of your domain and controls how search engine crawlers navigate your site. It follows the Robots Exclusion Protocol (RFC 9309), the official internet standard for crawler communication. Googlebot checks for this file before crawling any other URL on your domain — making it one of the most powerful (and dangerous) files in your technical SEO toolkit.

The file lives at https://yourdomain.com/robots.txt and contains a series of rule blocks. Each block opens with a User-agent line that targets a specific crawler — or the wildcard * for all bots — followed by Allow and Disallow directives that define exactly which URL paths that bot may or may not visit.

Robots.txt syntax: all key directives explained

Robots.txt uses a compact vocabulary. Understanding each directive prevents the most costly mistakes — including accidentally blocking your entire site or leaking admin paths to the public web.

Path matching is case-sensitive and prefix-based. Disallow: /Photo/ does not block /photo/ — a subtle distinction that causes real indexing problems on Linux-hosted sites. You can also use the wildcard * inside paths and the $ end-of-URL anchor, though support varies by crawler. Consult Google's robots.txt specification for the full matching rules Googlebot applies.

Why robots.txt matters for SEO

Search engines allocate a crawl budget to every domain — roughly the number of URLs they will fetch during a given period. Large e-commerce catalogues and content-heavy sites feel this limit acutely. If Googlebot spends half its budget crawling /wp-admin/, /?s= search result URLs, and endless filtered collection pages, it has less time for your most commercially important content.

"A broken robots.txt file is one of the few technical SEO issues that can completely de-index a site overnight. Treat it with the same care as your sitemap."

The flip side is equally dangerous. An overly aggressive robots.txt that blocks large sections of your site prevents Googlebot from seeing your noindex meta tags — because a blocked page can never be fetched, so the meta tag inside it is never read. This creates a paradox where pages you intend to de-index actually persist in search results. The correct pattern is: allow crawling, control indexing with <meta name="robots" content="noindex"> or the X-Robots-Tag HTTP header.

Platform-specific robots.txt examples

WordPress robots.txt example

WordPress typically serves a virtual robots.txt generated by your SEO plugin. A solid WordPress configuration blocks the admin area, login pages and internal search results while leaving all content, media and REST API endpoints open for discovery. The critical rule most guides miss is Allow: /wp-admin/admin-ajax.php — without it, interactive features and some page builders stop working for authenticated users while bots are watching.

Shopify robots.txt example

Shopify generates a default robots.txt for every store and — since late 2021 — allows merchants to customise it via theme templates. The default ruleset sensibly blocks /admin, /cart, /checkout and /orders. The most common advanced Shopify use case is extending the disallow list to cover faceted navigation URLs (collection filters) that produce thousands of near-duplicate pages and bleed crawl budget.

Next.js robots.txt example

Next.js 13+ has a built-in robots.ts file convention in the app/ directory that generates robots.txt at build time. For older projects, the popular next-sitemap package handles both sitemap and robots.txt generation. Either way, the underlying rules you design with this generator can be directly translated into the configuration your framework expects.

How to block AI crawlers in robots.txt

Since 2023, a new category of bots has changed the robots.txt landscape: AI training crawlers. Companies including OpenAI, Anthropic, Google, Apple and Amazon all deploy bots that collect web content for large language model training. Unlike search crawlers, these bots do not send traffic back to your site — they simply extract content. Many website owners and publishers now actively block them.

Most common robots.txt mistakes that hurt SEO

Most robots.txt problems are not caused by ignorance of the format. They are caused by copy-paste errors, staging configurations being promoted to production, or well-intentioned rules with unintended side effects. Here are the most costly mistakes and how to avoid them.

Robots.txt vs noindex: which to use?

This is one of the most misunderstood distinctions in technical SEO. Robots.txt controls crawling — whether a bot is allowed to fetch a URL. The noindex meta tag or X-Robots-Tag HTTP header controls indexing — whether a fetched page appears in search results.

The critical conflict: if you block a page in robots.txt, Googlebot cannot fetch it, which means it cannot read your noindex tag. The result is that the page may remain indexed indefinitely based on external link signals alone. Google's official guidance recommends allowing crawling and using proper noindex signals for any page you want removed from search results.

How to test your robots.txt file

Paste your file into the Check & Validate mode on this tool for an immediate syntax check. The validator flags missing User-agent lines, malformed Disallow rules, dangerous block-all patterns and missing sitemap lines before anything goes live.

Once you deploy your file, open Google Search Console → Settings → robots.txt report. This shows exactly how Googlebot reads your live file and lets you test specific URLs against your current rules. Monitor Coverage and Crawl Stats reports over the following days to confirm that Googlebot can still access your highest-priority pages.

How to upload and deploy your robots.txt file

Best practices for SEO-friendly robots.txt files

• Keep it small and commented. A concise, well-commented robots.txt is easier to audit during migrations. Add a short comment above each disallow block explaining why it exists.
• Block low-value, high-volume URL groups. Internal search results, pagination beyond page 5, filter combinations and tracking parameter URLs are common candidates for blocking.
• Never block CSS, JS or image assets Google needs to render pages. Blocking theme files or media libraries prevents Google from assessing your page quality accurately.
• Always include up-to-date sitemap URLs. If you run multiple sitemaps (posts, products, images, news), include each one on a separate Sitemap: line.
• Review robots.txt before every site migration. Staging configurations that reach production — even briefly — can trigger a significant drop in crawl coverage.
• Test after every deployment. Use the validation mode above to do a pre-flight check, then confirm with Search Console after each change.
• Decide your AI bot policy intentionally. If you do not want AI companies collecting your content for model training, add explicit rules for GPTBot, CCBot, Google-Extended and other AI agents that are relevant to your situation.

Need an XML sitemap to pair with your robots.txt? Use the free XML Sitemap Generator on WebToolTrix to build and download a clean sitemap in minutes. And if you want to check what meta tags search engines are reading on any of your pages, the Meta Tag Checker gives you a full breakdown without any browser extension.