What Is Robots.txt? A Complete Guide to Writing the Perfect Robots.txt File

What Is Robots.txt, Exactly?

A robots.txt file is a plain text file placed at the root of a website (e.g., https://example.com/robots.txt) that tells web crawlers—also called bots or spiders—which parts of the site they are allowed to access and which parts they should skip. It follows the Robots Exclusion Protocol, a standard that has been around since 1994 when Martijn Koster first proposed it on the www-talk mailing list.

The important thing to understand upfront is that robots.txt is a suggestion, not a security mechanism. Well-behaved bots like Googlebot and Bingbot respect these directives. Malicious scrapers or email harvesters generally do not. If you need to actually block access to sensitive content, you should use server-side authentication, not robots.txt.

That said, for SEO purposes, robots.txt is one of the most powerful tools you have. It controls where search engines spend their limited crawl budget—the number of pages a search engine will crawl on your site within a given time frame. For small sites with a few dozen pages, crawl budget is rarely an issue. But once your site grows to thousands or millions of URLs (think e-commerce catalogs, user-generated content, or parameter-heavy filtered pages), directing that crawl budget becomes critical.

How Search Engine Crawlers Use Robots.txt

When Googlebot (or any compliant crawler) visits your domain for the first time, the very first thing it does is request /robots.txt. Before it looks at your homepage, your sitemap, or any of your content, it checks this file to understand the ground rules.

Here is what happens step by step:

1. The crawler sends an HTTP GET request to https://yourdomain.com/robots.txt.
2. If it gets a 200 response, it parses the directives and stores them for that host.
3. If it gets a 404, the crawler assumes everything is open and crawls freely.
4. If it gets a 5xx server error, most crawlers will wait and retry. Google treats this as a temporary "disallow all" and pauses crawling until the file becomes available.

This last point is worth highlighting because it catches people off guard. If your server is misconfigured and returns a 500 error for robots.txt, Google may stop crawling your entire site until it resolves. I have seen this happen during server migrations when the new configuration forgot to handle the robots.txt path properly.

After fetching the file, the crawler caches it. Google typically re-fetches robots.txt about once a day, though this can vary. This means that if you make changes to your robots.txt, there can be a delay of up to 24 hours before crawlers pick up the new rules.

Syntax Breakdown: Directives You Actually Need

The robots.txt syntax is straightforward, but there are a few nuances that matter. Let us go through each directive.

User-agent

This specifies which crawler the rules apply to. The wildcard * targets all crawlers. You can also target specific bots:

User-agent: *
Disallow: /private/

User-agent: Googlebot
Allow: /private/special-page.html

User-agent: Bingbot
Disallow: /temp/

When multiple user-agent groups exist, each crawler looks for the most specific match. Googlebot will follow the Googlebot block, not the wildcard block. If there is no specific block for a given crawler, it falls back to the wildcard rules.

Disallow

Tells crawlers not to access a specific URL path. The path matching is prefix-based:

• Disallow: /admin/ — blocks anything starting with /admin/
• Disallow: /search — blocks /search, /search-results, /search?q=test, etc.
• Disallow: / — blocks the entire site
• Disallow: (empty value) — allows everything (same as not having the rule at all)

Allow

This is particularly useful when you want to disallow a directory but permit specific files within it:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

The Allow directive is officially supported by Google and Bing. It was not part of the original 1994 standard, but it has been widely adopted and was included in Google's formal specification of robots.txt parsing rules published in 2019.

Sitemap

While not technically part of the original robots exclusion protocol, the Sitemap directive is universally recognized by major search engines:

Sitemap: https://example.com/sitemap.xml
Sitemap: https://example.com/sitemap-posts.xml
Sitemap: https://example.com/sitemap-products.xml

You can list multiple sitemaps. This is a convenient way to point crawlers to your sitemap without needing to submit it manually through search console tools. Note that the sitemap URL must be a full, absolute URL.

Wildcards: * and $

Google and Bing support two pattern-matching characters that go beyond the original specification:

• * matches any sequence of characters. Disallow: /*.pdf$ blocks all URLs ending in .pdf.
• $ indicates the end of a URL. Without it, Disallow: /page would block /page, /page2, and /page/subpage. With $, Disallow: /page$ blocks only /page exactly.

A practical example: if your site generates PDF versions of every page and you do not want those indexed:

User-agent: *
Disallow: /*.pdf$
Disallow: /*?print=true

Writing Your First Robots.txt File

If you are starting from scratch, the process is simple. Create a plain text file named robots.txt (all lowercase, no extension tricks) and place it at the root of your domain.

Here is a sensible starting point for most websites:

User-agent: *
Allow: /

Disallow: /api/
Disallow: /admin/
Disallow: /private/
Disallow: /tmp/
Disallow: /*?utm_*
Disallow: /*?ref=*

Sitemap: https://yourdomain.com/sitemap.xml

Let me break down the reasoning:

• Allow: / explicitly allows crawling of the site. This is technically the default behavior, but stating it makes the intent clear.
• /api/ blocks API endpoints. These return JSON or XML data, not human-readable pages, so there is no reason for search engines to index them.
• /admin/ and /private/ block administrative or sensitive areas. Remember, this does not secure them—it just tells crawlers to stay away.
• /*?utm_* and /*?ref=* block URLs with tracking parameters. Without this, search engines might discover hundreds of duplicate pages that only differ by their UTM tags.
• Sitemap points crawlers to your sitemap file so they can discover all your important pages efficiently.

Allow and Disallow: Real-World Examples

Theory is one thing; let me share some scenarios that come up repeatedly in practice.

Scenario 1: E-Commerce Faceted Navigation

Online stores often generate thousands of filtered URLs. A product category page might have filters for color, size, price range, brand, and sorting options. Each combination creates a unique URL, but the content is largely the same.

User-agent: *
Disallow: /*?color=
Disallow: /*?size=
Disallow: /*?sort=
Disallow: /*?price_min=
Disallow: /*&color=
Disallow: /*&size=
Allow: /products/
Allow: /categories/

Sitemap: https://shop.example.com/sitemap.xml

This approach keeps the main product and category pages crawlable while preventing the crawler from getting stuck in a near-infinite loop of filter combinations. I worked on an e-commerce site a few years ago that had over 2 million crawlable filter URLs for just 15,000 products. After adding these rules, their crawl efficiency improved dramatically, and new products started appearing in search results within days instead of weeks.

Scenario 2: Staging and Development Environments

If your staging site is publicly accessible (a bad practice, but it happens), you absolutely need to block all crawlers:

User-agent: *
Disallow: /

Two lines. That is it. But be extremely careful: if you deploy this robots.txt to production by accident during a release, you will de-index your entire site. I have seen this happen more than once, usually during CI/CD pipeline migrations. Always add a check in your deployment process that validates the robots.txt content before pushing to production.

Scenario 3: Blocking Resource-Heavy Crawling

Some crawlers can be aggressive. If your server logs show a specific bot hitting your site too hard:

User-agent: AhrefsBot
Crawl-delay: 10

User-agent: SemrushBot
Crawl-delay: 10

User-agent: *
Allow: /

The Crawl-delay directive asks bots to wait the specified number of seconds between requests. Note that Googlebot does not respect Crawl-delay—you need to set crawl rate in Google Search Console instead. But Bing, Yandex, and many SEO tool bots do honor it.

Platform-Specific Configurations

WordPress

WordPress generates a virtual robots.txt by default, but you can override it with a physical file. Here is what I recommend for most WordPress sites:

User-agent: *
Allow: /wp-content/uploads/
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /wp-includes/
Disallow: /wp-content/plugins/
Disallow: /wp-content/themes/
Disallow: /wp-json/
Disallow: /feed/
Disallow: /comments/feed/
Disallow: /*?s=
Disallow: /*?replytocom=
Disallow: /tag/*/page/
Disallow: /author/

Sitemap: https://yoursite.com/sitemap_index.xml

The /wp-admin/admin-ajax.php allow rule is important because many WordPress themes use AJAX calls for frontend features, and blocking this endpoint can break functionality that search engines need to render your pages properly.

Next.js / React SPAs

Single-page applications present unique challenges. The content is rendered client-side, and many framework-generated routes should not be indexed:

User-agent: *
Allow: /

Disallow: /api/
Disallow: /_next/
Disallow: /admin/
Disallow: /dashboard/
Disallow: /*?preview=
Disallow: /*?draft=

Sitemap: https://yourapp.com/sitemap.xml

For Next.js specifically, blocking /_next/ is debatable. The static assets in this directory are needed for rendering, and Google can typically handle them. However, you definitely do not want the /_next/data/ JSON files indexed as separate pages. A more precise approach:

User-agent: *
Allow: /_next/static/
Disallow: /_next/data/
Disallow: /api/

Sitemap: https://yourapp.com/sitemap.xml

ASP.NET Core / MVC Applications

For .NET applications, you typically want to block API endpoints, authentication routes, and any admin panels:

User-agent: *
Allow: /

Disallow: /api/
Disallow: /Identity/
Disallow: /Admin/
Disallow: /Account/
Disallow: /*?returnUrl=
Disallow: /Error

Sitemap: https://yoursite.com/sitemap.xml

7 Common Mistakes and How to Avoid Them

1. Using robots.txt to hide sensitive content

This is the number one misconception. Blocking a URL in robots.txt does not prevent it from appearing in search results. If other sites link to that URL, Google may still show it in results with the note "A description for this result is not available because of this site's robots.txt." The URL is visible; only the crawling is prevented. Use authentication or noindex meta tags for genuinely sensitive content.

2. Blocking CSS and JavaScript files

In 2015, Google explicitly warned against blocking CSS and JavaScript resources. Modern search engines render pages to understand their content, and if they cannot load your stylesheets or scripts, they see a broken page. This can significantly hurt your rankings.

3. Forgetting the trailing slash

Disallow: /admin blocks /admin, /admin/, /admin/users, and also /administration. If you only want to block the admin directory, use Disallow: /admin/ with the trailing slash.

4. Having multiple robots.txt files

There should be exactly one robots.txt file per origin (protocol + domain + port). Having different robots.txt files for subdomains is fine (blog.example.com/robots.txt and shop.example.com/robots.txt are separate), but do not try to place robots.txt files in subdirectories. Crawlers only look at the root.

5. Deploying the staging robots.txt to production

As mentioned earlier, this can be catastrophic. Add a CI/CD check that verifies robots.txt does not contain Disallow: / for all user-agents in your production deployment pipeline.

6. Not specifying a sitemap

While you can submit sitemaps through Google Search Console and Bing Webmaster Tools, including the Sitemap: directive in robots.txt is free insurance. Any crawler that fetches your robots.txt automatically discovers your sitemap.

7. Ignoring the crawl budget implications

If your site has millions of pages (or generates them dynamically through filters, pagination, or search results), not managing your crawl budget through robots.txt means crawlers may spend their time on low-value pages instead of your important content. This is the most subtle mistake because its effects are gradual—you will not notice it until you realize your new product pages are taking weeks to get indexed.

Testing and Validating Your Robots.txt

After writing your robots.txt, you need to test it. Here are the tools I recommend:

• Google Search Console → robots.txt Tester: This is the most reliable tool for testing how Googlebot specifically will interpret your rules. It lets you enter URLs and see whether they are blocked or allowed.
• Bing Webmaster Tools: Has a similar testing feature for Bingbot.
• Browser check: Simply navigate to https://yourdomain.com/robots.txt in your browser. If you get a 404 or a page that is not plain text, there is a problem.
• Server log analysis: After deploying changes, monitor your server logs for a few days. Check which bots are accessing which URLs and whether your rules are working as intended.

A quick sanity checklist before deploying:

1. Is the file named exactly robots.txt (lowercase)?
2. Is it at the root of your domain?
3. Does it return a 200 HTTP status code?
4. Is the content type text/plain?
5. Is the file encoded in UTF-8?
6. Are there no BOM (Byte Order Mark) characters at the start?
7. Have you tested the critical allow/disallow rules?

Robots.txt vs Meta Robots vs X-Robots-Tag

One of the most common sources of confusion is the relationship between these three mechanisms. They serve different purposes:

The critical gotcha: if you block a page in robots.txt, Google cannot crawl it, which means it cannot see the noindex meta tag on that page. If other sites link to that blocked page, Google might still index the URL (without content). To prevent indexing, you need to allow crawling and use noindex on the page itself.

This is counterintuitive, but it is one of the most important things to understand about technical SEO. Robots.txt and meta robots work together, not as alternatives.

Generate Your Robots.txt with Toolomix

If you prefer not to write your robots.txt from scratch, you can use our Robots.txt Generator to create a properly formatted file in seconds. The tool lets you configure rules visually, add multiple user-agent blocks, and download the result ready to upload to your server.

Pair it with our Sitemap Generator to make sure crawlers can discover all your important pages, and our Meta Tag Analyzer to verify your meta robots tags are configured correctly across your site.

Getting your technical SEO foundation right—robots.txt, sitemap, meta tags—does not require deep expertise. It requires attention to detail and the right tools. Once these pieces are in place, you can focus on what actually drives rankings: creating content that genuinely helps people solve their problems.