Robots.txt Generator
Generate robots.txt from domain, user-agents, allow/disallow paths, sitemap, and crawl-delay—built on Toolomix servers.
Practical tips
Start with a wildcard agent
A rule for the * user agent applies to crawlers that do not have a more specific block.
List your sitemap
Including a Sitemap URL helps crawlers discover your XML sitemap from robots.txt.
Publish and verify
After upload, open https://yoursite.com/robots.txt to confirm the live file matches what you generated.
Multiple user agents
Add separate agents such as * , Googlebot, or Bingbot with their own path rules.
Allow and disallow paths
Declare which paths each agent should or should not request.
Sitemap and crawl-delay
Optionally add a Sitemap line and a crawl-delay between 0 and 3600 seconds.
Copy or download
Copy the generated text or download a robots.txt file to place at your site root.
What this tool does
Robots.txt Generator turns form fields—domain, user agents, disallow/allow paths, sitemap, and optional crawl-delay—into a downloadable robots.txt body.
The output is plain-text directives for crawlers. You still must host the file on your own origin; Toolomix does not publish it for you.
How to use
Enter your domain
Provide the required website domain or full URL.
Add agents and paths
List user agents and the disallow/allow paths for each.
Optional sitemap and crawl-delay
Add a Sitemap URL and a crawl-delay between 0 and 3600 seconds if needed.
Generate, then copy or download
Submit the form, review the generated text, then copy it or download the file to your site root.
Supported inputs and output
Input: domain (required), userAgents, disallow paths, allow paths, sitemap URL, crawlDelay (0–3600). Posted to /seo-tools/GenerateRobotsTxt.
Output: generated robots.txt plain text you can copy or download—not an automatic submission to search engines.
Processing and privacy
Generation runs on Toolomix servers. Your form values are sent over HTTPS to build the file content.
See the Privacy Policy for how Toolomix handles requests and logs. This tool does not register your site with Google or other crawlers.
Limitations
Known limits
- Produces text only—you must upload robots.txt to your own host
- Does not submit or verify the file with Google Search Console
- Crawl-delay is ignored by some major crawlers
- Robots.txt is not a security boundary
- Domain field is required for generation
Troubleshooting
If generation fails, confirm the domain field is filled and try again. Check that path entries use site-relative paths (for example /admin).
If crawlers still visit blocked URLs, remember robots.txt is advisory—use authentication or server rules for true access control, and verify the live file at /robots.txt.
Frequently asked questions
Where is robots.txt generated?
On Toolomix servers. The form posts to /seo-tools/GenerateRobotsTxt; the response text is assembled server-side (StringBuilder), not built only in the browser.
Is a domain required?
Yes. The domain/website URL field is required before generation.
Does this submit my robots.txt to Google?
No. The tool only produces robots.txt text for you to host. It does not ping or upload to Google Search Console or any crawler.
Where do I put the file?
Save it as robots.txt in your website root so it is reachable at https://yoursite.com/robots.txt.
What is the crawl-delay range?
The UI accepts crawl-delay from 0 to 3600 seconds. Some crawlers, including Googlebot, ignore crawl-delay.
Does robots.txt enforce access control?
No. It is a request that well-behaved crawlers usually respect; it is not authentication or a firewall.
Can I define different rules per crawler?
Yes. Add multiple user agents, each with its own allow and disallow paths.
Explore more free tools
Use Robots.txt Generator
Enter your domain and rules, generate robots.txt on Toolomix servers, then copy or download the file.
