🤖

Generate robots.txt files to control search engine crawlers and improve your site's SEO performance.

Domain/Website URL Enter your website's domain or full URL

User Agents

Disallow Paths

Paths you want to block from search engines

Allow Paths

Specific paths you want to allow (optional)

Sitemap URL URL to your XML sitemap (optional)

Crawl Delay (seconds) Delay between crawler requests in seconds (optional)

📄

Copy the generated robots.txt content to your website's root directory

💡

Best practices for creating effective robots.txt files

Consider blocking: /admin, /private, /temp, /cgi-bin, /wp-admin, /wp-content/uploads

Use "*" for all crawlers, or specify individual bots like "Googlebot", "Bingbot"

Always include your sitemap URL to help search engines find your content

Place robots.txt in your website's root directory (e.g., https://example.com/robots.txt)