Robots.txt Generator

Create and optimize your robots.txt file to guide search engine crawlers. Prevent indexing of sensitive areas and highlight your sitemap.

Global Configuration

Sitemap URL

Helps crawlers find your content faster.

User Agent

Disallow Paths

Allow Paths

Preview

User-agent: *
Disallow: /admin
Disallow: /api
Allow: /

Content Validated

Important Tips

Robots.txt is publicly visible. Don't hide sensitive file names here.
The GPTBot rule is useful if you want to prevent AI models from scraping your site.
Always place the robots.txt in your root directory: /robots.txt.