Robots.txt Generator
Optimize your site's SEO by guiding search engine crawlers.
Local-first
Optional
20 chars·3 words·3 lines
Drop file or paste textrobots.txt
User-agent: *
Allow: /
Disallow: /admin
Disallow: /api
Disallow: /private
74 chars·10 words·6 lines
Ready to copyHow this tool works
What it accepts, how to use it, and where your input is processed.
Show GuideHide Guide
How this tool works
What it accepts, how to use it, and where your input is processed.
Show GuideHide Guide
SEO Optimization
Guide search engine crawlers like Googlebot to index the right parts of your site, while protecting sensitive areas from exposure.
Crawler Efficiency
Save crawl budget by instructing bots to ignore non-essential paths or redundant parameters that don't need indexing.
Need an extra feature or found an edge case?Send Feedback
Related tools
Open after you finish the primary task.
ShowHide
Related tools
Open after you finish the primary task.
Meta Tag Generator
WebGenerate SEO-friendly meta tags for your website to improve search engine rankings.
Open tool
Curl to Fetch Converter
WebConvert standard curl commands into modern JavaScript Fetch API syntax instantly.
Open tool
URL Component Parser
WebBreak down any URL into its protocol, host, path, query parameters, and hash components.
Open tool