SEO Tools
Robots.txt Generator
Build a clean robots.txt file for common crawler rules.
robots.txt output will appear here.
Use one Allow or Disallow path per line.
What robots.txt controls
robots.txt provides crawl directives to compatible search-engine bots. It does not provide authentication or hide private content.
Crawl-delay support varies
Some crawlers ignore Crawl-delay. Use it only when you know the target crawler supports it.