robots.txt Generator
Build a robots.txt with allow/disallow rules.
This free robots.txt generator builds a valid robots.txt file that tells search-engine crawlers which parts of your site they may or may not access. Set a default allow or refuse policy, a crawl delay and a sitemap, then restrict directories or tune individual search robots — Google, Bing, Yahoo, Baidu, Yandex and more — each to allowed, refused or same-as-default.
Beyond the basics, one click blocks the major AI training crawlers (GPTBot, ClaudeBot, CCBot, Google-Extended and others) or drops in a battle-tested WordPress ruleset. The preview updates live with inline validation. Place the result at the root of your domain, e.g. https://example.com/robots.txt.
Related tools
JSON Formatter
Format, beautify & validate JSON data.
Base64 Encoder / Decoder
Encode and decode Base64 text instantly.
URL Encoder / Decoder
Percent-encode and decode URLs safely.
Regex Tester
Test regular expressions with live match highlighting.
JWT Decoder
Decode JWT header and payload claims.
Timestamp Converter
Convert Unix timestamps to dates and back.