Presets · AI crawlers · tested before you upload
Robots.txt generator
Pick a starting point, list what crawlers should stay out of, and get a robots.txt you can copy, test and upload.
Your robots.txt appears here
Choose the options and press Make robots.txt.
Questions about making a robots.txt
How do I create a robots.txt file?
Choose the options above, copy the result into a plain text file named robots.txt, and upload it to the root of your site so that it opens at https://yourdomain.com/robots.txt. Then run it through the checker to see what each crawler may reach.
What should a basic robots.txt contain?
For most sites, very little: a User-agent: * group that allows everything, the private areas you do not want crawled, and a Sitemap line with the full URL of your sitemap. An empty or missing file also allows everything.
How do I edit robots.txt on Shopify or WordPress?
WordPress serves a virtual robots.txt that SEO plugins such as Yoast and Rank Math let you edit, or you can upload a real file to the site root, which takes precedence. Shopify generates the file itself; to change it, add a robots.txt.liquid template to the theme.
Does blocking AI crawlers remove my site from ChatGPT or Perplexity?
Blocking the training crawlers does not. ChatGPT search uses OAI-SearchBot and Perplexity uses PerplexityBot, and the generator leaves both allowed. Block those two as well only if you do not want to be cited in their answers.