Make a robots.txt file and decide which bots get in.

Pick a starting point, add your sitemap, choose which paths and AI crawlers to keep out, then test any page against the result. The file is made in your browser and nothing is uploaded.

  • Free, no sign-up
  • Made in your browser
  • Test any URL

Start from

Your site

Rules for all crawlers

Paths to keep crawlers out of, and exceptions inside them. Leave empty to allow everything.

Bing and some others honour crawl-delay. Google ignores it.

AI crawlers

Choose what to keep out. These are requests: well-behaved bots follow them, others may not.

Choose crawlers one by one
AI training and data collection
AI search and assistants

Blocking these can remove your pages from AI answers and citations.

robots.txt

          
Save it as robots.txt in the root of your site, so it opens at https://example.com/robots.txt.

          
    
          
    Test a page against this file
    Test a robots.txt you already have
    While this box has text, the test above checks your pasted file instead of the one made here.
    
        
    
      
    
      
    
    
    
  
    
    
    How to create a robots.txt file
    
      
    1. Pick a starting point
      Choose Allow everything, a CMS preset or Block everything, then add your site address and sitemap.
    2. Set the rules and the bots
      List paths to keep crawlers out of, and tick the AI crawlers you want to block. Read the checks for mistakes.
    3. Test, then upload
      Try a few pages in the tester. Download the file and put it at the root of your site, at /robots.txt.
    
    
    A robots.txt file tells crawlers which parts of a site they may fetch. Search engines read it before they crawl, and a growing number of AI companies publish crawler names you can allow or block in the same file. This generator writes the file for you, explains the risky choices, and lets you test a page against it before you upload anything.
    What it does
    • Starts from sensible presets for WordPress, Shopify and online stores, or from allow-all or block-all.
    • Blocks AI training and data collection crawlers such as GPTBot, ClaudeBot, Google-Extended, CCBot and Applebot-Extended in one click, or one by one.
    • Keeps AI search and assistant bots in their own list, so you can stay visible in AI answers while opting out of training.
    • Adds Sitemap lines and an optional crawl-delay.
    • Tests any path against the file for Googlebot, Bingbot, GPTBot and others, using the matching rules in the robots.txt standard, and tells you which line decided the answer.
    • Warns about mistakes that cost traffic: blocking the whole site, blocking CSS and JavaScript, or relative sitemap addresses.
    Limits, honestly
    • robots.txt is a request, not a lock. Compliant crawlers follow it. A bot that ignores it can still fetch your pages, so protect private content with a login.
    • Blocking a page in robots.txt stops crawling, not indexing. A blocked page can still appear in search results if other sites link to it. Use a noindex tag on a crawlable page to keep it out.
    • Crawler names change. The AI bot list reflects names that are published today, so check each company’s documentation from time to time.
    • Google ignores crawl-delay. It is honoured by Bing and some other crawlers.
    • Shopify builds its own robots.txt from a theme template. The preset here is a starting point for that template, not a replacement.
    
    
    Robots.txt Generator: questions and answers
    
      
    What is a robots.txt file?
    It is a plain text file at the root of your site, at /robots.txt, that tells crawlers which paths they may fetch. Search engines read it before crawling. It is public and every visitor can open it.
    How do I block AI crawlers like GPTBot and ClaudeBot?
    Add a User-agent line for each bot with Disallow: / under it. This tool does it with one button for the main AI training crawlers, and lets you pick bots individually. Bots that follow the standard will skip your site. Those that ignore it will not be stopped.
    Will blocking AI bots hurt my SEO?
    Blocking AI training crawlers does not affect your Google or Bing rankings. Blocking AI search bots, such as OAI-SearchBot or PerplexityBot, can remove your pages from AI answers. Google-Extended is a control token for Gemini training and does not affect Search.
    Where do I put the robots.txt file?
    In the root of your domain so it opens at https://yourdomain.com/robots.txt. Each subdomain needs its own file. On WordPress you can upload it by FTP or use an SEO plugin. On Shopify you edit the robots.txt.liquid template.
    Does Disallow remove a page from Google?
    No. It only stops crawling. A blocked page can still be indexed from links elsewhere, without a description. To keep a page out of search, leave it crawlable and add a noindex tag, or password-protect it.
    What does User-agent: * mean?
    The asterisk means all crawlers that have no group of their own. A bot that has its own group, such as GPTBot, follows that group and ignores the rules under the asterisk.
    Should I add my sitemap to robots.txt?
    Yes. A Sitemap line with the full address helps every crawler find your pages. You can list more than one.
    Is my data sent anywhere?
    No. The file is generated in your browser and nothing you type is uploaded.
    Can I test the robots.txt I already have?
    Yes. Open “Test a robots.txt you already have” under the tester, paste your current file, then type a path or a full page address and pick a crawler. It tells you whether that page is allowed or blocked and which rule decided it, using the same longest-match rule Google uses. Nothing is fetched or sent.
    
    
    More free tools
    
      
    llms.txt GeneratorDescribe your site for AI assistantsOpen tool →JSON-LD Schema GeneratorFAQ, product, article and moreOpen tool →Website Launch ChecklistGo-live checks, including SEOOpen tool →
    
    

    Available for full-time and freelance
    Need a custom tool, site or store?
    I scope, design and build tools, websites and Shopify stores for teams in India and abroad. Send a short brief and I will reply within 48 hours.
    Book a 20-min callPick a slot that suits youhello@aashishpandey.com
    Send a briefRésuméLinkedInWhatsApp