Flash deal ends in--:--:--— use GEO10 for 10% off

LLM InfrastructureFree · runs in your browser

Opinionated policy → full ruleset

AICrawlRule Generator

Pick an opinionated policy and emit matching robots.txt and llms.txt rules for nine AI crawlers including GPTBot, ClaudeBot, PerplexityBot, Google-Extended and Applebot-Extended. Why this matters →

Paths blocked universally

Bot policies

OpenAI training (GPTBot)

Opts out of training. Does not affect live ChatGPT answers.

Allow

OpenAI live (ChatGPT-User, OAI-SearchBot)

Blocks live browse and SearchGPT. Usually you want these ALLOWED.

Allow

Anthropic training (ClaudeBot)

Opts out of Claude model training.

Allow

Anthropic live (Claude-Web)

Blocks live Claude browse. Usually ALLOWED.

Allow

Perplexity (PerplexityBot, Perplexity-User)

PerplexityBot is training, Perplexity-User is live queries.

Allow

Google-Extended (Gemini training)

Opts out of Gemini training without affecting Google search.

Allow

Applebot-Extended (Apple Intelligence training)

Opts out of Apple AI training without affecting Siri/Spotlight.

Allow

Common Crawl (CCBot)

Large public web archive. Used by most LLM training sets.

Block

ByteDance (Bytespider)

Aggressive scraper. Most operators block it.

Block
User-agent: *
Disallow: /admin/
Disallow: /cart/
Disallow: /checkout/
Disallow: /account/
Disallow: /api/

User-agent: GPTBot
Allow: /
Disallow: /admin/
Disallow: /cart/
Disallow: /checkout/
Disallow: /account/
Disallow: /api/

User-agent: ChatGPT-User
Allow: /
Disallow: /admin/
Disallow: /cart/
Disallow: /checkout/
Disallow: /account/
Disallow: /api/

User-agent: OAI-SearchBot
Allow: /
Disallow: /admin/
Disallow: /cart/
Disallow: /checkout/
Disallow: /account/
Disallow: /api/

User-agent: ClaudeBot
Allow: /
Disallow: /admin/
Disallow: /cart/
Disallow: /checkout/
Disallow: /account/
Disallow: /api/

User-agent: Claude-Web
Allow: /
Disallow: /admin/
Disallow: /cart/
Disallow: /checkout/
Disallow: /account/
Disallow: /api/

User-agent: PerplexityBot
Allow: /
Disallow: /admin/
Disallow: /cart/
Disallow: /checkout/
Disallow: /account/
Disallow: /api/

User-agent: Perplexity-User
Allow: /
Disallow: /admin/
Disallow: /cart/
Disallow: /checkout/
Disallow: /account/
Disallow: /api/

User-agent: Google-Extended
Allow: /
Disallow: /admin/
Disallow: /cart/
Disallow: /checkout/
Disallow: /account/
Disallow: /api/

User-agent: Applebot-Extended
Allow: /
Disallow: /admin/
Disallow: /cart/
Disallow: /checkout/
Disallow: /account/
Disallow: /api/

User-agent: CCBot
Disallow: /

User-agent: Bytespider
Disallow: /

How it works

Pick a policy for each bot group and list any paths you want blocked universally. The generator emits a robots.txt and matching llms.txt ## Bot policy block with the right User-agent rules for GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-Web, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, CCBot and Bytespider.

Sensible defaults

  • Allow ChatGPT-User and Perplexity-User. These represent real user queries and blocking them removes you from live answers.
  • Block CCBot and Bytespider by default. Common Crawl is the main training corpus for open models and Bytespider is aggressive with few benefits.
  • Universal blocks for cart, admin, account. These should be Disallowed for every user-agent regardless of policy.

Pair with

Once generated, simulate access with the AI Content Access Simulator, validate syntax with the LLMs.txt Validator, and cross-check against your sitemap with the Sitemap vs LLMs.txt Consistency Checker. Strategy reading: what is llms.txt.

Related tools

LLM Infrastructure

AI Content Access Simulator

Simulate exactly which URLs GPTBot, ChatGPT-User, ClaudeBot, Claude-Web, PerplexityBot, Google-Extended, Applebot-Extended, CCBot and Bytespider can reach on your site.

LLM Infrastructure

LLMs.txt Generator

Generate a standards-compliant llms.txt file for your site in seconds. Includes the standard header, policy blocks for GPTBot, ClaudeBot, Google-Extended and PerplexityBot, and a copy-paste output ready to drop at your domain root.

LLM Infrastructure

Sitemap vs LLMs.txt

Cross-check your sitemap.xml and llms.txt for contradictions, missing primary sources and phantom URLs so AI crawlers receive consistent signals.

LLM Infrastructure

LLMs.txt Validator

Paste an existing llms.txt file and we'll validate the syntax, flag unknown directives, warn about common mistakes, and tell you whether GPTBot, ClaudeBot and the other major AI crawlers will parse it correctly.

From the Knowledge Hub

Tutorials

What Is llms.txt?

llms.txt is the robots.txt of the AI era - a single file at the root of your domain that tells LLMs what to read, what to ignore, and how to attribute your content. Here's the exact spec, a working template, and the GEO upside.

AI Search

How AI Assistants Choose Citations

Reverse-engineered: the exact ranking signals ChatGPT, Claude, Gemini and Perplexity use to decide which sources end up in the citation chips at the bottom of an answer - and how to engineer your content to win them.

Want this done for you?

Ship the full GEO playbook in 14 days

Geolify GEO packages bundle every tool on this site into a 14-day done-for-you build - llms.txt, schema, entity strength, content overhaul, citations and the measurement stack. From $499.

Buy GEO PackagesSee Pricing← All tools

Explore More Packages

Combine services for maximum AI visibility.

GEO Packages

Keyword-targeted AI visibility across ChatGPT, Claude, Gemini & more.

From $499Learn more

Local GEO

City-level AI visibility for local service businesses.

From $199Learn more

AI SEO Boost

Authority links & guest posts to amplify your GEO results.

From $39Learn more
Compare All Pricing