AI.txt Checker
Check if a website provides AI crawler rules and analyze its AI.txt file for completeness and best practices.
What is AI.txt?
AI.txt is a proposed standard that helps website owners communicate their AI usage policies to AI crawlers, large language models, and other AI systems. Similar to how robots.txt tells search engine crawlers what they can access, AI.txt specifically addresses AI-related activities like training, summarization, and indexing. By placing an ai.txt file at the root of your domain, you make your content policies machine-readable for AI agents.
Why Should Websites Create AI.txt?
Control AI Usage
Specify whether AI systems can crawl, train on, or summarize your content. Set clear boundaries for how your data is used.
Agent Management
Allow or block specific AI agents like OpenAI, Anthropic, or Google Gemini from accessing your content.
Future-Proofing
As more AI crawlers emerge, having an AI.txt ensures your policies are discoverable without updating multiple systems.
Complement robots.txt
While robots.txt handles search crawlers, AI.txt specifically addresses AI-related use cases that robots.txt wasn't designed for.
How Do AI Crawlers Use AI.txt?
AI crawlers and language models check for AI.txt at /ai.txt or /.well-known/ai.txt when visiting a domain. The file tells them what types of AI activities are permitted and which agents are allowed or blocked. This is increasingly important as AI companies develop responsible crawling practices.
AI.txt Example
site: example.com name: Example Website ai_crawling: allowed ai_training: disallowed ai_summarization: allowed allowed_agents: openai anthropic google-gemini blocked_agents: unknown-ai contact: https://example.com/contact sitemap: https://example.com/sitemap.xml
Also check your LLM.txt
LLM.txt is a complementary standard focused on providing context to large language models. Use our LLM.txt Checker to analyze your LLM.txt file.