In brief: llms.txt is a proposed guide to website content. robots.txt controls crawling preferences; an XML sitemap lists URLs. Decide separately about public search, model training and protected material. None of these files guarantees mentions, visitors or enquiries.
A business publishes useful product information, yet an AI service does not find it or cannot read it. Before adding a file, establish the problem. Is the page public and accessible? Does it answer the question? Is an access rule preventing retrieval? A missing llms.txt does not answer those questions.
For a Vienna B2B business, start with a bounded task: select important documents, establish the intended public access and collect observations you can explain. That does not amount to a promised visibility jump within 48 hours.
What llms.txt is, and what the three files do
The llms.txt v2 proposal was updated on 10 August 2026. It is a proposed format, not a mandatory standard for all AI search providers. A small Markdown file offers context and links; it does not replace accurate, maintained source material.
| File | Purpose and limitation |
|---|---|
| robots.txt | Crawling preferences for relevant bots; not protection for confidential data. |
| XML sitemap | Offer URLs for discovery; no guarantee of indexing or ranking. |
| llms.txt | Optional content guidance for tools that use it; not a general AI ranking requirement. |
An agent may retrieve documents to support a current answer. A maintained guide can help that workflow if the tool actually uses it. Crawling, serving as an answer’s source and training a model are different operations.
Google Search ignores llms.txt, including for its generative features. Introduce it for a concrete need of another tool rather than presenting it as a mandatory Google SEO measure. Google’s generative search guidance.
Which bots serve which purpose?
“Allow or block AI” is too broad a decision. Check each provider’s roles and current guidance before changing rules.
OpenAI: search, training and individual user actions
OAI-SearchBot supports ChatGPT search; GPTBot supports potential model training. Their controls are independent. Blocking the search bot can still leave navigational links. ChatGPT-User handles certain user actions, does not determine Search inclusion, and robots.txt may not apply to those actions. OpenAI crawler documentation.
Anthropic and Perplexity
ClaudeBot gathers potential training content, Claude-SearchBot supports search and Claude-User retrieves at a user’s request. Anthropic describes robots.txt compliance for these bots. Anthropic’s documentation.
PerplexityBot supports search results, rather than foundation model training. Perplexity-User retrieves for users and, according to its provider, generally ignores robots.txt. Perplexity crawlers.
Google and Apple
GoogleOther is a general crawler, including for internal research. Google-Extended controls the described Gemini training and grounding uses; it is not a separate HTTP User-Agent or a Google Search ranking signal. Google’s crawler list.
Applebot crawls. Applebot-Extended controls training use of collected data rather than crawling, and blocking it does not exclude pages from Apple search results. Applebot guidance.
These are separate choices. A bot name in a log also proves neither the sender’s identity nor a later recommendation of your business.
Four preparation steps
1. Select public content and goals
Begin with documents that answer actual customer questions: services, requirements, product details, processes or existing evidence. Record each document’s URL, owner, currency and intended public use. Ten important documents might be a manageable first selection; that is an example, not a requirement.
Keep confidential proposals, personal data and internal calculations out of public guides. An accurate service page is more useful than a collection of unsupported marketing promises.
2. Check access before changing rules
The responsible person checks the HTTP response, redirects, delivered content, robots.txt and relevant CDN or firewall rules. A successful response must also contain the expected material. A login, error document or challenge may interfere with the intended retrieval.
Do not allow every request carrying a plausible User-Agent. When an exception is needed, use the provider’s current verification information and limit it to agreed public access. Disabling the entire security layer is not a condition for appearing in AI search.
Illustrative separate controls, not a complete replacement for an existing robots.txt:
User-agent: OAI-SearchBot
Allow: /
User-agent: GPTBot
Disallow: /This choice permits the named search crawler and signals an exclusion to the named training crawler. It does not secure private documents or automatically account for other providers and existing rules.
3. Prepare a small guide you can maintain
The proposal allows llms.txt at the root or within a section such as /docs/, covering that path. An H1 name, short summary and grouped links make its purpose understandable.
The following navigation example uses existing English pages. It does not claim Markdown endpoints exist or that search providers use the file. An implementation following v2 would need appropriate, maintained Markdown versions behind its links.
# Canem Errant
> Information about marketing services and their assessment.
## Services
- [GEO diagnostics](https://canemerrant.com/en/services/geo-audit/): Scope and process.
## Guides
- [GEO](https://canemerrant.com/en/blog/geo-generative-engine-optimization-guide/): Visibility and measurement.
- [AI visibility](https://canemerrant.com/en/blog/ai-visibility-check-business/): Questions and observations.Some tools additionally use a combined file such as llms-full.txt. Establish a specific need first. More text guarantees neither access nor selection as a source, and duplicate versions must remain current together.
4. Observe content and actual use separately
Recheck links and currency when services or documents change. Record the question, system, date, visible source and, where available, an attributable website visit.
A request is not a citation. A citation is not a click, and a click is not a suitable project. When observations change, assess other explanations such as new content or provider changes too. A sequence in time alone does not establish an effect of llms.txt.
Paid and confidential content
Decide deliberately which information is public and which stays behind a login. Protected documents need authentication, authorisation and appropriately restricted delivery. robots.txt is public and cannot prevent unauthorised access. Google on robots.txt limitations.
Permission to crawl public information is separate from a website visitor’s consent to analytics or advertising. A training preference does not replace legal or technical assessment of actual data use either.
Illustrative scenario: B2B documentation. A manufacturer publishes suitable technical information and contact options. A guide points to current versions, while private customer drawings remain protected. Observe access, possible citations and suitable enquiries without promising a fixed increase.
Illustrative scenario: paid engineering content. A consultancy publishes a summary, keeps complete calculations behind a client login and sets bot preferences separately. This alone provides no evidence of falling or increasing subscriptions.
Three tasks for this week
- Select: name important public documents and the people responsible for maintaining them.
- Agree: record intended search access, training preferences and protected areas separately; the responsible person investigates possible access problems.
- Observe: establish an initial record and a simple observation log. Implement an optional file only for a concrete need.
For the wider assessment, use the GEO guide and AI visibility check.
For the separate assessment of visitor data, see first-party data, Consent Mode, cookie banners and data protection in online advertising. Measurement work has its own purpose and agreed scope.
Frequently Asked Questions
Is llms.txt mandatory now?
No. It is a proposed format. Establish which tool needs it and how you would maintain it. Google Search does not need or use it for visibility or rankings.
Is llms.txt the new robots.txt?
No. llms.txt provides content guidance; robots.txt states crawling preferences for named bots. A guide changes no access permissions and does not replace existing rules.
What happens to ChatGPT search if GPTBot is blocked?
GPTBot and OAI-SearchBot are controlled independently. A GPTBot block concerns the described training use, not automatically the search role. Check OAI-SearchBot and actual access rules separately; search inclusion is still not guaranteed.
Can a file protect images and confidential content?
No. Publicly accessible images are not protected by the file. Confidential information needs effective access protection. Keep it out of public lists and do not rely on llms.txt or robots.txt as authentication.
Must images and videos be included?
Link content that helps a particular tool complete a real task. A maintained text description can be useful. Listing every media file is not a general requirement and does not guarantee use in answers.
Does llms.txt replace SEO or the XML sitemap?
No. An accessible, helpful website and relevant search signals remain separate tasks. An XML sitemap supports URL discovery; llms.txt optionally guides other tools. Neither guarantees indexing.
Where does the file belong, and what about subdomains?
The proposal permits the root or a subpath covering its section. The file is named llms.txt and contains Markdown. You can link relevant external resources; such a link grants no access rights on those hosts. Check the content and rules of each subdomain. A file on the main host does not automatically authorise every host.
Can I limit training while allowing search?
Providers with separate roles allow the respective preferences to be set separately. Check current provider guidance. This is neither a universal block for all models nor a promise that existing training will be reversed.
Establish a useful next step
Our GEO diagnostics can assess public content, access and traceable observations within an agreed scope. An initial enquiry establishes the task; diagnosis, implementation and ongoing measurement are agreed separately. Contact us →
