Before publishing a website, most teams check the visible things first: the headline, the hero image, the form, the navigation, and the mobile layout. Technical SEO usually comes next: title tags, meta descriptions, canonicals, redirects, schema, and broken links.
Now there is another layer worth checking before launch: whether the site is easy for AI-oriented tools, crawlers, and answer systems to understand.
That does not mean chasing a new magic file or promising that a page will be cited by an AI answer engine. It means checking the boring infrastructure that decides whether a page can be crawled, interpreted, summarized, and connected to the right entity.
Two files sit near the center of that review: `robots.txt` and `llms.txt`.
`robots.txt` controls crawler access. `llms.txt` is an emerging convention for giving AI tools a concise, Markdown-based guide to the most important parts of a website. They are not the same thing, and neither one replaces good content, structured data, or clean technical SEO.
What llms.txt is
`llms.txt` is a text file placed at the root of a domain, usually at a URL like:
```txt https://example.com/llms.txt ```
The idea is simple: give large language models and AI agents a short, readable map of the website. A useful file usually explains what the site is, what it offers, and which pages matter most.
A basic `llms.txt` file might include:
```md # Example Company
Example Company builds project management software for small engineering teams.
Key pages
- [Product overview](https://example.com/product)
- [Pricing](https://example.com/pricing)
- [Documentation](https://example.com/docs)
- [Security](https://example.com/security)
- [Support](https://example.com/support)
```
That is the right mental model: not a sitemap dump, not a hidden SEO trick, and not a replacement for crawlable HTML. It is a curated orientation layer.
For a SaaS site, documentation hub, product catalog, agency site, publication, or support-heavy website, that orientation layer can be useful. It tells a machine which parts of the site are central and which pages carry the best explanation of the brand, product, or topic.
What llms.txt is not
The most important thing to understand about `llms.txt` is what it cannot do.
It does not force AI systems to crawl your website. It does not guarantee inclusion in AI answers. It does not override `robots.txt`. It does not fix thin content, vague positioning, missing schema, broken canonicals, or blocked resources.
Treat it as a clarity file, not as a ranking file.
That distinction matters because AI search and answer visibility still depends on the same fundamentals that have always made pages easier to trust and retrieve: crawl access, indexability, helpful content, internal links, structured data, entity consistency, and a clean technical surface.
Why check AI crawler access before launch
Pre-launch reviews often happen under pressure. A landing page is approved. A migration is ready. A client wants the new site live. A product team has a launch date.
That is exactly when small technical mistakes become expensive.
A staging rule can survive into production. A `noindex` tag can be left behind. A canonical can point to the old URL. `robots.txt` can block a directory that now contains important content. A firewall, CDN rule, or bot protection layer can make the page technically public but difficult for crawlers to access.
For classic SEO, those mistakes can reduce crawlability and search performance. For AI-oriented discovery, they can also make the page harder to retrieve, summarize, or understand.
Before publishing, you want to answer a few plain questions:
- Can crawlers access the page?
- Is the page allowed to be indexed where indexing matters?
- Are important resources available?
- Does `robots.txt` reflect intentional policy rather than old launch debris?
- Does `llms.txt` exist, and if it does, is it useful?
- Do schema and visible content agree on what the page is about?
- Are the most important pages discoverable through internal links?
That is AI readiness in practice. It is not mystical. It is a quality-control pass for machines.
Start with robots.txt
The first file to check is still `robots.txt`.
You can usually find it at:
```txt https://example.com/robots.txt ```
A simple file may look like this:
```txt User-agent: * Allow: /
Sitemap: https://example.com/sitemap.xml ```
A more restrictive file may include blocked directories:
```txt User-agent: * Disallow: /admin/ Disallow: /internal/ Disallow: /search/ ```
The problem is not that restrictions exist. Many sites have good reasons to block internal search pages, admin paths, duplicate URLs, or low-value generated pages.
The problem is accidental restriction.
A pre-launch review should look for rules that block important public pages, product paths, documentation, blog content, or resources needed to understand the page.
Examples worth investigating:
```txt Disallow: / ```
```txt Disallow: /blog/ ```
```txt Disallow: /docs/ ```
```txt Disallow: /products/ ```
```txt Disallow: /_next/ ```
```txt Disallow: /assets/ ```
Blocking a framework asset folder is not always harmful, but it deserves review. If crawlers cannot access CSS, JavaScript, images, or rendered content needed to understand the page, the audit is not complete.
Check AI-specific crawler rules carefully
Some websites now include rules for AI-related bots and crawlers. These may include broad AI crawler policies or specific user agents from search engines, AI companies, SEO tools, and data providers.
A simplified example might look like this:
```txt User-agent: ExampleAIBot Disallow: /
User-agent: * Allow: / ```
There is nothing inherently wrong with blocking certain crawlers. Some publishers and companies intentionally restrict AI training, AI crawling, or non-search bots. That is a business, legal, and content policy decision.
The issue is whether the decision is intentional.
If your company wants maximum discoverability in AI-oriented surfaces, blocking relevant crawlers may work against that goal. If your company wants stricter control over AI access, the block may be correct. Either way, the rule should not be an accident copied from an old template.
When reviewing AI crawler access, document three things:
- Which user agents are blocked.
- Which user agents are allowed.
- Who made the policy decision and why.
That last point is not a technical nicety. It prevents the same argument from returning every time traffic changes, a crawler appears in logs, or leadership asks why the site is not showing up in a new discovery surface.
Do not confuse robots.txt with noindex
One of the most common technical mistakes is using `robots.txt` as if it were an indexing control.
It is not.
`robots.txt` tells crawlers which URLs they may access. A `noindex` directive tells compliant search engines not to index a page. Those are different controls.
A page can be blocked in `robots.txt` and still be discovered as a URL from links elsewhere. If the crawler cannot access the page, it may never see a page-level `noindex` tag. That is why sensitive pages should not rely on `robots.txt` alone.
For pre-launch SEO and AI readiness, check both layers:
```html <meta name="robots" content="noindex" /> ```
and HTTP headers such as:
```txt X-Robots-Tag: noindex ```
Then ask whether those rules belong on the production page.
A common launch failure looks like this:
```html <meta name="robots" content="noindex, nofollow" /> ```
That tag may have been useful on staging. On a live marketing page, it can quietly undermine the entire launch.
Check whether llms.txt exists
After reviewing `robots.txt`, check:
```txt https://example.com/llms.txt ```
There are three possible outcomes.
The first is a useful `llms.txt` file. That is ideal if the site has enough structure to justify one.
The second is a clean 404. That is not automatically a failure. `llms.txt` is still an emerging convention, not a universal requirement.
The third is a server error, redirect loop, blocked response, broken template, or file full of stale links. That is the result worth fixing.
A bad `llms.txt` file is worse than no file because it creates false confidence. If it points AI tools toward old URLs, dead documentation, retired product names, or staging environments, it adds confusion instead of reducing it.
What a useful llms.txt file should include
A strong `llms.txt` file is short, selective, and written for orientation.
It should answer:
- What is this website?
- Who is it for?
- What are the most important sections?
- Which URLs explain the product, service, documentation, policies, or support paths?
- Which pages should a machine read first to understand the site?
For a SaaS company, useful links may include:
- Product overview
- Pricing
- Documentation
- API reference
- Security page
- Changelog
- Support center
- About page
- Terms and privacy pages
For a publication, useful links may include:
- Editorial mission
- Topic hubs
- Author pages
- Recent analysis
- Corrections policy
- Contact page
For an ecommerce site, useful links may include:
- Main category pages
- Buying guides
- Shipping information
- Returns policy
- Support page
- Brand or manufacturer information
The file should not include every URL on the site. That is what a sitemap is for.
llms.txt vs sitemap.xml
A sitemap is a discovery file for URLs. It helps search engines find pages.
`llms.txt` is closer to an editorial guide. It explains which pages matter and why.
A sitemap says:
```txt Here are URLs on this site. ```
A good `llms.txt` file says:
```txt Here is what this site is about, and here are the best pages for understanding it. ```
That is why dumping a sitemap into `llms.txt` misses the point. The file should reduce cognitive load, not create another crawl queue.
llms.txt vs robots.txt
`robots.txt` is about access. `llms.txt` is about orientation.
`robots.txt` answers:
```txt May this crawler access this URL? ```
`llms.txt` answers:
```txt What should an AI system know first about this site? ```
They should not contradict each other.
For example, do not list a documentation page in `llms.txt` if `robots.txt` blocks the entire `/docs/` directory. Do not list a retired product page. Do not list canonical URLs that redirect several times before resolving. Do not include staging, preview, or parameter-heavy URLs.
The files should tell the same story.
Check canonical URLs
AI readiness still depends on classic canonical hygiene.
A page may be crawlable and well-written, but if its canonical points somewhere else, crawlers receive a mixed signal.
Review the canonical tag:
```html <link rel="canonical" href="https://example.com/final-page/" /> ```
Then check whether it matches the page you actually want discovered.
Common issues include:
- Canonical points to staging.
- Canonical points to the old domain.
- Canonical points to the homepage.
- Canonical points to a similar but weaker page.
- Canonical uses HTTP instead of HTTPS.
- Canonical redirects before resolving.
- Canonical is missing on duplicate-prone templates.
If the page is important enough to publish, it is important enough to canonicalize cleanly.
Check schema and entity clarity
AI systems need context. Search systems need context. Humans need context.
Structured data helps when it matches the visible page and describes the right entity. For many websites, relevant schema types may include:
- `Organization`
- `WebSite`
- `WebPage`
- `Article`
- `Product`
- `SoftwareApplication`
- `FAQPage`
- `BreadcrumbList`
- `Person`
The schema does not need to be excessive. It needs to be accurate.
For example, a Chrome extension landing page may reasonably use `SoftwareApplication` schema. A blog post should use article-oriented schema. A company homepage should make the organization clear. A product page should not hide the product name in metadata while the visible page talks vaguely about “the platform.”
Good entity clarity comes from repetition with purpose:
- The brand name appears in visible copy.
- The product or service category is stated plainly.
- The author, company, or publisher is identifiable.
- The page explains who the content is for.
- The structured data agrees with the content.
- Internal links connect related pages.
That is not keyword stuffing. It is reducing ambiguity.
Check content density before expecting AI visibility
A thin page is hard to summarize well.
If a page only has a hero statement, three feature cards, and a contact button, there may not be enough substance for a search engine or AI answer system to understand when the page is useful.
Before launch, ask whether the page answers the questions a qualified reader would naturally have:
- What is this?
- Who is it for?
- What problem does it solve?
- How does it work?
- What makes it different?
- What are the limitations?
- What should I do next?
- Where can I verify claims?
- Where can I find pricing, documentation, support, or examples?
This is where many AI readiness audits become content audits. The machine-readable layer cannot compensate for a page that says too little.
Review internal links
Crawlers do not understand a site only by reading one URL. They also learn from how pages connect.
Before publishing, check whether the new page is linked from relevant places:
- Homepage
- Product pages
- Documentation
- Blog posts
- Resource hubs
- Navigation
- Footer
- Related articles
- Comparison pages
- Support content
A page that exists only in a sitemap may technically be discoverable, but it is not strongly integrated into the site.
Internal links also help define relationships. A guide about AI crawler access should link to related pages about technical SEO, schema, launch QA, and AI readiness. A product page should link to documentation, pricing, support, and use cases.
Machine understanding improves when site architecture reflects topic relationships.
Check response codes and redirects
Before launch, verify that important URLs return the right status codes.
The page should usually return:
```txt 200 OK ```
The `llms.txt` file should return either a valid response or a clean, intentional 404 if the site does not use one.
Watch for:
```txt 500 Internal Server Error ```
```txt 403 Forbidden ```
```txt 401 Unauthorized ```
```txt 302 temporary redirect chains ```
```txt 404 Not Found on linked canonical pages ```
A crawler or AI-oriented tool cannot reliably interpret a site if the technical surface is unstable.
Redirects deserve special attention after migrations. If `llms.txt`, canonical tags, schema URLs, internal links, and sitemap entries all point to slightly different versions of a page, the site is sending mixed signals.
Common llms.txt mistakes
The most common `llms.txt` problems are editorial, not technical.
### 1. Treating it like a ranking hack
If the file exists only because “AI SEO needs it,” it will probably be shallow. The file should help a machine understand the site. That requires careful selection and plain language.
### 2. Dumping every URL
A long list of every blog post, filter page, tag archive, and product variant defeats the purpose. Keep the file curated.
### 3. Linking to weak pages
Do not send AI tools to pages that barely explain the product, lack context, or duplicate other pages. Link to the strongest explanatory assets.
### 4. Forgetting to update it
A stale `llms.txt` file can outlive a rebrand, migration, pricing change, or documentation restructure. Add it to launch and migration checklists.
### 5. Contradicting robots.txt
Do not list URLs that crawlers are blocked from accessing. That creates confusion and wastes the file.
### 6. Linking to non-canonical URLs
Use final, canonical, HTTPS URLs. Avoid tracking parameters and redirect-heavy paths.
### 7. Overwriting judgment with automation
A generator can help draft the file, but a human should decide which pages matter. The file is an editorial artifact as much as a technical one.
A practical pre-launch AI crawler checklist
Use this checklist before publishing a new website, landing page, product page, documentation section, or major content update.
### Crawl access
- Open `robots.txt`.
- Check whether important public directories are blocked.
- Review any AI-specific crawler rules.
- Confirm that blocked user agents reflect intentional policy.
- Check whether CDN, firewall, or bot protection rules interfere with access.
### Indexing controls
- Check for page-level `noindex`.
- Check for `nofollow` where it may affect discovery.
- Review `X-Robots-Tag` headers.
- Confirm that staging rules were removed from production.
- Make sure pages that should be private use proper protection, not just `robots.txt`.
### llms.txt
- Check whether `/llms.txt` exists.
- Confirm it returns a clean response.
- If missing, decide whether the site needs one.
- If present, check whether it is concise, current, and useful.
- Remove stale, duplicate, redirected, or non-canonical URLs.
- Make sure listed pages are not blocked by `robots.txt`.
### Canonicals and redirects
- Confirm the canonical URL matches the intended production page.
- Avoid canonicals pointing to staging, old domains, or unrelated pages.
- Check for redirect chains.
- Use HTTPS canonical URLs.
- Keep sitemap, internal links, schema, and `llms.txt` aligned.
### Structured data
- Validate JSON-LD, Microdata, or RDFa.
- Use relevant schema types.
- Make sure schema matches visible content.
- Connect entities with stable `@id` values where appropriate.
- Avoid adding schema for content that is not visible or true.
### Content and entity clarity
- State the main topic directly.
- Identify the brand, product, author, or organization.
- Explain who the page is for.
- Answer obvious follow-up questions.
- Add examples, support links, documentation, or FAQs where useful.
- Avoid vague marketing copy that gives machines little to work with.
How to check this with Crowra
You can review these signals manually, but manual checks often create a scattered workflow: one tab for `robots.txt`, another for schema validation, another for metadata, another for links, another for content notes, and another for export.
Crowra is designed for the moment before a page goes live.
Open the page in Chrome, run Crowra, and review the active page beside the site itself. Crowra checks SEO metadata, robots directives, schema, links, accessibility, technical health, and AI / GEO readiness signals such as `llms.txt`, AI-bot access in `robots.txt`, content density, E-E-A-T hints, and entity clarity.
That matters because AI readiness is not one checkbox. A page can have `llms.txt` and still fail because the canonical is wrong. It can have schema and still fail because the content is thin. It can be beautifully written and still fail because `robots.txt` blocks the directory.
The useful audit is the one that puts these signals in the same review surface.
When you do not need llms.txt yet
Not every website needs `llms.txt` on day one.
A small one-page brochure site may not gain much from it. A temporary campaign page may not need it. A private web app behind authentication may have no reason to publish one.
You should consider creating `llms.txt` when the site has:
- Multiple important content sections.
- Documentation or support pages.
- Product or pricing pages.
- Editorial content.
- Public policies.
- A clear brand or entity footprint.
- Content that AI tools may need to summarize accurately.
The larger and more structured the site, the more useful a curated orientation file becomes.
What to fix first
If your audit finds many issues, do not start with the trendiest one.
Fix crawl and indexability problems first. Then fix canonicals and redirects. Then fix schema and entity clarity. Then improve content depth. Then refine `llms.txt`.
A good order of operations looks like this:
- Make sure the page can be reached.
- Make sure it is allowed to be indexed if indexing is intended.
- Make sure the canonical is correct.
- Make sure the visible content explains the page clearly.
- Make sure structured data matches the page.
- Make sure internal links support discovery.
- Make sure `llms.txt` points to the best canonical resources.
That sequence keeps the audit grounded. A perfect `llms.txt` file cannot rescue a blocked or unclear page.
Final pre-publish review
Before clicking publish, run through these questions:
- Is the page crawlable?
- Is it indexable where it should be?
- Are important resources available?
- Does the canonical point to the correct URL?
- Does `robots.txt` reflect intentional crawler policy?
- Are AI crawler rules documented?
- Does `/llms.txt` exist if the site needs one?
- Is `llms.txt` concise and current?
- Do schema and visible content describe the same entity?
- Are internal links strong enough for discovery?
- Does the page answer real reader questions?
- Can the audit be exported or shared with the team?
That last question is practical. Technical reviews are only useful if they survive handoff. Developers, marketers, founders, SEO consultants, and content teams need a shared record of what was checked and what still needs work.
The bottom line
`llms.txt` is worth checking, but it should not become another superstition in the SEO toolkit.
The real work is making a site understandable: accessible to crawlers, clear in its entities, consistent in its technical signals, and useful in its content.
AI systems may change quickly. The fundamentals behind machine-readable websites change more slowly. A clear page, a clean crawl path, accurate schema, intentional crawler rules, and a useful orientation file are all part of the same discipline.
Before publishing, do not ask only whether the page looks ready.
Ask whether a machine can understand what it is, whether it is allowed to access it, and whether the site points it toward the right context.