AI Discovery Files

Why AI crawlers need discovery files

Traditional search engines crawl your HTML pages and extract meaning from the content. AI systems work differently. Large language models and AI assistants benefit from structured, machine-readable summaries that describe your business in a format optimized for ingestion.

Without discovery files, AI systems must piece together your identity from scattered HTML pages, meta tags, and schema markup. Discovery files provide a single, authoritative source of truth about your business that AI crawlers can consume in one request.

What is llms.txt?

The llms.txt file follows the llmstxt.org specification, an emerging standard for providing LLM-readable content. It is a Markdown file served at your site root that gives AI systems a brief, structured profile of your business.

CitedPro generates your llms.txt from the information you enter in the Setup tab. It includes your business name, description, contact details, and links to key pages.

Example llms.txt structure

# Your Business Name

> A brief description of what your business does.

## Contact

- Phone: (555) 123-4567
- Email: hello@example.com
- Website: https://example.com

## Social

- Facebook: https://facebook.com/yourbusiness
- LinkedIn: https://linkedin.com/company/yourbusiness

## Links

- [About Us](https://example.com/about/)
- [Services](https://example.com/services/)
- [Contact](https://example.com/contact/)

What is llms-full.txt?

The llms-full.txt file is a comprehensive version that includes everything from llms.txt plus all the detailed sections you have filled out in Setup. This gives AI systems the complete picture of your business in a single file.

In addition to the basic profile, llms-full.txt includes:

• Services: Each service with its description and pricing
• Products: Product listings with prices and availability
• FAQs: All question/answer pairs
• Reviews: Customer testimonials with ratings
• Team members: Names, roles, bios, and expertise
• Service areas: Geographic regions you serve
• Contact details: Complete contact information

AI crawlers that want a quick summary can use llms.txt. Those that want the full context use llms-full.txt.

How files are served

Both files are served virtually through WordPress. No physical files are written to your server's filesystem. CitedPro intercepts requests to /llms.txt and /llms-full.txt via WordPress's parse_request hook and serves the content dynamically with the correct text/plain content type.

This approach has several advantages:

• No file permission issues
• No conflicts with hosting environments that restrict file writes
• Content always reflects the latest business information
• Works on any WordPress hosting setup

Caching

File content is cached using WordPress transients with a 1-hour expiration. This means the files are regenerated from your Setup data at most once per hour, keeping server load minimal while ensuring changes propagate within 60 minutes.

File locations

Robots.txt integration

When your AI discovery files are generated, CitedPro automatically adds pointers to them in your robots.txt output. This helps AI crawlers discover your files even if they do not know to look for them by default.

# CitedPro AI Discovery Files
# llms.txt: https://yoursite.com/llms.txt
# llms-full.txt: https://yoursite.com/llms-full.txt

Requirements

Files can only be generated when the minimum required business information is present. CitedPro checks for:

• Business name (required)
• Business description (required)

If either field is empty, the Dashboard will prompt you to complete the Setup tab before files can be generated.

Auto-regeneration

Your AI discovery files automatically regenerate when you update business information in the Setup tab. Because the files are served from cached transients, changes take effect within one hour of saving. The transient cache is also cleared immediately when you save changes to the Setup tab, so in practice updates are near-instant.

Manual regeneration

You can force a regeneration at any time from the Dashboard:

1. Go to CitedPro → Dashboard
2. Click the Regenerate Files button
3. The transient cache is cleared and files are rebuilt on the next request

Verification

After generating your files, verify they are accessible:

1. Visit yoursite.com/llms.txt in your browser. You should see a Markdown-formatted business profile.
2. Visit yoursite.com/llms-full.txt to confirm the full version includes your services, FAQs, and other detailed sections.
3. Check yoursite.com/robots.txt to confirm the file pointers are present.
4. On the CitedPro Dashboard, verify the file status indicators show green.

Best practices

• Keep information current: Update your Setup data whenever your business details change. Outdated information in AI responses is worse than no information.
• Be specific in descriptions: "Full-service WordPress security agency for e-commerce sites" is more useful to AI systems than "Web services company."
• Add comprehensive FAQs: The more questions you answer in your Setup, the more queries AI systems can accurately respond to about your business.
• Include pricing: When possible, add pricing information to services and products. AI assistants frequently receive pricing-related queries.
• Fill out all sections: The more complete your llms-full.txt, the more context AI systems have when deciding whether to cite or recommend your business.