PluginsArmorProCitedProBoostPro
PricingDocsBlogAboutGet Started

Robots.txt Management

Configure AI bot access with automatic robots.txt directives.

Last updated Feb 3, 2025

How It Works

The robots.txt file tells web crawlers which parts of your site they can access. CitedPro extends this to include directives specifically for AI bots.

When you generate files, CitedPro intelligently merges AI bot directives with your existing robots.txt rules — preserving everything you've already configured.

AI Bot Directives

CitedPro adds directives for known AI crawlers:

# CitedPro - AI Bot Directives
# Generated: 2025-02-03

User-agent: GPTBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: Claude-Web
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Amazonbot
Allow: /

User-agent: Google-Extended
Allow: /

What Each Bot Does

BotCompanyPurpose
GPTBotOpenAITraining data collection
ChatGPT-UserOpenAIReal-time browsing for ChatGPT
ClaudeBotAnthropicClaude AI training and browsing
PerplexityBotPerplexityAnswer engine indexing
Google-ExtendedGoogleGemini/Bard training data
AmazonbotAmazonAlexa and AI services

Blocking Specific Bots

If you want to block certain AI bots from crawling your site:

  1. Go to CitedPro → Bot Control
  2. Find the bot you want to block
  3. Click Block
  4. The bot will be added to your robots.txt with Disallow: /

Important

Blocking AI bots means they won't crawl your content, but it also means AI assistants may not be able to recommend your business. Block selectively.

Preserving Existing Rules

CitedPro never overwrites your existing robots.txt rules. It:

  1. Reads your current robots.txt
  2. Identifies the CitedPro section (marked with comments)
  3. Updates only that section
  4. Leaves all other rules untouched

If you have custom rules for Googlebot, Bingbot, or other crawlers, they remain exactly as configured.

Manual Editing

If you need to manually edit AI bot directives:

  1. Open your robots.txt file via FTP or your hosting file manager
  2. Find the section between # CitedPro - Start and # CitedPro - End
  3. Make your changes
  4. Save the file

Tip

If you manually edit the CitedPro section, your changes may be overwritten when you regenerate files. For persistent custom rules, add them outside the marked section.

Verifying Your Robots.txt

  1. Visit yoursite.com/robots.txt
  2. Look for the CitedPro section with AI bot directives
  3. Verify Allow/Disallow rules match your preferences

Troubleshooting

Changes Not Appearing

  • Clear any caching plugins
  • Clear server-level cache (Varnish, nginx)
  • Wait for CDN cache to expire (Cloudflare, etc.)

Cannot Write to robots.txt

If CitedPro can't update robots.txt, check:

  • File permissions (should be writable by web server)
  • WordPress has write access to the root directory
  • No other plugin is locking the file
Previous← File GenerationNextBot Tracking →