Check whether AI crawlers (GPTBot, ClaudeBot, PerplexityBot and more) can access your site. CLI, library, MCP server and GitHub Action.
Eight SEO and GEO agent skills for WordPress: AI crawler log forensics, page citability scoring, brand visibility in AI answers, robots.txt and llms.txt governance, migration redirects, SEO plugin driving, and agency portfolio reporting. Every analysis ends in a self-contained HTML report.
The best AEO/GEO skill for Claude Code. Audit, fix, and monitor your website's visibility across ChatGPT, Claude, Perplexity, Gemini, and Google AI Overviews. 100+ optimization techniques, 33 evidence collectors, 4-vector GEO Score. Built on Princeton KDD 2024 paper. Free, MIT.
See which AI crawlers can read your site — GPTBot, ClaudeBot, PerplexityBot & 20 more. Curated, operator-sourced bot list + a zero-dep CLI and GitHub Action to audit robots.txt, test reachability, read access logs, and gate it all in CI.
Validates whether GPTBot, ClaudeBot, PerplexityBot and other LLM crawlers are allowed by a robots.txt file.
Complete database of AI search engine crawler user-agents (GPTBot, PerplexityBot, ClaudeBot) with robots.txt configuration examples
Agent-native multi-bot web crawler simulator. See your site through the eyes of Googlebot, GPTBot, ClaudeBot, and PerplexityBot.
Visual robots.txt manager for the AI era. Block or allow GPTBot, ClaudeBot, PerplexityBot, and 20+ AI crawlers with one click.
Four free MCP tools to check whether AI engines can find, read and recognise a website. No account, no API key, no LLM calls.
User-agent simulator for major LLM crawlers: measures response code, content length, and JS-dependency
AI Observatory — verified AI and search crawler retrieval monitoring for business websites
GitHub Action: fail the build when robots.txt or a firewall blocks GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot or PerplexityBot. Checks llms.txt. Zero dependencies.
Free no-tracking browser utility for drafting and auditing AI crawler robots.txt rules.
21 AI crawler user-agents we detect, and which 14 of them actually reached one real site over 74 days. First-party detection list plus first-party access logs. Not observed is written "not observed", never 0.
Record which AI crawlers read your pages. Forwards GPTBot, ClaudeBot, PerplexityBot and friends to rankveo, and nothing else.
Check whether AI crawlers can access a URL (robots.txt) plus page legibility signals. Zero dependency CLI and library.
robots.txt for AI crawlers: copy-paste templates that allow AI search bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot) and let you block AI training (GPTBot, ClaudeBot, Google-Extended), a verified AI crawler list, an llms.txt template and access tests. For generative engine optimization (GEO) and AI SEO.
See which search and AI crawlers (GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot…) your firewall or CDN turns away, even when robots.txt allows them. CLI + GitHub Action. By CMO Owls.
Machine-readable dataset of AI crawler user-agents — GPTBot, ClaudeBot, PerplexityBot, Google-Extended, Bytespider, and more. Includes robots.txt recipes and weekly verification.
Audit robots.txt for sitemap hygiene, crawl mistakes, and explicit AI crawler policy.
To associate your repository with the perplexitybot topic, visit your repo's landing page and select "manage topics."