AI crawler analytics
See GPTBot — and every AI crawler — in your server logs
GPTBot is OpenAI’s training crawler — and it’s only one of the 20+ AI bots reading your site. AI crawler analytics parses your logs, classifies each agent by why it’s there — training, search indexing, or a live answer fetch — and audits your robots.txt so you never block the bots that put you in answers.
Who’s crawling your site
Classify 20+ AI agents by what they’re actually doing
Raw logs tell you a bot showed up. They don’t tell you it matters. MentionFlow classifies every AI agent by purpose — training, search index, or live answer fetch — so a spike in AI bot traffic becomes a story you can act on, with a traffic timeline and a drill-down per agent.
Connect nginx, Apache, Cloudflare, Vercel or Fastly — or paste a log sample. No script on your pages, and the classification updates as new agents appear.
Illustrative — 20+ AI agents classified as training, search-index, or live answer fetch.
Should you block GPTBot?
Blocking GPTBot does not remove you from ChatGPT
This is the single most expensive misconception in AI SEO. Blocking GPTBot only opts you out of training future OpenAI models. The bots that decide whether ChatGPT can cite you are OAI-SearchBot (the search index) and ChatGPT-User (the live fetch while answering). Block those by accident and you disappear from answers.
| OpenAI agent | What it does | If you block it | Effect on answers |
|---|---|---|---|
| GPTBot | Trains future OpenAI models | Opts out of training | No effect |
| OAI-SearchBot | Builds OpenAI's search index | Removes you from the index | Removes you |
| ChatGPT-User | Live fetch while answering | Blocks live retrieval | Removes you |
robots.txt audit
A robots.txt audit that reads your actual top pages
The audit evaluates your robots.txt to RFC 9309 against the pages that actually matter — your real top URLs — not the file in the abstract. It tells you, in plain language, when a rule is quietly costing you answers: “blocking OAI-SearchBot removes you from answers; blocking GPTBot only opts out of training.”
You leave knowing exactly which directive to change, on which path, and what it will and won’t do — instead of copy-pasting a robots.txt from a blog post and hoping.
Illustrative — evaluated against your real top pages, not just the file in the abstract.
Crawl → citation join
Which crawled pages actually earn citations
Everyone can see that a bot fetched a page. Only MentionFlow can tell you whether that fetch turned into a citation — because we already collect the citations from your brand monitoring data. The crawl-to-citation join matches the two.
The payoff is focus: see the high-value pages that get crawled but never cited, catch wasted crawl budget spent on 404s, and send AI bots to the pages that actually convert a crawl into an answer.
Illustrative — matches pages AI bots fetch against pages AI answers cite.
Frequently asked questions
What is GPTBot?+
GPTBot is OpenAI's web crawler that collects public pages to help train future models. It is not the same as the bot that fetches pages while ChatGPT answers a live question. GPTBot identifies itself with the GPTBot user agent and respects robots.txt, so you can allow or disallow it independently of OpenAI's other crawlers.
Should I block GPTBot?+
Blocking GPTBot only opts your pages out of training future OpenAI models — it does not remove you from ChatGPT's answers. The bots that put you in answers are OAI-SearchBot (which builds the search index) and ChatGPT-User (which fetches pages live while answering). If your goal is AI visibility, block GPTBot at most, and keep OAI-SearchBot and ChatGPT-User allowed. If your goal is to withhold content from training, blocking GPTBot is the correct lever.
What is the difference between GPTBot, OAI-SearchBot and ChatGPT-User?+
GPTBot crawls for model training. OAI-SearchBot crawls to build and refresh OpenAI's search index — the pool ChatGPT draws answers from. ChatGPT-User is the live fetch that happens the moment a user asks something and ChatGPT retrieves a page to cite. Blocking GPTBot does not affect answers; blocking OAI-SearchBot or ChatGPT-User removes you from them.
What is AI crawler analytics?+
AI crawler analytics reads your server logs and shows which AI bots visit your site, how often, which pages they fetch, and — critically — why each bot is there: training, search indexing, or a live answer fetch. MentionFlow classifies 20+ AI agents by purpose, flags wasted crawl budget on 404s, and audits your robots.txt against your real top pages.
How do I connect my server logs?+
Connect through an integration — nginx, Apache, Cloudflare, Vercel or Fastly — or paste a log sample to get started in seconds. MentionFlow parses the entries, identifies each AI agent, and builds a traffic timeline with per-agent drill-downs. No tag or script to install on your pages.
Can you tell which crawled pages actually get cited?+
Yes — this is the crawl-to-citation join, and it's unique to MentionFlow because we already collect the citations. We match the pages AI bots fetch against the pages AI answers actually cite, so you can see which crawls turn into citations, which high-value pages are crawled but never cited, and where to focus. Nobody who only reads logs can compute this.
Next: AI brand monitoring to track the answers, AI content optimization to earn the citation, and the generative engine optimization guide for the strategy. Generate a spec-compliant file with the free llms.txt generator.
Find out who’s really reading your site
Connect your logs and see GPTBot, OAI-SearchBot, PerplexityBot and 20+ agents — classified, audited, and joined to citations.
Start your free trial