MentionFlow AI
Open app
DocsIntegrations

Crawler ingest

Get your AI-crawler logs into MentionFlow so you can see which AI bots visit your site. A one-click Cloudflare Worker, a log paste or upload, or a continuous feed.

AI assistants do more than answer questions about your site. Their bots also crawl it. Which bots reach you, and which get blocked, shapes whether you show up in AI answers at all. Agent analytics turns that invisible bot traffic into a dashboard. First it needs your server's crawler logs. Here are the three ways to hand them over.

Whichever you pick, MentionFlow keeps only what it needs. It stores recognized AI agents only, and only the path they requested. Never query strings, never anything that could identify a visitor.

Three ways to send your logs

1. One-click Cloudflare Worker (easiest). If your site is on Cloudflare, paste a Cloudflare token once. MentionFlow deploys a small Worker that streams crawler hits to it. Your token is used once to deploy and never stored. The Worker signs in afterward with its own ingest-only API key.

2. Paste or upload your access logs. Already have server logs? Drop them in. Up to 5 MB, in standard nginx/apache combined format or NDJSON. Good for a quick one-time look without wiring anything up.

3. Continuous REST feed (for engineers). POST batches of events to https://app.mentionflow.ai/api/v1/crawler-events?brand=<your project id> from your own pipeline, up to 10 MB per batch. That keeps the view always current. Send your API key as Authorization: Bearer mf_…. The Connections tab on Agent analytics shows your exact endpoint with a copy button, a link to the page that makes the key, and a button that copies the whole guide for your developer.

Good to know

  • The Agent analytics dashboard itself is a Growth-plan feature.
  • Ingest works on any plan that can create an ingest-scoped API key. So you can start collecting logs before the dashboard is available to you.
  • Size caps: log paste or upload 5 MB; REST batch 10 MB.
  • Rate: the REST feed accepts 120 requests a minute per key, on its own bucket separate from your read requests. Over that you get a 429 with Retry-After. Nothing retries for you, so send batches rather than one request per hit, and wait out a 429. See Limits.

Related