MentionFlow AI
Open app
DocsIntegrations

Crawler ingest

Three ways to get AI-crawler logs into MentionFlow for agent analytics — a one-click Cloudflare Worker, pasted or uploaded access logs, or a continuous REST feed.

Agent analytics needs AI-crawler logs. There are three ways to get them into MentionFlow, from a one-click Worker to a continuous REST feed, and all of them store only what is needed.

What it does

It ingests AI-crawler logs — records of AI agents visiting your site — so they can feed MentionFlow's agent analytics.

How to use it

There are three ingest paths:

  1. One-click Cloudflare Worker. Your pasted Cloudflare token is used once and never stored, and a fresh ingest-scoped API key is minted as the Worker's secret.
  2. Paste or upload access logs — up to 5 MB, in nginx/apache combined format or NDJSON.
  3. Continuous REST feed — POST batches to /api/v1/crawler-events (up to 10 MB per batch).

Across all three paths, only recognized AI agents are stored, and only paths — never query strings or PII.

How it's computed

Ingested records are filtered to recognized AI agents before anything is stored, and the stored request is reduced to its path only — query strings and PII are dropped at ingest. The Cloudflare path treats your token as single-use: it is used to deploy the Worker and never persisted, while the Worker authenticates with its own ingest-scoped API key.

Limits

  • The agent-analytics dashboard is a Growth-plan feature.
  • Ingest itself works with an ingest-scoped API key on any plan that can create one.
  • Log paste/upload cap: 5 MB; REST batch cap: 10 MB.

Related