Methodology

How MentionFlow measures AI search

Visibility data is only useful if you can defend it. This page documents exactly how we sample each engine, how every metric is computed, and where the limits are. When the methodology changes, we version it and annotate your charts.

Sampling

• Every tracked prompt runs on every selected engine once per day (plan-dependent cadence), per configured region with its local language and Google location targeting.
• We sample consumer surfaces wherever they diverge from developer APIs — our published bake-off found official-API ChatGPT answers agreed with the real surface only 42–67% of the time on brand presence.
• Raw answers are stored verbatim and re-scored whenever our extraction improves (it has happened twice already; both re-runs are in the changelog).
• All metrics report on rolling windows — never single runs. Windows with fewer than 5 samples are flagged low-confidence.

Extraction

• Two passes: a deterministic alias matcher decides presence, then an LLM pass ranks every brand in the answer (tracked and untracked), assigns sentiment, and quotes the verbatim evidence behind it.
• mention position means rank among all brands in the answer — what a buyer actually sees.
• Measured on a hand-labeled golden set: presence F1 100, sentiment agreement 84% and climbing as the set grows.
• Sponsored content policy: ads that engines display inside answers are captured, excluded from every metric, and shown separately on each answer's receipt.

Per-engine method

ChatGPT

Method: The real consumer chatgpt.com surface, sampled through managed browser infrastructure (logged-out sessions, geo-targeted per region). We tested this against the official API with web search: the API rarely searched (8% vs 100%), cited ~1 source vs 13–14, and ran an older model — so we sample what users actually see.
Fidelity: Exact surface, current consumer model. We record the model version on every run.
Caveat: ChatGPT now shows sponsored ads inside answers. We capture them, exclude them from every metric, and surface them separately as ad intelligence.

Gemini

Method: The consumer Gemini interface, sampled via managed scraping — not the developer API.
Fidelity: Exact surface, including citation cards and tables.
Caveat: Consumer model routing changes without notice; the recorded model version tracks it.

AI Overviews

Method: Real Google SERPs with AI Overviews, geo-targeted per region and language.
Fidelity: Exact. This is the actual consumer surface.
Caveat: AI Overviews don't trigger on every query; an empty result is itself a data point we record.

AI Mode

Method: Google's conversational AI Mode surface, sampled per region and language.
Fidelity: Exact surface, markdown and references preserved.
Caveat: A young surface — Google iterates on it quickly; expect more answer churn than AIO.

Perplexity

Method: Official Perplexity Sonar API. Citations are returned natively by the engine.
Fidelity: Very high — the API is the same engine behind perplexity.ai, so no scraping needed.
Caveat: Pro-subscriber model variants may differ from the default Sonar model we sample.

Claude

Method: Anthropic API with the web search tool enabled. Sampled weekly by default.
Fidelity: High.
Caveat: Small consumer-search share and ~40× the per-answer cost of other engines — hence weekly cadence, documented rather than hidden.

Metric definitions

Presence Rate	`runs where you appear ÷ all runs` The honest, unweighted baseline.
Visibility Score (0–100)	`100 × Σ 1/log₂(1 + position) ÷ runs` First mention earns full weight; later mentions decay logarithmically. Absence scores zero.
Share of Voice	`your mentions ÷ all mentions in your competitor set` Zero-sum by construction — gains are taken from someone.
Citation Share	`runs citing your domain ÷ runs with any citation` Only answers that cite sources count in the denominator.
Sentiment Index (0–100)	`50 + 50 × (positive − negative) ÷ all sentiment-bearing mentions` 50 is neutral. Computed only on runs where you appear.

Known limits

• LLM answers are non-deterministic; day-to-day movement inside a window is expected noise. Judge trends, not ticks. Measured on our own tracked brands (1,034 prompt×engine consecutive-day pairs, July 2026): 92.6% kept the same brand presence between consecutive days. Per-engine flip rates ranged from ~4% (Perplexity) to ~11% (AI Overviews). We re-run this number as the dataset grows.
• We sample logged-out, so personalization and memory are excluded. We consider this a feature for measurement, and we say so out loud.
• Engines ship model updates without notice. We record the model version on every run and annotate charts when versions shift.
• US-English sampling by default; additional locales are sampled separately, never blended.

Questions about the method?

We'd rather lose a deal than fudge a metric. Ask us anything.

[email protected]