Site health

Before an assistant can cite your page, its crawler has to be able to fetch and read it. Site health is your own site seen the way those bots see it: every page we could reach, with its status, speed, and readability, plus the checks — robots.txt, llms.txt, and bot-protection walls — that quietly decide whether AI can read your site at all. It's the difference between "our content isn't good enough" and "the bots never got in."

What it does

MentionFlow crawls your site the way an AI bot does — our own MentionFlowBot, honoring robots.txt, capped, and disclosed — and measures every page it reaches. Everything here comes from our crawl, never a search engine's index, so the page only ever says a page was crawled, never "indexed."

The two tabs

• Overview — the health of the crawl itself: four tiles above the full page list.
• AI readiness — per-AI-bot robots.txt verdicts, llms.txt presence, and challenge / rate-limit (bot-wall) detection.

How to use it

• Read the tiles.
  • Pages crawled — how many of the discovered URLs came back, broken out as ok · redirects · failed · queued.
  • Avg TTFB — server response time, measured by our crawler on every crawled page, with a fast / moderate / slow distribution.
  • Avg LCP and Avg FCP — paint timings from our own headless-browser pass. These are a nightly, rotating sample that grows toward full coverage, not a measurement of every page — so each figure carries its N-of-M coverage, and a page that hasn't been sampled yet shows an em-dash, never a guessed number. These come from our own browser, not Google PageSpeed or CrUX.
• Work the page list. Every discovered page is a row: status (ok, redirect, robots-disallowed, fetch-failed, HTTP error, or queued), TTFB, size, word count, text-to-HTML ratio, and internal / external link counts. Sort by path, status, TTFB, or paint; filter by status (all / crawled / redirects / queued / failed); search the roster; and page through it at 10, 50, or 100 rows at a time (50 by default).
• Heed the warnings first. When your site challenged our crawler on some pages (a bot wall or rate limit), or robots.txt was unreachable so the crawl couldn't run, that banner is the story of the page — AI crawlers likely hit the same wall, which can silently suppress your visibility. An unreachable robots.txt is reported as unknown, not as a verdict.
• Crawl on demand. Crawl now hands the crawl to the server and returns immediately — discovery walks robots.txt → llms.txt → the sitemap-index tree, then measures each page, and you're free to navigate away while it runs. Crawls also run on a schedule; each page row carries when it was last crawled.
• Export. A CSV of the full page list is available once anything has been crawled.

How it's computed

• TTFB is measured on every page the crawler fetches. LCP and FCP are sampled by a headless browser on a rotating subset each night, so coverage builds over time; the tiles state exactly how many of the eligible pages were measured, and unsampled pages render an em-dash. See data honesty.
• Discovery is capped (roughly 300 URLs per crawl); if a crawl found more URLs than it kept, the crawl-status line next to the tabs discloses the truncation ("sitemap lists N, crawl cap keeps M") rather than pretending the list is complete.

Limits

• Site health is a plan feature — available on Growth and above (Growth, Advanced, and every agency plan). On plans without it, the page shows a labelled sample preview and an upgrade prompt; demo mode is open so you can see it before you buy.
• Paint metrics (LCP / FCP) are a rotating nightly sample, not a full-site measurement — read them as a trend across the sampled pages, not a verdict on every URL.
• Everything is from our crawl of your site, bounded by robots.txt and the discovery cap — this is not a search-index audit.

Related

• Crawlers — which AI agents actually fetch your site, from your server logs (the other half of "can AI read me").
• Page tracker — which of your own pages go on to earn citations.
• Brand and knowledge — the Identity tab's Site card links here.
• Data honesty — the em-dash and N-of-M coverage rules the tiles follow.