Site health

Before an assistant can cite your page, its crawler has to fetch and read it. Site health is your site seen the way those bots see it: every page we could reach, with its status, speed, and readability, plus the checks that decide whether AI can read you at all. It is the difference between "our content isn't good enough" and "the bots never got in."

What it does

We crawl your site the way an AI bot does, with our own MentionFlowBot. It honors robots.txt, it is capped, and it is disclosed. Everything here comes from our crawl, never a search engine's index, so the page only ever says a page was crawled, never "indexed."

The two tabs

• Overview: the health of the crawl itself, four tiles above the full page list.
• AI readiness: per-AI-bot robots.txt verdicts, llms.txt presence, and challenge or rate-limit (bot-wall) detection.

How to use it

• Read the tiles.
  • Pages crawled: how many discovered URLs came back, split into ok, redirects, failed, queued.
  • Avg TTFB: server response time, measured on every crawled page, with a fast / moderate / slow spread.
  • Avg LCP and Avg FCP: paint timings from our own headless-browser pass, not Google PageSpeed or CrUX. They are a nightly, rotating sample that grows toward full coverage, so each figure carries its N-of-M coverage. An unsampled page shows an em-dash, never a guessed number.
• Work the page list. Every discovered page is a row: status (ok, redirect, robots-disallowed, fetch-failed, HTTP error, or queued), TTFB, size, word count, text-to-HTML ratio, and internal and external link counts. Sort by path, status, TTFB, or paint. Filter by status (all / crawled / redirects / queued / failed). Search the roster. Page through it at 10, 50, or 100 rows at a time, 50 by default.
• Heed the warnings first. Your site may have challenged our crawler on some pages, through a bot wall or a rate limit. Or robots.txt was unreachable and the crawl could not run. That banner is the story of the page: AI crawlers likely hit the same wall, which quietly suppresses your visibility. An unreachable robots.txt is reported as unknown, not as a verdict.
• Crawl on demand. Crawl now hands the crawl to the server and returns at once, so you can navigate away while it runs. Discovery walks robots.txt, then llms.txt, then the sitemap-index tree, then measures each page. Crawls also run on a schedule, and each row shows when it was last crawled.
• Export. A CSV of the full page list is ready once anything has been crawled.

How it's computed

• TTFB is measured on every page the crawler fetches. LCP and FCP are sampled by a headless browser on a rotating subset each night, so coverage builds over time. The tiles state how many eligible pages were measured, and unsampled pages render an em-dash. See data honesty.
• Discovery is capped at roughly 300 URLs per crawl. When a crawl finds more URLs than it keeps, the crawl-status line next to the tabs discloses it: "sitemap lists N, crawl cap keeps M".

Limits

• Site health is a plan feature, on Growth and above (Growth, Advanced, and every agency plan). Other plans see a labelled sample preview and an upgrade prompt. Demo mode is open so you can look before you buy.
• Paint metrics (LCP / FCP) are a rotating nightly sample. Read them as a trend across the sampled pages, not a verdict on every URL.
• Everything comes from our crawl of your site, bounded by robots.txt and the discovery cap. This is not a search-index audit.

Related

• Crawlers: which AI agents actually fetch your site, from your server logs. That is the other half of "can AI read me".
• Page tracker: which of your own pages go on to earn citations.
• Brand and knowledge: the Identity tab's Site card links here.
• Data honesty: the em-dash and N-of-M coverage rules the tiles follow.