SEO practitioners already think in a pipeline: crawl, index, rank. AI answers have their own version, and it is worth learning as its own model, because each stage is a separate place to win or lose. An engine crawls your page, retrieves it into an answer, and then cites you, or cites a competitor instead. Miss the first stage and the last two are impossible.
The three stages, and why they are separate
A citation is the visible end of a chain most tools only show you the tip of. Pulling the stages apart is the whole game.
- Crawl. An AI bot fetches your page. GPTBot and Google-Extended crawl to build model knowledge; OAI-SearchBot, PerplexityBot, ClaudeBot and their siblings fetch at answer time, seconds before a response is written. If a bot never reaches the page, nothing downstream can happen.
- Retrieval. The engine decides your page is worth pulling into the answer it is building. Being crawled is not being used. A page can be fetched every day and never make it into a single answer.
- Citation. The answer names you, describes you, and sometimes links you. Or it names a competitor who said the same thing more clearly.
The reason to separate them: a drop in visibility has a different fix at each stage. If bots cannot reach you, no amount of better content helps. If you are crawled but never retrieved, the content is the problem. If you are retrieved but a rival is cited, you are close, and the fix is specificity.
Stage 1: get crawled
This is the webmaster's stage, and the most common own-goal. Things to check:
- Do not block the AI crawlers you want. A surprising number of sites disallow GPTBot or PerplexityBot in robots.txt, often by accident through a blanket rule, and then wonder why they are invisible in AI answers. Decide deliberately which bots you allow.
- Be reachable and fast. Answer-time retrieval bots are impatient. A page behind a slow response, heavy client-side JavaScript, or an aggressive bot wall can time out before it is read.
- Emit an
llms.txtand keep your sitemap honest. Point engines at your best pages, and ping IndexNow on publish so Bing, which feeds ChatGPT search, reindexes quickly. - Watch who is actually crawling you. A caching layer or CDN can serve bots without your origin ever seeing them, so treat any crawl count as a floor, not a census. But knowing which AI bots reach you, and which pages they take, tells you where the funnel even starts.
Stage 2: get retrieved
Being fetched is table stakes. Retrieval is about being the page an engine wants to build an answer from. Two things move the needle here.
Be answer-shaped. Retrieval favors pages that already contain the answer in a liftable form: a clear heading that matches the question, a direct paragraph under it, a list, a table, a specific number. Content an engine has to infer loses to content it can lift.
Be verifiably fresh. When answers are built by retrieval, recency is a real signal, and engines can only see the freshness you emit. A page rewritten last month with no dateModified reads as abandoned. We covered the evidence and the fix in content freshness and AI citations.
Stage 3: get cited
You are retrieved and still not named. This is the closest and most frustrating gap: the engine looked at you and picked someone else. It almost always comes down to specificity.
AI answers cite the source that states the exact thing the answer needs. The page with a specific, dated, attributable figure gets quoted; the page that said "many buyers now use AI" does not, even if it ranks higher in classic search. Own the specific claim: the stat, the definition, the named comparison, the dated fact. That is what gets pulled into the sentence.
If you want a structured way to find these gaps, that is exactly what an AI citation gap analysis is for: the prompts where a rival is named and you are not.
Reading the timing: correlation, not proof
Here is where the chain gets genuinely useful, and where it is easy to fool yourself. Because each stage is time-stamped, you can watch the sequence: an engine crawled your page on the 3rd, first cited it on the 10th. A seven-day lag. Do that across your pages and you get a feel for how long your content takes to travel from fetched to cited, and whether a change actually shortened it.
What you cannot do is call that proof. A shorter lag after a freshness push is a strong hint, not a verdict. Plenty else moved in those seven days. The honest way to use this is the way a good analyst uses any time series: as evidence to prioritize and investigate, not a single cause to declare. Show the timing, label it as timing, and let the reader make the call. For how we keep every number honest, see how we measure.
One caveat worth stating plainly: "first cited" means the first citation you observed. If you started tracking a prompt last week, a page cited months ago will look artificially late. Treat first-observed as a floor, not a birth certificate.
The one-page checklist
- Crawl: AI bots allowed in robots.txt, page fast and reachable, llms.txt present, sitemap and IndexNow current.
- Retrieval: the answer is on the page in liftable form, freshness signals emitted, structured data in place.
- Citation: you own the specific claim the answer needs, stated more clearly than anyone else.
- Read it honestly: use the crawl-to-citation timing to prioritize, never to declare a single cause.
The teams that win AI search are not the ones with the most content. They are the ones who can see the whole chain and fix the exact stage that is broken. That is the loop llemmy is built to run: watch the crawl, the retrieval and the citation on one timeline, and hear the day something moves.