What is actually different
The model is the same family of model. Almost everything around it is not. A consumer app may search the web before answering and may not; may carry memory of the person asking; may apply a system prompt that shapes tone and hedging; may be running that person in an experiment bucket; and it knows their approximate location. An API call has none of that unless the caller adds it deliberately, and the caller can only add some of it.
That is why a report and your own phone can disagree on the same afternoon. Neither is lying. They are two different samples of a system that is not deterministic in the first place.
What a dated sample is genuinely good for
- Direction over time. Same question, same wording, same schedule. If you were named in four of twenty questions in July and eleven in September, something changed, and the captured answers say what.
- Comparison against competitors on the same run. Every business in that answer was sampled under identical conditions, so share of voice is a fair internal comparison even though it is not the app.
- Evidence you can act on. The cited links in an answer tell you which pages the model reached for, which is a concrete list of places to be present.
- A dated record. An answer captured on a date and stored is checkable later. A screenshot someone took on their phone is not.
What nobody should claim
- That a report reproduces what a specific customer saw in the app. It does not, and a vendor who says otherwise is overselling.
- That a single run proves anything. Model answers vary between identical calls. One absence is noise; four weeks of absence is a finding.
- That Google's AI Overviews shown in any dashboard are live. They are read through a search API and are usually cached, so treat them as up to a week old unless the tool stamps them otherwise.
- That a score is the truth. The score is a summary of the sample. The sample is the evidence.
The two-minute hand check
Open the app you care about in a logged-out window so your history is not in play, ask the question the way a customer would, and do it three times. Write down who was named each time. If your business is absent all three times, that is consistent with the sample, and the fix is the same either way: give the web more, better, more specific evidence about what you do and where. If you are named in the app but not in the sample, the gap is usually personalization or location, and the direction over the next month is still the number worth watching.
What each method can and cannot tell you
Use this before you argue with a report. Most disagreements are a question being asked of the wrong method.
| What you want to know | A dated API sample | Typing it into the app yourself |
|---|---|---|
| Exactly what one specific customer saw | No. Cannot be reconstructed by any tool. | Closest available, but still only what you saw, with your location and history. |
| Whether the model names you at all on a standard question | Yes, and repeatably, on the same wording every time. | Yes, once, for that moment. |
| Whether that changed over three months | Yes. This is the thing the method exists for. | Only if you kept dated notes every time, which almost nobody does. |
| Which pages the answer drew on | Yes, when the answer cites links; they are captured with it. | Sometimes, if the app shows sources that session. |
| How you compare to five named competitors on the same question | Yes, all sampled under identical conditions in one run. | In principle, but you would have to read and tally every answer by hand. |
| What Google's AI Overview says right now | Read through a search API and typically cached, so treat it as up to a week old. | Live, but only for your location and your session. |
The honest summary: an API sample is a controlled experiment on a moving system, and the consumer app is the system. Use the sample for direction and evidence, use your own phone for a reality check, and distrust anyone who promises the two are the same.
You are looking for the term
AI visibility monitoring
Doing this deliberately has a name: AI visibility monitoring. A fixed question set, asked the same way on a schedule, with the answers and their citations kept and dated. The discipline is in the fixed wording and the dates, not in the dashboard.
- Why did my report say I was not mentioned when ChatGPT just named me?
- Most often personalization or location. The app knows roughly where you are and may remember you; a clean API call knows neither unless the location is written into the question. Watch the direction across several weeks rather than any single reading.
- Does asking the model more times make the sample more accurate?
- It reduces noise, which is not the same as removing the gap between the API and the app. More repetitions give you a more stable estimate of what the model does with that question; they do not tell you what any given person's app did.
- Should I trust a tool that claims to check the real ChatGPT app?
- Ask how. If the answer involves scraping a logged-in consumer session, ask what happens to the numbers when that breaks or gets blocked, and whether the account's memory and location are shaping every result you are being sold.
- 01Models and API referenceOpenAI Platform
- 02Messages API referenceAnthropic
- 03Gemini API modelsGoogle AI for Developers
- 04Google AI Overview API documentationSerpApi
Every link above was opened and confirmed reachable on September 1, 2026.
The automated version
See what ChatGPT, Claude, Gemini and Perplexity say about your business
The free scan runs the buyer questions for your trade and city across all four assistants, shows who gets named instead of you, and lists the sources those answers came from. No signup, about thirty seconds.
Run my free scan