Why ChatGPT gives a different answer every time (and how to measure)

Ask ChatGPT "what's the best time tracking app for freelancers?" and you'll get a list. Ask again in a new chat and the list may change: a different order, a new name, one missing. If you're trying to find out whether AI assistants recommend your product, this is confusing. Which answer is the real one?

The short version: none of them and all of them. Each answer is a sample. This post explains why answers vary and how to measure your visibility in a way that holds up anyway.

Reason 1: models choose words with some randomness

Language models write one piece at a time, and at each step they pick from several likely options. Products like ChatGPT are set up to add some variety, so two answers to the same question are rarely identical. When several products are about equally good fits, which ones make the list, and in what order, can change from one answer to the next.

This is a feature for users (answers feel natural, not robotic) and a headache for measurement. It means a single answer can't tell you much on its own.

Reason 2: web search results change

When an assistant searches the web before answering, it builds its answer from the pages it finds. Search results change from day to day: new articles get published, pages get updated, rankings shift. A comparison post that ranked well last week might not be fetched this week, and the products it recommended drop out of the answer with it.

That's also good news: it's one reason your work on the sources engines cite can show up in answers without waiting for a new model.

Reason 3: the model itself changes

The companies behind these assistants update their models regularly. A new version can know about newer products, weigh sources differently or write shorter lists. Citemark records which model version wrote each answer so you can see when a change in your numbers lines up with a model update.

Reason 4: wording, context and memory

Small differences in the question change the answer: "best" vs "cheapest", "for startups" vs "for agencies". In the consumer apps, the conversation so far, the user's location and any saved memory can also shape the answer. Two people asking the "same" question can get different results.

That's why it matters to fix the exact wording of each question you track, and to track several phrasings of the questions that matter most.

How to measure anyway

Variation doesn't make measurement pointless. It means you measure the way you'd measure anything noisy: repeat it, keep the method fixed, and look at averages and trends.

1. Keep the questions fixed. Track the same wording every time, so a change in the answer is a change in the engine, not in your question.
2. Ask repeatedly. Your most important questions should be asked every day; the rest at least weekly.
3. Count shares, not single answers. "Named in 7 of 10 answers this week" is meaningful. "Named in this one answer" is not.
4. Weight by rank. Being named first matters more than being named fifth. Our score weights each answer by your position.
5. Treat errors as no data. When an engine fails to answer, leave that answer out. Counting it as "not mentioned" makes outages look like drops.
6. Read trends over weeks. Look at 7-day and 30-day views, and expect a change you make to take weeks to show.

What a real change looks like

A real shift usually shows up as a steady move in the trend line across several questions and days, often on one engine first. A one-day dip on a single question is almost always noise. When you do see a sustained change, open the answers on either side of it and compare: who appeared or disappeared, and which cited pages changed? That's usually where the explanation is.

API answers vs what users see

Tools like Citemark ask the engines through their official APIs, with web search on where the engine supports it. That's close to what people see in the consumer apps, but not identical: the apps add personal context, memory and their own settings. We think that's the right trade-off, because the API gives a consistent, repeatable view across days, which is what you need to see trends. Every answer is stored in full so you can read exactly what the engine said.

A quick checklist

If you're setting up your own measurement, by hand or with a tool, check these before you trust the numbers:

• Each question has one fixed wording, written the way a buyer would ask it.
• Each engine you track is asked the same questions, on a regular schedule.
• Failed answers are excluded, not counted as misses.
• You record your position in each answer, not only whether you were named.
• You store the full answer text and the cited pages, so you can go back and read them.
• You look at weekly and monthly trends before drawing conclusions.

Common mistakes

The most common mistake is reacting to a single answer. Someone on the team asks ChatGPT a question, doesn't see the product, and a project starts. Ask the same question five times first and count. The second is changing the question wording every week, which makes trends meaningless. The third is comparing numbers from different tools: each picks different questions, engines and weights, so their scores aren't comparable. Compare your numbers with themselves over time.

The bottom line

One answer is an anecdote. The same questions, asked consistently over weeks, give you a trend you can act on. If you'd like to see where you stand today, run the free AI visibility check, then track the questions that matter on a paid plan to watch them move.