GuidesNovember 10, 2026 · 4 min read
How to track AI answers before and after a messaging change
Changed your homepage, pricing or positioning? How to measure whether ChatGPT, Gemini and Claude describe and recommend you differently afterward.
Key takeaways
- Record a baseline before you ship anything: the same buyer questions, asked several times on each assistant, with the exact words AI uses about you.
- Answers that search the web can change within weeks of a page being re-read; what a model knows from training changes more slowly, so measure the two separately.
- Judge the change by the trend against your competitors over several weeks, never by one answer.
- The tracker you need keeps every answer's full text and sources, so before and after can be compared line by line.
To know whether a messaging change reached AI answers, start measuring before you change anything. Pick the buyer questions that matter, ask ChatGPT, Gemini and Claude each one several times, and save four things: how often you're named, your position, the exact sentence used to describe you, and the pages cited. Then ship the change, keep asking the same questions the same way on a schedule, and compare the trend with your competitors. One answer proves nothing. A few weeks of the same questions does.
This guide covers why the change takes time to show up, how to set up a fair before and after, which numbers to watch, and what to look for in a tool. We build Citemark, one of those tools, and we say so where it matters.
Why messaging changes take time to show up
AI answers draw on two sources, and they move at different speeds.
- Live web search. When an assistant searches before answering, it reads pages as they are now. Once the AI search bots re-read your updated page, these answers can change within days or a few weeks.
- What the model already knows. Without a web search, the answer comes from training data. That only changes when the model is updated, which can take months, and it reflects how the whole web describes you, not just your site.
Two practical points follow. First, make sure AI search bots can read the new page at all: the AI crawler checker shows which bots your robots.txt allows. Second, your old message lives on in review profiles, directories and old comparison articles. Update those too, or the old wording keeps coming back.
Step 1: Choose the questions and the words to watch
Use 10 to 20 buyer questions, mixed across three kinds:
- Discovery: "best [category] for [audience]", "[competitor] alternatives".
- Comparison: "[you] vs [competitor]", "which is better for [use case]".
- Description: "What is [you] and who is it for?", "How much does [you] cost?".
Then write down what you're changing, as 3 to 5 phrases you want AI to start using. "Built for agencies", "free plan", "SOC 2 compliant" or a new category name are typical. These phrases are your before-and-after markers. Choosing buyer questions helps with the list.
Step 2: Record a baseline
Before you ship, ask every question three times on each assistant, in fresh chats. For each answer, save:
| Field | Example |
|---|---|
| Date, assistant and model version | Nov 10, ChatGPT |
| Named, and position | Yes, #3 |
| The sentence that describes you | "Acme is a simple tool for small teams" |
| Your marker phrases used | None |
| Competitors named | Rival A, Rival B |
| Pages cited | Two list articles, Rival A's pricing page |
If you can, keep collecting for a week or two before the change. Answers vary from run to run, and a longer baseline tells you what normal looks like. Why answers vary explains how much.
Step 3: Ship the change where AI reads
Change the pages AI is most likely to read: your homepage, pricing page, comparison pages and docs. Then update the third-party pages that describe you: review profiles, directories and any list articles you can get edited. The review sites playbook and comparison pages AI cites cover both. Note the date you shipped each change, so you can line it up with the results.
Step 4: Measure after, the same way
Keep the questions, assistants and schedule exactly the same. Then watch five numbers:
- Share of answers that name you, overall and per assistant. Share of voice explains how to calculate it.
- Average position when you're named.
- Phrase adoption: how many answers now use your marker phrases.
- Citations of the changed pages: are your new pages showing up as sources?
- Competitors' share: your control. If everyone rose at the same time, it was probably a model update, not your change.
Check weekly. Expect answers that search the web to move first, often within one to four weeks, and model knowledge to follow later. Watch for old facts that linger, such as a price you dropped, because AI repeats them until the pages it reads change.
What to look for in a tool
A spreadsheet works for a handful of questions; the manual way explains when it stops working. For a real before and after, a tracker needs five things: the same questions on a fixed schedule, the full text of every answer, the model version, the cited sources, and enough history to compare months, not days.
| Tool | Entry price | Notes for before-and-after tracking |
|---|---|---|
| Citemark | $19/mo Starter, $49/mo Pro | 50 questions on ChatGPT and Gemini, top 10 daily; Pro adds Claude and 100 questions. Keeps every answer, its sources and a year of history (two on Pro) |
| Otterly.AI | $29/mo Lite | 15 prompts; Gemini (+$9/mo) and Claude (+$29/mo) are add-ons |
| LLMrefs | $79/mo | 500 prompts across the major assistants |
| Peec AI | $95/mo Starter | 50 prompts on 3 models you choose; Claude on Enterprise only |
| Profound | Custom quote | Enterprise platform |
Prices are from each vendor's own pricing page, checked on November 4, 2026. They change, so check before you buy. For a broader comparison, see the best AI visibility tools for founders.
A worked example
Imagine a form builder that rewrites its homepage around "built for agencies". In the baseline, it's named in 2 of 30 answers to "best form builder for agencies", and no answer uses the word agencies about it. Four weeks after the change, and after its review profiles are updated, it's named in 7 of 30, with "agencies" in 5 of those descriptions. Meanwhile its main rival's share stays flat. That's a result you can trust: the same questions, a stable control, and a clear change after the ship date.
Start with a baseline today
Run the free AI visibility check to see where you stand on 5 buyer questions in about a minute, with no sign-up. To track the same questions every day before and after your change, start a 7-day Pro trial with no card.