Choosing an AI visibility tool? Start with what you should track

Our clients have started asking us: "Does ChatGPT recommend us?" For most of 2025, we couldn't answer that with any confidence.

We looked at the tools on the market. Most are priced on how many prompts you track, which AI engines you track them on, and how many brands or projects a plan includes:

Prices from each vendor's own pricing page, checked 6 November 2026. Vendors change their packaging often — Profound removed its self-serve plans in September 2026 — so check before you buy.

Entry plans include between 15 and 100 prompts. How many you need depends on how many personas, buying stages and locations you track for each client. Agency and white-label options exist, but most vendors don't publish their prices.

We built our own for four reasons:

1. Price. Per-prompt pricing across a whole client list added up.
2. Recommendations. The tools we looked at were good at reporting where a client stood, but gave little guidance on what to do to improve it.
3. Proof. We wanted to test changes and show whether they made a difference — then keep measuring over time, not just once.
4. Change. The AI engines keep evolving. What works today may not be what works next year. Measuring continuously across all our clients gives us an up-to-date view of what is working now, and that is how we decide what to do next.

We also wanted the measurement to sit next to the work: the client's board, their site, their Search Console data and their tests. So we built AI Visibility into Pulse, the platform we run Traction on.

This article is for agency owners weighing the same decision. It covers the questions you're probably asking, how we answered each one, what the tool looks after, and what we haven't solved yet.

These come from agency forums, industry coverage and our own conversations. We've grouped them, and under each one we say how Pulse handles it.

They do. In a study by SparkToro and Gumshoe, ChatGPT and Google's AI gave the same list of brands in under 1% of runs, across nearly 3,000 runs of the same prompts. One agency CEO told Digiday that three tools given the same prompts gave three different answers.

So one check can't tell you much. What you can measure is how often a business appears across many answers. That rate is far steadier than any single answer or any "AI ranking", but it still moves — and the fewer answers behind it, the more it moves.

What our own data shows. Since July 2026, Pulse has asked ChatGPT and Claude "Who are the leading digital marketing agencies in Christchurch?" three times a week. In about 1 run in 5, the answer flipped from the run before — naming Traction when it hadn't, or dropping us when it had. And in Claude's last 10 answers, no two lists of agencies were the same.

Pool the answers, though, and the picture settles. We track eight non-branded questions that have run since July. Each week they produce around 40 answers between the two engines, and Pulse works out the share of those answers that named Traction.

• Week by week, from August to October, that share swung between 22% and 50% — roughly 9 answers in 40 one week and 20 in 40 another.
• As a four-week rolling average — each week combined with the three before it, about 160 answers — the share only moved between 31% and 39%.

The more answers behind a number, the less it swings by chance. In our data, single answers and single weeks moved a lot; four-week rates moved much less. That's why Pulse puts the answer count next to every number, flags any week with fewer than 20 answers, and its overview pools the last four weeks rather than reacting to one.

How Pulse handles it

• Every question runs three times a week, on each AI engine, in each location we track. One answer is one data point; the measure is the share of answers over a week.
• Results are reported weekly and never smoothed. Every percentage carries its answer count. Under 20 answers in a week gets a low-confidence flag; a cell with fewer than 5 answers is marked "few answers".
• Pulse doesn't report a "rank in AI". In the SparkToro study, the same list in the same order came up in fewer than 1 in 1,000 runs, so position within one answer isn't a measure we use.

The AI companies don't share query data, so every tracked prompt is, in a sense, synthetic. Which questions you choose decides what the numbers say.

How Pulse handles it

• Questions are written for a persona, a buyer journey (a persona's goal, like "fix our SEO") and a stage of that journey — from Unaware through Aware, Consideration, Evaluation and Decision.
• When we suggest new questions, the evidence comes first: Google Ads Keyword Planner volumes, the client's own Search Console queries for the pages in that topic, and their Google Ads search terms. These tools show the broad topics people search for, not the exact questions they ask AI. Search volume is only recorded for short phrases, and questions put to AI are longer and more specific. So an AI drafts up to 8 questions from that evidence, each written to cover the whole topic rather than repeat the short phrase. Volumes never come from the AI, and an idea with no evidence is dropped.
• Each suggestion shows a Demand gauge (relative to the strongest question in the same batch, with its sources) and an Ease gauge (from one live test answer).
• A question-quality review checks realism, persona and stage fit, and duplicates. Nothing changes until a person accepts it.
• Branded and non-branded questions are reported separately. A branded question — one that contains the client's name — will almost always name them. Since mid-October, our branded questions named Traction in 98% of answers; our non-branded questions named us in 20%. Mix the two and every number looks better than it is, so visibility is reported on non-branded questions by default.
• Branded questions still earn their place, for a different job: checking that the AI engines, and their web searches, have read the right information from the client's website. "Who founded Traction Marketing?" and "How much does Traction Marketing charge?" test whether the answers get the facts right. On ours, the answers linked to our website 89% of the time, and none was flagged as inaccurate.

They measure different things, so Pulse reports each one separately.

How Pulse handles it — three separate measures:

• Named: the answer names the business.
• Recommended: the answer names the business and puts it forward as a choice. It can never be higher than Named.
• Cited: a page on the client's own website is linked in the answer. This is measured only against answers where the AI actually searched the web — otherwise there was nothing to link to.

They can be far apart. Asked "Who are the leading digital marketing agencies in Christchurch?", the engines named Traction in 69% of answers — but recommended us in only 0–8%. Named on its own would suggest a much stronger position than the recommendation rate shows.

Each measure moves through steps — Not yet, Emerging, Established, Leading — with its own bars, because recommendation rates run much lower than naming rates. Across our non-branded questions from July to mid-October: named 29%, recommended 7%.

Pulse also combines the three into an AI visibility score out of 100 for a single headline, and shows the three measures beside it. Inside each journey stage, the score is weighted to suit the stage: at Evaluation, being recommended counts most; at Unaware, simply being named counts most.

A brand name can also be an everyday word. Ours is: "traction". In our own answers this was rare — of 337 answers that contained the word, 7 weren't about us — but a plain text match would have counted all of them.

How Pulse handles it

• Every mention needs name evidence: the actual words from the answer (the name, a known alias, or the domain). Each tag in the app shows its proof line, and a tag without evidence is flagged.
• An AI classifier judges each mention — was it a recommendation, a passing mention, positive or negative, accurate or not.
• Citations are read from the engine's own citation data, never guessed from the answer text.
• For off-site pages, a page only counts as the client when there's positive evidence — a profile the client links to, a link back to their site, or their city or street. A business with the same name in another city or country is left out.
• A plain use of the word — "gaining traction" — is never counted as a mention of the business.

Not exactly. Studies by tool vendors report low overlap between API answers and app answers. The vendors publishing those studies sell app-scraping tools, so treat the exact figures with care.

How Pulse handles it

• Pulse asks ChatGPT and Claude, with web search switched on for every question and an approximate location (we track Christchurch and New Zealand). Without web search, the answer comes only from what the model learned in training.
• Each answer records whether the AI actually searched and whether the location was applied. It matters: even with web search switched on, ChatGPT didn't search in about 1 answer in 10. That's also why Cited is measured only against answers where the AI searched.
• The engines can give very different results. On the same non-branded questions over the same weeks: "Which Christchurch marketing agency is best for lead generation?" — Claude named us in 55% of answers, ChatGPT in 5%. "A digital marketing agency in Christchurch that specialises in B2B companies" — 76% against 42%. "Which agencies in New Zealand have proven case studies?" — 0% against 18%.
• If an engine fails or drops out, the trend only uses engines present in every week, the dashboard shows a coverage warning, and admins get an alert. We added this after an engine outage went unnoticed in July.

(Gemini, Perplexity and Google AI Overviews are not tracked yet — see "What we haven't solved".)

It can. If new questions score higher than the existing ones, the overall number rises even when nothing else has changed.

How Pulse handles it

• Questions added in the same week form a question set. The chart marks the week with "+9 questions added", and each set can be shown as its own line.
• Changes on the tiles compare like for like: only questions with answers in both periods count.

This happened to us. In the week ending 18 October we added nine questions, and our overall score jumped from 32 to 49. Our original eight questions, on their own, had actually fallen from 32 to 24. Pulse showed both — the tooltip reads "Same 8 questions: 32 → 24".

Partly. AI referrals are still a small share of website traffic — about 0.4% of sessions across our clients' websites in 2026. But across those sites, visitors who arrived from ChatGPT converted at 7.0%, against 4.5% from organic search and 6.2% from paid search. That held for 9 of the 16 clients with enough ChatGPT traffic to compare, not for all of them. One possible reason is that people arrive having already shortlisted through a conversation with AI, but our data can't show that directly.

In New Zealand, people send almost 10 million ChatGPT messages a day, and 35% of users surveyed had used it to compare tradie quotes.

How Pulse handles it

• We don't claim AI visibility causes revenue. Pulse measures whether the client is named, recommended and cited.
• The same platform holds the client's Search Console, Google Ads and analytics data, so we read AI visibility alongside branded search and organic trends rather than in isolation.
• Every test reports an observed change, never a cause (see question 9).

There is overlap with SEO. But AI answers also draw heavily on other people's websites: directories, reviews, forums and press. In Ahrefs' study of 75,000 brands, branded web mentions correlated much more strongly with AI Overview visibility than backlinks did. Reddit sits at or near the top of the most-cited sources in several studies.

Our own data says the same. Across 954 answers to our non-branded questions, the engines linked to 421 different websites — and only 1.1% of those links pointed to tractionmarketing.nz. The most-cited sites were directories and "top agencies" lists: Semrush Agencies, top5.nz and Clutch among them. We're testing the best ways to influence that right now, and we'll share what works.

How Pulse handles it — it looks after both sides:

On the client's site

• Silos: Pulse reads the site, groups pages into topics with a pillar page, and shows which journey stages each topic covers — and the stage pages that are missing. People can add, move and remove pages by hand, and a re-scan never undoes their choices.
• Foundations: checks that the facts AI needs are on the site and correct — business name, address, phone, founders, year founded, area served, services and profile links — and whether AI crawlers (GPTBot, ClaudeBot, PerplexityBot and others) are allowed in.
• Structured data map: shows schema coverage by section, whether it links into one connected graph, and the gaps.
• Search Console overlay: flags "striking distance" queries for each topic.

Off the client's site

• One scan fills seven boxes: citations and directory listings, case studies, testimonials and reviews, press and media, forums, awards and partnerships, and social and video.
• Directories: ranks the directories AI answers actually cited, and checks whether the client is listed and the details are correct.
• Forum opportunities: a forum thread that AI read in several answers over several weeks becomes a finding, filed under the right topic.
• Competitor discovery: every organisation named in an answer is listed, so the team can start tracking a competitor in one click.

How Pulse handles it

• Findings → recommendations: 13 detectors look for specific patterns — "competitors named, you're not", "named but rarely recommended", "a site AI cites that you're not on", "your page was used but your name wasn't said". Recommendations must point to real findings and to an active playbook, or they're rejected.
• Recommendation → task: one click puts a master task on the client's board, with a planned test attached.
• Task → test: measurement starts when the task is marked done. The questions under test are locked, results are read at 4, 8 and 12 weeks, and targeted tests are compared against the client's other questions as a control.
• On the chart: numbered markers show the week each piece of work went live, so the client can see what happened next.
• A test that worked can be saved as a new playbook for the next client.

Web search on the major APIs costs roughly US$10–14 per 1,000 searches, plus the model's own usage.

Our real numbers:

• Each answer costs about US$0.05–0.08, web search included.
• Tracking Traction itself — ChatGPT and Claude, three times a week — has cost about US$11–17 a month in AI usage since August. In July, when we ran every question daily, it was about US$30.
• Every run has a hard US$10 cap. If a run reaches it, Pulse stops and marks the run as partial.
• Every costly button shows an estimate and asks for confirmation first. The site and off-site scans cost from nothing (the structured data map) to about US$0.90 (a full off-site scan).
• We moved from daily runs to three a week in July 2026. That cut the cost by about 57%. The trade-off is fewer answers each week, which is one reason a test on a single question can come back inconclusive (see below).

Most of the work is around the calls:

• Reading answers correctly — names, aliases, look-alike businesses, citations.
• Enough answers to tell a real change from noise.
• Engines that rename models, change their search tools or go down.
• Fair comparisons when the question set changes.
• Turning numbers into work, and work back into numbers.

Building this, and keeping it running as the engines change, is ongoing engineering work. We took it on for the reasons at the top of this article: cost, better recommendations, proof that changes work, and a continuous view across all our clients.

To be clear: Pulse isn't for sale. It's the platform we use to deliver for our own clients. We're sharing how we built it because the questions are the same whichever way you go.

• Engine coverage. Pulse tracks ChatGPT and Claude today. Gemini, Perplexity and Google AI Overviews / AI Mode are not tracked yet.
• API vs app. We measure through the engines' APIs with web search on. That's consistent and repeatable, but it isn't identical to what a logged-in person sees in the app, with their history and memory.
• One answer per run. Each question gets one answer per engine per run; we build the sample over the week rather than asking the same question several times at once. That keeps costs down, but a test on a single question and a single engine can still come back inconclusive.
• The bars and weights are judgement. The thresholds for Emerging, Established and Leading, and the stage weightings, were set from our own data. We'll recheck them as more clients build up history.
• Attribution. We show what changed after the work, not that the work caused it.
• Client-facing reports. The dashboards are for our team; client report exports are on the list but not built.

Whatever you track becomes what you work on. So the question that matters most is whether you're tracking the right thing: the questions your client's buyers actually ask, at the point in their decision when they ask them.

Our own numbers show how much the choice matters. Since mid-October, our branded questions named Traction in 98% of answers; our non-branded questions named us in 20%. Same business, same weeks — the questions alone made the difference.

So before you compare tools, or start building one, decide what to track:

1. Whose questions? A real buyer persona, not "anyone searching".
2. What are they trying to get done? The goal behind the question — "fix our SEO", "find a Google Ads agency we can trust".
3. Where are they in the decision? An early, problem-aware question and a "who should we hire" question need different answers, and success looks different at each.
4. Is there real demand behind it? Evidence from Keyword Planner, Search Console or Ads search terms — not a guess, and not a volume an AI made up.
5. Is it branded or non-branded? A question that already contains your client's name will flatter every visibility measure. Track branded questions for a different reason — to check the AI has the facts right.
6. Which measure matters at that stage? Pulse weights naming most at the early stages and recommendation most at evaluation and decision. These weights are our judgement (see "What we haven't solved").

That's why Pulse starts from personas, buyer journeys and stages, builds its question suggestions from real search evidence, and weights the score by stage (see questions 2 and 3 above).

Once you know what you want to track, any tool you're considering should be able to answer these:

• Can I track my questions — by persona and stage — or only the vendor's prompt library?
• How many answers sit behind each number, and do you show it?
• Is web search switched on, and do you record when it wasn't used?
• How do you know a mention is really my client?
• What happens to the trend when I add questions, or an engine changes?
• Can I get from a finding to a task, and from a task to a measured result?

These are the questions we had to answer for Pulse.

• SparkToro and Gumshoe study, reported by Search Engine Journal, January 2026 — 2,961 runs across 12 prompts and 3 platforms. Not peer reviewed; Gumshoe sells AI tracking.
• Digiday, 1 May 2026, "Marketers question expensive AI visibility tools as inconsistent results fuel skepticism".
• Surfer, "LLM scraped AI answers vs API results" (vendor study).
• OpenAI survey reported by IT Brief NZ, 3 September 2026.
• Ahrefs, May 2025, 75,000 brands — branded web mentions 0.664 vs backlinks 0.218 (correlation, not cause).
• Peec AI, March 2026, 30M sources (vendor study).
• Vendor pricing pages, checked 6 November 2026: Otterly.AI · Peec AI (currency not named on the page) · Profound (agency plan) · Scrunch · Semrush · Ahrefs (Brand Radar · pricing) · SE Ranking. Monthly billing unless stated.
• Published pricing, read 6 November 2026: OpenAI web search US$10 per 1,000 calls; Anthropic web search US$10 per 1,000 searches; Google Gemini grounding US$14 per 1,000 requests after the free allowance.