How to Run Test Conversations for an AI Visibility Audit

Run AI visibility test conversations with a versioned scenario library, documented execution rules, and one complete record for each conversation. Preserve the questions, provider responses, sources, run conditions, and reviewed outcomes so another person can inspect the result and assess whether later cycles are comparable.

Key Takeaways

• Define a versioned scenario library and documented execution rules before running tests.
• Keep a frozen benchmark for comparison and a separate exploratory set for new questions.
• Create one record per conversation, with its complete evidence, provider surface, run conditions, and reviewed outcomes.
• Keep brand references, qualified visibility, recommendations, citations, and retrieved sources separate.
• Compare cycles using documented versions, conditions, repetition counts, and review rules. Use the quality checklist before reporting findings.

---

Introduction

This guide provides a practical protocol for teams running an internal AI visibility audit. It covers scenario setup, execution, conversation logging, review, and comparison, with a downloadable workbook and a process-quality checklist.

It continues the preparation covered in brand discovery, persona design, intent mapping, and question development.

---

1. Define the scenarios and version the question library

For each scenario, record the target brand, persona, buyer intent, question context, approved opening questions, conversation goal, follow-up rules, and stop condition. Assign identifiers to the scenario and each approved question version.

Separate the library into two sets:

• Frozen benchmark: the approved questions and variants used for comparison over time.
• Exploratory: new questions, competitors, capabilities, or hypotheses that may inform a future benchmark version.

Preserve previous wording, effective dates, and the reason for each material change. Promote exploratory questions into a new benchmark deliberately, with their earlier records still identifiable.

Worked example: setting up a scenario

This example is illustrative, not a measured client result.

Choose the turn cap and testing depth for the scenario and decision. This example does not establish a required number of variants, turns, or repetitions.

---

2. Run each scenario on the intended provider surface

Identify the provider, product surface, and interface being tested. A consumer chat product, provider API, conversational-search interface, and search-answer block should remain distinguishable in the records.

For an API test, preserve exposed model, tool, retrieval, and system-instruction settings. For a consumer interface, record observable model information, locale, account or personalization state, retrieval state, and whether the test starts a fresh conversation. Mark unavailable conditions as unavailable instead of inferring them from the answer.

Apply the approved question and variant plan consistently. When the surface supports conversation, write follow-ups from the persona’s goal and the answer received. Preserve each follow-up so another reviewer can see how the conversation developed.

Stop when the scenario’s goal or turn cap is reached. Record the reason. For a search-answer surface, preserve the generated answer and sources when present. Record an absent answer block or failed test separately from an evaluated answer in which the brand is absent.

Automation can help submit questions, assign identifiers, capture timestamps, and save evidence. Review remains necessary for ambiguous outcomes and follow-ups that depend on the meaning of an answer. Choose an execution method that preserves the surface and rules being tested.

---

What should we define and log from each AI conversation for AI visibility testing?

Define the scenario, approved question version, provider surface, execution rules, and outcome definitions before testing. Log each complete conversation, available citations and retrieved sources, run conditions, reviewed brand treatment, and comparison limits under a stable conversation ID.

Create one record for each conversation on a defined surface and repetition. Give reruns separate conversation IDs. Individual turns belong to that conversation’s evidence; they should not become additional conversations in the denominator of a conversation-level metric.

The scenario register preserves the approved question and buyer context. The conversation log connects that question version to what actually happened:

Keep complete transcripts and source records in the log or in linked evidence files. A summary can accompany them, but it should not replace them.

Distinguish “none observed” from “unavailable.” For example, an answer with no visible citations differs from a surface whose citation evidence was not captured. For unknown numeric fields or timestamps, leave the cell blank and explain the gap in the review notes.

The source ecosystem study provides an example of reporting saved citation and search-result evidence with coverage and interpretation limits.

---

4. Review outcomes using shared definitions

First establish whether the record contains an answer eligible for evaluation. Keep no-answer records and failed tests separately identifiable, and leave their answer-based brand outcomes unevaluated.

For evaluated answers, record brand appearance, buyer-introduced versus independently surfaced treatment, evidence state, qualified visibility, recommendation, citations, and competitor treatment separately. A citation does not establish a brand recommendation, and a name in a list does not automatically establish qualified visibility.

Apply the written criteria in What Is Qualified Visibility in AI Search?. Keep ambiguous distinctions unresolved and attach the passages supporting each classification.

Before comparing reviewed results, have reviewers classify a shared sample, discuss disagreements, and preserve agreed interpretations. Record changes to the definitions. If those changes affect earlier classifications, determine which records require another review.

---

How do we run an internal AI visibility audit in a way that keeps scenarios and records consistent over time?

Keep a versioned scenario register, a frozen benchmark question set, shared outcome definitions, and complete records for each provider surface and testing cycle. Preserve historical wording and review rules, record changed conditions, and compare matched eligible conversations before reporting a trend.

Before reporting change, check what remained fixed and what changed:

Keep exploratory results outside the benchmark trend until introduced through a new benchmark version. Compare matched groups of records where possible, and make missing or changed coverage visible.

Plan another cycle when implementation work, provider or market changes, or a defined decision warrants testing. Set the cadence and repetition depth for that purpose rather than treating a particular schedule or run count as universally sufficient.

Describe the outcome within the tested questions, surfaces, and conditions. A changed result after a content update does not, by itself, establish that the update caused it.

---

Can you recommend a checklist for auditing the quality of our internal AI visibility process?

Use a checklist covering scenario versioning, provider surfaces, recorded conditions, complete evidence, execution rules, reviewer consistency, and comparison limits. Identify missing evidence or changed conditions, and resolve gaps that affect the conclusion or state their limits before reporting results.

Use this checklist before reporting findings or comparing cycles:

• Scenarios are defined: each has a buyer context, goal, approved questions, follow-up rules, and stop condition.
• Versions are preserved: benchmark, exploratory, question, and definitions versions remain identifiable without overwriting historical records.
• Surfaces are explicit: the provider and product interface match the scope being reported.
• Conditions are recorded: observable settings and timestamps are saved, with unavailable information identified.
• Evidence is complete: each record retains the full conversation and available source evidence, linked through stable IDs.
• Execution is documented: variant use, repetitions, stopping reasons, and buyer introduction of the brand can be inspected.
• Unevaluated records are identifiable: failed tests and absent answers are separated from evaluated brand absence.
• Review is traceable: classifications use written definitions, evidence passages, reviewer annotations, and calibrated interpretations.
• Comparison limits are visible: version changes, coverage, denominators, and changed conditions are disclosed before interpreting a trend.

Resolve gaps that affect the reported conclusion or state the resulting limitation alongside the finding. Checklist completion does not establish a universal reliability threshold.

---

What a Complete Conversation Can Reveal

The historical Freshdesk examples show why full conversations and provider boundaries matter. For one early-stage question, ChatGPT returned a framework without naming a platform, while Claude named Freshdesk in a peer list. Those were separate observations from defined test conditions.

Another recorded conversation developed as follows:

The third turn should remain distinguishable from treatment before the buyer named the brand. Preserve those turn-level distinctions when applying the conversation-level review rules. These historical examples do not establish a required provider set, run count, or result for another brand.

---

Frequently Asked Questions

How many reruns does each question need?

Set the repetition plan for the decision and scenario before testing, and record the actual counts by question variant and provider surface. There is no universal count that makes a result reliable. Use repeated observations for consequential findings and disclose the limits of small or uneven samples.

Are API and consumer-interface answers interchangeable?

Treat them as different test surfaces unless your evidence supports a specific comparison. Record the interface and exposed settings for each result, and keep their findings identifiable rather than pooling them as the same buyer experience.

Does a failed test count as brand absence?

No. A failed test or absent generated answer cannot establish how the brand was treated in an answer. Report it as unevaluated, preserve the reason, and show evaluation coverage alongside rates calculated from eligible answers.

What if the provider does not expose search results or citations?

Record that evidence as unavailable. If a complete answer was captured, its brand treatment may still be evaluated, but source-level findings must stay within the evidence the surface exposes. Do not turn missing source records into a claim that no sources were retrieved or cited.

---

Use the AI Visibility Testing Log

Download the AI visibility testing log to put this protocol into practice. The workbook contains:

• Scenario register: approved question versions, buyer context, variants, goals, follow-up rules, stop conditions, and testing plans.
• Conversation log: one row per conversation, with run conditions, evidence records, reviewed outcomes, and comparison limits.

Both sheets include illustrative entries, blank rows, and field definitions. The conversation examples show an evaluated answer with no brand reference and a separate unevaluated no-answer record. Replace them with your own scenarios and test evidence.

Once the records are reviewed, continue to How to Turn AI Visibility Test Data Into Actionable Findings. See the complete audit process for how the preparation, testing, and analysis steps connect.

If you would rather see what your brand’s test results look like before running this yourself, fill out the form below.