Holdout control

A holdout control is a group of items deliberately left untouched by a change, so that their movement can be compared with the items that were changed.

Example

You rewrite the pages that target 20 prompts and leave 10 similar prompts alone. Visibility then rises on both groups after an engine update. Much of the rise is likely the update rather than your rewrite, and the difference between the two groups is a better estimate of what the rewrite did.

Why it matters

AI engines change underneath you through model updates and retrieval changes. Without a control group, any engine-wide shift during the measurement period is credited to your work, in either direction. Comparing the change in the treated group with the change in the control group, a method known as difference-in-differences, removes much of that shared movement.

A control group works best when it resembles the treated group and its selection is fixed in advance, so it cannot be chosen after the fact to flatter the result.

How Stellarcast handles it

For each published fix, Stellarcast holds out about a fifth of the prompt set as controls: prompts in the same engine and market that the fix does not touch. The selection is deterministic, so the same fix always gets the same holdout, and lift is calculated as the change in treated prompts minus the change in controls. Each result carries a MEASURED, LIKELY or TOO-EARLY label.

Related terms

• Attribution window
• P25-P75 range
• Answer variability
• AI share of voice
• How Stellarcast measures AI visibility