Holdout testing

Holdout testing is an experimental project: we design a correct test with a control (holdout) group — at the user, segment level or via ad systems' platform holdout tools; we define metrics, compute the needed size and duration for statistical power, run it and analyze the incremental effect (lift) with a confidence interval and significance. Applicable to ads, CRM campaigns, mailings, discounts. Honestly about the nature, this is key: even a correct holdout gives a PROBABILISTIC result with an interval, not an exact number; the conclusion is valid for the test's audience, channel and period and need not repeat in another context. Honestly about the method's cost: a holdout means deliberately NOT acting on part of the audience during the test — a 'forgone' effect there, the price of reliability; in return you stop paying for the non-incremental. Honestly about requirements: a sufficient audience volume for significance and the ability to technically isolate a control are needed; on small volumes the test will not work. Honestly about discipline: the size and duration are set in advance, you cannot 'peek' and stop on a random spike — otherwise false conclusions. Honestly about data: correct conversions and a clean group split are needed (garbage in, garbage out). Honestly about the essence: this is effect measurement, NOT running campaigns and NOT a growth guarantee — the test says whether the intervention works, while decisions are yours. An important boundary: this is an audience/user holdout, not geo-experiments (by regions — separate), not attribution/MMM (correlational — separate) and not running ads/CRM. If the audience is small or a control cannot be isolated, the test is premature. Picture this: instead of 'the mailing brought 100 orders' you learn how many of them are really thanks to it and how many would have happened anyway. The base price starts from 30,000 ₽ per test; it depends on design and scale.

• You do not know what really works versus what the system credits.
• You pay for conversions/orders that would have happened without the intervention.
• You assess the effect of mailings, discounts and campaigns by gut feel.
• There is no control group for an honest comparison.

• Holdout test design (users/segment/platform holdout)
• Defining metrics and computing size/duration for power
• Clean isolation of the control group
• Running and controlling test cleanliness
• Analyzing incremental effect (lift) with an interval and significance
• A report with the result, limitations and validity conditions
• Reviewing results with you
• Recommendations for next tests

• A causal estimate of the intervention's effect (with an interval)
• You understand what works and what is just credited
• Savings on the non-incremental in the future
• A base for decisions and next tests (growth — separately)

• We clarify the hypothesis, audience, metrics, scale; plan the design
• We isolate the control, run it, do not 'peek'
• We analyze lift and significance, compile a report, review with you

• Will a holdout give an exact and final answer?

  No. It is a probabilistic result with a confidence interval, valid for the test's audience, channel and period. In another context the effect may differ — it is the best causal estimate, not an eternal guarantee.

• Is the control group lost orders?

  Partly: a holdout means deliberately not acting on part of the audience during the test — a 'forgone' effect there for measurement cleanliness. In return you stop paying for the non-incremental in the future. A sufficient volume is needed for significance.

• How is it different from geo-experiments?

  A holdout splits the audience/users; geo-experiments split regions (separate) when there is no user-level data. Both are causal methods; the choice depends on whether a control can be technically isolated at the people level or only geography.