Incrementality testing

Incrementality testing is an experimental project: we design a correct experiment (geo-holdout, on/off, control and test groups), define metrics and the required size/duration for statistical power, run it and analyze the result — estimating the incremental lift with a confidence interval and statistical significance. Unlike attribution and MMM (correlational estimates), an experiment gives the closest-to-causal estimate of effect. Honestly about the nature, this is key: even a correct experiment gives a PROBABILISTIC result with a confidence interval, not an exact number; the conclusion is valid for the test's conditions, audience and period and need not repeat exactly in another context — it is not an eternal guarantee. Honestly about requirements and the method's cost: a real holdout is needed — i.e. deliberately not showing ads to part of the audience/geo; this is temporary 'forgone' exposure, the price of measurement cleanliness; sufficient scale and traffic are needed for significance — on small volumes the test gives no conclusion. Honestly about discipline: you cannot 'peek' and stop the test on a random spike — that inflates false conclusions; predefined size and duration are needed. Honestly about the essence: this is measurement of causal effect, NOT running ads and NOT a growth guarantee — the test says whether a channel/campaign actually works, while decisions and scaling are done by you. Requirements: the ability to organize a holdout, access to data. An important boundary: these are incrementality experiments, not attribution/MMM (correlational — separate, complement well) and not running ads. If there is no scale/traffic or a holdout is impossible, the test is premature — we say so honestly. Picture this: instead of 'the account credits the channel with 100 sales' you learn how many of them are really incremental and how many would have happened anyway. The base price starts from 35,000 ₽ per test; it depends on design, scale and duration.

• You pay for 'conversions' that would have happened without ads.
• Attribution and accounts overestimate channel effect.
• You do not know the real causal contribution of a campaign.
• You make scaling decisions on correlations, not effect.

• Designing the experiment (geo-holdout, control/test groups)
• Defining metrics, sample size and duration for power
• Running and controlling test cleanliness
• Analyzing incremental lift with a confidence interval
• Statistical significance assessment
• A report with the result, limitations and validity conditions
• Reviewing results with you
• Recommendations for next tests

• An estimate of real causal lift from ads (with an interval)
• You understand what really works and what is just credited
• Scaling decisions on effect, not correlation
• A base for next tests (growth and running — separately)

• We clarify the hypothesis, metrics, holdout feasibility and scale; plan the design
• We run the experiment, control cleanliness, do not 'peek'
• We analyze lift and significance, compile a report, review with you

• Will the test give an exact and eternal answer?

  No. Even a correct experiment gives a probabilistic result with a confidence interval, valid for the test's conditions, audience and period. In another context the effect may differ — it is not an eternal guarantee but the best causal estimate available today.

• Is a holdout lost sales?

  Partly yes: a holdout means deliberately not showing ads to part of the audience/geo during the test — 'forgone' exposure, the price of measurement cleanliness. In return you stop paying for non-incremental conversions in the future. Sufficient scale is needed for significance.

• How is this better than attribution and MMM?

  Attribution and MMM are correlational estimates. An experiment gives the closest-to-causal estimate of effect. They work best together: experiments verify, attribution/MMM scale the picture. The test itself does not run ads or guarantee growth.