GuidesTrack and report

Incrementality testing: how to run a test

Incrementality testing measures the conversions your marketing caused, by comparing a group that saw it with a similar group that did not. Pick one question, choose a holdout or geographic design, run it long enough to reach a stable result, and compare outcomes. Attribution tells you who got credit. This tells you what changed.

Updated November 16, 20266 steps5 min read

Check my tracking →

$1 today. Cancel anytime before day 7.

Incrementality test, step by step
  1. 1Pick one questionDoes this channel add sales we would not have had?
  2. 2Choose a designUser holdout, geo split, or platform lift study.
  3. 3Size and set durationEnough conversions and long enough to see a signal.
  4. 4Run it untouchedNo budget or creative changes mid-test.
  5. 5Compare outcomesMeasure the difference between test and control.
  6. 6Act on the liftAdjust budget or targets, then repeat.
On this page
  1. What incrementality means
  2. Choose a design
  3. Size it and run it clean
  4. Read the result
  5. When not to run a test
  6. Use it, and know the limits
  7. Questions

What incrementality means

Incrementality is the extra outcome caused by a marketing activity, over what would have happened anyway. If you pause retargeting and sales barely move, retargeting was claiming credit for customers who were coming back regardless.

This is the difference from attribution, which assigns credit to touchpoints a customer already had, and from an A/B test, which compares two versions of something that both run. An incrementality test compares running against not running. The glossary entry on incrementality covers the term, and iROAS is the return measure it produces.

The practical use is to correct over-crediting. Branded search, retargeting and email often show strong platform results because they reach people already close to buying. A test shows how much of that is caused and how much is harvested.

You will also see the term used for individual campaigns, audiences and even creatives. The logic is identical: compare what happened with the activity against a credible picture of what would have happened without it. What changes is the cost and difficulty of building that comparison.

Choose a design

A user-level holdout randomly withholds ads from a share of eligible people and compares their conversions with the exposed group. Meta and Google both offer conversion lift studies that do this inside the platform, usually with eligibility rules on spend and conversion volume.

A geo test splits regions: ads run in some markets and are held back in matched ones. It works when you cannot randomize people, and for channels like TV or out-of-home, but needs enough markets that behave alike. Platform-run lift studies are the easiest start. Geo tests are the most platform-independent.

Some teams use a simpler approximation: turn a channel off for a few weeks in a market and watch total sales. It is rough, because other things change too, but it is better than trusting platform reports alone. A matched-market design makes it more rigorous.

Is your tracking counting every conversion?

Runs tracking health checks on your pixels and tags. Sends a weekly performance review.

Size it and run it clean

Decide the smallest lift you would act on, then check you have enough conversions in each group to detect it. Small accounts often cannot power a test on a niche campaign, so test the biggest budget line first.

Run it for at least one full buying cycle, often several weeks. Freeze budgets, creative and targeting for the test’s channel during the window, and write down any promotion or outage so you can explain surprises.

Document the plan before the test starts: the hypothesis, the groups, the dates, the metric and the lift that would change your decision. Writing it first keeps you from choosing a flattering reading once the numbers arrive.

Read the result

Lift is the difference in conversions per person (or per region) between test and control, divided by the control rate. Look at the confidence interval, not just the point estimate. A result of “+8 percent, but the range includes zero” is not a win.

Convert it to cost per incremental conversion: spend in the test divided by the extra conversions. Compare that with the platform’s reported cost per conversion. The gap is how much of the platform’s credit is real.

Be careful with small differences. A lift of two or three percent may be real and still not worth acting on, once you consider the cost of the channel. Decide ahead what counts as meaningful, and treat inconclusive results as information too.

When not to run a test

Do not run one when volume is too low. If a campaign produces a handful of conversions a month, a holdout will not tell you anything with confidence. Test a larger spend line, or run a longer period.

Do not run one when you cannot hold the other variables steady. A big promotion, a product launch or a price change in the middle of the window muddies the result. Pick a quiet period.

And do not treat a single test as permanent. Marketing effects change with creative, season, competition and audience saturation. Retest your biggest lines once a year, or after a major change.

If you cannot test, use cheaper checks: compare branded and non-branded performance, look at what happens to total sales when a campaign pauses for a planned reason, and watch new-customer share. They are weak evidence, but better than none.

Use it, and know the limits

Use the result to move budget, set a target, and calibrate your attribution numbers. Retest after big changes. Marketing mix modeling answers a broader question across many channels and is covered under MMM.

Tests have limits: one-off, one channel, one period. Sprites reads your connected channels and can set up a recurring performance review, so the lift you measure can be compared with live platform numbers over time.

Share the result in plain language: “Pausing this campaign would have cost us about this many sales a month, at this cost each.” Executives act on that sentence faster than on a lift percentage and a confidence interval, though you should keep both on file.

Questions

What is the difference between A/B testing and incrementality testing?

An A/B test compares two versions of something that both run. An incrementality test compares running a campaign against holding it back. The first picks a winner. The second shows whether the spend added any sales.

How to prove incrementality?

Randomize who sees the marketing, keep a comparable group that does not, and measure the difference in conversions. If you cannot randomize people, use matched geographies.

Does MMM measure incrementality?

It estimates it statistically from historical spend and sales across channels. A holdout test measures it directly for one channel. Teams often use tests to check and tune the model.

What does incrementality testing mean?

It means measuring the sales or leads your marketing actually caused, by comparing exposed and unexposed groups, rather than counting all conversions that followed an ad.

Keep going

Have Sprites check your tracking

  • Runs tracking health checks on your pixels and tags
  • Reads GA4, Search Console and every connected ad account
  • Sends a weekly performance review