Attribution and incrementality answer different questions. Attribution asks "which campaign should get credit for this observed install?" — a credit-assignment problem. Incrementality asks "how many of these installs happened BECAUSE of the campaign, that wouldn't have happened otherwise?" — a causal-impact problem. Attribution can give credit to a campaign that contributed nothing to the install (the user would have installed anyway from organic). Incrementality measures only the actual causal lift.
The standard mechanism is a holdout test
- Define a target audience (e.g., users in a specific country and demographic).
- Randomly split into control group (sees no ads from this campaign) and treatment group (sees the campaign).
- Run for a meaningful period (typically 2-6 weeks).
- Measure install / conversion rates in both groups.
- Incremental conversions = treatment conversions − control conversions.
The key constraint: the control group must genuinely not see the ads. Easy in walled gardens (Meta, TikTok, Google) where the platform can guarantee exclusion. Hard for cross-network programmatic where you can't easily exclude users.
The brutal truth most incrementality tests reveal: a substantial fraction of "attributed" installs are not actually incremental. The user would have installed anyway — from organic search, App Store featuring, word-of-mouth, brand awareness — and the campaign just happened to be present in the journey. Typical incrementality tests find 30-60% of attributed installs are organic. For mature brand-driven apps the fraction can be even higher.
Types of incrementality tests
- Geo-based holdout: hold out certain geographic markets, run campaigns in others, compare. Easy to implement, less precise (geo-level confounds).
- User-level holdout (PSA / ghost bids): platform serves a public-service ad (PSA) to the control group while the treatment group sees your real campaign. Same impression exposure, different creative. Cleanest causal estimate; Meta and Google support this natively.
- Conversion-lift studies: walled gardens (Meta, Google) offer first-party tools that randomize at user-level inside their platform and report lift directly.
- Cross-channel incrementality: hardest — requires user-level identifier across channels post-ATT, which iOS makes difficult.
When incrementality matters most
- Big budget allocation decisions: "Should we add $5M / month to Meta UA?" Run incrementality test before committing.
- Retargeting / re-engagement campaigns: these are most likely to claim credit for already-engaged users who'd return anyway. High organic-claim rate.
- Brand campaigns: where attribution metrics are weakest, incrementality is the only meaningful measurement.
- Channel-mix evaluation: comparing "$X of Meta vs $X of TikTok" makes sense only with comparable incrementality measurement.
When incrementality is overkill: small-budget tactical decisions, individual creative tests, daily bid adjustments. Stick with attribution for those.
Incrementality test types compared
| Test type | How it works | Precision | Where it works |
|---|---|---|---|
| Geo holdout | Hold out entire markets; compare to live ones | Lower — geo-level confounds | Any channel |
| User-level (PSA / ghost bids) | Control sees a public-service ad; treatment sees the real campaign | Highest — same exposure, randomized | Meta, Google |
| Conversion-lift study | Platform randomizes users internally and reports lift | High | Walled gardens |
| Cross-channel | User-level holdout spanning networks | Hard to achieve post-ATT | Limited on iOS |
User-level holdouts give the cleanest causal estimate; geo tests are easiest to run but least precise. Post-ATT, cross-channel incrementality is the hardest to execute because user-level matching across networks broke.