A/B test causal impact analysis
A valid A/B test is more than a significant p-value. The causal story depends on clean assignment, stable exposure, and a decision rule the business can act on.
Assignment integrity
Check randomization balance, bucketing stability, cross-device identity, and excluded traffic before interpreting lift.
Outcome integrity
Separate leading indicators from durable outcomes. Confirm that metric movement is not caused by broken logging or delayed ingestion.
Business interpretation
Translate effect size into revenue, retention, or risk terms, then compare the result against rollout cost and guardrail damage.
Common failure modes
- Users receive both variants through multiple surfaces.
- Stopping early after a favorable interim read.
- Optimizing one metric while guardrails silently degrade.