← All analytics projects
03
Project 03 · Experimentation

A/B Test Decision Calculator

Decide whether an observed conversion lift is statistically reliable, commercially useful, and supported by an adequately planned sample.

ExperimentationConfidence intervalsPowerPractical significance
Control rate11.52%
Treatment rate12.80%
Relative lift+11.1%
Two-sided p-value2.01e-18z = 8.757
Experiment workspace

Test the observed result

Uses a two-sided pooled two-proportion z-test and an unpooled confidence interval for treatment minus control.

Cumulative experiment counts

Edit all four values; results update immediately.

Effect and uncertainty

Absolute lift+1.28%
95% CI low+0.99%
Incremental conversions1,280

95% confidence interval: [+0.99%, +1.57%]

Observed conversion rates

Control11.52%
Treatment12.80%
Pre-registration

Plan the next sample before seeing the result

Equal-sized groups, two-sided α = 0.05, normal approximation to the two-proportion test.

Sample-size assumptions

Required sample per arm14,745

Total visitors: 29,490. No adjustment for attrition, repeated peeking, unequal allocation, or multiple metrics.

Method

What the calculator assumes

Core assumptions

  1. Independent randomized users
  2. Concurrent experiment variants
  3. Pre-specified conversion event and window
  4. Adequate expected successes and failures
  5. No unplanned repeated stopping

Controlled decision brief

This Vercel edition generates no live LLM prose. The decision label is a deterministic rule applied to the displayed statistics and your practical threshold.

Confirm guardrail metrics, randomization quality, and experiment duration before rollout.