Evaluation & Decision Checklist

How to use this

Start after Build & Launch has handed over the verified launch record. Validate the result only when the planned measurement window or a valid intervention condition has been reached. Decision implementation can be confirmed before the later post-implementation read, so complete items when their campaign event occurs. Progress is saved in this browser on this device and is not shared. Use Reset for a new campaign only when the current campaign is fully recorded.

Commercial impact standard

Normalize attributed revenue before projecting it. Use eligible visitors when the experiment platform provides a trustworthy visitor denominator. Use eligible sessions consistently when visitor data is unavailable, and record why.

90-day projected incremental revenue = (winner attributed revenue per eligible visitor - control attributed revenue per eligible visitor) x forecast eligible visitors over the next 90 days

Example: a winner at $3.50 per eligible visitor versus a control at $3.20 produces a $0.30 difference. Across 200,000 forecast eligible visitors, the central 90-day projection is $60,000. This is an illustrative conditional estimate, not a benchmark, secured result, or guarantee.

Use 90-day projected incremental revenue only when the winner gates and the revenue source, assignment, denominator, and comparison are valid. Use 90-day revenue scenario when the result or revenue effect is still uncertain. Use No commercial impact claim when the experiment or revenue data cannot support a responsible calculation.

Use this checklist to evaluate the campaign against its locked measurement plan, choose the evidence-supported decision, confirm the decision reached production, and carry the learning into the next campaign.

Reset for a new campaign

1. 1. Validate the result

   Confirm tracking and variant allocation remained valid Record the actual date range, exposure, sample, conversions, and primary and guardrail results Log promotions, traffic shifts, outages, campaign changes, stock issues, and tracking issues Confirm whether the planned duration and sample gates were met Confirm whether the result holds on mobile Label incomplete evidence as directional rather than a winner Apply the pre-agreed decision rules without moving the goalposts

   Required output: A clean, caveated result read using the pre-agreed decision rules.

2. 2. Quantify commercial impact

   Record control and winner attributed revenue, eligible visitor or session counts, attribution source and definition, and currency Calculate attributed revenue per eligible visitor for each arm, or revenue per eligible session when visitor data is unavailable, and record the absolute difference without mixing denominators Forecast the next 90 days of eligible traffic from a named complete baseline and record adjustments for seasonality, promotions, stock, traffic mix, and other known events Calculate the central 90-day commercial-impact projection with the pre-agreed formula and eligible traffic only Record the lower and upper impact range from a defensible experiment confidence interval where available; otherwise state that the range is unavailable and document the material caveats Label the result as 90-day projected incremental revenue, a 90-day revenue scenario, or no commercial impact claim, without presenting forecast revenue as secured or guaranteed

   Required output: One reproducible commercial-impact record with source values, denominator, formula, central estimate, available range, assumptions, caveats, currency, and label.

3. 3. Decide and compound

   Choose Roll out, Keep control, Iterate, Re-run, or Monitor Record the decision, reasoning, commercial-impact label and projection where applicable, and next owner For a valid winner, approve it as the live baseline and new control; otherwise confirm the current control remains the baseline Pull a post-implementation read after about two weeks and schedule a realized 90-day comparison for shipped winners when data remains available, or record why either follow-up does not apply Feed losses and near-misses into the next test for the same angle Move to the next angle only when the current angle is exhausted Record the learning, implementation status, caveats, and next action in the campaign or round log

   Decision output: The evidence-supported decision, reasoning, next owner, and required production action are recorded.

4. 4. Confirm the decision was applied

   Confirm the assigned implementation owner has acknowledged the recorded decision and required action Apply the recorded decision to production: roll out the winner, retain the control, implement the approved iteration or re-run, or keep the monitored state Verify the live audience, variant, and configuration match the recorded decision Remove obsolete experiment code where applicable and confirm tracking remains valid Record the final live state and effective time for the post-implementation read

   Close gate: Production matches the documented decision, and the result, caveats, implementation status, final live state, and next action are recorded.

Refresh data for the next campaign