Rules
Part of Reading ecommerce conversion optimization with a skeptical eye
Ecommerce Conversion Optimization Checklist for Test QA
Use this ecommerce conversion optimization checklist to QA test assignment, ecommerce events, orders, and checkout error recovery before trusting results.
Use this ecommerce conversion optimization checklist to QA a test before trusting its results. Check that visitors receive the intended variant, events and orders record consistently, and customers can recover from input errors. It applies to live or staged tests, not a general marketing strategy review. If a core record is unverified, fix implementation and rerun affected checks before interpreting results.
Check the test in real customer paths
Exercise the journey from product view through checkout with synthetic and representative records to find implementation defects, not to identify a winning variant. Include repeat visits and relevant cross-device behavior, because results are untrustworthy if exposure or measurement changes unexpectedly. Use this preflight checklist before comparing outcomes.
- Variant delivery:Confirm that eligible visits receive the intended control or variant, and that the rendered experience matches the test definition.
- Assignment persistence:Check that a visitor or other defined test unit does not switch variants unexpectedly on repeat visits.
- Journey coverage:Exercise the affected path through product view, add-to-cart, checkout start, and checkout completion. Include payments, cancellations, returns, and service interactions when the test could affect them.
- Page behavior:Check page speed and the relevant interface at the viewport and journey stages used in the test. A page change that affects crawlable content also needs the ecommerce SEO checklist before testing begins.
- Traffic changes:Log changes to paid ecommerce advertising during the test. A shift in who arrives at the site can look like a variant effect.
A tester such as Optimizely, VWO, or Adobe Target can be checked alongside web analytics such as Google Analytics 4. Compare what the customer actually received with the test and analytics records, including the assigned experience and recorded exposure. A dashboard showing a running experiment is not, by itself, proof that exposure was recorded correctly.
Verify assignment before comparing outcomes
Check assignment records for the defined test unit and confirm that the assigned variant persists across repeat visits. Assignment connects a visitor's experience to the outcome attributed to it, so verify that connection before reading results or treating any observed difference as meaningful.
If a visitor is assigned on one page but an order is measured under a different or unstable identity, the variant-to-outcome link may be unreliable. Check these items for both control and variant before interpreting any outcome. Together, they establish whether assignment, delivery, and recorded exposure describe the same customer experience.
- The recorded assignment identifies the intended test and variant.
- Repeat visits do not produce unexplained changes in variant.
- The delivered experience agrees with the recorded assignment.
- Variant counts match the planned allocation, or a difference has an identified explanation.
- The same person or session is not exposed to overlapping versions in a way the design did not allow.
Treat unexplained imbalance, variant overlap, or inconsistent assignment as a defect to investigate, not as evidence that one experience is winning. Check the test implementation and the records that create the assignment before moving to outcome analysis. If a change is made, repeat the assignment checks against the corrected version before using the records to compare outcomes.
Reconcile ecommerce events and orders
Reconcile ecommerce events and orders by writing down each test outcome's precise event definition and comparing it with site records. Confirm its data source, numerator, denominator, and analysis window, then verify that the event appears in test records and reconciles with relevant orders for the test.
Use the same definitions in the test view and ecommerce analytics, rather than assuming matching labels mean the same thing. Check the ecommerce events by name and confirm that each record matches the measure described in the test plan before comparing outcomes across both versions.
- Product view:Does the record correspond to the product view defined for the test?
- Add-to-cart:Is the event recorded when the customer adds the item, rather than at an unrelated interface action?
- Checkout start:Can the recorded event be tied to the intended checkout journey?
- Checkout completion:Does the event match a completed order, and is it present for both variants?
- Average order value, refund rate, and payment failure:Are the underlying order and payment records available to verify the measure being used?
Inspect missing or duplicate events and orders that do not match the stated definitions. Check cancellations and returns where they affect the measure. If checkout completion is missing for one variant, fail the outcome and data-quality checks and hold the result. Correct the event, then confirm it against order records before comparing totals.
Test checkout error recovery as a customer would use it
Test checkout error recovery as a customer would use it by checking whether messages name the problem and give a next step.
For each input error affected by the test, check the message and recovery behavior in both versions, including whether customers can identify the issue and find a safe correction.
W3C's Understanding Success Criterion 3.3.3 says to suggest a known correction after an input error unless security or purpose would be compromised. WCAG 2.1 tests cover 3.3.1 Error Identification and 3.3.3 Error Suggestion. The criterion provides a check for messages and recovery, not a reason to assume either version is working correctly.
WCAG 2.1 also covers message text through 1.4.3 Contrast (Minimum). For each affected input error, check that customers can identify the issue, find a safe correction, retain completed work, protect sensitive information, and access recovery across devices and assistive methods. Verify the experience in the relevant customer path rather than relying on the text alone.
Checkout error recovery checklist:
- Describes the problem in plain text and identifies the affected field.
- Suggests a valid format or a clear next step when a safe correction is known.
- Preserves unrelated completed information instead of clearing the customer's work.
- Protects sensitive information in both the message and the recovery behavior.
- Places focus where a customer can find and correct the problem.
- Works with keyboard access, a screen reader, zoom, and narrow screens.
Test errors deliberately in both versions, not only the one the team expects customers to use. ADA.gov's web guidance on forms highlights labels, keyboard access, and clear instructions for accessible forms. The UK government's accessible forms guidance also provides an official accessibility reference. If a message is unclear, inaccessible, or gives no safe correction, record the failure and fix the interface before relying on a test that includes it.
Decide whether the result can be trusted
Decide whether the result can be trusted by monitoring customer harm, payment defects, service, returns, fraud, and technical quality during the test. The decision depends on both customer experience and whether the records still represent the test as defined throughout the test.
Reconcile relevant events against analytics records. Stop for a material truth, security, accessibility, or customer-control failure, regardless of performance. A favorable conversion result does not make a broken or harmful experience acceptable, and it does not remove the need to verify the underlying records.
When a check fails, identify the affected part of the implementation, correct it, and rerun the checks that depend on it. For example, if an event definition changes, verify the event in the test records and against orders again before comparing outcomes. If checkout error recovery changes, repeat the input-error checks across the relevant variants and access methods.
Use this release decision checklist to keep checks tied to the result: verify that customers received the intended experience, reconcile the records with the stated event definitions and relevant orders, and hold the result if a core record remains unverified before any rollout decision.
- If an event definition changes, verify the event in test records and against orders again before comparing outcomes.
- If checkout error recovery changes, repeat the input-error checks across the relevant variants and access methods.
- If a core record is unverified, fix implementation and rerun affected checks before interpreting results.
Keep the test plan, code or implementation version, event definitions, quality checks, results, limitations, and decision together so the team can trace what was actually tested. The essential QA question is narrow: did customers receive the intended experience, and do the records accurately describe what they did? If either answer is uncertain, treat the result as unverified rather than as a basis for rollout.
