What this competency actually requires

A/B Testing is the professional capacity to design randomized experiments that yield trustworthy, ethical, and decision-relevant evidence. Competence is visible when a practitioner can explain where the evidence came from, why the method fits the decision, which assumptions carry the conclusion, and how uncertainty should affect action.

The intellectual foundation includes Fisher’s randomization logic and the large-scale online experimentation methods developed by Kohavi and colleagues (Fisher, 1935) (Kohavi & Xu, 2020).

Research foundations

Good analysis begins before a model is selected. A reproducible workflow therefore preserves source data, documents transformations, versions code or formulas, and separates exploratory work from confirmatory claims (Kohavi & Xu, 2014).

Uncertainty has several layers. Sampling error is only one. They report practical magnitude and plausible ranges, examine exceptions, and identify conditions under which the recommendation would change (Simmons & Simonsohn, 2011).

Ethical reasoning is part of analytical quality.

A defensible working method

Write the hypothesis, mechanism, primary metric, guardrails, unit of randomization, minimum detectable effect, sample requirement, duration, and stopping rule before launch. Validate assignment and instrumentation; monitor sample-ratio mismatch; analyze by assigned group; and report uncertainty and practical impact.

Before delivery, another qualified person should be able to reproduce the main result and challenge the strongest assumption. It should state limitations in language proportionate to the decision rather than burying them in technical notes.

1. A/B Testing framing: define the decision, outcome, unit of analysis, time horizon, and cost of error.
2. Audit: verify provenance, definitions, permissions, missingness, and measurement quality.
3. Analyze: establish a baseline, choose methods by assumptions, and retain a transparent trail.
4. Challenge: test alternatives, edge cases, sensitivity, and subgroup performance.
5. Act and learn: assign ownership, monitor consequences, and revisit the model when conditions change.

Failure modes and professional judgment

Peeking, metric switching, novelty effects, interference, attrition, and unplanned subgroup searches can manufacture a persuasive but unstable win. In A/B Testing, these defects are consequential because they can redirect resources and distribute harm.

A mature practitioner actively searches for disconfirming evidence. When evidence is weak, the correct output may be a smaller decision, a reversible pilot, better measurement, or no recommendation yet (Imbens & Rubin, 2015).

Credibility grows from traceability rather than certainty theater.

Evidence of mastery

Evidence of A/B Testing mastery includes a well-scoped question, data dictionary, assumption register, reproducible analysis, appropriate validation, plain-language decision memo, and post-decision monitoring plan. It also shows restraint: the practitioner knows which conclusions are supported, which are provisional, and which remain outside the evidence (American Statistical Association., 2016).

For leaders, the practical test is whether the competency improves decisions over repeated cycles.

Questions for review

Before approving work in A/B Testing, reviewers should ask: Is the decision question explicit? Are the data and definitions fit for that question? Could another analyst reproduce the result? Were reasonable alternatives tested? Are uncertainty and limitations visible where the recommendation appears? Is the action proportionate to the strength of evidence? For A/B Testing, these questions turn quality assurance into a practical discipline rather than a ceremonial sign-off.

The review record should preserve the original question, source dates, exclusions, transformations, analytical choices, contrary findings, reviewer comments, and final disposition.

Evidence, controls, and decision use

A/B testing estimates the causal effect of an intervention by randomly assigning eligible units to alternatives. Before launch, specify the hypothesis, unit of randomization, eligibility, exposure definition, primary metric, guardrails, minimum detectable effect, power, duration, and stopping rule. Controlled online experiments require trustworthy assignment and instrumentation; a sample-ratio mismatch is a diagnostic warning, not a minor reporting issue (Kohavi, 2009). Analyze participants in their assigned groups unless the design justifies another estimand. Report effect size and confidence interval, not only significance. Check novelty, learning, weekday cycles, and interference, and retain null or harmful results. A decision record should explain whether the observed effect is practically meaningful and whether the tested population and implementation match rollout.

Applied review protocol

Use an experiment review packet with the assignment code, eligibility query, exposure event, metric definitions, power calculation, planned duration, and analysis script. Before reading treatment effects, test allocation ratios, pre-treatment balance where relevant, logging completeness, and whether any unit appeared in multiple variants. Interpret the primary outcome together with guardrails for reliability, accessibility, complaints, revenue quality, or longer-term behavior. If the result is inconclusive, report the interval and decide whether more information is worth the delay; do not relabel it a win based on a secondary metric. Archive the design and result in an experiment registry so later teams can find null effects and avoid repeated testing of the same weak idea.

Limitations and misuse

Repeated peeking, optional stopping, many variants, and many outcomes inflate false positives unless handled by the design or analysis. Bots, caching, cross-device exposure, and network spillovers can break assumptions. Short tests may miss retention or downstream harm. Do not generalize beyond the tested treatment, population, and period without additional evidence.