In brief: A useful A/B test compares simultaneous variants on randomly assigned groups. Before launch, define the hypothesis, primary metric, useful effect size, sample size and stopping rule. If too few eligible participants are available, start with customer observation and confirmed usability problems. A non-significant result does not prove there is no effect.
A team changes a headline, sees more enquiries over three days and declares a winner. The improvement disappears the following week. The message might have helped, or the audience might have changed. A planned comparison is needed to separate these explanations.
Test when the answer can change a business decision. Button preferences matter less than an unexplained delivery charge, but an attractive hypothesis still needs evidence.
How many participants does a test need?
Sample size depends on baseline conversion and the minimum detectable effect, or MDE. Choose it before launch, based on the value and cost of implementation. Moving from 2% to 2.4% is 20% relative growth and 0.4 percentage points, not 20 percentage points.
These planning estimates assume two independent, equally sized groups, a two-sided significance level of 5%, 80% power and a normal approximation for two proportions without continuity correction. Counts are rounded up to the next hundred. They are calculated estimates, not industry requirements or guaranteed outcomes.
| Baseline → target CR | Participants per variant / total |
|---|---|
| 1% → 1.2% | 42,700 / 85,400 |
| 2% → 2.4% | 21,200 / 42,400 |
| 3% → 3.6% | 14,000 / 28,000 |
| 5% → 6% | 8,200 / 16,400 |
| 10% → 12% | 3,900 / 7,800 |
The outcome here is whether an independent participant converts. Revenue, repeated purchases, multiple variants and different statistical methods need their own calculation. Repeated visits by one person cannot simply count as independent new people.
With 10,000 eligible participants a month and a 50/50 split, the 2% example needs more than four months just to recruit the sample, plus any conversion delay. Whole-shop traffic is not the same as traffic eligible for a checkout or landing-page test. Microsoft’s preparation framework covers goals, audience, metrics and power; the table is our calculation under the stated assumptions.
Four errors that make results unreliable
Stopping when the percentage looks good. A fixed-horizon test needs its planned stopping rule. Two to four weeks may cover relevant weekly behaviour, but cannot substitute for sample size. Sequential analysis allows different monitoring rules when selected and implemented in advance. A purchase or B2B sale may also arrive after the visitor’s session.
Not knowing what changed. A new headline, image and form together test a package. That is legitimate if the decision concerns the whole proposition. To isolate one element, use a separate experiment or an appropriate factorial design.
No business hypothesis. Write the reason: removing a question that is unnecessary before first contact may help qualified visitors complete the form because they no longer have to find unavailable information. Select the target effect from the economics, rather than promising a convenient percentage.
Unequal measurement. One variant may load analytics later or lose payment events. Compare expected allocation with observed group counts. Sample Ratio Mismatch is a warning to investigate the assignment or data process before trusting an effect estimate. Keep assignment stable for the chosen unit.
Monitor failures and guardrails while the test runs. A critical checkout regression can justify an early stop, but does not make the surviving variant a statistically proven winner. Microsoft’s guidance during experimentation explains why repeated checking needs appropriate statistical handling.
What should be tested first?
Prioritise observed barriers and the cost of testing them. Potential impact is a hypothesis, not a known uplift.
| Element | A concrete reason to investigate |
|---|---|
| Headline and offer | Visitors cannot explain the product or terms |
| Hero image/video | It contradicts the ad or fails to show the product |
| Reviews and cases | A specific trust concern remains before action |
| Form fields | Reproducible abandonment or unnecessary questions |
| CTA wording | People cannot tell what follows the click |
| Page length | Essential answers or the next step are difficult to find |
| Button colour/design | Poor contrast or controls that are hard to notice |
| Footer | Important conditions are hidden where visitors do not look |
Fix accessibility defects whether or not a test is planned. For related work, see the creative testing framework and CPA review.
Two teaching examples
Fashion shop: place a relevant review beside the first purchase action to address an observed trust concern. Keep prices and acquisition conditions comparable. Measure confirmed purchases and returns, then decide from the interval and business value. Use the planned readout to support the decision.
B2B enquiry: compare a long form with clearer stages that preserve entered data. Define a qualified enquiry with sales before measuring completion, and allow for follow-up delay. More forms with poorer lead quality may be a worse outcome. Assess enquiry count and quality together.
Choosing a tool and budgeting the work
Google Optimize and Optimize 360 closed on 30 September 2023. They cannot be included as current free tools.
Compare platforms by editing, URL redirects, developer support, analysis method, collision prevention, exports and QA. The official VWO and AB Tasty pricing links currently lead to Wingify; confirm the current product and quote rather than relying on an old separate price list.
On 28 September 2026, Convert’s public trial page says plans start at $399/month. Treat this USD figure as a dated starting-price example. Verify annual billing, participant limits, features, tax and total cost before purchase; it is not a fixed price for your project.
Custom allocation through GTM or site code also costs development, stable assignment, event instrumentation, QA, maintenance and rollback. Check processing terms, consent, site speed and team access. Neither a paid platform nor a free tag manager makes the experiment automatically compliant or free of visual flicker.
Showing A one week and B the next is a before/after observation. Season, ads, payday and assortment can explain a difference; it is not equivalent to simultaneous random assignment.
When customer observation is the better first step
If the necessary sample takes longer than the decision allows, observe how people select delivery, correct an address, find evidence and submit a form. Heatmaps show interactions and recordings show sequences, not intentions. A few recruited participants can reveal a usability problem; five do not estimate its prevalence reliably. Apply data-processing and recording-masking requirements.
There is no universal ban below 5,000 visitors. Frequent actions or large effects may be measurable sooner, while rare purchases need longer. After fixing a confirmed defect, monitor sales and lead quality; before/after growth alone does not establish the entire causal effect.
What to prepare this week
Today: record eligible traffic, baseline CR, conversion delay and implementation cost. Verify equivalent event collection in both groups.
Tomorrow: write three hypotheses and choose a primary business metric with guardrails for margin, lead quality, speed and returns.
This week: plan one hypothesis, sample, allocation, exclusions and stopping rule. Define actions for benefit, harm and insufficient evidence. If the timeline is impractical, use qualitative diagnosis first. The output should support a decision, not merely display a percentage.
Frequently Asked Questions
How long should an A/B test run?
Until the planned sample and relevant behaviour cycle are covered. Two to four weeks may fit some tests but do not guarantee sufficient data. Include weekends, seasonality and conversion delay; the analysis method also determines the stopping rule.
Can several tests run simultaneously?
Yes, with controlled overlap, allocation and interactions. Different URLs do not guarantee independence when one buyer encounters several experiments. Running tests in sequence may be easier for a small team.
What does a non-significant result mean?
The chosen analysis did not provide sufficient evidence of an effect. The effect may be small, the sample inadequate or the change unhelpful. Examine uncertainty and MDE. Demonstrating practical equivalence needs its own criterion.
How does A/B testing differ from multivariate testing?
A/B compares complete variants. A multivariate or factorial experiment studies combinations and interactions. Required traffic depends on the design and analysis, rather than a universal multiplier.
Do we need a developer?
Some text changes can use a visual editor. Forms, payments, stable assignment and events often need technical work. Every approach still needs mobile QA, correct measurement and rollback.
Canem Errant’s CRO work connects an observed barrier to a hypothesis and decision rule. Describe the page or checkout, eligible traffic and confirmed conversion rate in the contact form so we can scope the next step.
