A/B Testing and Conversion Optimisation for Business Websites: Deciding with Evidence, Not Taste

Many website decisions are made the same way: someone believes the button should be a different colour, the headline shorter, or the form should have fewer fields. Whoever speaks loudest or holds the highest rank usually wins. The results are sometimes good, sometimes not, and hardly anyone knows for sure why.

A/B testing offers a calmer way: show two versions to different visitors at the same time, then see which more often produces the action you want. Instead of arguing, you let visitor behaviour answer. But A/B testing is also easy to misuse, and many convincing looking results are in fact misleading.

This article explains how to start testing soundly for the websites and online stores of small and medium businesses: when testing is worthwhile, how to frame a hypothesis, what to measure, how long a test should run, and how to avoid wrong conclusions. We give no firm improvement figures, because every site differs.

Summary

• An A/B test compares two versions on randomly split visitors to see which performs better
• Start from problems found through data and observation, not from taste
• Change one thing at a time and define the success measure before the test begins
• Give the test enough time and enough visitors before concluding
• If traffic is too small, improvements based on observation are often more useful than testing

What an A/B test is, in simple terms

Imagine a physical shop that wants to know whether putting a featured product near the door raises sales. The fair way is to arrange two similar shops differently in the same period and compare the results. If you change only one shop and compare it with last month, the result is mixed up with season, promotions, and weather.

An A/B test on a website works on the same principle. Each visitor is randomly assigned to version A, usually the page currently running, or version B, the variant under test. Because both groups see the page in the same period, a sufficiently large difference in results can be attributed to that change, not to other factors.

There are other variants, such as testing several versions at once or testing several combined elements. For most businesses, a simple A/B test with two versions is enough and far easier to interpret correctly.

When testing is worthwhile, and when it is not

Testing needs an adequate number of visitors. If your page is visited by only a few dozen people a day and few of them take the target action, a difference between two versions will only become visible after a very long time, and even then it is fragile. In this condition, test results are more often coincidence than reality.

When traffic is small, there are more effective ways: fix the obviously visible problems. Watch session recordings, check the page on a phone, read questions that often reach customer service, and interview a few customers. Problems like an obscured button, a confusing form, or an unclear price usually need no testing; they just need fixing.

Testing is most useful when the problem is clear but the solution invites several reasonable opinions, and when the expected impact is large enough to justify the effort. Choosing a button colour rarely belongs there; changing the offer, the order of checkout steps, or how prices are shown is more often worth testing.

Start from the problem, then frame a hypothesis

A good test starts from an observation, not an idea. For example, analytics data shows many visitors fill their cart but stop at the shipping page. From that comes a suspicion: the shipping cost is only visible at the end, which surprises buyers.

Turn that suspicion into a testable hypothesis: if we show the estimated shipping cost on the product page, more visitors will complete the purchase because they are not surprised at the end. A hypothesis names the change, the reason, and the expected result. The reason matters because it makes you learn something even when the result is not as hoped.

Write down the hypothesis and success measure before the test runs. Setting the measure after seeing the results opens the door to contrived justification: if the main measure does not rise, there is always another measure that happens to rise.

Choosing the success measure

The primary measure should be one and directly tied to the business goal: the percentage of visitors who complete a purchase, send a quotation request form, or sign up. Shallower measures such as clicks on a button can rise easily without bringing real benefit.

Besides the primary measure, set guardrail measures, things that must not get worse. For example, if you simplify a form to get more submissions, also check the quality of the leads that come in. An increase in quantity accompanied by a drop in quality is not a win.

Make sure tracking works before the test starts. Many tests are wasted because the event being measured is not recorded correctly in one version. Test the tracking by performing the target action yourself on both versions and checking that everything is recorded.

How long a test runs

The biggest temptation is to stop the test as soon as one version looks ahead. In the first few days, numbers rise and fall by chance, and a version that appears to win often loses its lead later. Stopping at a favourable moment is the most common way to get a false win.

Decide in advance how many visitors or how long the test will run, then stick to it. Run at least one full weekly cycle so weekday and weekend patterns are represented, and longer if your business has a long purchase cycle. Testing tools generally provide a sample size calculator that can help.

Avoid running tests during unusual events such as a big campaign, a long holiday, or a technical disruption. Those events change the mix of visitors and can shift results in ways unrelated to the change being tested.

Reading results without being fooled

Test results usually come with a measure of statistical confidence. Understand its meaning in outline: how likely it is that the difference seen is due only to chance. High confidence means the difference is probably real, but does not guarantee that the impact is large or meaningful for your business.

Look at the size of the effect, too. An improvement that is real but very small may not be worth the cost of implementing it. Conversely, a result that looks large but rests on little data should be treated as a hint to retest, not as proof.

If you run many tests or dig into many visitor segments at once, the chance of finding coincidental patterns rises. Breakdowns by device or traffic source may be used to generate ideas, but the findings need their own test before being trusted.

What is worth testing first

• The value statement at the top of the page: do visitors immediately understand what you offer
• The offer and how it is presented, such as a guarantee, free shipping, or a trial
• The number of fields and the order of steps in a form or checkout
• How prices, extra fees, and payment options are shown
• Social proof such as reviews, customer counts, or partner logos, where genuine
• The text and placement of the main call to action button

Rank by estimated impact and ease. Changes with high impact that are easy to implement go first. Working through the list of ideas this way stops the team spending weeks on tests whose result will be small at best.

One test from start to finish

As a hypothetical illustration, a sports equipment store finds that many visitors stop at the cart page. Session recordings show visitors scrolling up and down looking for shipping cost information. The hypothesis: showing the estimated shipping cost and arrival estimate directly in the cart will make more visitors continue to payment.

The team sets the primary measure as the percentage of carts that continue to payment, the guardrail measures as cancellation and return rates, and a duration of three weeks to cover several weekly cycles. Tracking is tested first on both versions, then traffic is split evenly at random.

When the period ends, the team reads the result against the measures already set, checks the guardrails, and decides whether to apply the change to all visitors. Whatever the result, the hypothesis, numbers, and lessons are recorded. This example is deliberately simple; the key is the orderly sequence of work, not the size of the change.

Technical points that make a test trustworthy

Make sure the split of visitors is truly random and consistent. A visitor must always see the same version during the test, including when they return the next day. An inconsistent split makes the experience confusing and ruins the data.

Watch out for flicker, where a visitor briefly sees the original before the test version appears. Besides being annoying, flicker affects behaviour and departs from the real experience. Tests run on the server side or with the right technique usually avoid this problem.

Filter out nonhuman traffic, such as search engine crawlers and bots, and internal traffic from your own team. Both can skew results, especially on sites with few visitors. Finally, check that both versions load at comparable speed, because a slower version will lose for reasons unrelated to its design.

Testing on apps and mobile experiences

The same principles apply to mobile apps, with a few notes. App updates take time to reach users, so tests are often run through remote configuration that turns on certain variants without releasing a new version. Make sure the app is designed to support this from the start.

App users also tend to be fewer and more loyal than site visitors, so changes to the early flow for new users, such as sign up and feature introduction, are often the most meaningful places to test. Also mind app store policies and privacy rules when collecting behavioural data.

When to stop, apply, or repeat

After a test ends there are three possibilities. If the new version is clearly better and the guardrails are safe, apply it to all visitors and monitor for a few weeks to make sure the result holds. If the results show no difference, choose the version that is simpler or cheaper to maintain.

If the result is inconclusive for lack of data, you may extend the test under rules set from the beginning, or test a bolder change so the difference is easier to detect. Changes that are too subtle are hard to prove, and often genuinely mean little to visitors.

Ethics, privacy, and keeping trust

Testing is not a licence to manipulate visitors. Avoid misleading patterns such as fake countdowns, untrue low stock claims, or hiding the option to unsubscribe. Besides damaging trust, such practices can violate consumer protection rules.

Mind privacy. Inform visitors about data collection through a privacy policy, ask for consent as the applicable rules require, and do not collect personal data that is not needed for the test. Refer to the personal data protection rules currently in force.

Finally, do not let testing make the experience feel inconsistent. Returning visitors should see the same version as on earlier visits, and all versions must work well on all devices.

Building a habit of learning, not just finding winners

The greatest value of testing is not a single win but the accumulating understanding of your customers. Keep a record of every test: the hypothesis, the change, the result, and what was learned. This record stops the team repeating the same test and reveals patterns about what matters to customers.

Failed tests are valuable, too. A result showing that a change has no effect frees you from discussing it further, and a result showing a change is harmful saves you from a mistake that could have been rolled out across the whole site.

Make testing part of a light working rhythm, for example one or two active tests at a time with a monthly review. A small team that consistently learns bit by bit usually outperforms one that runs one big test a year.

Common mistakes to avoid

• Stopping the test as soon as one version looks ahead
• Testing many changes at once and then claiming to know which had an effect
• Setting the success measure after seeing the results
• Testing on traffic that is too small and treating the result as certain
• Ignoring the quality of results in order to raise the number of actions
• Not recording results so lessons are lost

First steps you can start this week

1. 01Decide on one primary action on your website and make sure its tracking is accurate
2. 02Find one point where many visitors drop off through data and session recordings
3. 03Write a complete hypothesis with reasoning and a success measure
4. 04Set in advance the duration or number of visitors needed
5. 05Prepare a simple log sheet to store each test and its lessons

Closing

A/B testing is a powerful tool for replacing debate with evidence, but only when run with discipline: a clear problem, a written hypothesis, a measure set beforehand, and the patience to wait for enough data.

If your traffic is not yet adequate, do not force testing. Fix the things that obviously look problematic, observe visitors directly, and save testing for when volume supports it. The sure improvement from basic fixes is often larger than what a rushed test promises.