My A/B testing guide

A year ago when I changed jobs I was faced with the joyful task of being part of the startup of our A/B testing program. Since we have several sites under our wings the methodology was very important because we wanted to draw conclusions from our tests to be applied to new websites and in some sense create a kind of “theory” on website design and content.

With 150 A/B tests and counting our methodology has brought us a step closer to understanding what affects a webpage conversion rate. This knowledge has made our life a bit easier since we can apply the findings on new projects with the comfort of knowing what conversion rate we can expect. Without going any deeper into the core of the methodology I thought I would give you my approach on A/B testing.

The below points are a guide of what you should think of when doing A/B split testing and are based on my own experience. Please feel free to comment on them if you don’t agree.

1. Be prepared that your testing will bring up new question that will need testing. So you are basically in a testing process rather than just one test.

2. Think of a concept you want to test. Is this concept aligned with what users want or is it based on your own cravings? Very good inputs for testing are user surveys and focus groups.

3. Look how much traffic you have on the page(s) that you will be testing on. Large amount of traffic allows you to do smaller changes but still see an effect.

4. If you have large amount of traffic consider doing A/B/C/n testing. This will allow you to cut down your testing time considerably.

5. Analyze where the traffic to the testing page(s) comes from (I mean URLs). If a larger part of the traffic is from Google look at the keywords which show user intentions. Try also to understand how this page fits in the user path. Ask the following question: Is this page critical for users to convert?

6. Make a wireframe of the page you want to test. Make your various versions stand out from the original page. If they are too similar you will probably not reach statistical confidence, hence not be able to draw conclusions.

7. This is a no brainer but I need to say it, design the page so it fits the rest of the site.

8. Communicate clearly to your coding guy (fronted) what the success metric is. Usually it is conversion or/and money but it could also be reaching a certain page or downloading something. News site for example have more page views per user as a success metric.

9. It is very common in the beginning of the A/B testing experience that you don’t track everything on your testing page, which is a mistake. As the saying goes, it is better to be safe than sorry, so you better put more tracking elements on a testing page than you thought of. This will avoid you to redo the test later.

10.  If you are working with segments, which you should, consider to divide your traffic into those segments. The segments could for example be based on entry keywords or previous page behavior.

11.  Front end code your testing page(s) and setup your testing tool. Direct 50% of the traffic to one version. If you have very large amount of traffic you can do the test on only a fraction of your traffic. For example I heard that Google only tests on 10 of their traffic. As testing tools you can use Google Optimizer or any other tool that fits the requirements but remember to do QA so everything runs smoothly when you launch your test.

12.  Launch the test. Keep an eye on it the first couple of hours so everything seems all right.

13.  Let the test run until you reach at least 90% confidence. Usually the confidence level is shown in the tool.

14.  When you reach confidence stop the test and do the analysis. Example of questions you should ask is; What do the numbers show? Did we reach our targets? Did we answer our hypothesis? Is this the best we can do or can we improve the numbers even more? What other questions type of questions did the test arise?

15.  Document your results and conclusion in an easy way so you can go back whenever you need to. Documenting is important if you do many A/B tests since you probably want to use your knowledge further down the road.

16.  At this point I must put in a warning since many people think that one test will be enough to get the lift you are looking for. Most of the times your results will be rather surprising which will more or less force you to do more tests. This is a good thing since the more you learn about your users the better conversion rate you will have.

17.  As a final point, do not forgett to make the A/B testing process fun. If you can involve your designer and coder in the beginning of the process it will make your testing people and your whole organization much smarter in the long run.

5 thoughts on “My A/B testing guide”

  1. My boss told me that we should do ab testing and I must say in theory it sounds easy but when you digg deep into it, well then it is a totally different ballgame. This list is a good starting point. What ab testing tool providers do you recommend?

Leave a Reply

Your email address will not be published. Required fields are marked *