The 3 most common A/B testing mistakes beginners make

Having done over 200 A/B and MVT tests I have gained testing experience the hard way. I have had my fair amount of success and failures. Working with companies that are just exploring the A/B testing possibilities I have noticed reoccurring mistakes that is based on inexperience.

If you are a newbie in this area I would recommend you read this post. If you are experienced I would suggest you read about the top 3 most converting headings. If you have any questions do not hesitate to ask.


1. Low traffic

The absolute most common factor I have encountered is that the traffic to the test is way too low. It is important to have the right amount of traffic to be able to draw accurate conclusions. With accurate I mean statistically correct conclusions. You need statistical confidence otherwise your test is inconclusive. In other words it means that the change can be due to chance. For example Google does AB and MVT tests on 5% of their searches, but since they have so many searches they reach statistical confidence within a very short time (probably one day).

How do you determine that you have the right amount of traffic? I suggest you look into your web analytics package and on page level. Check how many visits you have a day and then calculate the amount of conversions. A rule of a thumb is to have around 300 conversions on each recipe you try before you conclude the test. If your conversion rate is 10% then you need 3000 visits to have the right amount of traffic.


2. To many recipes

AB testing an idea is always the way to go instead of having the HIPPO (Highest Paid Persons Opinion) dictate what the users want, which is a sure path for disaster. The second most common testing mistake is to have too many testing ideas that you try at once. This means that your traffic will be split into many recipes forcing you to run the test for a very long time. A recommendation is to test more than two recipes only when you have more than a 600 visitors to your testing page. Review your ideas by looking at user insight, my next point.


3. No clear idea or possibly the wrong idea

Testing an idea is always good but how do you know if you are testing the correct idea or if you are just wasting time and resources? Well this is where your user insight comes into place. You need to know what your users want to be able to compile the correct test. If you don’t you will make many tests that will be more or less useless. For example a well known poker grand tested 6 different versions at once. Three of them were very similar and the testing results were inconclusive.


User insight is derived by many different tools and methods. User studies such as, surveys, focus groups, user testing, eye tracking, keyword analysis, surfing patterns, interviews etc shall all be used to gain the knowledge of what your users want. This method is a never ending story and distinguishing the winners from the losers. I am a strong believer of doing continually user studies, just like Google does.


Of course you might also be very lucky hitting that idea that will give you a fantastic increase.


So now that you know the most common A/B testing mistakes you are ready to go. You can also read the more hands on post on how you start an A/B test.

Top 10 Tips for Testing Success

I have been testing for a long time by now, but this video is still interesting, wether you are a beginner or a newbie in the AB testing field.

You might know Eric Peterson, one of these guys that are kind of famous within the web analytics field. I have seen him several times and I think he has a great passion for what he does. I definately think he is a very good for the whole conversion and web analytics field. Have a look at his two videos and dont forget to comment.


My life as a Web Analyst – April 2011

It has been a while since I wrote a post like this. I guess the main reason for it is because I have been very very busy. Actually so busy that I had nightmares. I tell you about them some other day. Anyway let me try to conclude the major things that happened since my last post. Of course I will not give away business secrets or internal stuff but all other things good and bad.

  • The Google farmer update beginning of March hit some of our sites badly. There were a bit of panic in the organization due to that.
  • We made my redesign release beginning of March on one of the sites that was hit badly – one week after the Google update. It was probably the smoothest release in the history of the company. No major bugs. Of course I had set up a good quality control before we went live, so we handled most bugs before the live release. One of the major effects of the redesign is that our rankings in Google came back with full force. We actually ranked even better than before. Imagine how happy my boss was!
  • The redesign actually increased our page views with 4% and the review pages became 40% more popular. We know that people looking at a review page are much more valuable.
  • Some key people left the company, creating a bit of a competence loss. This affected my job in terms of not getting my stuff out. Although last week it seems we have made some major progress so I am happy that my programmers are producing.
  • We had a visit from our Florida office. Great to bond with people you only talk on the phone with.
  • Landing pages. I have done a lot of landing pages. The first one we tried was a real hit. The conversion rate went up with +33% which means big bucks. Needless to say there were a lot of smiles.
  • Lots of meetings during the last three weeks. I think meetings are good when you have something to say. Of course like every other organization the meetings do not always turn out to be super productive but I think looking back we needed to have them.
  • I was involved on a market analysis. We talked to owners, analysts, and sales people etc to get a good picture on where the market is heading within a two year period.
  • I am executing the conversion plans I created two months ago. To give you a brief picture on what that means, it is a plan with a lot of actions. The actions are a mix of various things. Almost all things that go on the website needs to be tested first so we don’t ruin the conversion. I must say I still get surprised how small stuff likes a badge or so can affect conversion so much.
  • I have scoped two new tools to try. One of them is a testing tool. I am now waiting for the implementation. I am a bit excited.
  • A new casino landing page is ready to go live soon. We have had a long discussion with a client on creating a kick ass landing page with a lot of behavioral targeting. Let’s see how it will perform.

That’s it. Of course there is daily stuff that I don’t mention but these are the major things. So what do you think? Are these post interesting to follow or have you fallen asleep? Do not hesitate to let me know.

What managers think of web analysts…but do not say

Since I have been working in the analytics area for some time now I have experienced many discussions with managers about web analytics and what a web analyst can contribute. The discussions have ranged back and forth mainly focusing on the fact of trying to harness the most of web analytics insight.

In most cases when you try to incorporate a new technology within an existing structure it will be a bumpy ride. There will be a lot of trial and error but eventually if your company is committed to extracting value you will succeed. In my case we ended up calling us conversion specialist hinting that we harvest web analytics for a conversion increase.

To help you in your process it is good to understand manager’s thoughts on the subject. Here come the five most common thoughts I have encountered.

1. “Unique visitors, visitors, average conversions I don’t really understand what he is talking about, maybe we don’t need a web analyst after all.”

a.       Recognized: When your manager do not ask analytics questions.

b.      Don’t talk metrics, talk user behavior. Start by showing the click density map. Make it less intimidating to explore the field of numbers.


2. “This is guy is just costing me money, we already have Google analytics and I know how to pull the numbers. I should fire him.”

a.       Recognized: When you are sitting in a strategically management meeting and the only questions you get is about visitors, page views and average conversion rate.

b.      This is a typical thought especially for smaller companies. It has a lot to do with the analytics maturity of the company (read my post about analytics maturity within organizations). To avoid it you need to first of all build segments of your users and start telling stories of their behavior. Second of all you need to move into AB testing. You need to show the value of pulling web analytics data and using it to directly increase the revenues.


3. “My web analyst is not as creative in our meetings as the other guys.”

a.       Recognized: When everyone throws ideas around but none of them are based on numbers.

b.      The bottom line is that everyone can throw out an idea. Even my 91 year old grandmother can express and idea on what we should do on our website, unfortunately not many of the ideas are based on facts. As a web analyst you have to be prepared to backup your ideas with data. You are the true master of data is you, and therefore build your case with data. Try to build an analytics culture by showing them the way.


4. “We already have a web analytics tool we do not need more tools so this guy should be happy as it is.”

a.       Recognized: When you ask for cheap tools and the only answer you get is that the budget is a bit strained at the moment.

b.      Remember that you want to show the connection between the money and your analytics knowledge. Preferably there should be a straight red line between the two.


5.       “Personas, why should we have that, we do fine anyway? He is just trying to justify his existence.”

a.       Recognized: Two months after you have presented your personas no one is using them.

b.      Try to base your ideas and improvements on your user profiles. Without understanding of your customers you will have a hard time to make improvements after all low hanging fruits are picked. Your team/company will have a hard time understanding where they should aim making your development process a wild west. It is your obligation as a web analyst to show how personas shall be used and tie them to the analytics data.