5 Questions to ask a web analyst

Recruiting someone can be daunting task, recruiting a web analyst is even a harder task since the position holds great importance in any type of company. To make it worse the recruiter usually has no idea what a web analyst truly does.

This post is about showing you the most important questions you should ask a web analyst to be able to evaluate their competence level. The questions are based on my own experience as a web analyst but also what other people within the field say.

If you are a web analyst you really have to think hard about these questions and how you have solved them so far. Try to be as honest as possible and if you have problems answering them don’t worry most web analyst work in silos but that does not mean you cannot explore new fields. The importance is that you have the passion to learn.

And now the questions:

Q1. How has your analytics insight contributed to the company?

You are looking for someone who can explain that the value of numbers not by report producing but rather explaining user behavior that then can be capitalized into a dollar value or another conversion metric. Many web analysts get stuck in the reporting mode, pulling a lot of reports on request. The problem is that the reports are just reports, they don’t help the company to develop their business.

A big bonus is if the web analyst has worked with AB testing by using the web analytics knowledge. Showing this skill will tell you that the analytics knowledge is truly being used in the business development process, which requires both insight in the business model and a good analytical skill.


Q2. What type of tools do you consider needing to make the best analysis?

This is not a task on counting the tools but rather get a sense if the person understands the complexity of doing good analysis. You will always need multiple tools to make that cutting edge analysis that will take you to the next level.


Q3. What has been the major obstacles in your web analytics role?

This is a rather interesting question since you will get many different answers depending on the type of organization the person has worked in. A big organization has its own demons while a smaller one has others. There is no “correct” answer for this question but it will rather point you towards the awareness factor. Awareness of the obstacles people can face and how they might have overcome them.


Q4. How do you convey your analytics insight?

Second in importance after being able to analyze the numbers correctly is to be able to convey the message to the appropriate stake holders. Remember that web analytics is not reporting it is making the numbers alive and ultimately making a difference in the bottom line of a company. Doing this involves cooperation with several different stakeholders. Being able to explain user behavior in an easy and intriguing way is almost a must.


Q5. In an ideal world how should web analytics be used?

The aim with this question is to show the analytical mind of the person and ultimately see how the person understands the development of analytics. It is not a must but a person that follows the industry knows where the market is heading, which is more and more data on all levels. This data is being used in driving business change. This question can also be a nice topic for the most common question of them all, how do you think you can help this company?


That’s my top 5. Of course you should combine these questions with other but on the web analytics field these are the most important ones. You will of course get spin off questions on the answers so asking these will keep your conversation going on for a good 30 minutes at least.

If you have any other questions you might think of please do not hesitate to comment. I love comments. They are rare but I love them.


Web analytics geek video

Have you ever wondered what a web analytics geek does on his spare time? Yes, of course he makes a web analytics animated video. The below video is actually pretty cool in a wierd way. It might be that I am a geek or it might be a new trend but you have to admire the effort. Actually it is made by a (new?) video service called xtranormal, which specializes in making animated video by the users.

Go ahead, watch it, it talks about a few of the important questions raised by web analysts and you get the robot voice as a bonus!


Yes, numbers lie

Have you ever read an article about a conversion increase? It is not uncommon to hear three figured increases, which sounds absolutely fantastic. With a three figured conversion increase it must mean that a lot of people are getting very rich, but of course the truth usually is a bit more complex.

I have stressed in my posts about the importance of having statistical significance in your tests before you draw conclusion but guess what, that is not always enough. You will also need to look at the number of conversions you get to understand your conversion increase.

First lets me briefly explain the significance concept. This is what Wikipedia says:

In statistics, a result is called statistically significant if it is unlikely to have occurred by chance alone, according to a pre-determined threshold probability, the significance level. The phrase “test of significance” was coined by Ronald Fisher: “Critical tests of this kind may be called tests of significance, and when such tests are available we may discover whether a second sample is or is not significantly different from the first.”

In plain English a statistical significance shows you how accurate your results are. So, if say you have a 95% statistical significance it should be read as, if you do this test a 100 times, 95 of these you will get the announced result.

Now, let me with an example show how statistical significance is important but not enough.

So, we wanted to try a new AB testing tool. We installed the code on the site, I read the manuals and I went ahead with a so called comma test. Actually it was a dot test. A dot test is when you put a dot somewhere on the site and then let it run as an AB test. In this way you should get the same conversion rate on both versions.

Below you can see the results of the test. Notice the fairly high amount of visitors on each version. I had several goal metrics. The Continue button, RefiForm2, RefiForm3, RefiForm4 and the thank you page. As you can see the Variation 1, which is the B version with the dot, had a 29,1% conversion on the continue button while the Original had 25,9%. Also the thank you page had 1,0% versus 0.6% for the original.

Both metrics point towards that this test increases conversion but of course you know better. Notice the confidence level. You need to reach five dots to be significant. None of these two metrics are significant, although the continue button shows a fairly healthy significance of 76%.

Now notice the RefiForm3. It shows significance with 99%. Does this means we should open an champagne and publish the results on a conference? Of course not. Even if the results show us a significant result if we look a bit deeper we can see that the winning version has only 22 conversions which is an extremely low number to draw conclusions from. As a rule of a thumb I usually look for 150 conversions or preferable around 300 conversions.

Further I have to consider what we are testing. Obviously there is no logic that a dot should increase conversion. Then I also look at how the test has performed during time. The graph above is a bit short but if you let it run for a while you should get a good illustration on how good the test has performed.

When interpreting testing results you have to consider several factors. Significance is a very important metric but not the only one. You have to combine it with the number of conversions and some time with a graph over how the test has done over time.

I hope that next time you come over a successful AB test that shows an incredible increase in conversion you should question the metrics. On your own tests, please do not try to be trigger happy, even if your stake holders push you for fast results. These things should be able to take time or there is a big risk of drawing the wrong conclusions.

To conclude you should:

  • Always look for significance in your testing.
  • Look at the number of conversions, are they to low?
  • Consider conversion over time, how it has changed.
  • Understand if the test makes sense.
  • And finally always consider the conversion interval.

If you have any questions do not hesitate to ask.