Web Analytics maturity as I have experienced it

I wanted to write this post to reflect over the experience that I have had so far over web data maturity on an organizational level. To become a data driven organization is not the easiest thing in the world but reaching this level though will lead to success.

The figure below shows how I have experienced the data maturity process during my time as a Web Analyst.

Panic, panic we need to measure now!

We need to measure our online activity – The first step is when people start to realize that they are so big that they need to measure their online activity. This comes usually as a realization when upper management start asking some of the more basic questions such as, what do people do on our site. This when they install a web analytics tool.

 

Is this data true or are they trying to trick me?

Can we trust the data? – The second step is to ask about the data. Can we trust the data? Is this data accurate? What does unique browser mean, why are these not the same as unique IP from our log servers? This type of questions is typical in the second phase as I have experienced it.

 

I am so smart if I have many reports

We need reports lots of them – When people feel satisfied with the data validation they realize they need lots of reports. So the Web Analyst guy becomes a reporting guy. I can assure you this is a big waste of time of the web analyst competence. All reporting should be done automatically since there are so many great software out there that will do it more accurate than letting a person to do it.

 

This Web Analyst sucks! He gives me so much data I feel dumb

What should I do with all the numbers? -Now you have all numbers you could ever wish for. Actually you have more numbers than you can handle and you are asking yourself why you ever hired a Web Analyst in the first place. Relax, first you need to realize that less data is better. Depending on how experienced your web analyst is ask him to only give you the minimum amount of data that is valid only for taking decisions on. Ask him also to give you his input on what to do since you will move a lot faster to the next level.

 

Wait a minute, maybe I should keep my Web Analyst, maybe I should promote him

Now I understand the numbers, how do I act? – The aha-experience hits you and you know how to interpret the numbers. You start asking the right questions and now you want to do something to improve your web business. You start looking at optimization.

 

My Web Analyst is gold worth, now I can get that promotion and bonus I was promised

How do we optimize? – Optimization is the nirvana of web analytics; it gives you the tool to do act. You start doing AB testing. This a small field in itself and there are several steps involved to have a good program in place but if you get here, you should be happy.

 

My boss said he promote me only if I can make our whole organization more effective

We need to coordinate our actions – Now that you feel comftable with split testing you move your horizon on an organizational level. So far usually only a few people within the organization reaches this level. The next step, which is rather hard, is to make your whole organization harvest the insights of good web analytics and AB testing.

 

I got the promotion now I sit on the board of directors helping them make strategic decisions based on the data

Let us use our data for strategic decisions – When you accomplish the previous task, which might take a long time to do, the insight will reach high level persons, people that want to use the data for strategic purposes, such as acquisitions and business development. This level is where you want to be in, only now you can title your organization as data savvy.

 

Now everyone needs the data all the time

How do we make the data on demand? – The next natural development is to make business critical data on demand. Each position needs their own set of data and therefore the web analyst plays a central role to be able to define the needs into data. You probably need to make investment in analytics tools for this level.

 

I think I am a rock star

Are we using data to make decisions all over? – People make decisions on data, congratulations you reached the highest web maturity level.

 

Everything we touch turns into gold

We are gaining market share – When you have climbed all the steps you reach the final goal, where you see increased market share and higher revenues.

Why we created a Data Warehouse (for the Web Analysts and everyone else)

On Thursday we launched our Data Warehouse. It was a special moment since it took us about a year of blood, sweat and tears to get all data in the same place. Much credit goes to Jon, Andrew, Harb and all the other guys involved in creating this masterpiece.

Why we created a data warehouse?

Our business are 100% online so we use mainly Omnitures various products to collect and analyze the data but it was not enough. The main problem is the lack of flexibility in the Omniture product suite hindering us to truly customize according to our need. We have about 60 websites that we keep track of and on top of that we have no control of the revenue data since it is tracked by a third party making our revenue based decisions more or less a guessing game. With the magic of the Data Warehouse we are able to eliminate this blindspot and make better business decisions.

What is the value of a data warehouse for a web analyst?

I must admit that in the beginning I had a hard time understanding the value of the data warehouse, although my attitude towards data is the more the better. I thought that the Omniture products were enough, and in some sense that is correct. The problem was that we have so many websites, and so many third party cooperation’s that when something went wrong no one could really make a good analysis. Now we can. Now I can see all the way from a visit to a purchase. Priceless.

So, how valuable is a data warehouse for a web analyst and can one get by with just a web analytics package? Well, I think it depends from case to case. As a general rule the more complex your business model is the bigger the need for a data warehouse.  At my previous employer we turned our tool into a data warehouse by customizing it to the extreme (thank you Yahoo Analytics for letting us do it). I know for example Dustin, the wholesaler of electronics are harvesting the potential of a data warehouse. For example by breaking down the figures they could micro segment their users sending out special offers depending on their interests and location.

How are we getting smarter by the data warehouse?

So far we have only had the thing for a few days but already I used it once. In future I see the data warehouse doing a lot of cool stuff such as correlation analysis of our top keywords and brands, projection of future and past trends and many more other things that does not come into my mind right now.

Of course it is like many others have pointed out, the tool itself is a dumb box, and it’s the people using it that will make the difference. Hopefully we will be able to get to the next level but I think road will be bumpy since it involves a mindset change for many people within the organization. If it is done the right way I think this will save a lot of money and energy by letting us to take the right decisions.

I think the web analytics market is going towards a more data warehouse approach

The field of web analytics is a fairly new one but fast evolving. I see more and more signals that the tools are becoming more sophisticated as their users, which is pushing the development towards a more data warehouse driven approach. For example the other day we had a representative of Web Trends that showed us their products and it was clear that they are going towards a data warehouse approach (and also copying Omniture).

I will most likely get back to the data warehouse when we have used it a bit more. Until then feel free to comment below with your own experience on your thoughts or experiences with a data warehouse.

Are you a Web Analytics Ninja? Take my quiz

In November I celebrated my third year as a Web Analyst and I must say I still love my work just as I did the first day. Just as this blog tries to give something back to the community I have trained people in real life to become good web analysts by sharing my knowledge. I am far from a teacher but I try to give it all and not hold back.

To be a good Web Analyst I think you need to be very broad in your knowledge. You need to know a bit of everything for example, how tracking works, how seo works, how to understand your customers, how to convince your coworkers and many other skills.

Rickard a collegue of mine was saying that Jon, another collegue, is a black belt in Excel. I thought that the black belt phrase is a good one to describe how good of an analyst you are, therefore I have created a “game” where you can find out if you deserve the black belt or not.

Download the file, print it out, count your points, and post your results in the comment field.

Of course you can download the file and post it on your blog. Creative Commons linces as they say.

Download Here

Post your score below

How I communicate the factors that affects our conversion

When working as a web analyst I have discovered that you are more than just a guy measuring your visitors and pulling out the stats. To be really good at your work I think one needs several skills among them understanding what affects your conversion. I constantly try to get involved in all parts of the business, from SEO to PPC to CRM and any other aspect that apply to the business model. I must say this is not an easy task but I think it gives me a lot of pleasure sniffing on other people’s area of expertise. What surprises me every time is that something that looks very simple for the outside might be very complicated when you dig deeper. Anyhow, the other day I was thinking of an easy way to display what affects our conversion and I came up with the following picture (I have removed a few factors but you still get the picture).

I think that displaying your most important factors that affect your conversion in this way makes it easier for everyone in the company to understand how you can improve. The hard part is to identify these factors, hence the argument that you have to know your company´s business model very well.

Each of the factors identified here can have several sub factors. For example, looking at your entry page you have a design factor, how you present your text and what you really say (the value proposition).

Factors Affecting Conversion Extended

So after finishing the drawing I realized two things. The first is that it gives you a very easy tool for communicating with various stake holders. The second thing was that it really showed the complexity of business optimization or in my case the complexity in my own role as a web analyst and conversion specialist. It was actually a bit of an aha-moment since to truly optimize our business I had to be that annoying Sherlock Holmes-type and not just mind my own business.

So, I would truly suggest you to make such a map show it to your various stake holders and ask them what they think. Even if they don’t like it, I think it will bring you a step closer to becoming that black belt web analytics expert.

My A/B testing guide

A year ago when I changed jobs I was faced with the joyful task of being part of the startup of our A/B testing program. Since we have several sites under our wings the methodology was very important because we wanted to draw conclusions from our tests to be applied to new websites and in some sense create a kind of “theory” on website design and content.

With 150 A/B tests and counting our methodology has brought us a step closer to understanding what affects a webpage conversion rate. This knowledge has made our life a bit easier since we can apply the findings on new projects with the comfort of knowing what conversion rate we can expect. Without going any deeper into the core of the methodology I thought I would give you my approach on A/B testing.

The below points are a guide of what you should think of when doing A/B split testing and are based on my own experience. Please feel free to comment on them if you don’t agree.

1. Be prepared that your testing will bring up new question that will need testing. So you are basically in a testing process rather than just one test.

2. Think of a concept you want to test. Is this concept aligned with what users want or is it based on your own cravings? Very good inputs for testing are user surveys and focus groups.

3. Look how much traffic you have on the page(s) that you will be testing on. Large amount of traffic allows you to do smaller changes but still see an effect.

4. If you have large amount of traffic consider doing A/B/C/n testing. This will allow you to cut down your testing time considerably.

5. Analyze where the traffic to the testing page(s) comes from (I mean URLs). If a larger part of the traffic is from Google look at the keywords which show user intentions. Try also to understand how this page fits in the user path. Ask the following question: Is this page critical for users to convert?

6. Make a wireframe of the page you want to test. Make your various versions stand out from the original page. If they are too similar you will probably not reach statistical confidence, hence not be able to draw conclusions.

7. This is a no brainer but I need to say it, design the page so it fits the rest of the site.

8. Communicate clearly to your coding guy (fronted) what the success metric is. Usually it is conversion or/and money but it could also be reaching a certain page or downloading something. News site for example have more page views per user as a success metric.

9. It is very common in the beginning of the A/B testing experience that you don’t track everything on your testing page, which is a mistake. As the saying goes, it is better to be safe than sorry, so you better put more tracking elements on a testing page than you thought of. This will avoid you to redo the test later.

10.  If you are working with segments, which you should, consider to divide your traffic into those segments. The segments could for example be based on entry keywords or previous page behavior.

11.  Front end code your testing page(s) and setup your testing tool. Direct 50% of the traffic to one version. If you have very large amount of traffic you can do the test on only a fraction of your traffic. For example I heard that Google only tests on 10 of their traffic. As testing tools you can use Google Optimizer or any other tool that fits the requirements but remember to do QA so everything runs smoothly when you launch your test.

12.  Launch the test. Keep an eye on it the first couple of hours so everything seems all right.

13.  Let the test run until you reach at least 90% confidence. Usually the confidence level is shown in the tool.

14.  When you reach confidence stop the test and do the analysis. Example of questions you should ask is; What do the numbers show? Did we reach our targets? Did we answer our hypothesis? Is this the best we can do or can we improve the numbers even more? What other questions type of questions did the test arise?

15.  Document your results and conclusion in an easy way so you can go back whenever you need to. Documenting is important if you do many A/B tests since you probably want to use your knowledge further down the road.

16.  At this point I must put in a warning since many people think that one test will be enough to get the lift you are looking for. Most of the times your results will be rather surprising which will more or less force you to do more tests. This is a good thing since the more you learn about your users the better conversion rate you will have.

17.  As a final point, do not forgett to make the A/B testing process fun. If you can involve your designer and coder in the beginning of the process it will make your testing people and your whole organization much smarter in the long run.

How I monitor a redesign release

I have worked as a web analyst for a while but I havent seen a post adressing the issue how to monitor a redesign release so I thought I would give you my view on how to monitor the immediate impacts of a redesign.

At the end of this post you will find a link to the excel document.

So, over the past six months we worked hard to make this really cool and useful redesign of our website. With a few hundred thousand of visits each day we had to get it right. The day of the launch came and within the hour I got the first question that would be repeated over the week “How does it perform?”. I looked at my web analytics package and could not see any immediate changes. Of course any of you out there that have had experience with redesign and in general hourly web analytics data know that the conversion rate, the visits, the bounce rate and such varies during the day and no immediate patterns are shown during such a short time.

Fast forward about 48 hours. The first numbers are showing up in my excel sheet and we could relax a bit. Everything looked good. Now let me get to the point, I composed an excel document to follow the effects of the redesign that I want to share with you. You can download the document at the end of the post.

On the top of the page I have collected the four most important metrics and I look at the averages per day. This way I can quickly see if the first day of the redesign looks fishy or not.

Monitor at least seven days, if you see major fluctuations do it for at least two weeks.

The second data set, the graphs, is the ones I like most since they are very simple to distribute and even the most novice can understand them with just a glance of an eye. The important part here is to calculate the standard deviation to have an upper and lower level, to understand if you should celebrate or investigate.

In the excel sheet you will also find the bounce rate. Of course if you want to have more metrics you can just create them, but these three are the most important according to my opinion.

Below you can see an example of the data I use to create the graphs.

I am not saying that this is the only way to do it, but I think it will be good for most cases. Of course I would never do a site wide redesign without testing it first, more on that another time.

Download the document here.