Why we created a Data Warehouse (for the Web Analysts and everyone else)

On Thursday we launched our Data Warehouse. It was a special moment since it took us about a year of blood, sweat and tears to get all data in the same place. Much credit goes to Jon, Andrew, Harb and all the other guys involved in creating this masterpiece.

Why we created a data warehouse?

Our business are 100% online so we use mainly Omnitures various products to collect and analyze the data but it was not enough. The main problem is the lack of flexibility in the Omniture product suite hindering us to truly customize according to our need. We have about 60 websites that we keep track of and on top of that we have no control of the revenue data since it is tracked by a third party making our revenue based decisions more or less a guessing game. With the magic of the Data Warehouse we are able to eliminate this blindspot and make better business decisions.

What is the value of a data warehouse for a web analyst?

I must admit that in the beginning I had a hard time understanding the value of the data warehouse, although my attitude towards data is the more the better. I thought that the Omniture products were enough, and in some sense that is correct. The problem was that we have so many websites, and so many third party cooperation’s that when something went wrong no one could really make a good analysis. Now we can. Now I can see all the way from a visit to a purchase. Priceless.

So, how valuable is a data warehouse for a web analyst and can one get by with just a web analytics package? Well, I think it depends from case to case. As a general rule the more complex your business model is the bigger the need for a data warehouse.  At my previous employer we turned our tool into a data warehouse by customizing it to the extreme (thank you Yahoo Analytics for letting us do it). I know for example Dustin, the wholesaler of electronics are harvesting the potential of a data warehouse. For example by breaking down the figures they could micro segment their users sending out special offers depending on their interests and location.

How are we getting smarter by the data warehouse?

So far we have only had the thing for a few days but already I used it once. In future I see the data warehouse doing a lot of cool stuff such as correlation analysis of our top keywords and brands, projection of future and past trends and many more other things that does not come into my mind right now.

Of course it is like many others have pointed out, the tool itself is a dumb box, and it’s the people using it that will make the difference. Hopefully we will be able to get to the next level but I think road will be bumpy since it involves a mindset change for many people within the organization. If it is done the right way I think this will save a lot of money and energy by letting us to take the right decisions.

I think the web analytics market is going towards a more data warehouse approach

The field of web analytics is a fairly new one but fast evolving. I see more and more signals that the tools are becoming more sophisticated as their users, which is pushing the development towards a more data warehouse driven approach. For example the other day we had a representative of Web Trends that showed us their products and it was clear that they are going towards a data warehouse approach (and also copying Omniture).

I will most likely get back to the data warehouse when we have used it a bit more. Until then feel free to comment below with your own experience on your thoughts or experiences with a data warehouse.

Are you a Web Analytics Ninja? Take my quiz

In November I celebrated my third year as a Web Analyst and I must say I still love my work just as I did the first day. Just as this blog tries to give something back to the community I have trained people in real life to become good web analysts by sharing my knowledge. I am far from a teacher but I try to give it all and not hold back.

To be a good Web Analyst I think you need to be very broad in your knowledge. You need to know a bit of everything for example, how tracking works, how seo works, how to understand your customers, how to convince your coworkers and many other skills.

Rickard a collegue of mine was saying that Jon, another collegue, is a black belt in Excel. I thought that the black belt phrase is a good one to describe how good of an analyst you are, therefore I have created a “game” where you can find out if you deserve the black belt or not.

Download the file, print it out, count your points, and post your results in the comment field.

Of course you can download the file and post it on your blog. Creative Commons linces as they say.

Download Here

Post your score below

How I communicate the factors that affects our conversion

When working as a web analyst I have discovered that you are more than just a guy measuring your visitors and pulling out the stats. To be really good at your work I think one needs several skills among them understanding what affects your conversion. I constantly try to get involved in all parts of the business, from SEO to PPC to CRM and any other aspect that apply to the business model. I must say this is not an easy task but I think it gives me a lot of pleasure sniffing on other people’s area of expertise. What surprises me every time is that something that looks very simple for the outside might be very complicated when you dig deeper. Anyhow, the other day I was thinking of an easy way to display what affects our conversion and I came up with the following picture (I have removed a few factors but you still get the picture).

I think that displaying your most important factors that affect your conversion in this way makes it easier for everyone in the company to understand how you can improve. The hard part is to identify these factors, hence the argument that you have to know your company´s business model very well.

Each of the factors identified here can have several sub factors. For example, looking at your entry page you have a design factor, how you present your text and what you really say (the value proposition).

Factors Affecting Conversion Extended

So after finishing the drawing I realized two things. The first is that it gives you a very easy tool for communicating with various stake holders. The second thing was that it really showed the complexity of business optimization or in my case the complexity in my own role as a web analyst and conversion specialist. It was actually a bit of an aha-moment since to truly optimize our business I had to be that annoying Sherlock Holmes-type and not just mind my own business.

So, I would truly suggest you to make such a map show it to your various stake holders and ask them what they think. Even if they don’t like it, I think it will bring you a step closer to becoming that black belt web analytics expert.

My A/B testing guide

A year ago when I changed jobs I was faced with the joyful task of being part of the startup of our A/B testing program. Since we have several sites under our wings the methodology was very important because we wanted to draw conclusions from our tests to be applied to new websites and in some sense create a kind of “theory” on website design and content.

With 150 A/B tests and counting our methodology has brought us a step closer to understanding what affects a webpage conversion rate. This knowledge has made our life a bit easier since we can apply the findings on new projects with the comfort of knowing what conversion rate we can expect. Without going any deeper into the core of the methodology I thought I would give you my approach on A/B testing.

The below points are a guide of what you should think of when doing A/B split testing and are based on my own experience. Please feel free to comment on them if you don’t agree.

1. Be prepared that your testing will bring up new question that will need testing. So you are basically in a testing process rather than just one test.

2. Think of a concept you want to test. Is this concept aligned with what users want or is it based on your own cravings? Very good inputs for testing are user surveys and focus groups.

3. Look how much traffic you have on the page(s) that you will be testing on. Large amount of traffic allows you to do smaller changes but still see an effect.

4. If you have large amount of traffic consider doing A/B/C/n testing. This will allow you to cut down your testing time considerably.

5. Analyze where the traffic to the testing page(s) comes from (I mean URLs). If a larger part of the traffic is from Google look at the keywords which show user intentions. Try also to understand how this page fits in the user path. Ask the following question: Is this page critical for users to convert?

6. Make a wireframe of the page you want to test. Make your various versions stand out from the original page. If they are too similar you will probably not reach statistical confidence, hence not be able to draw conclusions.

7. This is a no brainer but I need to say it, design the page so it fits the rest of the site.

8. Communicate clearly to your coding guy (fronted) what the success metric is. Usually it is conversion or/and money but it could also be reaching a certain page or downloading something. News site for example have more page views per user as a success metric.

9. It is very common in the beginning of the A/B testing experience that you don’t track everything on your testing page, which is a mistake. As the saying goes, it is better to be safe than sorry, so you better put more tracking elements on a testing page than you thought of. This will avoid you to redo the test later.

10.  If you are working with segments, which you should, consider to divide your traffic into those segments. The segments could for example be based on entry keywords or previous page behavior.

11.  Front end code your testing page(s) and setup your testing tool. Direct 50% of the traffic to one version. If you have very large amount of traffic you can do the test on only a fraction of your traffic. For example I heard that Google only tests on 10 of their traffic. As testing tools you can use Google Optimizer or any other tool that fits the requirements but remember to do QA so everything runs smoothly when you launch your test.

12.  Launch the test. Keep an eye on it the first couple of hours so everything seems all right.

13.  Let the test run until you reach at least 90% confidence. Usually the confidence level is shown in the tool.

14.  When you reach confidence stop the test and do the analysis. Example of questions you should ask is; What do the numbers show? Did we reach our targets? Did we answer our hypothesis? Is this the best we can do or can we improve the numbers even more? What other questions type of questions did the test arise?

15.  Document your results and conclusion in an easy way so you can go back whenever you need to. Documenting is important if you do many A/B tests since you probably want to use your knowledge further down the road.

16.  At this point I must put in a warning since many people think that one test will be enough to get the lift you are looking for. Most of the times your results will be rather surprising which will more or less force you to do more tests. This is a good thing since the more you learn about your users the better conversion rate you will have.

17.  As a final point, do not forgett to make the A/B testing process fun. If you can involve your designer and coder in the beginning of the process it will make your testing people and your whole organization much smarter in the long run.

How I monitor a redesign release

I have worked as a web analyst for a while but I havent seen a post adressing the issue how to monitor a redesign release so I thought I would give you my view on how to monitor the immediate impacts of a redesign.

At the end of this post you will find a link to the excel document.

So, over the past six months we worked hard to make this really cool and useful redesign of our website. With a few hundred thousand of visits each day we had to get it right. The day of the launch came and within the hour I got the first question that would be repeated over the week “How does it perform?”. I looked at my web analytics package and could not see any immediate changes. Of course any of you out there that have had experience with redesign and in general hourly web analytics data know that the conversion rate, the visits, the bounce rate and such varies during the day and no immediate patterns are shown during such a short time.

Fast forward about 48 hours. The first numbers are showing up in my excel sheet and we could relax a bit. Everything looked good. Now let me get to the point, I composed an excel document to follow the effects of the redesign that I want to share with you. You can download the document at the end of the post.

On the top of the page I have collected the four most important metrics and I look at the averages per day. This way I can quickly see if the first day of the redesign looks fishy or not.

Monitor at least seven days, if you see major fluctuations do it for at least two weeks.

The second data set, the graphs, is the ones I like most since they are very simple to distribute and even the most novice can understand them with just a glance of an eye. The important part here is to calculate the standard deviation to have an upper and lower level, to understand if you should celebrate or investigate.

In the excel sheet you will also find the bounce rate. Of course if you want to have more metrics you can just create them, but these three are the most important according to my opinion.

Below you can see an example of the data I use to create the graphs.

I am not saying that this is the only way to do it, but I think it will be good for most cases. Of course I would never do a site wide redesign without testing it first, more on that another time.

Download the document here.

Hello world!

My first post a second time. I mean I used to blogg in the beginning of days but stopped due to drought in the idea department but now I am back! I want to give something back to the community of e-business and more specifically the web analytics world and the e-business development, therefore I will blog about stuff like web analytics and e-business. I will try to be as personal as possible and keep it short and concice.