Aaron Gray will be speaking about “Web Analytics” at SearchFest 2010, which will take place on March 9th at the Governor Hotel in Portland, Oregon. Tickets are available now. To purchase, please click the following link.
1) Please give me your background and tell us what you do for a living.
Educationally, my background is in psychology and sociology. Almost immediately after finishing my undergrad work I was wooed (distracted?) by interactive media and the web – where I’ve been professionally ever since. Because of my interest and background in psychology, I was immediately drawn to understanding how and why people interacted with interactive systems and web sites, and how to optimize those experiences to produce better commercial results. For a while, I focused on usability engineering and interaction design. As a result, I founded Tweak Interactive in 2000 to create great online customer experiences for, primarily, Northwest companies. My interests soon turned to data, and how data about behavior online, in conjunction with knowledge of human psychology, can be used to generate analytical insights which themselves are used to drive continuous, incremental improvements to the commercial performance of websites. Based on that interest, I left Tweak and went to work for a Portland startup called WebCriteria, which had the first commercial technology for measuring the ability of web site users to complete a defined process or journey online.
WebCriteria was eventually acquired by Coremetrics, one of the big 3 commercial web analytics players. I stayed with Coremetrics for a few years before moving on to another local Portland company, Webtrends. At Webtrends, I helped formulate and built the consulting practice that designs global analytics solutions for some of the largest brands in the world. I also had a heavy hand in the creation of the current Open Exchange partner integration framework. I was at Webtrends for nearly five years, and in that time I had many roles – always focused on building or growing new teams and operational processes. I left last summer after it became clear I was no longer going down a path that fit me, personally or professionally.
Since June 2009, I’ve been consulting at Greater Returns, helping companies get better value from analytics and e-commerce enabling technology investments. Sometimes that means fixing an implementation that has fallen out of tune; sometimes that means helping to create processes for the governance of analytics solutions; sometimes it means helping companies define the right measurements for their business model; and sometimes it means helping companies select the right solutions for their needs. No matter what I do, I’m always focused on finding ways to help my clients get better value from their analytics and e-commerce investments.
2) How much data is necessary to make a landing page “test” statistically significant?
The real answer, paradoxically, is "as much as is needed." In all seriousness, it depends on the product or method you’re using. If you’re using a packaged testing solution, like Google Web Site Optimizer, Maxymizer, or Autonomy Interwoven’s Optimost, they’ll pretty much manage that for you. If you’re doing a home-grown a/b split test (which I advocate) there are tricks for telling when you’ve got enough data. Essentially, you set the test up so that your a and b populations only represent, say, 20% of your traffic — 10% to each. If "a" is your "original" design, then that population will see the same site version as the other 80% of your traffic who are not part of the test (actually, you can call them the "c" group). After launching the test, wait for key ratios, such as conversion rate, in group "a" and "c" to settle out to be the same. When they do, your test is valid enough. If they don’t, you’ve got something wrong in your set up, and you need to figure out what that is and fix it.
Statisticians will scream at me for recommending this approach, but I’ve seen it work many times. A disciplined application of continuous home-grown a/b testing will yield far better returns than undisciplined use of a commercial a/b or multivariate testing solution.
On the other hand, Google Website Optimizer says it needs about 100 "conversions" (completions of the goal you’re optimizing for) per version to get good results.
3) What are the pluses and minuses of Google’s Website Optimizer for landing page testing?
Well, on the plus side it’s free. And then there’s the fact that it’s free. Also, it’s free. It will let you do a/b or multivariate testing, and only requires about 1000 views of the test page per week for a valid test. It’s pretty simple to set up.
On the down side, it may not be robust enough for more advanced organizations. And some organizations, especially government agencies and financial services companies are sensitive to having their data in the cloud, and even more so with Google. But, then again, it’s free.
Todd Mintz knows PPC…knows Social Media…knows SEO…knows Blogging…knows Domaining…and knows them all real well. He runs growth marketing for )and is also a Director & Founding Member of SEMpdx: Portland, Oregon’s Search Engine Marketing Association, and he can be found here on Twitter and Facebook.