The Internet Marketing Driver: Glenn Gabe's goal is to help marketers build powerful and measurable web marketing strategies.

Friday, July 06, 2007

Multivariate Testing with Google Website Optimizer – Increase Conversion Rate and Take Guesswork Out of the Equation

Multivariate Testing Using Google Website OptimizerWebsite and campaign optimization has become an extremely hot area of web marketing recently. My guess is that you’ve heard the terms split testing, A/B testing, multivariate testing, Taguchi method, etc. thrown around on blogs and at conferences recently. What do these terms mean?? They all refer to the concept of scientifically testing your marketing campaigns in order to increase conversion (whether that’s increasing sales, downloading a whitepaper, filling out a contact form, subscribing to your RSS feed, etc.) This post is intended to be an introduction to multivariate testing using Google’s free optimization tool called Google Website Optimizer. I’ll explain more about it a little later on.

Let’s start with a quick definition of multivariate testing:
Multivariate testing enables you to test several components of a website to determine the optimal combination for increasing conversion. But unlike a traditional split test, which tests one creative versus another, multivariate testing enables you to test the combination of elements on a page and then will determine the optimal combination of those elements for increasing conversion.

Here’s a quick example:
Let’s say you are running a paid search campaign and you have set up a landing page specifically for your paid search visitors. The page currently converts at .75%. You’re not thrilled… In addition, your analytics package shows that you have a 60% bounce rate on the landing page. Again, you’re not thrilled. With multivariate testing, you can take that landing page and then determine key components of the page that you would like to test in order to increase conversion. For example, you might want to test the header graphic, the headline, and a call to action on the page. So, you have 3 components (or page sections) to test and then you can create multiple versions of each component. Let’s say you tested 4 versions of each component, which would mean that you have 81 possible combinations of elements. Your testing application would automatically change the content for different visitors and then measure the effectiveness of each element and the combination of elements. Cool, right? Note, the more components you test and more versions of components will extend the length of your test. There needs to be statistical significance in order to accurately determine the best combination of elements.

Enter Google Website Optimizer:
There are several tools on the market to help you with multivariate testing, including Offermatica and Optimost (probably the most popular services.) However, you might want to consider a great starter application before diving into the more complex applications listed above. That’s where Google Website Optimizer comes in. It’s Google’s free optimization tool that does a great job with basic multivariate testing. It’s a great way to start your in-house testing program. Heck, it’s free! I’ll take you through a quick example below.

A Real World Example, Step by Step:
One of my clients sells software (B2B) and they noticed a lot of people visiting the demo page, which also has customer testimonials as part of the page. After utilizing their web analytics package to analyze the past few months of activity, I could clearly see that visitors weren’t taking the next step and clicking the call to action to buy the software from the page. So, to increase conversion, we decided to use multivariate testing using Google Website Optimizer.

Here is what I did and how it worked:

1. I broke down the demo page into the core components we wanted to test. This ended up being the header graphic, the headline, and 2 calls to action (one at the top of the page and one below the fold). Note that mapping out your test is probably the toughest part since if you test the wrong components, then you probably won’t achieve your goals. :-) I recommend including everyone at this stage, including the designers, the marketing group, your VP, developers, etc. You don’t have to take everyone’s recommendation, but it typically gives you a good view of the possibilities.

2. Access Google Website Optimizer, which can be found in your AdWords account under campaign management. Create a new experiment and follow the steps for setting up your experiment. There are 5 steps and I briefly touch on each of them below.

3. Click the button for identifying your experiment pages. This will include the page you are testing and the conversion page.

4. Name your experiment and locate your test page and conversion page.

5. The second step (at least in Google Website Optimizer) is to plan your experiment. I briefly explained my client’s experiment above, but this is the step where you figure out which components to test in order to increase conversion. I determined that the header graphic, the headline and 2 calls to action would be the components to test and that we would test 3 different versions of each (giving us 81 possible combinations).

6. Add the JavaScript tags to both the test page and the conversion page. Google gives you detailed instructions for doing this, so I won’t add each step of accomplishing this task. Basically, you’ll be adding some JavaScript to each page and then identifying the unique sections of your page that you will be testing. Once you identify a section, such as the headline, then you will add specific JavaScript so Google Website Optimizer can swap out content on the fly for testing purposes. If you are familiar with HTML and JavaScript, this process is fairly simple. If you aren’t familiar with HTML and JavaScript, then have a developer help you.

7. Then you can click “Check tags on page” once they are uploaded and Google will automatically check them for you. A nice feature…Note, you can also check local pages, if needed by browsing your computer for the webpages.

8. Now is the fun part. You will create the different versions of each component. Don’t be shy here…make sure each version is significantly different or your results may not tell you anything! Remember, we are testing conversion, not how pretty something is. To me, I love this part of website testing. Why? Because it’s sometimes the ugliest component that pulls the best response. Nothing drives designers crazier that showing them an ugly element that converted twice the rate of the prettiest graphic. :-) And remember you are always starting with the original page and elements as a comparison.

9. Once you create and enter each variation for your test components, then you can preview the experiment. You can also tell Google Website Optimizer how much traffic to use for the test. For example, 50% of the traffic to the page, 75%, or 100% of the traffic. You can determine this, based on your exact experiment.

10. Preview your experiment. Having 4 components and 3 variations of each component yields 81 possible combinations. And, Google Website Optimizer enables you to view each combination in a slick preview screen. You can use dropdowns to select each combination or specific versions of each component. **Note, if you see something wrong, change it now. Close out the preview and click the back button to re-edit your versions. If you go live with the experiment, you will NOT be able to change the component versions.

11. When you are ready and everything looks perfect, simply click the Launch Now button to start your test. Now all you need is traffic. :-O By the way, you should start with a page that gets a lot of traffic. In order to scientifically determine which combination of elements works best, you need enough traffic to thoroughly test all of the combinations. The more components and variations you have, the more traffic and time you will need.

Go have a snack, grab lunch, hit the local bar, or wherever you want to boast about the multivariate test you just set up. :-) Check back a few hours later and you can start to review the reporting for your experiment. I plan to write a post that extensively covers the reporting in Google Website Optimizer, but in a nutshell, you can see the following statistics:

Under the Combinations Tab:
* which combinations are performing the best
* which combinations are performing the worst
* each combination’s chance of beating the original
* each combination’s chance of beating the other combinations
* observed improvement over the original
* conversions per visitor

Under the Page Sections Tab:
* the estimated conversion rate for each element within each component
* each variation’s chance of beating the original
* each variation’s chance of beating the other combinations
* observed improvement over the original
* conversions per visitor
* section relevance rating, which basically tells you how important variations in that section are to the overall experiment.

The results of my experiment:
And why I love multivariate testing…we never would have picked the combination that pulled the best response.

1. A header graphic that was relatively cluttered from a design standpoint, but one that builds credibility, performed the best. Needless to say, the designers never would have chosen this header. :-)

2. All 3 of the headlines performed well. The original actually performed better than the rest, but not much better…

3. A simple call to action (as compared to the other variations) performed the best at the top of the page.

4. For the lengthier call to action below the fold, a conversational call to action performed the best. This played on the fact that if you actually got down to the second call (below the fold), then you probably went through most of the content on the page.

Taking Action:
Based on the multivariate test, the optimal combination (out of 81 possible combinations) showed a 60% observed improvement over the original. So we reviewed the results and recently implemented the winning combination.

In Closing…
I hope this post helped you understand what multivariate testing is, how you can use it, and leaves you wanting to set up your first experiment. What I like best about scientific marketing is that you take opinion out of the equation (or mostly out of the equation). If your VP loves one piece of creative and you believe it’s not the right one…test it! It’s hard to argue with real data…and the reporting can act as nice buffer, which will help you keep your job! ;-)


Labels: , , , , , , , , , ,

If you enjoyed this post and you need assistance
with your online marketing projects,
then contact Glenn Gabe today>


  • At 6:07 AM, Anonymous Bryan Eisenberg said…


    Great job explainer Google Website Optimizer. We have also created 7 free resources for Google Website Optimizer for people to get a quick boost to their conversion rates.

  • At 8:10 AM, Blogger Glenn Gabe said…

    Thanks Bryan and I'm glad you posted a link to the 7 free resources for Google Website Optimizer. I hadn't seen that page on your site yet and I will definitely take a closer look at the resources...

    Also, being a big fan of your books Call to Action and Waiting for Your Cat to Bark, I'm honored that you stopped by my blog! You can consider me an evangelist for Persuasion Architecture. :-)

    I plan to write more posts about Google Website Optimizer soon, since I think website optimization is a critical component to the success of any online business (yet it remains one of the most underutilized tools in the web marketing arsenal.)

    I'm looking forward to hearing more from you!

  • At 3:47 AM, Anonymous Alasdair Bailey said…

    Great posting - good explanation of the principals. Your UK and european readers might be interested to hear of Maxymiser - we're a homegrown Multivariate testing platform and can also perform segmentation testing and behavioural targeting.

    check our blog out at or our company page at

  • At 7:13 AM, Blogger Glenn Gabe said…

    Hi Alasdair. Thanks for posting the link and I just checked out your site.

    How does your platform compare to Google Website Optimizer and the more advanced services like Offermatica and Optimost? I'm sure my readers would like to know.

    Thanks again.

  • At 7:12 AM, Anonymous Mike Duncan said…

    Great post! This is the nuts and bolts overview I wish I had earlier to pitch to my business stakeholders. I have a post that is kind of a next step for developers implementing some of the more "gotcha" details at Google Website Optimizer Hacks your readers might also enjoy.


  • At 8:44 AM, Blogger Glenn Gabe said…

    Hi Mike.

    Glad you liked the post. You mentioned your pitch to business stakeholders. Did they push back when you brought up multivariate testing? I think the path of least resistance is to implement a first test and have everyone involved in that first test (designers, marketers, developers, the VP, etc.) Then you can give everyone a vested interest in the results.

    Regarding your post, very cool and I'm sure many implementing MV testing with Google Website Optimizer have not taken it to that level. I also develop with (using so your hacks are extremely relevant to what I do. I'm looking forward to reading more of your posts. Expect some comments from me!

  • At 9:54 AM, Blogger Chris said…

    Nice job explaining GWO, Glenn.

    We recently interviewed the GWO product manager, Tom Leung. Your readers might be interested in it.

    We've also developed a simple calculator to determine how valuable multivariate testing could be for your business.

  • At 2:49 PM, Blogger Glenn Gabe said…

    Thanks Chris and I'm glad you posted a link to the interview. I'll also check out your calculator.

    I especially like the quote from Tom Leung at the end of the interview about Google Website Optimizer creating an "everyone wins" situation. I completely agree.


Post a Comment

Links to this post:

Create a Link

<< Home