B2B Marketing 101: A/B Testing

April 19, 2013 | by | Category:

My marketing professor used to answer every question about marketing tactics the same way – “it’s testable.” Her voice still echoes in my head every time I consider a subject line or a new channel, and I still open my notes from her class every once in a while. So in my biased opinion (what opinion isn’t biased?), testing is one of the core marketing arts that any marketer should master and practice regularly.

As marketing athletes we’re expected to plan, run and execute marketing campaigns that deliver high quality leads to our sales team, but we’re only marketers — not miracle makers (or fortunetellers) — so testing is our way to use data to tell predict the future.

MarketingProfs Mouse Pad

Testing can be done on any piece of marketing asset and its roots are actually in direct mail marketing, where armies of marketers used to run robust testing on envelop sizes, font sizes, colors, copy and creative. With the emergence of online marketing and the ability to track, measure and report on every single pixel on your website, testing is now easier than ever. But it has also become an overwhelming task that leaves marketers helpless in the face of too many possibilities.

A/B versus multivariate testing

By its definition, A/B testing is a test designed to compare two versions of a marketing piece (web page, email, etc.) to determine which version performs better. Multivariate testing is a process by which more than one component of a website is tested at the same time. You can think of multivariate testing as multiple A/B tests running on the same piece of content at the same time.

Multivariate testing can theoretically test the effectiveness of limitless combinations. The only limits on the number of combinations and the number of variables in a multivariate test are the amount of time it will take to get a statistically valid sample of visitors and computational power. (Wikipedia)

As you can probably see, multivariate testing requires a dedicated software as well as someone who can manage and run the tests. It also requires enough traffic to get to a statistically significant result, so if you don’t want to wait a few months before you can make a decision and would like to start testing some of your marketing activities right away, A/B testing is the way to go.

8 rules for A/B testing

First posted on SearchEngineWatch – read the full article.

  1. Hypothesis – every test starts with a hypothesis. It doesn’t have to be sophisticated or smart, but it has to be clear.
  2. One variable per test – the rule of thumb for variables is that the number of variables you test is always one less than the number of variations. Or in other words, for every tested variable you will have two variations.
  3. Clear and aligned success metric – this is your “decision rule,” the one metric that will tell you which variation performed better.
  4. Volume and statistical significance – to make sure your results are not a result of chance.
  5. Test group and splits – to ensure the user experience stays intact.
  6. Randomization – to eliminate variables that will affect the results.
  7. Common sense – because it’s about making things better, not worse
  8. Documentation – for the future generations.

A/B testing checklist – 8 steps for carrying out an A/B test

Read the details on How I Used Optify and Google Experiments to Run a Landing Page A/B Test.

  1. Choose the variables to be tested
  2. Set up the hypothesis for each variable
  3. Define the Decision Rule
  4. Design the test
  5. Determine the sample size
  6. Choose the significance level
  7. Conduct the test
  8. Make the decision on each variable

A/B testing resources and solutions