“One problem we have in marketing is that adequacy is the enemy of excellence. In many cases, we’re losing the most money by using campaigns that are working, but are not working well enough.”
Dr. Flint McGlaughlin, CEO of MECLABS
In a 2009 New York Times profile, Marissa Mayer—Google’s vice president of search products and user experience—recounts discovering that Google users were more likely to click on a toolbar that was a greener shade of the current blue. The team liked the existing color, so Mayer decided to split the difference and make the toolbar a hue exactly between the two colors. Unsettled by her gut decision she decided to test 41 gradations between the two blues to see which got the most clicks.
Early and often
You’ve probably heard the somewhat cynical remark: Vote early and often. At Masterworks we like to say: Test early and often. As soon as you make something new for an email or landing page start testing it, and once a test has won set up another test to try to improve on the last version.
What you should be testing
If testing 41 shades of blue seems like an onerous task, consider that Google’s bottom line is attached to people clicking text ads, so it’s in their best interest to enhance for clicks.
You should test the user actions that most directly affect your bottom line. It might be a headline, a call to action button or the way you sign off your emails.
How to run an A/B test
An A/B test can be easily implemented. Here is an example of one in six steps:
- Decide the most important thing to test—what optimization will have the biggest impact on your bottom line?
- Write a measurable test question—it should always be a sentence that starts with “which,” e.g. “Which button color will get the most donations?”
- Create treatments—the presentation of your test, e.g.
- Get a lot of observations of your treatments—statistical validity is it’s own article, for now, just get as big a sample size as you can and show one half treatment A and the other half treatment B.
- Determine the impact of your test—did one treatment get more donations? Why?
- Repeat with a different treatment—e.g.
Testing resources
There are a lot of resources out there to help you easily create A/B tests.
- Testing: An easy way to test emails is just split your list using tools from your email service provider. A tool we recommend for landing pages is Optimizely. It’s simple interface makes it incredibly easy for anyone to run an A/B test, even with no knowledge of HTML. Other good tools include Visual Website Optimizer and Google Website Optimizer.
- Analytics: Make sure you’re using a good analytics tool—like Google Analytics—so you can track your tests accurately.
- Inspiration: If you need some ideas for what to test, follow these testing sites: Which Test Won, ABtests, Visual Website Optimizer’s Split Testing blog.