3 Things Marketing Experts Don't Tell You About Split Testing
by Jim Stone
Published on this site: November 10th, 2005 - See
more articles from this month
I've been split testing for a while now, and even use some
pretty advanced methods these days. It has worked out well
for me, but I still remember many early disappointments.
Many very smart marketers will tell you there's a lot of
power in A/B split testing.
They'll even tell you that you CAN get 1000% improvements
by changing a single word - or even a single letter.
These marketers are right. But if you think they're telling
you the whole story, you're in for a rude awakening! I found
out there were a few things the experts hadn't told me.
Here are three of the most important things:
- It's More Difficult To Set Up And Run A Simple Split
Test Than You Might Think:
In order to set up a good split test you must:
* upload a new sales page with just one change to it. You
might also have to create two different "thankyou"
pages, depending on how you set up your tracking.
* get a script that splits traffic between the two pages.
Ideally this traffic splitter will split the traffic randomly,
and not just in an "every other" fashion.
* make sure your script sets a cookie on the visitor's machine
and serves the same page to them the next time they come
to the site.
* put a tracking system in place that will track your visitors
and your sales for both pages separately.
* set up a database (perhaps).
And getting set up is just part of the hassle. It's also
difficult to run the test and interpret your results.
One time I set up a test, and started sending traffic to
the pages being split tested. Then I noticed a typo on ad
B. So I corrected it. Now, "strictly speaking"
I shouldn't have changed anything during the test, but I
was probably ok doing this, even though it's not technically
good form.
That wasn't the problem. Here's the bad part: when I made
the change to my page, my editing software (without my knowing
it) changed the names of the fields in my opt-in form on
page B, and, as a result, it didn't work any more.
I didn't realize the mistake until a week later. During
the week I had marveled at how ad A had done so well, while
ad B pretty much just stood still.
That was one week wasted. My results were meaningless.
(I don't know about you, but this kind of thing happens
to me all the time. These are the stupid pesky gremlins
that infest every new thing I try. Why can't things ever
go as planned?)
Other things can happen, too. Ad campaigns can stop. Hurricanes
can strike. Terrorists can strike. Christmas can strike.
Your page can get listed on the front page of Yahoo, and
suddenly you're getting a different kind of traffic - and
lots of it. All of these things can create "noise"
that affects how your ads perform.
You also have to know how long to run your test before your
results are "statistically significant".
Truly, simple split testing can be something of a hassle.
But the hassle isn't really the worst part about split testing.
Here's something else the split testing "gurus"
haven't told you:
- You Can't Consistently Get Great Results On Your Own.
If you look at your landing page, and ask yourself, "How
can I improve this?" You will probably come up with
something to test. And then you will test it.
Good for you, right? You were Johnny on the spot. You got
going right away. You didn't sit around debating the merits
of your choice. You took action. What could be wrong with
that?
The problem is you probably came up with some safe little
change. Some timid little tweak. And it probably didn't
help much, if any at all (and it might have hurt). It probably
wasn't a golden nugget that significantly boosted your response.
It might have been, but it probably wasn't.
The truth is, to get best results, you need to get others
to help you brainstorm for testing ideas.
But the experts haven't told you that.
And there's one more thing they haven't told you:
- With Simple Split Tests, You Should Be Prepared For
Lengthy Strings Of Disappointing Tests - Even When You Generate
Great Ideas.
Did you know that most of the changes you make to your landing
page will not help?
According to Charles Holland of Qualpro (Breakthrough Business
Results With MVT), if you run 25 split tests, about 13 won't
show any significant difference. And 6 of the proposed changes
will have a negative effect. That leaves about 6 to have
a significant positive effect. That's 6 improvements out
of 25 tests.
But it gets worse. Did you know it takes about 100 ideas
to get 25 elements that you can easily split test? That's
because 75% of the ideas you will generate for improving
your ad will be things that are too expensive or time consuming
to implement, or can't be combined with other things you
want to include on your page.
That means you must generate 100 testing ideas to get 6
that will actually make a positive difference on your page.
Please, re-read that sentence, and let it sink in this time.
It's that important.
If you have ever been discouraged by a string of 5 or 6
A/B split tests that made no difference, all in a row, you
know in your heart that these numbers are very close to
the mark. You might have felt alone - just an unlucky sap
for whom nothing ever goes right - but you were actually
just experiencing normal, run-of- the-mill split testing.
The experts didn't tell you that, either.
They didn't tell you how much of a hassle it can be, they
didn't tell you what you need to do to generate good ideas,
and they didn't tell you how long it can take to get meaningful
results.
So How Do You Deal With These Problems?
- The hassle. You're just going to have to bite
the bullet. Split testing is one of the best ways to improve
your profits. Get some good software, and some good information
about running and interpreting your tests, and just dig
in. This doesn't exactly made it any easier, but at least
you've been warned.
- The help. Imagine what kind of ideas you would
get if you assembled 20 people in a room. Other marketers,
potential customers, existing customers, maybe your mother.
And you ask them the question "How can I improve this
sales page?" And you let them throw ideas out, and
let other people's ideas inspire new ideas, and you do this
until you have 100 ideas - ideas that come from many perspectives,
from people with nothing to lose, and from people who are thinking more and more "out
of the box" as the session goes on.
If you could do THIS, then you would have a good chance
of finding some truly good testing ideas.
Such a big brainstorming session is impractical for most
people. That's true. But it can't hurt to be told what the
ideal is. At least now you'll be thinking of ways you can
approach this ideal.
For instance, you might consider checking out what your
competitors are doing. You might pay better attention to
the marketing books you've read. You might solicit ideas
from discussion forums.
And maybe, just maybe, you will go to the trouble of actually
getting a few friends and relatives to sit around with you
and brainstorm a bit.
- The discouragement. First, it helps just to be
told what to expect. Winning ideas are rare, but they usually
deliver big results when they come. So the hunt is worthwhile,
but it can be discouraging.
But there's actually something you can do to change the
situation in your favor.
There's a new method of split testing called "accelerated
split testing", or "Taguchi testing", or
"Multi-variate testing". It allows you to take
all your ideas - the good ones and the bad ones - and test
them all at once. Your chances of hitting a winner (or two
or three) somewhere in that jumble is actually very, very
good. If you set your tests up right, you should get significant improvements from almost every single test.
So there you go. Three problems the experts haven't told
you about, and three ways to deal with the problems.
Now you've been told.
Jim Stone is an accelerated split testing expert. He has
developed an accelerated split testing software product, and
writes regularly on the subject. You may receive his free
course: "10 Steps To More Sales With Accelerated Split
Testing" at: http://www.splittestaccelerator.com
|
