Why Testing is Hard

"The question is, if everyone wants to test, then why isn’t everyone testing? The answer is that testing is simple in concept but really, really hard to execute. But not for the reasons you think."

One of the joys of being in the digital space is the ability to test. Yet – if you ask many marketers about what they need to do more of in the next year – testing invariably comes out at or near the top. An effort we all seem to want to do, yet where few of us seem to feel success.

The question is, if everyone wants to test, then why isn’t everyone testing? The answer is that testing is simple in concept but really, really hard to execute. But not for the reasons you think.

To set up a digital test is fairly simple – come up with a variation of your current program, run it to a limited percentage of your audience and then measure the results. If the results look promising, run it to a larger percent of your audience and see if the same direction hold true. If so, roll it out to the winner. (Yes, I know – most people skip that second step.)

So if the testing process is so easy, why is it so hard to execute? The answer is simple…school.

Think about your life for a second. At about the age of 4 or 5, you’re shepherded into a classroom of other kids. While some kids never really break out of the fingerpaint and daydreaming stage (ie – creative people), most kids learn quickly that you get rewarded by knowing more than your compatriots. Good grades are the key to attention and (hopefully) positive reinforcement from your parents. You study the books, memorize what they say, then regurgitate what you’ve learned.

Testing throws all of that training out the window. Testing says “I don’t really know what the answer is…” which is enough to stop most people in their tracks. But what testing really says is “I don’t really know what the answer is…but I can try to find out.” This, of course, is the sweet spot for all the nerdy science kids in the classroom. Most marketing people I’ve met are NOT the nerdy science types.

Successful testing means that you have to embrace the idea that you are wrong about something. Think for a second about all the times you’ve been wrong in your life. Does it bring you feelings of great joy, laughter, piles of cash and romantic journeys? Didn’t think so.

We’ve been conditioned to think that being wrong is BAD. But successful testing means that you have to embrace the idea of being wrong. It also means that you embrace the idea that there may not actually be a better answer. Because if you’re always developing winning tests, it means you’re not testing far enough outside your comfort zone.

I had a team who embraced the unknown and really got into testing. (I was comfortable with the idea of them being wrong.) One day – after looking over lots of data – I decided to throw my hat into the ring and develop a test. I was sure my test would win. I even bet my team dinner that I would win. Then the test results came in an – as you may have guessed – I did not win. In fact, my test got thrashed against the champion. It was a terrible feeling but it taught me a valuable lesson -

What I think does not matter. What matters is what I can prove.

The team became a powerhouse of testing – we tested everything. When we ran out of the basics, we made up crazy stuff to test. All because we embraced the unknown and wanted to know a truth that was unvarnished by our personal bias and/or expertise.

Which leads us to the second problem with testing.

If you accept that testing can lead you to a place where what you think can be replaced by what you know, you’ll end up butting heads with many, many traditional marketing people who live too far inside their own heads and think they are marketing to themselves.

How many of you have been part of meetings where some person (usually VP or above) droned on about “I think, I think, I think, I think, I think.” One day I had one of those meetings until I finally stood up and said “I don’t really care what you think.” Unfortunately, I neglected to add the “prove” part. Hint – this is not a tactic that will result in career longevity.  A better response to someone who is stuck in their own world is “Great! Let’s test that!” If their idea is a winner, then you all win. If it’s a loser, you can always drag it out the next time that person gets a little too aggressive with their own opinion.

Being wrong is hard. Testing is all about being wrong. Embrace it. After all…you might learn something.