A data-driven approach to communications

This blog post mentions a small app I wrote to do some testing. If you want to download it, it’s available on Github here.

Messaging and positioning

Three wholesale mlb jerseys weeks ago I took a look at our messaging and positioning for my new company, Insidr. In other words, I had to figure out how to optimally present our company and product: the words to use, the example to cite, the feelings to emote, etc. I wanted to be very analytical about it and arm myself (and my team) with as much data as possible to inform decisions (and let us call bullshit when needed). What follows is the process I used, from start to finish, to make an analytical decision about an emotional question.

Cast a wide net

I wanted to force myself to try lots of things and at least allow for the chance to be surprised. I wrote (read: stole from someone else) a 5 part sentence that included all the key elements I wanted to test: our target customer, a reference point to explain what we do, our point of differentiation, why customers should believe we can do what we claim, and what benefit our customers will get.

I think it’s safe to say that if you can nail all these things, you’re going to be in a great spot to communicate what you’re all about. I wrote 3 ideas for each of these 5 pieces and made sure they could be plugged into the following sentence model:

For [target] [company] is [reference] that [differentiator] because [believe], so that you can [benefit].

Here’s a wholesale nba jerseys (made up) example: For [people that like hamburgers] [Hamburgler] is [a way to eat more often] that [has indexed the world’s burger joints] because [we had a great idea about finding burgers], so that you can [avoid overcooked beef].

With 3 versions of each piece, I suddenly had 243 versions of positioning statements for Insidr!

Multi-variant testing

Next I We needed a way to test all of these ideas and measure the results. I wrote a small app that would create two versions of the full positioning statement using randomly chosen variants (without any overlap) and let a respondent vote on which one they preferred. Here’s what it looks like:

Each time someone choses an answer, the winning and losing variants for each piece of the sentence are logged in a database for safe keeping.

Next I setup a Mechanical Turk task that gave Turkers a short explanation of what they were about to see, then loaded the voting app in an iframe. I turned it on and went to sleep.

Because wholesale jerseys China I wanted to avoid people randomly clicking over and over to make money, I limited it to one vote per user. I also limited the task to people in the US, Canada and the UK. This meant it took about Startup 3 days to get reasonably significant results. To my mild surprise, there were clear winners and losers.

Refining and more testing

There were two themes that emerged from the multi-variant tests for Insidr: positive and goal-oriented statements did best. That may sound obvious, but it’s not vente du viagra. Until this point our positioning for Insidr had been, in many ways, exactly the opposite. We were, if not negative, at least confrontational in regards to the companies we were supporting. We were also cheap jerseys very tactical and thought it was important to make it immediately clear to customers exactly what we did and how we did it. The data seemed to be suggesting the exact opposite.

I got 168 results from MT after a few days and in each case the results were pretty similar, with the best variant winning 65-70% of the time and the worst winning 30-35% of the time. I did the confidence intervals on these and got back between 85-99% confidence that the best version was beating the 2nd best version. That was enough for me to move on to the next step.

Next I setup a new and more detailed test. This time we created three short positioning statements: one using our existing (tactical, confrontational) positioning, one with a purely positive and goal-oriented positioning, and one hybrid of into the two. We used these statements as a basis for demand testing asking respondents if they would use the service described, how often and whether or not they would pay for it.

Again I used Mechanical Turk to drive responses and now added SurveyGizmo to the mix to handle the questions, which let’s you do a little bit of scripting using custom URL parameters to change pieces of the survey. I setup a custom redirector to split test traffic to different survey versions that each used one of the three positioning statements to describe a “hypothetical website.” Another Mechanical Turk task and we were off to the races.

Again, there were clear winners and losers, and the data from the multi-variant test held up. Positive and goal oriented was the way to go with about 95% confidence.

Acting on the data

Now that we had a clear signal about what to do, it was time to roll the changes into our product and marketing. First up were our front page and search ads. Both were updated with new messaging (and the front page got a new design) that was positive and goal-oriented to in the point of being almost vague. Before the tests that jerseys led to this point I’m pretty confident we never would have thought this made any sense, but we trusted the data and tried it anyway.

The results were shocking. Literally overnight our search ad click through rates went up by 135% and our front page bounce rate went down by 36%. The combined effect was almost a 200% improvement on what we were doing before. To add some cheap jerseys icing on the cake, the improvement in the quality (CTR) of our ads resulted in the CPC’s dropping by about 50%. Not bad for a couple weeks’ work.

Here’s an example of an old ad. It’s pretty tactical, and maybe a bit negative (certainly not overwhelmingly positive):

And here’s a new ad. It gets rid of the details about how Insidr works and focuses just on a benefit to the customer, even to the point of being unclear. The new ad KILLS the old ad:

Here is the old version of the front page. It’s pretty detailed and leads with a very negative question. We thought this was a great way to empathize with a frustrated consumer:

Here is the new version of the front page. It’s got a big, happy looking image and makes a lot of nice sounding promises without much information to back them up. It KILLS the old front page:

If you wan to see the code I used for the multi-variant testing, it’s on Github here.