I’ve recently been learning Ruby on Rails and after hearing the term for a long time have actually begun to understand what “test-driven development” is. The idea is that you define success for a piece of an application by writing a test that “uses” the application in predefined ways and tells you whether or not it produces the expected results. Before I go on I should admit that in an effort to learn how to make things in Rails faster, I’ve actually not written any tests. Bad, I know.
I think the idea of test-driven development is pretty cool. It forces you to set specific goals, makes it clear when you achieve them, and hopefully makes achieving them even easier and faster since the scope is so clearly defined. Thinking about it more, there are probably other parts of life and certainly work that would make sense to approach this way. One that came to mind and made for a catchy title was testing itself, in the product and user sense.
Surveys, A/B tests, multi-variant tests and many other methods of learning from customers have become wildly popular, and rightly so, thanks to the work of Steve Blank, Eric Ries, Sean Ellis (who first taught me about it) and others in what’s generally been termed “The Lean Startup” movement. Like any good fad, I think that some followers have fallen victim to function following form — in other words, I’ve seen a bunch of people use these tactics for the sake of using them. One instance of this that sticks out is running tests where virtually no result is going to teach you anything.
The most egregious offender has to be surveys, I think because they’re the easiest to create and the most vulnerable to lazily copying from others. Every question on a survey — like every line of code in an application — has an implicit, real cost. Usually it’s a lower response rate, which means you get less information and it takes longer to get it. So a great deal of effort should be put into eliminating unneeded questions, just like a great deal of effort is often put into eliminating unneeded code. Or (here comes the theme) in a test-driven development world hopefully the unneeded code is never written in the first place, because you’re writing code based on test requirements and writing tests requirements based on what’s needed. It’s a beautiful system.
Back to the surveys. There are several “standard” survey questions that I see almost everywhere and have a way of creeping into new surveys as you write them: gender, age, income, location, referral source, etc etc. Certainly all of these can be valid survey questions, but they’re probably not all important all of the time. Unfortunately it’s really, really easy to add them to your product survey and some services like SurveyMonkey will even automate it for you. Pretty soon this turns into survey-by-committee and you’re asking 30 questions with boondoggles from every part of the company added. Biz dev wants to know about the users’ favorite websites, marketing wants to know about age and gender, sales wants to know about purchase intent for mid-size cars (wtf?) and the list goes on. Meanwhile, if the scope of the survey had been properly defined, it would be clear that none of these were actually required; or, if these were actually required, it would have been made clear sooner and maybe someone would have fought against it, thanks to a test.
This is a trivial example only to make a point, but I think it’s an important point that comes down to two ideas for me:
- Keep in mind that just because something is easy to do doesn’t mean it’s low risk
- Upfront investment in making sure everyone agrees on the goals can pay huge dividends later
Testing to accelerate learning is great and more companies should do more of it, but adding unneeded complexity (for you and for your users) through more survey questions or button colors doesn’t always improve your tests.