Finding the next great thing is hard

Here's the rub with so-called Web 2.0 product development, in my eyes:

Most people who develop Internet products and services these days still do so based on their personal assumptions about what potential customers want.

Ask anyone with online experience who the standardbearers of Web 2.0 innovation are, and you'll hear, more often than not, one name: 37signals. It's the company that designed streamlined Web productivity services such as Basecamp, Backpack and Writeboard.

I like these guys, from what I know of them. The 37signals principals just plain build and launch stuff, believing others will be interested in it because they are. They proudly proclaim the simplicity and focus of their methods, and I admit the lack of distractions -- from market research, corporate stakeholders, downstream customers and such -- sounds refreshing.

No market research? Jason Fried says it himself: "research isn't our thing."

That's where I struggle. I wish I could be so confident that I know what customers want based on what I want. Much of what makes product development successful anywhere -- especially online -- is the right mix of two things:

• Instincts for understanding customer wants. Yes, I said wants. Forget about needs. Any true human need (I mean the basics such as shelter and health) that could reasonably be met by an Internet service probably already has been. Almost all product development I see these days aims at wants, many trivial.
• Almost religious reverence for learning from trial and error, and adapting while it's feasible. If you aim a product at a want, and find out you missed, adjusting your aim becomes more expensive as time passes.

I also believe well-conceived customer research hones those instincts up front, and keeps some trials from becoming errors. To me, customer development (aka research) leads to product development, which may or may not lead to business development. The business flows if the product actually satisfies the customer wants identified in research, and then only if the research revealed wants profound enough that their satisfaction is worth enough to enough people.

When I build things online, I'm seldom spending my own money. When others sign the checks, they become stakeholders with an investment to protect. They must always be persuaded that enough people will want the resulting product or service to generate a return on their investment in a satisfactory time period. (Unfortunately, haggling over satisfactory ROI time frames has a way of stifling innovation.)

Friends already know I'm no fan of "soft launches." To me, a soft launch is what happens when you think you have a hot idea, but can afford neither the research to test it nor the marketing to promote it. All that's left is to put it out there timidly, and see if anyone nibbles.

Good research provides valuable fuel for innovation, and can build the confidence necessary to take risks with development and marketing money.

But what makes good research? Every method we know of reaching out to potential customers, be they everyday consumers or high-power business clients, seems watered down:

• Surveys: Phone surveys cost a lot. Mail surveys don't always get good response rates. Web-based surveys are tough to verify and localize. And people feel as though they're over-surveyed, so it's tough to get a good sample of replies especially from the most desirable potential customers -- those with money to spend.
• Market data analysis: Sure, you can find out how many people in a given ZIP Code, state or nation exhibit likely luxury car-buying tendencies. But how often do market characteristics and demographics really give you the clues you need to innovate? They're more useful when you need to figure out where and how to deploy an existing consumer product. With new Web products, chances are your customer base will be scattered amid a long tail of demographic profiles that isn't easily picked up in broad market analysis.
• Focus groups: Group dynamics can so easily ruin learnings. It seems almost every group has an "alpha" participant that sways gentler minds in the room. Unfortunately, as often as not it's the facilitator.
• Inbound calls, mail, e-mail: Generally, these are existing customers of existing products -- not always the best group to listen to when you are trying to innovate. Right, Clay Christensen?
• User testing: Call it what you want: usability testing, one-on-one interviews, invitational beta tests. User testing works best, I believe, on products that have been in the field a while -- at least long enough to allow you to test a mix of current and potential users. I love to do user testing on an existing product with an upgrade in the works. Then you can really learn what needs improvement and what is best left alone. Usability testing on prototypes, even on paper, can help solve pressing interface issues, but won't tell you much about the likelihood of customers engaging with a brand new product.

It seems we've hit a loop: we need good research to spur innovation with the best chance of business success, but common research methods all have flaws that could make results misleading, thus reducing the chance of business success.

Do we just go with the gut? No. I still think those people who sign the checks in the corporate world will not accept that. And I don't think 37signals has enough job openings for all of us.

Instead, I think we take some logical steps and work for the best:

• Spend what it takes to get high-quality research methodology in any form. It's a competitive field. Shop around. Check references, asking especially for examples of research where the outcome was improved business metrics, product loyalty and/or satisfaction.
• Combine research methods to mitigate the weaknesses of any one. Surveys can provide a dumbass test for any ideas or hunches you get from focus groups. But no matter how many people in a survey tell you they just gotta have the product you're building, body language and facial expressions in usability tests during beta or after launch can speak volumes on how emotionally engaged users will be.
• Any interaction with current or potential users of a product is better than no interaction. Poorly crafted user testing may not yield quantifiable results, but hearing what people say about and seeing what people do with your product always feeds instincts.
• Don't let the researchers be the only ones who see the customers in focus groups or user testing. Stakeholders never learn as much from reading a research briefing after the fact as they do from watching people while they are participating in the study.
• Remember that customer research will not always yield logical results. You're asking people what they think. People aren't always logical, and frankly, they don't always think before they tell you what they think. So, Mr. Spock, research may be better at refining the gut feelings in your human half than the reasoning skills in your Vulcan half.

Ultimately, customer research is just conversation with structure, more listening than talking. The more conversations you have with more people about your idea -- and the more you listen -- the better you will sense its likelihood of success.