Skip to content

Stuck in the Building

October 16, 2012

This is a long piece, examining biases and heuristics that undermine customer discovery.  More like an essay than a blog post, but, I hope, useful for people who are trying to get the most out of talking with potential customers.

“There are no facts inside your building, so get the heck outside”

Steve Blank, in The Startup Owner’s Manual

“Gentlemen, our resources are limited….at present these resources are undertaking to speak to local people, and to act on what they find.  This has led us to confusion.”

“Confusion?” says Marland.

“Confusion sir.  By relying on the testimony of individuals, we have exposed ourselves to the prejudices of those individuals, and we have thus been chasing our tails.

Lloyd Shepard, in The English Monster

Talking to customers early and often is the lean startup movement’s prescription to cure the disease of founder fantasy.  The disease is real enough.    Creative people tend to believe their ideas; insightful people have faith in their insights.  Entrepreneurs are generally insightful and creative, so it’s normal for these people to believe they are right and that their product will be the next big thing.  Too often, this leads to startups based on plausible but mistaken ideas about who customers are and what they want – a fatal flaw if not rapidly corrected.

How does talking to lots of customers address this problem?  Three main ways:

  1. If enough customers tell you you’re wrong, you might start to believe them.
  2. With enough data, anyone can do a better job at pattern matching.
  3. If you have to talk with scores of people, you’ll be forced to move beyond your circle of friends and their friends, which will expose you to responses from more objective strangers and therefore to the real level of interest potential customers may initially have in your ideas.

All of that is helpful, just not terribly helpful.    It fails to address some of the key reasons behind “false positives” – validations of hypotheses that are actually invalid.  Behavioral economists have done a fairly thorough, science-based examination of how people make decisions, and have concluded that many of the tools we use for decision making simply and demonstrably lead us astray.  In the customer discovery context, customer interviews will frequently seem to validate wrong hypotheses.  Customers’ predictions about their future buying behavior will be wrong; customer’s insights into their own decision process will be flawed; customer’s inferences about how other people may act will be fictional.  Worse, this is not a peripheral phenomenon.  You can interview fifty people, and forty of them will tell you they will buy when it’s ready; but only one will, or none.  Worse still, you, the entrepreneur, are subject to the same errors and decision-making weaknesses, which compound the errors made by your potential customers.

In behavioral economics, the rules of thumb that we habitually use in making decisions are known as “heuristics.”  Some heuristics lead to good decisions in some situations, bad decisions in others; some nearly always lead to bad decisions, and a few usually improve decision making.  We’re commonly aware of many of these heuristics, such as overconfidence, over-valuing sunk costs, being paralyzed by too many choices, using hindsight, ignoring the importance of base rates, etcetera.  The science of behavioral economics has focused on identifying and defining these heuristics, unveiling where they lead to mistakes, measuring the biases they inject into decisions, and pointing decision-makers toward strategies that minimize errors. This article focuses on five heuristics that are less well known but that play a big role in undermining the value of customer discovery:

  • The Availability Heuristic
  • The Curse of Knowledge
  • Confirmation Bias
  • The Halo Effect
  • Representativeness, and it’s subset, the Conjunction Fallacy

Availability Heuristic

It’s not news that many mistakes get made because people are mentally lazy.[1]  Amos Tversky and Daniel Kahneman were pioneers in elaborating, refining, and measuring this phenomenon, which leads us toward certain conclusions, not because they are based on the preponderance of the evidence, but instead because they are based on the evidence that comes most easily to mind.  For example, we think robberies are much more frequent if we read about one yesterday, or if for some reason we have a vivid memory of one.  We overestimate the frequency of terrorist attacks and celebrity divorces because examples come so easily to mind.

This heuristic effects, and can even be used to manipulate, how people see themselves. In a classic study, researchers examined their subject’s views about how assertive they were.  Think for a moment how you would answer the question:  “Are you an assertive person?”  To answer in an unbiased way, you would have to review a long series of interactions you’ve had with people, over many years, assess them for assertiveness, and come up with some kind of tally and scoring mechanism.  Instead of doing all that, most people, when faced with this question, substitute an easier one:  “What examples come to mind of me being assertive or not, and what can I conclude from them?”  If you happened to have stayed quiet at two business meetings this week, and waited patiently for a table at a restaurant, and backed down in an argument with your spouse, then you would be likely to conclude that you are, in fact, a meek person – even if you were Donald Trump.

Conversely, if you have a harder time recalling something, you will underestimate it.  In the study, two groups of people were asked to recall examples of being assertive, and then to assess whether they were assertive people.  The first group was asked for four examples, the second group for twelve examples.  The first group found it easy to remember four good examples, and they generally concluded that they were assertive.  People in the second group found it difficult to come up with twelve examples, and generally concluded that they were meek.[2]

In customer discovery, both the entrepreneur and potential customers will judge value propositions, features, and the likelihood of success based on examples that spring to mind.  One entrepreneur we know was trying to support a manager who was moving to a different role.  The manager said he wanted a way to pass on operational information.  The entrepreneur asked the manager to recall circumstances where he had passed on this kind of information.  Through the discussion they realized that the ability to make and capture checklists would be very useful – so useful that the manager said he would buy a product like that.  Based on this and similar interviews, the entrepreneur went ahead and developed the software.  But when he went back to the potential customers, they weren’t interested in buying.  The fact that potential customers could recall instances where the product would have been useful biased them toward believing that they would use it frequently.  Unfortunately, those instances were too rare to justify buying and learning to use the software.

The Curse of Knowledge

While it’s easy to make mistakes because you can only remember a few examples of something, the opposite is also true – knowing a lot about their area of expertise can lead entrepreneurs into error.  Psychologist Elizabeth Newton’s doctoral thesis, “Overconfidence in the Communication of Intent: Heard and Unheard Melodies”[3] has spurred two decades of research, shedding light on everything from education techniques to advertising, to relationship counseling.  Her work was based on a simple, easily repeatable experiment:  Two people are given a list of popular songs.  One person chooses a song, and attempts to communicate it by tapping on the table between them, and the second person attempts to guess the song.  In consistent tests, the tapper predicts that the listener will identify the song about half the time – a 50% accuracy rate.  In fact, listeners correctly identify the song at a rate of 2.5% – twenty times less often than predicted.   The disparity arises because the tapper hears the song – without the song in his head, he couldn’t tap it out accurately.  But once it’s there, he cannot unlearn it.  He loses the ability to listen to the taps as the listener hears them, as simply taps, so he loses the ability predict how accurate the listener will be in identifying the song, and wildly overestimates how predictable the song is from the taps. This has broad implications.

When an entrepreneur talks about a prospective product she has it firmly in mind.  She can’t unlearn what she “knows” about the product, and will overestimate, by a factor of 20, the hypothetical customer’s likelihood of understanding what it is.  This problem may be even worse, because while the listener in Dr. Newton’s experiments were trying to match the tapping to a song thy are already familiar with, the customer is trying to match an unknown product to an unmet need that is itself difficult to pin down.  In customer discovery interviews, the curse of knowledge leads entrepreneurs to extreme overconfidence about their how well they have communicated their plans.  It is very difficult for entrepreneurs to discount a customer’s comments for the likelihood that those comments are based on misunderstandings.

Confirmation Bias

In a seminal 1991 article, the psychologist Daniel Gilbert founded modern study into the mechanics of belief.[4]  Gilbert brought current psychological research to bear on an argument between the philosophers Descartes and Spinoza.  Descartes believed that people first perceive and/or comprehend something (an object or an idea) and then assess whether or not they believe it.  Spinoza argued that the act of perceiving/comprehending involves believing, and that rejecting an assertion or questioning a perception only comes later.

The research strongly supports Spinoza.  When you hear something, you automatically, unconsciously, see it as true.  Critical systems, which judge the truth of statements, operate more consciously and only go into action after the words are understood and believed – comprehension and belief arrive first and together; judgment and disbelief follow.  Some consequences are that if people are told something and then distracted, the distraction will interfere with critical thinking and they will be more likely to believe it.  Considerable evidence shows, for example, that people are more likely to believe the claims in commercials if they are tired or distracted.

In customer discovery interviews, customers are working to understand what the entrepreneur is describing.  That work involves believing that what the entrepreneur says is true.  If the use scenario has multiple steps, or the product has multiple features, then the customer is likely to unconsciously stipulate the truth of the first step or feature, just so that she can comprehend it well enough to make sense of the next step of feature.

The Halo Effect

The Halo Effect was first described and tested in detail by the psychologist Solomon Asch in 1946.[5]  Kahneman describes the effect as “exaggerated emotional coherence.”  Asch’s experiments showed that we use first impressions, or primary impressions, as guiding principles to interpret additional information, striving to create a coherent “story” about whoever or whatever we are trying to comprehend.  If you appreciate Lance Armstrong for his work with cancer patients, you will tend to interpret doping allegations as sour grapes from his lesser rivals.  Conversely, if you think of him as egotistical or coldly mechanical, you’ll be more likely to think he cheated.  This effect manifests itself almost instantly.  In one of Asch’s key experiments, he gave subjects descriptions of two people, and asked them to write short descriptions of each one.  Here are the people:

Alan: intelligent – industrious – impulsive – critical – stubborn – envious

Ban: envious – stubborn – critical – impulsive – industrious – intelligent.

The descriptions are identical, but the order of presentation makes all the difference.  Subjects had much more positive impressions of Alan than Ben just because they were exposed to the good traits first.  Interestingly, Alan’s negative traits were interpreted as less negative in the light of the positive ones.  “Well, of course Alan is critical; he’s smarter and works harder than other people, so it’s natural.”  The same effect operates in reverse for Ben “Yes he’s industrious, he has to work hard to be better than everyone else.”

In customer discovery, the Halo Effect distorts reactions both overall and in respect to features or aspects of the plan.  Potential customers can think “This entrepreneur seems really smart, her idea must be good.”  The can also let a positive impression of one part of the business model color the others:  “Customers will love this, so partners should be found easily.

The Conjunction Fallacy

Tversky and Kahneman’s most famous experiment was called the “Linda Experiment.”  In one version, large groups of people are presented with a description of Linda and a question.  Here’s the description:  “Linda is thirty-one years old, single, outspoken, and very bright. She majored in philosophy.  As a student, she was deeply concerned with issues of discrimination and social justice, and also participated in antinuclear demonstrations.”

Here’s the question:

Which alternative is more probable:  1) Linda is a bank teller.  2) Linda is a bank teller and is active in the feminist movement.

Respondents overwhelming pick #2 – it fits the description so much better.  Of course, it is logically impossible.  #2 is a subset of #1, so there is no way #2 is more probable.

Twenty five years of experiments that follow up on the Linda experiment have yielded some important conclusions.  From a customer discovery perspective, maybe the most important one is that adding details to a story has a contradictory effect – on one hand, it makes the story seem more likely to be true.  On the other hand, it makes the story LESS likely to be true.  Describing a fantasy product in detail “It’ll do this, this and this, people will buy it here, and take it out to use when this happens…” conjures up a coherent, reasonable scenario.  The better an entrepreneur elaborates this kind of story, the more likely potential customers will buy into it. Not because it is true, but because people naturally jettison the hard question “Are each one of these assertions true?”  and substitute an easier question “Does this story as a whole make sense?”  Avoiding the conjunction fallacy requires keeping things as simple as possible.

Conclusion

Steve Blank’s definition of a startup is a “temporary organization designed to search for a valid business model under conditions of extreme uncertainty.”  It’s the “extreme uncertainty” part that amplifies the effect of every one of these fallacies and heuristics.  Uncertainty is why entrepreneurs need to do customer discovery in the first place.  But customers are uncertain too.  They may have very powerful needs that will ultimately catapult the business to greatness, but those needs are probably poorly defined, because if they were well-mapped, then established companies would already be filling them.

Customer discovery is designed to map new territory, and new territory is exactly where people are most likely to misunderstand themselves, substitute easier questions for harder ones, buy into false but coherent stories, and leap to conclusions based on inadequate evidence.  Talking to lots of customers is a start, but only a start, down the path toward a valid business model.


[1] Of course, there are good reasons for mental laziness.  Perhaps the most important one is that using less effort means that people can make decisions more quickly.  In numerous situations, coming to a quick decision, even if it may not be right, is preferable to coming to no decision at all.

Availability: A Heuristic for Judging Frequency and Probability,” Amos Tversky and Daniel Kahneman 1973.

[2] Norbert Schwartz et al., Ease of Retrieval as Information. http://sitemaker.umich.edu/norbert.schwarz/files/91_jpsp_schwarz_et_al_ease.pdf

[4] How mental systems believe. Gilbert, Daniel T. American Psychologist, Vol 46(2), Feb 1991, 107-119. doi: 10.1037/0003-066X.46.2.107

[5] Forming Impressions of Personality,”   Solomon E Asch, Journal of Abnormal and Social Psychology 41 (1946) http://www.all-about-psychology.com/solomon-asch.html

About these ads
5 Comments
  1. Hi, only yesterday I found this blog and I must say: thank you and please keep writing.
    Yes, maybe I’m from a faraway country (Peru) and still don’t have a company but I appreciate people who can go beyond the manual and update the situation with facts instead of sticking to dogma.

    Reality is, here over South America, founders are ALMOST NEVER advised to dig deeper into their customers’ minds or check their actual needs or context (some do but it’s their own initiative). There’ve been some efforts like Lean Startup Machine but their goal is getting founders to interview or survey people and report their answers. I’ve also seen Steve Blank’s Lean Launchpad team presentations on Slideshare (where they must interview a few dozens of people over five weeks) and some of them do a better job at finding customers’ true motives. Still no solid advice on how to do it though.

    Customer discovery needs more depth in order to create more useful startups. And yes, I also agree that it’s not enough – the risks of team, money and tech are still there – but getting the foundations right (value for customers’ real needs) might yield better results.

    • Thanks for your comment, we’re been trying to post a few times per week, but hoping at the same time to maximize the value of the posts. If you like it, please share with your friends.

      I agree that customer discovery “needs more depth in order to create more useful startups.” – It’s a powerful methodology, but also, I think, quite difficult to implement – very different from “listening to the customer” for an established product, but in practice, too easy to conflate the two.

  2. Thanks Juan Carlos, That’s what we’re trying to do – operationalize customer discovery, and move toward reducing other risks. Check out our program at http://flashpoint.gatech.edu/

  3. I haven’t looked at other posts yet, but I was vectored to this one from a conversation about good techniques for Customer Discovery. This essay paints a very thorough and helpful picture of the reason that false positives abound.

    If I may, I’d like to be prescriptive regarding how one can minimize the biases introduced by the above factors and others such as your subject wanting to be polite. Here is how I have my teams overcome these biases:

    A. Ask for operational FACTS before product/feature OPINION. (note that most of the examples used above are asking the prospective customer to predict their future behavior rather than describe their past behavior).

    Triangulate information with additional questions (of course, about fact!). (“Wow, this happens nearly hourly. So, it’s nearly lunchtime, how many times has it happened this morning?”)

    B. Invoke the sales model. I find this to be the real shortcoming of “The Four Steps…” and others who discuss customer discovery. I learned early on that the only true expert is someone who is ready to write a check. Now, even I would poke holes in my last statement, but I use it because it gets most practitioners in the right mindset.

    I don’t EVER tell a prospective customer when I’m trying to set up a meeting “Don’t worry, I’m not trying to sell them something.” Quite the contrary, I make it clear how important candid answers are because if they describe a need and agree that we will fulfill it, I am definitely coming back to ask for the order.

    By the way, “invoking the sales model” does not mean “selling” your product. The focus still needs to remain on the problem-to-be-solved. Besides making sure the prospect realizes you want them t think as a future customer, the key “sales model” element is trial closing. Look for ways for the prospective customer can start “buying” through commitments.

    One example in the above post does describe the situation where a prospective customer said they’d buy but when it was time to buy, they didn’t. I’ve seen this, too. I’ve seen it with a product that already existed, too!

    The key is to go back to Step A. above: ask for FACTS in when you first hear it. Confirm why they said they would buy it. Understand how they solve the problem today and what they paid for it. Understand the cost of NOT solving the problem. If it’s a big ticket item, understand the last time they purchased something in the $ range. What was the procurement process then? Has it changed? If you have facts about their behavior and then hear their opinion and it doesn’t match with the facts, that is when you can drill down and really learn. If you don’t have the facts, you don’t know when your missing some important data.

    I agree with the post that speaking with prospective customers in any manner can be fraught with creating false positives. However, I know that FPs can be greatly minimized by recognizing the problem and really understanding how purchase decision have been made and will be made through a discussion with prospective customers.

    C. Always interview as a team…and get quotes. Two sets of ears will hear things more clearly and avoid the misinterpretation. Also, it’s easier to clarify as the second party.

    These are the same discussions your salesforce is going to have to have in the future. Why not have them now? Know Now. Don’t settle for guess.

    I write about the practice and tactics of customer discovery at http://www.telleen-lawton.com based on my 25 years in the trenches.

    • David, your thoughtful comment fell through the cracks, and I just saw it today. Thanks, and sorry about the belated reply.

      I appreciate your tactics, and think they’re exactly right in the right context. But that context is AFTER the company has discovered a vein of demand. Once that happens, having two people listen, asking questions about past behavior rather than predictions, triangulating, etcetera, are what’s needed to figure out how the product needs to be shaped, features, how the company needs to relate to customers, etcetera. But that’s all in what Peter Thiel would call the “one to x” phase. In the “zero to one” phase, the nature of the improvement goal (or the sum of the improvement goal and the constraints that prohibit the customer from fulfilling it) are not clearly known to the customer. In that situation, the tactics you describe are good for getting facts, but the facts aren’t meaningful because there’s not yet a context.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

Join 168 other followers

%d bloggers like this: