Critics of qualitative research point to small sample sizes that allow anomalies to be presented as typical, while critics of qualitative studies cite over-generalisation and inability to capture evocative individual stories that drive true insight.
Jeff Sauro says why not quantify your qual data? – a quick extrapolation from small samples can give you a broader steer for a given confidence interval.
A more statistically robust approach perhaps is to run some qual research and follow up with surveys to substantiate (or disprove) the hypotheses the qual research threw up. Or to start with quant and then follow up with qual to further explore findings (which you start with is a whole other post and depends on your world view [pdf]). Either way. the importance of such a ‘mixed-method’ research approach is to get some numbers to check your thinking and make sure you are not biased (humans are notoriously bad at thinking statistically).
One approach I find particularly appealing is to ask both an open and closed question (such as Net Promoter Score) during research. This helps to anchor rich human stories to broader ratings-style data. It also allows you to incorporate both the long and short term into your research, for example when exploring the experience vs memory paradigm .
Ford pioneered an efficient process based on the division of labour and time and motion studies. It was great for mass producing the same stuff. Marketing’s job was to convince the masses they all wanted this same thing.
But the outcomes weren’t always great and in these instances no amount of marketing was going to convince people otherwise…
Costly mistakes in public services prompted a bit of a re-think in how to approach project delivery.
From the 1990s, there have been increasing moves in the public sector in the US and other western economies to address outcomes as a key part of performance measurement. This has been in part because services focusing on outputs often required costly review when user needs and benefits had not been sufficiently considered.
Measuring outcomes in this way highlighted other knock-on benefits that would help to sell in projects and secure funding:
Advantages could also be seen in the added value that could be modelled and estimated in terms of population outcomes. For example, learning or health incomes impacted on improved productivity, increased tax revenue, fewer days lost through sickness and disability, and so on.
Today a quick search reveals such public organisations as the World Bank, NHS, NICE and IDEA all talking the outcomes talk.
Meanwhile in business production was loosened up with Just in Time management philosophy. More nimble, modular production, making only what is necessary when it was needed. At the consumer coal face, marketing departments were testing emotional response to marketing messages, although not necessarily to the products and services themselves.
The emotional outcome
We are now in a third phase which goes beyond the ‘build first, market later’ approach and builds on lean processes to achieve it. Now a good outcome must also consider the psychological dimension – how people experience your service. While seemingly a no-brainer, the challenge is encoding this into the production process.
Michael Fassbender as the emotional robot, David, in Prometheus.
Recently User Experience (UX) principles and design thinking has been sneaking into the business vernacular. UX comprises various sub-disciplines: usability (making an interface easy and intuitive to navigate) but also considerations of context, intent, environmental factors that add up to an overall experience.
Of course experience is a subjective thing and your product may not be received in the way intended because everyone’s mental experiencing equipment is different. An experience is a function of a product or service in terms of this mental equipment.
This underlines the importance of setting up multiple feedback loops to help us check how our product and services are being experienced, something that we cannot necessarily know in advance.
With beginnings in software land, agile processes are now slowly finding their way in other areas of business. In a nutshell, you launch a minimum viable product or service, test user (emotional) response and repeat.
This trend can only be set to continue. Digital tech is relatively plastic building material and is making possible a new generation of digitally-powered products and services that are driven by achieving the right emotional outcome.
Sounds intuitive enough, but here is some research to back it up.
The difference between good and bad experience equated to a 19.5% point difference in customers saying they would recommend a business and shop there again (customers having good experiences were 9.9% more likely than average to recommend and repeat purchase compared to those experiencing the worst customer experience who scored -9.6%).
Behavioural economics is so hot right now. It has also been a big part of the inspiration for nativeye. I studied economics at university which assumed perfectly rational humans as a foundation for many of its models. At the same time I also took psychology modules which dealt with the highly irrational, aka real-life human brain. The two disciplines dealt with similar subjects but seemed very far apart. Behavioural economics has mainstreamed to fill this gap and is required reading for anyone involved in researching people.
Introducing tonight’s double act – your brain
BE books such as Nudge and Predictably Irrational have been top sellers. I am working my way through the latest and probably greatest, Thinking fast and slow, byBE’s godfather Daniel Kahneman. It deals mainly with the flaws in our intuition and our apparatus for perceiving and making sense of the world (our Systems 1 and 2). System 1 is concerned with intuitive thought, it is immediate, reactive and always on. System 2 is the policeman – it calculates whether what System 1 reports is true and makes corrections where required. It is also pretty lazy.
System 2 uses a bunch of heuristics (rules of thumb) to speed up its job and this is where all sorts of biases creep in. Here are some examples:
Disproportionate effect of emotion on risk taking decisions (e.g. loss aversion)
Affect bias (“I like this company, i think I’ll invest in them” – rather than crunching the numbers)
Availability bias (“That air crash in the news was terrible. Air travel is really unsafe.”)
Anchoring – being unduly influenced by a previous reference point
We are not aware that we do these things, and when questioned, System 2 tends to make up stories to explain them away. We like (and like to tell ourselves) stories that make sense, that are coherent, and that provides us with ‘cognitive ease’, whether our not they are actually ‘true’.
“The conscious rational brain isn’t the Oval Office. It’s actually the press office issuing explanations for actions we’ve already taken”
Rory Sutherland, Vice Chairman, Ogilvy & Mather and President, IPA
What does it mean for market research and your business?
This tendency to post-rationalise our decisions and actions clearly raises serious questions about how we go about researching these same actions and decisions. As usual Rory Sutherland is miles ahead. His concern is that traditional, ‘after the fact’ survey-based research is missing a trick, and that businesses can benefit from research that is closer to the point of decision.
“As you get closer to the point of decision, there are factors at work which never really appear in conventional market research. They are contextual factors, social factors”
This is taken from an interview with Research Magazine and the whole interview is worth a read, exhibiting Sutherland’s entertaining ‘after-dinner speech’ style.
Some thoughts on how to measure the moment
So how do we go about short-circuiting this storytelling and get through to people’s true motivations?
Capture emotions ‘in the moment’ which are associated with a particular decision or action (rather than the explanation after the fact)
Pay more attention to the context in which decisions are made (e.g. environment and location, time constraints, social pressure, mood)
Studying behaviour (based on the counter-intuition that behaviour predicts preference rather than the other way around)
Or hire a neuroscientist and by-pass the problem and probe directly into the brain