About 25 years ago, I had the privilege of shadowing a drug safety expert for 12 months as part of an industrial experience placement incorporated into my biochemistry degree course. The guy I was following coached me in research study design, data analysis, and effective reporting writing, all in an environment that was highly regulated, expensive to operate in, and therefore extremely intolerant of anything other than precise and objective thinking.
A lot of the knowledge and skills I acquired back then and tuned over the years are directly relevant to what I do as an analyst today. Much of our work at Freeform Dynamics, for example, is based on the same experimental process of using observation to develop then iteratively test, refine and retest hypotheses until a reasonable degree of confidence is gained into what’s going on within the area of investigation.
To do this well means keeping an open mind and avoiding the temptation of getting too attached or committed to any particular hypothesis. It is all too easy to fall into the trap of focusing on the data that fits your pet theory and not even consider alternative inputs or explanations. The mentor I mentioned earlier used to tell a story that illustrates the pitfalls in this area very well, and although some might find it distasteful, it is pretty effective at getting the message across, so here it is anyway (apologies to anyone offended, but remember it was being told 25 years ago).
A scientist called together a group of his peers to witness an experiment in which he would prove a hitherto unknown relationship between the motor and sensory capability of the average house spider. It was an important event. Once the community got sight of this experiment, the grants would come rolling in to fund further research into a new convergent theory of arachnid physiology and neurology.
The scientist began by explaining that in preparation for the experiment, he had trained a group of spiders to come to him when called. To illustrate what he meant, he tipped one of these spiders, Horace, out of its glass jar onto the lectern. Then sure enough, when the scientist called him, the little creature scuttled across to the sound of his name.
The scientist then lifted Horace up and pulled off one of his legs. It didn’t make much difference, though, Horace still scuttled across the lectern to the sound of his name, with no noticeable decrease in speed.
However, as the scientist repeated the procedure, pulling off another leg each time around, Horace’s progress across the lectern got progressively slower and clumsier. Nevertheless, he still came when he was called, and even with one leg left, Horace still managed to drag himself across the lectern to the sound of the scientist’s voice.
Then, to finish the experiment, the scientist removed the spider’s last remaining leg, set him down and called him for one final time. Nothing. Horace just sat there not moving at all. “And there we have it”, said the scientist triumphantly, “conclusive proof that if you pull all of the legs off a spider, it goes deaf”.
I know, it’s a horrible and cruel story, but you have to admit that it sums up a lot of the way in which we often see observations from research misinterpreted, particularly when the interpreters are looking to drive an agenda or support a position that they have dug themselves into. What we are talking about here is the practice of selecting and bending research findings around the argument you are trying to make, even to the extent of ignoring more obvious interpretations that are staring you in the face.
It’s a problem that research centric analyst firms like ourselves struggle with from a differentiation perspective quite frequently. People do not always appreciate the difference between the objective research studies we conduct to genuinely test hypotheses and generate new insight, and the simplistic and/or divisive pseudo-research we sometimes see commissioned by PR departments to support a particular marketing message or value proposition. We ‘no-bid’ on projects like the latter all the time, as do other analyst firms we know, and such work typically ends up being conducted by generic market research firms (e.g. on a price per question basis) who are happy to leave interpretation up to the client or, worse still, the client’s PR agency.
Differentiating between good and the not so good research can be a real challenge though, and I sympathise with those out there who struggle with this ongoing problem. We as Freeform seem to do pretty well with getting our stuff accepted, but it is a shame when you see good quality intelligence and insights being dismissed from any source by those tarring all market research with the same brush.
Related to this is the question of independence, with the argument sometimes heard that it is not possible for sponsored research to be objective. This is again perfectly understandable. After all, why would an IT vendor, for example, fund a Community Research study that seriously tested a hypothesis that had shaped its go-to-market thinking? And by ‘tested’ here, I mean researching questions designed to challenge as well as support the hypothesis.
In order to understand this, you have to overcome the notion that vendors are continually looking to trick or mislead people into buying their products and services. The truth is that this is the harder and riskier way of doing things. It is far easier to sell something that is naturally and comfortably aligned with customer expectations and interests, particularly if you are a major vendor that thrives or flounders on the strength of your longer term relationships with major accounts. Against this background, the engagements we undertake are in the spirit of a mature win-win mindset, so while we do not pretend that vendors have no interest in tangible benefits from sponsorship of research, unless there is equal or greater benefit likely to emerge for the buyer/user community, we don’t get involved.
Fortunately, as we found when researching The Technology Garden, CIOs, architects, operations professionals and so on are as interested in vendors understanding and meeting their needs as the vendors are themselves. In practice, there is therefore a lot less conflict between the interests of buyers and sellers than you might imagine if you focus on areas such as planning, strategy, business cases, best practice, etc, which is the level at which we generally work.
More often than not, the real value that comes out of objective research is alignment of interests by segment or scenario. What I mean by this is probably best illustrated with an example.
One of the research studies we conducted last year was sponsored by a company called Momote, a mobile middleware/platform provider. An important foundation upon which the Momote proposition is based is the need for platforms to be flexible to support initial tailoring of solutions then subsequent modification as business needs evolve. It sounded sensible, but so too did a couple of alternative approaches that were more prescriptive, or ‘plug and go’, in nature, so it was important to challenge the hypothesis in different contexts. To cut a long story short, the research told us the hypothesis held true for organisations with either very dynamic or complex field service operations, but not for those in which field work was highly repetitive and driven by relatively static schedules.
On face value, you might question the value of this research for the sponsor, in that the evidence suggested that its approach was overkill for a pretty large chunk of the market. However, it meant that Momote could now identify, qualify and articulate its proposition much more precisely and effectively to those most likely to buy. From the buyer/user perspective, we were then able to advise on the type of solution most likely to meet business requirements based on the nature of operation, all driven by input from peers within the service management community.
So, my request AR to professionals out there is to make sure you yourself understand the different styles of research that are conducted in the market place and the associated benefits and limitations of the various approaches. I am happy to discuss this further with anyone who is interested in a more in-depth exchange.
In the meantime, if someone can come up with a more tasteful alternative to the Horace story, or would like to contribute similar stories that illustrate practical issues around research that are easy for non-specialists to understand, all input will be gratefully received.
0 comments:
Post a Comment