About 25 years ago, I had the privilege of shadowing a drug safety expert for 12 months as part of an industrial experience placement incorporated into my biochemistry degree course. The guy I was following coached me in research study design, data analysis, and effective reporting writing, all in an environment that was highly regulated, expensive to operate in, and therefore extremely intolerant of anything other than precise and objective thinking.
A lot of the knowledge and skills I acquired back then and tuned over the years are directly relevant to what I do as an analyst today. Much of our work at Freeform Dynamics, for example, is based on the same experimental process of using observation to develop then iteratively test, refine and retest hypotheses until a reasonable degree of confidence is gained into what’s going on within the area of investigation.
To do this well means keeping an open mind and avoiding the temptation of getting too attached or committed to any particular hypothesis. It is all too easy to fall into the trap of focusing on the data that fits your pet theory and not even consider alternative inputs or explanations. The mentor I mentioned earlier used to tell a story that illustrates the pitfalls in this area very well, and although some might find it distasteful, it is pretty effective at getting the message across, so here it is anyway (apologies to anyone offended, but remember it was being told 25 years ago).
A scientist called together a group of his peers to witness an experiment in which he would prove a hitherto unknown relationship between the motor and sensory capability of the average house spider. It was an important event. Once the community got sight of this experiment, the grants would come rolling in to fund further research into a new convergent theory of arachnid physiology and neurology.
The scientist began by explaining that in preparation for the experiment, he had trained a group of spiders to come to him when called. To illustrate what he meant, he tipped one of these spiders, Horace, out of its glass jar onto the lectern. Then sure enough, when the scientist called him, the little creature scuttled across to the sound of his name.
The scientist then lifted Horace up and pulled off one of his legs. It didn’t make much difference, though, Horace still scuttled across the lectern to the sound of his name, with no noticeable decrease in speed.
However, as the scientist repeated the procedure, pulling off another leg each time around, Horace’s progress across the lectern got progressively slower and clumsier. Nevertheless, he still came when he was called, and even with one leg left, Horace still managed to drag himself across the lectern to the sound of the scientist’s voice.
Then, to finish the experiment, the scientist removed the spider’s last remaining leg, set him down and called him for one final time. Nothing. Horace just sat there not moving at all. “And there we have it”, said the scientist triumphantly, “conclusive proof that if you pull all of the legs off a spider, it goes deaf”.
I know, it’s a horrible and cruel story, but you have to admit that it sums up a lot of the way in which we often see observations from research misinterpreted, particularly when the interpreters are looking to drive an agenda or support a position that they have dug themselves into. What we are talking about here is the practice of selecting and bending research findings around the argument you are trying to make, even to the extent of ignoring more obvious interpretations that are staring you in the face.
It’s a problem that research centric analyst firms like ourselves struggle with from a differentiation perspective quite frequently. People do not always appreciate the difference between the objective research studies we conduct to genuinely test hypotheses and generate new insight, and the simplistic and/or divisive pseudo-research we sometimes see commissioned by PR departments to support a particular marketing message or value proposition. We ‘no-bid’ on projects like the latter all the time, as do other analyst firms we know, and such work typically ends up being conducted by generic market research firms (e.g. on a price per question basis) who are happy to leave interpretation up to the client or, worse still, the client’s PR agency.
Differentiating between good and the not so good research can be a real challenge though, and I sympathise with those out there who struggle with this ongoing problem. We as Freeform seem to do pretty well with getting our stuff accepted, but it is a shame when you see good quality intelligence and insights being dismissed from any source by those tarring all market research with the same brush.
Related to this is the question of independence, with the argument sometimes heard that it is not possible for sponsored research to be objective. This is again perfectly understandable. After all, why would an IT vendor, for example, fund a Community Research study that seriously tested a hypothesis that had shaped its go-to-market thinking? And by ‘tested’ here, I mean researching questions designed to challenge as well as support the hypothesis.
In order to understand this, you have to overcome the notion that vendors are continually looking to trick or mislead people into buying their products and services. The truth is that this is the harder and riskier way of doing things. It is far easier to sell something that is naturally and comfortably aligned with customer expectations and interests, particularly if you are a major vendor that thrives or flounders on the strength of your longer term relationships with major accounts. Against this background, the engagements we undertake are in the spirit of a mature win-win mindset, so while we do not pretend that vendors have no interest in tangible benefits from sponsorship of research, unless there is equal or greater benefit likely to emerge for the buyer/user community, we don’t get involved.
Fortunately, as we found when researching The Technology Garden, CIOs, architects, operations professionals and so on are as interested in vendors understanding and meeting their needs as the vendors are themselves. In practice, there is therefore a lot less conflict between the interests of buyers and sellers than you might imagine if you focus on areas such as planning, strategy, business cases, best practice, etc, which is the level at which we generally work.
More often than not, the real value that comes out of objective research is alignment of interests by segment or scenario. What I mean by this is probably best illustrated with an example.
One of the research studies we conducted last year was sponsored by a company called Momote, a mobile middleware/platform provider. An important foundation upon which the Momote proposition is based is the need for platforms to be flexible to support initial tailoring of solutions then subsequent modification as business needs evolve. It sounded sensible, but so too did a couple of alternative approaches that were more prescriptive, or ‘plug and go’, in nature, so it was important to challenge the hypothesis in different contexts. To cut a long story short, the research told us the hypothesis held true for organisations with either very dynamic or complex field service operations, but not for those in which field work was highly repetitive and driven by relatively static schedules.
On face value, you might question the value of this research for the sponsor, in that the evidence suggested that its approach was overkill for a pretty large chunk of the market. However, it meant that Momote could now identify, qualify and articulate its proposition much more precisely and effectively to those most likely to buy. From the buyer/user perspective, we were then able to advise on the type of solution most likely to meet business requirements based on the nature of operation, all driven by input from peers within the service management community.
So, my request AR to professionals out there is to make sure you yourself understand the different styles of research that are conducted in the market place and the associated benefits and limitations of the various approaches. I am happy to discuss this further with anyone who is interested in a more in-depth exchange.
In the meantime, if someone can come up with a more tasteful alternative to the Horace story, or would like to contribute similar stories that illustrate practical issues around research that are easy for non-specialists to understand, all input will be gratefully received.
Thursday, February 14, 2008
Saturday, February 02, 2008
The analyst watcher blind spot
There has been something bothering me for a while now, and I haven’t been able to put my finger on it until recently.
The problem is to do with ‘analyst watcher’ firms, particularly independent agencies and consultants that provide advice and other services to vendors on how best to engage with and exploit the IT industry analyst community. This is essentially a sub-segment of the analyst community itself, in that such organisations pretty much do the same job as industry analysts. We all look at solutions, who is providing them, and try to give a steer on who to engage for what and why, either at a category or individual provider level.
As soon as I realised that analyst watchers were analysts, it helped me to understand how to interpret their output and activities. Anyone who knows me will be aware that analyst watching is a bit of a hobby of my own and, indeed, Helen and I conducted a pretty thorough review of the analyst landscape before we designed Freeform Dynamics. That’s how we ended up with a firm and business model that is reasonably well differentiated to those who know us.
This last statement, particularly the reference to “those who know us”, brings me back to the thing I was struggling with. Some analyst watchers out there seem to have difficulty understanding what we do, why we do it, and why it matters to their clients. In fact, I have read various pieces of commentary recently questioning the value and relevance of smaller analyst firms – let’s call them ‘boutiques’ – in general. Related to this is talk about whether the use of media channels and Web 2.0 style delivery is as effective from an influencing perspective as advice driven through commercial subscriptions and consulting services by– let’s call them – the ‘big incumbents’.
The problem is, though, you end up trying to compare apples to oranges, and this is the trap some of the analyst watchers out there have fallen into as they attempt to assess boutiques using the same criteria they apply to the big incumbents. One of the highest weighted factors in this is direct participation of the analyst firm at the tail end of the strategic IT investment cycle, i.e. when vendors are competing head to head to win major IT procurement decisions in large accounts. With this kind of measure central to the assessment, it is no wonder that some analyst watchers struggle to see the relevance of boutiques. Even at a collective level, the direct involvement of boutiques in this context cannot seriously be compared to that of the big incumbents.
Most of the analyst watchers have not surprisingly reached this same conclusion, and as both their thinking and often their own propositions revolve around monitoring and measuring influence defined in this way, those pesky boutiques are simply viewed as an irrelevance, or perhaps ‘inconvenience’ would be a better term.
Why inconvenience? Well because there is something about many of us little guys that makes us difficult to ignore or write off completely. Even though we don’t fit into the traditional influencing models and scoring mechanisms that are often central to the services provided by analyst watchers to their vendor clients, their clients still, annoyingly, seem to take a lot of us seriously. And it’s doubly annoying because the reasons for this are often too slippery to tie down, and if you can’t tie them down, you can’t measure and score them, so how can you work them into your service offerings and make money from advising on them? Business is business, after all.
So let’s take a look at some of these more slippery factors.
As someone who carried a sales quota for many years of my career, one of the things I learned was that the earlier in the decision making cycle you can engage, the more chance you stand of closing a deal. One of the strongest positions to be in, in fact, is when you are the person (or company) that has educated the prospect at the point of them considering whether they should be heading in a particular direction or not - i.e. long before they even consider which technologies, products or services will ultimately be required. If you can get people thinking in the ‘right way’ at this point, i.e. orientate their mindset or frame a decision in line with your own view of the world (or that of the company you are representing), then you can out-position the competition even before they get involved. Conversely, walking into an opportunity late in the day when mindsets and thinking have already been set unfavourably to your proposition makes winning the business extremely difficult, even if you wave around an analyst report that positions what you are selling in the top right hand corner.
The reality, of course, is that this kind of ‘upstream’ shaping of mindsets, thinking and objectives that has such a huge bearing on ‘downstream’ decisions made later, is influenced by a whole range of inputs. This is something we discuss in a report we have just published in collaboration with our friends at The Register, in which we look at the market 'buzz' surrounding 25 global tech brands (see here). As part of this, it was fascinating to see the range and diversity of information sources that have a bearing on the decision making process when considered in its entirety (i.e. upstream as well as downstream). Now I don’t want to make any claims here as this upstream part of the equation is extremely complex, especially when you consider the interplay between influences (i.e. influencer networks), but it does put more traditional downstream analyst activity into its proper overall perspective.
And here’s where we close the loop with analyst boutiques. The point is that many boutiques play predominantly in the upstream space, with hard hitting blogs, far reaching media relationships, and deep community ties into various buyer communities and constituencies.
As I said earlier, though, figuring out how to measure all this is pretty near impossible. When several tens of thousands of people (we track the stats) read our analysis of BEAs acquisition by Oracle, for example, what difference does it make to procurement behaviour within the BEA and Oracle customer base? When the aforementioned ‘buzz report’ is circulated to a subscription list of 18,000 IT professionals as a free download, with no copy protection, how much easier or harder does the job of some IT sales people become? I certainly couldn’t quantify the answers to these questions, but to argue that such upstream activity has no influence at all is as impossible to defend as claiming that boutiques can compete with the bigger firms on direct downstream participation in major deals.
Nevertheless, appreciating the impact still depends on subjective value judgements, which is the main contributing factor to the analyst watcher blind-spot.
Fortunately for us, analyst relations professionals working in a vendor environment do not seem to suffer from this same impediment. We actually have no problem at all justifying our existence with people in this position, and I know from my dealings across the analyst community that other boutiques we rub shoulders with have the same positive experience. I think the difference is that in-house AR guys do not have the luxury of ignoring or dismissing activity they intuitively know to be important just because it is difficult to measure. Regardless of the views they might hear from some analyst watching advisors, they still therefore give us the time and access to executives and privileged information we need to do our jobs.
The other reason quite a few boutiques tend to be entertained and promoted by in-house AR people is the value their executives and other spokespeople get from a different kind of dialogue. The average level of ‘been there, done that’ experience and general level of talent within smaller firms tends to be higher than in larger ones, and analysts are far less constrained in the way they interact. This is not a criticism of larger firms (I don’t want to fall into the apples versus oranges trap myself), it’s simply a natural consequence of the scale of operations and nature of the traditional analyst business model.
If you pull all this together, the message is that it is important to appreciate that boutiques are simply different, and trying to assess their merits and value using the same criteria and approach you would for larger firms means you potentially fail to appreciate what they have to offer and why they can be important.
I would also like to stress that my comments on analyst watchers are provided in the spirit of constructiveness. Most are making the effort, and that is to be applauded, but I think it is important for some with a noticeably larger blind spot than others to realise that their thinking is sometimes actually behind that of the clients they are trying to advise. I personally believe there is an opportunity here for advisory firms that broaden their view and take upstream influence and influencer networks a bit more seriously.
The problem is to do with ‘analyst watcher’ firms, particularly independent agencies and consultants that provide advice and other services to vendors on how best to engage with and exploit the IT industry analyst community. This is essentially a sub-segment of the analyst community itself, in that such organisations pretty much do the same job as industry analysts. We all look at solutions, who is providing them, and try to give a steer on who to engage for what and why, either at a category or individual provider level.
As soon as I realised that analyst watchers were analysts, it helped me to understand how to interpret their output and activities. Anyone who knows me will be aware that analyst watching is a bit of a hobby of my own and, indeed, Helen and I conducted a pretty thorough review of the analyst landscape before we designed Freeform Dynamics. That’s how we ended up with a firm and business model that is reasonably well differentiated to those who know us.
This last statement, particularly the reference to “those who know us”, brings me back to the thing I was struggling with. Some analyst watchers out there seem to have difficulty understanding what we do, why we do it, and why it matters to their clients. In fact, I have read various pieces of commentary recently questioning the value and relevance of smaller analyst firms – let’s call them ‘boutiques’ – in general. Related to this is talk about whether the use of media channels and Web 2.0 style delivery is as effective from an influencing perspective as advice driven through commercial subscriptions and consulting services by– let’s call them – the ‘big incumbents’.
The problem is, though, you end up trying to compare apples to oranges, and this is the trap some of the analyst watchers out there have fallen into as they attempt to assess boutiques using the same criteria they apply to the big incumbents. One of the highest weighted factors in this is direct participation of the analyst firm at the tail end of the strategic IT investment cycle, i.e. when vendors are competing head to head to win major IT procurement decisions in large accounts. With this kind of measure central to the assessment, it is no wonder that some analyst watchers struggle to see the relevance of boutiques. Even at a collective level, the direct involvement of boutiques in this context cannot seriously be compared to that of the big incumbents.
Most of the analyst watchers have not surprisingly reached this same conclusion, and as both their thinking and often their own propositions revolve around monitoring and measuring influence defined in this way, those pesky boutiques are simply viewed as an irrelevance, or perhaps ‘inconvenience’ would be a better term.
Why inconvenience? Well because there is something about many of us little guys that makes us difficult to ignore or write off completely. Even though we don’t fit into the traditional influencing models and scoring mechanisms that are often central to the services provided by analyst watchers to their vendor clients, their clients still, annoyingly, seem to take a lot of us seriously. And it’s doubly annoying because the reasons for this are often too slippery to tie down, and if you can’t tie them down, you can’t measure and score them, so how can you work them into your service offerings and make money from advising on them? Business is business, after all.
So let’s take a look at some of these more slippery factors.
As someone who carried a sales quota for many years of my career, one of the things I learned was that the earlier in the decision making cycle you can engage, the more chance you stand of closing a deal. One of the strongest positions to be in, in fact, is when you are the person (or company) that has educated the prospect at the point of them considering whether they should be heading in a particular direction or not - i.e. long before they even consider which technologies, products or services will ultimately be required. If you can get people thinking in the ‘right way’ at this point, i.e. orientate their mindset or frame a decision in line with your own view of the world (or that of the company you are representing), then you can out-position the competition even before they get involved. Conversely, walking into an opportunity late in the day when mindsets and thinking have already been set unfavourably to your proposition makes winning the business extremely difficult, even if you wave around an analyst report that positions what you are selling in the top right hand corner.
The reality, of course, is that this kind of ‘upstream’ shaping of mindsets, thinking and objectives that has such a huge bearing on ‘downstream’ decisions made later, is influenced by a whole range of inputs. This is something we discuss in a report we have just published in collaboration with our friends at The Register, in which we look at the market 'buzz' surrounding 25 global tech brands (see here). As part of this, it was fascinating to see the range and diversity of information sources that have a bearing on the decision making process when considered in its entirety (i.e. upstream as well as downstream). Now I don’t want to make any claims here as this upstream part of the equation is extremely complex, especially when you consider the interplay between influences (i.e. influencer networks), but it does put more traditional downstream analyst activity into its proper overall perspective.
And here’s where we close the loop with analyst boutiques. The point is that many boutiques play predominantly in the upstream space, with hard hitting blogs, far reaching media relationships, and deep community ties into various buyer communities and constituencies.
As I said earlier, though, figuring out how to measure all this is pretty near impossible. When several tens of thousands of people (we track the stats) read our analysis of BEAs acquisition by Oracle, for example, what difference does it make to procurement behaviour within the BEA and Oracle customer base? When the aforementioned ‘buzz report’ is circulated to a subscription list of 18,000 IT professionals as a free download, with no copy protection, how much easier or harder does the job of some IT sales people become? I certainly couldn’t quantify the answers to these questions, but to argue that such upstream activity has no influence at all is as impossible to defend as claiming that boutiques can compete with the bigger firms on direct downstream participation in major deals.
Nevertheless, appreciating the impact still depends on subjective value judgements, which is the main contributing factor to the analyst watcher blind-spot.
Fortunately for us, analyst relations professionals working in a vendor environment do not seem to suffer from this same impediment. We actually have no problem at all justifying our existence with people in this position, and I know from my dealings across the analyst community that other boutiques we rub shoulders with have the same positive experience. I think the difference is that in-house AR guys do not have the luxury of ignoring or dismissing activity they intuitively know to be important just because it is difficult to measure. Regardless of the views they might hear from some analyst watching advisors, they still therefore give us the time and access to executives and privileged information we need to do our jobs.
The other reason quite a few boutiques tend to be entertained and promoted by in-house AR people is the value their executives and other spokespeople get from a different kind of dialogue. The average level of ‘been there, done that’ experience and general level of talent within smaller firms tends to be higher than in larger ones, and analysts are far less constrained in the way they interact. This is not a criticism of larger firms (I don’t want to fall into the apples versus oranges trap myself), it’s simply a natural consequence of the scale of operations and nature of the traditional analyst business model.
If you pull all this together, the message is that it is important to appreciate that boutiques are simply different, and trying to assess their merits and value using the same criteria and approach you would for larger firms means you potentially fail to appreciate what they have to offer and why they can be important.
I would also like to stress that my comments on analyst watchers are provided in the spirit of constructiveness. Most are making the effort, and that is to be applauded, but I think it is important for some with a noticeably larger blind spot than others to realise that their thinking is sometimes actually behind that of the clients they are trying to advise. I personally believe there is an opportunity here for advisory firms that broaden their view and take upstream influence and influencer networks a bit more seriously.
Subscribe to:
Posts (Atom)
Posts
