What’s the point of running an AR program if you’re not measuring the outcomes?

Marketing is all about results – there’s no point doing something if it doesn’t deliver results, and there’s not much point doing something if you can’t measure those results. So it’s often astounded me that otherwise savvy marketing folks are quite happy to invest in AR, but not in measuring the outcomes.

As the saying goes, the definition of stupidity is doing the same thing again & again, but expecting different results. And so it goes that when you’re not prepared to examine your outcomes, then you’re bound to repeat the mistakes of the past.

You might think this is a bit self-serving, given that I’m about to bang on about a product that addresses this issue, but the fact is that monitoring, measuring & modifying AR programs has significant benefits for ICT vendors and analysts alike.

What got me thinking down this track was doing all the development work for our annual study of AR effectiveness in APJ, Understanding the Influencers 2012, which goes into the field early next month. This will be the 10^th year of running Understanding the Influencers, and every year it has been tweaked to better reflect the market, better meet the needs of our clients & make it easier for analysts to provide feedback.

This year, the changes are slight, but the level of detail & insight we get now compared to when we launched in 2003 is incredible. Last year, we introduced some new questions about analyst use of social media, discussed in this blog post and this one, so this year we’ll be able to provide some comparative analysis of the changes in analyst behaviour.

The point I’m making is that just as this study has had to continue to evolve, so should AR programs. Just because something was working last year doesn’t mean that it is this year. Or that just because you think something is working, that necessarily means it is…

Too many AR programs focus on counting activities eg number of analysts briefed, number of white papers purchased etc rather than measuring outcomes eg positive report mentions, positive analyst recommendations in deals etc. That’s not to say it’s not important to know what activities you’ve undertaken – it is – but it’s not a measure of success.

Nor am I trying to suggest that measuring the success of AR programs is simple – it isn’t. There is no silver bullet, no single metric which can define success in AR, and that’s a frustration for many. Rather, any AR practitioner worth their salt will use a number of measures – some of them subjective, some of them objective – to build an overall dashboard.

Whichever way you look at it, analyst perception studies are an important element of that. They don’t provide all the answers, but they provide many of them, and they provide the opportunity to get insight which is very specific to an individual vendor, and to identify issues which are actionable.

So what’s the benefit to a vendor from an analyst perception study?

Objective, third-party validation

Most vendors value analyst reports because they provide an independent assessment of markets or technologies, and analyst perception studies such as Understanding the Influencers are just the same. Individual responses are not identified, so analysts can be completely open & honest in their assessment – and believe me, they mostly are – so feedback doesn’t get moderated by politeness or commercial imperatives as it might in a direct conversation.

Depth, detail & richness of data

Syndicated studies such as Understanding the Influencers are structured to capture feedback from as many analysts as possible in a short period of time so that sensible comparisons with competitors can be made and enable detailed analysis. In 2011, we received responses from 140 analysts at 20 different firms in 12 countries, covering more than 70 different vendors & service providers, which provides a wealth of perspectives. The ability to slice & dice this data is significant.

Tailored, actionable advice

This ability to cut the data many ways provides us with the ability to drill down into issues specific to each client, and recommend appropriate actions to address problems and leverage opportunities. Apart from the quantitative research, we also conduct 20+ in-depth interviews with senior analysts, in which they provide very specific feedback on what’s working, and what’s not.

Insight into analyst thinking

Apart from measuring a number of key attributes to evaluate vendor AR programs, we also ask a number of questions about the analyst’s involvement in & likelihood of recommending vendors in a deal. While this is an indicator rather than an absolute, it quite clearly shows whether a vendor is getting traction at the “business” end, or not.

And what’s the value to an analyst in responding?

The analyst’s voice is heard

Analysts tend to have strong opinions, and most aren’t backward in coming forward. They want vendors to understand that analysts’ information needs are different to other audiences, but they don’t have the time or energy to tell them one by one why their AR program sucks, or how they can improve it. But by spending 15 minutes to complete a survey once a year, they can get their message across load and clear.

The analyst/AR ecosystem benefits

Many analysts value AR and know that a good AR program can make the analyst’s job easier. They also know that AR pros are powerful advocates for analyst research & insight inside the vendor organisations. They know that if they help the AR pros do their jobs, AR will help the analysts build their profile, and so the process continues.

There are many analysts who have participated in every study I’ve conducted over the past nine years, and most of them are the heavy hitters of the APJ analyst business. These are the analysts who put up their hands year after year to participate in 15-minute qualitative interviews, on top of completing the online survey. They know that it’s a good investment of their time, and some of them just like to give a little bit back.

As usual, that’s the perspective through my lens. I see plenty of upside in analyst perception studies for vendors and analysts alike, but what do you think? All benefit? Or is there a downside? Join the conversation below.

Or if you just want to know more about Understanding the Influencers, click here.

Cheers,

Dave