Skip to content

Posts tagged ‘AR measurement’

How measurement of analyst relations programs has evolved to provide clarity in an uncertain world

The business of influencing the influencers has never been straightforward. Just getting agreement in the analyst relations community about what we mean by “influence” has always been the subject of lengthy & sometimes heated debate, but the evolution of new analyst business models, research methodologies and information delivery processes have added further layers of complexity.

AR professionals continue to be challenged by tightened budgets at the same time that the scope of their target audience is growing. Even the most disciplined AR pros who focus tightly on their key influencers sometimes find themselves distracted by their stakeholders pushing new players into the mix, and the deafening noise of others promoting their opinions via social media.

A well-structured targeting & tiering process is a key element in ensuring that you’re focusing on the analysts that matter to your company, however you define influence and however broad or narrow the scope of your AR program. I’ve written previously about why targeting is important and how to go about tiering analysts.

But then you still need to determine whether you’re getting cut-through with these analysts. And that’s a bit harder.

There are a bunch of things that come into play when you’re trying to measure an AR program. Some of them are subjective, some of them are a little more concrete. As I wrote a couple of years ago, there is no single metric which will define success in AR – and counting the number of briefings conducted is certainly not that! – but smart AR pros will use a range of measures to build an overall dashboard.

A big piece of the puzzle is some sort of objective measure of how well your AR program is working, and what it is actually achieving. How better to do this than asking the analysts themselves?

So here comes the self-serving bit. Intelligen has been running AR effectiveness studies since 2003, annually surveying analysts right across the Asia/Pacific & Japan region. Not surprisingly, we think that this is a really important tool for keeping track of the impact of vendor AR vendor programs, and so do many of our clients, who subscribe year after year.

Our study is called Understanding the Influencers, and it has constantly evolved to reflect the changing nature of the analyst business in APJ, plus the changing requirements of our vendor AR clients.

Over the past few months, we have undertaken a complete review & revision of Understanding the Influencers. We have sat down with analysts to understand what’s important to them, and how to make this study more relevant for them. We have talked with AR and marketing pros across the region to understand the sorts of insights they need, and how to deliver them.

So here are the big changes (and what hasn’t changed):

A new survey platform, optimised for today’s web

Although Understanding the Influencers started as a phone survey, we took all the quantitative aspects online fairly early – in 2005 – and since 2006, the study has been hosted by Novagenus. Over the past few months, the outstanding team at Novagenus have built a complete new survey platform for us in HTML5, optimised for modern browsers and tablets. The new layout is clean and simple to use, minimising the amount of time it takes busy analysts to respond. For the first time, analysts will be able to write in the names of vendors not included in our standard segment lists.

A new survey questionnaire which reflects today’s AR world

We’ve tweaked the survey questions many times over the years to expand coverage and address new influences such as social media, but this year we put everything on the table and analysed the value of every question. The core attributes we use to measure AR effectiveness remain (plus one new one), but other questions have been removed, updated, simplified and/or clarified. We believe that this will not only make it easier for analysts to respond, but will also enhance the quality of the results.

Native language support

For the first time, we are effectively running three separate versions of Understanding the Influencers this year – in English, Chinese and Japanese. While analysts in China and Japan generally read & write English fairly well, particularly in the large global firms, we know that it will be easier and less time-consuming for them to be able to engage in their native languages. We’re hopeful that this will also increase responsiveness from local research firms where English skills are not so strong.

了解决策影响者们 2014

情報発信者を理解する 2014

Qualitative insights remain important

A significant differentiator for Understanding the Influencers has always been the fact that we overlaid the quantitative survey with qualitative depth interviews with more than 20 senior analysts across the region. This approach has allowed us to provide deeper insight into why specific AR programs are working (or not) and what other issues are important to analysts. We will continue to conduct these interviews with leading analysts who really understand AR, plus capture comments in the online survey in English, Chinese and Japanese.

More tailored analysis for vendors

One of the challenges of this type of study is that every vendor segments its target markets differently, and drives its AR program differently as a result. What’s important to one vendor will be less so to another, and the tools used will vary depending on size, resources and geographic focus. At the end of the day, we have to categorise both vendors and analysts according to their technology & business focus, what we call capture segments. It is impossible to get that 100% right, but we think we’ve improved our ability to map vendors with their target analysts by revising and expanding the capture segments. While we’ve always provided our vendor clients with customised analysis, we’ll be able to add a great deal more precision to that this year.

Earlier timescale for research & deliverables

We got a lot of feedback from analysts that our November/December schedule for fieldwork often conflicted with their need to finish research projects by the end of the year, so we’ve moved the start date to early October. This means that not only can we leave the analysts to finish their research in peace, but we can also deliver results for our vendor clients a month earlier. This ties in more neatly with typical planning stages which often take place in January.

Understanding the Influencers 2014 goes live next Tuesday October 7. We’re confident that all the changes we’ve made will drive up response rates and provide our vendor clients with deeper & more detailed insights on how to engage more effectively with the growing group of influential analysts in the APJ region.

If you’d like to know more, send me an email.




10 things I’ve learned about AR & the analyst business – and that you should know too…

It’s that time of year when just about everyone in the analyst business and the broader technology industry comes up with their prognostications and predictions for the year ahead. Inevitably, many of those will prove wildly inaccurate, overly optimistic or simply embarrassing.

So rather than fall into that trap, I decided to cast my mind back and consider what I have learned about analyst relations and the analyst business in APJ over 10 years of running the only independent AR consultancy in the region (that milestone ticked over in November), working with dozens of vendor clients and engaging with hundreds of analysts.

Here’s my list. I’ve written about some of these issues before at length – you’ll find more detail on previous posts. And while I’ve thought a bit about the rankings, this is just my perspective. Don’t be afraid to give me your thoughts.

1.         It’s the relationship, stupid

AR is all about creating a two-way dialogue between a vendor and an analyst. Relationship builders take the time to understand the analyst’s interests & needs, and personalise the engagement accordingly, but they’re also pretty good at creating internal executive support for AR, which is when the magic happens. A good relationship builder with a weak story and/or content will mostly do better than an average engager with great content

2.         Most analysts are decent human beings

Yes, some are arrogant, anally-retentive or just downright difficult, but 99% of the time they’re trying to do the right thing for their clients. An engagement approach that recognises this can turn an adversary into an advocate. Analysts also need to have some relationship skills, and dickheads just don’t last.

3.         Vendors will continue to under-invest in APJ

I’ve written about this before, and sadly it’s just a reality of the technology business in this region – it applies as much to overall marketing & sales investment as it does specifically to AR. Vendors have a poor track record of making decisions at HQ which don’t take into account the growth & needs in emerging markets, and the current global economic situation isn’t going to change that. (But reading the goat entrails available to me, I feel cautiously optimistic that the needle is swinging back a little in 2013, after a very lacklustre 2012).

4.         Many vendors just don’t “get” AR – nor do some analysts

Some vendors will never “get” AR, simply because they don’t try to understand the value that analysts provide, or how they are differentiated from other influence communities. Their loss. On the flip side, some analysts fail to see that most AR professionals are advocates for analysts, not gatekeepers – despite all the evidence to contrary. It’s not a perfect world…

5.         Analyst targeting is the most important element of any AR program

Full stop. Understanding your audience is the cornerstone of any marketing or influence program, and AR is no different. Targeting is the first step, tiering is the second, engagement approach follows. All analysts have different needs & require different approaches, regardless of their prioritisation.

6.         Influence is global/regional, but engagement is local

Many analysts engage with clients right across the world, not just in their home countries or cities, and it’s important to understand where individual analysts have impact. But it’s much more important to engage with analysts in their own timezones. Regardless of how you manage that, you can’t assume that information will trickle down to an influential analyst who’s sleeping when you decide to run a briefing.

7.         Training spokespeople to engage with analysts is a no-brainer

Why wouldn’t you want to give your executives the best possible preparation for engaging with key influencers? Dealing with analysts is not that complex, but it is not innate, and spending a few hours upfront demystifying the analyst business yields immediate benefits and also avoids embarrassing outcomes.

8.         Measuring results is critical to AR success

You might consider this a bit self-serving, considering I provide a measurement program, but really – if you’re not measuring what you’re doing, then what are you doing? Don’t try to measure everything, and focus on measuring where you’ve actually influenced analyst perceptions – this is where you’ll demonstrate value to your internal stakeholders (and holders of the purse-strings).

9.         Some vendors will continue to confuse running analyst summits with an AR program

Sad, but true. An analyst summit is a one or two-day event which provides the opportunity to showcase your key messages, introduce some key customers, dig into some nitty-gritty around technology or go-to-market strategy, and develop relationships between key executives and analysts. An AR program is a day-to-day interaction process which ensures that analysts get the information they need, when they need it – and ensures those relationships prosper. To do the first without the second is a waste of time, effort & money.

10.       Change in the analyst landscape is a natural state

It is for every other business, so why not analysts? Firms will continue to grow & prosper. Some will be acquired because they offer something different, others because they have lost focus but retain analyst value and/or an interesting client base. Analysts will continue to become disgruntled with their employers, then quit to explore new markets, business approaches and delivery models. And so the cycle continues…

Just one more thing, which doesn’t require a number of its own…

In 2013, doing AR will continue to be fun/challenging/rewarding/ frustrating/boring/exciting/bloody hard work/just a breeze… Pick your adjective – it will be all of the above, and more, but the one thing I hope is that for AR pros and analysts alike : it will be worthwhile!! And fun, of course…

So tell me what you think. Have I missed anything? Would you rank these points differently?



What’s the point of running an AR program if you’re not measuring the outcomes?

Marketing is all about results – there’s no point doing something if it doesn’t deliver results, and there’s not much point doing something if you can’t measure those results. So it’s often astounded me that otherwise savvy marketing folks are quite happy to invest in AR, but not in measuring the outcomes.

As the saying goes, the definition of stupidity is doing the same thing again & again, but expecting different results. And so it goes that when you’re not prepared to examine your outcomes, then you’re bound to repeat the mistakes of the past.

You might think this is a bit self-serving, given that I’m about to bang on about a product that addresses this issue, but the fact is that monitoring, measuring & modifying AR programs has significant benefits for ICT vendors and analysts alike.

What got me thinking down this track was doing all the development work for our annual study of AR effectiveness in APJ, Understanding the Influencers 2012, which goes into the field early next month. This will be the 10th year of running Understanding the Influencers, and every year it has been tweaked to better reflect the market, better meet the needs of our clients & make it easier for analysts to provide feedback.

This year, the changes are slight, but the level of detail & insight we get now compared to when we launched in 2003 is incredible. Last year, we introduced some new questions about analyst use of social media, discussed in this blog post and this one, so this year we’ll be able to provide some comparative analysis of the changes in analyst behaviour.

The point I’m making is that just as this study has had to continue to evolve, so should AR programs. Just because something was working last year doesn’t mean that it is this year. Or that just because you think something is working, that necessarily means it is…

Too many AR programs focus on counting activities eg number of analysts briefed, number of white papers purchased etc rather than measuring outcomes eg positive report mentions, positive analyst recommendations in deals etc. That’s not to say it’s not important to know what activities you’ve undertaken – it is – but it’s not a measure of success.

Nor am I trying to suggest that measuring the success of AR programs is simple – it isn’t. There is no silver bullet, no single metric which can define success in AR, and that’s a frustration for many. Rather, any AR practitioner worth their salt will use a number of measures – some of them subjective, some of them objective – to build an overall dashboard.

Whichever way you look at it, analyst perception studies are an important element of that. They don’t provide all the answers, but they provide many of them, and they provide the opportunity to get insight which is very specific to an individual vendor, and to identify issues which are actionable.

So what’s the benefit to a vendor from an analyst perception study?

Objective, third-party validation

Most vendors value analyst reports because they provide an independent assessment of markets or technologies, and analyst perception studies such as Understanding the Influencers are just the same. Individual responses are not identified, so analysts can be completely open & honest in their assessment – and believe me, they mostly are – so feedback doesn’t get moderated by politeness or commercial imperatives as it might in a direct conversation.

Depth, detail & richness of data

Syndicated studies such as Understanding the Influencers are structured to capture feedback from as many analysts as possible in a short period of time so that sensible comparisons with competitors can be made and enable detailed analysis. In 2011, we received responses from 140 analysts at 20 different firms in 12 countries, covering more than 70 different vendors & service providers, which provides a wealth of perspectives. The ability to slice & dice this data is significant.

Tailored, actionable advice

This ability to cut the data many ways provides us with the ability to drill down into issues specific to each client, and recommend appropriate actions to address problems and leverage opportunities. Apart from the quantitative research, we also conduct 20+ in-depth interviews with senior analysts, in which they provide very specific feedback on what’s working, and what’s not.

Insight into analyst thinking

Apart from measuring a number of key attributes to evaluate vendor AR programs, we also ask a number of questions about the analyst’s involvement in & likelihood of recommending vendors in a deal. While this is an indicator rather than an absolute, it quite clearly shows whether a vendor is getting traction at the “business” end, or not.

And what’s the value to an analyst in responding?

The analyst’s voice is heard

Analysts tend to have strong opinions, and most aren’t backward in coming forward. They want vendors to understand that analysts’ information needs are different to other audiences, but they don’t have the time or energy to tell them one by one why their AR program sucks, or how they can improve it. But by spending 15 minutes to complete a survey once a year, they can get their message across load and clear.

The analyst/AR ecosystem benefits

Many analysts value AR and know that a good AR program can make the analyst’s job easier. They also know that AR pros are powerful advocates for analyst research & insight inside the vendor organisations. They know that if they help the AR pros do their jobs, AR will help the analysts build their profile, and so the process continues.

There are many analysts who have participated in every study I’ve conducted over the past nine years, and most of them are the heavy hitters of the APJ analyst business. These are the analysts who put up their hands year after year to participate in 15-minute qualitative interviews, on top of completing the online survey. They know that it’s a good investment of their time, and some of them just like to give a little bit back.

As usual, that’s the perspective through my lens. I see plenty of upside in analyst perception studies for vendors and analysts alike, but what do you think? All benefit? Or is there a downside? Join the conversation below.

Or if you just want to know more about Understanding the Influencers, click here.