Picture for a moment (if you will) the time, effort and money spent to build a survey draft, field it, tabulate and analyze data – only to find that the final overall metrics or scores the researcher us were faulty or misleading.
Not only has the researcher wast their effort and their client’s money but perhaps more importantly, they have squander respondents’ valuable time, to say nothing of the cost of incentives, etc.
Why is choosing the right customer
metric so important to researchers? (In this discussion, when we say customer metric” we are referring to overall customer satisfaction or customer loyalty metrics.) Many companies place great emphasis on KPIs like overall satisfaction or loyalty scores and include them in their corporate goals and compensation and/or bonuses.
The ultimate marketing tool to drive success is phone lists. Your business will connect with the right audience, meaning that the phone number list engagement will grow, conversion rates will increase, and the marketing will be further effective. Targeted outreach due to a well-compiled accurate list guarantees much speedier growth and increased sales.
But what if the system of gathering and measuring these corporate goals and the resulting employee compensation was flaw? Not a good pricament to be in.
There are two metrics
Which incorporate a five-point scale, that have been us for many decades without problems or flaws (provid the research is done correctly): the “gold standard” American Customer Satisfaction Index (ACSI) customer satisfaction question – “How satisfi are you with the following product/service?” – and also a standard loyalty metric question – “How likely are you to purchase X products or services in the future?
I was employ as a research consultant for nearly four years at Kaiser Permanente, where I was responsible for research, metrics and executive interface for its largest lines of business (national accounts, strategic accounts, labor and trust, etc.).
When I first join Kaiser, executives were complaining that Kaiser’s Net Promoter Score (NPS) appear to be very unprictable. I inform them that NPS was a volatile metric and that we should adopt a different one.
The problem was
Employee and executive compensation and scorecards were ti to NPS, so naturally the executives were hesitant to change. I suggest that we do a real-world customer metric comparison by having two loyalty metrics in the surveys versus just the NPS.
For a couple years, we us hello world! both the standard ACSI five-point loyalty metric question, “How likely are you to purchase X products or services in the future?” simultaneously with the NPS metric question “On a scale of 0 to 10, how likely are you to recommend Kaiser Permanente to a friend or colleague?
At the end we closely examin
the differences between the two metrics (especially since employee and executive compensation were bas on them). The results were conclusive: We found the NPS score to be of no value to Kaiser and further that it actually provid negative value because of its volatility. (While the NPS score fluctuat wildly, all the other question scores in the survey remain almost identical over the same period.)
Let’s discuss the numerous significant flaws of the NPS scoring system and why NPS system fail:
Begin with the NPS question and wording itself. The NPS question is bas around measuring the respondent’s likelihood to recommend. First of all, many respondents would never naturally or formally recommend in the first place, so this is not a good question to ask. Therefore, a score of 9 or 10 doesn’t mean respondents will actually recommend. Rather, likelihood to purchase (vs. recommend) is a better indicator in most cases.
For an article publish in 2008 in the MIT Sloan Management Review (“Linking customer loyalty to growth”) a study found no phone number sa evidence that NPS was the best prictor across customers’ future loyalty intentions. The authors also attempt to find a link between NPS and growth, a part of the NPS measure that has been attractive to companies.
They examin data from more than
They then add in the growth rates for those companies. None of the range of metrics they examin, including NPS, was found to be a good prictor of growth. As the authors note, “Even when ignoring statistical significance (the likelihood that the correlations occurr by chance), Net Promoter was the best prictor in only two out of 19 cases.” They conclude that, “bas on our research it is difficult to imagine a scenario in which Net Promoter could be call the superior metric.”
Also another example of problems with “likely to recommend” comes from a study titl, “Measuring customer satisfaction and loyalty: improving the ‘Net-Promoter’ Score,” by Schneider, Berent, Thomas and Krosnick (2008), who found satisfaction is a stronger prictor than the likelihood of recommending.
Measuring NPS is simply a case of asking the following question: How likely is it that you would recommend X to a friend or colleague? It uses an 11-point scale of 0 (not at all likely) to 10 (extremely likely) and, bas on their responses, customers fall into one of three categories.
Promoters respond with a score of 9 or 10; Passives respond with a score of 7 or 8; Detractors respond with a score of 0 to 6. Net Promoter Score is calculat by subtracting the percentage of Detractors from the percentage of Promoters.
The percentage of Passives is not us in the formula
For example, if 10% of respondents are Detractors, 20% are Passives and 70% are Promoters, your NPS score would be 60 (70-10). Here’s one example in which the NPS calculation provides flaw computational results: a combination of 20% Promoters, 80% Passives and 0% Detractors gives the same score as 60% Promoters, 0% Passive and 40% Detractors!
Another NPS score computational issue is that NPS is not symmetrical (top-two boxes minus the bottom six).
The NPS 11-point scale lacks labels on the scale to guide respondents (including no neutral label).
Well-known researchers (such as Jon Krosnick, see below) have complet extensive studies mentioning that respondents have problems differentiating beyond a seven-point scale.
In fact, this author cannot differentiate using an 11-point scale so I don’t expect my respondents too either. To illustrate why an 11-point scale is so difficult to differentiate, the example above shows how ridiculous an NPS scale would look like if you put labels on the scale. Do you realistically expect your respondents to be accurate with this NPS scale, label or unlabel?
A score of 0 on the scale throws off respondents who are us to seeing a 1.
In addition, rucing an 11-point scale to three points increases the statistical variability.
NPS scores vary by industry.
Sample sizes ne to be increas for NPS
In a July 2007 Journal of Marketing article (“A longitudinal examination of Net Promoter and firm revenue growth”) the authors offer empirical evidence using data from 21 firms and 15,500 respondents: “We find that when making an ‘apples and apples’ comparison, Net Promoter does not perform better than the American Customer Satisfaction Index (ACSI) for the data under investigation. [NPS creator Fr] Reichheld acknowlges the ‘imperfections’ in the analytics that were us to support Net Promoter. Unfortunately, the statistics matter.
