“You cannot manage what you cannot measure.” Those are well known and accepted words of wisdom that have been taught to thousands of MBA students for decades. There is a lesser popular truth though—that measuring wrong things can really hurt your business. Definition, selection and design of appropriate, balanced and concise metrics, as well as the processes for continuous delivery of these metrics, are the key management challenges for every organization.

1.      Focusing on the “wrong” metrics will create unintended results.

One of the more graphic examples of “wrong” metrics is a ratio of successful convictions, widely used to measure success or reputation of public prosecutors. This one causes US taxpayers to waste hundreds of millions of dollars in court costs, compensation for wrongful imprisonment and lost productivity every year. Essentially this metric is measuring a percentage of tried cases that result in conviction against the total number of cases tried by the prosecutor. The higher the percentage of convictions, the more “successful” the prosecutor is considered to be regardless of how well justice is served, how many lives are destroyed and how much financial damage is inflicted.

Another example is the evaluation of a Product Manager performance based on the attainment of product forecast goals. While I am familiar with a popular definition of a Product Manager’s role as a “Product CEO,” the organizational reality does not often support this definition, as product managers rarely have administrative authority to enforce their decisions and act mostly as influencers. It is intellectually dishonest to keep them accountable for a result of a sum of aggregated decisions made by a multitude of people, but most importantly it does not help to bring desired improvements in products performance.

2.      Focusing on unbalanced metrics will promote bad behavior.

Performance is often measured by a singular metric, yet people are rarely expected to behave one-dimensionally. Everyone knows that a lot of digressions will be forgiven to a salesman who consistently makes his quota, even though his lack of desire and/or skill to forecast costs your company serious hits to profit margin. Imagine it is the end of a quarter and you are deeply discounting your product in a desperate attempt to make your company revenue numbers, just to see your “best” performer bringing in a “bluebird” deal you had no visibility of. He just caused you to give away profits, and you did not need to sacrifice for “please sign today” deals. A secondary measurement attached to accuracy of forecast and associated with commission structure can dramatically improve a company’s profitability.

3.      Concise metrics promote action.

Conversely, the convoluted metrics are a waste of time, expense and opportunity. Many Customer Satisfaction measurements are falling into this category because they are often too general, and the best cause of action you may take is to do more studies. Most companies do not even consider competitive influences on their customer’s assessment of their satisfaction with their products or services. Unless feedback from customer analysis of every key component of customer experience is continuously conducted, and in relation to competitive options available to the customers, it is very difficult to figure out why overall Customer Satisfaction is moving higher or lower, who should take any action and what kind of action should be taken.

In conclusion, I would like to suggest that any performance metric has to be evaluated in a holistic model as it is very easy to come up with a clever way to improve one aspect of a specific performance at a detriment of the long-term well-being of the company as a whole.

This analysis was updated. Click to see the new report

I am getting ready to buy one, but buying one is just a beginning of the experience. The trick is to select the DMR that will give me more joy than a headache. Somebody said that there are 3 kinds of people:
1. people who learn from other people mistakes
2. people who learn from their own ones, and
3. people who never learn.
Let’s try to be the first kind of people and learn from others about their DMR experiences.

Let’s start with filtering the DMRs that have most customer reviews available on Social Media venues as there is a safety in numbers. I am not saying “Eat shit – 5,000,000 flies can’t be wrong!”, but there is a value in statistically representative information and it is much more difficult to plant a large number of reasonably descriptive customer reviews – just ask the Idiot Marketers who tried and got caught.

Comparing the “stars” of these receivers does not reveal much

as all of them sport 3.5-4 stars forcing me to sift through hundreds of reviews to decipher which one would give me the most satisfaction with the least risk and headache. This problem provided motivation for development of Opinion Miner® software and applications that are using it to produce the following score card.

The green shadowed cells indicate the highest score for an attribute and the red one highlight customer disappointment.

This makes my selection much easier as I can see that Roku XD delighted their customers with most of the attributes important to them. More than any other receiver we considered. However I also have information to make this decision personal, not just following the math – I do not buy from a company that disappoints their customers with Customer Support that makes…..

Drum rolls please!

The winner of 2010 Piplzchoice Award in the Digital Receiver Category is Apple TV 2010

I have written and published reviews online about books and products for over 10 years. Primary motivation for doing it is to share my experience with other consumers to help them make a choice, and with manufacturers to learn about customer experience with their product. I know that only a tiny number of consumers find it important to write the reviews as on the average well under 1% of a product customers review them, but my motivation is the same as most reviewer’s:

“Fully 90% write reviews in order to help others make better buying decisions, and more than 70% want to help companies improve the products they build and carry.”

Sounds like a truly win-win proposition until dishonest, idiot marketers come with “brilliant” idea to manipulate customer’s feedback. We will never know what financial impact was experienced by Belkin after their reviews manipulating scheme was publicly revealed, but the brand reputation stain keeps lingering years after it became public. Just Google “Belkin reviews scandal”

Belkin became a “poster child”, but it surely is not the only perpetrator of public trust. There is a long list of slimy morons who figured out it is easier to brake the law (yes, it is illegal) than to produce really good product the customers would be inspired to write about. I can’t imagine that economic value of results produced using cheap manipulation can possibly justify the legal and financial risk involved, yet in spite of all scandals and successful legal proceedings the practice continues, and cheap labor providing sites are flooded with ads like this

I don’t want to single out travel industry, but hotels seem to be the worst offenders as these ads are running for years

All of these activities are unethical and illegal, practiced openly through Social Media channels and yet did not destroy public trust in value of Customer Reviews.

Are consumers dumb and unaware of chicanery or is it a testament to human resiliency and faith? My bet is on the second and we will continue to work on development of algorithms to help us minimize influence of the disingenuous product reviews.

Today we rolled out a new version of our home page. That change signify the change in our focus from monitoring huge number of products, to quantify their product reputation measurements, to deep analysis of customer stories that is more resemble win/loss reporting. We also expanded our model to enable analysis of services.

We have all experienced frustration and disappointment of seeing the results of our efforts being wasted, the recommendations we have made not being followed on, insights we discovered not acted upon, and application systems we implemented not adopted by user communities, to name a few. If we assume for the purpose of this discussion that our efforts produced quality results, the problem that caused our unfortunate experiences is the problem of knowledge transfer between experts and managers in organizational decision making processes.

One of the Next Market Research (NGMR) LinkedIn group members shared with us a link to the study by Professor Martin J. Eppler of School of Communication Sciences, University of Lugano.

This paper is an excellent resource and I wish I would have access to it much earlier in my career. I’ll quote:

The efficient and effective transfer of experiences, insights, and know-how among different experts and decision makers is a prerequisite for high-quality decision-making and coordinated, organisational action (Straub and Karahanna, 1998).

For one reason or another, my experiences of these transfers were not often very effective, and most of these times, both experts and decision makers failed to build common context for the knowledge transfer.

The process of knowledge communication hence requires more reciprocal interaction between decision makers and experts because both sides only have a fragmented understanding of an issue and consequently can only gain a complete comprehension by iteratively aligning their mental models. All of this means that when we communicate knowledge, we are still communicating information and emotions, but we also create a specific type of context so that this information can be used to re-construct insights, create new perspectives, or acquire new skills.

I have been guilty on both side of the table: as an expert I often discounted value of my knowledge assuming that what I have learned is obvious to everyone around me, and as a decision maker I never had enough time and attention span to really understand intricacies that did not seem to address business problems I had to wrestle with.

How do you deal with this?

The positive effects of Word of Mouth references in customer acquisition (btw I hate that term) are very well documented.  Often I see the term “peer to peer” marketing being used in the same context interchangeably, however not being a marketing professional I am not sure if there is a difference or if they are really synonymous. Wikipedia defines WoM Marketing as

Word-of-mouth marketing, which encompasses a variety of subcategories, including buzz, blog, viral, grassroots, cause influencers and social media marketing, as well as ambassador programs, work with consumer-generated media and more, can be highly valued by product marketers. Because of the personal nature of the communications between individuals, it is believed that product information communicated in this way has an added layer of credibility. Research points to individuals being more inclined to believe WOMM than more formal forms of promotion methods; the receiver of word-of-mouth referrals tends to believe that the communicator is speaking honestly and is unlikely to have an ulterior motive (i.e. they are not receiving an incentive for their referrals).^[2]

Customer Reviews, describing personal experiences, opinions and recommendations of individual customers, are one of the best examples of WOMM. Amazon pioneered the approach and now there are many retailers like NewEgg, Best Buy and others, with technologies from BazaarVoice and PowerReviews that collect, manage and publish Customer Reviews.  I have both contributed and used them as guidance for my purchases for many years, and even though I understand that the reviews sometimes tell more about the reviewer than the product reviewed, I still find them the best tool for reduction of purchasing decision uncertainty. I know some tech pros and gadget mavens, who’s advice is sought and respected by many, to use customer reviews as an important part of their product evaluation process.

Consumers have no squabbles over paying for independent advice and recommendation, often called unbiased which is incorrect as such a thing does not exist IMHO. The Consumer Report was a very successful example and provided great service to generations of shoppers who subscribed to their magazine and its online version, however their model has difficulties to cover an ever expanding breadth of the products offered, and it does not really deal with customer experiences. The point however is that their approach is not misleading – you, the customer, pay them to learn their opinion and recommendation and thus the only incentive is to provide you with good and honest information.

I want to make very clear that I am not attacking profit motives or the marketing profession. I love profits when they are honestly earned by providing quality customer experiences, and I love marketing that helps me find providers of such experiences. The problems arise when some people or companies decide to focus on deception instead. Many years ago, one of my good friends shared with me her great admiration for Amway products. I was very grateful for her zeal to “help” me find a good product, until I realized the concealed motivation. Needless to say, we are no longer good friends, just acquaintances and I would never buy anything associated with the Amway brand. That is not to say that Word Of Mouth Marketing cannot be incentivised, just that marketers have to understand that it could become a double-edged sword and can easily create unintended adverse consequences. It also creates a challenge for us, at Amplified Analytics, to develop an effective approach to weigh authenticity of reviews we analyze for producing Product Reputation metrics.

We continue to test and debug the PRMIR (Product Reputation Market Intelligence Reporter), to get it ready for private beta. Original scenarios were focused on use of the Product Reputation data by product marketing managers of Consumer Electronics manufacturers to spot “systemic” failures of the products they are responsible for, and provide them with information to assist in developing corrective actions. However as I test the PRMIR, I keep coming up with new ways to focus and filter the information from different perspectives. The consumer focus (i.e. “what is the most reputable 40″ flat screen HDTV?”) is the most obvious example.

Best 40 inch TV

All 30 of very popular Asus netbooks have very high reputation for their functionality and design, but over half of them have some support problems. If I was responsible for Asus technical support or Customer Experience management, I would surely want to look at the source of that information (verbatim). The problem is that these metrics are based on 4,882 customer reviews and any meaningful analysis would take a significant amount of time to locate specific references to the support issues.  The Support “filter” below helps to focus on the subject of investigation.

Asus Support Filter

Click on the image to enlarge.

I found an interesting post by Jeffrey Henning today. The article is touching on emotional attachment people have for different measurement  scales used in Market Research. I can see how easily it come into play as we try to find one method to fit variety of research projects. Every method or tool has it’s limitations and therefore the challenge is to find the most appropriate one for the task at hand. Jeffrey quotes Brad Borther who provides an excellent advise:

Ten-point scale: “A five-point scale is totally inappropriate for customer satisfaction studies. Why? It lacks enough granularity and robs companies of a burning desire to take corrective action. It commonly leads executives to believe that ‘80% rate us four or five; that’s great, let’s move on,’ without realizing that it simply means that 80% are at least somewhat satisfied. Further, many people will never rate anything a ‘five,’ resulting in ‘four’ including those who are really very satisfied and those who are only somewhat satisfied. To avoid this topping effect, use at least a 10-point scale and count nine and 10 ratings as fully satisfied. This will also allow easier analysis of what bottom-line effects satisfaction has, since such tools as regressions work better with a more granular score.” – Brad Bortner, principal analyst with Forrester Research, “Best Practices: Why Customer Satisfaction Studies Fail“

Since our approach to measuring Product Reputation (delta between Customer Expectations and Customer Experience) is focused on competitive position of multiple products within their category, and our method does not require to ask people to measure it,  I have decided to use “0″ to “2″ balanced scale with 2 decimal points for more granularity. It is interesting how infrequently people want to challenge a value of our methodology or accuracy of our analysis, compared to the selection of the measuring scale. By now I gave up any attempts to change their mind. We arrive to the scores using our algorithms to analyze Customer comments and reviews, not by asking them to measure according to any scale, therefore is much easier for us to recalculate Product Reputation scores to appear in a customer “favorite” scale. The integrity of the finding is not compromised by the conversion.

I wish all religious wars could be settled that easily.

I wrote this story awhile ago, before we had this blog started, to eventually publish here.

It all started in August 2007, when I had to travel across the continent almost every week and started shopping for a lightweight laptop computer suitable for business travel.

While selecting parameters for my planned purchase in terms of specifications was not too difficult, if you know what you want, predicting the quality of your ownership experience may prove to be much more complicated. I’ve been an online shopper for a long time, but the recent cost of shipping and the hassle of dealing with less and less responsive customer service reps, started to outweigh the original savings and conveniences.  As a result I became much less impulsive with my online purchases – in this economy there is less money to spend and less time to waste, so I figured a small investment into initial research would be a good thing.

I found three major sources of information that were quite useful to assist me in making the choice:

1. Product specifications provided by the Original Equipment Manufacturer that are part of advertising and marketing collateral and designed to create our expectations of functionality and performance, but provide little help to gauge the probability of these expectations being met;
2. Editorial Reviews provided by magazine and online publishers that offer us a glimpse of potential user experiences which are quite valuable, but substantially removed from the regular consumer environment – the editors test carefully selected and tuned equipment provided at no cost to them by the OEM, and don’t have to deal with fulfillment, delivery and customer service issues. Unfortunately Consumers Reports did not review the laptops I was interested in and their recommendations were not available. A relatively quick tour of a few popular web sites helped me to create a short list of the two laptops that met my requirements;
3. Consumers (Users’) Reviews provided by actual purchasers of the product who share their personal experience and rate their satisfaction with this product. There is a relatively high probability that one will be very satisfied with a product that has been rated very highly by most reviewers. Adversely, you will do well avoiding products that are rated very low by most reviewers.

Two laptops that I had short-listed for purchase, based on product specifications and editorial reviews, had very similar reputation ratings of 3.5 stars out of 5, which is not perfect, but acceptable. So what is the next step? Toss a coin? Is it safe to assume that these two products have the same reputation and would be equally satisfying purchases?

It turns out that this would have been a very bad assumption.

Usually I lack in patience (those who know me, please stop laughing – “Good men know their limitations”), but this time I decided to assess whether the reasons, that prevented these laptop users from giving the highest satisfaction ratings of 5, are “showstoppers” for me, or not – we all have our own limits of tolerance to different experiences. I ended hunting for and reading through dozens, or sometimes hundreds of reviews and found out that the most negative experiences with laptop #1 were centered around overheating issues and resulting customer service hassles, where negative reports of laptop #2 were focused on order processing and fulfillment problems.

I had invested 8-10 hours of my time in research and spent $120 more than originally anticipated, to purchase the laptop #2 from an online retailer that had it in inventory, to bypass the fulfillment problem and have been enjoying my laptop without any reliability problems. I hear the laptop #1 OEM has finally found workarounds for the overheating problems and is now working to pacify many of their very vocal and unhappy customers.

As smug as I am about this experience, the efforts required to do such research require too much time and patience and I wondered how much easier it would be to have more meaningful product reputation ratings. So I looked, but could not find anything unbiased, consistent and verifiable to make it work for me. That is how my new project, Amplified Analytics was born. Please look around the site and let me know if it makes any sense to you. I would love to hear your experiences related to product reputations and user reviews.

Very interesting results of the survey:

A Social Media Survey conducted on behalf of PRWeek and MS&L by PRWeek and CA Walker found that marketers don’t make changes to their products based on customer feedback, despite monitoring feedback being one of the most common business uses of social media in the first place.

The survey found that 70% of marketers say they’ve never made a change to a product or marketing efforts based on feedback from consumers on social media sites.

I have to second Larry Malloy’s comment.

I believe there’s two reasons for this.First, we are still in the early stages of social media as a marketing tool. I believe as the technology matures, potentials are stretched, metrics are determined, and processes are developed this will change.

Second, there could be a disconnect between marketing and product management (you said the survey polled senior level marketers). As a product manager, I often used social media throughout the product lifecycle, and the executives I reported to often did not know where the new product ideas came from. And, what I learned through social media, I often further tested through more traditional marketing technologies like surveys, customer visits, interviews, etc.

Most Product Management and Marketing executives I have talked to are interested in listening, but have no strategy, processes, methodologies or best practices to act on customer feedback. Most tools available today are not providing particularly actionable data either. I am not sure what would or should come first, but without these elements you cannot produce any ROI. I attempted to come up with a “calculator” to measure an impact of customer feedback on product profitability, but it is just a rudimentary attempt for discussion and anybody who wants a copy can find it here.