We finally got enough customer feedback to see the evidence that iPhone 4S substantially improved customer experience compared to iPhone 4 models. Apple resolved antenna issues completely and as the result significantly improved reception/signal clarity. Customers love new keyboard experience and their usability satisfaction ratings are exceedingly high. The results are based on Opinion Mining of 4,489 Apple iPhone customers published online.

The image above highlights the attributes of customer’s experience most important to them as they have articulated in their feedback. No keywords were used during the analysis to identify these attributes, and no questions were asked to influence the answers, as surveys are not our business or part of our opinion mining methodology. Click on the image to enlarge it.

This morning, I had an interesting experience. Among many emails, tweets and webinar offerings, I managed to expose myself blindly to two that focused on one challenge I have to overcome the most – talking too much and listening too little. Interestingly enough, neither of the messages was specifically targeting individual or style shortcomings, and the subject line of these presentations was not about listening skills. If this is not a moment of serendipity, then I don’t know what is. BTW it also synch with my favorite definition of serendipity: “…is when you come to look for a needle in a stack of hay, and you end up finding a farmer’s daughter.”

The first piece of content is a video from http://www.entselling.com/ that talks about challenges of entrepreneurial selling and is not focused on the selling or listening style at all. It is very good and I strongly recommend it to any startup team, but the listening piece resonated with me the most. I’ve been trained on the importance of this skill for selling many years ago, and judging by my performance at the time, I have even learned to apply it. However, as it may be obvious to people who know me, it is not one of my natural qualities . The more I get excited about the subject of conversation, the less patient I get with listening to my conversation partners, particularly if I think I already figured out what they are trying to communicate.

I may be right about that, but it doesn’t create a great conversation experience, nor does it make them feel that they have been heard and that I actually do understand their concerns or problems I am proposing to address with my product or service. Apparently, it is a very common problem undermining many startup founders who are understandably excited about their creations to a detriment of their potential customers’ comfort, and subsequently a sales success. Maybe I should start looking for a startup founders “shut up and listen” support group. Please let me know if such a group exists.

The second piece is even more interesting and was presented by Rebel Brown at the Defy Gravity webinar sponsored by TreeHouseInteractive. Rebel is a very dynamic and passionate speaker, and she was talking about many marketers trying to use social media as traditional content broadcasting channel. The main lesson I took out of this presentation is about a challenge of institutionalized listening that needs to become a part of personalized conversation if a brand wants to be successful in social media. I suppose that no brand will be able to survive without social engagement with their customers, as the customers are creators of a brand.

“Advertising can help you sell good products, but only your customers can help you build a great Brand!”

The challenge is in learning what is important to your customers and communicating with them about this, as opposed to focusing on your product or your brand. I think the most difficult part is to not assume that we already know what it is, and not be afraid to learn from these communications.

Rebel also made a great point about the practice of counting followers and “likes” as a result of social media efforts. I will paraphrase it here as, “Do not confuse tactical metrics with actual meaningful results.” Let’s face it – these only exist because they are easy to count. Their relevance to business outcomes is very questionable, and the only thing they help to learn is how to manipulate or game the counting mechanisms.

I don’t think a “real” marketer can be helped by a mere support group or volunteer 12-step program. Perhaps there is an opportunity for a true “rehab.” All you need is a recovering marketing celebrity lending their name to this venture.

“You cannot manage what you cannot measure.” Those are well known and accepted words of wisdom that have been taught to thousands of MBA students for decades. There is a lesser popular truth though—that measuring wrong things can really hurt your business. Definition, selection and design of appropriate, balanced and concise metrics, as well as the processes for continuous delivery of these metrics, are the key management challenges for every organization.

1.      Focusing on the “wrong” metrics will create unintended results.

One of the more graphic examples of “wrong” metrics is a ratio of successful convictions, widely used to measure success or reputation of public prosecutors. This one causes US taxpayers to waste hundreds of millions of dollars in court costs, compensation for wrongful imprisonment and lost productivity every year. Essentially this metric is measuring a percentage of tried cases that result in conviction against the total number of cases tried by the prosecutor. The higher the percentage of convictions, the more “successful” the prosecutor is considered to be regardless of how well justice is served, how many lives are destroyed and how much financial damage is inflicted.

Another example is the evaluation of a Product Manager performance based on the attainment of product forecast goals. While I am familiar with a popular definition of a Product Manager’s role as a “Product CEO,” the organizational reality does not often support this definition, as product managers rarely have administrative authority to enforce their decisions and act mostly as influencers. It is intellectually dishonest to keep them accountable for a result of a sum of aggregated decisions made by a multitude of people, but most importantly it does not help to bring desired improvements in products performance.

2.      Focusing on unbalanced metrics will promote bad behavior.

Performance is often measured by a singular metric, yet people are rarely expected to behave one-dimensionally. Everyone knows that a lot of digressions will be forgiven to a salesman who consistently makes his quota, even though his lack of desire and/or skill to forecast costs your company serious hits to profit margin. Imagine it is the end of a quarter and you are deeply discounting your product in a desperate attempt to make your company revenue numbers, just to see your “best” performer bringing in a “bluebird” deal you had no visibility of. He just caused you to give away profits, and you did not need to sacrifice for “please sign today” deals. A secondary measurement attached to accuracy of forecast and associated with commission structure can dramatically improve a company’s profitability.

3.      Concise metrics promote action.

Conversely, the convoluted metrics are a waste of time, expense and opportunity. Many Customer Satisfaction measurements are falling into this category because they are often too general, and the best cause of action you may take is to do more studies. Most companies do not even consider competitive influences on their customer’s assessment of their satisfaction with their products or services. Unless feedback from customer analysis of every key component of customer experience is continuously conducted, and in relation to competitive options available to the customers, it is very difficult to figure out why overall Customer Satisfaction is moving higher or lower, who should take any action and what kind of action should be taken.

In conclusion, I would like to suggest that any performance metric has to be evaluated in a holistic model as it is very easy to come up with a clever way to improve one aspect of a specific performance at a detriment of the long-term well-being of the company as a whole.

Tablet’ Market Segment update

The last month was full of action if you follow the tablets market segment. HP first started the price discounting to “gain market share” and then dropped the “bomb” of discontinuing the TouchPad they introduced only 42 days before. The judgment is still out whether we witnessed results of really agile decision making or an example of exceptionally bad corporate self-distraction. The initial stock market reaction seems to support the latter hypothesis, and current history of HP boardroom soap opera episodes provides enough clues. The new CEO apparently wants the company out of consumer products businesses.

There is an update – It appears the decision produced a shark bait effect and caused a number of class action suites on behalf of shareholders.

In this installment of online market research update, I would like to explain the process we follow to generate these reports.

We start at the Product Reputation screen and enter name/model of a tablet and click the “Submit” button. When the metrics for the entered tablet show up on the right side of the screen, we click on the “Compare with other products” button to expose all tablets in our database.

At this date, we monitor customer generated content for 32 tablets and the Market Intelligence report will show their reputation which is calculated by processing the text of customer reviews with our opinion mining software. Keep in mind that no questions were asked about their experiences and no customer was ever contacted by us to solicit their opinions. Surveys are not our business!

We use the “Customize your report” button to select only the tablets that were updated (new customer reviews were published online) within the last 30 days and have greater than 25 customer reviews published. The total number of reviews analyzed is 9,003. The result of the customization is displayed below. Please note that the metrics are calculated on all aggregated reviews published by the customers from the time a tablet became available for purchase.

We export the CSV file and generate WoM (Word of Mouth) Share chart and Trend based on the exported data. Not surprisingly, Apple iPad earns the largest share of customer feedback at 17%, although the number is much smaller than its market share and substantially dropped from 35% only two months ago.

To discover what attributes of their customers’ experience are important, to measure how important these attributes are to the customers, and what the difference between their expectations and their experiences with each attribute is, we focus on five of the most reviewed tablets for a more detailed Customer Intelligence Analysis.

We use a two-point scale to visualize that difference.

0=unacceptable / 1=experience meets expectations / 2=delighted

Customers “say” that Usability (11.64%), Reliability (11.32%), Quality of Construction (9.5%) and Display (5.4%) are the most important attributes of their experience with the tablets. While all participants are providing Usability experience well above their customer’s expectations, HP TouchPad and Apple iPad 2 are the leaders in this category. Motorola and Toshiba are considered the most “reliable” tablets by their customers, while Samsung Tab continues to struggle.

Here is the access to the tablets’ online marketing research dashboard that allows to see actual customer’s feedback if you click on a specific bar.

The last chart I would like to offer depicts customer affinity for tablet’s operating system and despite all brouhaha about the TouchPad, WebOS still earns the highest score from its customers.

This online market research, administered by Amplified Analytics, is valid as of August 5, 2011 and is the result of the review of different tablet brands’ segment of the market. It is based on the analysis of customer feedback from 8,241 tablet users who expressed their sentiments on the product.

The chart above depicts the Word of Mouth (WOM) Share for ten of the top tablet brands in terms of customer satisfaction (CSI).

The chart above illustrates shows customer satisfaction with Operating System of their respective Tablet.

The scores were algorithmically produced by the use of Opinion Mining software that conducts an analysis of customer feedback that is published online by the customers themselves; no customer was personally contacted to provide their opinions.  The relative percentages were not based on customer’s answers to biased survey questions. It is not Amplified Analytics’ business to conduct surveys.

It is also important to note that the Apple iPad2 was excluded from the list of tablet brands analyzed because it significantly dominates the market, thus making the comparison of customer satisfaction meaningless. The Apple brand seems to have its own market that is incomparable to others.

The chart below shows a comparison of specific attributes (reliability, portability, display, etc) of leading tablets. These attributes came from the customers when they shared their experience using the brands. The green line athwart denotes the relative importance of each attribute to customers. Check the methodology used in this link: Opinion Mining. Attributes with less than 1% importance were not included in the graph.

From the graph, we can see that Reliability is the most important attribute, with 11.7% of total opinions.

Yet, customers of Samsung Galaxy continue to get disappointed for the second month of measuring it since it was first introduced. Its Customer Support, with an importance rating of 1.76%, exceeds customer expectations by 8% while the Display attribute, with a 7.42% score, is on top of the competition.

You can access this dynamic Customer Intelligence dashboard by clicking on this link and CustomerSay! Verbatim by clicking on a specific bar of this chart. “Attributes” and “Products” selection windows allow for focus on your area of interest.

The tablets market segment is fun to watch. While there is no doubt that Apple “owns” the segment it is interesting to note that it has the lowest satisfaction score compared to the competition.

Given the enthusiasm of Apple fans and popularity of the original iPad, I wonder if the bar was set too high for many iPad 2 purchasers. Digging deeper into the details of Customer Experiences we can see that a lot of negative comments are focused on quality of the camera embedded into the tablet.

Indeed the “focused” listening provides specific metrics that show difference between customer expectations and their actual experience with this attribute of Customer Experience.

The analysis shows that Blackberry Playbook is a clear leader when it comes to Customer Experience with picture quality. Considering widely held belief (which I do not share) that Apple does not do market research it would be interesting to see if they address the camera/picture quality issue in the next edition of this popular product.

We used Opinion Miner® software to analyze 1,103 customer generated reviews, published online before May, 29 2011, of the tablets listed above to generate these findings. The scores are calculated to the two point scale from 0 (unacceptable) to 2 (delighted) with 1=100% satisfied (i.e. experience matches expectations).

The Playbook introduction caused a lot of controversy well before it was actually released with speculations about its battery life problems wildly spread all over internet. The decision to go with its proprietary operating system was also met with a lot of criticism in social media. However since the Playbook started shipping and customers had a chance to form their own opinion, the results of their opinions analysis did not produce a crystal clear picture yet.  One thing seems to emerge though – it is a lot better product than most people have expected.  This writing is based on the analysis of 886 customer reviews of their experience with these tablets.

Below is a screenshot of the dashboard with a subset of attributes that are most important to customers (statistically determined). The bars represent measurements of the difference between customer expectations and there experience with each attribute.

It appears that Battery life issue was either blown out of proportions by pre-release speculations, or miraculously corrected by RIM engineers before the product started shipping, as the Playbook customers report that it exceeded their expectations by 21%.

QNX operating system appears to be a big hit with the customers as it exceeded their expectations by 35% and scored at much higher customer perception levels than iOS and Android.

What is surprising that overall general satisfaction score from the Playbook customers is quite low in spite the fact the most of the important attribute’s scores are positive.  When I clicked on the general satisfaction bar of the Playbook, the negative opinion snippets (customer insight) was focused around the lack of content and applications for the device.

Word of Mouth (WOM) Analysis

Customers’ perception of mobile phones

(February 24, 2011)

Source Data:

• Social Media – customer generated content found on Amazon, Best Buy, Cnet and other popular consumer sites.

Total of 115 mobile phones from 9 major manufacturers were analyzed for this report.  18,194 customer reviews were located, processed and analyzed to measure difference between customer expectations and their experiences. The measurements are represented by a 2 point scale from 0 (unacceptable) to 2 (delighted), where 1 indicates that customers experienced exactly what they have expected.

As customers continue to buy these products and publish their reviews online, these findings, and conclusions drawn from them, may change over time. Judging by these products’ WoM history, no severe fluctuations in perception are likely within a monthly period.

Methodology:

• Only customer’s generated content is analyzed. Consumers’ opinions, without ownership reference, are not part of this analysis. The customer generated content was located, authenticated, de-duped and aggregated for the analysis.
• The content of the customer reviews is processed by Opinion Miner© software to measure the Customer Satisfaction Index and its components (Functionality, Reliability, Support scores) of the discussed products.
• The findings are organized to facilitate discovery of the products that have negative impact on a Brand Equity, compare Customer Perception of the Brands and specific products that roll up to these Brands.
• Discovery of specific product attributes and customer insights that help to understand why customers selected to purchase this product or passed it up for a competitor are not part of this analysis, but can be performed using Customer Intelligence dashboard.
• Access to verbatim of the analyzed customer reviews can be obtained by following this link and entering the name of the phone associated with these reviews.

Findings

RIM has the highest average Brand reputation score (1.16) which means that Blackberry phones exceed average customer’ expectations by 16.1%, while Nokia and Motorola brands where most disappointing to their customers.

Apple and HTC brands are the only two other brands that on average exceeded their Customer expectations, yet further analysis shows that Samsung Galaxy S phones are among the most reputable products of this segment.

In order to understand the impact of specific products on Brand Equity we have applied additional filters:

1.     All metrics have to be above 1(instead of just CSI) and

2.     Only phones that are currently being sold are being considered.

The results are showing five phones, currently sold, that have the highest average Customer Satisfaction scores.

It is not surprising to find two Blackberry phones in the group, as the RIM is the top Brand when it comes to average customer satisfaction. However the fact that Sony and Samsung phones are among the top 5, yet the Brands have overall low standings, is telling that the “dogs” of these manufacturers are dragging down their Brand Equity.

This begs the question – which Sony and Samsung phones are these “dogs”, that destroy their Brand Equity?

To answer this question we need to apply a different filter that would expose the 5 lowest CSI scored phones from these Brands.

Functionally, only Samsung Glide was really an unsuccessful phone as indicated on the chart below.

A closer look at the Reliability reputation of these phones reveals the reason for customer disappointment.

All of this analysis was performed by using Market Intelligence Dashboard

Here are some static examples (screen-shots) of customer feedback analysis for E-Readers and Tablets. The reason I mixed them together is because I have found that customers often investigate one before they purchase another. It appears that this category “divide” is largely historic and the two may eventually merge into one, just like laptops and notebooks have done.

This is the result of automated analysis of customer reviews published online. Opinions are measured on the scale from “0″ (unacceptable) to “2″ (delighted), with “1″ indicating satisfaction (expectation=experience). The importance is expressed by the line across bars and measured by a weighted average of total number of opinions about specific attribute of a specific product.

List of Attributes is extracted from customer’s conversations, not preconceived by any individual.

The dashboard controls allows a user to select and filter subsets of data for more targeted analysis. Below is an example of one-to-one comparison between Amazon Kindle and B&N Nook Color

Further “drilling” into the 5 (for example) most “important” to the E-Readers customers attributes may be achieved by selecting these attributes at the top left list of Attributes. The screen-shot below reveals that while both e-readers exceed their customers Usability expectations by a wide margin, Amazon holds 3% lead over the Nook Color which is within margin of error (+/- 7.2%). However its Reliability (second most important attribute) is 12% higher than Amazon Kindle. Both e-readers offer their customers a very good value exceeding their expectation by a wide margin, but the Nook pulls ahead again when it comes to Quality of Display.

This analysis was updated. Click to see the new report

I am getting ready to buy one, but buying one is just a beginning of the experience. The trick is to select the DMR that will give me more joy than a headache. Somebody said that there are 3 kinds of people:
1. people who learn from other people mistakes
2. people who learn from their own ones, and
3. people who never learn.
Let’s try to be the first kind of people and learn from others about their DMR experiences.

Let’s start with filtering the DMRs that have most customer reviews available on Social Media venues as there is a safety in numbers. I am not saying “Eat shit – 5,000,000 flies can’t be wrong!”, but there is a value in statistically representative information and it is much more difficult to plant a large number of reasonably descriptive customer reviews – just ask the Idiot Marketers who tried and got caught.

Comparing the “stars” of these receivers does not reveal much

as all of them sport 3.5-4 stars forcing me to sift through hundreds of reviews to decipher which one would give me the most satisfaction with the least risk and headache. This problem provided motivation for development of Opinion Miner® software and applications that are using it to produce the following score card.

The green shadowed cells indicate the highest score for an attribute and the red one highlight customer disappointment.

This makes my selection much easier as I can see that Roku XD delighted their customers with most of the attributes important to them. More than any other receiver we considered. However I also have information to make this decision personal, not just following the math – I do not buy from a company that disappoints their customers with Customer Support that makes…..

Drum rolls please!

The winner of 2010 Piplzchoice Award in the Digital Receiver Category is Apple TV 2010