This is a new analysis of customer feedback, which is available online, about their smart phones. At this time we are tracking and analyzing comments from 37,110 customers on 136 mobile smart phones. I decided to filter out the phones which were not updated with new customer comments during the last 30 days to insure that these phones are still available on the market. The resulting Product Reputation report is available at Market Intelligence.

I selected the most reviewed phones for each operating system to take a close look at what attributes are important to the smart phone customers. As customers keep posting their reviews and forum comments about their experience with the phones they chose,  Reliability remains the most important specific attribute that dominate the conversation as 15.22% of all opinions mined is focused on it.

Let’s face it, anybody who buys a smart phone and pays for the service expects to be able to use their phone every time they want to. Apple clearly outshines competition by exceeding customers’ expectations of Reliability by 10%. HTC-HD7 (Win7 OS “representative”) meets their customers’ expectations: however Android (HTC Thunderbolt), Symbian (NokiaN8) and RIM’s Blackberry Curve 9330 are a disappointment to customers who selected to purchase these phones.

Overall customers are satisfied with their decisions to various degrees, but Apple iPhone users are reporting that the phone exceeded their expectations by 42%. Not surprisingly they also are the most satisfied with the choice and quality of the Applications available to them, their Usability and Web Browsing Experience. Since I personally have never purchased an Apple product, nobody can accuse me in the Apple bias: however this phone has earned a remarkable reputation by managing not to disappoint its user in a single attribute of customer experience.

Nokia N8 leads customer satisfaction in Battery Life, exceeding expectations by 14%, Call Quality, Music Player experience, Sound and Video Quality. However, it also disappoints their customers where it really counts – poor Customer Support, inadequate Keyboard, Operating System experience and Web Browsing.

Below is the list of top 19 Customer Experience Attributes by their importance to the customers as they opined in their comments and reviews. Our methodology does not utilize surveys, focus groups, panels or other forms of leading questions/bias forming market research tools. I have filtered out any Attribute with importance below 0.35% that may be very valuable for Product Marketing analysis, but not very meaningful for general consumption. The complete list is available on request:

1. General Satisfaction (~CSI) – 16.96%
2. Reliability – 15.22%
3. Usability – 8.97%
4. Battery Life – 5.42%
5. Screen/Display experience – 4.44%
6. Call Quality – 3.78%
7. Customer Support – 3.39%
8. Style/Design – 3.19%
9. Picture Quality – 1.94%
10. Feature Set – 1.92

The selection of phones for the comparative analysis would vary based on criteria important to a person who conducts the research – I wanted to compare a single representative phone per operating system and you may want to find the best Android phone for example. The Attributes and their importance may vary based on such choice as customers “conversations” could yield substantially different results.

“You cannot manage what you cannot measure.” Those are well known and accepted words of wisdom that have been taught to thousands of MBA students for decades. There is a lesser popular truth though—that measuring wrong things can really hurt your business. Definition, selection and design of appropriate, balanced and concise metrics, as well as the processes for continuous delivery of these metrics, are the key management challenges for every organization.

1.      Focusing on the “wrong” metrics will create unintended results.

One of the more graphic examples of “wrong” metrics is a ratio of successful convictions, widely used to measure success or reputation of public prosecutors. This one causes US taxpayers to waste hundreds of millions of dollars in court costs, compensation for wrongful imprisonment and lost productivity every year. Essentially this metric is measuring a percentage of tried cases that result in conviction against the total number of cases tried by the prosecutor. The higher the percentage of convictions, the more “successful” the prosecutor is considered to be regardless of how well justice is served, how many lives are destroyed and how much financial damage is inflicted.

Another example is the evaluation of a Product Manager performance based on the attainment of product forecast goals. While I am familiar with a popular definition of a Product Manager’s role as a “Product CEO,” the organizational reality does not often support this definition, as product managers rarely have administrative authority to enforce their decisions and act mostly as influencers. It is intellectually dishonest to keep them accountable for a result of a sum of aggregated decisions made by a multitude of people, but most importantly it does not help to bring desired improvements in products performance.

2.      Focusing on unbalanced metrics will promote bad behavior.

Performance is often measured by a singular metric, yet people are rarely expected to behave one-dimensionally. Everyone knows that a lot of digressions will be forgiven to a salesman who consistently makes his quota, even though his lack of desire and/or skill to forecast costs your company serious hits to profit margin. Imagine it is the end of a quarter and you are deeply discounting your product in a desperate attempt to make your company revenue numbers, just to see your “best” performer bringing in a “bluebird” deal you had no visibility of. He just caused you to give away profits, and you did not need to sacrifice for “please sign today” deals. A secondary measurement attached to accuracy of forecast and associated with commission structure can dramatically improve a company’s profitability.

3.      Concise metrics promote action.

Conversely, the convoluted metrics are a waste of time, expense and opportunity. Many Customer Satisfaction measurements are falling into this category because they are often too general, and the best cause of action you may take is to do more studies. Most companies do not even consider competitive influences on their customer’s assessment of their satisfaction with their products or services. Unless feedback from customer analysis of every key component of customer experience is continuously conducted, and in relation to competitive options available to the customers, it is very difficult to figure out why overall Customer Satisfaction is moving higher or lower, who should take any action and what kind of action should be taken.

In conclusion, I would like to suggest that any performance metric has to be evaluated in a holistic model as it is very easy to come up with a clever way to improve one aspect of a specific performance at a detriment of the long-term well-being of the company as a whole.

Tablet’ Market Segment update

The last month was full of action if you follow the tablets market segment. HP first started the price discounting to “gain market share” and then dropped the “bomb” of discontinuing the TouchPad they introduced only 42 days before. The judgment is still out whether we witnessed results of really agile decision making or an example of exceptionally bad corporate self-distraction. The initial stock market reaction seems to support the latter hypothesis, and current history of HP boardroom soap opera episodes provides enough clues. The new CEO apparently wants the company out of consumer products businesses.

There is an update – It appears the decision produced a shark bait effect and caused a number of class action suites on behalf of shareholders.

In this installment of online market research update, I would like to explain the process we follow to generate these reports.

We start at the Product Reputation screen and enter name/model of a tablet and click the “Submit” button. When the metrics for the entered tablet show up on the right side of the screen, we click on the “Compare with other products” button to expose all tablets in our database.

At this date, we monitor customer generated content for 32 tablets and the Market Intelligence report will show their reputation which is calculated by processing the text of customer reviews with our opinion mining software. Keep in mind that no questions were asked about their experiences and no customer was ever contacted by us to solicit their opinions. Surveys are not our business!

We use the “Customize your report” button to select only the tablets that were updated (new customer reviews were published online) within the last 30 days and have greater than 25 customer reviews published. The total number of reviews analyzed is 9,003. The result of the customization is displayed below. Please note that the metrics are calculated on all aggregated reviews published by the customers from the time a tablet became available for purchase.

We export the CSV file and generate WoM (Word of Mouth) Share chart and Trend based on the exported data. Not surprisingly, Apple iPad earns the largest share of customer feedback at 17%, although the number is much smaller than its market share and substantially dropped from 35% only two months ago.

To discover what attributes of their customers’ experience are important, to measure how important these attributes are to the customers, and what the difference between their expectations and their experiences with each attribute is, we focus on five of the most reviewed tablets for a more detailed Customer Intelligence Analysis.

We use a two-point scale to visualize that difference.

0=unacceptable / 1=experience meets expectations / 2=delighted

Customers “say” that Usability (11.64%), Reliability (11.32%), Quality of Construction (9.5%) and Display (5.4%) are the most important attributes of their experience with the tablets. While all participants are providing Usability experience well above their customer’s expectations, HP TouchPad and Apple iPad 2 are the leaders in this category. Motorola and Toshiba are considered the most “reliable” tablets by their customers, while Samsung Tab continues to struggle.

Here is the access to the tablets’ online marketing research dashboard that allows to see actual customer’s feedback if you click on a specific bar.

The last chart I would like to offer depicts customer affinity for tablet’s operating system and despite all brouhaha about the TouchPad, WebOS still earns the highest score from its customers.

The Playbook introduction caused a lot of controversy well before it was actually released with speculations about its battery life problems wildly spread all over internet. The decision to go with its proprietary operating system was also met with a lot of criticism in social media. However since the Playbook started shipping and customers had a chance to form their own opinion, the results of their opinions analysis did not produce a crystal clear picture yet.  One thing seems to emerge though – it is a lot better product than most people have expected.  This writing is based on the analysis of 886 customer reviews of their experience with these tablets.

Below is a screenshot of the dashboard with a subset of attributes that are most important to customers (statistically determined). The bars represent measurements of the difference between customer expectations and there experience with each attribute.

It appears that Battery life issue was either blown out of proportions by pre-release speculations, or miraculously corrected by RIM engineers before the product started shipping, as the Playbook customers report that it exceeded their expectations by 21%.

QNX operating system appears to be a big hit with the customers as it exceeded their expectations by 35% and scored at much higher customer perception levels than iOS and Android.

What is surprising that overall general satisfaction score from the Playbook customers is quite low in spite the fact the most of the important attribute’s scores are positive.  When I clicked on the general satisfaction bar of the Playbook, the negative opinion snippets (customer insight) was focused around the lack of content and applications for the device.

Word of Mouth (WOM) Analysis

Customers’ perception of mobile phones

(February 24, 2011)

Source Data:

• Social Media – customer generated content found on Amazon, Best Buy, Cnet and other popular consumer sites.

Total of 115 mobile phones from 9 major manufacturers were analyzed for this report.  18,194 customer reviews were located, processed and analyzed to measure difference between customer expectations and their experiences. The measurements are represented by a 2 point scale from 0 (unacceptable) to 2 (delighted), where 1 indicates that customers experienced exactly what they have expected.

As customers continue to buy these products and publish their reviews online, these findings, and conclusions drawn from them, may change over time. Judging by these products’ WoM history, no severe fluctuations in perception are likely within a monthly period.

Methodology:

• Only customer’s generated content is analyzed. Consumers’ opinions, without ownership reference, are not part of this analysis. The customer generated content was located, authenticated, de-duped and aggregated for the analysis.
• The content of the customer reviews is processed by Opinion Miner© software to measure the Customer Satisfaction Index and its components (Functionality, Reliability, Support scores) of the discussed products.
• The findings are organized to facilitate discovery of the products that have negative impact on a Brand Equity, compare Customer Perception of the Brands and specific products that roll up to these Brands.
• Discovery of specific product attributes and customer insights that help to understand why customers selected to purchase this product or passed it up for a competitor are not part of this analysis, but can be performed using Customer Intelligence dashboard.
• Access to verbatim of the analyzed customer reviews can be obtained by following this link and entering the name of the phone associated with these reviews.

Findings

RIM has the highest average Brand reputation score (1.16) which means that Blackberry phones exceed average customer’ expectations by 16.1%, while Nokia and Motorola brands where most disappointing to their customers.

Apple and HTC brands are the only two other brands that on average exceeded their Customer expectations, yet further analysis shows that Samsung Galaxy S phones are among the most reputable products of this segment.

In order to understand the impact of specific products on Brand Equity we have applied additional filters:

1.     All metrics have to be above 1(instead of just CSI) and

2.     Only phones that are currently being sold are being considered.

The results are showing five phones, currently sold, that have the highest average Customer Satisfaction scores.

It is not surprising to find two Blackberry phones in the group, as the RIM is the top Brand when it comes to average customer satisfaction. However the fact that Sony and Samsung phones are among the top 5, yet the Brands have overall low standings, is telling that the “dogs” of these manufacturers are dragging down their Brand Equity.

This begs the question – which Sony and Samsung phones are these “dogs”, that destroy their Brand Equity?

To answer this question we need to apply a different filter that would expose the 5 lowest CSI scored phones from these Brands.

Functionally, only Samsung Glide was really an unsuccessful phone as indicated on the chart below.

A closer look at the Reliability reputation of these phones reveals the reason for customer disappointment.

All of this analysis was performed by using Market Intelligence Dashboard

In the latest twist in the ongoing battle between RIM and Apple for the hearts of customers, RIM managed to retake the No. 1 ranking in Customer Satisfaction

A new study of 14,701 customer reviews spanning 108 mobile phone models revealed that Blackberry phones gained ground by exceeding customer expectations 8% more than Apple iPhones.

The most interesting rivalry is developing between the latest popular offerings: Blackberry 9800 Torch and the iPhone 4-32GB. Torch is opening a significant lead in perception of Reliability and Customer Support. On the other hand, iPhone delights their customers with screen quality, design and software stability.

Analysis Methodology: For this study we aggregated consumer-generated reviews published through November 15, 2010 on multiple popular public sites. This data was analyzed with our proprietary Opinion Miner software. The complete data set of competitive products, services and brands selected by Amplified Analytics’ customers is collected, monitored for updates, processed and analyzed through a paid arrangement. Source Data and attributes analysis for each model is available by request at info@amplifiedanalytics.com

Here is the link to a full copy of this report

As of this date Piplzchoice Award research is read by 15,126 people. In this study we analyzed Mobile Phones. We are currently tracking 98 products in these categories and analyzed 7,529 reviews written by their customers. However some of these products have not accumulated enough reviews to produce statistically representative and accurate metrics, so we filtered them out of the competition. The second round disqualified any product that failed to meet Customer Expectations with its Functionality, Reliability or Support.

Blackberry Torch 9800 is a newcomer that quickly captured the top spot for it’s Functionality and Reliability reputation.

RIM Blackberry Torch 9800
2010 Piplzchoice Award winner
59% above average Customer Satisfaction in its Category
The winners are chosen by their customers

This Phone’ Reliability score is 42% higher than the category average. See mobile phone brands analysis study below. For the free full copy of this study contact greg@amplifiedanalytics.com

Mobile Phone Brands Analysis

We keep working on new methodologies for measuring impact of Word of Mouth on Brand equity. Only a few quarters ago CE manufacturers named their laptops and netbooks as two very separate categories of products, however after release of iPad this distinction started to melt away. This is the analysis of 25,4169 customer reviews of 232 laptop and netbook computers from 9 brands. The measurements represent difference between customer expectations and customer experiences using 2 point scale

The assessment was administered on reviews published before September 28, 2010. The chart below represents Customer Satisfaction Indexes for these brands.

• As all brands display a very close, above 1, mean value which makes it safe to conclude that most customers had their expectations exceeded.
• All brands have “negative” outliers, but their number is small
• Samsung emerges as a leader with a large majority of it’s customers are highly positive about their experience – 50% are very concentrated at higher than average values
• Acer and Apple display a “long tail” of disappointed customers, while Sony has the weakest brand satisfaction ratings.

Let me know if you want to see the results for Functionality, Reliability and Support reputation analysis and I will email it to you.