Your old smartphone contract is about to run out, and you’re in the mood to shop around. You start by reading a few expert reviews. They provide tables with benchmarks, list advanced features, and showcase “low light” camera shots of bobblehead dolls or coffee mugs. One review even compares a phone’s design to the “elegant curves” of a famous actress.
Just as you’ve begun daydreaming about holding the actress phone yourself, a user comment snaps your attention back to the screen. “HORRIBLE battery life!” it reads, directly contradicting a point made by the expert. Concerned, you start reading dozens of user reviews, where you find hapless customers reporting “tragically low” talk time, and customer service calls that “had me stuck on the phone for two hours.”
So, who to trust? Experts or users? A month ago, we featured the top smartphones by Smart Rating, a mix of expert review scores, features, and technical specifications. But what about the experience of the average user? At FindTheBest, we compared our Smart Ratings against user ratings across several different smartphone categories. To begin, here’s a graph comparing average user ratings (blue) with Smart Ratings (red). (Smart Ratings have been normalized to a 1 to 5 scale.)
Notably, Sony phones have the greatest disparity between scores—users rate Sony handsets nearly a full point higher (+0.81) than the experts. Meanwhile, Apple‘s scores skew the opposite direction—users tend to rate iPhones 0.70 points lower than the experts. What accounts for the difference?
“I think a lot of professional reviews pay a lot closer attention to industrial design,” said Michael Gorman, a senior associate editor at Engadget. “We’re looking for a way to differentiate these handsets. The user experience is not drastically different between flagship phones. You know, everybody’s using the same Gmail app.” Instead of price or common apps, explained Gorman, experts tend to look more closely at things like hardware design or image quality in extreme lighting conditions, things of comparatively less importance to casual consumers. “Reviewers wax poetic about industrial design…other people are just worried about, ‘am I going to get little dings and dents and is it going to break when I drop it?'”
Still, Apple’s lower average user review scores may simply be a curse that comes with success. “Apple’s just such a big name in the industry they’ve become more of a lightning rod as a result,” said Gorman. Case in point: Apple phones receive disproportionately more 1-star user ratings than any of their rivals. For every three extremely positive user ratings, Apple receives one angry 1-star review, with a user condemning Apple’s “walled-garden” philosophy or “inadequate screen size.” Sony has far fewer highly-vocal detractors.
Meanwhile, reviews for Sony’s popular Xperia line showcase the different mindsets of users and experts. For the flagship Xperia Z, users compliment the “great screen” and “superb camera,” while experts observe the “camera is slow” (CNET), the design too “boxy” (Laptop Mag), the processor “dated” (PC Mag, CNET), and the viewing angles too narrow (Laptop Mag, GoodGearGuide). Several experts also report that the chassis (expert speak for the outer framework of the phone) attracts fingerprints too easily. For the tech geek, these are entirely relevant details. For most consumers, none of these critiques really matter. (See Xperia Z expert and user reviews side-by-side…under the “Reviews” section.)
Sony’s popular Xperia line
Next, let’s look at mobile operating systems. How do experts and users compare here?
Predictably, we see the same trend with iOS that we saw for Apple above: high expert scores next to lower user scores. This time, however, it’s BlackBerry that features notably higher scores from users. Browse user reviews for BlackBerry’s most popular phones, and you’ll find a devoted user base that loves BlackBerry keyboards (whether the physical keyboard of the older Curves or the virtual keyboard of the Z10) while celebrating BlackBerry’s business-friendly interface. Flip over to expert reviews, however, and you’ll see an obvious theme emerge: experts can’t wholly recommend BlackBerry-based phones next to Android and iOS, citing the lower quality and quantity of apps. Still, these professional complaints might not matter for a certain subset of business users. The longtime BlackBerry loyalist doesn’t know—and doesn’t care—about the apps he or she is missing out on.
The BlackBerry Z10
One other note on operating systems: even if Microsoft has failed to win a significant portion of the market, experts and users tend to give Windows phones high marks. In fact, Windows emerged as the highest user-rated OS of the big four.
Users and experts both love top Windows phones
Finally, let’s take a look at carriers. This comparison is a bit trickier than the others, especially given that phones are judged primarily on design, features, and general operation, and (typically) less so on call performance and signal strength. Still, we set aside four groups of phones, each comprised of handsets exclusive to their respective carrier. We then performed the same comparison of user ratings and expert ratings.
Verizon-only phones squeak by with the gold medal from experts, but intriguingly, T-Mobile-only phones win among users. Whether the results here have most to do with customer service, reception quality, location (a large number of tech writers live in New York or the Bay Area; users live everywhere), or exclusive deals (the user-favorite Sony Xperia Z is exclusive to T-Mobile) is anyone’s guess.
– – – –
We asked around for a few more perspectives on expert vs. user reviews. Here’s what we heard.
Pro Expert
“When you’re talking about battery life, when you look at an expert review, they’ve outlined exactly what they’ve done to test the battery life and explain what they did to test it and how long it lasted. When you see user reviews, people will just say, ‘yeah I had no problem using it all day it’s been great.’ But you don’t know how they use it, if your use case is similar to theirs.” – Michael Gorman, Engadget
Pro User
“Most recently the Galaxy S4’s smart pause received a lot of buzz from both experts and social media (mostly driven by Samsung) but our analysis of consumers found that less than 5% even mention it and of those over half are disappointed with the feature, either because of poor performance or significantly reduced battery life.” – John Feland, CEO of Argus Insights
“User reviews provide much more information on the long-term experience of using a gadget—how build quality held up over time, how easy it was to engage tech support, and so on. Journalists often put far, far too much emphasis on smartphone benchmark performance. In most cases, there’s only one smartphone component that warrants empirical lab testing, and that’s the battery.” – Jon Phillips, editor-in-chief of TechHive and PCWorld
A Mix of Both
“When looking to buy a new smartphone consumers should really ask themselves one question to guide their purchase research: ‘Do I typically ask people to help me with my smartphone or do I typically help others with their smartphones?’ Reading reviews from other consumers….will help [new users] avoid painful experiences like ‘group text messaging doesn’t work right’ or ‘contacts and music/pictures/videos don’t transfer correctly.’ For the smartphone-savvy consumers, however, professional reviews will likely help [more].” – Todd Day, Frost & Sullivan senior industry analyst.
“As far as meaningful metrics when shopping for a phone, I think consumers should enlist the full marketplace of information. Find a few tech sites that speak to them on a meaningful level. The smaller tech blogs might get into the weeds of component minutia, but more lifestyle-focused tech sites—like TechHive and Wired and Wirecutter—will do a better job of making spec and component information relevant and meaningful.” – Jon Phillips
This article was written for TIME by Ben Taylor of FindTheBest.