 | How accurate is ResellerRatings.com?March 16, 2009 – 4:24 pm by Jon Bach |
The website ResellerRatings.com has been around for a very long time, and is the de facto standard for checking out just about any online retailer in the industry. It provides a place for unbiased reviews of companies by people who have purchased in the past. However, in the last few years it has become increasingly easy to get a 9/10 score or better. Can the scores still be trusted, or has ResellerRatings.com become nothing more than a marketing platform? If so, how is that possible if the reviews cannot be affected by the companies being reviewed?
I very much like the idea of aggregate rating systems. I trust the wisdom of a crowd more than I do any individual, provided the data is properly compiled. For example, I have always found it odd for someone to have a brand loyalty based on their own experience only.
| Person: | I always use Brand-X memory, it’s definitely the best in my experience. |
| Me: | How many computers have you built with it? |
| Person: | Three, and I’ve never had a single failure. |
Why would this person base his decision on such a small set of experiences when he can look at reliability statistics based on thousands of computers? I believe it is the same reason why people buy based on word of mouth alone. A small set of personal experiences are more real to us than “distant” data, even if it is far more accurate. While I understand this phenomenon as human nature, I still find it odd!
ResellerRatings.com represents a great concept that provides both aggregate data, and personal reviews. I’ve personally followed it for over 10 years now, and have watched it change over time. As a system builder, we use it quite heavily as a trusted place to send our customers who want unbiased information about our company. I’ve been frustrated in the last year by reports of some of our competitors “gaming” the system. Companies can try to post fake reviews, but that’s easily caught (I believe). But what if they are providing incentives for good reviews? Or even harder to track, what if they are only asking already happy customers to provide feedback? While these things are obviously against the rules at ResellerRatings.com, they can be difficult, if not impossible to enforce. It is frustrating to me, to think that other companies could be cheating their way into the same scores that we earn rightfully. It undermines the very purpose of the reviews, and takes away from the credability of ResellerRatings.com.
One other important way of determining the quality of a company is to take a look at BBB reports. A large number of complaints filed with the BBB is a strong indicator of a company that doesn’t care. They typically represent a situation that has gone horribly wrong, and even after discussion, the customer still feels that the company is not being fair. I’ve been in only a few of these situations over the years, and they’re very unpleasant. If a company has a large number of BBB complaints, they should also have a poor score at ResellerRatings.com, right? I decided to take a snapshot of our industry, to see if there is indeed a correlation between the two.
| BBB Complaints | ResellerRatings | |||
| 36 months | 12 months | Lifetime | 6 months | |
| Alienware | 853 | 130 | 7.76 | 9.36 |
| AVA Direct | 7 | 7 | 9.47 | 9.32 |
| CyberPower | 480 | ??? | 7.51 | 8.31 |
| Dell | 11630 | 4589 | 4.05 | 2.45 |
| Digital Storm | 14 | 6 | 8.96 | 9.67 |
| IBuyPower | 256 | ??? | 7.13 | 8.44 |
| Puget Systems | 3 | 1 | 9.9 | 9.71 |
| Velocity Micro | 24 | 7 | 8.48 | 8.83 |
| Vigor Gaming | 6 | ??? | 9.61 | 9.89 |
I plotted this data on a graph, showing both the number of complaints at the BBB, and the DISTANCE from a perfect score at ResellerRatings.com. For example, since Puget Systems holds a 9.9 lifetime score, our value would be 0.1. At first, the results were hard to see. I changed to the “lifetime” scores for the company, since that’s a number that is not as easily skewed by any recent events. I also changed the plot to a logarithmic scale, and was amazed at the correlation!
This is great news for trusting the scores at ResellerRatings.com (but only the lifetime scores). We have a second set of data, which lines up perfectly. But take note of the logarithmic scale — this is very important! In this case, every line is 10 times the value of the one before it. What does this tell us? It tells us that even VERY slight differences in ResellerRatings.com score are very important. A score in the low to mid 9’s is a big difference from a score in the upper 9’s.
I do think ResellerRatings.com has some work they need to do. Even if you’re not cheating, it is far too easy to obtain a 9/10 score, and it is far too easy for recent reviews to skew the score shown. If they don’t address this soon, it will begin to impact the lifetime scores. For now, the lifetime score is a much better indicator, unless you truly believe that something in the company changed in the last 6 months.
Are there other ways you objectively research the quality of a company? I’d love to hear about it!
Share this blog post!
9 Responses to “How accurate is ResellerRatings.com?”
What about dividing units sold by BBB complaints to get an average “complaint per unit sold”? Then you would not have to use the log graphs…
By Paul on Mar 17, 2009
@Paul
I thought about that, but there were two problems: 1) I could not find units sold for many, if not most of the companies. 2) I suspect that that type of comparison wouldn’t be valid if you’re comparing different types of companies. For example, Puget vs Dell. On the one hand, Dell has a full-out assembly line going and likely has less problems per unit built. On the other hand, they’re large enough that if there is a problem, I bet customers are more likely to go to the BBB more quickly, simply because they have no voice.
You are right though, there is a component of “per unit sold” that should be considered here. I just worry that it opens up more opportunity for subjective interpretation.
By Jon Bach on Mar 17, 2009
True enough.
I vaguely remember my college days and there were many many techniques to generate valid statistical surveys (eliminating cross correlation or some such thing!)
Anyway, you make good stuff, mine still going strong! I had an occasion to open the case to add another drive, and you guys did some super cabling work.
By Paul on Mar 18, 2009
Jon,
I appreciate your analysis but I think it is flawed because it omits an important aspect of our site that only came about in the past 9 months.
Our launch of the Merchant Member program changed the game. Merchants are now working harder than ever to resolve customer complaints at ResellerRatings. What would be a bad review in the past (or a negative BBB complaint), is now getting posted as a bad review but then later edited by the reviewer into a good review after the merchant has resolved their complaint.
We do this by:
-Listing and featuring our parcipating merchant members on our homepage
-Creating a merchant member list page, which ranks merchant members by how active they are at ResellerRatings
-Our 48 hour preview system which gives merchants 48 hours to see reviews before the public, incentivizing them to resolve complaints quickly.
-The ability to view invoice numbers, contact reviewers, resolve issues.
-The participation badge on reviews pages: this green checkmark indicates a merchant’s participation, or a red exclamation point indicates that they don’t participate.
So, merchants are now working hard to resolve customer complaints at ResellerRatings. This will skew the score upwards for participating merchants, but that’s a good thing. It tells consumers that by using our site, they can get resolution to their complaints, and it rewards merchants with good ratings for doing what they should be doing – providing good quality service.
Scott Wainner
Founder & CEO
ResellerRatings.com
By Scott Wainner on Mar 22, 2009
Thanks Scott! I appreciate you taking the time to comment. In fact, I’ve always appreciated how you are open to discussion. Keeping ResellerRatings.com both fair, and attractive to merchants is no easy task, for sure.
The ability for customers to update their reviews is for sure a good thing. It keeps companies engaged at settling for no less than complete satisfaction, and that’s a win for everyone. At the core, I think that ResellerRatings.com serves two purposes:
1) Provide information about companies to potential customers
2) Provide an avenue of recourse for existing customers.
I think that your points above are great when it comes to purpose #2. Interestingly, it works almost a little too well
I have personally found that once you have achieved a 10/10 score, you become a target for customers that haven’t gotten everything they wanted (even if you’ve done nothing wrong). But that’s fine, and that’s healthy.
However, I would argue that keeping some record of the original rating would be useful when a customer updates a review. This is because when it comes to purpose #1, the ability to manipulate past reviews is a tool that can be abused in the hands of a dishonest company. Essentially, they can “pay off” their customers to change their review. I think it would be useful to see how many reviews are getting updated for a company, even if it doesn’t factor into the original score. After all, its better to not do something wrong in the first place, so companies should be rewarded for a lack of bad reviews in the first place.
This difference alone could account for the difference in severity between ResellerRatings.com scores and BBB complaints. A BBB complaint never goes away, even if the issue is resolved. A bad review can be erased entirely, and I’m not convinced that’s best for the consumer.
No matter what the case though, keep up the good work! As one of the highest rated companies on your website, you most certainly have our support, even when we have some constructive criticism.
By Jon Bach on Mar 22, 2009
Jon
As a new customer to your company, I’m looking forward to the continued excellent experience I’ve found so far. It’s been very good, and I haven’t even received the product yet. With my long years of extensive business background, both in large corporate and most independently owned environments, I’ve encountered the entire scale of business reliability and support. It’s only in recent years that the advantages of sites like ResellerRatings.com have been available to provide the broad concensus of public experience, instead of having to ferret out reliable sources independently.
But as with any user feedback system, your concerns of manipulation are very valid, and appropriately so, since you have worked diligently to achieve your current positive rating. It’s good to see that Scott has addressed this with the addition of the ability to update a negative review with a positive response if the problem is appropriately solved. However, again the issue of manipulation can be appropriately raised as you have done.
One way to address this would be to install two tracking methods. The first would be as before, with only the initial complaints tracked and not updates available. The second would be to track the resolution of those negative complaints. This way the viewer would be able to see initial performance statistics, and then the followup service statistics and gauge one against the other.
From a company standpoint, this also would provide not only an incentive to solve problems once they occur, but also to correct the cause and create a more problem free environment so they don’t occur in the first place!
By RIchard on Apr 6, 2009
Jon,
I was surprised to see your analysis of reseller ratings which I came across while goggling reseller ratings (the way I get to every web site these days ) What surprised me is that I have been looking for a new computer by a higher end manufacturer then the Dell & Micron I have. I look periodically and have not read great things about Alienware post merger and definitely not that good when it comes to Cyberpower and ibuypower , which I believe are the same company. I would venture to say that it is unexplainable that Alienware would have a higher rating then Velocity based on what I have read on pretty much all the computer websites. I had noticed that all 3 are now getting remarkably higher scores the last 6 months which I attributed solely to gaming the system. I have not heard of your company, but it’s nice to know there is another boutique company out there. Thanks for sticking your neck out and tackling a topic that needs to be discussed as reseller ratings is indeed the number one source for checking on the viability of a company for those in the know.
Regards,
Mark
By Mark Feldman on Sep 3, 2009
Nice post. http://www.pugetsystems.com is amazing.
By Jim on Mar 8, 2010
My webstore is a little over a year old and I am starting to get some reviews on Reseller Ratings. So far so good but what happens when someone makes a complaint about my store and I am not a paying member? Do you offer a free membership that can address complains or do I have to pay to play???
By Lawson on Mar 20, 2010