Home > Puget Systems Blog > How accurate is ResellerRatings.com?

How accurate is ResellerRatings.com?

Jon Bach (President)

How accurate is ResellerRatings.com?

Posted on March 16, 2009 by Jon Bach

[ View All Blog Posts ]


The website ResellerRatings.com has been around for a very long time, and is the de facto standard for checking out just about any online retailer in the industry. It provides a place for unbiased reviews of companies by people who have purchased in the past. However, in the last few years it has become increasingly easy to get a 9/10 score or better. Can the scores still be trusted, or has ResellerRatings.com become nothing more than a marketing platform? If so, how is that possible if the reviews cannot be affected by the companies being reviewed?

I very much like the idea of aggregate rating systems. I trust the wisdom of a crowd more than I do any individual, provided the data is properly compiled. For example, I have always found it odd for someone to have a brand loyalty based on their own experience only.

Person: I always use Brand-X memory, it's definitely the best in my experience.
Me: How many computers have you built with it?
Person: Three, and I've never had a single failure.


Why would this person base his decision on such a small set of experiences when he can look at reliability statistics based on thousands of computers? I believe it is the same reason why people buy based on word of mouth alone. A small set of personal experiences are more real to us than "distant" data, even if it is far more accurate. While I understand this phenomenon as human nature, I still find it odd!

ResellerRatings.com represents a great concept that provides both aggregate data, and personal reviews. I've personally followed it for over 10 years now, and have watched it change over time. As a system builder, we use it quite heavily as a trusted place to send our customers who want unbiased information about our company. I've been frustrated in the last year by reports of some of our competitors "gaming" the system. Companies can try to post fake reviews, but that's easily caught (I believe). But what if they are providing incentives for good reviews? Or even harder to track, what if they are only asking already happy customers to provide feedback? While these things are obviously against the rules at ResellerRatings.com, they can be difficult, if not impossible to enforce. It is frustrating to me, to think that other companies could be cheating their way into the same scores that we earn rightfully. It undermines the very purpose of the reviews, and takes away from the credability of ResellerRatings.com.

One other important way of determining the quality of a company is to take a look at BBB reports. A large number of complaints filed with the BBB is a strong indicator of a company that doesn't care. They typically represent a situation that has gone horribly wrong, and even after discussion, the customer still feels that the company is not being fair. I've been in only a few of these situations over the years, and they're very unpleasant. If a company has a large number of BBB complaints, they should also have a poor score at ResellerRatings.com, right? I decided to take a snapshot of our industry, to see if there is indeed a correlation between the two.

  BBB Complaints ResellerRatings
  36 months 12 months Lifetime 6 months
Alienware 853 130 7.76 9.36
AVA Direct 7 7 9.47 9.32
CyberPower 480 ??? 7.51 8.31
Dell 11630 4589 4.05 2.45
Digital Storm 14 6 8.96 9.67
IBuyPower 256 ??? 7.13 8.44
Puget Systems 3 1 9.9 9.71
Velocity Micro 24 7 8.48 8.83
Vigor Gaming 6 ??? 9.61 9.89


I plotted this data on a graph, showing both the number of complaints at the BBB, and the DISTANCE from a perfect score at ResellerRatings.com. For example, since Puget Systems holds a 9.9 lifetime score, our value would be 0.1. At first, the results were hard to see. I changed to the "lifetime" scores for the company, since that's a number that is not as easily skewed by any recent events. I also changed the plot to a logarithmic scale, and was amazed at the correlation!

This is great news for trusting the scores at ResellerRatings.com (but only the lifetime scores). We have a second set of data, which lines up perfectly. But take note of the logarithmic scale -- this is very important! In this case, every line is 10 times the value of the one before it. What does this tell us? It tells us that even VERY slight differences in ResellerRatings.com score are very important. A score in the low to mid 9's is a big difference from a score in the upper 9's.

I do think ResellerRatings.com has some work they need to do. Even if you're not cheating, it is far too easy to obtain a 9/10 score, and it is far too easy for recent reviews to skew the score shown. If they don't address this soon, it will begin to impact the lifetime scores. For now, the lifetime score is a much better indicator, unless you truly believe that something in the company changed in the last 6 months.

Are there other ways you objectively research the quality of a company? I'd love to hear about it!



Share this blog post!

Paul

What about dividing units sold by BBB complaints to get an average "complaint per unit sold"? Then you would not have to use the log graphs...

Posted on 2009-03-17 17:55:16

@Paul

I thought about that, but there were two problems: 1) I could not find units sold for many, if not most of the companies. 2) I suspect that that type of comparison wouldn't be valid if you're comparing different types of companies. For example, Puget vs Dell. On the one hand, Dell has a full-out assembly line going and likely has less problems per unit built. On the other hand, they're large enough that if there is a problem, I bet customers are more likely to go to the BBB more quickly, simply because they have no voice.

You are right though, there is a component of "per unit sold" that should be considered here. I just worry that it opens up more opportunity for subjective interpretation.

Posted on 2009-03-17 18:00:36
Paul

True enough.

I vaguely remember my college days and there were many many techniques to generate valid statistical surveys (eliminating cross correlation or some such thing!)

Anyway, you make good stuff, mine still going strong! I had an occasion to open the case to add another drive, and you guys did some super cabling work.

Posted on 2009-03-18 23:56:03

Jon,

I appreciate your analysis but I think it is flawed because it omits an important aspect of our site that only came about in the past 9 months.

Our launch of the Merchant Member program changed the game. Merchants are now working harder than ever to resolve customer complaints at ResellerRatings. What would be a bad review in the past (or a negative BBB complaint), is now getting posted as a bad review but then later edited by the reviewer into a good review after the merchant has resolved their complaint.

We do this by:
-Listing and featuring our parcipating merchant members on our homepage
-Creating a merchant member list page, which ranks merchant members by how active they are at ResellerRatings
-Our 48 hour preview system which gives merchants 48 hours to see reviews before the public, incentivizing them to resolve complaints quickly.
-The ability to view invoice numbers, contact reviewers, resolve issues.
-The participation badge on reviews pages: this green checkmark indicates a merchant's participation, or a red exclamation point indicates that they don't participate.

So, merchants are now working hard to resolve customer complaints at ResellerRatings. This will skew the score upwards for participating merchants, but that's a good thing. It tells consumers that by using our site, they can get resolution to their complaints, and it rewards merchants with good ratings for doing what they should be doing - providing good quality service.

Scott Wainner
Founder & CEO
ResellerRatings.com

Posted on 2009-03-22 23:31:03

Thanks Scott! I appreciate you taking the time to comment. In fact, I've always appreciated how you are open to discussion. Keeping ResellerRatings.com both fair, and attractive to merchants is no easy task, for sure.

The ability for customers to update their reviews is for sure a good thing. It keeps companies engaged at settling for no less than complete satisfaction, and that's a win for everyone. At the core, I think that ResellerRatings.com serves two purposes:

1) Provide information about companies to potential customers
2) Provide an avenue of recourse for existing customers.

I think that your points above are great when it comes to purpose #2. Interestingly, it works almost a little too well :) I have personally found that once you have achieved a 10/10 score, you become a target for customers that haven't gotten everything they wanted (even if you've done nothing wrong). But that's fine, and that's healthy.

However, I would argue that keeping some record of the original rating would be useful when a customer updates a review. This is because when it comes to purpose #1, the ability to manipulate past reviews is a tool that can be abused in the hands of a dishonest company. Essentially, they can "pay off" their customers to change their review. I think it would be useful to see how many reviews are getting updated for a company, even if it doesn't factor into the original score. After all, its better to not do something wrong in the first place, so companies should be rewarded for a lack of bad reviews in the first place.

This difference alone could account for the difference in severity between ResellerRatings.com scores and BBB complaints. A BBB complaint never goes away, even if the issue is resolved. A bad review can be erased entirely, and I'm not convinced that's best for the consumer.

No matter what the case though, keep up the good work! As one of the highest rated companies on your website, you most certainly have our support, even when we have some constructive criticism.

Posted on 2009-03-23 03:41:07
RIchard

Jon
As a new customer to your company, I'm looking forward to the continued excellent experience I've found so far. It's been very good, and I haven't even received the product yet. With my long years of extensive business background, both in large corporate and most independently owned environments, I've encountered the entire scale of business reliability and support. It's only in recent years that the advantages of sites like ResellerRatings.com have been available to provide the broad concensus of public experience, instead of having to ferret out reliable sources independently.

But as with any user feedback system, your concerns of manipulation are very valid, and appropriately so, since you have worked diligently to achieve your current positive rating. It's good to see that Scott has addressed this with the addition of the ability to update a negative review with a positive response if the problem is appropriately solved. However, again the issue of manipulation can be appropriately raised as you have done.

One way to address this would be to install two tracking methods. The first would be as before, with only the initial complaints tracked and not updates available. The second would be to track the resolution of those negative complaints. This way the viewer would be able to see initial performance statistics, and then the followup service statistics and gauge one against the other.

From a company standpoint, this also would provide not only an incentive to solve problems once they occur, but also to correct the cause and create a more problem free environment so they don't occur in the first place!

Posted on 2009-04-06 18:59:25
Mark Feldman

Jon,

I was surprised to see your analysis of reseller ratings which I came across while goggling reseller ratings (the way I get to every web site these days ) What surprised me is that I have been looking for a new computer by a higher end manufacturer then the Dell & Micron I have. I look periodically and have not read great things about Alienware post merger and definitely not that good when it comes to Cyberpower and ibuypower , which I believe are the same company. I would venture to say that it is unexplainable that Alienware would have a higher rating then Velocity based on what I have read on pretty much all the computer websites. I had noticed that all 3 are now getting remarkably higher scores the last 6 months which I attributed solely to gaming the system. I have not heard of your company, but it’s nice to know there is another boutique company out there. Thanks for sticking your neck out and tackling a topic that needs to be discussed as reseller ratings is indeed the number one source for checking on the viability of a company for those in the know.

Regards,

Mark

Posted on 2009-09-04 02:44:55
Jim

Nice post. www.pugetsystems.com is amazing.

Posted on 2010-03-08 16:03:50

My webstore is a little over a year old and I am starting to get some reviews on Reseller Ratings. So far so good but what happens when someone makes a complaint about my store and I am not a paying member? Do you offer a free membership that can address complains or do I have to pay to play???

Posted on 2010-03-20 16:41:38

This Web site is terrible. The only way to have good FB on this site is to promote it. They essentially hold your business hostage until you work for them by telling your customers to leave feedback there. Take a look at Best Buy for example they don't participate and have a 2 FB. We all know that is BS. If only 2 out of 10 transactions went well they would have been out of business the 1st day they opened.

Posted on 2010-12-22 22:56:23

Lawson, Matt -- You guys realize that THIS website is not ResellerRatings, right? I just wrote about them.

Posted on 2010-12-22 23:37:02
Hung Lui

I usually do my shopping with bigger online companies, but decided to take a chance to shop from small website. I used ResellerRatings to review the site. The small company had over 700 ratings mostly 5. But I had such a bad experience I couldn't believe the ratings. Only looking more closely, the reviews were posted daily for the last couple of months. I began to suspect the posts are inaccurate because I also learned the company is actually one little store. There were so many good reviews you couldn't see the more accurate bad reviews. I've also find the same type of posts on other review websites. Hundreds of good reviews on multiple sites for one store. However, the bad reviews are all similar to the experience I had. Are there actually companies now that sell ways to fake reviews on reputable sites like Resellerratings? I also don't believe that one little store can write all those good reviews on their own.

Posted on 2011-02-11 14:19:37
Youdontneedthis

I'll tell you what I don't like about resellerratings dot com. I don't like them automatically creating an account for me, simply because I leave a review for a store I bought from.
I'm sick and tired of arrogant companies deciding for me what I want. Nowhere do they ask if you would like an account created. They just tell you that one will be created.
And they don't offer an option to close your account either.

I left a review of a store because I thought I was doing them a good turn, but thanks to pushy companies like reseller ratings, I won't be leaving reviews for stores anymore.

Posted on 2011-10-04 22:47:23
Ratingsarenotaccurate

I agree with your complaint.  Another big issue, (in my opinion), is that happy customers don't typically leave reviews therefore the ratings are skewed.  I don't even pay attention to reviews unless the store has tons of them, then I might skeptically read them.

Posted on 2012-02-13 21:21:29
Rrisgod2011

Warning!  Test of 15 years of research and development from resellerrattings.com's ratting system.  This only a test, result of the test should be determined by customers and merchants.  Any one can do this same test and judge the result on their own.Keep in mind all reviews are posted on one email account, one computer, same IP address unchanged and no reboot of the computer system.  And yes we know that RR takes up to 48hrs to post reviews, and give merchant that has an membership the chance to dispute or fix any issue they have with customer before the reviews are posted.1.  Test #1:  Posted 3 5/5 star positive review on 3 merchant that are not member of the resellerrattings.com and has an over all average review of 1/5 stars.  After 5 days none of the review showed up, and only email that was sent by RR was your ratting will take 48hrs to show up...(Some will say, you see the system is working!  Please wait for the second test...)2.  Test #2:  Posted on the same email address after the fist 3 good reviews.  The next 3 reviews are done on new merchant that has 0 ratings or has an average review of 1/5 stars and are still not members of resellerrattings.com.  Those 3 reviews posted are negative with 1/5 stars.  Result was all 3 review showed up almost instantly if not less then 24hrs.  Again this is done on the same computer with same email address and same IP address as the first 3 review.(Surprised?  This was expected before the test was done)3.  Test #3:  Posted 4 review this time on merchant that is an member of resellerrattings.com.  2 negative, 2 positive.  After 48hrs, the 2 positive reviews showed up and the 2 negative review did not.  No email was sent by resellersrattings.com till this day for those reviews.(... use your own judgment on this one  :P)The result seem that merchants that are not members will build up negative reviews much faster then positive reviews and are unable to fix or respond to those reviews until they become a member.  In which RR will then do its job and sent out emails to fix any unreal ratings.We will keep doing those type of test and build up an database with review date, email, and results for public viewing."Please keep in min this is only a test!"  And should not be used to determine any claim or actions.

Posted on 2011-10-11 08:50:17
winterband

Just look at how the reputation pirates deal with merchants who refuse to pay their tribute as if the merchant did not care about their customers.  Reseller ratings are scum.

http://urgentwarnings.blogspot...

Posted on 2012-02-13 21:42:09
Guest

Resellerratings.com is an extortion company and has become a national spttoon. They allow anonimous unsubstantiated reviews posted by whoever, including competitors. No invoice number is required. Anonomity is a key - there is no responsibility for reviewer and it is free. You do not offer free shipping? - I will drop a bomb on you on resellerratings.com. They exploit freedom of speech, which is converted to a freedom to smear. Seller even can not file a lawsuit against slanderer due to anonimity. Resellerratings.com also makes sure all negative posts are showing first on the list and positive ones are delayed or not listed at all. Query on company name shows negative lies about the company on the first page. They defend themselves fiercely on all forums, trying to prove they bring some value to shoppers. What a disgraceful business!

Posted on 2012-04-04 15:33:16

Resellerrating is pure scam. They are engaged in negative and unethical marketing practices. They call you/ send emails to you to get enrolled in their paid membership program since your company has started getting some reviews.

When you deny them for their paid membership they will start deleting your positive reviews living back only the negative ones.

I wonder why Google rank them high and even consider ratings left on this site? ?

Posted on 2013-04-08 08:10:19
ROGER

SITES LIKE RESELLER RATING USE THE SAME EXTORTION PAY TO PLAY AS BBB DOES SEE 60 MINUTES REPORTS ON HOW THEY MANIPULATE RATINGS - THAT'S WHY THE GRAPH ABOVE LOOKS THE SAME

Posted on 2013-04-16 16:33:34
See a problem on this page? Let us know.