Ubuntu Software Center

Are ratings calculated by a weighted average?

Asked by Ingo Gerth

Recently I have seen many apps on the software centre which seem to be rated rather poorly and score only few stars (3 or less). However, when clicking on the app, the most helpful comments are actually comments with a very good rating (4 stars or more).

This makes me wonder on the accuracy of how the ratings are calculated, as they currently seem to give the wrong picture. A few questions:

* To start with, is there any weighting at all, or how is the average computed?
* Do ratings of older versions get less weight, or are they removed from the statistics alltogether? This would make sense since oftentimes 1 star is given when an app doesn't work, but it might work at a later point in time. The rating would not be fair anymore.
* Are ratings weighted according to helpful comments? This could make them more accurate.
* Do ratings expire after a while?

One website that commputes a weighted average is IMDB. They have a very nice average. We should make ratings fair for app developers, so weighting might be an idea for Ubuntu as well (if it is not already there, in that case I apologize for the stupid question).

Thanks in advance,
Ingo

Question information

Language:
English Edit question
Status:
Solved
For:
Ubuntu Software Center Edit question
Assignee:
No assignee Edit question
Solved by:
Ingo Gerth
Solved:
Last query:
Last reply:
Revision history for this message
Aaron Peachey (aaronp) said : 		#1

Hi Ingo,
Currently the rating shown for each application is simply an average of the ratings that have been given to the application. Taking a quick look at the code, I can't see anywhere where this is dependent on previous versions and it appears to simply be an average of the ratings given to the app.

If you would like to visit the following spec page
https://wiki.ubuntu.com/SoftwareCenter/RatingsAndReviews#Presenting_reviews
you can see that there probably needs to be some discussion or suggestions on a solution to the way ratings are averaged/weighted - are you in a position to provide some mathematical alternatives? The weighting that IMDB uses appears to be a secret (to stop people from finding out how to stack votes on their work) so we would need to come up with something else that works.

In our current top-rated we do use a dampened rating which attempts to weight the scores in a 'fairer' manner by determining the true score rather than an average. In essence it stops apps with only a handful of ratings from achieving the same score as apps which may have the same raw average but with alot of ratings, since having alot of ratings is a more reliable measure.
If I remember rightly, this was the inspiration for the formula currently used to determine top rated:

http://evanmiller.org/how-not-to-sort-by-average-rating.html

Revision history for this message
Ingo Gerth (igerth) said : 		#2

Hi Aaron,

thanks a lot for the answer! The article is a very good one and as it shows there is a need for a smarter way to do rating, which applies to all ratings I think. Some of the issues mentioned in my question are particular to our case, so they need special attention in my opinion.

Unfortunately I'm not anywhere near an expert in statistics, so I can not provide mathematical alternatives. But I am sure we could find some skilled people in the community. And some of the issues, like transitory ratings between versions, are less mathematical issues that just need a proper discussion to come to a good approach I think.

So if you agree there might be a need to do some improvements to the ratings, I'd be glad to be involved in the discussion, but unfortunately can not give much mathematicl insight.

Thanks,
Ingo