A good algorithm for ranking gems

Question

I built a site that let's you freely add Rails-related gems, and tag those gems. The next feature I'd like to add is gem ranking. I've considered some of the common systems, like stars and up-voting, but have decided a simple system won't work well enough.

Here are a couple scenario's I want the ranking algorithm to work well for:

• There is a popular gem named GEM1 that does XYZ. GEM1 has been released for a year and ranks the highest among gems that do XYZ. A new gem, GEM2, that also does XYZ, is released and is clearly better than GEM1. GEM2 should be able to outrank GEM1 quickly.
• There is a high-ranking gem named GEM1. A new version of Rails is released and GEM1 doesn't work for this version. As a reasonable amount of time goes by (prob the same amount as it would take for the majority of people to start using the new Rails version) and GEM1 still doesn't support this version, it should lose its high ranking.

Some info we have available to us for gems are:

• number of downloads
• number of downloads for the current version
• if the gem is hosted on github, we have number of forks and watchers (although a gem shouldn't outrank another gem b/c it's hosted on github)

This info could be part of the algorithm (along with up-votes or stars or whatever we need to collect from the users).

So my question is, can you think of a good algorithm (not exact of course, just describe it) that could handle these scenarios, or at least be able to offer some good advice, or point me in the right direction.

Answer 1

http://ruby-toolbox.com/categories.html is an excellent site that does something similar. They simply rank on downloads, but it is not always true that the most downloaded one is the best IMO. Activity on github etc should be shown because I am not likely to try something new that is not actively being developed.

I would think that downloads should count for something and perhaps decay on a logarithmic scale. development activity should count too. In addition you can add user reviews and incorporate that too. I think that these metrics can be tweaked with certain weights to see what would rank things most appropriately. Perhaps too you can invite the authors to write what their gem offers that others do not in the same class.

I think to handle the gem that suddenly does not handle a new framework etc is to have a downvote button where users can downvote and give a reason from a canned list you provide. If a gem suddenly starts getting down voted, does not have recent activity and the downloads are decreasing, it would then fall. You can then indicate this with a down arrow indicating it is trending down. The most popular reasons can be listed.

Answer 2

Number of downloads isn't, in my mind, a very useful meter. There are probably code snippets out there that haven't worked for years, but are still being tried (i.e. downloaded) by thousands of people each month.

1. Ask the submitter to identify what version of Rails the code was made in. While it may not solve all of your problems because they can't predict what later version of Rails will break it, it takes two seconds to specify that information, and it could save lots of time down the road. Display this on the download page.
2. Ask the downloader to rate the code... execution speed, execution reliability, code readability, ease of installation, documentation, etc.