Calculate comparative rank of things compared two at a time

Question

How to calculate the weights of 10 things that express a person's relative preference when the input data is as follows:

The list of 10 things are presented in random pairs (ignoring order, so A vs B might be shown, or B vs A, but not both; and not an item with itself);

A is better than B
D is better than F
H is better than B
C is better than J

etc.

Is there a way to rank or weight the 10 items from data like this? And is there a way to shorten the sequence from 10! / (2! (10 - 2)!) =45 questions.

Answer 1

TL;DR: Compare each element of a list to each other element, and assign a weight or value to the choice that wins out. Then rank each element by how many "wins" it achieved.

These are really good valid responses that call out limitations that need to be brought up, but they provide no solution. I offer one...

When making relative comparisons, and in this example, ranking based off a subjective preference, there will no doubt be a lack of consistency that will break the transitivity integrity. But, you can overcome this. How? Well, using what is suggested in the question: Weights.

Compare each element of a list to each other element, and assign a weight or value to the choice that wins out. Then rank each element by how many "wins" it achieved.

And each win does not have to weigh the same. E.g., if one element beats a significantly high ranked or top ranked element, it gets additional points or something. Idk, go nuts!

So, this is not a scientific unequivocal ranking system, (which you are never getting anyways without an objective measurement) but I think with this approach, you can get pretty darn close.

I wrote a simple java command line program that does this comparing fruit. Below is my solution that works "good enough" for what I was looking for. I hope someone else is able to find this helpful.

package com.example;

import java.util.Comparator;
import java.util.List;
import java.util.Scanner;
import java.util.stream.Collectors;
import com.google.common.collect.Lists;
import lombok.Getter;
import lombok.Setter;

public class ComparativeRanking {

    public static void main(String[] args) throws Exception {
        ComparativeRanking instance = new ComparativeRanking();
        final List<Fruit> fruits = Lists.newArrayList(
                new Fruit("banana"),
                new Fruit("apple"),
                new Fruit("kiwi"),
                new Fruit("strawberry"),
                new Fruit("mango"));
        Fruit[] fruitArray = fruits.toArray(new Fruit[0]);
        instance.compute(fruitArray);
    }

    void compute(Fruit[] fruitArray) throws Exception {
        for (int i = 0; i < fruitArray.length; i++) {
            for (int k = i + 1; k < fruitArray.length; k++) {
                Fruit choice1 = fruitArray[i];
                Fruit choice2 = fruitArray[k];
                System.out.println(String.format("Enter '1' for %s and '2' for %s", choice1.name, choice2.name));
                Scanner scanner = new Scanner(System.in);
                int selection = scanner.nextInt();
                switch (selection) {
                    case 1:
                        choice1.addWin();
                        break;
                    case 2:
                        choice2.addWin();
                        break;
                    default:
                        throw new Exception("Hey idiot, pick 1 or 2.");
                }
            }
        }

        List<Fruit> fruitsRankedByWins = Lists.newArrayList(fruitArray)
                .stream()
                .sorted(Comparator.comparing(Fruit::getWins).reversed())
                .collect(Collectors.toList());

        System.out.println("Result:");
        for (Fruit fruit : fruitsRankedByWins) {
            System.out.println(String.format("Fruit: %s | Wins: %s", fruit.getName(), fruit.getWins()));
        }
    }

    @Getter
    @Setter
    static class Fruit {
        String name;
        int wins;
        public Fruit(String name) {this.name = name;}
        void addWin() {
            wins++;
        }
    }
}

Answer 2

In order to rank data using rules like this the relation "better" must be transitive.

so A vs B might be shown, or B vs A, but not both

This is a step in the right direction, but this is not quite transitive. You can still do this:

A is better than B
B is better than C
C is better than A

Now you have a cycle, so the three rules above cannot be used to rank items.

When there are no cycles you can make orders consistent with the rules. However, the ordering may not necessarily be unique. For example, if your rules look like this

A is better than B
C is better than B

and you need to order {A, B, C} both "ACB" and "CAB" are consistent with your rules.

If you are looking for any consistent ordering based on the rules, you can first construct a tree induced by your "X is better than Y" rules, and then use that tree to decide the relative ordering of your ten items.

Answer 3

You're making a dangerous psychological assumption here: you assume that the "better than" relation is transitive, so that from

A > B
B > C

directly follows that

A > C

That is, without loss of generality, not the case for humans preferences. So this puts your whole approach in a questionable light.

However, if you can justify that assumption, yeah, this breaks down to a sorting problem. There's a truckload of sorting algorithms out there, that just rely on the existence of an ordering relation ">". Your weight is then just the position in the sorted list.

Every real-world programming language out there probably has a library that contains at least one sorter algorithm; most languages also have a way of specifying which function to call to compare two elements, so this really boils down to calling that sorter function with the unsorted list and the function to call when comparing two things.

If the assumption that the "better than" relation is transitive cannot be made, things get a lot more complex. Basically, you can build a directed graph and try to find paths through it, but that graph might not be cycle free, and there might be no definite possibility to assign any weights.

There's decades of psychological method research on how to test such things; I'd recommend finding a university that offers psychology programmes and asking the people there.