The Statistical Case for Global Rankings

[macky]

And I agree about missing the competitions. If I just got really good results in the previous competition I would want to go to another competition as soon as possible to try to break them again. If I would fail, I wouldn't care about my "ranking in the statistics" at all.

Arnaud, I think you meant here that you don't care much about rankings in general, but let me answer the people who have suggested that no one would skip a competion because of a ranking, specifically one that is just on the statistics page. I also finally formulated my answer to Stefan's question

What is the question that the new ranking is supposed to answer?

which I really should haved done from the start.

I want this ranking to give a reasonable prediction of a competitor's average if ey were to do one competition round tomorrow, given that ey is in eir typical competition condition. The average-of-5 ranking doesn't do this well for two reasons: a number of top-100 cubers have one or two exceptionally fast averages, and some competitors are inactive or have recently been performing noticeably worse than their best average. Essentially, the average-of-5 ranking is a ranking by all-time best competition performance, whereas what I want is a ranking by recent average competition performance. I would use this new ranking, for example, to predict placements if the World Championship were to be held in a week. In my opinion, such a ranking would be both more interesting and more important than the average-of-5 ranking.

If you go to a large number of competitions a year, say 15+ 3-round ones, then an average ranking which tracks some fixed percentage of your rounds will tend to give a result pretty close to what is expected

We could also cap it from both sides. Not just minimum 3 like my previous suggestion but maybe also maximum 10. So from those 45 averages, averages the best 10.

The "half" and 3 and 10 are of course arbitrary and up for discussion, could also be "square root" of number of averages, minimum 5, and maximum I guess isn't needed when using square root .

Or simply average the best five averages, I think that's been mentioned before and I quite like it for both its value and its simplicity.

I like all these ideas. Simply taking some number of best averages would avoid the problem of possibly slower early-round averages, though I would still discard at least the fastest average for the reason mentioned above.

If someone always gets 13 seconds averages, I'd say he can get 13 seconds averages. If someone got seven 15 seconds averages and seven 11 seconds averages, I'd say he can get 11 seconds averages. Yes, it's not as consistent as the first guy, but I guess I care more about capability than consistency. One could argue that the single best average shows capability even better, but I do care about consistency as well, and would prefer "reasonably consistent" capability over one-time-exceptional capability.

A single number can't describe both capability and consistency. We've seen from our discussion so far that, once you start adding several desirable criteria, it's not easy to come up with a ranking system that's not arbitrary. If we abandon the idea of a single ranking, there's a much simpler approach. For each competitor's page, provide a link to a "detailed statistics" page that uses all averages within the past year to compute and display any number of useful charts and values: frequency plot of averages (say within 0.25 ranges), mean, SD, median and quartertiles, number of rounds, etc. A separate customizable ranking page could then allow any ranking method we've already talked about or someone may later suggest. For now, provide several options, and let the community decide which ranking methods are useful. The discussion of whether to choose one method for the official global ranking may be more appropriate after the community has understood the various methods available.

 

[AvGalen]

Thanks Macky, answering that question made the whole thread a lot more clear.

What this thread needs now is the following:
- Randomly pick 10 competitions from this year and calculate the "average average" in several different ways based on the results of the competitors in the 5 competitions before that
- Compare the "average average" with the actual results from these competitions to see which way(s) of calculating provide meaningful results

For this, database access, statistical knowledge and "Mathlab" and time/motivation will be needed. I personally only have 1.5 out of these 4 requirements

 

[macky]

I bet Stefan has all 4, even if StefanPochmann denies it.

 

[qqwref]

ey [...] eir

oh god my eyes ;_;

I want this ranking to give a reasonable prediction of a competitor's average if [th]ey were to do one competition round tomorrow

Aha. Then how about this:
- Take all averages in the past, say, 6 months. If we don't have at least, say, 5 rounds, choose the most recent 5 rounds instead.
- Calculate a weighted average, with a total weighting of 1, as follows: the first (i.e. most recent) is worth k; the second, k(1-k); the third, k(1-(k + k(1-k))); and so on, giving each average a weight of k times the remaining weight, except that the last two rounds should be weighted equally. k should be a number between 1/2 and 1, chosen based on exactly how much you want past rounds to be counted.

 

[macky]

I want this ranking to give a reasonable prediction of a competitor's average if [th]ey were to do one competition round tomorrow

Thanks Macky, answering that question made the whole thread a lot more clear.

Of course, that was just my formulation, decided after figuring that "best overall" performance within the past 12 months is harder to define. We can also include ranking methods that attempt to provide some definition for this.

oh god my eyes ;_;

Spivak pronouns amuse me.

Aha. Then how about this:
- Take all averages in the past, say, 6 months. If we don't have at least, say, 5 rounds, choose the most recent 5 rounds instead.
- Calculate a weighted average, with a total weighting of 1, as follows: the first (i.e. most recent) is worth k; the second, k(1-k); the third, k(1-(k + k(1-k))); and so on, giving each average a weight of k times the remaining weight, except that the last two rounds should be weighted equally. k should be a number between 1/2 and 1, chosen based on exactly how much you want past rounds to be counted.

Yeah, some weighted mean average should definitely be one of the options.

I might do it, though I hope someone else will do it first...

OK, I'm going to hack together some php/MySQL to try to generate rankings under several of the methods mentioned. Could you email me the source code for statistics.php to use as a reference?