The Statistical Case for Global Rankings

[macky]

Read Ravi's post. Then read my post, reproduced below. This deserves a separate discussion, and, in my opinion, is a better solution than changing the competition format.



Performing when it counts is of course important. The problem with comparisons to other sports is that speedcubing inherently involves chance. I don't know how much of a top cuber's time variation is from pure chance, but it's considerable. Even when performing the very best, a top CFOPer's times will vary > 0.5 second just by the difficulty of last layer cases--and we aren't even considering F2L. With the continuing clustering of times near the top, this variation is starting to become enough to significantly affect the ranking in a competition.

I thought about all this back in WC07 and with my prayer before WC09. Because of complications such as those mentioned in this thread, I'm fine having the winner of the last round be the winner of the competition. Or if an organizer feels strongly, he can use the fastest average or the mean of averages. In any case, there's no rule change needed.

The winner of a competition is the cuber who performed "the best" (in a predefined sense) at that competition, not necessarily the best cuber. I therefore don't see too much need to change the competition format so that the best cuber has a better chance of winning the competition.* What I've suggested instead since 2007 is an official yearly overall ranking (see this thread for an example).

* But I'm sure people will soon be needing my prayer more often!

 

[Faz]

So, you're suggesting having a yearly overall ranking for each event, WCA recognised, like what JustinJ did in his thread? So, you would take all of the cuber's averages, and determine the mean. Eg: "The 2010 champion for pyraminx is Yohei Oka - 4.00 average time"

Sounds like a cool idea. Maybe I just misread the thread totally, but it sounds like this is what you're proposing.

 

[RCTACameron]

Judging by JustinJ's thread, Faz would probably be the first in everything.

If someone only went to one competition in a year, and had a lucky average, then luck would still play a big part in the results.

However, I think this is a great idea, and it would be interesting to see who comes out on top. I hope that the WCA does this.

 

[macky]

If someone only went to one competition in a year, and had a lucky average, then luck would still play a big part in the results.

We'd come up with a minimum number of competitions to avoid a situation like this.

For now, though, I want to hear what people think about having such a ranking without worrying too much about the implementation. Average of 5 is great for determining rankings within a competition, and so a world ranking by average of 5 is great for comparing best competition performances. But many people agree that average of 5 isn't a satisfactory measure of a cuber's true ability. This is where a ranking like this would come in. Some questions:
- (Would such a ranking be useful/interesting? Well yeah.)
- Is it worth being displayed on the WCA database as a separate official ranking?
- Is it just as important as the best average of 5 ranking?
- Is it important enough to warrant giving some title based on it? On an annual basis?
- (Would it be hard to agree on a consistent ranking method? Are there other practicality considerations?)

 

[ExoCorsair]

Wouldn't it a combined average of all past competition results discourage newer cubers from participating, though (at least, until they got faster)?

 

[macky]

Wouldn't it a combined average of all past competition results discourage newer cubers from participating, though (at least, until they got faster)?

Only use results from the last X competitions or from the last Y months. As long as the competitor is improving, the global average decreases with more results using either scheme. Using last X competitions would discourage cubers who are out of practice and can't match their last official results from competing again unless they improve. So we'd need to use some combination of number of competitions (with a minimum) and time period.

But these things are mere details. I want more feedback on the main idea.

 

[hawkmp4]

How much different would the highest ranks of this new system be than the all time list? For example, thirty of the top 31 3x3 average results were set in 2010. Forty-five of the top 50, and 78 of the top 100. Only 3 in the top 100 were from 2008, none before then. So it's likely that some of the 2009 results are still within a calendar year. I like the idea, though, especially for the people who aren't in the highest tier of cubing.

EDIT: I think I misunderstood...feel free to ignore this post. I'm going to reread things.

 

[PhillipEspinoza]

This kinda info would be useful to know, but why can't we just put it under the Statistics Page? You know, kinda like how they have the BLD Success Streak with a minimum of 5 solves.

 

[macky]

This kinda info would be useful to know, but why can't we just put it under the Statistics Page? You know, kinda like how they have the BLD Success Streak with a minimum of 5 solves.

That's perfectly possible. It depends on how important you think this type of ranking is compared to the best competition result (i.e. avg of 5) ranking.

 

[hawkmp4]

I'd be interested in seeing something like the mean of the last x averages in competition, within the last y months. That'd give a more accurate picture of the current cubing community. The 'last y months' part would knock inactive cubers (e.g., Nakajima) out and the 'last x averages' would reward consistent cubers over ones who get lucky on an average. I guess x and y would just have to be played with to see what gives a good result.

 

[Ravi]

I'm not sure that directly averaging results is the best way to do it, since that could potentially discourage people from competing. I find it fairly plausible that, for example, on December 27, 20xx, Speedcuber A may have a yearly average of 9.93 seconds compared to 10.04 for Speedcuber B. Speedcuber A would then have two compelling reasons not to go to the Geographical Region C Open on December 28, 20xx: staying ahead of Speedcuber B and staying under 10 seconds. A better scheme, IMO, would never penalize people for competing more. At the same time, though, we don't want to give an massive advantage to someone who goes to twenty competitions a year compared to someone who only goes to one or two but is as fast or faster. Here's one scheme (intended more as an example than anything else):

For any average of x seconds, assign a value of 10^-x points. Add up these values over a one-year period. Then, to make the numbers more life-sized, take the negative log (to base 10) of the total. Thus, ten 12s would count as equal to one 11. This may (or may not) work decently among the top few competitors, but I don't think this exact system would work particularly well as a whole, because I don't think a 37-second average is worth a hundred million times more than a 45-second one. Feel free to refine.

I do like the idea in principle though, and I think something of the sort should be implemented even if the competition format is changed (which looks unlikely at the moment). Point to consider: Is there any reason to do this ranking only on a yearly basis? I think it would make a lot of sense to have a continual year-to-date ranking (as many sports already do) with awards, if any, given at the end of each calendar year.

Another idea I really like: a formula to objectively compare historical speedcubing results, with extra value to results achieved earlier. For example, which is more impressive: Macky's 14.52 average at Caltech Fall '04 or faz's 8.52 average at New Zealand Champs '10? Furthermore, which is more impressive: Macky's 14.52, 15.28, 15.38, and 15.68 topping the "until 2004" average list, or faz's 8.52, 9.21, 9.37, 9.38, and 9.82 topping today's average list? [Based on the trajectory of my thread, I think I should point out that these last questions are rhetorical and NOT INTENDED TO START A FLAME WAR between Mackyites and fazz0rs.]

 

[macky]

For any average of x seconds, assign a value of 10^-x points. Add up these values over a one-year period. Then, to make the numbers more life-sized, take the negative log (to base 10) of the total. Thus, ten 12s would count as equal to one 11. This may (or may not) work decently among the top few competitors, but I don't think this exact system would work particularly well as a whole, because I don't think a 37-second average is worth a hundred million times more than a 45-second one. Feel free to refine.

I do like the idea of a point system. Let me think more.

Point to consider: Is there any reason to do this ranking only on a yearly basis? I think it would make a lot of sense to have a continual year-to-date ranking (as many sports already do) with awards, if any, given at the end of each calendar year.

That sounds good.

Another idea I really like: a formula to objectively compare historical speedcubing results, with extra value to results achieved earlier. For example, which is more impressive: Macky's 14.52 average at Caltech Fall '04 or faz's 8.52 average at New Zealand Champs '10? Furthermore, which is more impressive: Macky's 14.52, 15.28, 15.38, and 15.68 topping the "until 2004" average list, or faz's 8.52, 9.21, 9.37, 9.38, and 9.82 topping today's average list?

How about the multiple of SD away from the mean of top 100 averages (best avg of 5 of the top 100 people, not results)? This gives a sense of impressiveness as perceived by the community at the time. Note that this accounts for the increasing difficulty of improvement and the greater talent pool today since both factors lead to lower SD. I can't think of any other measure that would correctly account for these factors now and in the future. In any case, this type of data belongs to the Statistics page.

[edit] I went ahead and calculated this (I thank emacs's keyboard macros).

Not enough data in 2004.

Top 100 up to 2005
Mean = 19.6730
SD = 2.3949
mean - macky (14.52) = 5.1530
(mean - macky) / SD = 2.1517


Top 100 up to 2006
Mean = 17.0077
SD = 1.5912
mean - Anssi (13.22) = 3.7877
(mean - Anssi) / SD = 2.3804

(My 13.34 here gives 2.2421 SD, more than my 14.52.)


Top 100 up to 2007
Mean = 14.8121
SD = 1.2073
mean - Gungz (11.76) = 3.0521
(mean - Gungz) / SD = 2.5280


Top 102 up to 2008 (because of ties)
Mean = 13.4360
SD = 0.9570
mean - Nakaji (11.28) = 2.1560
(mean - Nakaji) / SD = 2.2529


Top 100 up to 2009
Mean = 12.2808
SD = 0.7329
mean - Tomasz (10.07) = 2.2108
(mean - Tomasz) / SD = 3.0165


Current top 101
Mean = 11.4573
SD = 0.7042
mean - faz (8.52) = 2.9373
(mean - faz) / SD = 4.1711



As I suspected, my 14.52 is farther away from the top 100 mean then than faz's 8.52 is from today's mean, but the massive clustering in today's ranking means that his average is farther away in terms of SD and so more impressive by this measure--as it should be in view of today's larger talent pool and faster times!

This analysis can of course also be done on a global ranking and would produce different results (e.g. Gungz didn't have many more chances to compete because of his military service), but this alone is interesting because the fastest average of 5 is what top speedcubers have always aimed at. For me there's some discrepancy between the SD distance and the degree of "holy s.." reaction, especially with Tomasz's 10.07, but this is probably because I was desensitized to seeing these kinds of times by then.

The shock factor shouldn't necessarily correspond to a measure like this, but are there other SD-like values that would be appropriate here?

 

[macky]

This is based on something Tyson pointed out to me. Can someone who knows statistics well help me out here?

For SD to work, should we take a bigger sample size to account for the bigger talent pool? Maybe top 1% of the ranking instead of a fixed number? Do we need to analyze the distribution of times to find an appropriate sample size to take? Can we do this in some non-arbitrary way? Are there other measures like SD that are appropriate here?

 

[qqwref]

A better scheme, IMO, would never penalize people for competing more.

Although this sounds on the surface like a good criterion, I don't think it necessarily is. If you don't want to penalize people for competing more, you simply cannot do any kind of *global* average of their times - it is always a possibility that in the next competition a cuber will do badly and make their average worse. Or, even worse, take this hypothetical: a cuber goes to 20 competitions in the year, each with one round of 3x3 (haha). So his average average will be the mean of those 20 averages, or something similar. But suppose this same cuber, just by luck, missed the 10 competitions in which he did the worst (and performed the same as before in the 10 he attended). So now his average average is better, assuming 10 competitions is enough. Of course, this cuber couldn't know that if he had attended those other 10 competitions as well his statistics would look worse.

I do have an idea of a statistic that would at least always make it worthwhile to attend another competition: for a given year, the ranking would be by the best (rolling) avg5 of consecutive averages, with the caveat that (a) a majority of those averages are in the correct year, and (b) all the averages are within one year of that year. (You could change it to make all averages within the year, but I think this could overly penalize people who don't attend a huge number of competitions. This way, going to one large competition a year would guarantee you at least a spot in the ranking.) Anyway, that way going to the next competition couldn't hurt your best avg-avg, because you have already completed it.

This is based on something Tyson pointed out to me. For SD to work, should we take a bigger sample size to account for the bigger talent pool? Maybe top 1% of the ranking instead of a fixed number? Can someone who knows statistics help me out here?

You should take a bigger sample size. The top 100 in 2006 was much more spread out than it is now, simply because there were fewer people with world-class skill; the top 100 is more close now. I think using a fixed percentage of the total cuber pool might help, but it's not perfect since ideally you'd only want to include people who you'd describe as reasonably close to world-class.

 

[Ravi]

If you don't want to penalize people for competing more, you simply cannot do any kind of *global* average of their times - it is always a possibility that in the next competition a cuber will do badly and make their average worse.

What's your view on a point system? That would avoid the problem you mentioned, since it's a global sum rather than an average (compare: ATP race in tennis, or http://primes.utm.edu/bios/top20.php). One objection against it would be that it might place too much value on the number of competitions one attends rather than the quality of one's results; there may or may not be a good way to fix this. (Appropriately chosen polynomials might work instead of the exponential I gave.) I think it would be good, though, to rank everyone who's completed at least one average within the year. This would be quite inclusive: all of the top 56* speedcubers by best average have competed at least once this year, although some (Gabriel Dechichi Barbar, Nakajima, Gungz, Shinichiro Sato, and Takumi Yoshida for example) have competed only once. Some kind of point system seemed like the simplest way to reward people for competing more without actually setting a quota (other than one average per year).

Of course, your idea, or something like it, could work too.



* ... and I'm 57th. Can I play the "competition-starved Midwesterner" card?

 

[qqwref]

If you don't want to penalize people for competing more, you simply cannot do any kind of *global* average of their times - it is always a possibility that in the next competition a cuber will do badly and make their average worse.

What's your view on a point system? That would avoid the problem you mentioned, since it's a global sum rather than an average (compare: ATP race in tennis, or http://primes.utm.edu/bios/top20.php). One objection against it would be that it might place too much value on the number of competitions one attends rather than the quality of one's results; there may or may not be a good way to fix this.

I think this is a pretty major objection and removes most of the value of a standard points system (i.e. any one where you accumulate points in each competition). If two cubers are exactly equal, wouldn't the winner essentially be the one who has been to the most comps in the year? The "most medals in 3x3" part of the statistics is like this and is clearly biased against good cubers who also attend a ton of comps. For winning to be possible with my system (let's give it a name for ease of discussion, perhaps majority-in-year average-average or MIYAA) you only need to do three 3x3 rounds a year; doing more competitions helps, but only in the sense that you have more chances to get a good result. I don't think MIYAA would have much bias against very good cubers who only go to one or two comps in a year.

 

[reThinking the Cube]

Mean of weighted results for the previous 100 weeks.

S = results, (can be either 5-avg or single solves)

N = #of weeks ago this result was entered.

For each result S, create (100-N) occurrences in the (for computational purposes only) data set of results[R].

Use the mean of R for your ranking value.

(qq) can probably explain it better than me.

 

[keemy]

reThinking:
not say you are wrong but one of the things I feel (and I think others) is that going to more competitions should not overly negatively effect your ranking. Also, going to only 1 competition getting an amazing result and not competing for a long time should not give you a great advantage. Unless I have misunderstood your proposal your system does both of these things thus making it undesirable.

simplified you want (if i read correctly) is

\( (\sum_{n=0}^{99} (100-n)(a_n))/(\sum_{n=0}^{99} (100-n)(R_n)) \)

where \( n= \) number of weeks ago, \( a_n= \) sum of results n weeks ago, \( R_n= \) number of results n weeks ago.

 

[reThinking the Cube]

reThinking:
not say you are wrong but one of the things I feel (and I think others) is that going to more competitions should not overly negatively effect your ranking. Also, going to only 1 competition getting an amazing result and not competing for a long time should not give you a great advantage. Unless I have misunderstood your proposal your system does both of these things thus making it undesirable.

simplified you want (if i read correctly) is

\( (\sum_{n=0}^{99} (100-n)(a_n))/(\sum_{n=0}^{99} (100-n)(R_n)) \)

where \( n= \) number of weeks ago, \( a_n= \) sum of results n weeks ago, \( R_n= \) number of results n weeks ago.

Hmmm. How to solve the problem of - "going to only 1 competition getting an amazing result and not competing for a long time should not give you a great advantage."?

Result based on X# of solves? Maybe this could be made to work, but I can see your point.

Here is another idea:
http://en.wikipedia.org/wiki/Elo_rating_system

 

[hawkmp4]

I don't see how ELO rankings could be used in a situation where we don't have head to head matches.