The Statistical Case for Global Rankings

[macky]

That was a more particular example, but in general, the proposals are asking competitors to perform well throughout several consecutive conditions. For various reasons, people like me rarely get the chance to focus during 5 consecutive rounds.

qq should correct me if I'm wrong, but my interpretation of (3) is a rolling avg5 of averages, meaning as usual the average of the middle 3 among some 5 consecutive averages. That means you can have rounds 2 and 3 of one competition and 3 rounds of the next, and have the first-round average of the second competition thrown out. While I sympathize, I don't see how you could better account for the conditions you described without either throwing out first-round results altogether (unreasonable since you can set first-round average PBs) or discarding more than 20% of average results on either side (allows for too many bad rounds).

In European competitions, first rounds are often split into several groups, with each group judging and scrambling for another. With two groups, you can end up judging and then having to immediately go without much warm-up. This certainly isn't as severe as the conditions that organizers face, but it should be pointed out that, even for ordinary competitors, there are sometimes inevitable less-than-ideal conditions.

 

[qqwref]

Often, I don't have normal conditions for my averages. By helping out at most competitions, I'm often in the position where I'm forced to do an early round average at an inconvenient time, and without warm-up. When I perform terrible, I perform really terrible, and it doesn't help when I'm not properly warmed up.

I agree that a system which took the mean of consecutive averages would disadvantage you, but there's no way to fix this without also providing a huge benefit to inconsistent but fast cubers. How can I distinguish someone who should be getting a 12 average every time (but sometimes does 15s due to bad competition conditions), compared to someone who is very inconsistent and almost always gets either a 12s or a 15s average? You might deserve a 12 on the rank, where he doesn't; but if you have the same results as him, how can we tell the difference? I think you just have to accept that helping out is something which may hurt your standing in a ranking like this.

If someone would currently start at 35 seconds average, next competition do 17, next 12, then 10, then 8.5 he still wouldn't be among the top of the list which seems just plain wrong to me.

I suppose you're saying that if his real times are around 8.5-10 then he should be near the top of the list. But if you just look at the averages he's done, you can't know that for sure; he only has two "good" averages recorded at all, and statistically - if we only look at the average times - we can't distinguish this guy from someone who normally gets 10-12 but had two very bad rounds and one very good round. For someone like this, I'd say they simply have to do well in more competitions to place well in a ranking like this. The goal is not to reward someone who is fast once or twice in competition, but to reward someone who is consistently fast, so it would be wrong to allow someone to do well once or twice and be catapulted to near the top.

 

[Bryan]

So what's the advantage of taking only the last 5 averages? If you have a really bad competition (poor lighting venue or something), then that's going to "haunt" you for a while. And if there's someone who's super fast, but only goes to one competition a year, they'll be unranked.

And like Lucas, I don't do any warmup.

 

[macky]

I agree that a system which took the mean of consecutive averages would

Wait, so you do mean "mean" here?

So what's the advantage of taking only the last 5 averages? If you have a really bad competition (poor lighting venue or something), then that's going to "haunt" you for a while.

(3) is a rolling average of 5 averages. This has the advantage that competing can only improve the ranking. The same can't be said for say the average of all averages within the last year. The number can be changed, but 5 corresponds to 2 competitions (edit: 3 in some countries), which seems reasonable--more than one, but not too many so a bad competition can't haunt you for too long.

And if there's someone who's super fast, but only goes to one competition a year, they'll be unranked.

I don't think results from a single competition (2 or 3 averages) would do justice to what purports to be a global ranking. The year-to-date restriction needs to be kept to filter out inactive cubers, so it seems impossible to solve that problem.

 

[Bryan]

The number can be changed, but 5 corresponds to 2 competitions (edit: 3 in some countries), which seems reasonable--more than one, but not too many so a bad competition can't haunt you for too long.

Sure, for 3x3, but in some places, you have to be extremely fast to get beyond the first round. Plus, this idea wouldn't work well for other events that just have 1 round. I would want something that would work for all events the same.

 

[qqwref]

I agree that a system which took the mean of consecutive averages would

Wait, so you do mean "mean" here?

Sorry, meant mean or average.

 

[AvGalen]

If someone would currently start at 35 seconds average, next competition do 17, next 12, then 10, then 8.5 he still wouldn't be among the top of the list which seems just plain wrong to me.

I suppose you're saying that if his real times are around 8.5-10 then he should be near the top of the list. But if you just look at the averages he's done, you can't know that for sure; he only has two "good" averages recorded at all, and statistically - if we only look at the average times - we can't distinguish this guy from someone who normally gets 10-12 but had two very bad rounds and one very good round. For someone like this, I'd say they simply have to do well in more competitions to place well in a ranking like this. The goal is not to reward someone who is fast once or twice in competition, but to reward someone who is consistently fast, so it would be wrong to allow someone to do well once or twice and be catapulted to near the top.

Yes, I am saying that this guy should be among the top. Otherwise people that improve quickly will always remain absent in this ranking untill they are well established. Someone like Faz (not going to many competitions, then being top of the world) would have shown up in this ranking untill after he had broken the world record 2 or 3 times. So basically this ranking wouldn't have acknowledged his (current) skill because he didn't go to competitions enough while improving.
In a sport that is still seeing WR's broken by 15% a year with a new World Champion every 2 years and WR-holder every year a system that doesn't account for improvement is significantly flawed.
Wouldn't the solution be simple enough to assign a weighing factor based on time? Last competition counts for 2.5, the one before that for 2 (then 1.5, 1, 0.5) or something like that? I will leave it to the theorists and number crunchers to determine if 2.5/2.0/1.5/1.0/0.5 are good values and if they should be applied before or after removing best/worst

(I am still reading Ravi's topic where the "first rounds are different from finals" is discussed in enough detail)

 

[qqwref]

It just seems that, to me, the idea behind a global ranking (separate from the overall WCA ranking; separate from the 2010-only WCA ranking) is to reward consistent speed - not just speed itself. You can't prove that you are consistently fast by only one or two good results. I don't think that waiting to have 4 decent averages recorded is too much to ask; Feliks had a sub-10 avg5 of averages after his 3rd competition.

 

[macky]

The number can be changed, but 5 corresponds to 2 competitions (edit: 3 in some countries), which seems reasonable--more than one, but not too many so a bad competition can't haunt you for too long.

Sure, for 3x3, but in some places, you have to be extremely fast to get beyond the first round. Plus, this idea wouldn't work well for other events that just have 1 round. I would want something that would work for all events the same.

These are also problems you can't solve as long as you have a global ranking system deserving of the name (though there must be another adjective that better describes what we're getting at...). Even if you consider all averages within the last year, there needs to be minimum requirement for the number of averages.

I think you're right that 3x3 is the currently only category where any global ranking makes sense because of practicality concerns (enough data for everyone fast). The problems you brought are good reasons why any other global ranking should appear in the Statistics page, if at all. For me, this also means that we should design the ranking system for puzzles that meets the practicality requirements. Then it makes sense for 3x3 now, and maybe in the future, it will make sense for other puzzles.

 

[reThinking the Cube]

Please study this:

http://chess.emrald.net/ratinginfo.php

Replacing the words *tactitician* with *cuber*, and *problem* with *puzzle*, yields an excellent description for a superior CUBER Rating System. All the other proposals are feeble by comparison.

Spoiler

This page is based on the FICS help page on Glicko Rating (http://glicko.com).

As you may have noticed, each cuber has a rating and an RD. RD stands for ratings deviation.
What RD represents
The Ratings Deviation is used to measure how much a cuber's current rating should be trusted. A high RD indicates that the cuber may not be competing frequently or that the cuber has not solved very many cubes yet at the current rating level. A low RD indicates that the cuber's rating is fairly well established. This is described in more detail below under RD Interpretation.
How RD Affects Ratings Changes
In general, if your RD is high, then your rating will change a lot each time you solve a cube. As it gets smaller, the ratings change per solve will go down. However, the scramble's RD will have the opposite effect, to a smaller extent: if it's RD is high, then your ratings change will be somewhat smaller than it would be otherwise.
How RD is Updated
In this system, the RD will decrease somewhat each time you solve a cube, because when you solve more cubes there is a stronger basis for concluding what your rating should be. However, if you go for a long time without solving any cubes, your RD will increase to reflect the increased uncertainty in your rating due to the passage of time. Also, your RD will decrease more if the scramble's rating is similar to yours, and decrease less if your scramble's rating is much different.
Mathematical Interpretation of RD
Direct from Mark Glickman:
Each cuber can be characterized as having a true (but unknown) rating that may be thought of as the cuber's average ability. We never get to know that value, partly because we only observe a finite number of solves, but also because that true rating changes over time as a cuber's ability changes. But we can estimate the unknown rating. Rather than restrict oneself to a single estimate of the true rating, we can describe our estimate as an interval of plausible values. The interval is wider if we are less sure about the cuber's unknown true rating, and the interval is narrower if we are more sure about the unknown rating. The RD quantifies the uncertainty in terms of probability:
· The interval formed by current rating +/- RD contains your true rating with probability of about 0.67.
· The interval formed by current rating +/- 2 * RD contains your true rating with probability of about 0.95.
· The interval formed by current rating +/- 3 * RD contains your true rating with probability of about 0.997.
For those of you who know something about statistics, these are not confidence intervals, but are called central posterior intervals because the derivation came from a Bayesian analysis of the problem. These numbers are found from the cumulative distribution function of the normal distribution with mean = current rating, and standard deviation = RD. For example, CDF[ N[1600,50], 1550 ] = .159 approximately (that's shorthand Mathematica notation.)
Credits
The Glicko Ratings System was invented by Mark Glickman, Ph.D. who is currently at Boston University.

 

[Stefan]

Before this thread becomes too old, I want to at least bring the main ideas to Stefan (who can access the WCA database).

Well, the export should give you access to all relevant data, too

(3) qq's majority-in-year average-average: Best (rolling) avg5 of consecutive averages
...
under which competing can only improve the ranking

Not quite. Yes, competing can't hurt *past* scores, but it *can* hurt a *future* score. Imagine your last three averages were somehow awesome. You could go to another competition in a few days, but you're out of shape and have no time to prepare (important exam coming up). If you compete, you'd destroy your chance at an awesome 5-streak. You might better leave out this competition and go to another in a month, for which you can prepare a lot.

And let me throw out another idea: Average the better half of averages in the past rolling year, minimum three. So if you had 14, average the best 7. If you had 4, average the best 3.

 

[qqwref]

Interesting proposal, Stefan. It would certainly do away with a single competition ruining your average (although a single competition could definitely hurt your average). I wonder if it would be fair to consistent people, though. By discarding only the bad averages, there is an incentive to have a high SD on your averages. It would mainly measure how good you do on the best half of your rounds, but would ignore how you perform on the worst half, and I think this might skew the results against people who always perform almost equally good.

I wonder if the "law of large numbers" is going to become very important in ideas like this. Anything which averages a variable number of results could have this problem. If you go to a large number of competitions a year, say 15+ 3-round ones, then an average ranking which tracks some fixed percentage of your rounds will tend to give a result pretty close to what is expected, given the distribution of your averages. But if you only go to 2 or 3 competitions, you could get lucky and do very well compared to what would be expected of you. Then there is a disincentive to go to more competitions - you can expect that your ranking would even out (i.e. get worse) if you do. Maybe such disincentives are things we should just accept, rather than trying to fix.

 

[Stefan]

I wonder if it would be fair to consistent people, though.

Well, it all depends on what we're after. What is the question that the new ranking is supposed to answer?

If someone always gets 13 seconds averages, I'd say he can get 13 seconds averages. If someone got seven 15 seconds averages and seven 11 seconds averages, I'd say he can get 11 seconds averages. Yes, it's not as consistent as the first guy, but I guess I care more about capability than consistency. One could argue that the single best average shows capability even better, but I do care about consistency as well, and would prefer "reasonably consistent" capability over one-time-exceptional capability.

So for me, the question this ought to answer is "What is someone (reasonably) capable of?" and I'm fine with ignoring bad outliers.

If you go to a large number of competitions a year, say 15+ 3-round ones, then an average ranking which tracks some fixed percentage of your rounds will tend to give a result pretty close to what is expected

We could also cap it from both sides. Not just minimum 3 like my previous suggestion but maybe also maximum 10. So from those 45 averages, averages the best 10.

The "half" and 3 and 10 are of course arbitrary and up for discussion, could also be "square root" of number of averages, minimum 5, and maximum I guess isn't needed when using square root .

Or simply average the best five averages, I think that's been mentioned before and I quite like it for both its value and its simplicity.

Maybe such disincentives are things we should just accept, rather than trying to fix.

Yeah, I agree. If someone really misses out on a competition just because of some ranking, I don't really care much about him. Also, as far as I understand, it's only suggested as an add-on, not a replacement, of the current ranking which would remain the main ranking.

 

[AvGalen]

(don't think I need to quote all of what The Pochman said)

This is starting to look good. Could a top 25 be calculated with some of the variables you mentioned?

And I agree about missing the competitions. If I just got really good results in the previous competition I would want to go to another competition as soon as possible to try to break them again. If I would fail, I wouldn't care about my "ranking in the statistics" at all.

Spoiler

If Stefan doesn't complain about the way I wrote his name, I will continue to call him The Pochman from now on

 

[Stefan]

Could a top 25 be calculated with some of the variables you mentioned?

I might do it, though I hope someone else will do it first...

Spoiler

If Stefan doesn't complain about the way I wrote his name, I will continue to call him The Pochman from now on

I hereby complain . Don't like how it sets me apart, already annoys me how Dene always calls me "Mr. Pochmann".

 

[AvGalen]

Spoiler

If Stefan doesn't complain about the way I wrote his name, I will continue to call him The Pochman from now on

I hereby complain . Don't like how it sets me apart, already annoys me how Dene always calls me "Mr. Pochmann".

Spoiler

while (DidHeComplain("The Pochman"))
{
OutputRandomWord(WordTypes.FourLetter);
}

 

[qqwref]

Spoiler

No! You can't make a while loop using a history-lookup function! It'll run forever!

 

[AvGalen]

Spoiler

No! You can't make a while loop using a history-lookup function! It'll run forever!

Exactly my point. StefanPochmann won't change his mind, so I will be cursing forever

 

[Stefan]

StefanPochmann

Gah, how about just Stefan?

back to topic?

 

[AvGalen]

StefanPochmann

Gah, how about just Stefan?

back to topic?

Stefan is the real person. He might change his mind with future insights.
StefanPochmann is a user (although some people claim he is a knowledgebase with a verbal spanking mode) that would only change his mind if proven wrong (some people claim these are minor updates or bugfixes), which would most likely be done by .... StefanPochmann.
(if you add a ? I feel obligated to answer)
but okay, Stefan it will be.

Back to topic indeed, although I will spend my time on the blindfolded topic I opened and on travelling again