The Statistical Case for Global Rankings

[keemy]

I thought about the ELO actually but I don't think it would work very well for cubing to put it simply there is a very large location bias (ex. faz who lives in Australia has basically no competitors remotely close to him in 3x3 would not be able to make his rating very high)

and I don't think your method can be salvaged without drastic change such that it would be easier to think of something else entirely.

so far the thing I like the most is qq's, ravi's is alright but it doesn't scale very well and someone could get a decently large advantage by going to enough competitions (ex A avgs 13 and B avgs 12 well if A just goes to enough competitions to get 10 times as many solves as B then A will be ranked better)

 

[Ravi]

I think this is a pretty major objection and removes most of the value of a standard points system (i.e. any one where you accumulate points in each competition).

Well, yes, if we're trying to make this totally independent of the number of comps attended. I guess a point system would be more a measure of "prolificness," so to speak. But perhaps it could be adapted by dividing the total score by the number of averages completed before converting back to units of time. That would give a sort of weighted average favoring the faster times. Do you think that's desirable at all? It would be rather slanted, but at least it wouldn't have as strong a tendency to discourage people from competing.

 

[qqwref]

Hmmm. How to solve the problem of - "going to only 1 competition getting an amazing result and not competing for a long time should not give you a great advantage."?

Either you have to create a bias towards people who attend more competitions, or you have to (implicitly or explicitly) require someone to have a certain number of results to ensure a place in the ranking.

Here is another idea:
http://en.wikipedia.org/wiki/Elo_rating_system

A system like this is not applicable to cubing because times are unaffected by the skill of the other people in the round. You could only really apply it to in-round/in-competition placements, and as keemy correctly pointed out you would still have many problems associated with people not leaving their region. Besides, why even consider such a subjective and opaque system when we can already objectively measure each solve to (near-)centisecond accuracy?

I guess a point system would be more a measure of "prolificness," so to speak. But perhaps it could be adapted by dividing the total score by the number of averages completed before converting back to units of time. That would give a sort of weighted average favoring the faster times.

Could be good, but it is dangerous to try to compromise between favoring those who attend few comps and those who attend many, because finding a balance that doesn't favor either side would be difficult. You'd also have to be careful to keep the measure simple, because using a statistic that is not computed in an obvious way leads to the community waiting for each new published ranking rather than actively trying to optimize their result (see: ELO), and that's boring

 

[Faz]

I highly doubt that someone would not attend any competitions because they did really well in one, just to get their name on the stats page >_>

 

[reThinking the Cube]

I don't see how ELO rankings could be used in a situation where we don't have head to head matches.

Here is the standard ELO rating formula:

Rnew = Rold + K(Tactual -Texpected)

Simply stated: new rating = old rating, changed by the difference between the actual results and the expected results, multiplied by a scaling factor, K.

ELO rating is actually an estimation of human performance.

We can therefore make CubELO = estimated time that this human will take to solve a cube.

This rating would be a *performance predictor*.

Glicko rating system is an optimized version of ELO, and uses rating deviation, and volatility factors, to get better predictions, but is more complicated.

Any and all official results by a competitor could be used to calculate their CubELO.

I thought about the ELO actually but I don't think it would work very well for cubing to put it simply there is a very large location bias (ex. faz who lives in Australia has basically no competitors remotely close to him in 3x3 would not be able to make his rating very high)

There is no location bias (as there would be in chess), since competitors are competing against the clock, and clocks are located everywhere.

A system like this is not applicable to cubing because times are unaffected by the skill of the other people in the round. You could only really apply it to in-round/in-competition placements, and as keemy correctly pointed out you would still have many problems associated with people not leaving their region. Besides, why even consider such a subjective and opaque system when we can already objectively measure each solve to (near-)centisecond accuracy?

reAnswered™ above.

 

[qqwref]

\( R_{new} = R_{old} + K(T_{actual} - T_{expected}) \)

Simply stated: new rating = old rating, changed by the difference between the actual results and the expected results, multiplied by a scaling factor, K.

ELO rating is actually an estimation of human performance.

We can therefore make CubELO = estimated time that this human will take to solve a cube.

This rating would be a *performance predictor*.

So you're not really proposing an ELO-type ranking system with points, but instead another statistical measure aimed at estimating a typical performance. Fair enough, but there are already several proposals for this (global-average of avg5, best avg5 of avg5, recent avg12 of avg5, etc.). Can you explain exactly how yours would be better than the other ones? How exactly would it work out mathematically, anyway? It seems like it might be equivalent to some other measure but I'm not sure since your formula is iterative.

 

[keemy]

I thought about the ELO actually but I don't think it would work very well for cubing to put it simply there is a very large location bias (ex. faz who lives in Australia has basically no competitors remotely close to him in 3x3 would not be able to make his rating very high)

There is no location bias (as there would be in chess), since competitors are competing against the clock, and clocks are located everywhere.

reAnswered™ above.

This would defeat the point in using something like the elo which is best used when you want to compared expected performance relative to other competitors which was really it's only upside (the scrambles wouldn't matter as much because anyone at the same comp would get the same scrambles in the final and thus their ranking would changed based on how they placed there)

What you want to do would regain a scramble bias (which was the only elo advantage) give a heavy incentive to not compete if you were out of shape (for the purpose of these rankings) and give a large advantage to people who only go to a few competitions and get good times.

 

[reThinking the Cube]

\( R_{new} = R_{old} + K(T_{actual} - T_{expected}) \)

Simply stated: new rating = old rating, changed by the difference between the actual results and the expected results, multiplied by a scaling factor, K.
ELO rating is actually an estimation of human performance.

We can therefore make CubELO = estimated time that this human will take to solve a cube.

This rating would be a *performance predictor*.

So you're not really proposing an ELO-type ranking system with points, but instead another statistical measure aimed at estimating a typical performance. Fair enough, but there are already several proposals for this (global-average of avg5, best avg5 of avg5, recent avg12 of avg5, etc.). Can you explain exactly how yours would be better than the other ones? How exactly would it work out mathematically, anyway? It seems like it might be equivalent to some other measure but I'm not sure since your formula is iterative.

CubELO would NOT be an average. It is better because I reThink it to be better, and my reasoning is better than flawless. Seriously, this is a dynamic performance predictor. What could be better than that? It is not, as you have possibly suggested, equivalent to any other measure that has been proposed so far.

 

[qqwref]

CubELO would NOT be an average.

Yes it is, if I'm interpreting it correctly. Your CubELO formula increases the "estimation" if you do worse and decreases it if you do better, does it not?

Here, from Wikipedia: "In mathematics, an average, or central tendency of a data set is a measure of the "middle" value of the data set." Emphasis added by me. The whole idea of any type of average is to predict a typical element of the data set (in this case a cubing performance).

 

[hawkmp4]

Even if we say we'll use some form of ELO (which just seems wrong to me, the usefulness of the ELO system comes from its ability to compare and rank, say, chess players, where wins and losses aren't a useful to directly evaluate a player's skill) that DOESN'T resolve ANY of the problems brought up in this thread- we'd still have to come up with a way to measure performance.

 

[reThinking the Cube]

CubELO would NOT be an average.

Yes it is, if I'm interpreting it correctly.

No it isn't, you are misinterpreting (again). An approximating Bayesian Updating Algorithm would be technically more correct. K factor scaling makes all the difference.

Even if we say we'll use some form of ELO (which just seems wrong to me, the usefulness of the ELO system comes from its ability to compare and rank, say, chess players, where wins and losses aren't a useful to directly evaluate a player's skill) that DOESN'T resolve ANY of the problems brought up in this thread- we'd still have to come up with a way to measure performance.

wt(DNF)?:confused:

 

[macky]

CubELO would NOT be an average.

Yes it is, if I'm interpreting it correctly. Your CubELO formula increases the "estimation" if you do worse and decreases it if you do better, does it not?

Here, from Wikipedia: "In mathematics, an average, or central tendency of a data set is a measure of the "middle" value of the data set." Emphasis added by me. The whole idea of any type of average is to predict a typical element of the data set (in this case a cubing performance).

Careful, I wouldn't argue with rTtC over technicalities; that's how he gets you.

CubELO would NOT be an average.

Yes it is, if I'm interpreting it correctly.

No it isn't, you are misinterpreting (again). An approximating Bayesian Updating Algorithm would be technically more correct. K factor scaling makes all the difference.

I have a bad feeling that he actually knows Bayesian Updating (easy to understand anyway) and purposefully misapplied the term here to something not at all Bayesian but with a similar feel.

 

[hawkmp4]

Even if we say we'll use some form of ELO (which just seems wrong to me, the usefulness of the ELO system comes from its ability to compare and rank, say, chess players, where wins and losses aren't a useful to directly evaluate a player's skill) that DOESN'T resolve ANY of the problems brought up in this thread- we'd still have to come up with a way to measure performance.

wt(DNF)?:confused:

Okay, one more try, and then I will just assume you're trolling. It's not that difficult.

Simply stated: new rating = old rating, changed by the difference between the actual results and the expected results, multiplied by a scaling factor, K.

Using the ELO system wouldn't resolve any of the issues brought up in this thread, because we would still have to decide on what we would use to determine 'results.' For example, best average in a competition? That favors competitors who went to comps with more rounds of 3x3, is that what we want to do? Average of averages? That brings up many of the issues already discussed here.

 

[qqwref]

CubELO would NOT be an average.

Yes it is, if I'm interpreting it correctly.

No it isn't, you are misinterpreting (again).

OK, so CubELO is *not* trying to estimate a typical performance? If it IS, then it is a type of average, whether you like it or not (that's just what the definition is). You said yourself: "CubELO = estimated time that this human will take to solve a cube." If you think this direct quote from your post is wrong, feel free to correct it. A measure can't fit into the statistician's definition of average and also not be a type of average.

This has nothing to do with the above paragraph, but just to clarify, what is the relation between \( R_{old} \) and \( T_{expected} \)?

Also, if you legitimately want us to consider your idea, we'd need an answer to this:

Can you explain exactly how yours would be better than the other ones? How exactly would it work out mathematically, anyway?

Nobody has the luxury of just being able to suggest a measure and have everyone accept it; the measure itself has to be good, at least with respect to some criterion that people care about. You say your method is a "dynamic performance predictor." Well, yeah, almost all averages are, and especially so for averages which weight the most recent results more, which I think yours would do for some values of K. Why is yours better than others?

 

[reThinking the Cube]

Spoiler

I don't see how ELO rankings could be used in a situation where we don't have head to head matches.

I thought about the ELO actually but I don't think it would work very well for cubing to put it simply there is a very large location bias (ex. faz who lives in Australia has basically no competitors remotely close to him in 3x3 would not be able to make his rating very high)

This would defeat the point in using something like the elo which is best used when you want to compared expected performance relative to other competitors which was really it's only upside (the scrambles wouldn't matter as much because anyone at the same comp would get the same scrambles in the final and thus their ranking would changed based on how they placed there)
What you want to do would regain a scramble bias (which was the only elo advantage) give a heavy incentive to not compete if you were out of shape (for the purpose of these rankings) and give a large advantage to people who only go to a few competitions and get good times.

Even if we say we'll use some form of ELO (which just seems wrong to me, the usefulness of the ELO system comes from its ability to compare and rank, say, chess players, where wins and losses aren't a useful to directly evaluate a player's skill) that DOESN'T resolve ANY of the problems brought up in this thread- we'd still have to come up with a way to measure performance.

Using the ELO system wouldn't resolve any of the issues brought up in this thread, because we would still have to decide on what we would use to determine 'results.' For example, best average in a competition? That favors competitors who went to comps with more rounds of 3x3, is that what we want to do? Average of averages? That brings up many of the issues already discussed here.

Technically it is more correct, to view cubing as a competition between a cuber and a scrambled puzzle. Competitors aren't really competing directly against each other(as in chess), but rather, they are competing directly against a scrambled puzzle (problem). The time required to solve the problem however, can be used as a comparative measure.

A cuber rating system would be analogous to this chess tactics rater http://chess.emrald.net/index.php
Here is the description of that site's rating system:http://chess.emrald.net/ratinginfo.php

\( R_{new} = R_{old} + K(T_{actual} - T_{expected}) \)

Simply stated: new rating = old rating, changed by the difference between the actual results and the expected results, multiplied by a scaling factor, K.
ELO rating is actually an estimation of human performance.

We can therefore make CubELO = estimated time that this human will take to solve a cube.

This rating would be a *performance predictor*.

but just to clarify, what is the relation between \( R_{old} \) and \( T_{expected} \)?

Texpected = Rold(ScrambleBias/ImportanceBias). K would be based on non-standard rating deviation of the competitor(which is also based on other factors such as new cuber/or length of time since last rating period), and rating deviation of the scramble (which could be determined from all rankable results recorded for this scramble). This is not the specific mathematical expression of this algorithm, but is just a general description of the *form* (a type of Kalman filter (http://en.wikipedia.org/wiki/Kalman_filter)

 

[macky]

Before this thread becomes too old, I want to at least bring the main ideas to Stefan (who can access the WCA database).

A continual ranking, possibly with recognition at the end of each calendar year for a given year. The question is the ranking system.

(1) Mean of the last X (say 5) averages. All averages must be within the last 12 months.
(2) Point system, e.g. Ravi's.
(3) qq's majority-in-year average-average: Best (rolling) avg5 of consecutive averages, with the caveat that (a) a majority of those averages are in the correct year, and (b) all the averages are within one year of that year.

The motivation for (2) over (1) was to make it always worthwhile to compete. Because point systems have an added layer of arbitrariness and favors frequent competitors, I think (3), under which competing can only improve the ranking, is the best solution we've seen so far.

Any more thoughts to add?

 

[Lucas Garron]

I don't think I have, so let me voice my personal objection against mostly 1) but also a bit of 3).

Often, I don't have normal conditions for my averages. By helping out at most competitions, I'm often in the position where I'm forced to do an early round average at an inconvenient time, and without warm-up. When I perform terrible, I perform really terrible, and it doesn't help when I'm not properly warmed up.
I don't really want to have to choose between being helpful at a critical time and making sure to care about my personal rank. I can really imagine having done well at a previous competition, but having Tyson getting ready to yell at me: "Lucas, we don't have time for this [organizers practicing]. Stop whining and do your first round solves right now."

That was a more particular example, but in general, the proposals are asking competitors to perform well throughout several consecutive conditions. For various reasons, people like me rarely get the chance to focus during 5 consecutive rounds.

 

[Lucas Garron]

I don't think I have, so let me voice my personal objection against mostly 1) but also a bit of 3).

Often, I don't have normal conditions for my averages. By helping out at most competitions, I'm often in the position where I'm forced to do an early round average at an inconvenient time, and without warm-up. When I perform terrible, I perform really terrible, and it doesn't help when I'm not properly warmed up.
I don't really want to have to choose between being helpful at a critical time and making sure to care about my personal rank. I can really imagine having done well at a previous competition, but having Tyson getting ready to yell at me: "Lucas, we don't have time for this [organizers practicing]. Stop whining and do your first round solves right now."

That was a more particular example, but in general, the proposals are asking competitors to perform well throughout several consecutive conditions. For various reasons, people like me rarely get the chance to focus during 5 consecutive rounds.

 

[Tyson]

Yeah, I have to actually agree with Lucas here. And me yelling at him is somewhat accurate.

Right now with how things go, I know that Lucas will easily advance into the second round even if his first round average is terrible by his standards. That's why I ask him to go first so we have staff for the first round.

Organizers of competitions have been sacrificing their times for the sake of the competition since competitions have been being run. The people who make the competition happen don't have the luxury of warming up for 15 minutes before the round, since they were dealing with the round before, and then have to go early to get the next round started.

But yeah, Lucas is a good sport.

 

[AvGalen]

WOW, high quality discussion in here.

I am a bit afraid to mix myself in here with trivial real-life scenario's, but the last example of Lucas is really obvious to me. Right now, the best of the best don't go full-out in the first rounds (just like many of the best sprinters don't). The purpose of the first round (qualifying) simply isn't the same as the purpose of the last round (winning).

Another factor that seems to be completely overlooked in this discussion is improvement. Although I think that it is almost impossible to enter the cube-world and improve to world level within a year, some people still manage to do this. If someone would currently start at 35 seconds average, next competition do 17, next 12, then 10, then 8.5 he still wouldn't be among the top of the list which seems just plain wrong to me.

I am afraid reality is just to complicated to get translated to math in this situation