The Statistical Case for Combined Finals

[incessantcheese]

i think that's the main issue here. a few of us feel that the winner of a competition is the person who stood up there in front of everyone and performed in that situation, while you feel a need to find a statistically "best" person out of the competition and give first place to him. it's not saying that one cuber is better than another - that's what the records pages are for. i would argue that for a lot of people, the records page matters a lot more than individual tournament wins and losses, and the best average is always shown on the records page.

 

[lilkdub503]

Surprisingly, I don't agree with qq on this one. I think a direct comparison to the Athletics events in Track and Field apply here.

Let's say Usain Bolt gets a world record time in the semifinals of the 100m of the London Olympics in 2012. Say, 9.51 or something (Sh*t, I can't believe that's even feasible). He doesn't win the gold medal, he doesn't get jack-sh*t if he doesn't get the fastest time in the final round. Let's say Tyson Gay runs 9.58 in the final, and Bolt runs 9.71. Gay walks away with the gold, Bolt walks away with the silver. End of story, it was about who got the best time when it mattered, when the lights, camera, and pressure were on. If we model our events and competitions similarly to athletics, than why should we change how our winners are decided?

 

[Ravi]

What if Rowe screwed up with a 16s average? Or a 14.5 average in the second round, and a 14 average in the third round? He was on a bad solve streak. But we all know that he is the best 3x3 cuber in the US, and he won and I think we all agree that he clearly deserved to win. If we did it your way, he wouldn't stand a chance.

Frankly, this could hardly be further from the truth. If Rowe averaged 16, or 14, or anything like that, then that would be a strong indication to me that he didn't deserve to win. What makes him the best speedcuber in the nation is not that he was able to average 10.93 once or sub-11 twice, but that he averaged well under 12 seconds in every single round. In fact, if we did it my way, Rowe would have had a much easier road to victory: since everyone else had a less-than-spectacular round or two, he could have afforded to slip once or twice himself. (In fact, any of his averages could increase by 2.88 seconds before dropping into second place. This means a 13.65 average in the final, which is slower than his worst single solve in the entire competition.) Also, remember that he only won two of the four rounds. If the cards had fallen differently, the final might not have been one of them.

To put it another way, let's compare the top three under my mean-average system:
Rowe Hessler (11.72, 11.04, 10.98, 10.93 -> 11.165)
Michal Pleskowicz (11.21, 12.85, 11.41, 12.07 -> 11.885)
John Tamanas (11.75, 10.83, 12.31, 12.85 -> 11.935)
In terms of single solves, Rowe had a mean of 11.20 and an SD of 1.34; Michal had a mean of 12.08 and an SD of 1.05; and John had a mean of 12.09 and an SD of 1.54. According to the Mathematica work I discussed in my first post, Rowe had an 86.1% chance to beat Michal and an 82.0% chance to beat John under the current system, compared to 98.5% and 96.7% respectively under my system*. So no, it is not true that my system would screw the top competitors like Rowe. Quite the contrary.

Also, note that under the best-average-whenever system proposed of Kir, riffz and qqwref, John Tamanas would be our national champion by virtue of his 10.83 average in round 2. While John does deserve better than the 7th place he got, I think most of us agree that Rowe deserves to win, and the best-average-whenever system fails to see that.



* The coefficient for "mean of 4 avgs of 5" is 0.238, from Mathematica:
StandardDeviation[Table[Mean[Table[avg[RandomReal[NormalDistribution[0, 1], 5]], {i, 4}]], {i, 1000000}]]
The probabilities come from the following codes:
N[CDF[NormalDistribution[0, Sqrt[1.34^2 + 1.05^2]*0.4765], 0.88]] -> 0.861002
N[CDF[NormalDistribution[0, Sqrt[1.34^2 + 1.54^2]*0.4765], 0.89]] -> 0.819895
N[CDF[NormalDistribution[0, Sqrt[1.34^2 + 1.05^2]*0.238], 0.88]] -> 0.98507
N[CDF[NormalDistribution[0, Sqrt[1.34^2 + 1.54^2]*0.238], 0.89]] -> 0.966513

 

[qqwref]

Let's say Usain Bolt gets a world record time in the semifinals of the 100m of the London Olympics in 2012. Say, 9.51 or something (Sh*t, I can't believe that's even feasible). He doesn't win the gold medal, he doesn't get jack-sh*t if he doesn't get the fastest time in the final round. Let's say Tyson Gay runs 9.58 in the final, and Bolt runs 9.71. Gay walks away with the gold, Bolt walks away with the silver. End of story, it was about who got the best time when it mattered, when the lights, camera, and pressure were on. If we model our events and competitions similarly to athletics, than why should we change how our winners are decided?

But why should he try in the early rounds? What's the point? He can do a casual run and still make it. The thing about running is that it's a race, and the winner of the finals is important because it's visually obvious. You can watch the race and see them ahead. But cubing isn't a race, and you don't have such an obvious and intuitive way of judging who's best. Times are entered in later and a ranking decides the results. Keep in mind that the first and last competitors in the finals of Nationals 2010 went about an hour apart. The way it's set up in many cases, it's not "perform the best NOW" as much as "do a few averages, but we will ignore all but the last one".

Anyway, giving the win to the best average in the competition would introduce an interesting dynamic to the finals. You'd know what bar there was to break from the previous results. Right now my experience as a finals spectator is that you're watching some fast people do fast solves, but you don't really have the memory to figure out exactly who wins, so all you can do is watch and then wait for the data entry guy to calculate who won. Not too exciting unless an amazing average pops up. But if everyone was trying to beat a certain mark, I think it would be more interesting because you'd be watching people try to break that record, and it could be exciting just like watching a WR attempt. It becomes easy to pay attention to and keep track of.

I think most of us agree that Rowe deserves to win

Rowe should have won!

 

[Ton]

I have long been of the opinion that a five-solve trimmed average measures a speedcuber’s skill too unreliably to decide the winner of a competition.

You forget a lot of other factors thats are decisive

1) A final is not a second or first round, the pressure on the top cubers is only when they enter a final. Hence a cuber that can take the pressure well has a better change
2) The scramble matters on the method you use -especially on a 2x2-, how slight the difference maybe. Currently a 0.01 can be decisive!
3) In a final we often do one solve at the time compared to other rounds, these are other circumstance. Especially is you compare them with a qualification round or a first round
4) In a final all competitors will have the same scramble, this is not true for the other rounds
... and there are more
These factors you did not take into account

So do you not convince me

 

[Ravi]

With such a small sample size (20 solves) you can't expect most of the people to get a really bad or a really good solve, and that would explain why many of the people had a moderately negative calculated kurtosis. Instead of just looking at how many people had a positive kurtosis, and saying that 9 of 16 is indistinguishable from 50%, what if we analyzed all 16 people together and took the average kurtosis? I get +1.15, supporting my conclusion that a typical person would have a higher-than-average kurtosis. My whole point was that outliers are more common than a normal distribution would imply (but still pretty rare in most cases) so you can't expect every sample of 20 solves to include one of them.

It may not entirely make sense to calculate the kurtosis of a sample including solves by different people with different average times (and I'm not sure whether this would be more likely to increase or decrease kurtosis), but I'll concede the point. Large samples are more likely to have positive kurtosis, hence the necessity for trimming.

It sounds to me like "very negatively affected in one of the rounds" is a euphemism for "not a consistently fast speedcuber." Am I missing something?

I don't mean that someone has a high SD or messes up a lot, I mean that someone is physically or mentally not in their best condition when a certain round takes place. I was once at a competition where the organizer forgot the scorecards where we were staying and a group of people (myself included) had to drive back and fetch them. When we got there we were pretty much yelled at for holding the competition up and had to do the 3x3 first round immediately. I was not warmed up, annoyed, and nervous, and of course I did badly (enough in fact to not make the next round). Are you going to say this is my fault for not being consistently fast? Cubers are people, not sources of statistical noise.

Not your fault, and this can of course happen in any round (and any sport). But I'm not sure that any averaging scheme can really fix that. My mean-average proposal is intended to average out any temporary changes as much as possible, but I think discarding all but the best individual average is an overreaction: see, for example, the closing remark in my last comment.

 

[Dene]

Oh I forgot another point. What about people that only get to do one average? It would be inconsistent for the winner to have 4 averages averaged, and for people in the first round only to have one average. Unless this inconsistency can be justified, this cannot happen.

 

[qqwref]

Oh I forgot another point. What about people that only get to do one average? It would be inconsistent for the winner to have 4 averages averaged, and for people in the first round only to have one average. Unless this inconsistency can be justified, this cannot happen.

I think the idea is that only people who make the final are worthy of being ranked at all.

But if you rank by the best average achieved, everyone who competed in the event can be included in a single overall rank. I think this would be pretty cool.

 

[riffz]

Oh I forgot another point. What about people that only get to do one average? It would be inconsistent for the winner to have 4 averages averaged, and for people in the first round only to have one average. Unless this inconsistency can be justified, this cannot happen.

I think the idea is that only people who make the final are worthy of being ranked at all.

Ew. That would discourage slower competitors from showing up to comps. Not a good idea. (I realize you aren't supporting this.)

 

[nck]

What about a fourth round for the top five cubers from the third round to do averages of 12? Or even for the top 3cubers if time is an issue.

 

[gasmus]

I agree an average of 5 is not completely fair but i dont think there is a better option. i would love if it was the best average of the whole competition as the winning average but that would only really be fair if everyone competed in every round(or at least the same number of people in the 2nd round onwards).

A mean of averages may be more accurate and fair but in my opinion would change how a lot of people compete, too much for what its worth. in my case(and i'm sure many other cases) i always save my energy for the finals, or maybe go for fast singles in the first and second rounds. with a mean of averages i would go for more consistent averages which in my opinion is much less fun. i much prefer the idea of everything relying on the finals. if any changes were to be made i think it should be to the finals only.

The most fair change i could think of would be best of 2(or more^^) averages of 5 for the finalists, though i'm sure that has its problems too.

btw, i dont think naming names is fair in this case since almost every competition doesnt have a clear winner. Both mine and Rowe's averages were all full step completely non-lucky solves so luck didnt come into it. the finals are just when it counts.

 

[Mitch15]

ill admit i didn't read all the extremely long posts of this thread, but i agree with garron about different rounds carrying different goals and meaning. and i definitely agree with separate rounds and a final round deciding all. that's how competitions work.

would you decide wimbledon on how who had the best set win ratio of the competition?
of course not. now i know this isn't a 1v1 situation. but there is a lot to be said about a final where you have no laurels to rest on and everything rests on your next 5 solves. cubing under pressure is a big part of who is a better cuber.

 

[qqwref]

would you decide wimbledon on how who had the best set win ratio of the competition?

Should the World Series teams be decided by who got the most points in their most recent game? Not all sports work the same, or should. Tennis is single elimination anyway, but many sports (including cubing) are not optimally run in such a directly competitive way.

 

[macky]

Performing when it counts is of course important. The problem with comparisons to other sports is that speedcubing inherently involves chance. I don't know how much of a top cuber's time variation is from pure chance, but it's considerable. Even when performing the very best, a top CFOPer's times will vary > 0.5 second just by the difficulty of last layer cases--and we aren't even considering F2L. With the continuing clustering of times near the top, this variation is starting to become enough to significantly affect the ranking in a competition.

I thought about all this back in WC07 and with my prayer before WC09. Because of complications such as those mentioned in this thread, I'm fine having the winner of the last round be the winner of the competition. Or if an organizer feels strongly, he can use the fastest average or the mean of averages. In any case, there's no rule change needed.

The winner of a competition is the cuber who performed "the best" (in a predefined sense) at that competition, not necessarily the best cuber. I therefore don't see too much need to change the competition format so that the best cuber has a better chance of winning the competition.* What I've suggested instead since 2007 is an official yearly overall ranking (see this thread for an example).

* But I'm sure people will soon be needing my prayer more often!

 

[qqwref]

Oh, here's another small reason to rank people in a competition by the best average they got: it would mirror the format of the WCA rankings (which are also best average, not most recent average) which as we all know are more important than any individual competition's results (except perhaps Worlds).

 

[reThinking the Cube]

Then any speedcuber can be represented essentially by two numbers: his or her mean and standard deviation. The purpose of taking averages, of course, is to decrease the standard deviation while preserving the mean. The question: How much do standard deviations decrease when averages are used?

Can you calculate comparisons using 15 solves, with the fastest (2) and slowest (3) solves discarded - leaving 10 for a competition trimmed average?

I like the idea of using the median average of 10 taken from 15 solves, even better than taking the best trimmed average, or a median average from 3 individual rounds of 5 solves.

I'd rather that the person who set the best average at the comp won, no matter what round it was in.

If the best average is going to win, then we should make it the ONLY average (all solves factoring into 1 competition average). By starting over each round, we make some results not as important as others, but the way things are formatted now, Lucas is correct:

An important fact to consider: In the earlier rounds of a tough competition, cubers often try to make it to the next round rather than a fast average.
Two DNFs are unacceptable, adding 2 seconds to your average can be. In the finals, however, it's about the maximum speed and all its risks.

You have a point: It would be (and is) unfortunate for a serious competitor to be eliminated early because of DNFs.

Possible Solution: double trim on 5th solve leaving a single median value as the basis to eliminate early. From that point on, averages (and competition rank) can be computed based on the averages of the median values gotten so far, after double trimming. This will prevent early elimination of serious competitor, unless they DNF in 3 out of the 5 attempts, and in that case, they deserve to be eliminated!

Ravi, there are two important factors that you are not considering. First: there is always a chance of a very nice time from a skip or a very bad time from a pop or similar problem. This even extends into averages as it is always possible in theory to get two DNFs (or bad times) or two skips (or very nice solves). Very bad times are more likely in a competition due to pressure, especially in a final like the one at Nationals 2010 where every competitor did their five solves in front of hundreds of people, the only one on stage in a silent and huge room. So the standard deviation approach doesn't do well. You can tell because your numbers give an average of 5 as varying LESS than a trimmed average of 5; this isn't true in real life as many competition averages are strongly hurt or helped by the removal of the best and worst times. So considering times to fall in a strict bell curve is an approach that may be easier but will not give a good model.

I entered into Mathematica all of the statistics for everyone who made the finals (20 solves for each person, except for the two who DNFed earlier), and calculated kurtosis values, which are listed below in the order of final rankings: ...<snip>

Thus it is only barely true that, as you argued, most real speedcubers' distributions in competition are more leptokurtic than a normal distribution, and that effect is almost entirely negated and even reversed in the process of trimming averages. Normal distributions aren't perfect models, but they're surprisingly accurate, especially when we're trimming the averages anyway.

What would the kurtosis values be for those competitors if using all 15 solves, the slowest 3, and fastest 2 solves were trimmed out first?

So each cuber who makes it to the final, and gets to do 3 averages, has an approximately 1% chance that one of their 3 averages has at least two skips. With 16 competitors making it to the final, the probability that at least one of those competitors gets at least one average with at least two skips is:

\( 1-(1-0.0099)^{16} \approx 0.147 \)

So this competition has an approximately 15% chance that the winner wins by luck, and not necessarily by skill.

How does double trimming (as described above) on a set of 15 solves compare with this?

Anyway, giving the win to the best average in the competition would introduce an interesting dynamic to the finals. You'd know what bar there was to break from the previous results. Right now my experience as a finals spectator is that you're watching some fast people do fast solves, but you don't really have the memory to figure out exactly who wins, so all you can do is watch and then wait for the data entry guy to calculate who won. Not too exciting unless an amazing average pops up. But if everyone was trying to beat a certain mark, I think it would be more interesting because you'd be watching people try to break that record, and it could be exciting just like watching a WR attempt. It becomes easy to pay attention to and keep track of.

Excellent point. This would be perfect if it incorporated all the solves, instead of just using the results from the best *lucky/easy* round.

 

[Tim Major]

I'd rather that the person who set the best average at the comp won, no matter what round it was in.

If the best average is going to win, then we should make it the ONLY average (all solves factoring into 1 competition average). By starting over each round, we make some results not as important as others, but the way things are formatted now, Lucas is correct:

An important fact to consider: In the earlier rounds of a tough competition, cubers often try to make it to the next round rather than a fast average.
Two DNFs are unacceptable, adding 2 seconds to your average can be. In the finals, however, it's about the maximum speed and all its risks.

You have a point: It would be (and is) unfortunate for a serious competitor to be eliminated early because of DNFs.

Possible Solution: double trim on 5th solve leaving a single median value as the basis to eliminate early. From that point on, averages (and competition rank) can be computed based on the averages of the median values gotten so far, after double trimming. This will prevent early elimination of serious competitor, unless they DNF in 3 out of the 5 attempts, and in that case, they deserve to be eliminated!

I agree with Kir, however, I don't think your solution is feasible due to practical reasons. Do you really think that would work in competitions? Why make a simple thing complex? What Kir is suggests is great I think, however, in Football/Baseball/Basketball/etc, even if team 1 beats team 2 sometime throughout the season, if team 2 wins in the final, then team 2 wins.
Whilst I agree with Kir, I think at the same time, it's more exciting, to have a final, that actually matters, and to have a winner determined by the winner of the final.
What are you going to do, if there is more than one group/heat in the earlier stage of an event. That would be unfair. If you have a cuber who averages 10, Vs. another cuber averaging 10, but they're in different groups/heats, maybe one of the two will be unfairly advantaged, by having an easier set of scrambles (I'm not saying this will always happen). If you're going to have whoever gets the best average throughout the whole competition wins, then there can't be separate groups/heats.
If a long jumper trips in all their runs on the day of the Olympics, they don't win if they had a better jump at a previous official competition.

Spoiler

I've convinced my self by these arguments, that I think whoever wins in the final, wins overall Sure sometimes "unfair" but not really. Such is life, and life isn't always fair.
Often in the final, there is way more pressure than the other rounds, and that's part of winning, you need to be able to hold up under pressure.

 

[qqwref]

If a long jumper trips in all their runs on the day of the Olympics, they don't win if they had a better jump at a previous official competition.

You need at least ONE good result in the competition...

(You did describe how the official overall rankings work, though And I'm a fan of those. I like the idea of trying to improve your personal best a lot more than the idea of trying to beat everyone else on these particular scrambles.)

 

[Gavin]

An important fact to consider: In the earlier rounds of a tough competition, cubers often try to make it to the next round rather than a fast average.
Two DNFs are unacceptable, adding 2 seconds to your average can be.

In the finals, however, it's about the maximum speed and all its risks.

I think this pretty much hits the nail on the head. At Nationals, I remember someone (might have been Rowe) saying that the first round average didn't really matter as long as they got into the next round. Now you could either think of it this way, or you could think of it as every round I have a chance to get a new WR average. I think at Nats though, more people were focused on winning the competition.

 

[Olivér Perge]

I see no problem with the current format. The final round counts, so do your best at it! If you are going for a good single, saving yourself or just practicing in the former rounds, that's up to you.

In my opinion it has never worked really well (at least for me). Like does anyone have a story when she/he wanted to go for a good single/average and got exactly what she/he wanted, not more and not less? I think expecting times is, what really killes the official solves. Overthinking is really bad for competing, in my opinion. (Two examples: Breandan at the world chamipionship: no one expected him to win, including himself, many expected Tomasz to win. At Austrian Open I was talking with Stefan Huber and he told me he wants a 10.5-ish average. Well, it didn't work out quite well. )

I don't like the idea of making the best average in the competition the winner average.