The Statistical Case for Combined Finals

I agree...A should advance.
At the Boulder 2009 comp, I ended up letting someone borrow my 2H cube, and I used my OH cube for 2H. I popped twice. If I hadn't popped, and instead got a normal solve (even a bad solve for me, 5 seconds slower than my average), I would have advanced to the second round. But I didn't, and I shouldn't have. I made a decision, and it led directly to poor performance. The competition rules shouldn't try to take away the risk I took.

In other cases, it may not be so easy to see...but why did competitor B pop? A combination of recklessness and inaccurate turning and a cube that was loose. It wasn't a random act of God, so to speak. Competitor B did things that led to his poor performance. He took a risk, and he lost the gamble.
 
A faster outlier not helping is perfectly consistent with "faster times can never hurt you"--it doesn't help, but it doesn't hurt either.

My point was/is - that 2 or more faster outliers could also help *unfairly*. That was relevant to the definition of 1.Fairness ....and all the rest of axioms. I have always understood your point(s). Do you understand mine?
No, you still don't understand what I said. Ravi, can you clarify what you meant by your fairness requirements? Did you mean: "Let S be a set of times, and let S' be another set formed by replacing one time in S with a faster time. Then S' is at least as good as S." ? If so, does what RTC is saying here make any sense?

Yes, that's what I meant by part 3 of my fairness requirements. (Parts 4 and 5 were more far-reaching but also more vague. Those were meant to carry double meanings: results should of course be ranked from fastest to slowest, but ranking/advancement systems should also be designed to make the best speedcubers most likely to get the best results.) I think RTC's point is that fast outliers, like slow outliers, can potentially provide an inaccurate view of how fast a competitor is. (Fair enough, although the outlier effect comes into play more often and to a greater extent on the opposite end.) His point is somewhat relevant to #4 and #5 under "fairness" in that ranking/advancement schemes should ideally dull the effects of freak outliers in both directions, not just one. I would say it's more directly relevant to the trimming section. IMO, lucky times are a relatively small issue (in fact, I think fast outliers are actually less common than a normal distribution would suggest). I think the main reason to trim fast times is to keep the symmetry with trimmed slow times: if we only trimmed the slow ones, then our averages would be unrealistically fast (and therefore incomparable with past records).

Still, though, RTC is welcome to clarify what he meant.

Item 2 under Fairness is " (2) that the rules must apply equally to all competitors."
Sorry, I inserted another item 2 after typing that. The "item 2" I was referring to there is now item 3, "that faster times can never hurt you and DNFs can never help." The original post is fixed now.

In the first couple of pages, I wasn't convinced that your suggestion was better enough than the current system to warrant a change. There were concerns brought up about cubers who go slower but safer in the first rounds, for example; your system is compatible with the average of 5 ranking, but it would certainly change competition strategies. If you already addressed this point, could you repost it? I can't find anything in this thread anymore.

This is probably the most substantial criticism of the mean-average system I've seen here. I'll try to address some potential effects of the change, both positive and negative, in another post (let's say tomorrow).
 
Dude, you still failed to argument against my example where a better cuber would be out for 0.01 if we used the median time...is it fair?

A: 10, 10, 11.5, 11.6, 12
B: 11.2, 11.3, 11.4, 15, 20

also, did someone read this?
To summarize: someone with a bad 2nd round would have minimal chances to win, in a strong competition (like needing a sub-9 average, dependind on the other people results, if we used Ravi's suggestion of taking the mean of 3 averages to determine the winner.

I read it, and I agree. He may have read it, but he's not interested in intelligent conversation anymore, just trolling.

No. I asked him my question first, and he has been avoiding answering me. Lighten up please.:p

Wait, who may have read what, and who asked whom what question? Pronouns are confusing.

About Pedro's first question, A should of course beat B.

About the second, I think it is absolutely fair for someone who performs poorly in the second round to suffer a lower placement, just as it is if that same person had performed poorly in the final round instead of the second. This is in accordance with my "a solve is a solve is a solve" guideline listed under "symmetry" in post number 103. As I've been saying since the first post of this thread, averaging more solves gives a greater chance for the best speedcuber to win. If Competitor D (or whoever) really is the fastest speedcuber in the competition, then there is a very small chance of him/her choking enough to lose under my plan--and a significantly smaller chance than there is under the current system. See for example my calculations in post 23:

According to the Mathematica work I discussed in my first post, Rowe had an 86.1% chance to beat Michal and an 82.0% chance to beat John under the current system, compared to 98.5% and 96.7% respectively under my system

Also note that a 14-second average can be enough to keep someone out of the finals in some competitions these days. Under my advancement system, the 14 would average out with the first-round result (11?) and make that competitor place ahead of someone who averaged, for example, 13 and 13. That is fair. But I do not believe averages of 11, 14, and 11 should beat averages of 11, 11, and 12. In my view, the winner should be the person who performed the best in the competition--the whole competition.

Suppose this scenario: [1st Round 3x3x3]
Competitor A gets (13.10), 13.30 ,14.51, 14.58, (14.90) = 14.13 avg (same as above)
Competitor B gets (9.20), 9.30, 14.50, 19.70, (DNF) = 14.50 avg (strong cuber+bad,bad luck)

Is it fair for B to advance because of the median (14.50 x 14.51)?

For this question, both competitors would also have the 2 slowest medians (14.50, 14.51), and since B clearly shows the potential to outperform A, I would favor B advancing. Yes, this IS fair. Using the median value here, would clearly be superior to using the 5-solve average to decide which competitor gets to advance.

What if A's third solve had been 14.49 instead of 14.51? Then A would beat B in spite of B's "potential to outperform." It seems what you mean by "potential to outperform" here is "best solve (or two)." Under the median system, B does not beat A because of the nines, but because his 14 just happened to be a little faster than A's 14. That's chance. I think we should dull the effects of chance by averaging more.
 
Last edited:
I have long been of the opinion that a five-solve trimmed average measures a speedcuber’s skill too unreliably to decide the winner of a competition.

My last thought on this topic

If I look at darts, these competitions (the big one) are more like a statistical format, with many rounds and attempt (legs). If cubers become closer and closer in their times at average you might consider other formats. This will always effect the lesser cuber as they will have less attempts or can not compete.

If you want to distinguish cubers they should compete 1 on 1 and use a format like darts. This can only be in a tournament format

Just my 5ct
 
Item 2 under Fairness is " (2) that the rules must apply equally to all competitors."
Sorry, I inserted another item 2 after typing that. The "item 2" I was referring to there is now item 3, "that faster times can never hurt you and DNFs can never help." The original post is fixed now.
In that case, RTC, I apologize for the confusion. But then I don't understand why the phrase "notwithstanding item 3" is necessary here at all. Perhaps I'm reading the criteria more strictly than Ravi intended.

Of course, RTC's post #104 still doesn't make sense since he only quotes items (1)-(3) and specifically suggests modifying (3):
1. Fairness. This means, for example, (1) that the overall result of the competition must be based on the individual results of the competitors and nothing else, (2) that the rules must apply equally to all competitors, and (3) that faster times can never hurt you and DNFs can never help.
I have a primary concern that I would like to include here. Currently, the trims for 5-solve average are being applied to prevent outliers (good&bad) from unfairly affecting the results. Choosing to trim 1 from each side of 5 solves is completely arbitrary, but does work well for the majority of cases. There is a problem though when a competitor gets more than one good or bad outlier in any one round. The outlier(s) in that case end up skewing the results *unfairly*. Once that solve, (which should be considered an outlier), gets included into any 5-avg, the current system cannot keep it from unfairly biasing all the results that are based on it. Suggest edit (3) to address this concern.
But that was a fault in the (format of the) argument, not in RTC's suggestion itself, and I shouldn't have bothered to point this out. If I start going after small things, he'll defeat me, no question. That's what happened to qq.

Seriously, RTC, I hope you're not trolling us. Because that would be too good.

Which competitor's performance should allow them to advance into the next round, A or B?
A obviously.
Why you no likeee B.Pedro?:confused:
You could have inferred from our posts that we would answer A. That's why I said in post #128 that you ignored this part:

macky said:
For a cuber that performs 100% CONSISTENTLY (every solve is the same time), the median would be EXACTLY THE SAME AS THEIR AVERAGE. Consistent cubers won't LOVE the median as much as inconsistent cubers, but they also should have no good reason to HATE it. Seems pretty *fair* to me, and to argue otherwise would appear to be unnecessarily *misguided*.
It seems like you understand that your system supports a very inconsistent cuber (messes up to 2/5 of solves) over a perfectly consistent cuber whose median is 0.01 worse than this cuber's. Then we can't really do anything to change your mind.
I respect your opinion - "If you mess up twice in a round, well, too bad.", but it doesn't HAVE to be that way.
Yes, there are situations where speedcuber who should have qualified is eliminated because of two unfortunate mistakes. Pedro has already addressed this. We, and I'm sure the majority of speedcubers, understand this and still don't agree with you. A similar consideration came up in 2004 when we decided stop awarding an extra solve for a POP. One argument was that a POP followed by a bad solve that gets trimmed would mean 2 bad solves out of 6, both not counting. Many members in the community thought that 33% inconsistency (less than your %40) was already too much to go unpenalized.

Your system would be great if all mistakes people ever made were timer failures. Competitors should then always perform around their true potential except for errors outside his control, so the median would be the best measure. In reality, most slow times are the competitor's fault to some degree. POPs and even corner twist DNFs are mistakes that can been avoided by better preparing the cube.
Now please actually address this.
 
Last edited:
Also note that a 14-second average can be enough to keep someone out of the finals in some competitions these days. Under my advancement system, the 14 would average out with the first-round result (11?) and make that competitor place ahead of someone who averaged, for example, 13 and 13. That is fair. But I do not believe averages of 11, 14, and 11 should beat averages of 11, 11, and 12. In my view, the winner should be the person who performed the best in the competition--the whole competition.

Yeah, I just made the number up. But even considering a not-that-bad average by D, 13.30, for example (suppose a pop/DNF and a bad lock-up, giving him a 14 time) and A messing up more (getting 11.9), D would still need a sub-10 average to win. If he's not capable at all of doing that, he wouldn't be really motivated in the final.

What I mean is that the mean-of-avgs system can be discouraging for people who get a 'bad' average, since it would pretty much ruin their chances of anything.

Michael/Kirjava idea of best avg in the comp seems not that bad, but I still don't like the fact that in the 1st round, where people may get different (and thus easier/harder) scrambles, a competitor may set an 'impossible' average.

If we were to use this system, I think we should do what I said somewhere here, using the best X results until the 2nd round to define who advances to the final. First round would obviously not change, but people with a good average would 'automatically' qualify.

Or we could do like athletics and have heats with 8 people, with the top-3 from each heat and the 4 other best results advancing :D
 
Michael/Kirjava idea of best avg in the comp seems not that bad, but I still don't like the fact that in the 1st round, where people may get different (and thus easier/harder) scrambles, a competitor may set an 'impossible' average.

I personally am still a bit hesitant on this idea, because of the approximately 15% chance that at least one competitor will get a, possibly, lucky enough average to skew their abilities and possibly win the competition for that reason, even if s/he is the fastest cuber there and would otherwise deserve the win.

With the current system the chance that at least one of 16 Fridrich solvers gets at least two skip solves in their final round average, possibly providing them enough of an advantage to win the competition from this luck alone, would be:
\( 1-(1-0.0033)^{16} \approx 0.05 \)

For the most part I agree with the arguments that the mean of 3 averages system does seem to give the better cubers a higher chance to win the tournament, but it also gives slightly less than top class cubers more of a chance to win the tournament as well, by luck. This could make competitions more exciting by allowing for underdog wins and upsets.

However, I would feel strange about celebrating the following results:
Cuber A 1st round, 2nd round, 3rd round averages: 10.00, 12.50, 13.00
Mean of 3 averages: 12.17

Cuber B 1st round, 2nd round, 3rd round averages: 12.40, 12.30, 11.90
Mean of 3 averages: 12.20

Cuber A wins under the mean of averages system, even though I would consider Cuber B to be a "better" cuber under competition conditions. I think such a scenario of averages is not too far fetched. Cuber A, perhaps, had a first round average with 2 skips (both PLL, one OLL and one PLL, both OLL, etc.). Based on Cuber A's global average of, probably, around 12.75 s/he received a significant advantage from the 2 skips over Cuber B who's global average is probably around 12.25 seconds.

Although I understand that luck is a part of cubing, it would seem strange that someone who, potentially, came in last place in the final with a 13.00 (not too unreasonable) could win the entire tournament because of a lucky first round average. Not only this, but Cuber A did progressively worse from round to round, while Cuber B did progressively better. I think doing "progressively" better should be rewarded over doing "progressively" worse when all other factors are otherwise equal. To me, and this is personal opinion, this makes the competition feel more like it brings out the best in each competitor.

I still think the current system, although it might not give the "better" cubers more chances to do well than the mean of averages system, gives the slightly less than top class cubers fewer chances to beat the "better" cubers due to luck alone. This is why I am in favor of the current system over the mean of averages system.

Chris
 
Last edited:
No matter how you try and remove it, luck will always influence results.

Michael/Kirjava idea of best avg in the comp seems not that bad, but I still don't like the fact that in the 1st round, where people may get different (and thus easier/harder) scrambles, a competitor may set an 'impossible' average.


People can still have different scrambles in the final, so this problem is not exclusive to the system I propose.
 
Cuber A 1st round, 2nd round, 3rd round averages: 10.00, 12.50, 13.00

[...]

Cuber A, perhaps, had a first round average with 2 skips (both PLL, one OLL and one PLL, both OLL, etc.). Based on Cuber A's global average of, probably, around 12.75 s/he received a significant advantage from the 2 skips over Cuber B who's global average is probably around 12.25 seconds.

First of all, let's be reasonable. There is no plausible way for a 12-13s performer to get a 10-second average based solely on luck. That would require something like the following:

13.00, 12.00, (5.00), (13.00), 5.00 -> 10.00

Of course, this assumes that the nonlucky solves actually averaged in the 12-13 range. Dropping this assumption, it is possible to conceive a slightly more plausible average like the following:

8.04, (7.89), 11.22, 10.74, (14.45) -> 10.00

but this still strikes me as highly unlikely, and at the very least it would require the cuber to perform much, much better than a 12-13s average would imply--not only on the 10 and 11, but also on the lucky solves, in order to make them that fast.

The question in question is this: How much can two lucky solves affect an average, or a mean average? I believe the effect is much less than you imagine.

Suppose, for example, that any given OLL/PLL skip cuts a given speedcuber's times by exactly 2 seconds. (This may be generous or slightly conservative, depending on the speedcuber, OLL/PLL, reaction times, etc.) Then, on the rare occasion when the cuber gets two skips within the same average in competition, we can imagine that his average drops by roughly 0.67 seconds. (In fact, it would sometimes be a smaller difference than this, since the counting lucky time might have otherwise been the slowest solve and therefore dropped--but let's assume 0.67.) Then, assuming there are three rounds in this competition, that speedcuber's mean average would drop by 0.22 seconds.

There's a trade-off here: single averages would give a small chance of a 0.67-second error, while mean averages give three times the chance of a 0.22-second error. (I think we can agree that the probabilities of three lucky solves in an average or two in each of two averages are negligible.) So which side of the trade-off is better? That is, which side gives a greater likelihood of a luck-based upset?

You assumed, quite falsely, that all finalists are fast enough to win with a single counting lucky time. It seems more natural to assume that the probability of being within 0.67 of the lead is roughly three times the probability of being within 0.22. (Actually, I would venture to say that it is somewhat more than three times, since competitors tend to thin out at the head of the pack and bunch up behind.) EDIT: A quick look down the speedcubing.com homepage gives 35 competitions, in which a total of 15 speedcubers were within 0.67 seconds and 3 within 0.22. This is a small sample, but the stats do support my assumption. So although the mean-average system would give competitors three times the chance to get lucky twice, there would most likely be at least three times as many people close enough to win with a 0.67 boost compared to a 0.22 boost. (A rough guess gives an expected value of 15/35 * 0.0033 = 0.00141 competitors passing the leader by luck under the current system, and 3/35 * 0.0099 = 0.00085 under my system.) Therefore I conclude that winning by luck is at least as unlikely, and probably more unlikely, under the mean average system compared to the single average system. In any case, the probabilities are on the order of a tenth of a percent, and we would be doing very well indeed if our competitions had anything near 99.9% accuracy.

Furthermore, I claim that lucky times are not only an extremely minor issue in competitions, but that they actually fall well inside a typical cuber's bell curve. For example, in the last few weeks, I've recorded 268 official-rules solves at home. My overall average of 268 is 12.37 seconds, and my SD is 1.70. But of all 268 solves, the best single time is 9.07, less than two standard deviations away from the mean. I don't recall whether the 9.07 was lucky, but there were several lucky times; they just weren't all that far from the mean. Theoretically, roughly 2.2% of times should be over two SDs away from the mean on each side of a bell curve. In my experience, virtually zero (in this case, exactly zero) solves are more than two SDs faster than average*. Conversely, there is much greater variance on the slower end of the distribution. What this means is that lucky times are a far less important issue than a bell-curve approximation would indicate.

As my statistics in posts #1 and #23 showed, mean averages are significantly more accurate than single averages assuming normal distributions. As I have argued previously, freak outliers on the slow end rarely affect that accuracy due to their rarity. And as I showed here, lucky solves do not make a difference either. Thus I conclude once again that mean averages provide a more accurate view of the relative speed of cubers in a competition than single averages do, not only assuming a normal distribution, but even when one considers unusually fast and slow times as well.




* Please tell me if this does not match your own experience.
 
In the first couple of pages, I wasn't convinced that your suggestion was better enough than the current system to warrant a change. There were concerns brought up about cubers who go slower but safer in the first rounds, for example; your system is compatible with the average of 5 ranking, but it would certainly change competition strategies. If you already addressed this point, could you repost it? I can't find anything in this thread anymore.

This is probably the most substantial criticism of the mean-average system I've seen here. I'll try to address some potential effects of the change, both positive and negative, in another post (let's say tomorrow).

Oops. Later today?
 
I have never been to a competition but it seems that the overall length of the event needs to be kept down but at the end of it you want the best performer on the day to win.

So is there a good reason why the final round/s (idk how many are typical) couldn't/shouldn't be averages of 12?
 
I see your point, Ravi.

I was gonna argument that a not-that-good avg ruins your chances of winning, if you're in a strong competition (11, 11, 11 x 11, 12.01, 10 (sub-10 avg needed to win)). But that sounds reasonable.

Just so I understand...your proposal is only for single competitions? To determine the winner? World rankings would still be avg5? Would we have a separate ranking for average of 15?

That would be hard, because not all competitions have 3 rounds, and some have 4.

Also, qualification into the 3rd and 4th rounds take into account the mean of the 2 or 3 previous avgs?
 
Just so I understand...your proposal is only for single competitions? To determine the winner? World rankings would still be avg5? Would we have a separate ranking for average of 15?

That would be hard, because not all competitions have 3 rounds, and some have 4.

Yes, I agree: it would make much more sense to leave the average rankings as they are. (There could, of course, be informal rankings for averages of 10, 15, 20, or whatever.) Macky started a separate thread on global rankings: http://www.speedsolving.com/forum/showthread.php?t=23261

Also, qualification into the 3rd and 4th rounds take into account the mean of the 2 or 3 previous avgs?

That was my intention, unless someone can see a reason not to do so.

I'll try to address some potential effects of the change, both positive and negative, in another post (let's say tomorrow).

Oops. Later today?

aaaaahhhh too busy with cubing, preparing for college, and living life to fulfill my forum promises. Sorry. Let's just say I intend to do justice to this at some point. In the meantime, others are of course welcome to weigh in.
 
Wow, long read, but I finished reading all of it carefully.

I hope to bring a fresh perspective into this discussion by stating a few points that will hopefully lead to discussion.

Analysis of the need for change
I hope we can all agree that the current system is actually working pretty well for both determining who proceeds to another round and for who wins. The most prominent example that has been brought forward and might possibly warrant a change was Breandans WC-winning average.
Back then, many people had tipped of Breandan as the outsider that did really well at home but couldn't handle the competition pressure (yet). His results during the competition were (as expected) relatively unstable, especially when compared with one of the other favorites "steady as a rock Zolnowski".
For all people there, the finals were as exciting as never before and even after everyone had done 3 and 4 solves, the winner had yet to be determined. If Zolnowski (or 1 of at least 3 others that still had a chance) had won, everyone would have thought he was the right person to become the WC. When Breandan turned out to have won everyone there thought he was the right person to win. Why? Because all of them had made it to the WC, through qualification and all other rounds and into the final. Nobody was interested in the results of those pre-final rounds anymore during the final. Breandan won because he was the best under the same circumstances as everyone else and the audience enjoyed the show.
This example alone should be a big warning sign not to change the current way of doing things.

Are there theoretical problems with the current system that could be improved?
Yes. (I was really tempted to keep it that short)
Ravi has posted a very interesting idea and reThinking the Cube proposed an interesting variation. The level of the discussion about luck, mean vs average and even peoples behavior (although distorted with some ego-bumping) was extremely high and convinced me that we should discuss and evaluate a better system untill

Reality (kicked me in the rear)
- If you are doing 5 solves in an average and you manage to screw 2 of them up so badly that it influences your results beyond repair you made a tactical error and will have to live with the consequences
- If you are doing 5 solves in an average and you manage to have 2 "undeserved" great solves that not only influence your own results but even change your ranking: Good for you, sucks for others, all in the game
- external circumstances (different scrambles in the same round, lighting-difference, helping with organising, and too many other things to list are a far bigger influence and problem
- Showing of by doing qualification rounds with beginner-method/wooden-cubes/keychain/one-hand is generally appreciated by the audience and enhances the fun-factor of a competition for many people
- Some people are consistant, others have a style or method that allows for greater variation within averages and between averages. This has been long understood by competitors and they have often accepted this as their strength/weakness

Conclusion
Keep the current system. The proposed changes are not solving enough real world problems to warrant a change that would influence the way competitors behave and experience during a competition. Dropping best and worst results solves almost all the problems and has been entranched in the community.

Other considerations
- 3x3x3 is special, so I understand that most of the discussion is about that, but to test theories it would be better to look at the extremes like 2x2x2 (skips) and magic (dnf)
- 3/5 is the preferred format for most puzzles (now also for megaminx) and I long for the day that it will be available for even more events like 6x6x6, 7x7x7 and 3x3x3 blind (I am serious about that last one, but please don't fill up this already hard to maneuver topic with replies to this)
- If you use a method that gives you big variations in time and you don't like that (4x4x4 reduction -> parities), then improve the method to get rid of the variations (learn all OLL's + parity), switch to a another method, or make up a better method. Another example would be to learn all 1LLL-algs for OLL's that lead to N-Perms (assuming N-Perms are your slowest PLL). Of course, most people just accept the variations in their times as a result of their method
- Even the best boxers walk into a lucky punch, the fastest skaters fall, the best cyclers get a flat tire, etc etc etc. Sport isn't an exact science and (luckily for the audience) it isn't predefined that the best person (on paper) will win in reality
 
Back
Top