The Statistical Case for Combined Finals

[Pedro]

If you had done really well in the 1st round, and both of the 2 worst times came in the 2nd round, then YES it would be fair to trim them. That is NOT how it is done now, but the main idea behind the extra trimming is to give the Cubers with the best overall performance at that point in the tournament, the fairest chance of advancing into the final round. It should be noted that even though you may have advanced into the final with 2 poor times in the 2nd round, those poor results could eventually impact on your overall performance, since you only get a finite# of trims (applied to all 15 solves) for calculating your tournament average (my way).

I don't see why it is fair. Why should my 1st round be a bias to my 2nd round? Different conditions (weather, audience, light, whatever) happen on each round, even our physical condition, cube condition is different. In my opininion, someone with a 10.xx and a 20.xx average shouldn't take the spot in the final from someone with two 12.xx averages.

The standard trimming that is done right now isn't considered to be a *discount*. Besides, all competitors' rounds would be trimmed the same, no matter which format. Trimming to median value will just weed out more of the bad/good luck solves that would have been used (IMO unfairly) to decide who gets to advance. I respect your opinion - "If you mess up twice in a round, well, too bad.", but it doesn't HAVE to be that way. Look at Sikan Li in round 2 of the Bayview Hills Open 2010. Trimming it my way, he advances to the finals, which I think would have been fairer, based on his OVERALL performance. And in the same tournament, an even clearer example would be Phillip Espinoza's final round disaster.

Sikan Li avg of 8/10 = 16.56, avg of 6/10 = 16.41
Cameron Brown avg of 8/10 = 16.08, avg of 6/10 = 16.10
Alexander Lin avg of 8/10 = 16.21, avg of 6/10 = 16.10

So, no, Sikan didn't deserver to qualify, in my view

Thank you for presenting this question. I am assuming these times are for 3x3x3 round 1. With my way there is indeed a .01 difference going to B. This will only actually make a difference, if and only if, the last spot for advancement into round 2 is taken by B. Now that scenario is not very likely, but in either case then, both A and B are the SLOWEST competitors coming from round 1, that can advance. Also note that those 2 poorer times of B will not go away, and may impact latter on (my way). It may also be the case that B is really faster than A, and just got a terrible round.

My main concern is to prevent the strongest competitors from being eliminated early simply due to bad luck.

Suppose this scenario: [1st Round 3x3x3]
Competitor A gets (13.10), 13.30 ,14.51, 14.58, (14.90) = 14.13 avg (same as above)
Competitor B gets (9.20), 9.30, 14.50, 19.70, (DNF) = 14.50 avg (strong cuber+bad,bad luck)

Is it fair for B to advance because of the median (14.50 x 14.51)?

For this question, both competitors would also have the 2 slowest medians (14.50, 14.51), and since B clearly shows the potential to outperform A, I would favor B advancing. Yes, this IS fair. Using the median value here, would clearly be superior to using the 5-solve average to decide which competitor gets to advance.

Hmm...so you get to decide what IS fair...

The problem with your proposal is exactly what I presented. Yes, it may qualify a good cuber with some bad luck, but it may qualify a bad cuber which had a 0.01 luck, so this is NOT FAIR to me

I say we keep the current system.

 

[Lucas Garron]

Don't forget Arrow.

There is no good ranking/advancement system.

Personally, I'd rather keep the current system the same until a much better idea comes along (i.e. not an arbitrary tweak). Macky's proposal seems like a better idea.

 

[Ravi]

There is no good ranking/advancement system

That sounds like a bit of a stretch to me. Certainly, there is no perfect ranking/advancement system, but by invoking Arrow's theorem you're assuming all of its hypotheses: "unrestricted domain, non-dictatorship, Pareto efficiency, and independence of irrelevant alternatives." I'm honestly not sure which of those hold and which fail for the current system, but I would describe the existing system as "good"--just not the best.

But this raises a very important question: What do we mean by a "good" ranking/advancement system? I've listed below several qualities that I believe a "good" system would need to have. Not all are rigorously defined, of course.

1. Fairness. This means, for example, (1) that the overall result of the competition must be based on the individual results of the competitors and nothing else, (2) that the rules must apply equally to all competitors, and (3) that faster times can never hurt you and DNFs can never help. Furthermore, (4) the ranking system should be designed to rank competitors from fastest to slowest as accurately as possible, within the the bounds listed below; and (5) the advancement system should similarly aim to advance the fastest competitors after each round.

2. Practicality. This means, for example, (1) that competitors should not be expected to complete excessively many attempts, nor should organizers be expected to schedule excessive amounts of competing time; (2) that the rules should be simple enough for most speedcubers to understand; and (3) that the results should be reasonably easy to calculate.

3. (Limited) forgiveness for outliers. This means (1) that DNFs and occasional outliers, which are generally considered relatively unimportant, should not affect a competitor's result drastically, notwithstanding item 3 under "Fairness" above; but (2) that this forgiveness (ie. trimming) should not be taken to the extent of invalidating significant, near-average solves. A good guideline provided by the current system is (3) that roughly the middle 60% of attempts should count.

4. Similarity to the current system. This means, in addition to "Compatibility" below, (1) that the overall concept of a competition should not be changed unnecessarily, and (2) that excessive rule changes should not be applied lightly. Furthermore, (3) no significant change should be made without some consensus.

Additionally, I think the following qualities would be preferable, although perhaps not entirely necessary.

5. Compatibility with previous records. Single-solve records will presumably not be affected, but (1) the existing average-of-5 format should be preserved. Furthermore, (2) it should be preserved not only in name; if records are based on averages of 5, then the competition should demand good averages of 5 for advancement and/or ranking, to avoid forcing a competitor to decide between competing for ranking and competing for records.

6. Symmetry and uniformity of rules. This means (1) that, psychology aside, all solves should preferably be equal in value; that is, so to speak, that "a solve is a solve is a solve," and (2) that as far as practical, the same rules should apply throughout the competition.

Please comment on the guidelines above, and I will edit accordingly. I would really like to reach a consensus on this.

 

[reThinking the Cube]

I'm honestly not sure which of those hold and which fail for the current system, but I would describe the existing system as "good"--just not the best.

I concur.

1. Fairness. This means, for example, (1) that the overall result of the competition must be based on the individual results of the competitors and nothing else, (2) that the rules must apply equally to all competitors, and (3) that faster times can never hurt you and DNFs can never help.

I have a primary concern that I would like to include here. Currently, the trims for 5-solve average are being applied to prevent outliers (good&bad) from unfairly affecting the results. Choosing to trim 1 from each side of 5 solves is completely arbitrary, but does work well for the majority of cases. There is a problem though when a competitor gets more than one good or bad outlier in any one round. The outlier(s) in that case end up skewing the results *unfairly*. Once that solve, (which should be considered an outlier), gets included into any 5-avg, the current system cannot keep it from unfairly biasing all the results that are based on it. Suggest edit (3) to address this concern.

3. (Limited) forgiveness for outliers. This means (1) that DNFs and occasional outliers, which are generally considered relatively unimportant, should not affect a competitor's result drastically, notwithstanding item 2 under "Fairness" above; but (2) that this forgiveness (ie. trimming) should not be taken to the extent of invalidating significant, near-average solves. A good guideline provided by the current system is (3) that roughly the middle 60% of attempts should count.

For a system to be *perfect*, it would have to trim ALL the outliers (good&bad), and NONE of the others. A reasonable ideal then, is to fairly trim as many of the outliers as possible, from all the rest. The current system, is deriving an average using the middle 60%, but this is an arbitrary compromise that is assumed to approximate the ideal. Sometimes it does, sometimes it doesn't. Anytime there is more than 1 good/lucky, or more than 1 bad/unlucky solve, in a round of 5, the arbitrary trimming for 5-avg. yields a result that is polluted. This is not trivial, since all tournament results, whether based on a single round, or an average of averages, and all rankings, are negatively affected by outlier polluted 5-avg. IMO - deriving some results based on medians, would be much better, since outliers would all get (Perfected/Unlimited) forgiveness.

Please comment on the guidelines above, and I will edit accordingly. I would really like to reach a consensus on this.

Most excellent guidelines, nice work. I can pretty much agree with all of it, except for the concern(s) mentioned above.

 

[macky]

1. Fairness. This means, for example, (1) that the overall result of the competition must be based on the individual results of the competitors and nothing else, (2) that the rules must apply equally to all competitors, and (3) that faster times can never hurt you and DNFs can never help.

I have a primary concern that I would like to include here. Currently, the trims for 5-solve average are being applied to prevent outliers (good&bad) from unfairly affecting the results. Choosing to trim 1 from each side of 5 solves is completely arbitrary, but does work well for the majority of cases. There is a problem though when a competitor gets more than one good or bad outlier in any one round. The outlier(s) in that case end up skewing the results *unfairly*. Once that solve, (which should be considered an outlier), gets included into any 5-avg, the current system cannot keep it from unfairly biasing all the results that are based on it.

(emphasis mine)

@reThinking the Cube: I don't think you understand Ravi. He's imitating the kind of axiomatic approach used for example for Arrow, beginning with the bare minimum requirements and trying to construct a method that satisfies them. The "fairness" you speak of has nothing to do with the very basic fairness that Ravi describes (just go through items 1, 2, and 3). You're letting your own (I'd say misguided) sense of fairness shape both your suggestions and your arguments, without much basis.

First, read these comments:

Hmm...so you get to decide what IS fair...

I find it silly to be talking about what's 'fair' or not.

Everyone is subject to the same rules, regulations, and procedures. It's not unfair for a good cuber not to advance if they don't perform well. It SUCKS, I understand that, but it's not unfair to them.

Then, back up. With the understanding that it's difficult to measure any fairness that goes beyond the bare minimum that Ravi suggests, what reasons can you provide to support that your method is more fair than the current system? I actually think your method is less fair because of the following comment from Pedro, and you need to address it:

The problem with your proposal is exactly what I presented. Yes, it may qualify a good cuber with some bad luck, but it may qualify a bad cuber which had a 0.01 luck, so this is NOT FAIR to me

Also, regarding what you wrote here (emphasis mine):

For this question, both competitors would also have the 2 slowest medians (14.50, 14.51), and since B clearly shows the potential to outperform A, I would favor B advancing. Yes, this IS fair. Using the median value here, would clearly be superior to using the 5-solve average to decide which competitor gets to advance.

"Potential to outperform" suggests that you want the best cubers to advance. I, and also Pedro from what I've read, believe that the cuber who performs "best" (in some predefined way) _at the competition_ should advance. I wonder if you're just starting with different idea of what a competition ranking should achieve. Neither goal conflicts with Ravi's fairness requirements.

In any case, I'd suggest writing up your argument in its entirety: what a competition ranking should achieve, your idea of fairness, and why your method is significantly more fair than the current system, enough to warrant a rule change.

 

[Pedro]

Another hard thing to define is what is an outlier. How far should it be from the median/average value to be considered so? I don't have a good answer...

For someone with a 11 seconds average, a 15 time is just terrible, but a non-lucky 9 is probably an outlier, since he/she gets it not that often. But is not that far from the average. Should it be counted out?

Also, different speeds generate different ranges of times. I average high-12's now (global average, I can do 11.50 when I'm doing good) and I get non-lucky times from 9 to 16. I can also mess up and get 17 or 18, which doesn't give an equal range in both directions (up and down from the average). How we decide what should and what shouldn't count?

Someone who is really consistent may complain about using just the median value. For example:
Competitor A: (11), 11.2, 11.5, 11.6, (11.8)
Competitor B: (10), 11.3, 11.4, 13.7, (15.8)

With the current system, A clearly beats B. Using just the median, B would get a better ranking and maybe kick A out of the next round.


One that that could work is making a competition ranking. Everybody does an average, and the best X advance (just like it is now). Then, in the second round, the top Y results advance. Then we'd have to go by best average in the comp to determine the winner. It would be kinda weird to have a different system to determine who advances and who is the winner.

I don't like this very much because someone who gets a really good average in the 1st round may not even try on the 2nd because he/she will surely qualify. I prefer "forcing" people to perform good in every round if they want to advance/win. If someone can do good in all rounds, he's probably a good cuber, which is what we (supposedly) want to find

 

[blade740]

For a system to be *perfect*, it would have to trim ALL the outliers (good&bad), and NONE of the others. A reasonable ideal then, is to fairly trim as many of the outliers as possible, from all the rest. The current system, is deriving an average using the middle 60%, but this is an arbitrary compromise that is assumed to approximate the ideal. Sometimes it does, sometimes it doesn't. Anytime there is more than 1 good/lucky, or more than 1 bad/unlucky solve, in a round of 5, the arbitrary trimming for 5-avg. yields a result that is polluted. This is not trivial, since all tournament results, whether based on a single round, or an average of averages, and all rankings, are negatively affected by outlier polluted 5-avg. IMO - deriving some results based on medians, would be much better, since outliers would all get (Perfected/Unlimited) forgiveness.
)

There are a few problems with your idea:

For a system to be *perfect*, it would have to trim ALL the outliers (good&bad), and NONE of the others

Taking the median would certainly trim more non-outlier times than the middle 3. Occasionally it would trim an extra bad solve, but for cubers who perform fairly consistently, it would trim average results unnecessarily.

Anytime there is more than 1 good/lucky, or more than 1 bad/unlucky solve, in a round of 5, the arbitrary trimming for 5-avg. yields a result that is polluted.

What about puzzles that have a 50% chance of parity, like 4x4 and square-1? A 50% chance of parity on any given solve means there's a 50% chance of 3 or more parities in an average of 5. What this means is that 50% of the time, the median solve will be a parity case (and so slower than the cuber's overall average) and half the time it won't (so it will probably faster than the cuber's overall average). This leaves the entire average up to chance, as it's basically a 5-second coin flip.

To give an example, what if two solvers' times in a 4x4 round were as follows:

A: 46, 47, 48, 53(p), 54(p)
B: 44, 45, 50(p), 50(p), 51(p)

Solver B is obviously the faster solver, but because he got 3 parity cases instead of two he is ranked lower. If solver B's third solve had not had parity, it would be closer to 45, putting him 3 seconds ahead instead of 2 behind.

 

[reThinking the Cube]

Originally Posted by Ravi
1. Fairness. This means, for example, (1) that the overall result of the competition must be based on the individual results of the competitors and nothing else, (2) that the rules must apply equally to all competitors, and (3) that faster times can never hurt you and DNFs can never help.

I have a primary concern that I would like to include here. Currently, the trims for 5-solve average are being applied to prevent outliers (good&bad) from unfairly affecting the results. Choosing to trim 1 from each side of 5 solves is completely arbitrary, but does work well for the majority of cases. There is a problem though when a competitor gets more than one good or bad outlier in any one round. The outlier(s) in that case end up skewing the results *unfairly*. Once that solve, (which should be considered an outlier), gets included into any 5-avg, the current system cannot keep it from unfairly biasing all the results that are based on it.

You're letting your own (I'd say misguided) sense of fairness shape both your suggestions and your arguments, without much basis.

Fair enough. You are entitled to our opinion. IMO - maybe YOU are letting your own (I'd say misguided) sense of fairness shape both your suggestions and your arguments, without much basis. That shoe fits you *fairly* better than me anyway.

The primary reason (basis) I expressed my concern in the above post, was that the definition of "1.Fairness" was being established, and I felt that the (I'd say misguided) way it was worded (i.e. "faster times can never hurt you", but - what about faster outliers not helping you?) created an exemption in the basic definition, for even considering that extra outliers being included in 5-avg, could even be an issue relating to fairness. This definition directly tied in with (3), and how outliers are to be trimmed *fairly* according to the very definition of 1.Fairness.

I think it was *fair* for me to comment, for the sake of trying to achieve a *fair* definition of *fairness*.

How is this answer NOT *FAIR* TO YOU?

IMO - using outlier polluted 5-avgs is NOT MORE FAIR than using averages/results/advancements based on some other methods with less outliers and more solves.

Misguided?

Can you please show me actual tournament examples where you feel outlier polluted 5-avgs yielded results that were *FAIRER* than using any other possible method?

Also, regarding what you wrote here (emphasis mine):
Originally Posted by reThinking the Cube
For this question, both competitors would also have the 2 slowest medians (14.50, 14.51), and since B clearly shows the potential to outperform A, I would favor B advancing. Yes, this IS fair. Using the median value here, would clearly be superior to using the 5-solve average to decide which competitor gets to advance.

"Potential to outperform" suggests that you want the best cubers to advance. I, and also Pedro from what I've read, believe that the cuber who performs "best" (in some predefined way) _at the competition_ should advance.

Yes, I agree. What we are debating, is the possibility of changing this *predefined way* to score the performances. Either way then, the one who performs best will advance.:tu

For a system to be *perfect*, it would have to trim ALL the outliers (good&bad), and NONE of the others

There are a few problems with your idea:
Taking the median would certainly trim more non-outlier times than the middle 3. Occasionally it would trim an extra bad solve, but for cubers who perform fairly consistently, it would trim average results unnecessarily.

If this is the only problem with my idea, then there is not really a problem at all. For a cuber that performs 100% CONSISTENTLY (every solve is the same time), the median would be EXACTLY THE SAME AS THEIR AVERAGE. Consistent cubers won't LOVE the median as much as inconsistent cubers, but they also should have no good reason to HATE it. Seems pretty *fair* to me, and to argue otherwise would appear to be unnecessarily *misguided*.

Anytime there is more than 1 good/lucky, or more than 1 bad/unlucky solve, in a round of 5, the arbitrary trimming for 5-avg. yields a result that is polluted.

What about puzzles that have a 50% chance of parity, like 4x4 and square-1? A 50% chance of parity on any given solve means there's a 50% chance of 3 or more parities in an average of 5. What this means is that 50% of the time, the median solve will be a parity case (and so slower than the cuber's overall average) and half the time it won't (so it will probably faster than the cuber's overall average). This leaves the entire average up to chance, as it's basically a 5-second coin flip.

The theoretical distribution based on parity does not actually correlate to real tournament results. I don't know why that is yet, and will not attempt to address this now, so that it doesn't act as a distraction. Besides, the parity problem to be solved first - is simply agreeing on the appropriate# of solves (for deciding higher SD contests), and that should be dealt with seperately.

Another hard thing to define is what is an outlier. How far should it be from the median/average value to be considered so? I don't have a good answer...

This is why I have been proposing to use medians for tournament advancement, because then you can confidently get a trim on ALL outliers, without having to worry about this definition. The median doesn't have to discriminate, like a trimmed average.

Someone who is really consistent may complain about using just the median value.
For example:
Competitor A: (11), 11.2, 11.5, 11.6, (11.8)
Competitor B: (10), 11.3, 11.4, 13.7, (15.8)

With the current system, A clearly beats B. Using just the median, B would get a better ranking and maybe kick A out of the next round.

What is wrong with this? No matter which format & definition of *best performance* is used - 5 solves is insufficient to fairly determine with 100% assurity, the *best performer*. To get it right as often as possible, is a reasonable goal. After 2 rounds, you have twice as many solves to work with (my way), and should be better able to determine which competitors should advance, based on their performance.

I will pose the following (3) questions, so that you might better understand my reasoning here, as this relates to global ranking formats as well.

1) What happens to the difference between (avg/median), as the number of solves , in the average, is increased from 5 to 100 (assuming 20%, or even better yet, all outliers (if they could be identified) trimmed off each side)?

2) Do you agree that the 100-avg will more closely approximate the 100-median for that larger set of solves?

3) Do you agree that the 100-median will become more and more accurate, as the number of solves is increased, and it is a more reliable reflection of that competitors average ability (neglecting outliers)?

Now take the same 100 solves and break them up into 20 trimmed 5-avg. Many of these 20 will be polluted by more than 1 good, or more than 1 bad, outlier. The average of 20 separate 5-avg will be skewed by some 5-avgs that have been polluted with extra outliers.

The rather obvious reason that median is better than sets of 5-avg, is that - the trims that are applied to any 5-avg in an earlier round cannot be redistributed later on, to better eliminate the outliers that are part of ALL the solves that were done by that competitor. Outliers don't have to cooperate with the arbitrary 5-avg format, so that at most, only 2 are present in any one round of 5. The median isn't affected by this lack of outlier cooperation, and that is a very good reason for using it.

 

[Pedro]

What is wrong with this? No matter which format & definition of *best performance* is used - 5 solves is insufficient to fairly determine with 100% assurity, the *best performer*. To get it right as often as possible, is a reasonable goal. After 2 rounds, you have twice as many solves to work with (my way), and should be better able to determine which competitors should advance, based on their performance.

And is one solve better to determine the best performer?

I still think that [11, 11.2, 11.4, 11.6, 11.8] should surely beat [11.2, 11.3, 11.3, 16.7, 19.8].

We use averages because one single value is not reliable (spell?). I'm not saying 3 values averaged is the perfect way, but I think it's better than just one.

If the best cuber is the one who can do good solve most of the time, taking the median doesn't show who is the best. In the example above, who has a better probability of getting a good time? I think it's the first one...

 

[Kirjava]

You guys are forgetting that the system has to be simple.

I like that the fact that this has become so convoluted that none of you have a solution that is universally accepted. This is another reason why none of the solutions will ever be implemented.

100+ posts on and you are yet to come up with something better than my original proposal that you all agree with.

 

[macky]

You guys are forgetting that the system has to be simple.

I like that the fact that this has become so convoluted that none of you have a solution that is universally accepted. This is another reason why none of the solutions will ever be implemented.

100+ posts on and you are yet to come up with something better than my original proposal that you all agree with.

No one except reThinking the Cube has suggested a new idea. The rest of us are for keeping the current system, having also considered the best average method suggested by you and a few others.

The primary reason (basis) I expressed my concern in the above post, was that the definition of "1.Fairness" was being established, and I felt that the (I'd say misguided) way it was worded (i.e. "faster times can never hurt you", but - what about faster outliers not helping you?) created an exemption in the basic definition, for even considering that extra outliers being included in 5-avg, could even be an issue relating to fairness. This definition directly tied in with (3), and how outliers are to be trimmed *fairly* according to the very definition of 1.Fairness.

A faster outlier not helping is perfectly consistent with "faster times can never hurt you"--it doesn't help, but it doesn't hurt either. You of all people certainly don't want to replace this with "faster times can only help you." You would then have to count every outlier for computing the average, the exact opposite of your suggestion. Again, I don't think you've understood Ravi. "Faster times can never hurt you" means the following very basic fairness requirement: "Let S be a set of times, and let S' be another set formed by replacing one time in S with a faster time. Then S' is at least as good as S."

For a cuber that performs 100% CONSISTENTLY (every solve is the same time), the median would be EXACTLY THE SAME AS THEIR AVERAGE. Consistent cubers won't LOVE the median as much as inconsistent cubers, but they also should have no good reason to HATE it. Seems pretty *fair* to me, and to argue otherwise would appear to be unnecessarily *misguided*.

It seems like you understand that your system supports a very inconsistent cuber (messes up to 2/5 of solves) over a perfectly consistent cuber whose median is 0.01 worse than this cuber's. Then we can't really do anything to change your mind.

I respect your opinion - "If you mess up twice in a round, well, too bad.", but it doesn't HAVE to be that way.

Yes, there are situations where speedcuber who should have qualified is eliminated because of two unfortunate mistakes. Pedro has already addressed this. We, and I'm sure the majority of speedcubers, understand this and still don't agree with you. A similar consideration came up in 2004 when we decided stop awarding an extra solve for a POP. One argument was that a POP followed by a bad solve that gets trimmed would mean 2 bad solves out of 6, both not counting. Many members in the community thought that 33% inconsistency (less than your %40) was already too much to go unpenalized.

Your system would be great if all mistakes people ever made were timer failures. Competitors should then always perform around their true potential except for errors outside his control, so the median would be the best measure. In reality, most slow times are the competitor's fault to some degree. POPs and even corner twist DNFs are mistakes that can been avoided by better preparing the cube.

 

[Ravi]

And is one solve better to determine the best performer?

I still think that [11, 11.2, 11.4, 11.6, 11.8] should surely beat [11.2, 11.3, 11.3, 16.7, 19.8].

We use averages because one single value is not reliable (spell?). I'm not saying 3 values averaged is the perfect way, but I think it's better than just one.

If the best cuber is the one who can do good solve most of the time, taking the median doesn't show who is the best. In the example above, who has a better probability of getting a good time? I think it's the first one...

This.

My mean-average proposal is actually very similar to RTC's, except for basically two things:

1. For advancement from the first round, averages are used instead of medians. I believe that averages are more accurate here, as Pedro argued above.

2. In later rounds, the cumulative mean of averages is used instead of the multi-trimmed average. I'm guessing his method is slightly more accurate here because of the (improbable but possible) situation of two outliers/DNFs in the same round, but I'm willing to sacrifice that accuracy margin for a significantly simpler system. In my system, for example, a time that is trimmed at any point is trimmed for good, and one that counts at any point counts for good; these are not always true of RTC's proposal. Also, my system's direct use of averages of 5 makes it highly compatible (see my criteria above) with the current records.

It is also important to remember why we trim averages. If real speedcubers had perfect bell-curve distributions, then there would be no need for trimming--in fact, as my statistics in the original post showed, untrimmed means would be slightly more accurate. But there are two features of real distributions which make trimming necessary. First, real cubers sometimes get DNFs. If we used untrimmed means, a DNF would immediately eliminate the cuber from the competition; a single solve should not have that much power. The second problem is that real cubers have more outliers than there are in bell-curve distributions. I'm not talking about when a 12-second cuber gets a 14 or 15. I'm talking about, well, mostly pops. When a cuber gets a time more than three or four standard deviations away from his/her mean (which in a normal distribution would happen, respectively, once out of 741 or once out of 31547 solves), it is generally fair to dismiss it as an anomaly. Pops are considered forgivable to a point, but there is little excuse for popping more than once within five solves. There's also little excuse for popping more than, say, twice within fifteen solves, so I think if anything we should trim less rather than more.

Also, it is not practical to simply trim everything we consider to be an outlier and nothing else. Some people may have two outliers in a competition, while others (most) may have none. Trimming outliers for some people but not others would obviously favor those who get outliers. It also wouldn't make sense to trim the greatest number of outliers that anybody gets in the competition, because that would allow one person to affect the results of others. That's why we need to set some rule standard rule about trimming outliers. It will be arbitrary, but it's necessary.

 

[Kirjava]

No one except reThinking the Cube has suggested a new idea.

I was blinded by the stupid.

 

[Ravi]

No one except reThinking the Cube has suggested a new idea.

Um, me?

You guys are forgetting that the system has to be simple.

Not all of us. See the following:

2. Practicality. This means, for example, (1) that competitors should not be expected to complete excessively many attempts, nor should organizers be expected to schedule excessive amounts of competing time; (2) that the rules should be simple enough for most speedcubers to understand; and (3) that the results should be reasonably easy to calculate.

My system would be identical to the current format, except that competitors in every round would be ranked based on the mean (or sum; it doesn't matter) of their averages so far. That's it.

And the benefits? My system is simple, fair, significantly more accurate than the current system or any other system proposed so far in this thread (with the possible exception of RTC's), and compatible with all existing records. Compared to RTC's system, it is nearly identical, except it is simpler and more compatible and it avoids medians, which seem to be the primary criticism against that system. Compared to the existing system, it is again nearly identical, except for a single simple change narrowly designed to make results more accurate without requiring any new solves or changing the trimming rules. Simply put, it fulfills all of my criteria better than any other system I have seen.

There, I've demonstrated my system's superiority. Now, instead of just saying "I don't see why we need to change," can someone provide a convincing argument that any other system is as good as (or better than) mine?

 

[reThinking the Cube]

You guys are forgetting that the system has to be simple.

What is simpler than taking the median? You don't even need a calculator for that.

I like that the fact that this has become so convoluted. that none of you have a solution that is universally accepted. This is another reason why none of the solutions will ever be implemented.

Think about it.

A faster outlier not helping is perfectly consistent with "faster times can never hurt you"--it doesn't help, but it doesn't hurt either.

My point was/is - that 2 or more faster outliers could also help *unfairly*. That was relevant to the definition of 1.Fairness ....and all the rest of axioms. I have always understood your point(s). Do you understand mine?

It seems like you understand that your system supports a very inconsistent cuber (messes up to 2/5 of solves) over a perfectly consistent cuber whose median is 0.01 worse than this cuber's.

OK, let's look at Pedro's 1st attempt to make this case. Please read it carefully.

Suppose this scenario:

Competitor A gets (13.10), 13.30 ,14.51, 14.58, (14.90) = 14.13 avg
Competitor B gets (13.20), 14.30, 14.50, 18.56, (19.70) = 15.79 avg

Is it fair for B to advance because of the median (14.50 x 14.51)?

Thank you for presenting this question. I am assuming these times are for 3x3x3 round 1. With my way there is indeed a .01 difference going to B. This will only actually make a difference, if and only if, the last spot for advancement into round 2 is taken by B. Now that scenario is not very likely, but in either case then, both A and B are the SLOWEST competitors coming from round 1, that can advance. Also note that those 2 poorer times of B will not go away, and may impact latter on (my way). It may also be the case that B is really faster than A, and just got a terrible round.

My main concern is to prevent the strongest competitors from being eliminated early simply due to bad luck.

Suppose this scenario: [1st Round 3x3x3]
Competitor A gets (13.10), 13.30 ,14.51, 14.58, (14.90) = 14.13 avg (same as above)
Competitor B gets (9.20), 9.30, 14.50, 19.70, (DNF) = 14.50 avg (strong cuber+bad,bad luck)

Is it fair for B to advance because of the median (14.50 x 14.51)?

For this question, both competitors would also have the 2 slowest medians (14.50, 14.51), and since B clearly shows the potential to outperform A, I would favor B advancing. Yes, this IS fair. Using the median value here, would clearly be superior to using the 5-solve average to decide which competitor gets to advance.

My answer strongly disputes Pedro's concern.

Ok, let's look at Pedro's 2nd attempt to argue the same point, and my answer to that.

Someone who is really consistent may complain about using just the median value.
For example:
Competitor A: (11), 11.2, 11.5, 11.6, (11.8)
Competitor B: (10), 11.3, 11.4, 13.7, (15.8)
With the current system, A clearly beats B. Using just the median, B would get a better ranking and maybe kick A out of the next round.

What is wrong with this? No matter which format & definition of *best performance* is used - 5 solves is insufficient to fairly determine with 100% assurity, the *best performer*. To get it right as often as possible, is a reasonable goal. After 2 rounds, you have twice as many solves to work with (my way), and should be better able to determine which competitors should advance, based on their performance.

I will pose the following (3) questions, so that you might better understand my reasoning here, as this relates to global ranking formats as well.

1) What happens to the difference between (avg/median), as the number of solves , in the average, is increased from 5 to 100 (assuming 20%, or even better yet, all outliers (if they could be identified) trimmed off each side)?

2) Do you agree that the 100-avg will more closely approximate the 100-median for that larger set of solves?

3) Do you agree that the 100-median will become more and more accurate, as the number of solves is increased, and it is a more reliable reflection of that competitors average ability (neglecting outliers)?

Now take the same 100 solves and break them up into 20 trimmed 5-avg. Many of these 20 will be polluted by more than 1 good, or more than 1 bad, outlier. The average of 20 separate 5-avg will be skewed by some 5-avgs that have been polluted with extra outliers.

The rather obvious reason that median is better than sets of 5-avg, is that - the trims that are applied to any 5-avg in an earlier round cannot be redistributed later on, to better eliminate the outliers that are part of ALL the solves that were done by that competitor. Outliers don't have to cooperate with the arbitrary 5-avg format, so that at most, only 2 are present in any one round of 5. The median isn't affected by this lack of outlier cooperation, and that is a very good reason for using it.

Once again, my answer strongly disputes that using the median should be considered any less fair than using outlier polluted arbitrary 5-avg.

Someone who is really consistent may complain about using just the median value.
For example:
Competitor A: (11), 11.2, 11.5, 11.6, (11.8)
Competitor B: (10), 11.3, 11.4, 13.7, (15.8)

If the current format was different, then someone who is a really good cuber, could strongly complain about using just the arbitrary 5-avg.

For example:
Competitor A: (11), 11.2, 11.5, 11.6, (11.8) (same as above) [11.43 avg]
Competitor B: (9), 9.3, 11.4, (DNF), 15.8 (i.e. Pedro going-for-gold, and popping on 4th solve) [12.17 avg]

Which competitor's performance should allow them to advance into the next round, A or B?

 

[Kirjava]

nonsense

om nom nom

why bother replying when you'll end up ignoring me anyway XD

 

[reThinking the Cube]

nonsense

om nom nom

why bother replying when you'll end up ignoring me anyway XD

LOL. Fair trim, with a simple & practical format. I can't argue with that anyway XD

 

[Pedro]

My system would be identical to the current format, except that competitors in every round would be ranked based on the mean (or sum; it doesn't matter) of their averages so far. That's it.

(...)

One problem I can see is that someone who got a bad 2nd round but still advances would have a very small chance of winning.

For example, people get this averages: (suppose a very strong competition)

A. 10.89
B. 11.01
C. 11.20
D. 11.50
E. 11.79
F. 11.93

2nd round:
B. 11.10
C. 11.15
A. 11.30
F. 11.49
E. 11.68
D. 14.30

I'm not sure how you sum/median the averages, but let's say we sum them up and divide by 2.

A: 11.10
B: 11.06
C: 11.18
D: 12.90
E: 11.74
F: 11.71

So competitor D is over a second behid the worst one of the others. Suppose everyone performs 0.5 worse than this median in the final round. D would need a sub-8 average to win.

Even with a less worse 2nd round (let's say 13.30), he'd still need a sub-9.

(going to read your first post to see what exactly you propose)

 

[macky]

A faster outlier not helping is perfectly consistent with "faster times can never hurt you"--it doesn't help, but it doesn't hurt either.

My point was/is - that 2 or more faster outliers could also help *unfairly*. That was relevant to the definition of 1.Fairness ....and all the rest of axioms. I have always understood your point(s). Do you understand mine?

No, you still don't understand what I said. Ravi, can you clarify what you meant by your fairness requirements? Did you mean: "Let S be a set of times, and let S' be another set formed by replacing one time in S with a faster time. Then S' is at least as good as S." ? If so, does what RTC is saying here make any sense?

No one except reThinking the Cube has suggested a new idea.

Um, me?

Sorry, I mean since Kirjava posted his idea / since RTC started on his. I think it's safe to say that most of us (except RTC) agrees on using the average of 5 for the first round. The question is what to do for the subsequent rounds.

In the first couple of pages, I wasn't convinced that your suggestion was better enough than the current system to warrant a change. There were concerns brought up about cubers who go slower but safer in the first rounds, for example; your system is compatible with the average of 5 ranking, but it would certainly change competition strategies. If you already addressed this point, could you repost it? I can't find anything in this thread anymore.

 

[reThinking the Cube]

A faster outlier not helping is perfectly consistent with "faster times can never hurt you"--it doesn't help, but it doesn't hurt either.

My point was/is - that 2 or more faster outliers could also help *unfairly*. That was relevant to the definition of 1.Fairness ....and all the rest of axioms. I have always understood your point(s). Do you understand mine?

No, you still don't understand what I said. Ravi, can you clarify what you meant by your fairness requirements? Did you mean: "Let S be a set of times, and let S' be another set formed by replacing one time in S with a faster time. Then S' is at least as good as S." ? If so, does what RTC is saying here make any sense?

1. Fairness. This means, for example,.....
<snip>
3. (Limited) forgiveness for outliers. This means (1) that DNFs and occasional outliers, which are generally considered relatively unimportant, should not affect a competitor's result drastically, notwithstanding item 2 under "Fairness" above; but (2) that this forgiveness (ie. trimming) should not be taken to the extent of invalidating significant, near-average solves. A good guideline provided by the current system is (3) that roughly the middle 60% of attempts should count.
<snip>
Please comment on the guidelines above, and I will edit accordingly. I would really like to reach a consensus on this.

Does it make sense now?