(80% crackpot idea and only 20% actually serious. But have a think. I wrote most of the first three sections in 2020 and figured I should just fill in the rest and post it now.)
tl;dr read the thing,it's not that long see title
In official competitions, results are usually ranked by "averages" of some sort: among the non-blindsolving events, it's mean-of-three (mo3) for 666, 777 and FMC and average-of-five (ao5) for the rest. (And for completeness, results are ranked by best-of-three for 3BLD, 4BLD, 5BLD and MBLD.) Unofficially, timers like csTimer, qqTimer, Twisty Timer, and so on primarily adopt the 5% trimmed mean when computing averages over large number of solves, which is a generalisation of the ao5: for measuring the average of \( n \) solves, we sort the input and drop \( \lceil n/20\rceil \) of the best and worst results each, then take the arithmetic mean of the remaining. For \( n\le20 \), this is equivalent to just dropping the best result and the worst result and computing the arithmetic mean of the remaining \( n-2 \) results.
Somewhat more recently, people have also suggested using something like the mean of ao5s, where you divide the input into non-overlapping sections of five results, compute the ao5 of each of those, then take the mean. This provides a "better" estimate of how one would perform in competitions than the 5% trimmed mean, since solves in competitions (for the events held in the ao5 format) are done in sets of five and the results of one round don't affect the others. One can also adjust solving style conditional on performance in the earlier solves (e.g. if you already have four extremely good solves, you "should" go high-variance on the last solve and hope for a fifth extremely good solve; this is super risky and I don't think anyone fast actually bothers), and the mean of ao5s method accounts for this possibility, whereas the usual 5% trimmed mean disincentivises doing anything that isn't useful in general.
csTimer also reports the mean of successes: you throw out all DNFs and compute the arithmetic mean of what's left. This seems to be somewhat popular among blindsolvers.
What I'm proposing here: the harmonic mean. Instead of directly computing the arithmetic mean, we first replace each entry with its reciprocal, take the arithmetic mean of that, and then take the reciprocal again. To wit, the harmonic mean of \( x_1,x_2,x_3,\cdots,x_n\in[0,+\infty] \) is
\( \displaystyle H(x_1,\cdots,x_n):=\frac1{\frac1n\left(\frac1{x_1}+\cdots+\frac1{x_n}\right)}=\frac n{\sum_{k=1}^n\frac1{x_k}} \),
where we do the thing teachers tell you not to do in school by assuming \( 1/0=\infty \) and \( 1/\infty=0 \). (Kids, please don't write this in your maths homework; it's extremely wrong when taken out of context.)
How the different types of averages deal with DNFs / large outliers
The arithmetic mean is completely nonrobust against DNFs—a single DNF invalidates the whole average. While this may be fine for small averages, this essentially guarantees that sufficiently large averages are always going to be DNF. (Find me someone who has an exactly zero DNF rate and I'll find you a liar.) Ao5 and the trimmed mean in general offer some robustness against DNFs; ao5 and ao12 can tolerate one DNF, ao50 can tolerate three DNFs, and ao100 can tolerate five DNFs. Mo10Ao5 ironically has less robustness against DNFs than ao50 despite having more non-counting solves: while each individual ao5 has a very low chance of being a DNF, it's not zero, and you're now calculating the mean of some input that can contain DNF entries.
The harmonic mean is maximally robust against DNFs. If every single result is a DNF, then, well, of course the harmonic mean will be a DNF as well—this is unavoidable for any reasonable notion of "average". However, as long as there is any one result that isn't a DNF, the harmonic mean will be finite. DNFs will no longer be infinitely punishing if you get too many of them, as long as you have a success somewhere.
As an example, say you did three 3BLD solves and they were 2:26.06, 2:23.63, and DNF. You've only got two successes, so this is a DNF mo3. However, the harmonic mean here is 3/(1/146.06+1/143.63+1/∞)=217.25, or 3:37.25. You no longer need 100% accuracy to get an average, although of course DNFs will still pull the average up.
To a lesser extent, all of the above remarks also apply to large outliers, e.g. solves that took twice as long as usual because your cube exploded or because you messed up near the end and had to redo the whole solve.
(Mean-of-successes has very strange behaviour with respect to DNFs. Maybe you think that since DNFs are always non-counting with mean-of-successes, they don't matter, but this also means you're not thinking hard enough. More in a later section.)
An alternative formulation of the harmonic mean
The harmonic mean can also be thought of as a "self-weighted" arithmetic mean where the weight of an entry is determined solely by itself (rather than being separately specified, or depending on the ranking). As a special case, if you weight an entry \( x_k \) by some power of itself, you get the Lehmer means. Weighting by the zeroth power (i.e. equal weights) just gives the arithmetic mean, whereas weighting by the \( (-1) \)st power gives the harmonic mean (for finite and positive input).
Qualitatively, what this means is that (from the point of view of self-weighted means) better times receive a higher weight, while worse times receive a lower weight. At the very extreme, DNFs receive "zero" weight.
(Which does not mean that DNFs don't matter! See previous section.)
Computational complexity
Rolling aoX computations involve a surprising amount of complexity, especially when the window size is large and the data set is even larger (e.g. rolling ao1000 in a session of 10000 solves). The naïve algorithm is to calculate the aoX independently for each sliding window, sort the entries within the window, then compute the trimmed mean. In our example of rolling ao1000 over 10000 solves, this uses around \( 10^8 \) operations. It's certainly possible to optimise this (e.g. with quickselect or binary search trees), but the resulting algorithms will be too complicated to explicitly describe in a single sentence.
In contrast, the arithmetic and harmonic means are much more straightforward, and have obvious O(1)-amortised-complexity algorithms: just subtract outgoing values and add incoming values. More generally, this applies to all f-means and self-weighted means, which both AM and HM are both of.
Computing rolling means-of-successes requires some bookkeeping, but is not substantially more complicated than rolling arithmetic means.
Caveat: The accumulation of rounding error is a significant concern with rolling harmonic means (and general f-means, self-weighted means) that is not present with rolling arithmetic means. It's still possible to compute rolling harmonic means with bounded rounding error and O(1) amortised complexity (see e.g. van Herk (1992)), but this introduces a lot of code complexity. (Bonus caveat: By "bounded rounding error", I mean that the error won't blow up to infinity as the data set size increases, with the window size kept fixed. This rounding error still grows with the window size, which I don't think is avoidable.)
One major disadvantage of the harmonic mean over ao5 and mo3 is the impracticality to compute with mental arithmetic; a calculator is practically a necessity. However, this should also rarely matter—it is already hard enough to compute ao5/mo3 mentally that people largely use calculators anyway. The obvious exception here is FMC, where the three results are small integers, usually close to each other.
Avoiding perverse incentives
A key feature you would expect out of an "average" is monotonicity: holding all the other results fixed, getting one worse result should lead to a worse average, and getting one better result should lead to a better average. Of the types of averages mentioned within this post, all of them except the mean-of-successes satisfy monotonicity.
Mean-of-successes leads to a perverse incentive: once you see that the timer has exceeded the current average, it's better to take a DNF rather than to complete a solve. If we want maximal robustness against DNFs, this leaves only the harmonic mean in consideration.
Consider also this scenario: A competitor has DNFed their 3rd and 4th attempts, and is about to start their 5th attempt. With ao5, they have little incentive to continue, because the ao5 is already guaranteed to be a DNF at this point. However, with the harmonic mean instead, they can still salvage the final result with a sufficiently good single. An uphill task, but fundamentally, not impossible—the competitor is incentivised to continue trying. (If you think DNFing is too extreme, this also applies to any situation that leads to much-worse-than-usual solves, e.g. a pop/lockup that took very long to fix.)
Continuity and smoothness
While the trimmed mean is continuous, it is not everywhere differentiable; the discontinuities in the derivative come from when a result transitions from being a counting time to a non-counting time or vice versa. This is actually beneficial for human calculation purposes (we are better at if-else than at division), but is mathematically inelegant. In contrast, the harmonic mean is infinitely differentiable as long as all the times are (strictly) positive.
Are you seriously suggesting that the WCA should switch to harmonic means?
No.
Unless…?
tl;dr read the thing,
In official competitions, results are usually ranked by "averages" of some sort: among the non-blindsolving events, it's mean-of-three (mo3) for 666, 777 and FMC and average-of-five (ao5) for the rest. (And for completeness, results are ranked by best-of-three for 3BLD, 4BLD, 5BLD and MBLD.) Unofficially, timers like csTimer, qqTimer, Twisty Timer, and so on primarily adopt the 5% trimmed mean when computing averages over large number of solves, which is a generalisation of the ao5: for measuring the average of \( n \) solves, we sort the input and drop \( \lceil n/20\rceil \) of the best and worst results each, then take the arithmetic mean of the remaining. For \( n\le20 \), this is equivalent to just dropping the best result and the worst result and computing the arithmetic mean of the remaining \( n-2 \) results.
Somewhat more recently, people have also suggested using something like the mean of ao5s, where you divide the input into non-overlapping sections of five results, compute the ao5 of each of those, then take the mean. This provides a "better" estimate of how one would perform in competitions than the 5% trimmed mean, since solves in competitions (for the events held in the ao5 format) are done in sets of five and the results of one round don't affect the others. One can also adjust solving style conditional on performance in the earlier solves (e.g. if you already have four extremely good solves, you "should" go high-variance on the last solve and hope for a fifth extremely good solve; this is super risky and I don't think anyone fast actually bothers), and the mean of ao5s method accounts for this possibility, whereas the usual 5% trimmed mean disincentivises doing anything that isn't useful in general.
csTimer also reports the mean of successes: you throw out all DNFs and compute the arithmetic mean of what's left. This seems to be somewhat popular among blindsolvers.
What I'm proposing here: the harmonic mean. Instead of directly computing the arithmetic mean, we first replace each entry with its reciprocal, take the arithmetic mean of that, and then take the reciprocal again. To wit, the harmonic mean of \( x_1,x_2,x_3,\cdots,x_n\in[0,+\infty] \) is
\( \displaystyle H(x_1,\cdots,x_n):=\frac1{\frac1n\left(\frac1{x_1}+\cdots+\frac1{x_n}\right)}=\frac n{\sum_{k=1}^n\frac1{x_k}} \),
where we do the thing teachers tell you not to do in school by assuming \( 1/0=\infty \) and \( 1/\infty=0 \). (Kids, please don't write this in your maths homework; it's extremely wrong when taken out of context.)
How the different types of averages deal with DNFs / large outliers
The arithmetic mean is completely nonrobust against DNFs—a single DNF invalidates the whole average. While this may be fine for small averages, this essentially guarantees that sufficiently large averages are always going to be DNF. (Find me someone who has an exactly zero DNF rate and I'll find you a liar.) Ao5 and the trimmed mean in general offer some robustness against DNFs; ao5 and ao12 can tolerate one DNF, ao50 can tolerate three DNFs, and ao100 can tolerate five DNFs. Mo10Ao5 ironically has less robustness against DNFs than ao50 despite having more non-counting solves: while each individual ao5 has a very low chance of being a DNF, it's not zero, and you're now calculating the mean of some input that can contain DNF entries.
The harmonic mean is maximally robust against DNFs. If every single result is a DNF, then, well, of course the harmonic mean will be a DNF as well—this is unavoidable for any reasonable notion of "average". However, as long as there is any one result that isn't a DNF, the harmonic mean will be finite. DNFs will no longer be infinitely punishing if you get too many of them, as long as you have a success somewhere.
As an example, say you did three 3BLD solves and they were 2:26.06, 2:23.63, and DNF. You've only got two successes, so this is a DNF mo3. However, the harmonic mean here is 3/(1/146.06+1/143.63+1/∞)=217.25, or 3:37.25. You no longer need 100% accuracy to get an average, although of course DNFs will still pull the average up.
To a lesser extent, all of the above remarks also apply to large outliers, e.g. solves that took twice as long as usual because your cube exploded or because you messed up near the end and had to redo the whole solve.
(Mean-of-successes has very strange behaviour with respect to DNFs. Maybe you think that since DNFs are always non-counting with mean-of-successes, they don't matter, but this also means you're not thinking hard enough. More in a later section.)
An alternative formulation of the harmonic mean
The harmonic mean can also be thought of as a "self-weighted" arithmetic mean where the weight of an entry is determined solely by itself (rather than being separately specified, or depending on the ranking). As a special case, if you weight an entry \( x_k \) by some power of itself, you get the Lehmer means. Weighting by the zeroth power (i.e. equal weights) just gives the arithmetic mean, whereas weighting by the \( (-1) \)st power gives the harmonic mean (for finite and positive input).
Qualitatively, what this means is that (from the point of view of self-weighted means) better times receive a higher weight, while worse times receive a lower weight. At the very extreme, DNFs receive "zero" weight.
(Which does not mean that DNFs don't matter! See previous section.)
Computational complexity
Rolling aoX computations involve a surprising amount of complexity, especially when the window size is large and the data set is even larger (e.g. rolling ao1000 in a session of 10000 solves). The naïve algorithm is to calculate the aoX independently for each sliding window, sort the entries within the window, then compute the trimmed mean. In our example of rolling ao1000 over 10000 solves, this uses around \( 10^8 \) operations. It's certainly possible to optimise this (e.g. with quickselect or binary search trees), but the resulting algorithms will be too complicated to explicitly describe in a single sentence.
In contrast, the arithmetic and harmonic means are much more straightforward, and have obvious O(1)-amortised-complexity algorithms: just subtract outgoing values and add incoming values. More generally, this applies to all f-means and self-weighted means, which both AM and HM are both of.
Computing rolling means-of-successes requires some bookkeeping, but is not substantially more complicated than rolling arithmetic means.
Caveat: The accumulation of rounding error is a significant concern with rolling harmonic means (and general f-means, self-weighted means) that is not present with rolling arithmetic means. It's still possible to compute rolling harmonic means with bounded rounding error and O(1) amortised complexity (see e.g. van Herk (1992)), but this introduces a lot of code complexity. (Bonus caveat: By "bounded rounding error", I mean that the error won't blow up to infinity as the data set size increases, with the window size kept fixed. This rounding error still grows with the window size, which I don't think is avoidable.)
One major disadvantage of the harmonic mean over ao5 and mo3 is the impracticality to compute with mental arithmetic; a calculator is practically a necessity. However, this should also rarely matter—it is already hard enough to compute ao5/mo3 mentally that people largely use calculators anyway. The obvious exception here is FMC, where the three results are small integers, usually close to each other.
Avoiding perverse incentives
A key feature you would expect out of an "average" is monotonicity: holding all the other results fixed, getting one worse result should lead to a worse average, and getting one better result should lead to a better average. Of the types of averages mentioned within this post, all of them except the mean-of-successes satisfy monotonicity.
Mean-of-successes leads to a perverse incentive: once you see that the timer has exceeded the current average, it's better to take a DNF rather than to complete a solve. If we want maximal robustness against DNFs, this leaves only the harmonic mean in consideration.
Consider also this scenario: A competitor has DNFed their 3rd and 4th attempts, and is about to start their 5th attempt. With ao5, they have little incentive to continue, because the ao5 is already guaranteed to be a DNF at this point. However, with the harmonic mean instead, they can still salvage the final result with a sufficiently good single. An uphill task, but fundamentally, not impossible—the competitor is incentivised to continue trying. (If you think DNFing is too extreme, this also applies to any situation that leads to much-worse-than-usual solves, e.g. a pop/lockup that took very long to fix.)
Continuity and smoothness
While the trimmed mean is continuous, it is not everywhere differentiable; the discontinuities in the derivative come from when a result transitions from being a counting time to a non-counting time or vice versa. This is actually beneficial for human calculation purposes (we are better at if-else than at division), but is mathematically inelegant. In contrast, the harmonic mean is infinitely differentiable as long as all the times are (strictly) positive.
Are you seriously suggesting that the WCA should switch to harmonic means?
No.
Unless…?
Last edited:
icon or go directly to 
