Coming up with a framework for evaluating methods

[Osric]

I'd like to see if we can collaborate to produce a framework for evaluating speedsolving methods.

The goals would be to have common metrics and language for talking about what's good and bad about a method, and a standard for describing the method in this framework, so that we can more easily explore refinements to existing methods or entirely new methods to figure out if they are suitable canidates to displace Fridrix (aka CFOP [sic]) or Roux.

Here are the kinds of things I think are important to evaluate the alternatives:

1- a move metric, composed of move*cost summed up over the course of the solve, where the costs are meant to reward moves we know are easy to train and penalize moves that we feel are difficult to train. Ideally we would construct the cost table by analyzing videos of the top 100 or 200 solvers in the world to concretely define what moves are trainable and what moves probably aren't; this will introduce a large bias based on the history of cubing so far but still is probably the best way to figure out move metrics.

2- a number of algorithms metric, to enable assessing the complexity of learning the system. this same metric needs to be applied to so-called "intuitive" steps to get a sense of the problem space of creativity required to execute the method.

3- some sort of visibility or recognition metric, that rates how difficult it is to recognize what the upcoming steps are (the next alg and the one after at least).

Do these seem reasonable? Is there more to add or things I've suggested that don't make sense?

Aside from having ways of evaluating the alternatives, it would be good to have a common standard for defining the methods as well.

For this, I suggest that each method really consists of a graph of cube states, connected by edges which are the algsets needed to get from one state to the next. For example one formulation of Roux would be first block -> second block -> corners -> edge orientation -> edges. The same method could be broken down into more nodes ilke this: first block -> square back [or front] -> second block -> corner orientation -> corners -> edge orientation -> edges. An algorithm should be described with pictures of these states and moves expected to get from one state to the next. For example:

scrambled->fb allow: U U2 U' F F2 F' R R2 R' r r2 r' D D2 D' M M' M2 B B' B2

fb->square back allow U U2 U' F F2 F' R R2 R' r r2 r' D D2 D' M M' M2 B B' B2

square back->ss allow U U' U2 R R' R2 r r' r2 M' M M2

ss->cmll orientation allow U U' U2 R R' R2 r r' r2 M' M M2

cmll orientation->cmll allow U U' U2 R R' R2 r r' r2 M' M M2

cmll->lse orientation allow U U' U2 M' M M2

lse orientation->solved allow U U' U2 M' M M2
​

This example definiton of a roux variation makes it clear what moves are expected for each stage and what the stage looks like. Then for each edge in the graph, an algsheet should be provided which defines all the situations and algorithms invovled if there are a managable number, as well as the frequency with which the cases are expected to occur and the move metrics for all the algorithms so that the expense of a stage can be the sum of the probabilities of the cases multiplied by their move metric assessment and recognition assessment.

Armed with all this information it seems to me that it'll be easier to explore whether it's worth learning a new thing - say mismatched centers for roux, or even colour neutrality - by comparing methods variations which do those things to ones which don't.

Keen to get feedback on whether this framework would enable better discussions and explorations of methods and their variations.

Thanks
Osric

 

[abunickabhi]

I feel evaluating methods is subjective and should not be framework bound.
Different methods can lead to discussions but should never be tested against a score drummed up by a metric or a framework imo.

 

[Osric]

The framework isn't meant to create a hard and fast judgement of the method, but to force us to provide the information needed to really reason about it.

For example, a method without algsheets, without average move counts, without states that you know you need to visit as you go from scrambled to solved ... is that even a method? How do you know what it is?

But if you follow the framework, you'll have algsheets, average move counts, intended states. Producing this information will enable actual evaluation and discussion of the method using common terms and metrics that make it easier to see "oh, this may be a good idea because I can get from state X to state Y with only 8 algs, whereas the old way required 42" ... simply put, if you could do CMLL for example in 8 short algs instead of 42 that would be undeniably an improvement. That would not be subjective, it would be a breakthrough.