Writeup
Thanks for running the contest! I spent way too much time on it
but had a lot of fun. Here are my notes.
Two-phase solver for 4x4x4
My approach for this contest was to implement a two-phase solver
for the 4x4x4. The first phase finds optimal and suboptimal
reductions (move sequences that take the input position to the
subgroup (U, F, R, D, B, L)), and the second phase uses standard
3x3x3 optimal solvers to finish the solution. Only the first
phase is new work, so I will focus on that here.
I must open by saying this is not a practical 4x4x4 solver;
using a high-powered machine (16 cores) with a lot of memory
(256GB) running nearly non-stop for nearly two months, I was
able to directly find only 42 actual reduction solutions.
However, through the use of near-miss solves and derived
solves, I was able to generate more near-optimal reduction
solutions that helped me find shorter solutions.
Pruning Tables
My reduction solver used two large pruning tables, one for the
edges and one for the middles. Since each table was too
large to fit into memory, rather than using a subgroup indexing
approach, I instead used a hashing approach with collisions
to generate an inexact pruning table. I allocated four bits
per hash entry; the value in those four bits is always the
least value of all positions that hash to that value. Given a
maximum distance for hash table generation, I simply filled
each table with that distance plus one, then did an iterated
depth-first exploration from solved, hashing each position
solved and updating hash table entries along the way. For my
edge pruning table, I used 5GB of RAM, and for my middle
pruning table, I used 215GB of RAM. I explored edges out to
a distance of 10, and middles out to a distance of 10.
For both edges and middles, most entries of the hashtable
were unset by the search from solved to a depth of 10, so
for most random positions, the pruning table returned an
immediate lower bound on distance of 11.
For edges, if I had spent the time to complete a search
from solved of depth 11, I would have probably sped up the
program a fair bit; it would probably have taken about
four days to generate this table. In hindsight this
would probably have been wise; the search through depth
10 for edges only touched a total of 2.3B out of the 430B
hash table entries. I guess I was in just too much of a
hurry to get results. I may do this and see how performance
differs in the coming weeks; this may make the solver
significantly faster! Writing this at the end of the contest,
I'm disappointed I did not consider this earlier.
The hash function I used implicitly reduced the positions by
48 ways of symmetry. Also, the moves I considered were
restricted to keep a particular corner fixed, giving me an
additional reduction by a factor of 24. Details of the hash
function themselves are somewhat intricate and I would be
willing to write them up if anyone has interest.
My code is written to support the slice turn metric, block
turn metric, and outer block turn metric; I have focused on the
slice turn metric for this contest but am considering exploring
the other metrics as well.
The target subgroup for edges for my pruning table is that where
edges are matched and the total edge parity is even. (Note that
rotations and conjugation by rotations/mirroring does not change
the parity of the edge permutation.) The target "subgroup" for
middles was with each face a solid color (but not necessarily to
have the colors in a solved position). I put subgroup in
quotes because of course the middle positions do not form a
group because not all pieces are distinguishable.
Designing pruning tables is still somewhat more of an art than
a science. I considered generating one big pruning table that
combined edges and middles, but experiments showed that this
pruning table got too "wide" too quickly, and I could get better
average distances by using separate edge and middle pruning
tables. I did not fully implement and optimize a combined
pruning table so I do not have conclusive evidence that it would
be worse.
Initial Results
These are the initial results based on just using the simple
reduction solver and an optimal 3x3x3 solver with no additional
improvements. The input positions were as follows:
!: R2 F2 Rw' R D' Uw' L B' L' Rw2 R2 F Uw2 Fw F2 Rw2 B D B2 Rw' F2 Uw2
B' Rw R U Fw' Rw' U' Rw' B' D F2 U Fw' R2 B L2 Uw L2 U D R U2 Uw2
2: B' Fw' L' D F B2 Uw2 D2 R2 L Fw2 F R' Uw' Fw2 B2 Uw Rw2 F L R D U Rw
B Fw' R2 Rw2 L' D' B D' F' L2 Uw2 Rw F2 Rw2 F Fw2 U F' Rw' R U
3: Rw F' Rw2 L' Fw2 F2 B2 U' R' L Uw B' R' Rw2 B Fw L2 Rw' Uw2 B' L B2
Fw2 Rw' U Fw Rw2 B R' Uw2 R' B Uw2 B2 Fw2 F2 D' R' B2 U Rw F' R2 Rw2 L
4: B R F' D U' R' B2 D2 B' R B' D2 F' B L2 F' D2 F' U2 R2 Uw2 Rw2 L' B
U2 L' B2 Uw2 B' R' D2 B2 Uw Rw2 R2 D2 B2 U' Fw' Rw' B' F2 Uw U' R B2
U' R F2 U
5: F2 L2 B' U2 D2 R2 U2 F U F2 R B R2 D2 B D L' U Fw2 D Rw2 R Fw2 R
Uw2 F2 R2 D Rw2 D2 U' Fw F L' Fw2 Rw' Uw F' D2 Rw' B Uw2 B L B U' F2
For all positions but number 2, the optimal reduction was of length
19; for position 2, the optimal reduction was of length 20. (Surprisingly
position 2 is also the position we found the shortest solution for.)
An example optimal reduction for the moves is shown here:
1: 2R2 R B2 2U2 2F 2U 2B' 2R2 2U2 D' 2R2 2F' B' L 2F2 2D 2L2 L' 2F' (19*)
2: 2R2 2F 2L 2D' R' D' F' 2U' 2L' 2U' F' U 2B 2R' 2F U B 2R D 2L (20*)
3: 2F L2 R' 2D' F 2D' D L' 2F2 D 2L 2U' 2L' 2B B2 2R' U' D' 2F' (19*)
4: R' B2 U' 2B' R 2U2 U2 B R 2F' R' 2U 2R D2 2F2 U' 2B' B2 2U (19*)
5: 2R' F 2D' 2L2 2F' 2D 2R 2F2 U2 B2 2L2 L' 2F2 2D' R2 U' 2B D 2F' (19*)
The count of direct reductions we found (limited to those sequences
that do not end in a face turn):
Code:
19 20 From 19 From 20 Best
1: 2 10 19+18=37 20+16=36 36
2: 0 2 20+16=36 36
3: 2 10 19+18=37 20+17=37 37
4: 2 6 19+18=37 20+17=37 37
5: 2 6 19+17=36 20+18=38 36
We completed reduction search through a depth of 19, and finished
a fair fraction (maybe 30%?) of the search through depth 20.
We optimally solved all the positions remaining after reduction,
and listed the best results in the table above (the format is
the reduction length, a plus sign, the best 3x3x3 result,
an equals sign, and then the resulting best sum. We list the
best overall result in the last column. As you can see we
were able to solve all positions in 37 or fewer moves, and
three of them in only 36 moves.
The optimal reduction solutions show that four of the five
positions have an optimal reduction length of 19. This means
it is more likely than not that the median reduction length
over all 4x4x4 positions is 19.
This result gave us an average of median 3 of 36.3, and an
overall average of 36.4. But we can do better.
Near Misses
Not all positions that were obtained by solving the edges and
middles separately but simultaneously were actually reduction
solutions. For instance, in some solutions the colors for the
middles were properly paired (for instance, the white middle was
not opposite the yellow middle). In some solutions, the middles
were properly paired, but the middles were in a mirror image
position. Finally, in some solutions, the total parity of the
corners and the combined edges was odd. When one of these
conditions was true, I considered the solution a "near miss";
this happened most of the time (about 87% of the time). Early
on, I ignored these near misses as they could not be solved
by a 3x3x3 solver. But later I found a way to exploit them.
First-phase solves always came in pairs (or larger sets).
The final move of any first-phase solve was always a slice move
(if it was not a slice move, the position before the final move
would also have been a first-phase solve). When I say slice
move I mean inner slice; an outer slice move is just a face move.
If moving an inner slice brings the position to a phase one
solution, then moving the opposite inner slice the other way
*also* brought the position to a phase one solution.
There are additional ways that first-phase solves can come in
bursts but they are less frequent so we will not discuss them
further here.
Given how much CPU time I was spending to compute the phase one
solutions, I decided to see if there was a way I could obtain
additional related solutions directly from those solutions, so
I took each phase one solution, trimmed off all but the first
eight moves, and ran it through the phase one solver again.
This generated a surprising wealth of additional solutions at
distances only a bit more than the original. The additional
solutions were not the trivial ones generated by appending
face moves, but were usefully different ones that, in some
cases, generated better overall solution lengths. Further, not
just the actual reduction solves themselves but in addition
the near misses also generated actual phase one solutions that
were only slightly longer. Finally, the additional solutions
were generated in seconds, so this was an easy way to leverage
a solution or near miss into many additional phase one solutions
that let me do more probes into phase two.
This table has a count of the phase-1 solutions found
organized by input sequence and length:
Code:
19 20 21 22 23 24 25 26
1: 2 10 112 1498 17002 19558 7023 18
+18=37 +16=36 +16=37 +16=38 +14=37 +14=38 +14=39 +17=43
2: 0 4 30 174 1174 2785 12 0
+16=36 +17=38 +13=35 +15=38 +14=38 +17=42
3: 2 14 119 850 6979 25281 62431 0
+18=37 +17=37 +16=37 +14=36 +14=37 +13=37 +13=38
4: 2 22 72 304 3037 7835 2181 490
+18=37 +17=37 +16=37 +16=38 +14=37 +14=38 +15=40 +16=42
5: 2 16 55 535 4317 9865 2362 2
+17=36 +17=37 +17=38 +15=37 +15=38 +14=38 +15=40 +18=44
With these additional probes we were able to find a
length 35 solution to input sequence 2 and a length 36
solution to input sequence 3.
Of course, with tens of thousands of candidate phase one
solutions, solving them all optimally with a 3x3x3 solver
would have taken a fair amount of CPU. But generally, once
an overall solution was known for the original 4x4x4
position, we no longer cared about longer solutions, so the
3x3x3 optimal solver could be instructed to only search to a
useful overall length, and give up if no such short solution
was found.
Let us deconstruct the two better solutions that were found
with this strategy. The length 35 solution to the second input
position was found as follows.
First, a length-20 reduction solution was found:
2R2 2F 2L 2D' R' D' F' 2U' 2L' 2U' F' U 2B 2R' 2F U B 2R D 2R
By trimming off two moves and running this through our reduction
solver again, we find the following reduction solution of length
22:
2R2 2F 2L 2D' R' D' F' 2U' 2L' 2U' F' U 2B 2R' 2F U B 2R D' L2 D2 2R
The phase one reduction is longer, but the 3x3x3 solution now is only
13 moves:
U L D R' F2 D' L2 F L F2 U' R B'
for a total of 35 moves.
For input 3, the process was as follows. We found the following
length-19 near-miss:
R' F' 2U' D' 2F2 2L2 2B 2L' F D 2R' B' L 2U2 2L2 2F 2D2 2L' 2U'
This near miss was rejected because of parity. But by trimming the
last seven moves and resolving with the reduction solver, we find the
following valid reduction of length 22:
R' F' 2U' D' 2F2 2L2 2B 2L' F D 2R' B' L2 R' 2U 2L 2F2 2R' 2F' 2R2 2F2 2L'
This has a 3x3x3 solution of only 14 moves, for a total of 36 moves:
L2 U' B2 D2 F2 R' D2 B2 D' F' U2 R' D' L2
Some questions remain that are worth exploring:
1. What is a good target subgroup for phase 1? I happened
to use one that ignored overall corner/edge parity
and middle color permutation, and it worked well, but
is there a larger subgroup that would provide more
opportunity to exploit near-misses for phase 2 probes?
2. Is there a better way to construct either separate or
combined hash functions for phase one search that
generates a higher overall depth, or that is faster
to compute?
3. What exactly is the relationship of all the solutions
generated by trimming and re-solving phase one
solutions and near-misses? Is there an alternative
or faster way to generate these that does not require
search?
4. Given the approximate nature of the pattern database,
what is the best way to compress this in order to
obtain higher average probe depth, without compromising
performance or random access?