RicoReco - 3x3 video reconstruction

rotobld

Member
Joined
Jul 28, 2017
Messages
23
WCA ID
2014IFAR01
Hi everyone!

I've built a new video reconstruction tool, and I'm very happy to show it :)
At RicoReco.rotohands.com you can see how the tool performs and what it could become.
In the future, you'll be able to upload your own solves and get their reconstructions easily.



Background
As some of you may know, I love working on new ways to make deliberate practice in cubing better (TrainBLD, CustomBLD). My dream has always been to reconstruct solves straight from video, without relying on smart cubes.
My first attempt was back in 2020. After a couple of weeks I realised I wouldn't be able to make it work (short demo).

With the current AI tools things became much easier, so I restarted the project in May 2026. I didn't have much time back then and didn't get very far. About a month ago I picked it up again.

Once I had an MVP that worked reasonably well, a friend who was at the Hungarian Open told me about flowtimer.io, which does exactly this. A week later, Jeff Park posted about his own version too.

First of all, huge congrats to both of them for getting it working. As someone who has done it too, I can say it's not easy and it takes more than "just vibe coding". it needs lot of effort and thinking to make it work.

So after a long time and a lot of effort, I'm proud to introduce RicoReco: a video reconstruction model that can analyse and decode 3x3 solves.

Where it's going

My vision is for it to help with BLD training, live streams solves reconstructions and later to work as an AI coach.

I put a lot of effort specifically into making it work in live-stream conditions. Most streams today are 30 fps (Euros 2026, Worlds 2025), and with high-TPS solves that makes decoding a real challenge. So I took a slightly different path from the others: I focused mainly on making it work at 1080p and 30 fps, so it can handle live streams of fast solvers (All the solves on the website were decoded at 30 fps, even when a 60 fps version of the video exists).

The first 80% is done, but the last 20% is quite hard. Right now it reconstructs 95%+ of solves that are 10 seconds and above. But on 3–5 second solves the success rate drops to about 70%. the reconstruction themself are not always 100% the same moves that the cuber applied. So, There is still work to do :)

The next steps are making the whole live-stream pipeline faster and more reliable, and building a platform where people can upload their solves (a solve currently takes about 30 seconds to decode on a laptop).

I wanted to finish the "upload your solve" feature before sharing, but I won't be able to keep working on it for the next month and a half. So for now I've made a small website that shows how the model performs and demonstrates its potential. I thought it was better to share it with you now :)

RicoReco.rotohands.com

How the pipeline works

  • A small model that takes a video (an ao5, for example) and finds the inspection, start and finish of each solve
  • A small model that finds a box around the cube
  • A pose model that outputs points on the cube, from which the sticker grid is found
  • A small CNN for reading the colours
  • From the starting state and the evidence in each frame, the candidate moves that were applied to the cube
  • A Viterbi algorithm that prunes the candidates that don't fit
  • A second pass for more accurate 3D tracking of the cube
The toughest part was getting the pose model to high accuracy and high recall. I first tried the smart-cube approach, but it overfitted too much and didn't work well. Then I generated 30k images in Blender to train the pose model. That was a big improvement, but it hit a ceiling. When I investigated, it turned out my Blender images weren't good enough. So I spent a lot of time making high-quality images with hands gripping the cube, fingers covering stickers and so on. That made the pose model great, and from there the rest went fairly well.

I'll write a deeper dive in the future when I have more time.

A special thanks to Stewy and Brest for reconstructing so many solves over the years. It really helped me evaluate the model during development. And thanks to reco.nz for hosting all the reconstructions.

I'm really excited for the future, and I'm sure speedcubing training and live streaming are going to change a lot
Feel free join the discord server and contact me
 
Back
Top