I started with 2 colours, W/Y then 4 W/Y/R/O then 6. I found it easier. But I do Roux, so I now work on a 5 colour neutrality scheme ie: W/Y on left then any of the remaining four as a base. It's probably easier, but the same principle.
Do the same scramble 6 times, each time starting the cross on a different side. Do an average of 5, play the color neutrality game on each solve. Do no-inspection solves. How fast are you when you stick to the [color] cross anyway?