After removing the plastic ring in the clock face, place a center clock beneath the area you're working on (the spaces where clocks go). The center clock piece you use should be the OPPOSITE color as the side you're working on. The magnets on the center clock will attract the magnets within the clock face, which ensures that they've correctly swapped polarity.
Also, I would work on magnets one position at a time, leaving the ring on top of others that you aren't working on. That way, you've isolated any magnet movement to your your area of focus.
And if you're just switching out the clocks by color and doing nothing else, it won't work because of incorrect magnet polarity.