About 3 months ago I committed myself to become fully color neutral and these days I practice starting on different colors on a regular basis.
I found that after you get over the initial scare, it becomes natural very quickly and it's a lot of fun!
My main ambition is to get a fully color neutral average of 12 on video (doing 2 solves on each color) that is close to my normal average (15s).
I think you misunderstand the point of being colour neutral. Either that, or you are going about proving it to yourself in completely the wrong way. If I get 10 crosses on white in an average of 12, I'm still colour neutral. White just happens to stick out to me in those 10 solves.
Forcing yourself to use a particular colour on a particular solve is not being colour neutral.
Finding a cross that looks nice to you and being able to solve it, whichever colour it happens to be, without that particular colour having a positive or negative effect in general on your solving times, is being colour neutral.