The averages don't really do color neutrality justice. Yes, if you can always find the best color on the cube, then you save two moves overall. I'm completely color neutral and I always have been, it never occurred to me to always use the same color. I'm not particularly fast overall, I'm sup-30 in practice, but I can reliably solve my cross in 5 seconds.
I basically scan the cube for an easy cross, looking for edge/center pairs solved, especially any side with more than one edge, even if the edges are not permuted correctly. If I don't find an "easy" color, I quickly look for a color that has obvious good edges, then pick it. That all happens in the first five seconds of inspection, then I plan the rest of my cross. I'm not really good enough to plan an extended cross unless it's really obvious, I'm hoping that as I get better I'll recognize X-cross situations more readily.
When I find an "easy" color it's mostly just a single corner/edge pair connected, but maybe one in three scrambles has two corner/edges connected and maybe half of those, the two pairs are permuted correctly. That leads to a seriously fast cross, maybe two seconds. When I don't find an easy cross, I'm pretty much in the same boat as any solver, but because I look for a color that has at least a couple of "good" edges, I can almost always solve my cross in seven moves or fewer. Seldom to I use more than 9 moves for my cross, excepting what I'll describe next.
The funny thing (and this is a point to Lucas in his "finger friendliness" argument) when I have an easy color, I usually use more fingertricks to solve the cross. More often than not the edges will connect with their center, but not be permuted properly, so I will use a 5 move fingertrick to swap two adjacent edges after putting the other two correctly, or use R' u R' u' to flip a single edge in place. An easy color can lead to longer crosses in ftm, but because of the finger tricks, still end up fast.
I can see how memorizing the order of your equatorial center colors can help in your cross solution if you always solve the same color, but it really is simple to look at the two E centers you can see and deduce the colors all the way around in a split second.
Overall, I think I'll say it definitely took me longer to get decent at cross solving than if I'd started single color, but I think the benefits outweigh the effort.
The "average" of two moves saved belies two salient points:
1) I don't believe most people solve their cross optimally. Lucas G. has spoken most to this point, he's an uber fast solver in my book, but relies on "finger frienliness" to solve quickly as opposed to in the fewest moves. So, if the optimal solution is 7 moves and you use 9, the savings of finding an easy color with an optimal solution of 4 moves, but you use 6 is a much more significant savings.
2) It belies "holy crap" factor that when you do find a super cool 3 move cross on blue, it a) allows a ton more time to find your first pair and beyond, and b) fires you up for a good solve.
I don't believe that overall I benefit more than two or three moves per solve by being color neutral, but when I do, it's really, really awesome.
I wall add a footnote that there are some drawbacks to color neutrality. Single color solvers seldom find they've placed four edges, correctly oriented and permuted, around the wrong center. I do that.
They also don't have to worry about forgetting the color they're solving, I do that too, but you just have to look at the U center to remind yourself. These are things I'm actively working on.
Finally, I have a question for other color neutral solvers out there. As an experiment for a friend on the forum I went three weeks only solving cross on white. I didn't find any difference overall in my times (two Sunday contest averages after two and three weeks) or my efficiency in the cross (subjective impression). I'd like to hear about any other color neutral solvers that have done something similar or encourage you to try it yourselves and report back.
