The corners are the only things you should need to sand down. The edges just have holes - no need to sand them.
And yeah, sanding them down by hand is pretty painful.
I just tried to do my second one before my competition this weekend, but I quit without really getting the bumps sanded down very well, so it didn't help all that much. You really have to get it down a long way before it helps enough.
I think I'm going to buy an electric sander this week sometime if I can find the time. Sanding by hand is just too painful.