With all the different internal clearances available to make sure that the bearing works within a very tight tolerance, on a very important part, doing a very important thing, I'd vote no.
Losing .0015" is a lot of polishing. A bright and shiny finish is much less important than the dimension. It's nice to have both, but the dimension is the critical one.