If the axle can fit easily through the speedo spacer, and the spacer is a snug fit in the speedo drive, the spacer is not out of spec.
Can you spin the drive dogs in the speedo drive by hand, with no tight spots or drag? The speedo drive itself is notoriously unreliable, the soft large ring gear inside being worn away until there is almost no engagement with the pinion, and subsequent lockup. I've now taken to buying used Smiths drives for different bikes (as long as the ratio is 15:12 (or same 1.25:1) just hoping to score a decent ring gear. Anyone with a good 15-tooth gear is welcome to contact me and make some money!
The speedo drive body can also spontaneously distort, begin rubbing on the shiny hub cover.
Assuming the speedo drive is good (and it's almost impossible to tell unless you drive out the rivets and take it all the way apart), more likely in your case is the "very thin washer" between the dummy axle and the brake drum as seen above. Or maybe you have a shot bearing?
Good luck, and please let us know what you find.