I went out and got a new 60 grit flap wheel yesterday and spent 20 mins on one of the drums this morning, just to see what difference a new flap wheel would make compared with my old, tired one.
You can see the results below. Top drum as they were with the old flap wheel. Bottom with the new one. Might be able to avoid the lathe after all.