Yes but not very much, those shaft couplers deal with a good amount of torquing already, and if you remove too much material, it could easily break/shear off if put under load. Plus a drill press isn't the most accurate tool for the job, if I were you, I would leave it alone.