You could make a bracket that fits snugly (glued perhaps) over the servo end and have a short rod (welded or bolted on) coaxial with the center of the servo axle and a matching hole in the large bracket (if possible with a bushing of bronze, brass, nylon or PTFE).
I've been thinking about the same problem during the last few days.
I somehow dislike glue and have no means to weld/brase.
My conclusion so far (wip and not yet implemented): create a C shaped piece from an aluminium foil (i've found 1.6 mm and seems good enough) so that it wraps itself on the side of the servo, the upper part screwed on the mounts, the lower part drilled and milled so that it can contain the head of a flat-head screw sticking out and holding in place a flanged ball bearing and the other side of the pivot bracket. The BB is kept in place by a nut and standard washer is inserted between the BB and the servo wrapping.
For improved stability, the "wrapping" can be drilled so that the 4 screws at the bottom of the servo are made to go through the wrapping as well.
The idea is an evolution of this product:
http://www.lynxmotion.com/p-399-injection-molded-servo-hinge-for-ball-bearing-two-pack-standard-hd.aspxThe BB can be found (at less than half the price) from hobbyking.
Nuts and screws from MCMaster.