Is the wheel directly under the pivot? Or does it trail behind the pivot?
Go to the hardware store (or search on-line) and look at casters noting where the wheel's axial is relative the the caster's pivot.
The wheel trails the caster's pivot.
I tried placing less weight on the caster but it didn't seem to help. I also installed a larger caster but the drifting behavior still remained. I installed a ball caster and this seemed to work much better but there is still a little drift to one side. I assume the drift is due to one motor turning very slightly faster that the other. The slight drift I can fix with code but I wonder if this might have been the root cause of the original caster's problem.
If there are any other suggestions or experience I'd still welcome it. I hate leaving problems unsolved!