Oh, boy... I"ve had that before. Honestly I can't really remember what my fault was.
Though, IIRC, it's the reason I do the PivotPoint => Center of Object. Then Resetxform.
If you for example rotated an object, then restxform it, it can go whack. Pffff, I can't be of more help really