I don’t think you are necessarily doing anything wrong, it just goes back to last drone position if I’m not mistaken.
I’m not at my PC right now, but if my view is “off” I press 5 on my numpad. Assuming I’m using the default command, it is Reset Drone Target offset.