Those times are when you go back to where you started. The snapping direction (horizontal/vertical) is based on which direction you have moved the most from where you started (pressed the mouse button). Maybe it is unexpected to you because you forgot where you started from?
Shift is used to allow dragging from anywhere within the rectangle.
Alt is used to suspend snapping
I wish I could have used the same modifiers that we use to constrain the direction of keyframes.
You can see the previous discussion here: