Here's the video. I had to shoot it from my phone because I needed to add the crosshair (superimposed by the monitor) so it's easier to see exactly where the center is.
So the first 6 seconds is moving right to left then right to left 100 counts, it works perfectly as you can see the crosshair ends up the same place every time at the top of the mountain.
From 8 seconds to 14 seconds I move up and down 100 counts while still having the mountaintop in the center, and clearly it moves a different distance for each movement (in this case resulting in a downwards movement in total).
Lastly from 19 seconds until the end I align the top of the monitor to the top of the mountain and again move 100 counts up and down (exactly same script as above), and as you can see it now works as intended.
So there's something happening when the camera is right behind you and you look "forward", while it works correctly if the camera is looking up or down. I hope you get what I mean