The problem was largely that without a reference point, I lost where I was looking and so I couldn't work out how to look back to where I wanted.

Its hard to say, I think I'd personally prefer the option where you push the mouse at the end of the screen and it scrolls, otherwise, the cursor just moves around in that view.