The problem was largely that without a reference point, I lost where I was looking and so I couldn't work out how to look back to where I wanted.
Its hard to say, I think I'd personally prefer the option where you push the mouse at the end of the screen and it scrolls, otherwise, the cursor just moves around in that view.
Bookmarks