As I suspected pixel by pixel scrolling turned out to be much more tricky than I had imagined.

It currently works if you are travelling up the screen, all the other axis are currently not working.
I'll come bakc to this later and fill out the details.