Following Harry's comment I tried to change my program to FullScreen mode and again tried SystemMemory:=True/False;
When True in Full screen mode I got a frame rate of about 25, as opposed to 1 when not full screen.
When False I get a frame rate of about 100 which is the same as when not fullscreen.

I think its wierd