I'm back from few weeks holiday. I have no idea what I am doing but somehow I manage to make some progress.

Stutter is fixed, frustrum culling is working much better too (only on one plane, need to add 4 more.