Checked wiki Quadtree might be good way with possibly frustum culling... Quadtree would simply mean dividing the map in many small maps, where frustum culling would tell which of the small maps are visible. Still trying to avoid too many frustum checks or it lose its purpose (speed). Because of heightmap i would make frustum checks 2 dimensional if at all (if camera is always top-view there's no need for frustum).

I only learned more about vertex arrays little while ago so haven't tried making this all that optimal, but i would imagine this needing several dynamic vertex arrays.