You should load all data when the application starts and unload it when your app terminates. Loaded data doesn't hurt performance.

However, you should limit the ammount of geometry you render every frame. Only draw object that are visible to the camera. You could use frustum culling for this.