You could use some quadtree/octree optimizations ( http://en.wikipedia.org/wiki/Octree ). This way you only access the objects that are visible to the player.

When objects are not visible and not nearby (say a few miles away) you don't see them, so you don't have to load them. You can load them in separate threads as you like to not stall your game. They call it content streaming. So the best optimization is just to load only the things that you can see or interact with.