I don't have experience in such advanced stuff, but it sounds like you might need a different approach. If object is too big for file then it is more than likely too big for video memory aswell - too slow to render if its at all possible to open.

I would think no game uses a single model for terrain or full buildings. Instead self made editors which combine shapes and many objects.

OBJ is big because it is text inside. 3DS and many others are binary but if you like the challenge (and isn't hard) you can make converter that reads OBJ and saves it again in binary format. Floating point numbers will each fit in 4 bytes instead of for example "900.512334" string that is separated by line change too.