The input queue works just great, but I have to make two more, one stack at least side where I store all objects that are too far away from the player to be interesting. Then you can go back and forth, everything will remain, and I can have thousands of objects costing only memory.

In the past, I would have used a linked list for this, but now I am using dynamic arrays instead. That allows me to do it with fewer memory operations (granted that I expand the list with many items at a time, which I do).