AFAIK both TObjectList and TDictionary are dynamic and designed for easy insertion and deletion. That is slow.

The best demonstrable game performance were data-pools, where no dynamic memory management is done: just activate and de-activate static memory.