Solved.

The offending line was:
procedure TVirtualFileSystem.GetBlockId(var existingblock: Boolean; iEndBlock: Integer; var blocklistsize: Integer; var BlockList: TBlockList; i: Integer; var b: Int64; var nb: Int64);
Where i did not make blocklistsize var.
As a result of that i and blocklistsize went out of sync. And when writing outside of BlockList it destroys the object based on TVirtualFileSystem.
Wow this was a hard one to debug. Better take more care in the future when using refactoring. And for now give i a more meaningfull name.