Versioned Data Storage

Hi and welcome to another of my articles. This time around I'm going to be covering a couple of topics. The first is pretty obvious from the title... versioned data storage, or to be specific, a simple method of allowing your software to provide a level of backwards compatibility with older versions of its data whilst simultaneously allowing it to update the store to reflect the format of the latest version *Deep breath*

To do this, without creating code that is awash with if..then..else and case statements, I'm going to use dynamic method calls. Not to be confused with the DYNAMIC keyword, these are calls to methods whose names we don't actually know at compile time. So you can experiment (or use the ideas) straight away, I'm also going to present a base class that handles the generic load/save routines for you... all you have to do is provide your implementation specific routines.

This article and the code should work 100% with Delphi (I can't comment on other flavours of Pascal... maybe others could post comments about the support for this technique within their favourite flavour).

So lets get started...

We, as games developers, are (from time to time) likely to require large local data stores for our various media elements. To handle our local storage requirements we have various options open to us. Streaming is one such option.

I'm sure a lot of people will be using (or have at least experimented with) Delphi's built in streaming mechanisms to store data... they are great, but they can have some severe limitations... largely related to the addition (or removal) of items of data. Many times in the past I've created datastores, found I needed to add extra fields to the objects in my code only to be faced with the prospect of recreating my data stores just to add the missing fields.

The answer... versioning... there are lots of possible options here, but I like simplicity where possible as its one less thing to go wrong, so my approach is to write a version header (a single integer) at the start of the objects data within the stream to indicate the version of the data it contains. Reading it back, I read the version indicator and then select the corresponding method for handling the stored data format. There is a little more to it (I don't use TPersistent and I certainly don't use Delphi's built in component streaming), but the crux of it is selecting the required methods when the data is being read and written.

I would imagine everyone is saying use 'if..then..else' or better 'case..'.

If you only have one or two versions these approaches are OK, but they can quickly become an ugly mess.

And thats only 7 versions... during the PGD competition I went through 6 versions of map data and 10 versions of editor data store. And all without actually changing a single line of the code that was handling previous versions or the main load and save routines (thats not strictly true as removing an item of data does require slight modification of older load and save routines, but we'll come to that later).

Before I go any further I think I should just point out the limitation that applies to using this approach... whilst it sounds great... 'you can call a method without knowing its name at compile time' you do have to know a little bit about the method... namely the parameters it expects and any return types for functions.