Tripping The Class Fantastic - Versioned Data Storage

A practical application of dynamic method calls

Hi and welcome to another of my articles. This time around I'm going to be covering a couple of topics. The first is pretty obvious from the title... versioned data storage, or to be specific, a simple method of allowing your software to provide a level of backwards compatibility with older versions of its data whilst simultaneously allowing it to update the store to reflect the format of the latest version *Deep breath*

To do this, without creating code that is awash with if..then..else and case statements, I'm going to use dynamic method calls. Not to be confused with the DYNAMIC keyword, these are calls to methods whose names we don't actually know at compile time. So you can experiment (or use the ideas) straight away, I'm also going to present a base class that handles the generic load/save routines for you... all you have to do is provide your implementation specific routines.

This article and the code should work 100% with Delphi (I can't comment on other flavours of Pascal... maybe others could post comments about the support for this technique within their favourite flavour).

So lets get started...

We, as games developers, are (from time to time) likely to require large local data stores for our various media elements. To handle our local storage requirements we have various options open to us. Streaming is one such option.

I'm sure a lot of people will be using (or have at least experimented with) Delphi's built in streaming mechanisms to store data... they are great, but they can have some severe limitations... largely related to the addition (or removal) of items of data. Many times in the past I've created datastores, found I needed to add extra fields to the objects in my code only to be faced with the prospect of recreating my data stores just to add the missing fields.

The answer... versioning... there are lots of possible options here, but I like simplicity where possible as its one less thing to go wrong, so my approach is to write a version header (a single integer) at the start of the objects data within the stream to indicate the version of the data it contains. Reading it back, I read the version indicator and then select the corresponding method for handling the stored data format. There is a little more to it (I don't use TPersistent and I certainly don't use Delphi's built in component streaming), but the crux of it is selecting the required methods when the data is being read and written.

I would imagine everyone is saying use 'if..then..else' or better 'case..'.

If you only have one or two versions these approaches are OK, but they can quickly become an ugly mess.

And thats only 7 versions... during the PGD competition I went through 6 versions of map data and 10 versions of editor data store. And all without actually changing a single line of the code that was handling previous versions or the main load and save routines (thats not strictly true as removing an item of data does require slight modification of older load and save routines, but we'll come to that later).

Before I go any further I think I should just point out the limitation that applies to using this approach... whilst it sounds great... 'you can call a method without knowing its name at compile time' you do have to know a little bit about the method... namely the parameters it expects and any return types for functions.

Now we've got the general idea, lets get into some code. The first thing we need to address is the issue about knowing the parameters and returns types when we finally call our method.

To handle this, we declare prototypes, just like we do for event handling.

In this example, we have declared two prototypes. One procedure and one function. As far as I know, these declarations can use any of the standard rules that apply when defining procedures and functions (variable parameters, constant parameters etc.), but I haven't tried every possible permutation so you may find you need to tweak your code if a particular combination isn't acceptable to the compiler.

Once we have our prototypes, we are pretty much set to start implementing the actual calls. There is just one more rule that we need to be aware of... for this to work, the methods you plan on calling in this way need to be declared as PUBLISHED. This is to ensure that the required information about the methods is available at runtime.

So lets make a call. The only things we need are a couple of variables that will be used during the process and the prototypes of the methods you want to call.

The variables...

The first (MethodPtr) is used to hold the result from the function that is used to locate the required method. The second (Method) is a base prototype equivalent to 'procedure of object'. It has two fields 'Data' and 'Code'. Code is used to provide the entry point for the required method (from MethodPtr) and Data provides a link to the object to which the method belongs. I've only ever been in a situation where Data is set to Self, so I can't comment any further on whether it works if you call methods in another object. However for the sake of clarity I have used a variable called 'targetObject' which we will for this article assume is set to 'Self'.

Anyhow, lets get the rest of the code covered. It is essentially quite simple.

The first thing we need to do is to try and locate the method we want to call. The name of this method is in the string variable 'methName'.

If methodPtr is nil, we haven't found the method... the course of action you take here will obviously depend on the application. If methodPtr is not nil, then we've found a method and we can get on and run it like this...

And that as they say is that. Functions will obviously require a variable to receive the result and the parameter lists will need to match the prototype declarations.

So, lets put this to some real use... a basic object that can be used as the basis for a versioned data store.

This unit is derived from my own streamed object. It just doesn't include some of the more exotic functions that make it more functional. It doesn't do anything fancy, but it works.

I'm not going to do a full breakdown of the code as for the most part it is pretty self explanatory and not too complex. The only thing I really want to discuss is the method search section of the load and save methods.

In both cases, the routines start at a high version (the current fObjectVersion in the case of save and the version read from the stream in the case of load) and work down to 1 looking for the methods load_verx and save_verx respectively.

This makes our data store capable of handling older formats and ensures that when we save the store (during development/editing) it is always stored using the latest data format.

And, in case you're new to streams, lets just mention TReader and TWriter.

TStream (and its descendants) doesn't actually have any methods for reading and writing say an Integer. You can use Read and Write, but then you have to concern yourself with addresses and variable sizes. The easier way (as I have done) is to use TReader and TWriter. These are helper objects that provide methods for direct storage and retrieval of data... TReader.readInteger and TWriter.writeInteger for example. This is not a perfect solution as by denying access to the source and destination streams within the load_verxx and save_verxx methods, life gets difficult if you want to store a stream within the stream, but the trade off is worthwhile because for the majority of the time your life is made considerably easier by using TReader and TWriter.

If you are wondering why you would want to save a stream within a stream... consider TCollection... you have a collection object within your data store object... the easiest way to load and save it is to use its own loadFromStream and saveToStream methods. One approach would be to use an intermediate stream (TStringStream) for example... save the object to the TStringStream and then save the TStringStream's 'DataString' property using TWriter.writeString. Its a bit clunky, but it is one approach to loading and saving streams within the load_verx and save_verx methods.

So thats the basics... now lets put it to use.

Here we have declared the object whose data we want to store. If you have a single version, its that easy. To use the object to store the data you create a stream and call the appropriate method (loadFromStream or saveToStream) like this.

Just a quick note in case its not obvious, you can of course store multiple objects in a single stream... you just have to be sure you read and write them in the same order.

Well, thats a basic case... so lets look at version 2. We have added the field 'MyIntData2'.

Notice that now we have 2 versions we have to override the default value of fObjectVersion in the constructor. This object will save the data using the latest version, so it automatically updates data stores when they are saved.

One thing you will notice is that when you load a version 1 stream, the version 2 fields are not initialised. This can be addressed by initialising them in the version 1 loader, or by modifying the load method to call an initialisation function that sets all fields to a known state.

Now lets look at the version 3 code. Version 3 removes an item of data. This is a little trickier since we have removed the field that holds the data once it is loaded. The item we will remove is fMyIntData. To achieve this, it is necessary to make a modification to the routines that load and save this item (in this case the load_ver1 and save_ver1 methods) and to declare a new set of load and save code for version 3 and beyond.

As you can see, removing items is trickier than adding, but I've found that I generally remove a lot fewer items than I add. Of course you can minimise your work by thinking ahead and designing your code before you get into the nuts and bolts of creating data stores. But if you do dive straight in and you find yourself with a lot of redundant items, I would advise removing them together. The reason is that everytime you remove an item, you need to rewrite the majority of the load_verx/save_verx code as shown in the example above. You will notice that when we add items we call the earlier routines (load_ver2 calls load_ver1 etc). But when we remove items its necessary to break the chain at the version they were removed in (so load_ver3 doesn't call earlier versions), this obviously necessitates writing a whole bunch of code that loads/saves all the fields you haven't removed.

That concludes this little introduction to dynamic method calls and their practical application in constructing versioned data stores. I hope you can see that there are many possibile applications of this ability to call methods dynamically at runtime, but care should be taken when considering whether it is appropriate. I haven't conducted any timings but my gut feeling is that this is not the quickest way in the world. Depending on your application, a big ugly IF..THEN..ELSE or CASE... may be a better approach. But where speed isn't important and you're IF..THEN..ELSEs are getting out of hand, this may be applicable.

With regards to the data storage method presented here, there are other ways to achieve this kind of thing, but with this system you have full control over whats saved and how, you don't have to expose every field as published/public in order to store it and above all, its easy to understand... even for beginners.

As usual, if you have any questions or comments, then please feel free to email me on athena at outer hyphen reaches dot com or post a comment on the article. Thanks for reading. Until next time... take care and happy coding :-)

Another Example

Following the publication of the original article, I was asked a couple of questions about how to use this, so I wrote a more practical application example. This is buried deep in the forums now, so I have included it here for easy reference.

This example is more closely related to the base class I use which includes one or two helper functions that cater for including string length checks and loading and saving streams within your object. These methods are as follows:-

The exact implementations of these aren't relevant at this time, but they exist as protected methods of TStreamedObject.

You will undoubtedly notice the clunkiness of loading/saving streams. This was mentioned in the original article.

So, lets consider a simple map (TMap)... the maps height and width are variable. The map consists of 3 layers. The first layer (TGroundLayer - Descended from TTileLayer) contains the base tile for each cell, the cells movement costs. The other two layers are TTileLayer objects that simply tell the engine what tiles to draw to create overhanging trees for example.

A note about TGroundLayer.load_ver1 and TGroundLayer.save_ver1... this arrangement is not optimal since you will iterate through the map twice (once in the inherited load/save from TTileLayer and then once in TGroundLayers routines), however it does mean that if you add/remove data to TTileLayer you only have to update TTileLayer and the changes will be reflected without having to change the code for TGroundLayer. You could reimplement the code from TTileLayer in the TGroundLayer routines like this:-

This would be quicker since you only iterate through the layers data once, BUT, the two lots of data become mixed and you must then take care to ensure that the load/save routines in TGroundLayer reflect any changes you might make to TTileLayer. This is easy with only a small number of fields and a single version... imagine a lot of fields and multiple versions and you could quickly get in a mess. So, I would advise that (whilst it is not optimal from a speed point of view) you keep the data seperated into their classes as I have done in the example (this is optimal from the point of view of ease of maintenance... not to mention the OOP paradigm).

So how would you use these objects...

Then loading and saving your entire map is as simple as using TMap's loadFromStream and saveToStream routines.

A map that is 10 x 10 (all data set to 0) with the name 'Test Map' occupies 832 bytes on the disk. Consider that the raw data we are storing is 10 x 10 x 2 x 4 (width x height x sizeOf(word) x (ground tile + ground movement + layer 1 tile + layer 2 tile)) = 800 bytes... its pretty good at keeping the size of your data stores down because it doesn't add too much baggage. In reality if all of your map cells (tiles and movement data) were set to 256, then an approximate size would be 10 x 10 x 3 x 4 (width x height x data size x number of layers/data sets) = 1200 bytes. The reason that the data size is 3 (as stored in the stream) rather than 2 (as stored in memory) is that TReader and TWriter store a byte that indicates the size of the data and then the actual bytes that represent the data, but this is based on the actual numerical value and NOT the variable size, so 0 will always be represented by 2 bytes whether it is stored by your object as a byte, integer, word or cardinal. 255 is represented as 2 bytes, but 256 thats 3 bytes... 1 for the size and 2 for the data itself.