My mighty persistency system - for anyone to use!

[Chebmaster]

* 23.11.2006: A small but significant step towards the true usability.
* 20.09.2006: Ported to Turbo Delphi.
* 13.09.2006: Resolved the conflict with Lazarus that prevented its code completion and Ctrl-click browsing from working.
* 01.09.2006: adressed one issue related to the usage convenience B).
* 31.08.2006: 4 bugs corrected, one of them severe ops:

Project page on SourceForge.net >>

It always irked me that every time you add a new data structure, you need to spend a lot of effort on the boring, non-creative background work of creating the save/load mechanism for it - if you wish it stay persistent. And the published properties are no help. Mainly because there is no mechanism to check if you forgot to include something.

I spent two months resolving exactly the problem of storing complex data structures and backward compatibility of data files. The main idea that all used types are registered, and divided to "binary" (which are saved/loaded as is) and "complex" (each has a custom loading/saving procedure). Then all fields are registered. While saving, all the adjacent "binary" fields are saved as a solid block of bytes, and for all the "complex" fields a corresponding procedure is called. For the string, for example, it first stores the length, then, if the string isn't empty, its contents - and so on.

Cross-links and circular object referencesa are resolved automatically.

Well, aood test example can say more than a thousand docs: feel free to play with it


Content of the Readme file:

1. This stuff was roughly cut from my game engine.
   Hence many extraneous stub functions.

2. The documentation is built into the mo_classes unit
   in the form of comments. See also the *.inc files
   it uses.

3. I should add that average saving/loading speed
   of 1.000.000 objects per second is quite real.
   This stuf is *fast*.

4. The {$minenumsize 4} directive in your program
   is ABSOLUTELY NECESSARY. Do not forget it, 
   or you'll get the corrupt data files that will give
   you free AVs instead of backward compatibility.

5. The AddLog() and Die() procedure variations
   in the mo_stubs_public.inc are a temporary stubs
   using  WriteLn(). You should replace these with
   your own code for displaying log/ error messages.

   The glUint and glFloat types are re defined in 
   mo_tmid_public.h - remove these if you plan 
   to use OpenGL.


6. The following files are taken directly from my
   engine (i.e. these are *shared* files) and thus
   are subject to modification without notice.
   Do not modify them if you want to keep your 
   version up to date (The updated versions may be
   "borrowed" from my CGE distributions, starting
   from the version included with the Test #13
   (NOT Test #12 or below!). As I write these lines,
   the Test #13 is not out yet.
   For example, my future plans include adding
   the compression of data files.

      cl_cfile_func.inc  
      cl_keys.inc
      mo_basket.inc
      mo_classes.pp
      mo_dyna.h
      mo_dyna.inc
      mo_dyna_1template.h
      mo_dyna_1template.inc
      mo_dyna_2template.h
      mo_registrator.inc
      mo_scenario.inc
      mo_trulypersistent.inc
      mo_typeprocs.inc

   the home of CGE is 
       http://chebmaster.narod.ru/cge/cge_e.html

7. The correct one approach is to use classes.
   The record types are not allowed (yet).

8. The mechanism of matching enumerated types by
   their value names is quite powerful, as you 
   can see. You can swap the valuse by places,
   add new ones at the beginning - the converter
   will always assign the correct one. Note, that
   if you *delete* the value, it will be replaced
   at loading with the value which ord() is 0.

9.The object structure can be cross-linked, 
   can contain circular references - the 
   processing mechanism will resolve everything
   automatically (although you may want to raise 
   the stack limit, because the alghoritm is
   recursive). 
   And do NOT touch the _BasketIndex field, or 
   things will go kaboom.

10.The built-in validator checks if the registered
   fields cover the entire instance, alsso if 
   there are any gaps between them, and gives
   comprehensible error messages, refusing to run
   if some clas is registered incorrectly.

[/b]

[Traveler]

Thanks for sharing

[Chebmaster]

It's my contribution, and my thanks to the people who created Free Pascal and The GIMP, libPNG and uncounted other GNU'd things...

But... Perharps, I should have published these from the start:

Strategy: Make your entire game world be contained in a single object, which contains other objects. The TArrayOfTrulyPersistent (a dynamic-array substitution class) should be of great help here. Definition may look a bit (or completely) unclear, since it is implemented via macros. ((look at TDyna and mo_Dyna_xxxxx files more closely - these are practically a substitutes for dynamic arrays, but have much more conveniencies built-in, like the "Last" property, reading the last element, "Add" and "Delete" methods, as well as some optimization towards the minimization of memory reallocation calls. )



text of the test program:
[pascal]{$appmode console}
{$m 10000, 300000000} //Otherwise it crashes with stack overflow (error 202)
{$mode delphi}
{$minenumsize 4}
uses
SysUtils, typinfo, mo_classes;
type
TMyEnum = (Svobu, Dabu, Zigzag, ZugZug);
TTestData = class (TTrulyPersistent)
a: string;
c: TMyEnum;
b: integer;
d, e: TTestData;
procedure RegisterFields; override;
end;
var
TestData: TTestData;
FileVersion: AnsiChar;
procedure TTestData.RegisterFields;
begin
RegType(TypeInfo(TMyEnum));
RegField('a', @a, TypeInfo(string));
RegField('c', @c, TypeInfo(TMyEnum));
RegField('b', @b, TypeInfo(integer));
RegField('d', @d, TypeInfo(TTestData));
RegField('e', @e, TypeInfo(TTestData));
end;
begin
Try
AddLog('Started.');
ClassesRegistrationStart;
RegClass(TTestData);
if not FileExists(ExtractFilePath(ParamStr(0)) + 'testsave.cge')
then begin
TestData:=TTestData.Create;
TestData.a:='My Nice AnsiString';
TestData.b:=777;
Testdata.c:=ZugZug;
TestData.d:=TTestData.Create;

//A circular reference. Scary?.. Hell, no!
TestData.d.e:=TTestData.Create;
TestData.d.d:=Testdata.d;
TestData.d.e.e:=TestData;

WriteLn('Writing...');
//Format 0 (zero, not o) is the simple but fast.
//format s includes MD5 sum calculation and checking.
SaveGame(TestData, 's', ExtractFilePath(ParamStr(0)) + 'testsave.cge');
WriteLn('Success.');
WriteLn;
TestData.Free;
end;
WriteLn('Loading...');
TestData:=LoadGame(ExtractFilePath(ParamStr(0)) + 'testsave.cge', 'TTestData') as TTestData;
WriteLn('Success.');
WriteLn('a = ', TestData.d.e.e.a);
WriteLn('b = ', TestData.d.e.e.b);
WriteLn('c = ', GetEnumName(TypeInfo(TMyEnum), ord(TestData.d.e.e.c)));
Except
WriteLn ('Error! ', (ExceptObject as Exception).Message);
End;
ReadLn;
end.
[/pascal]


[pascal] TTrulyPersistent = class(TObject)
protected
...
public
...
procedure RegisterFields; virtual; abstract;
procedure AfterLoading(); virtual; //is called after the object is loaded.
procedure AfterConstruction(); override;
end;

...

procedure SaveGame(o: TObject; version: AnsiChar; TargetName: AnsiString);
function LoadGame(SourceName: AnsiString; ClassName: WideString): TObject;

...

// ****************** INIT PROCEDURES ********************************
procedure ClassesRegistrationStart;
//!!! procedure ClassesRegistratonEnd;

{ RegClass must be called for each descendant of TTrulyPersistent.
The best place to do it is the initialization procedure of the module.
Automatically calls the "Register" constructor of a class.
}
procedure RegClass(C: CTrulyPersistent);

// ****************** FIELD PROCEDURES *********************************

{
These intended to be called only and only from within the
"Register" constructor of TTrulyPersistent descendants.
(Why didn't I make them its methods...?)
Types should be registered first. )
}
procedure RegField(name: string; Pf: pointer; PI: PTypeInfo);
{ example: RegField('Merry rabbits', @bunnies, TypeInfo(TBunnies));
-- where "bunnies" is a field of type TBunnies}

//Skipped fields: for texture ids, and other stuff that
// cannot be safely restored from the save.
procedure RegSkip(name: string; Pf: pointer; PI: PTypeInfo);
//For pointers and non-registrable classes (technically, pointers too)
procedure RegSkipPtr(name: string; Pf: pointer);

// ****************** TYPE PROCEDURES ************************************

{ These could be called from anywhere after the classes registration started.
Even from the "Register" constructor.}

{ This bunny registers binary types (i.e. ones that could be
dumped into a stream directly, without caring about their structure).
...I wish TypeInfo or TypeData structures had a size field
so that pointing it manually would be unnecessary. Dreams, dreams... }
procedure RegType(Info: PTypeInfo; Size: Integer); overload;

{ This bunny acts the same as the one above, with two following exceptions:
A). It assumes type size is equal to pointer size (4 bytes), and
B). It could be used to register enumeration types which need
a special approach (saved as DWORDs, loaded either as
DWORDs or via special converting routine - depending on the
parsing results).
(Do NOT forget to use the $MINENUMSIZE 4 compiler directive!)}
procedure RegType(Info: PTypeInfo); overload;

{ This bunny allows to deal with complex types like strings,
dynamic arrays - even non-persistent objects! - by introducing
a custom processing procedure (see template in mo_typeprocs.inc).
Its only limitation is that it assumes base type to be a pointer,
so field size is always pointer-sized (4 bytes on 32-bit platform).}
procedure RegType(Info: PTypeInfo; Proc: TCustomTypeProcessingProc); overload;

{ These bunnies register types of corresponding classes.
Note, that persistent classes are registered automatically
when RegClass() is called. So you need to evoke
RegType(C: CTrulyPersistent) manually only in cases of
cross-reference (while class A has fields of type B,
and class B - fields of type A)}
procedure RegType(C: CTrulyPersistent); overload;

// *********** FIELD SAVING / LOADING ROUTINES **********************
{ These can be used in the custom processing procedures}
Function SizeToBufferIndex(s: integer): integer; inline;
procedure WriteInt(i: integer); inline;
function ReadInt: integer; inline;
function PeekInt: integer; inline; //read value but not remove it from buffer

//Don't forget! !!! len is not in bytes but in 32-bit words!
procedure WriteBin(p: pointer; len: integer); inline;
procedure ReadBin(p: pointer; len: integer); inline;
procedure SkipBin(len: integer); inline;
// These three work for anything 32-bit: glFloat, cardinal, integer...
procedure WriteDword(p: pointer); inline;
procedure ReadDWord(p: pointer); inline;
procedure SkipDword(); inline;

//resizes the string prior to writing, to align its size to 32-bit boundary.
// probably, can cause a memory re-allocation call.
procedure WriteAnsiString(v: AnsiString); inline;

//doesn't need the abovementioned trick, because of the additional
// trailing zero (PWideChar compatibility). This extra character
// (two bytes) isn't counted, and in the worst case scenario
// it'll be all that will be rewritten.
procedure WriteWideString(v: WideString); inline;

procedure SkipAnsiString; inline;
procedure SkipWideString; inline;
function ReadWideString(): WideString; inline;
function ReadAnsiString(): AnsiString; inline;
procedure WriteNil; inline;

procedure ReadPersistent(var o: TTrulyPersistent); register;
procedure WritePersistent(o: TTrulyPersistent);

...
[/pascal]

If you want to add more complex types (although that wouldn' be highly advisable) - look the mo_typeprocs as an example.
[pascal]
{ ************** COMPLEX TYPE PROCESSING PROCEDURES ********}
(*
NOTE:
Since the reading and writing is always performed
in 32-bit words, AND it is assumed that *anything*
is aligned to the 32-bit boundary,
the data types whose size isn't a multiply of 4
are a big bad no-no.
Be especially watchful for the packed records.
************************************************** ************
*)
(*
//Procedure template:
procedure RP_ (PField: pointer; OP: TFieldOperation); typeprocscall;
begin
case op of
fio_Load:
fio_Save:
fio_Skip:
end;
end;
*)
...
procedure RP_Persistent (PField: pointer; OP: TFieldOperation); typeprocscall;
var
o: TTrulyPersistent;
begin
case op of
fio_Load: ReadPersistent(TTrulyPersistent(PField^));
fio_Save: WritePersistent(TTrulyPersistent(PField^));
fio_Skip: begin
ReadPersistent(o); //..load it..
MemoryLeakSuspected:=Yes;
//..And do NOTHING.
// All the persistent objects are stored in the special array
// and dealt with special alghorithm.
end;
end;
end;[/pascal]

In the end I want to add, that I plan to evolve this thing, together with my entire engine, and correct any bugs/shortcomings I might have missed. I dreamed of creating it for years.

[dmantione]

Brilliant! This is really the kind of stuff you can be a language more powerfull.

I had a look, but unfortunately you sabotaged your code to be compiled anything besides Windows.

Some comments:
- Would it be possible to make the code do endian conversions? So for example the format on disk is always big endian?
- The registerfields doesn't look nice, it would be much nicer to do it with RTTI. I don't know if compiler can do it, but it can likely be made to output the right information to make this no longer necessary.

[Chebmaster]

your code to be compiled anything besides Windows.

Untrue. It should go for anything intel-32 bit. For example, Linux or FreeBSD. I, myself, indend my engine to support both Windows and Linux, butt the latter shleduled for the later time, because I currently don't have the free hard disk to install the second OS.

- Would it be possible to make the code do endian conversions? So for example the format on disk is always big endian?

Theoretically - yes. Practically... I do not have that kind of hardware to test it, to begin with (note: for Russia the statement "There Is No Such Thing As Macintosh" is 99.9% true).

The cge-file header includes some reserved (zero) fields and contains platform info (sizes of pointer, integer, etc) but this information isn't currently used.

In short, the support of little-endian is theoretically possible but is above my abilities (at least, for now). The same could be said for the 64-bit platforms.

- The registerfields doesn't look nice, it would be much nicer to do it with RTTI.

I wish it was possible... But utfortunately, it is not
RTTI contains ONLY info on published fields.

This is really the kind of stuff you can be a language more powerfull.

The full RTTI containing info on all the fields, types and type sizes - *that* would truly make it more powerful. :roll:

[dmantione]

your code to be compiled anything besides Windows.

Untrue. It should go for anything intel-32 bit. For example, Linux or FreeBSD. I, myself, indend my engine to support both Windows and Linux, butt the latter shleduled for the later time, because I currently don't have the free hard disk to install the second OS.

Perhaps, but if you code like:
[pascal]
{$ifndef win32}^
{$fatal AS OF JUNE 2006 LINUX SUPPORT IS TEMPORARILY DOWN}^
{$ifndef linux}^
{$fatal NOT PORTED TO THIS PLATFORM YET!}^
{$endif}^
{$endif}^
[/pascal]

... it is not going to compile on any Intel 32-bit.

- Would it be possible to make the code do endian conversions? So for example the format on disk is always big endian?

Theoretically - yes. Practically... I do not have that kind of hardware to test it, to begin with (note: for Russia the statement "There Is No Such Thing As Macintosh" is 99.9% true).

Well, it is interresting because it would remove one of the most annoying porting tasks away from the programmer and would therefore make code automatically portable.

This is really the kind of stuff you can be a language more powerfull.

The full RTTI containing info on all the fields, types and type sizes - *that* would truly make it more powerful. :roll:

The compiler can be changed... It'll be quite some work though :?

[Chebmaster]

Perhaps, but if you code like:

Oops... This was meant for the window manager of the CGE core - its Linux version is currently terribly out of date...

You can replace this code with
[pascal]
{$ifdef cge}
{$ifndef win32}
{$fatal AS OF JUNE 2006 LINUX SUPPORT IS TEMPORARILY DOWN}^
{$ifndef linux}
{$fatal NOT PORTED TO THIS PLATFORM YET!}
{$endif}
{$endif}
{$endif}[/pascal]

I updated the archive. Now it should kick and scream only if compiled as a part of my engine.

[Chebmaster]

I have found and corrected 4 bugs.

The download address is the same (see the first post).

1: The boolean type wasn't registered by default.
2: It was impossible to refister any dynamic array fields
3: It was impossible to register any fields of class types that weren't descendants of TTrulyPersistent.
4: At registering a class, the temporary instance was being deleted via regular destructor. A very nasty bug, caused AV at registering any class that deallocated something in its destructor.

[Chebmaster]

01.09.2006:
- added ability of nested registering
(i.e. you now can call RegClass()
from inside the REgisterFields() method.

[Chebmaster]

September, 13 of 2006: A big, fat gingerbread fou you all, people!
I've resolved, at last, the conflict with Lazarus that prevented its code completion and Ctrl-click browsing from working at all. You need Lazarus 0.9.16 or newer for this to be true.

Download page (the old link is not valid anymore).