Handling huge amounts of data

[cairnswm]

I was reffering directly to the problem you mentioned with ini files - not the whole idea of bulk data stores.

Ini files are nice and I use them a lot - but I dont think you can really expect high performance from them. If you've got a 54MB data file then it really should not be in ini format but in some custom binary format of your own - and then my code is meaning less.

But for small games with limited data requirements there is nothing better than an ini file - its user editable and easy accessible etc. I'd never thought of the Array problem before but when you brought it up I thought it was a great idea to work with and is somethign I will be using reqularly in the future.

[Chesso]

Yeah for using with ini, it could be very usefull.

[Nitrogen]

A warning about ini files: Dont use them if you have more than about 100 bits of data you want to store and retrieve. They become REALLY slow after a while (because of all that string matching), then you have to rip them out and start all over again.

I usually build a ridiculously complex level editor which saves game data in binary form, then read it back into a game class inside the game..

However, it becomes a real pain in the ass when you change the format or even just insert a small bit of data anywhere in the file...

Moral of the story: Use file data types which can cope with change (ie. chunks)

[cragwolf]

For users of Free Pascal (not sure about Delphi), in the Sysutils unit you'll find the following function:

function SScanf(const s: String; const fmt: String;
                const Pointers: Array[] of Pointer): Integer

which is very useful for scanning a string, and extracting information from it in a format that you have the freedom to specify. Very simple to use, too. For example, to parse the following string: s = '200x300+10-10' (a common geometry specification) you would call something like:

SScanf(s, '%dx%d%d%d', [@FWidth, @FHeight, @FLeft, @FTop]);

which would return FWidth = 200, FHeight = 300, FLeft = 10, and FTop = -10.

[Chesso]

That sounds quite interesting, so I could pull a line from a file and use that on it, I'm not sure as of yet if it is or has an equivalent in Delphi either.

But could you show me a direct example involving , seperated data like such:

Draw_Box=20, 20, 100, 100

Which relates to X1, Y1, X2, Y2 (for a framed, or filled rectangle for example).

[Setharian]

it has no equivalent...and sscanf is not the fastest routine either....chunky structure is the best...4 bytes Identifier, 4 bytes Size + x bytes data...allows the insertion of new chunk types without breaking an older loader....

[cronodragon]

I'm using XML files. Are good for structure hierarchies, and are very easy to parse. I also made my own parser/generator, but you can find many components out there. The benefit of XML is that you can also read and write in many different languages, allowing easy data exchange between systems and applications.

[Chebmaster]

When I wrote my first (anf for now the only) game in TurboPascal for MS-DOS, it had a fixed format with a hard-coded limits of object numbers, based on the static arrays. I used a simple BlockWrite() plus additional checksum for all the arrays representing the game objects. Notable is the fact that the whole level was stored in a savegame (around 40 Kbytes total) -- you didn't need the level file to load a game saved while roaming it.
...oh yes, and that game was a 3D FPS.

But now I, even before I started to flesh out my new engine, I anticipated the problem and developed my persistency system (chepersy). Should save a lot of effort on the date storage part, including all the compatibility and performance problems. Nobody seems to be interested, though, despite the fact that it could help a lot of people :cry:

I, honestly, plan to use it for everything, from the save games to 3d models. Because the other known solutions -- Ini files, raw binaries, TPersistent-based streams -- didn't statisfy me. And now, when I almost finished chepersy, they feel like a stone age tools to me.

If somebody at least tried to use chepersy for his own level editor... Honestly, It would benefit him greatly (me too). :roll:

[cragwolf]

Assuming there's a space between the comma and the next number, and s = '20, 20, 100, 100':

SScanf(s, '%d, %d, %d, %d', [@X1, @Y1, @X2, @Y2]);

I agree it's not the fastest way to read data, but if you want to parse a string it can be a useful function.

[cronodragon]

I made my own parsing routine:

[pascal]procedure ExplodeText(
const Text: string;
var Dest: array of string;
const Separator: string = '';
TrimSpaces: Boolean = True
);

var
IText: Integer;
Part: string;

begin
IText := 1;

if Separator = '' then
begin
// Interpret spaces as separators.

while IText <= Length(Text) do
begin
Part := '';

while (IText <= Length(Text)) and (Text[IText] <= #32) do
begin
Inc(IText);
end;

while (IText <Length> #32) do
begin
Part := Part+Text[IText];
Inc(IText);
end;

SetLength(Dest, Length(Dest)+1);
Dest[High(Dest)] := Part;
end;

end else
begin

while IText <= Length(Text) do
begin
Part := '';

while (IText <= Length(Text)) and (Copy(Text, IText,
Length(Separator)) <> Separator) do
begin
Part := Part+Text[IText];
Inc(IText);
end;

SetLength(Dest, Length(Dest)+1);
if TrimSpaces then
Dest[High(Dest)] := Trim(Part) else
Dest[High(Dest)] := Part;
Inc(IText, Length(Separator));
end;

end;

end;[/pascal]

You can use any space (<= #32) as separator, or a given string and remove spaces surrounding the results.

Any syntax error might be due to the translation I just made to post it here.