Handling huge amounts of data

[marmin]

In a game, the positions , characteristics etc. of 'enemies', (the raw data) , how are they managed by the developer(s)? Is a database used? Or simply a text file that is readed into the program? What is the most common practice.

[czar]

Depends on what you are doing.

In many games it is possible to randomly generate objects characteristics in the game.

However, in the past I have used ini files, textfile, binary files and databases (Access) to store game data.

It all boils down to you requirements.

Databases are nice if you need to store quite a bit and you want to be able to do SQL queries.

[tanffn]

I usually create my own data structure and ~simply dump it to a file (WriteBlock), it is also easy to stream that data using TCP for remote file loading.

[BaronBigman]

How do you send information using a stream?

[AthenaOfDelphi]

The basics of streaming go something like this:-

var
 myFileStream : TFileStream;
begin
 myFileStream:=TFileStream.create('C:\MyFile.dat',fmCreate);
 myFileStream.writeInteger(intData1);
 myFileStream.writeString(strData1);
 myFileStream.free;
end;

To read the data back, you then do something like this:-

var
 myFileStream : TFileStream;
begin
 myFileStream:=TFileStream.create('C:\MyFile.dat',fmOpenRead);
 intData1:=myFileStream.readInteger;
 strData1:=myFileStream.readString;
 myFileStream.free;
end;

The really good thing about streams is that if you have your load and save methods encapsulated as procedures that take the stream as a parameter like this:-

procedure saveMyData(dst:TStream);

You can save the data to a TMemoryStream, TStringStream (any descendant of TStream)... this then allows you to do things like stream the data over a TCP/IP connection for example.

My one bit of advice would be to include versioning information in the stream, so that when you change the data you are storing, you can read the version and then decide how handle it. I know, IIRC, chebmaster has made his persistency system available... whether its appropriate for what you want, only you can decide. I personally favour a more simplistic approach that uses dynamic method calls, to read the version information and then call a method to handle that particular version.

If you want more information on this, let me know and I'll post something.

Hope this helps.

[Chesso]

I usually use my own format in a text file and create a parser for it, but lately iv'e gone with INI files and they work fairly well and are structured well enough.

Although it would be nice if I could....

[PLAYER]
position=0, 0, 0

You know like have comma delimited values in a single keyvalue, although I could read it in as a string and parse it myself, I might aswell just not use INI at all in that case lol.

[czar]

You could use a bits, sort of like the way colour is represented $AABBCCDD then you could store a number up to 256 quite easily.

[Chesso]

But it'd still need to be parsed either pay .

I suppose if it was like this:

POSITION=00250015

So it can up to atleast 4 figures, and if the zero's are there, I know exactly where the numbers will be (either the first 4 or the last 4).

But I think using comma's is a little easier to read, if need be.

[cairnswm]

Why not make your own ini class that allows an array of string for a value - how about like this:

[pascal]
unit GameIni;

interface

uses
IniFiles, Classes;

Type
TGameIni = Class(TiniFile)
Function ReadArrayString(const Section, Ident: String; Default : Array of String) : TStringList;
End;

implementation

{ TGameIni }

function TGameIni.ReadArrayString(const Section, Ident: String;
Default: Array of String): TStringList;
Var
Values, S : String;
SA : TStringList;
I : Integer;
begin
Values := ReadString(Section,Ident,'');
SA := TStringList.Create;
If Values = '' then
Begin
For I := 0 to High(Default) do
SA.Add(Default[I]);
End
Else
Begin
Values := Values + ',';
While Pos(',',Values) > 0 do
Begin
S := Copy(Values,1,Pos(',',Values)-1);
SA.Add(S);
Values := Copy(Values,Pos(',',Values)+1,Length(Values));
End;
End;
Result := SA;
end;

end.
[/pascal]

and you'll be able to use it like this:
[pascal]
procedure TForm1.Button1Click(Sender: TObject);
Var
Ini : TGameIni;
begin
Ini := TGameIni.Create(ExtractFilePath(Application.ExeNam e)+'/Game.ini');
ListBox1.Items.AddStrings(Ini.ReadArrayString('Tes t','List',['a','b','c']));
end;
[/pascal]

[Chesso]

Never thought of doing that, I usually use the quite fast Copy() function (instead of slower bogged down ones that eventually call Copy()) and just find the position of the first comma, grab what I need, remove anything before, go to the next comma, remove anything before and so on.

I made a little app that did this once to convert a 54mb file worth of data into a small (few mb's) sql like data syntax and it could do it roughly in 4 seconds .

P.S. And that's just using the basic file methods, rather than going down any further, I could never understand how to use filestreams and what not.