Handling huge amounts of data

[Chebmaster]

Things like this
[pascal]while (IText <Length> #32) do
begin
Part := Part+Text[IText]; [/pascal]
kill performance on the spot. Brutally.

EACH time you add a char to a string, a call to realloc its memory is performed. So you can imagine how "fast" will this code execute.

Ideally, you should make two passes. The first one determines subsstring lengths and positions, the second pass allocates memory by calling SetLength for destination strings (once!) and copies the characters.

As a person who is greatly annoed by the lagre save/load times, I pay a big attention to such matter. There are fields that do not worth optimization. This is *not* one of these.

P.S. My game, even on the 80386, performed quicksave and quickload as fast as you can tap a key - unlike, for example, doom/doom2. So I know what I am talking about.

[Chesso]

What I do is basically use Pos (or similar), Copy and Delete.

So if it's a line like this:

Bleh=40, 40, 40, 400

I do this:

Get the position of the first ,.
Copy everything beforehand.
Remove everything beforehand.
Rinse and repeat.

Dunno if it's the best/fastest method but it seemed pretty quick for my from my tests (compared to other methods and using other functions).

Faster than Q_Strings too.

[Nitrogen]

I'd guess that XML is only slightly faster than INI files, and dont forget all those bulky XML libraries required to read it.

XML is too much of a buzz word, people throw it in because they can and because they think its a cool thing to use, not because it's best suited to their program (there are exceptions of course).

[Chesso]

XML from what I know and have heard, is basically not that great performance wise, I high doubt it outperforms ini files, or atleast wouldn't if ini file handler was improved (as the basic fact is, an INI file is simpler then an XML file, atleast as far as I know).

But people shouldn't be using it for it's performance, but for it's ease of structure.

If you want performance, XML isn't the way to go, nor is INI.

[cronodragon]

Things like this
[pascal]while (IText <Length> #32) do
begin
Part := Part+Text[IText]; [/pascal]
kill performance on the spot. Brutally.

EACH time you add a char to a string, a call to realloc its memory is performed. So you can imagine how "fast" will this code execute.

Ideally, you should make two passes. The first one determines subsstring lengths and positions, the second pass allocates memory by calling SetLength for destination strings (once!) and copies the characters.

Thanks for pointing that out. I still haven't concentrated on lower level optimizations... I'm solving the general problems for my engine, and making all the code to be as useful as possible. For example that function I posted helps to parse data in many different situations. And while reading you comment, I was thinking that is not hard to optimize that function, if the problem is allocating one character at a time, I would just allocate in big blocks (i.e. 256 bytes) and finally cutting it down to the required size. On the other hand, I wouldn't use parsing functions on places that are time critical anyway. It always depends on the given problem.

XML from what I know and have heard, is basically not that great performance wise, I high doubt it outperforms ini files, or atleast wouldn't if ini file handler was improved (as the basic fact is, an INI file is simpler then an XML file, atleast as far as I know).

But people shouldn't be using it for it's performance, but for it's ease of structure.

If you want performance, XML isn't the way to go, nor is INI.

That's true. And XML is for easy exchange of data between systems.

[Paizo]

i have a question about large data management,
I need to use a database of cards for a cards game (like magic), at the moment the data is in a .xls file (around 900kb).

information like name of the card, text, name of the images...

i think to save it in .csv and read it with delphi, the question is:

can i use the power of sql query on this file "somehow"?
Do you have any suggestions how to better store this data and manage it?

[AthenaOfDelphi]

Hi Paizo,

What kind of queries do you want to run against the data?

You can use local files to provide data via TClientDataset, but I'm not sure about querying it as I've never used that mechanism for data storage.

But, depending on the kind of information you want to search on (and the number of cards in the list and how quickly you need an answer) you should be able to do it without a database.

The key issues though are the kind of searches you want to run (some examples would help), the number of cards and the amount of time you are prepared to spend searching. I guess the answers to these questions will depend on the kind of game you're writing.

If you can provide a little more information then it will help immensely :-)

[Paizo]

ok, i'm here

i want to use only the 'select' command for 2 services:

a)you are looking for a card in the database, you don't know the name of the card but you know that (for example) the card is green and have a certain icon. So something fast for searching cards

b)every players have their own decks with n > 59 cards, not random, a player choose them as he wish. So at the start of the game i have to load all data about this cards (ie images and text) and store that data in (for example) a Timagelist on a custom sprite engine.

hope its clear.

[AthenaOfDelphi]

Certainly is... I'd say that whilst you don't want users sitting around waiting, searching manually (i.e. do a whole bunch of comparisons against your in memory data) isn't going to be too slow for your application.

So with that in mind, you could use a collection.

Create an object descended from TCollectionItem that holds the information about the cards, then write yourself a tool to manage the collection. Players decks could also be held as a collection... maybe create an object call TDeck descended from TCollection that implements the methods you need to search through the cards.

Once you've loaded the deck, you could run through the items it contains and extract the information you require for things like your image lists etc. Again, this shouldn't take too long to execute because the data is all held in memory.

I'm guessing someone will popup and say that this isn't an optimal solution... but from what you've said its more than adequate for your purposes.

Just my $0.02 ;-)

[Nitrogen]

Yea, you not going to be able to get at the data inside an .xls file (text, images etc) without serious effort, or converting it to another format like you suggest..

And a fully fledged database is WAAY overkill for your requirements...

The quickest and easiest would be to use something like the Collections mentioned above.