Handling huge amounts of data

[cairnswm]

Here is another option for large datasets. I used these components a long time ago D1/D2 days and I thought they were great.

http://rmarsh.com/2001/06/18/qdb-qui...se-components/

[cronodragon]

Things like this
[pascal]while (IText <Length> #32) do
begin
Part := Part+Text[IText]; [/pascal]
kill performance on the spot. Brutally.

EACH time you add a char to a string, a call to realloc its memory is performed. So you can imagine how "fast" will this code execute.

Ideally, you should make two passes. The first one determines subsstring lengths and positions, the second pass allocates memory by calling SetLength for destination strings (once!) and copies the characters.

Just to inform I made the optimization of that code

I'm allocating a big block to build the string Part, and then I cut it creating another string with the standard Copy() function. In total I make just two allocations in that function, and the speed of the program didn't changed significantly on my 4 years old computer. That should mean using Part := Part+Text[IText] is not a brutal performance killer as suggested. Maybe it is already being optimized by the compiler.

Anyway, that helped me to clear a little my code. Thanks.

[Paizo]

ok i made some test.
I read the text file containing the spoiler and save it in a record as i want. the record is (names are in italian .P ):

Type
  TCardRecord = Record
    Serie : String[25];
    Categoria : String[25];
    Corporazione : String[30];
    Nome : String[30];
    Testo : Array [0..900] of Char;
    L : String[2];
    Errata : Array [0..900] of Char;
    R : String[3];
    C : String[3];
    S : String[3];
    A : String[3];
    V : String[3];
    P : String[2];
    Tipo : String[25];
    Grafico : String[20];
    Cerchietto: String[40];
  End;

then i try to implement binary search as suggested, but i don't understand this:

FSize := FileSize(F);//file size in bytes, return 2047
RSize:=Sizeof(TCardRecord);//Record size in bytes, return 2030

the file i created is around 4 mb; why filesize return this value? (i can read it all with no errors)

[AthenaOfDelphi]

If a file variable links to a typed file (ie f : file of TCardRecord), then the position and size functions work on record numbers.

So, from that I'd say you have 2047 records in your card list.

If you then use Seek(0) (IIRC) you will end up at record 0... the start of the file. Seek(2046) will take you to the last record in the file and Seek(2047) will take you past the end of the file ready to write another record.

To get the file size in bytes, you must connect to it with a file of byte or textfile.

[Paizo]

ok thanks; i thinked so but by reading some manuals they always say that filesize return the size in bytes (for untyped and typed files). Nice to know that