Page 2 of 5 FirstFirst 1234 ... LastLast
Results 11 to 20 of 45

Thread: Handling huge amounts of data

  1. #11
    Legendary Member cairnswm's Avatar
    Join Date
    Nov 2002
    Location
    Randburg, South Africa
    Posts
    1,537

    Handling huge amounts of data

    I was reffering directly to the problem you mentioned with ini files - not the whole idea of bulk data stores.

    Ini files are nice and I use them a lot - but I dont think you can really expect high performance from them. If you've got a 54MB data file then it really should not be in ini format but in some custom binary format of your own - and then my code is meaning less.

    But for small games with limited data requirements there is nothing better than an ini file - its user editable and easy accessible etc. I'd never thought of the Array problem before but when you brought it up I thought it was a great idea to work with and is somethign I will be using reqularly in the future.
    William Cairns
    My Games: http://www.cairnsgames.co.za (Currently very inactive)
    MyOnline Games: http://TheGameDeveloper.co.za (Currently very inactive)

  2. #12

    Handling huge amounts of data

    Yeah for using with ini, it could be very usefull.

  3. #13

    Handling huge amounts of data

    A warning about ini files: Dont use them if you have more than about 100 bits of data you want to store and retrieve. They become REALLY slow after a while (because of all that string matching), then you have to rip them out and start all over again.

    I usually build a ridiculously complex level editor which saves game data in binary form, then read it back into a game class inside the game..

    However, it becomes a real pain in the ass when you change the format or even just insert a small bit of data anywhere in the file...

    Moral of the story: Use file data types which can cope with change (ie. chunks)
    My site: DelphiTuts.com (coming soon)...

    Download Font Studio 4.21 here.

  4. #14

    Handling huge amounts of data

    For users of Free Pascal (not sure about Delphi), in the Sysutils unit you'll find the following function:

    Code:
    function SScanf(const s: String; const fmt: String;
                    const Pointers: Array[] of Pointer): Integer
    which is very useful for scanning a string, and extracting information from it in a format that you have the freedom to specify. Very simple to use, too. For example, to parse the following string: s = '200x300+10-10' (a common geometry specification) you would call something like:

    Code:
    SScanf(s, '%dx%d%d%d', [@FWidth, @FHeight, @FLeft, @FTop]);
    which would return FWidth = 200, FHeight = 300, FLeft = 10, and FTop = -10.
    [size=10px]"In science one tries to tell people, in such a way as to be understood by everyone, something that no one ever knew before. But in poetry, it's the exact opposite." -- Paul Dirac[/size]

  5. #15

    Handling huge amounts of data

    That sounds quite interesting, so I could pull a line from a file and use that on it, I'm not sure as of yet if it is or has an equivalent in Delphi either.

    But could you show me a direct example involving , seperated data like such:

    Draw_Box=20, 20, 100, 100

    Which relates to X1, Y1, X2, Y2 (for a framed, or filled rectangle for example).

  6. #16

    Handling huge amounts of data

    it has no equivalent...and sscanf is not the fastest routine either....chunky structure is the best...4 bytes Identifier, 4 bytes Size + x bytes data...allows the insertion of new chunk types without breaking an older loader....

  7. #17

    Handling huge amounts of data

    I'm using XML files. Are good for structure hierarchies, and are very easy to parse. I also made my own parser/generator, but you can find many components out there. The benefit of XML is that you can also read and write in many different languages, allowing easy data exchange between systems and applications.

  8. #18

    Handling huge amounts of data

    When I wrote my first (anf for now the only) game in TurboPascal for MS-DOS, it had a fixed format with a hard-coded limits of object numbers, based on the static arrays. I used a simple BlockWrite() plus additional checksum for all the arrays representing the game objects. Notable is the fact that the whole level was stored in a savegame (around 40 Kbytes total) -- you didn't need the level file to load a game saved while roaming it.
    ...oh yes, and that game was a 3D FPS.

    But now I, even before I started to flesh out my new engine, I anticipated the problem and developed my persistency system (chepersy). Should save a lot of effort on the date storage part, including all the compatibility and performance problems. Nobody seems to be interested, though, despite the fact that it could help a lot of people :cry:

    I, honestly, plan to use it for everything, from the save games to 3d models. Because the other known solutions -- Ini files, raw binaries, TPersistent-based streams -- didn't statisfy me. And now, when I almost finished chepersy, they feel like a stone age tools to me.

    If somebody at least tried to use chepersy for his own level editor... Honestly, It would benefit him greatly (me too). :roll:

  9. #19

    Handling huge amounts of data

    Assuming there's a space between the comma and the next number, and s = '20, 20, 100, 100':

    Code:
    SScanf(s, '%d, %d, %d, %d', [@X1, @Y1, @X2, @Y2]);
    I agree it's not the fastest way to read data, but if you want to parse a string it can be a useful function.
    [size=10px]"In science one tries to tell people, in such a way as to be understood by everyone, something that no one ever knew before. But in poetry, it's the exact opposite." -- Paul Dirac[/size]

  10. #20

    Handling huge amounts of data

    I made my own parsing routine:

    [pascal]procedure ExplodeText(
    const Text: string;
    var Dest: array of string;
    const Separator: string = '';
    TrimSpaces: Boolean = True
    );

    var
    IText: Integer;
    Part: string;

    begin
    IText := 1;

    if Separator = '' then
    begin
    // Interpret spaces as separators.

    while IText <= Length(Text) do
    begin
    Part := '';

    while (IText <= Length(Text)) and (Text[IText] <= #32) do
    begin
    Inc(IText);
    end;

    while (IText <Length> #32) do
    begin
    Part := Part+Text[IText];
    Inc(IText);
    end;

    SetLength(Dest, Length(Dest)+1);
    Dest[High(Dest)] := Part;
    end;

    end else
    begin

    while IText <= Length(Text) do
    begin
    Part := '';

    while (IText <= Length(Text)) and (Copy(Text, IText,
    Length(Separator)) <> Separator) do
    begin
    Part := Part+Text[IText];
    Inc(IText);
    end;

    SetLength(Dest, Length(Dest)+1);
    if TrimSpaces then
    Dest[High(Dest)] := Trim(Part) else
    Dest[High(Dest)] := Part;
    Inc(IText, Length(Separator));
    end;

    end;

    end;[/pascal]

    You can use any space (<= #32) as separator, or a given string and remove spaces surrounding the results.

    Any syntax error might be due to the translation I just made to post it here.

Page 2 of 5 FirstFirst 1234 ... LastLast

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •