Results 1 to 10 of 10

Thread: Help large multidimensional arrays

  1. #1

    Smile Help large multidimensional arrays

    I'm trying to create a three dimensional array, I assume it needs to be a pointer. I'm trying to create an array that hold the x,y,z coordinates in the array while inputing the color number like this:

    var
    threeDcolornumberRED : array[-32768..32767,-32768..32767,-32768..32767] of byte;
    threeDcolornumberGREEN : array[-32768..32767,-32768..32767,-32768..32767] of byte;
    threeDcolornumberBLUE : array[-32768..32767,-32768..32767,-32768..32767] of byte;
    ---
    threeDcolornumberRED[X,y,z]:=255;
    threeDcolornumberGREEN[X,Y,Z]:=255;
    threeDcolornumberBLUE[X,y,z]:=255;

    first question is that possible using negatives in the array structure? I got it to compile but it would always crash with access violation error.
    not sure if its the way i was defining the pointer, or am i just running out of memory? I have 8 mb running on windows 7 using free pascal 2.6.2 (win32)

    I need this to work so that I can convert the 2d version of this AI program i made with pixels representing organisms that move around and interact. I want to use the array data to determine their location and also allow them to move through this array I can figure out the details, just need help with defining the array or maybe a different approach.....Id hate to have to write to a file and read it to get the data working right...{slow}

    Not sure if I've given enough info but maybe some one has some ideas. I have tried multiple tutorials on arrays, nothing i tried worked right, sometimes it would compile but would always crash no matter what i tried I even made the arrays smaller and non negative to see if that was the problem didn't matter. So if you could show how you define the pointer stuff as well would be helpful. Thanks in advance folks.
    Last edited by siv@; 18-01-2015 at 05:52 PM. Reason: correction

  2. #2
    Quote Originally Posted by siv@ View Post
    first question is that possible using negatives in the array structure? I got it to compile but it would always crash with access violation error.
    not sure if its the way i was defining the pointer, or am i just running out of memory? I have 8 mb running on windows 7 using free pascal 2.6.2 (win32)
    Negative bounds are OK, but each such array needs 256GB of memory so 8GB is not enough.

    May be you can try to make your organisms represented by array of record with coordinates within the record. It may work if organism quantity not too high.

  3. #3
    PGD Staff / News Reporter phibermon's Avatar
    Join Date
    Sep 2009
    Location
    England
    Posts
    524
    currently you have 3 dimensions each of 65536 elements which I'm guessing you don't want.

    The structure you have at the moment would be one used for volumetric data, like the data you get brain MRI scan, 65536 images, all of them 65536 by 65536 pixels. An absolutely *huge* amount of data!

    Some algorithms for AI that you might of worked on store an image and the 'organisms' are only pixels in that image and data about them isn't stored anywhere else (such as the classic 'game of life')


    However you will have far, far faaaar less organisms than you will have pixels, so it's more efficient to store data per organism :

    Code:
    type
      TOrganism = record
        PositionX : integer;
        PositionY : integer;
        PositionZ : integer;
        ColourR : byte;
        ColourG : byte;
        ColourB : byte;
      end;
    Now lets say you have 1000 organisms in your world :

    Code:
    var
      Organisms : array[0..999] of TOrganism;

    you can access them like this :

    Code:
    Oganisms[22].PositionX := 10;

    Now to render the organisms, rather than iterate (loop) over every pixel in your array and draw its colour, you iterate over your organisms and draw them at their stored position.

    A rough example in 2d :

    Code:
    For I := 0 to high(organisms) do
    begin
      Image.Pixels[organisms[I].PositionX, organisms[I].PositionY].Red := organisms[I].ColourR;
      Image.Pixels[organisms[I].PositionX, organisms[I].PositionY].Green := organisms[I].ColourG;
      Image.Pixels[organisms[I].PositionX, organisms[I].PositionY].Blue := organisms[I].ColourB;
    end;
    Per pixel access will probably be more like : Image.Pixels[2,2] := RGB(RedValue,GreenValue,BlueValue) rather than a seperate byte for each colour but you get the idea.

    Basically flip your logic on it's head, rather than storing all the data on every pixel which as a whole contain all your organisms, just store the data on your organisms and save yourself a ton of memory and speed (it's a lot quicker to draw 1000 pixels than it is 65536x65536 pixels)

    You will probably have more questions so please feel free to ask


    (long story short, you can use arrays like you're trying to use them but you're using such a huge, huge, huuuge number of entries that it's just going to crash and fall over. Whatever it is that you're trying to do, I can promise you that there are far easier ways to do it, just give us more info)
    Last edited by SilverWarior; 18-01-2015 at 07:35 PM. Reason: Wraping code into code fields
    When the moon hits your eye like a big pizza pie - that's an extinction level impact event.

  4. #4
    okay was wondering, i see too much memory needed, so I will try the record method.

  5. #5
    thanks for the explanation yeah the 2d version was similar...just over thinking it with 3d version lol okay, will post my results with this method...and eventually post a new thread on the program mentioned...I may have more questions on some of my procedures I plan to implement in this program. I thought my 'look' procedure for the organisms would be much easier in the large array but...i guess I'll have to rethink how i'll do that, on the 2d version i cheated and just used getpixel lol

  6. #6
    @phibermon
    While your proposed solution is perfectly acceptable for 2D owrlds it might not be verry practical for 3D worlds. Why? In 2D world there is no importance of which wixel representing some unit you draw first and which last. But in 3D world the order of drawn units is verry important so that units that are closer to the viewer arent actually blocked by units that are farther away but were rendered later.

    So better approach would be to use combination multiple two dimensional arrays (each array sotres dta for onw Z plane) and one one dimeansional array to store pointers to these two dimensional arrays and actually controll the Z order of the multiple planes that the two dimensional arrays represent.

    This is aproximately how I would do it. WARNIG the code is untested and uncomplete, but it should point out the basic design.
    Code:
    type
      TOrganism = class
        PositionX: integer;
        ColorR : byte;
        ColorG : byte;
        ColorB : byte;
      end;
    
      TWorldLine = class(TObject)
      private
        FYPos: Integer;
        FOrganisms: Array of TOrganism;
      protected
        function GetOrganism(XPos: Integer): TOrganism;
        procedure SetOrganism(XPos: Integer; AOrganism: TOrganism);
      public
        constructor Create(AYPos: Integer);
        procedure AddOrganism(AOrganism: TOrganism);
        procedure DeleteOrganism(XPos: Integer);
        property Organism[Index: Integer]: TOrganism read GetOrganism write SetOrganism;
      end;
    
      TWorldPlain = class(TObject)
      private
        FZPos: Integer;
        FWorldLines: Array of TWorldLine;
      protected
        function GetOrganism(XPos, YPOs: Integer): TOrganism;
        procedure SetOrganism(XPos, YPos: Integer; AOrganism: TOrganism);
      public
        constructor Create(AZPos: Integer);
        procedure AddOrganism(AOrganism: TOrganism; YPos: Integer);
        procedure DeleteOrganism(XPos, YPos: Integer);
        property Organism[XPos, YPos: Integer]: TOrganism read GetOrganism write SetOrganism;
      end;
    
      TWorld = class(TObject)
      private
        FWorldPlains: Array of TWorldPlain;
      protected
        function GetOrganism(XPos, YPos, ZPos: Integer): TOrganism;
        procedure SetOrganism(XPos, YPos, ZPos: Integer; AOrganism: TOrganism);
      public
        constructor Create;
        procedure AddOrganism(AOrganism: TOrganism; YPos, ZPos: Integer);
        procedure DeleteOrganism(XPos, YPos, ZPos: Integer);
        property Organism[XPos, YPos, ZPos: Integer]: TOrganism read GetOrganism write SetOrganism;
      end;
    
    implementation
    
    procedure TWorldLine.AddOrganism(AOrganism: TOrganism);
    begin
      //
    end;
    
    constructor TWorldLine.Create(AYPos: Integer);
    begin
      FYPos := AYPos;
    end;
    
    procedure TWorldLine.DeleteOrganism(XPos: Integer);
    begin
      //
    end;
    
    function TWorldLine.GetOrganism(XPos: Integer): TOrganism;
    var I: Integer;
    begin
      result := nil;
      //Iterate thorugh all stored organisms
      for I := Low(FOrganisms) to High(FOrganisms) do
      begin
        //Check to see if there is organizm with certain X position
        if FOrganisms[I].PositionX = XPos then
          result := FOrganisms[I];
      end;
    end;
    
    procedure TWorldLine.SetOrganism(XPos: Integer; AOrganism: TOrganism);
    begin
      //
    end;
    
    { TWorldPlain }
    
    procedure TWorldPlain.AddOrganism(AOrganism: TOrganism; YPos: Integer);
    begin
      //
    end;
    
    constructor TWorldPlain.Create(AZPos: Integer);
    begin
      FZPos := AZPos;
    end;
    
    procedure TWorldPlain.DeleteOrganism(XPos, YPos: Integer);
    begin
      //
    end;
    
    function TWorldPlain.GetOrganism(XPos, YPOs: Integer): TOrganism;
    var I: Integer;
    begin
      result := nil;
      for I := Low(FWorldLines) to High(FWorldLines) do
      begin
        //Check to see if Y world plane does exist
        if FWorldLines[I].FYPos = YPos then
        begin
          //Retrieve organizsm through organizms property
          result := FWorldLines[I].Organism[XPos];;
        end;
      end;
    end;
    
    procedure TWorldPlain.SetOrganism(XPos, YPos: Integer; AOrganism: TOrganism);
    begin
      //
    end;
    
    { TWorld }
    
    procedure TWorld.AddOrganism(AOrganism: TOrganism; YPos, ZPos: Integer);
    begin
      //
    end;
    
    constructor TWorld.Create;
    begin
      //
    end;
    
    procedure TWorld.DeleteOrganism(XPos, YPos, ZPos: Integer);
    begin
      //
    end;
    
    function TWorld.GetOrganism(XPos, YPos, ZPos: Integer): TOrganism;
    var I: Integer;
    begin
      result := nil;
      for I := Low(FWorldPlains) to High(FWorldPlains) do
      begin
        if FWorldPlains[I].FZPos = ZPos then
        begin
          result := FWorldPlains[I].Organism[XPos,YPos];
        end;
      end;
    end;
    
    procedure TWorld.SetOrganism(XPos, YPos, ZPos: Integer; AOrganism: TOrganism);
    begin
      //
    end;
    Main advantage of this design is that it actually alows dynamic memory alocation as needed.
    Last edited by SilverWarior; 18-01-2015 at 09:03 PM.

  7. #7
    Quote Originally Posted by siv@ View Post
    i guess I'll have to rethink how i'll do that, on the 2d version i cheated and just used getpixel lol
    Perhaps my suggested approach would come usefull as from the usage point of view it alows you to interact with organisms like if they were stored in 3D aray. The only exceptions are routines for adding and removing them.
    And since in my design I treat organisms as class objects you can simply move organizm from one cell to another by simply canging the references to it. You don't need to copy its data from one part of memory to another as it woulld be with standard arrays.

  8. #8
    PGD Staff / News Reporter phibermon's Avatar
    Join Date
    Sep 2009
    Location
    England
    Posts
    524
    I took an educated guess that our friend isn't ready for such complexity yet and tried to give him/her a window to the next level of understanding. You've got to walk before you can run

    You know I develop 3D engines / ray-tracers and all manner of 3D related code? as well as having two degrees - one in quantitative mathematics and another in computer graphics and visualization? Please understand that I'm not trying to 'show off' but I must make it clear that I know perfectly well what is suitable for 3D.

    If I'm not describing KD/Oct trees or managing entities on the GPU using transform feedback then I'm probably not doing so for a good reason

    Your structure assumes too much at this stage and assumes that this volumetric representation is what the user requires and isn't in fact a misunderstanding. It is a more optimal approach in terms of memory use should you be looking to maintain a 'volumetric cloud' of data but is far from the correct approach should this be the case.

    Let's get some more details first I think it's safe to assume it's a misunderstanding at this stage given the lack of objection to Mirage's suggestion of storing data per organism and my attempt to give an example of that in a way that doesn't exceed the current knowledge of the user, you may of scared them away with your fancy structures


    --

    And rendering back to front vs front to back? in a simple software renderer without a z-buffer then you do indeed want to render back to front as you suggest but throw in a depth buffer as exists on all 3D hardware ever then you actually want to render front to back so that fragments that are obscured by foreground objects can be checked against the z-buffer and not be renderered at all (and indeed any modern attempt at writing a 3D software engine would include a z-buffer)

    The *only* exceptions to this rule are when rendering transparent fragments and even then modern techniques such as depth peeling and stochastic transparency can't be said to be rendering from back to front, depth peeling rendering in both directions at the same time and stochastic requiring no specific rendering order.
    Last edited by phibermon; 18-01-2015 at 10:51 PM.
    When the moon hits your eye like a big pizza pie - that's an extinction level impact event.

  9. #9
    Using an array of organism records will work better for your problem, but for storing large amounts of volumetric data, you could try diving it into chunks. The world from the original post could be represented as a 256x256x256 array of chunks, each sized 256x256x256 (that's 16MiB if each entry is 1 byte). When a chunk is no longer needed, you can save it to file and unload it from memory - and then restore it when the need arises. This can cause some hiccups when you read from disk, but otherwise grants you the possibility to operate on volumetric data, while keeping the memory usage a bit more reasonable.

    @down: Minecraft actually uses 16xALLx16 chunks, so there is no partitioning along the Y-axis. But yeah, the general principle is pretty much the same.
    Last edited by Super Vegeta; 18-01-2015 at 11:29 PM.

  10. #10
    PGD Staff / News Reporter phibermon's Avatar
    Join Date
    Sep 2009
    Location
    England
    Posts
    524
    aka Minecraft, what vegeta (the prince of all saiyans) describes is generally considered to be the best current approach to handling the vast amounts of data (well, until we have voxel hardware in a few more years)

    @up: I didn't know the chunk size. Notch is my hero, bit disappointed that his spaceship coding game didn't see the light of day, would like to do something similar one day
    Last edited by phibermon; 18-01-2015 at 11:43 PM.
    When the moon hits your eye like a big pizza pie - that's an extinction level impact event.

Tags for this Thread

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •