Help large multidimensional arrays

[phibermon]

currently you have 3 dimensions each of 65536 elements which I'm guessing you don't want.

The structure you have at the moment would be one used for volumetric data, like the data you get brain MRI scan, 65536 images, all of them 65536 by 65536 pixels. An absolutely *huge* amount of data!

Some algorithms for AI that you might of worked on store an image and the 'organisms' are only pixels in that image and data about them isn't stored anywhere else (such as the classic 'game of life')


However you will have far, far faaaar less organisms than you will have pixels, so it's more efficient to store data per organism :

type
  TOrganism = record
    PositionX : integer;
    PositionY : integer;
    PositionZ : integer;
    ColourR : byte;
    ColourG : byte;
    ColourB : byte;
  end;

Now lets say you have 1000 organisms in your world :

var
  Organisms : array[0..999] of TOrganism;

you can access them like this :

Oganisms[22].PositionX := 10;

Now to render the organisms, rather than iterate (loop) over every pixel in your array and draw its colour, you iterate over your organisms and draw them at their stored position.

A rough example in 2d :

For I := 0 to high(organisms) do
begin
  Image.Pixels[organisms[I].PositionX, organisms[I].PositionY].Red := organisms[I].ColourR;
  Image.Pixels[organisms[I].PositionX, organisms[I].PositionY].Green := organisms[I].ColourG;
  Image.Pixels[organisms[I].PositionX, organisms[I].PositionY].Blue := organisms[I].ColourB;
end;

Per pixel access will probably be more like : Image.Pixels[2,2] := RGB(RedValue,GreenValue,BlueValue) rather than a seperate byte for each colour but you get the idea.

Basically flip your logic on it's head, rather than storing all the data on every pixel which as a whole contain all your organisms, just store the data on your organisms and save yourself a ton of memory and speed (it's a lot quicker to draw 1000 pixels than it is 65536x65536 pixels)

You will probably have more questions so please feel free to ask


(long story short, you can use arrays like you're trying to use them but you're using such a huge, huge, huuuge number of entries that it's just going to crash and fall over. Whatever it is that you're trying to do, I can promise you that there are far easier ways to do it, just give us more info)

[siv@]

thanks for the explanation yeah the 2d version was similar...just over thinking it with 3d version lol okay, will post my results with this method...and eventually post a new thread on the program mentioned...I may have more questions on some of my procedures I plan to implement in this program. I thought my 'look' procedure for the organisms would be much easier in the large array but...i guess I'll have to rethink how i'll do that, on the 2d version i cheated and just used getpixel lol

[SilverWarior]

i guess I'll have to rethink how i'll do that, on the 2d version i cheated and just used getpixel lol

Perhaps my suggested approach would come usefull as from the usage point of view it alows you to interact with organisms like if they were stored in 3D aray. The only exceptions are routines for adding and removing them.
And since in my design I treat organisms as class objects you can simply move organizm from one cell to another by simply canging the references to it. You don't need to copy its data from one part of memory to another as it woulld be with standard arrays.

[phibermon]

I took an educated guess that our friend isn't ready for such complexity yet and tried to give him/her a window to the next level of understanding. You've got to walk before you can run

You know I develop 3D engines / ray-tracers and all manner of 3D related code? as well as having two degrees - one in quantitative mathematics and another in computer graphics and visualization? Please understand that I'm not trying to 'show off' but I must make it clear that I know perfectly well what is suitable for 3D.

If I'm not describing KD/Oct trees or managing entities on the GPU using transform feedback then I'm probably not doing so for a good reason

Your structure assumes too much at this stage and assumes that this volumetric representation is what the user requires and isn't in fact a misunderstanding. It is a more optimal approach in terms of memory use should you be looking to maintain a 'volumetric cloud' of data but is far from the correct approach should this be the case.

Let's get some more details first I think it's safe to assume it's a misunderstanding at this stage given the lack of objection to Mirage's suggestion of storing data per organism and my attempt to give an example of that in a way that doesn't exceed the current knowledge of the user, you may of scared them away with your fancy structures


--

And rendering back to front vs front to back? in a simple software renderer without a z-buffer then you do indeed want to render back to front as you suggest but throw in a depth buffer as exists on all 3D hardware ever then you actually want to render front to back so that fragments that are obscured by foreground objects can be checked against the z-buffer and not be renderered at all (and indeed any modern attempt at writing a 3D software engine would include a z-buffer)

The *only* exceptions to this rule are when rendering transparent fragments and even then modern techniques such as depth peeling and stochastic transparency can't be said to be rendering from back to front, depth peeling rendering in both directions at the same time and stochastic requiring no specific rendering order.

[Super Vegeta]

Using an array of organism records will work better for your problem, but for storing large amounts of volumetric data, you could try diving it into chunks. The world from the original post could be represented as a 256x256x256 array of chunks, each sized 256x256x256 (that's 16MiB if each entry is 1 byte). When a chunk is no longer needed, you can save it to file and unload it from memory - and then restore it when the need arises. This can cause some hiccups when you read from disk, but otherwise grants you the possibility to operate on volumetric data, while keeping the memory usage a bit more reasonable.

@down: Minecraft actually uses 16xALLx16 chunks, so there is no partitioning along the Y-axis. But yeah, the general principle is pretty much the same.

[phibermon]

aka Minecraft, what vegeta (the prince of all saiyans) describes is generally considered to be the best current approach to handling the vast amounts of data (well, until we have voxel hardware in a few more years)

@up: I didn't know the chunk size. Notch is my hero, bit disappointed that his spaceship coding game didn't see the light of day, would like to do something similar one day