Thyandyr's voxel engine

[Thyandyr]

DVE - Delphi Voxel Engine

https://github.com/Thyandyr/DVE

[Thyandyr]

26/10/17 Didn't get anything done today. Trying to figure out best way to do chunk management.
29/10/17 Some progress on chunk management. Next need to make my OpenGL code compatible with arbitrary amount of objects. Too late in the night to start that.

[Ñuño Martínez]

I see it is minecrafty. I'm not saying its bad.

Since you're not using any framework, may be it can be used with SDL or Allegro or GLScene, can it?

Also, did you plan to make it compatible with FreePascal?

[Thyandyr]

I see it is minecrafty. I'm not saying its bad.

Since you're not using any framework, may be it can be used with SDL or Allegro or GLScene, can it?

Also, did you plan to make it compatible with FreePascal?

It looks like that yes, but it won't be minecrafty. Gamewise, I am juggling between Dwarf fortress, Master of Magic, UFO etc. It'll definitely be a management game. I like stuff like evolution and fuzzy logic and geological evolution. I have no idea how those ingredients turns into a game :D Maybe it'll just be my own sandbox for weird ideas.

The engine itself should end up being relatively reusable so that I can use it for various things. Also because I do not know what I am doing, and I always forget everything so it needs to be something that is relatively easy to jump onto. I do not know what it entails to make it compatible with free pascal.

[Thyandyr]

In reference to freeing all the TChunks in TDictionary<string, TChunk>, how do I do it?

I don't see a way to just do


for I = 0 to count-1 do
TDict.Items[0].Free

as it is not sorted or indexable in that way

Do I need to extract all the keys and then iterate with that list? That hardly seems like a smart way to do it.

OK this works


for Chunk in TDict.Values do Chunk.Free

[Thyandyr]

SO I'm adding lot of single values into one VBO. Things work OK until somewhere slightly above

Vertices:467544
Indices :100188
Total :567732

some (full) chunks just seem to be missing from where it should be on screen.

So I draw 23 chunks size 11x11x11 and it all shows great, but 24 chunks and some start to disappear. If I draw 30 chunks it looks like swiss cheese!

[Akira13]

In reference to freeing all the TChunks in TDictionary<string, TChunk>, how do I do it?

I don't see a way to just do


for I = 0 to count-1 do
TDict.Items[0].Free

as it is not sorted or indexable in that way

Do I need to extract all the keys and then iterate with that list? That hardly seems like a smart way to do it.

OK this works


for Chunk in TDict.Values do Chunk.Free


Uhhh... why aren't you just using a TObjectDictionary with "doOwnsValues" set in the ownerships parameter in the constructor? That's what they're there for. That way, it works exactly like a TObjectList that has OwnsObjects set to true, and you don't need to do anything at all in your destructor as all the dictionary values will be freed automatically.

As far as your VBO problem, what kind of GPU are you using?

[Thyandyr]

I have several dictionaries that juggle values between them, so I want to manage the lifetime of the contents myself. But when closing the program I rather empty all the dictionaries. I had a code looping the keys and then getting the values and then freeing the values. that resulted in mem leaks. But I found I can just iterate the values directly and free them. I just haven't used dictionary before. Anyway it is solved.

For graphics card I have Nvidia GFX 860M or something like that. Laptop. I didn't see any notes about maximum size in the OpenGL documentation. Will continue on this issue tonight.

[SilverWarior]

I'm not sure exactly how TDictionary works but in general when you are removing items from Lists it is always better to go from last item to first instead of always removing first item in a loop. Why?
Because when you remove first item from the list all other items needs to be shifted in order to fill the hole that you caused by removing that one item. This results in lots of unneeded data moving in the list internal array.

[Thyandyr]

I'm not sure exactly how TDictionary works but in general when you are removing items from Lists it is always better to go from last item to first instead of always removing first item in a loop. Why?
Because when you remove first item from the list all other items needs to be shifted in order to fill the hole that you caused by removing that one item. This results in lots of unneeded data moving in the list internal array.

I agree and would have done it 'from count-1 downto 0' but that is not possible with TDictionary. Anyway freeing the Chunks is only for the program shutdown so performance is not that important anyway.

Also I think if I do

for Chunk in TDict.Values do Chunk.Free

it still leaves the keys in, so the size of TDict size is not updated. Might be the size is not updated even if I removed the key as well, since there is some method to it called TrimExcess

Didn't work much today.

Fixed one bug, but made no progress on the case of too large VBO size causing full chunks disappear.

Line of 11x11 size chunks along X axis. If I decrease the count by few they all fill a nice solid line as supposed to. Too many like here, and something gets corrupted. Note the closest artifact, the whole chunk is not missing, only half of it (It is hollow since none of the cubes that should not be visible are not drawn)

1491

[laggyluk]

Why keeping all this in single vbo?
btw occlusion culling will be possible with this approach?

[Thyandyr]

Why keeping all this in single vbo?
btw occlusion culling will be possible with this approach?

I'm learning by doing and just wanted to see what happens and how much can push to one VBO. I will only remember things that I learn if I have first hand experience of the 'why' of things. Seeing something work or fail is essential to me.

Sure occlusion calculations will be possible, currently I only draw voxel surfaces that have solid-gas interface (assuming atmosphere... should be called transparent-opaque interface I guess. Actually you draw glass and water too, so should be called change-of-light-refraction interface heh heh), plus chunk edges. Occlusion on larger scale will be necessary once I manage to draw more stuff on screen. I have camera and chunk locations (and voxels' too). I don't know how it's normally done but I thought do to the large scale occlusion per-chunk and not per voxel.

[Thyandyr]

1492

Oh yeah! Long and sturdy like my d***

Problem was I had indices data type as UInt16

I can now get back to chunk manager. And after that multiple VBOs. And then compacting the data; I don't need full single for texture type or texture coordinates and so forth. Then culling.... and...

[Thyandyr]

I have elementary chunk management. It is slow but it works. It is fun to fly (backwards) and to see chunks get deleted in the distance from where you came and new ones to appear under you.

[Thyandyr]

I think next I will work on moving the code that generates the vertex and indices arrays, from TChunk to TChunkManager. The latter has easy access to surrounding chunks so I can easier do some culling when drawing at the edge of the chunk. I thought at the same time I would also optimize the attribute data. https://www.khronos.org/opengl/wiki/Vertex_Specification_Best_Practices#Attribute_size s Not sure how to 'cheat' and insert four bytes instead of single, and how get 'use' them as bytes instead os a single in the shaders.




Also, is there a better way to get undefined entry from TDictionary? Feels kind-a odd, that part getting a key/id from ListLoad


procedure TChunkManager.ListLoadChunksProcess(const Steps: Cardinal);
var
I: Integer;
aID: String;
C: TChunk;
begin
for I := 1 to Steps do
begin
aID := '';


if ListLoad.Count > 0 then // If we have entries in the ListLoad
for aID in ListLoad.Keys do // Get one of them
break;


if aID <> '' then // If we do have Chunk to load
begin
ListLoad.Remove(aID); // Remove the Chunk ID from ListLoad
C := ChunkLoadCreate(aID); // Load Chunk from disk or create it
ListLoaded.Add(C.ID, C); // Add to list of loaded chunks
end
else // Break out, nothing to process
break;
end;
end;

[Thyandyr]

Seems TDictionary frees memory as item count goes down.

[Akira13]

Seems TDictionary frees memory as item count goes down.

lol how else would you expect any kind of storage class ever to possibly work?

Also as far as the chunk post you made above this one... The kind of use case you're outlining there is far better suited to a TStack than a TDictionary. Not that you have to use one or the other... you could for example have some kind of global TDictionary to manage the overall voxel chunks by string ID, and a local TStack instantiated within some kind of renderer class that references the same values as the TDictionary. (Assuming that the chunks are classes and can be directly assigned without creating copies... if they're records you'd obviously need to make the TStack's specialization a pointer type instead.)

[Thyandyr]

TStack would be better for those lists that do not need to search for specific content, which is most of the lists in the Chunk manager. I think I'll update it (as you see I have no pre-knowledge on anything). Actually I need it to be TDictionary. But anyway, what is the difference between TStack.Pop and TStack.Extract? Docs say both remove and return.

TDictionary has method called TrimExcess made me think maybe it does not release the used memory automatically. Which would be nice in my case. Had it been so I could had the benefits of both the 'static' (or increase on demand) size and arbitrary indexes.

The load, unload etc lists will pump up and down in size constantly. Wouldn't it be beneficial if they grew in size but never shrank even when empty, or took the theoretical maximum size and kept at that.

[Thyandyr]

Achieved:

TChunkManager per frame update (TChunks load, create, unload etc. few items per frame)
Each TChunk in separate VBO
Frustrum culling on Camera change (TChunk.InFrustrum = boolean). VBO stays in but does not get drawn if not in frustrum.
VBO freed on TChunk unload


Managed to introduce annoying stutter effect on fast camera movements.

[Thyandyr]

I'm back from few weeks holiday. I have no idea what I am doing but somehow I manage to make some progress.

Stutter is fixed, frustrum culling is working much better too (only on one plane, need to add 4 more.

[Thyandyr]

...frustrum culling is working much better too

Now with 4 planes and distance is handled by other means. Nice job me!

[Thyandyr]

https://youtu.be/9FWw1iZrERc

[SilverWarior]

Nice progress there.

[Thyandyr]

Changed from keeping track of neighbouring chunks and accessing their data, to n+2^3 array that just keeps extra layer (redundant) of data around the real data. for chunk edge drawing and occlusion. Works much better.


https://youtu.be/M8QN7E5QHOQ

[Thyandyr]

Will put this eventually to GitHub once it is decent enough and I have learned to use Github. I did start a project and made readme, but no code yet. Link in initial post.

[Ñuño Martínez]

I want to see that sources... ;)

[Thyandyr]

I want to see that sources... ;)

See initial post for link to github. Not all of the source is nice yet. Still figuring out things myself.

[Thyandyr]

I worked a bit tonight and I did get the initial save/load (binary) chunks to disk to work.

Did have some challenges from the fact that the world size is not artifically limited.

The maximum world size is Int64 chunks per axis.

Chunk size then is freely selectable, but with a limit of 2 642 243^3 (but in practice something like 16^3 to 32^3).

[Thyandyr]

Benchmarks

Chunks cubed: 21
Chunk count: 9261
Chunk blocks cubed: 16
Blocks array(+2) size: 5832

Initialization etc: 190 ms

Occupy ListLoad: 4 ms
Per item: 0.000431918799265738 ms

Creating chunks, please wait
Create chunks one at a time: 10989 ms
Chunks processed: 9261
Per item: 1.1865889212828 ms
Chunks with solids created: 544
Per created: 20.2003676470588 ms

Save chunks disk bulk: 10 ms
Items: 544
Per item: 0.0183823529411765 ms

Save chunks file locations: 11 ms

Create vertices, Indices: 965 ms
Items: 544
Per item: 1.77389705882353 ms
Slowest item: 2 ms

Render (all loaded): 49 ms
Items: 544
Per item: 0.0900735294117647 ms

Render (all loaded): 1 ms
Items: 544
Per item: 0.00183823529411765 ms

Updated frustrum: 0 ms

Render (in frustrum): 1 ms
Items: 185
Per item: 0.00540540540540541 ms

[Chebmaster]

Is any sort of LOD planned?
I mean, those chunks missing their meshes: it would look nicer if instead of holes you rendered solid "plugs" as high as the highest block of the chunk missing its mesh yet. Then retract them into the ground smoothly after the mesh is generated.
Also, there could be other ways: (Planet Explorers had this implemented): LODs like, in the distance meshes are built by joining 8 adjacent blocks into one -- this way you can have a 4-chunk LOD mesh at the price of one.

[Thyandyr]

Yeah this is one of the issues I will address next.

[Ñuño Martínez]

(...) Not all of the source is nice yet. (...)
I did a fast read and it looks much nicer than most other open sourced projects out there. It is even possible to figure the purpose of each unit!

Keep it.

[Thyandyr]

Is any sort of LOD planned?
I mean, those chunks missing their meshes: it would look nicer if instead of holes you rendered solid "plugs" as high as the highest block of the chunk missing its mesh yet. Then retract them into the ground smoothly after the mesh is generated.
Also, there could be other ways: (Planet Explorers had this implemented): LODs like, in the distance meshes are built by joining 8 adjacent blocks into one -- this way you can have a 4-chunk LOD mesh at the price of one.


I did a fast read and it looks much nicer than most other open sourced projects out there. It is even possible to figure the purpose of each unit!

Keep it.

Wow nice to hear. Especially since I consider the code quite messy. I am learning as I go, so some of it is at 'unclean' state. But as I get more comfortable with these things, the codebase also gets cleaner.

I have been having busy times during holidays and now at work... but good sign is that I am looking forward to when I get to work on the engine.

I will next concentrate on world class, and with that it allows smarter order of creating the chunks. Recursive chunk creation following the earth surface, so we can skip all chunks in the sky and underground, and thus be much faster and avoid the 'holes' that Chebmaster pointed out, and create underground in the background.

Then perhaps few more terrain types to make it a bit less unpleasant to look at. After that I guess would be the fun part of adding a bit variation to the landscape creation.

[Thyandyr]

Damn Rimworld. Had to uninstall it. Maybe progress ahad?