Smoothening tile edges

[User137]

I'm working on a tile-based game at the moment, and could use some hints on making it look better. Rendering is using GLSL pixel shader for these 6-sided hexagon tiles. I let there be 4 texturemaps for the terrain, so i can blend them without too much performance loss. Basically fragment shader does:

if (color.w > 0.5) {
gl_FragColor = color * (
ts.x * texture2D(tex0, texCoord) +
ts.y * texture2D(tex1, texCoord) +
ts.z * texture2D(tex2, texCoord) +
ts.w * texture2D(tex3, texCoord) );
} else {
gl_FragColor = vec4(0.0);
}
So i can "break" 1 edge that is exposed to air, to make it look flatter as seen in the attached screenshot. I render 2 dirt tile and 1 stone tile on right side of them. The top-left corner use alpha 0.2 and bottom mid-left corners are set alpha -0.2, that's variety i can let there be... But it didn't work as well as i hoped. Instead of having 1 sharp edge, there now is 2 smaller ones. Ok that looks better than that 1 big sharp edge, when zoomed further out, but is there anything i can do to make it rounder/flatter? I could propably modify color.w component to start alpha 0 from that edge at 0.5, scale it differently. But then it wouldn't be solid edge anymore.

edit: Scaling was easy to do so attached comparison what it would look like

vec4 col = vec4(color.xyz, (color.w-0.5)*2.0);

[phibermon]

Don't forget that in order to draw the hexgons, they're deconstructed into triangles and you're essentially setting the alpha level on a shared vertex between two triangles which causes exactly what you describe. if you render the scene in wireframe that should make it more obvious.


You don't have to shade your fragments in the classical sence, IE you can control the behaviour with properties that don't come along with the vertices.

for example, if you wanted to fade the edge of a hex out or blend it etc following the edge, you'd instead control the blending as a function of the distance to that line.

Not the only solution but in a really basic way :

Pass the two points that make up a hex edge as uniforms into your shader.


In your pixel shader you can then do some simple maths to work out the fragments distance to the line and then shade it accordingly, You can then get any result you like.


You could also tesselate the hexes for more than 6 triangles, say spitting those 6 into 3 each, this will give you a greater vertex density bringing you closer to the look of per-pixel but signifigantly quicker (note any situation where a shader's main bottleneck is the fragment shader, you should look to see if you can move certain things to the vertex/evaluation/control shaders and then come closer to the per pixel result by evenly tesselating your source geometry. Check out 'Opacity Maps', specifically a similar optimization made with opacity mapped particle systems)

You could also render all hexes on the outside as textured circles of radi equal to that of the hex, with half of the circle fading opacity to zero. the circles would then be aligned (by querying the surrounding blank hexes) so that the faded side points out, this would give a feathered but much smoother looking edge when 'zoomed' out by larger factors.

You could also 'mask' by rendering over the edges of the hexes, like a 'fog of war' on lots of rts titles. Then you don't have to worry about smoothing the hexes, instead worry about drawing a smooth border. This is actually what I'd do because you could quickly identify long rows of hexes in the map by walking around it's edge, then you could draw one big overlay/mask over that group, giving an even smooth all the way down etc

Alternativley you could render the map as a low resolution, filtered, alpha texture (where 0 is no hex etc) Then as you render your actual map, you translate the map coord to this texture and then sample it, using the result as the alpha value in your hex render. the filtering from a low resolution texture will greatly smooth out the edge. If you do this you should scale the co-ordinates slightly so the mask is referenced slightly smaller than the actual hexes, this would be like 'feathering' the alpha edge, encroaching the blend further into the visible hexes (if you get me)


Edit : For anybody reading unfamiliar with shaders, here's me practicing some tutorial writing again - Fragment Shader = Pixel Shader. Control and Evaluation shaders are new additons in GL4.0, seperate from Geometry shaders you may of heard of. Control+Evaluation lets you take arbitary source vertices (called a 'patch' in this context) apply the tesselation hardware and get varied (yet still subject to the hardwired tesselation routines) output containing a higher number of vertices. You can control all per vertex data that gets pumped in and effect new, greater number of vertices that pop out of the tesselation. Geometry shaders are vaugely similar in that you can generate or bypass output via code but they are a lot more relaxed and varied in what they can do. They can also be used to tesselate although its very slow, hence the tesselation control/evaluation shaders which are specifically intended for the task of subdividing surfaces. An example of the difference is that a geometry shader could output cubes if you input points, or output 3D fishes if you input lines where as the tesselation shader couldn't. It could however take a simple cube and output a smoothed cube (smoothed edges or offset the new points into a sphere). So can the Geometry stage with some trickery (but not as quick),

[User137]

Isn't it out of options to have unique uniforms passed per tile? I am rendering hundreds of tiles on single call. Tessellation is nice idea, but my knowledge on the subject is non existent ::) It might require some higher level graphics card, or high OpenGL version. Also doubling the triangle count might double the rendering time, depending on implementation, graphics card etc. Would be nice to have high compatibility even with the poorest laptops with integrated cards.

I'm not sure if i can use opacity map. Firstly it would have to be massive texture for all tiles on screen at once, and secondly it needs to be changeable at realtime. Current terrain textures are size 256x256 that repeats, and the hexagon is made of 6 triangles connected to the middle vertex. I would guess the resolution of opacity map would have to be at least 6x6 pixels per tile... Well, propably even 512x512 would be enough for that, although the realtime requirement would not be met. 6000x6000 texture or so for whole tilemap would be out of question, but i still haven't set any world sizes in stone.

I think i figured a nice alternative, that i will simply not render those outside triangles, and doing this triangle elimination on the CPU. I could remove 3 or 4 triangles, 3 on straight, 4 on corner. This will let me have really flat surfaces on all 6 sides, and i could make even big circle shapes with no holes in the edges. It will make interesting math for collision with these "half-tiles" though. I can also get rid of all alpha-related special treatment in fragment shader, making rendering even faster.

Oh, there is yet another option... I could add 1 triangle to fill the gap between 2 hexagons. Brilliant...

[User137]

And i must say that over 700 views in less than 24 hours is unexpected :o I don't know if you even realize how much that is on any forum scale. Stop lurking and start posting, now! There are no bad replies.

[Rodrigo Robles]

And i must say that over 700 views in less than 24 hours is unexpected :o I don't know if you even realize how much that is on any forum scale. Stop lurking and start posting, now! There are no bad replies.
The subject is very interesting but, most people, like me, doesn't have sufficient knowledge about shaders to help.

[pstudio]

And i must say that over 700 views in less than 24 hours is unexpected :o I don't know if you even realize how much that is on any forum scale. Stop lurking and start posting, now! There are no bad replies.
I'm not even sure what effect you are aiming for :?
Do you want all the edges of the whole map to be smooth or every tile edge? Or is it something else?

[User137]

I'm making a circle shaped world. I'm not far enough to show the idea with game yet, but this should give you the idea:
http://upload.wikimedia.org/wikipedia/commons/thumb/9/94/MagicHexagon-Order7.svg/500px-MagicHexagon-Order7.svg.png
Center of gravity is in the middle of the planet, and player feet are always towards it. Imagine walking on that "spiky" surface, there are these "holes" on all 6 sides of it. And i believe this is solved by adding triangles outside the hexagons to fill the gaps.

And not just the planet surface, but some caves inside. If not natural caves, then digged ones. Idea similar to
http://i146.photobucket.com/albums/r280/KrunkSplein/Screenshots/terraria_fullscreen.jpg

Hmm.. know what, i would get much easier by doing this with square tiles. I could add triangles between edges with it aswell. And it might even look much better, not to mention easier math around it.

1 tile from 4 triangles and i could also leave 2 triangles hidden depending on corner, instead of adding new ones between tiles. This would also increase the rendering performance. I have to consider the options...
http://i.stack.imgur.com/ssaaS.png

[Carver413]

well I suppose that could work but your still going to end up with some ugly spots as you can't make a perfect circle with any shape you chose. you could apply a granular aproach to it and acheve a much rounder cut off.
measure the distance of each point from the center of the world to calulate the cutoff point. you could then apply some sort of linear interpolation to give it a more controlled fade.
1156

[User137]

I made a simple planet and tunnel generator, so here is few screenshots.
http://i41.tinypic.com/35knxp3.png
http://i41.tinypic.com/2ez7cll.png
Tile-radius of this planet is 240. Overall 600x600 tiles are reserved for space building room. Went down to 12 fps rendering all at once ;D

[phibermon]

If the framerate is proportional (linear) to the number of tiles you render then that suggests that you are not using any spatial partitioning schemes to cull rendering to the visible hexagons?

If you are you may want make sure it's functioning correctly.

Can I ask what version of GL you code for, your CPU and GPU?

12fps for a a screen of 2D triangles at the perceived density from your screenshots is well below what I'd expect from even mobile phone class hardware.

If you're not using spatial partitioning, a quad tree should be perfectly good enough.

The basic principal is to represent the bounds of the world as a quad, then divide that into four smaller quads, and add to each of these a list of hexegons in that quad. then keep on splitting and adding in this fashion until you reach some pre-defined minimum size (Say the size of one hex etc).

Then when you render a frame, you test your screen quad against the top 'node' in the tree and determine which quadrant it's in, then you can immediatly bypass 3/4 of the hexes from render. Then test against the 4 child nodes of that quad etc etc

Because of the squared nature of dividing the space by 4 each time, you reach the smallest nodes (leaf nodes) in just a small handful of checks. (obviously you test if the screen is inside a node OR overlapping the node, you may be itterating into more than one child node at a time, but it will beat brute force quite early on in a graph)

(just for those reading if you already know this)

EDIT : Apologies! I just saw you screenshot, you're zoomed right out so obviously spatial partitioning won't help there ;)

you could however render the map at a certain resoution to a texture and then switch to rendering that texture at a certain level of zoom. if you required the display to still show realtime changes on the map, you could stagger the updating of this renderbuffer texture over a number of frames (so as not to impact framerate). So you'd have good framerates when zoomed out in terms of the interface and any other things you wish to render at the cost of a lower percieved framerate for the rendered map.

[SilverWarior]

If you are designing your game to use hexagon tiles, then it might not be bad to check how Widelands (Settlers 2 clone) map is gnerated. While Widelands use triangles (vertex approach) for map representation you can always create hexagon tiles by combining 6 triangles.

On link below you can see a bit more information about how it is done, including the way for calculating regions. I hoe it would come in handy to you:
http://widelands.svn.sourceforge.net/viewvc/widelands/trunk/doc/geometry/index.xhtml

[User137]

I could let you test this version (at least while i keep it online): https://docs.google.com/file/d/0B7FII3MhcAHJNVQtaEY0VTB4emc/edit?usp=sharing
Actually i'd hope it to be tested just to verify there's no major bugs in nxPascal. The controls don't allow zooming, just drag-moving with mouse. F7 toggles for windowed fullscreen mode.

Full earth screenshot was special case with no limiter. Smaller game view is rendering 41x41 area, which is propably much bigger than needed at the moment anyway. I should be fine without splitting optimizations. I get max 60+ fps with it, at 0-1% cpu use. I need to look closer to zooming levels when i get to playable stuff. Also 1 reason the FPS went slower on full planet view, was especially after i finalized the texture blending. It is calculating texture opacity for each vertex per frame, by adding and normalizing texture opacities of self+3 neighbouring tiles.

And yes, i scrapped the hexagon idea and used squares and triangles with this. Because with triangles there are actually 8 different "sides" in use, which is smoother than hexagons 6. It calculates the corner cutting in planet generation phase at least, not realtime, and puts the angle in 1 shortint variable. Also there is no static vertex-array, but the triangles are queued same way as my renderer class does. There's hard cap of 1000 vertices and 1000 vertex indices for glDrawElements(). That is another thing new, so i don't have to do the heavy texture-opacity calculation so many times on CPU. So pretty much everything is reconstructed per frame.

If someone was interested in joining the game project, you can send me a PM ;) I'm working on it with 1 other person at the moment, with growing documentation and stuff in google docs. I don't want to reveal the game more in public yet, and i don't have any opinion yet on wether it will be free/opensource or anything else.

[phibermon]

I'd love to work with people on projects but in terms of games there's only one game I have my heart set on and I code my engine with it in mind. It requires the rendering of at least one planet (and possibly a moon) from space, all the way down to walking on the surface. It involves time travel, a deep mystery, a planet wide search, a lovable AI companion and a shocking twist. I have my story and I know how it'll play just depends on how much of my ideas I can get into it on my own :)