Very large terrain : 4096x4096x10m, precision problem on distant locations?

[njord]

Hello all guys!

 

I have created the largest terrain possible in LE which is 4096x4096x10m (in fact, by modifying sbx with notepad, u can make it even larger).

 

On distant locations (relative to origin), I see artifacts on shadow maps and UV assignments as well as geometry degredation which I believe all due to floating point precision loss.

 

Here is the related video, although you can check it out yourself by creating a terrain in Editor. Parts of terrain at locations (0,0,0), (7500,0,7500), (15000,0,15000), (20480,0,20480). :

 

 

ftp://81.214.87.33/

user : katron

pass : katron2012

 

file : "LE 4096x10m map precision problem - DIVX.avi"

 

What are your comments on this issue ?

(One possible dirty solution : I can move the terrain and objects on it instead of moving the cam )

 

PS: I encountered precision problem on my previous job, where we were implementing a google earth clone 3D app. We were visualising the whole earth along with a tiny house or road sign in the same scene. We solved the issue by utilizing "camera relative rendering" & "using double precision math on vector calculations taking place on cpu" & "enabling consistent floating point mode on direct3D".

[Pixel Perfect]

Flexman has probably done the most work on this and is best able to advise you. Hopefully he'll see this and pass some knowledge on

[Flexman]

You can't move the terrain object.

 

However if you drop LE terrain and use a mesh (or tiles) attached to pivots (and parent zone contents to that tile) then you can approximate camera relative rendering.

 

There's no easy or clean solution to this with LE. In Unity it's pretty much the same deal except you have control over terrain tile positioning and async loading to make it all seamless.

 

Changing the scale doesn't help either, it just offsets the problem.

 

Short of having a build of LE that uses double precision (not easy as a lot of shaders which use world co-ordinates will be effected) you have to do a huge amount of work and design around it. And It's not something I wanted to do so I adopted a half-measure with the intention to fix it fully in version 2 of our project.

 

Best you can do is mitigate problems.

• Shorten the camera near and far distance as much as you can
  
• Modify shaders to add more depth bias based on distance from camera
  
• User the material overlay flag where appropriate
  
• Render the scene in layers and blend them
  

 

And how you tackles those depends on your project and the content. You can't (sadly) create the largest world you can in the editor and expect shadows to be stable once you're past 20,000 units from the world origin.

 

 

The techniques you mention are solid (camera relative rendering, double precision) , but can't be easily applied to LE. I understand LE3.0 has large world rendering on the roadmap but is down the "to do" list.

 

 

Our current project is vehicle based (a helicopter) which has the advantage in that you're typically inside an object that obstructs close up views of the main 3D world. And often these imperfections are only visible in outside 3rd person views. When the camera is in such a mode the camera settings are adjusted slightly, also we had to build the models in a way that minimised jitter (objects passing through each other). On the plus side it did make the models more efficient.

 

For a world exploration type game I think one of the better approaches is to forget LE terrain and use imported meshes but that comes at the expense of the vegetation system and you can't use the editor to build your worlds in an easy fashion.

 

GROME is an expensive tool but is built for tiling,painting and exporting individual zones as meshes (now with a direct to Unity export feature). It's well suited to a tile based world.

 

Feel free to ask questions, I'll keep monitoring this thread.

[njord]

Thank you very much for your long explanation Flexman. Indeed we are trying to create a terrain for a helicopter simulation too It should be 30x30 km so the furthest location from origin will be 15000,15000 (plus the chopper can elevate into sky like 5000 or 10000 meters). By turning off terrain shadows and ssao, also by adjusting camera planes to 0.35,7500 now we have a rather acceptable level of visual artifacts & bugs, spiced with proper DOF and fog.

 

However, can you explain more about your advices below ?

 

• Modify shaders to add more depth bias based on distance from camera
  
• User the material overlay flag where appropriate
  
• Render the scene in layers and blend them
  

 

We will also try building simple models without intersecting polygons as you suggest. I will post how much it will help.

 

PS: our terrain heightmap is 512x512x100m now (I modified 512x512x10m sbx to have a instead 100m resolution grid). This causes some small differences between the phy and gmf of the terrain. However since we use LE for only visualisation, it seems ok for us.

[njord]

Hello again! We are creating a helicopter simulation as I told, so by modifying the near range, disabling SSAO and terrain shadows I have managed to reduce the precision loss problems to an acceptable level on a 30kmx30km terrain (1024x1024 resolution, 32m each tile length). Also by increasing the slope on the water-land intersection geometry, my waterplane flickering is reduced too.

 

Thanks to Flexman

 

However, as the project progressed, we have modelled a detailed helicopter, and when the pilot (camera) sits inside it, all the tiny buttons and gauges on the panel JITTER on far locations ! So I need further optimizations to further mitigate this problem.

 

A dirty solution : I scaled up ONLY the helicopter by 5-10 times. From the point of view of the camera, everything seems properly scaled, but in fact, I am inside a giant helicopter and all the buttons and gauges are like 5 m away from me. Since they are both bigger and further, no problem in visual scale, unless you take a look from outside It is still not a problem, if i look from outside, i can scale back to normal size.

 

However, when i land, or collide with an object, - since my collision geo is still small and it should stay that way, objects penetrate through the helicopter. To work around this, i need a special material that will render the helicopter over/after all other geometry. First of all , I want to ask you, how to make a "depthtest off" and "render last" material. As i have posted on another topic, I have managed to come up with a material like below :

 

texture0="abstract::blackwhite.dds"
color=1.0,1.0,1.0,1.0
depthmask=1
colormask=1
depthtest=0
castshadows=0
invisible=0
specular=0.5
gloss=0.5
bumpscale=0.5
zsort=1
cullface=1
overlay=0
blend=0
shader="abstract::mesh_diffuse.vert","abstract::mesh_diffuse.frag"

 

depthtest=0 and zsort=1 : this makes the entity to be visible even when it is occluded by another object, great! However the entity is slightly transparent! If I make zsort=0, than entity is rendered last, even when it is actually in front of me.

 

Q1. What is the proper way of "no depthtest & render last" == make something properly "always visible"?

 

--------------------------

 

This is my quick and dirty solution of course, so I want to learn more about what you have gone through Flexman, if you don't mind

 

Can you or anyone answer these questions also ?

 

Q2.Converting LE source code into double is not easy right ? Also you couldn't "fully" do it as i understand, did it help a bit anyway? Do you recommend me to follow the same path by obtaining the source code and modifying?

 

Q3."Modify shaders to add more depth bias based on distance from camera" == what do you mean by this, which shaders specifically, can u post a very simple example or give more insight to get me started ?

 

Q4."User the material overlay flag where appropriate" it is used by decals as far as i know, to translate the decal surface a little bit closer to the camera. How can it help me ? Shall I write my "gauge, button, joystick" materials this way ? But, my problem with the helicopter cockpit surfaces is not flickering, it is the geometry degredation mostly. Vertices just move, its like they are not made of a solid matter

 

Q5."Render the scene in layers and blend them" how can i do that, what is the reasoning behind it/how will it help ?

 

Thank you very much for reading all this and noob questions if any

Cheers!

 

visuals :

(On the second image, black and white cube is between terrain and helicopter. But as you see it is always fully visible even occluded. However, it is transparent as well. I dont want it to be.)

[wh1sp3r]

I am not sure about what i say now, BUT

 

you should use near camera more than 1.0, why ?

 

Let's say, our depth map is 8 bit, that's 256 levels.

We set near 1 and far 10, that's 10 levels.

 

10/256 = 0.039 per bit.

 

when you set near to 0.1 and far 10, that's 100 levels

 

100/256 = 0.39 per bit .. worse precision.

 

Shadows will be more squared. There will be bigger steps in depth map.

[njord]

Are you suggesting I should make it 1.0 m mate ? But inside the helicopter, i barely have 1m space for the pilot to sit. So if i make the near plane a large value like that, I wont be able to see the gauges buttons and joysticks in front of me, they will be clipped. I can scale the model up, there comes my Q1 above

 

By the way :

Right now my camera near clip is 0.35 and far clip is 500-1000-10000 - i tried different values. 0.35 is the highest value i can reach when DAYNIGHT script skybox is still visually bug free. If you change near plane to 0.40 lets' say, skybox cubic planes start to disappear.

 

In other words, decreasing far clip plane did not help with jitter much i guess, but increasing near plane from 0.1ish values to 0.35 did help.

[wh1sp3r]

i don't know, how other developers do it, but there is a trick. Let's say, you can make cockpit 2 times bigger but with depth write disabled.

If i remember, Josh is using it too for handswithweapon model in LE. Not sure, how it will look like with cockpit, lol.

[njord]

I have looked at the materials of that gmf mate (LESDK 2.5\Models\Weapons\smg). There is nothing special in there, like disabling depth writing or testing.

[Flexman]

This is the topic for a small book. All these experiments and more were tested in Patrick Cozzi's book "3D Engine Design for Virtual Globes", the books website has small examples for experimenting with depth precision, camera distances, scale. Just to prove the point you can't have your cake and eat it. I highly recommend this book for anyone seriously interested in the problems of terrain and scale.

 

If you have a cockpit in world space that moves away from the origin (this applies to any 3D engine) you will get vertex jitter. That's it. Obtaining a source code license to try and rebuild the engine for 64bit math is not practical. Both in terms of license cost and ability required - I'm sure Josh uses some arcane matrix shortcuts and it won't be enough simply substituting one type to another.

 

So that's the negative. Now the positive.

 

Your solution would be to render the cockpit it in it's own world space. The one in combat helo never leaves the world origin, sames goes for guns and knives. This is layering. It's no different from pasting old school 2D cockpits over the screen, only difference is the 2D image is a 3D model with lighting. Getting everything to match up, exterior models with interior models, lighting etc.is the hard work.

 

That is the solution for any kind of inventory or vehicle cockpit. It doesn't work so well for exteriors but could be done in a similar fashion.

 

The panacea is to create a tiled/paged environment so the viewer never strays too far from the origin. Think old scrolling tiled games where you can explore maps bigger than the screen. You have a slightly easier time with this in UnityPro, something I'll be covering in a future book. I hope to have a working Unity example ready in time for publication but it's not a trivial thing. You can't apply transformations to Leadwerks Terrain and you can't have more than one Terrain object so you need to build your own system from scratch. But you have to ask yourself if the game justifies it. Really the easiest thing is just make a smaller game and all the problems go away. Battlefield 3 uses art tricks to make you think you're going faster and further than you are but you're still moving around very small areas, even in jet fighters. That's art trickery and a whole topic of the illusion of speed.

 

Here's some more practical help and I don't want to spend more time on this as it really is a never-ending topic.

 

One of the things I did to fudge depth writing on SOME objects to create a second overlay material that was only applied to problem objects (such as rockets inside pods). My MESH.FRAG is far from standard but after the BLOOM conditional I have the following kludge which is applied to an exterior container (eg. rocket pods).

 

// I need the depth offset proportional to the camera distance from the vertex
#ifdef LW_OVERLAY2
 float dist = length(modelvertex.xyz - cameraposition ) / 512.0;
 float z = DepthToZPosition(gl_FragCoord.z);
 z = z - dist;
 gl_FragDepth = ZPositionToDepth( z );
#endif

[Pancakes]

Wow Flexman you are very impressive!

[Guest Red Ocktober]

However, as the project progressed, we have modelled a detailed helicopter, and when the pilot (camera) sits inside it, all the tiny buttons and gauges on the panel JITTER on far locations

 

i ran into this same issue in another engine... the farther away i got from the "world center", the more "jitter" i got when the view was inside of the moving vehicle...

 

the (less than optimal) solution i came up with was to offset all scenery objects once the vehicle went beyond a certain distance... this worked (with a hardly noticeable flicker when everything moved)... basically, the world moved past the world center as the view stayed pretty much at the world center, or within 200 meters...

 

maintaining the logic becomes a chore though... an object fired from a moving vehicle immediately prior to offset change sometimes could present a problem...

 

--Mike

[Flexman]

Offsetting the "world" as you describe Red is a valid technique, it's one way tiled worlds work. What you're wanting to do is add an extra co-ordinate space to everything.

• World co-ordinates (e.g lat/lon)
  
• Tile co-ordinates (e.g. tile[12][13])
  
• camera space co-ordinates (e.g a Vec3() position )
  

And everything is tied up in a manager to deal with it. Some commercial games exhibit small glitches when crossing these tile co-ordinate boundaries and have to adjust the world. Spatial audio glitches are a clue as the "listener" can lag a little. Sometimes you get rubber-banding in an external view for no apparent reason.

 

Parent everything within tile space to the tile and you move the tile - thus moving everything with it. You'll be able to do this in my procedural dungeon. In the final part I plan to show how you can create infinite dungeons by exploiting "cells" as a virtual co-ordinate system. Currently each wall is parented to the room mesh, move the room and all the lights, props, wall, go with it. Each room lives in a virtual "cell" space. Replace room with terrain tiles and it's similar, if less sophisticated.

[Guest Red Ocktober]

thx Flex, for the info...

 

--Mike