Editor (Foliage): Buffer resize error

[klepto2]

When using the vegetation system with large terrains it happens very fast that this error occurs:

I can reproduce this with these simple steps:

1. Create a terrain with at least 4096 * 4096
2. Add a foliage layer (u can use the crate.gltf)
3. Fill the terrain with that layer (this should work)
4. Set the density to 1.0 
5. The editor works some time then the error message will appear

It looks like a system memory problem, not a gpu problem.

From time to time i get this error as well, when using eg: to many probes but with the vegetation system it is the safest way to reproduce it.

 

[Josh]

This means that malloc or realloc are returning NULL. How much RAM do you have?

[Josh]

Your signature says 16 GB is that correct? This is caused by either not enough memory, or memory fragmentation. There's probably not much I can do about it.

[klepto2]

Ok, yes it is only 16gb, but it also occurs on another machine where i have 32gb (just abit later).

The bigger problem, is that even maps loaded ingame will crash with this error and it can't be workaround.  This increaes the ram requirement to massive amounts for the engine and the published games. While i agree that 16 gb is not much, what will happen, if you later integrate streaming terrains or full planets? 

There must be a solution, maybe splitting the workload to slower chunks of memory? as it seems a ram problem, not agpu problem it should be solvable.

 

[Josh]

How often do you restart your computer? I have 16 GB RAM and have never experienced this.

[klepto2]

I can reproduce it with the above steps everytime after a fresh restart. Sometimes it needs a bigger terrain or more dense vegetation. 

[CJO Games]

I can recreate this - with the added following scenarios...

.paint any foliage to the maps edge and this will occur the editor will crash to desktop or the editor will turn blank white or blank black. AMD graphics card will report a driver error and reset to standard settings Nvidia will not report a driver error but the symptoms are the same.   

.fill the terrain with density set to any level and then attempt to change this level - the same crash will occur as above

I am using terrains of 2048x2048 with imported RAW maps from L3DT

 

System specs are: Windows 11 Intel i3 10105f 16 GB RAM AMD RX6600 8gb GPU  Resizeable BAR active in driver.

Windows 11 i3 10105 16gb RAM Nvidia GTX1650 4gb (No resizeable bar or equivalent)

 

Thea.zip

[Josh]

I have added the size of the requested memory allocation into the error message. I believe the biggest buffer a material layer on a 4096x4096 terrain will be 256 MB.

[Josh]

2 hours ago, CJO Games said:

paint any foliage to the maps edge and this will occur the editor will crash to desktop or the editor will turn blank white or blank black. AMD graphics card will report a driver error and reset to standard settings Nvidia will not report a driver error but the symptoms are the same.   

.fill the terrain with density set to any level and then attempt to change this level - the same crash will occur as above

I followed these instructions with no problems, using an AMD 6600.

[klepto2]

Details: 

Terrain-Size is 4k

used memory of the app is around 1.3 gb and there is more than 10 gb free mem availbale. the first fill with a density of 2.0 is working, lowering it to 1.0 leads to the state above. if i remember correctly it the allocation is 4gb not 256mb as you stated.

 

[Addition:]

It also seems to be dependent on how the terrain is layouted. In the above case it is just a fresh created flat terrain. With a heightmap applied and some height and slope settings the error comes earlier, sometimes already with the initial fill command. It also depends how many layers are already there. The number of variations seems to play no role.

[Josh]

Whoa, that is only 2 MB.

I don't think the problem is memory fragmentation.

[Josh]

The editor now includes some extra info in these errors so at least we can tell if it's malloc or realloc that are failing.

[CJO Games]

Will keep an eye out for the next time it occurs.

[klepto2]

with the same steps as i mentioned above. 

[klepto2]

Maybe a small hint, when changing the density, the virtual memory goes up to more than 32gb and then the app crashes with the error.  

these are some virtual memory stats i measured:

• Starting the Editor: virtual memory  ~ 1gb
• Creating the terrain (4k): virtual memory ~ 2.4gb
• Adding a layer with 3 variations: virtual memory ~7gb
• Decreasing the density to 1.0 : virtual memory stays at ~7gb
• Using Fill (no crash): virtual memory stays at ~7gb
• Increasing the density back to 2.0: virtual memory increases up to max available (here 32gb) and the app crashes.

 

[Josh]

7 hours ago, klepto2 said:

Maybe a small hint, when changing the density, the virtual memory goes up to more than 32gb and then the app crashes with the error.  

Yeah, that is an interesting detail... 

[Josh]

When you do this, are you holding down the spinner arrow so the density changes gradually many times? That's the only way I was able to raise the memory up much. It appears there could be a mem leak, and then when the density is set a lot of times, the effect is magnified.

[klepto2]

I played around with some other terrain sizes, and even with a 512*512 sized terrain and i get the same results when switching the density from 2.0 to 1.0 and back. 

I can do this either by the spinner or by directly entering the density in the textfield, so in theory it should be only triggered once.

With the 512 sized terrain, the virtual memory is always increased by 10gb as soon as I switch to "1.0" density. Switching back to a higher value doesn't lower the virtual memory, but then switching back to 1.0 it grows again by 10gb. Also loading a map ingame will lead to the same issue.

[klepto2]

ok, tested it with the latest build.

The good news:

• on smaller maps up to 1024 * 1024 i can't trigger the buffer resize error anymore.
• the virtual memory goes up to a crazy amount but after finishing the memory goes down again

The bad news:

• on larger maps the virtual memory still raises beyond the maximum and triggers the buffer resize error
• the error now appears a lot later but still appears
• when the error message appears, the memory goes back to normal but the app still crashes
  • thats far better than before as now the app doesn't "kill" the whole os anymore

[klepto2]

ok, maybe this helps to track the error down:

i first thought the error is related to the density, but made some small tests and i found the following things:

The reason for the error seems to be more complex than i thought: 

• Create a terrain of any size
  • to get the error it should be bigger than 512*512
  • otherwise the memory will raise but in handable sizes (7-10gb)
• Add a Layer:
  • No error and instant visualisaton (no memory usage as well)
    • change the density only to 1.0 or any value you like
    • Click Fill
      • Layer is displayed instant 
      • no mentionable increase in memory
    • Now you can clear the layer, change density and fill it again, still instant and no memory usage
  • With error and high memory usage Case: 1 
    • make the same steps as above, but instead of clearing the layer after the first steps 
      • change the density to a lower value
    • memory usage goes up to crazy amounts ( i have raised my virtual memory to 100gb for testing this)
    • resize error occurs.
  • With error and high memory usage Case: 2
    • This case is more like the "No Error" steps
    • this time we will change the maxslope to lets say 20 before clicking Fill
    • same result as in case 1

[CJO Games]

Loading a map from the editor has now slowed down significantly - looking at the windows basic task manger ram use is very high and the system as a whole has slowed down:

Interestingly with the client apparently running in the background the CPU use seems high too:

The next image is with the map loaded in and the editor 'at rest' with a much smaller level of RAM being used in this state:

 

 

In the following images I change the grass density from 2.0 to 1.0:

 

As reported everything jumps up significantly until the buffer resize error pops up:

 

 

Pressing ok and leaving the editor alone results in a crash to desktop with the client still running but the editor closed.

 

The map is the one attached above the painted textures are 1024 in size from the Ambient CG assets add on. The models (grass trees etc.) are imported from Leadwerks Vegetation/nature dlc.

 

The map size is 2048 x 2048 with a height setting of 20000. Its a L3DT professional map imported as a RAW16.

 

If this is repeated enough times the map will become corrupted i.e. it will no longer open in the editor.

 

 

Edited July 3 by CJO Games
Various typos corrected

[CJO Games]

For completeness, altering the grass density in the other direction from 2.0 to 3.0 and 3.0 to 4.0 whilst slow doesn't cause the above buffer resize error.

The other way around also works increasingly slowly until the density of 1.0 is selected and then the above error is recreated.

The good news is its no longer causing my graphics card driver to crash and isn't causing an overall OS crash either. 

[Josh]

Okay, if I create a 2048x2048 terrain, add one layer, then change the density from 2.0 to 1.0, then the memory usage skyrockets to something like 12 GB. That's definitely not right...

[Josh]

Okay, I found something quite bad.

A lot of multi-threading in Ultra is done by adding a function to a queue of functions to be executed in the other thread, and then passing that batch of functions at a specific time, like this:

auto pixmap = visibilitymap->Copy();
world->physicsthreadmanager->AddCommand(std::bind(&Physics::PhysicsMeshLayer::SetVisibilityMap, meshlayer->physicsmeshlayer, pixmap));

The problem is that in the MeshLayer::SetInstanceVisible method, I was doing this every time the visibility of an instance was changed. If the mesh layer was filled, then this resulted in a new copy of the visibility map being made for every single instance. So if you had a 2048x2048 terrain with density set to 1.0, it would result in 4,194,304 pixmaps being created, with a size of 256 x 2048 bytes each, for a total of 2 terabytes of memory. 

The solution is to add the mesh layer into an STL set when the visibility map is modified, and then in the World::Update method, just add an instruction to update the visibility map of each mesh layer once:

void World::UpdateMeshLayers()
{
	for (auto meshlayer : modifiedmeshlayervisibility)
	{
		auto pixmap = std::dynamic_pointer_cast<Pixmap>(meshlayer->visibilitymap->Copy());
		physicsthreadmanager->AddCommand(std::bind(&Physics::PhysicsMeshLayer::SetVisibilityMap, meshlayer->physicsmeshlayer, pixmap));
	}
	modifiedmeshlayervisibility.clear();
}

I apologize for this oversight.

[CJO Games]

2 tb of memory! No wonder it was struggling! ha ha!

No worries about the oversight at all it's good that you found the problem - I'll wait for the update to come down the pipe... Cheers!