Okay, I found something quite bad.
A lot of multi-threading in Ultra is done by adding a function to a queue of functions to be executed in the other thread, and then passing that batch of functions at a specific time, like this:
auto pixmap = visibilitymap->Copy();
world->physicsthreadmanager->AddCommand(std::bind(&Physics::PhysicsMeshLayer::SetVisibilityMap, meshlayer->physicsmeshlayer, pixmap));
The problem is that in the MeshLayer::SetInstanceVisible method, I was doing this every time the visibility of an instance was changed. If the mesh layer was filled, then this resulted in a new copy of the visibility map being made for every single instance. So if you had a 2048x2048 terrain with density set to 1.0, it would result in 4,194,304 pixmaps being created, with a size of 256 x 2048 bytes each, for a total of 2 terabytes of memory.
The solution is to add the mesh layer into an STL set when the visibility map is modified, and then in the World::Update method, just add an instruction to update the visibility map of each mesh layer once:
void World::UpdateMeshLayers()
{
for (auto meshlayer : modifiedmeshlayervisibility)
{
auto pixmap = std::dynamic_pointer_cast<Pixmap>(meshlayer->visibilitymap->Copy());
physicsthreadmanager->AddCommand(std::bind(&Physics::PhysicsMeshLayer::SetVisibilityMap, meshlayer->physicsmeshlayer, pixmap));
}
modifiedmeshlayervisibility.clear();
}
I apologize for this oversight.