Streaming Terrain Data Preparation
Entry posted by Josh ·
2,580 views
Being able to support huge worlds is great, but how do you fill them up with content? Loading an entire planet into memory all at once isn't possible, so we need a system that allows us to stream terrain data in and out of memory dynamically. I wanted a system that could load data from any source, including local files on the hard drive or online GIS sources. Fortunately, I developed most of this system last spring and I am ready to finish it up now.
Preparing Terrain Data
The first step is to create a set of data to test with. I generated a 32768x32768 terrain using L3DT. This produces a 2.0 GB heightmap. The total terrain data with normals, terrain layers, and other data would probably exceed 10 GB, so we need to split this up into smaller pieces.
Loading a 2 GB file into memory might be okay, but we have some special functionality in the new engine that can help with this. First, some terminology: A Stream is an open file that can be read from or written to. A Buffer is a block of memory that can have values poked / peeked at a specific offset. (This is called a "Bank" in Leadwerks 4.) A BufferStream is a block of memory with an internal position value that allows reading and writing with Stream commands. We also have the new StreamBuffer class, which allows you to use Buffer commands on a file on the hard drive! The advantage here is you can treat a BufferStream like it's a big block of memory without actually loading the entire file into memory at once.
Our Pixmap class allows easy manipulation, copying, and conversion of pixel data. The CreatePixmap() function can accept a Buffer as the source of the pixel data. The StreamBuffer class is derived from the Buffer class, so we can create a StreamBuffer from a file and then create a 32768x32768 pixmap without actually loading the data into memory like so:
auto stream = ReadFile("Terrain/32768/32768.r16"); auto buffer = CreateStreamBuffer(stream,0,stream->GetSize()); auto pixmap = CreatePixmap(32768, 32768, TEXTURE_R16, buffer);
So at this point we have a 32768x32768 heightmap that can be manipulated without actually using any memory.
Next we are going to split the pixmap up into a bunch of smaller pixmaps and save each one as a separate file. To do this, we will create a single 1024x1024 pixmap:
auto dest = CreatePixmap(1024, 1024, TEXTURE_R16);
Then we simply walk through the original heightmap, copy a 1024x1024 patch of data to our small heightmap, and save each patch as a separate file in .dds format:
CreateDir("Terrain/32768/LOD0"); for (int x = 0; x < pixmap->size.x / 1024; ++x) { for (int y = 0; y < pixmap->size.y / 1024; ++y) { pixmap->CopyRect(x * 1024, y * 1024, 1024, 1024, dest, 0, 0); dest->Save("Terrain/32768/LOD0/" + String(x) + "_" + String(y) + ".dds"); } }
We end up with a set of 1024 smaller heightmap files. (I took this screenshot while the program was still processing, so at the time there were only 411 files saved.)
Creating LODs
When you are working with large terrains it is necessary to store data at multiple resolutions. The difference between looking at the Earth from orbit and at human-scale height is basically like the difference between macroscopic and microscopic viewing. (Google Earth demonstrates this pretty well.) We need to take our full-resolution data and resample it into a series of lower-resolution data sets. We can do that all in one go with the following code:
int num = 32; // =32768/1024 int lod = 0; while (num > 0) { CreateDir("Terrain/32768/LOD" + String(lod+1)); for (int x = 0; x < num / 2; ++x) { for (int y = 0; y < num / 2; ++y) { auto pm00 = LoadPixmap("Terrain/32768/LOD" + String(lod) + "/" + String(x * 2 + 0) + "_" + String(y * 2 + 0) + ".dds"); auto pm10 = LoadPixmap("Terrain/32768/LOD" + String(lod) + "/" + String(x * 2 + 1) + "_" + String(y * 2 + 0) + ".dds"); auto pm01 = LoadPixmap("Terrain/32768/LOD" + String(lod) + "/" + String(x * 2 + 0) + "_" + String(y * 2 + 1) + ".dds"); auto pm11 = LoadPixmap("Terrain/32768/LOD" + String(lod) + "/" + String(x * 2 + 1) + "_" + String(y * 2 + 1) + ".dds"); pm00 = pm00->Resize(512, 512); pm10 = pm10->Resize(512, 512); pm01 = pm01->Resize(512, 512); pm11 = pm11->Resize(512, 512); pm00->CopyRect(0, 0, 512, 512, dest, 0, 0); pm10->CopyRect(0, 0, 512, 512, dest, 512, 0); pm01->CopyRect(0, 0, 512, 512, dest, 0, 512); pm11->CopyRect(0, 0, 512, 512, dest, 512, 512); dest->Save("Terrain/32768/LOD" + String(lod + 1) + "/" + String(x) + "_" + String(y) + ".dds"); } } num /= 2; lod++; }
The LOD1 folder then contains 256 1024x1024 heightmaps. The LOD2 folder contains 64, and so on, all the way to LOD 5 which contains the entire terrain downsampled into a single 1024x1024 heightmap:
Now we have a multi-resolution data set that can be dynamically loaded into the engine. (If we were loading data from an online GIS data set it would probably already be set up like this.) The next step will be to set up a custom callback function that handles the data loading.
-
3
2 Comments
Recommended Comments