Package plugins and Quake files now supported
Package plugins are now supported in Ultra Engine 1.0.2. This allows the engine to handle other package formats besides just ZIP. In order to test this out, I created the Quake Loader plugin, which currently supports the following formats used in the original Quake game:
- PAK
- WAD
- BSP (textures)
- SPR
- LMP
Why Quake? Well, the original Quake game is what got me into game development through modding, but Quake is also great for testing because it's so weird. This game was written before hardware accelerated graphics existed, so it has a lot of conventions that don't match modern 3D standards:
- All animation is vertex morphing (no skeletal animation)
- Texture coordinates are in pixels / integers instead of floating point values
- Three different types of "packages" (PAK, WAD, BSP), the latter two of which can be stored inside the first (packages-in-packages)
-
Image files are stored seven different ways:
- LMP image files
- SPR sprite files
- Color pallette ('@')
- Mipmapped texture in WAD ('D')
- Console picture ('E')
- Status bar pictures ('B')
- Mipmapped textures stored in BSP
Each of the image storage methods store data in slightly different ways. Sometimes the width and height are packed into the image data, sometimes they are found elsewhere in a separate structure, sometimes they are hard-coded, and sometimes you have to guess the image dimensions based on the data size. It's a mess, and it seems like this was made up on-the-fly by several people, but this was cutting-edge technology at the time and I don't think usability outside the specified game they were working on was a big concern.
There are some interesting challenges there, but this forms the basis of several dated but influential games. As time went by, the design became more and more sane. For example, Quake 3 just stores textures as TGA images in a ZIP file with the extension changed to PK3. So I figure if I can make the plugin system flexible enough to work with Quake then it will be able to handle anything.
Originally I tried implementing this using HLLib from Nem's Tools. This library supports Quake formats and more, but I had a lot of problems with it, and got better results when I just wrote my own PAK loading code. WAD files were an interesting challenge. There is no real Quake texture format, so loading the raw texture data from the package made no sense. The most interesting breakthrough in this process was how I handled WAD textures. I finally figured out I can make the WAD loader return a block of data that forms a DDS file when a texture is loaded, even through the texture name has no extension. This tricks the engine into thinking the WAD package contains a DDS file which can easily be loaded, when the reality is quite different. This introduces an important concept, that package plugins don't necessarily have to return the exact data they contain, but instead can return files in a ready-to-use format, and the game engine will never know the difference.
The resulting plugin allows you to easily extract and load textures from the Quake game files. This code will extract textures from a single WAD file.
#include "UltraEngine.h" using namespace UltraEngine; void main(int argc, const char* argv[]) { // Load Quake file loader plugin auto qplg = LoadPlugin("Plugins/QuakeLoader"); // Load FreeImage texture plugin auto fiplg = LoadPlugin("Plugins/FITextureLoader"); // WAD to download WString wadfile = "bigcastle"; // Download a WAD package DownloadFile("https://www.quaketastic.com/files/texture_wads/" + wadfile + ".wad", wadfile + ".wad"); // Load the package auto wad = LoadPackage(wadfile + ".wad"); // Create a subdirectory to save images in CreateDir(wadfile); // Read all files in the package auto dir = wad->LoadDir(""); for (auto file : dir) { // Load the image auto pm = LoadPixmap(file); // Save as a PNG file if (pm) pm->Save(wadfile + "/" + file + ".png"); } OpenDir(wadfile); }
All the files found in the WAD package are saved as PNG images. BSP files will work exactly the same way.
This code will load a Quake package, extract all the textures from all the maps in the game, and save them to a folder on the desktop. This is done by detecting packages-in-packages (WAD and BSP), which return the file type '3', indicating that they can be treated both as a file and as a folder. Since none of these package formats use any compression, pixmaps can be easily loaded straight out of the file without extracting the entire BSP. Since Quake PAK files don't use compression, the whole system just turns into a very complicated blob of data with pointers that store data all over the place:
#include "UltraEngine.h" using namespace UltraEngine; void main(int argc, const char* argv[]) { // Load Quake file loader plugin auto qplg = LoadPlugin("Plugins/QuakeLoader"); // Load FreeImage texture plugin auto fiplg = LoadPlugin("Plugins/FITextureLoader"); // Path to Quake game WString qpath = "C:/Program Files (x86)/Steam/steamapps/common/Quake"; //WString qpath = "D:/SteamLibrary/steamapps/common/Quake"; // Load game package auto pak = LoadPackage(qpath + "/id1/PAK1.PAK"); //Change the current directory to load files with relative paths ChangeDir(qpath + "/id1"); // Create a folder to save images in WString savedir = GetPath(PATH_DESKTOP) + "/Quake"; CreateDir(savedir); // Load the main package directory auto dir = pak->LoadDir("maps"); for (auto file : dir) { // Print the file name Print(file); // Get the file type int type = FileType("maps/" + file); // If a package-in-a-package is found load its contents (BSP and WAD) if (type == 3) { auto subdir = pak->LoadDir("maps/" + file); for (int n = 0; n < subdir.size(); ++n) { // Load the texture (plugin will return DDS files) auto pm = LoadPixmap("maps/" + file + "/" + subdir[n]); // Save to PNG if (pm) pm->Save(savedir + "/" + subdir[n] + ".png"); } } } OpenDir(savedir); }
Here is the result:
Since all of these images can be loaded as pixmaps, does that mean they can also be loaded as a texture? I had to know the answer, so I tried this code:
#include "UltraEngine.h" using namespace UltraEngine; int main(int argc, const char* argv[]) { //Get the displays auto displays = GetDisplays(); //Create a window auto window = CreateWindow("Ultra Engine", 0, 0, 1280, 720, displays[0], WINDOW_CENTER | WINDOW_TITLEBAR); //Create a world auto world = CreateWorld(); //Create a framebuffer auto framebuffer = CreateFramebuffer(window); //Create a camera auto camera = CreateCamera(world); camera->SetClearColor(0.125); camera->SetPosition(0, 0, -2); camera->SetFov(70); //Create a light auto light = CreateBoxLight(world); light->SetRotation(45, 35, 0); light->SetRange(-10, 10); light->SetColor(2); //Create a model auto model = CreateBox(world); // Load Quake file loader plugin auto qplg = LoadPlugin("Plugins/QuakeLoader"); // WAD to download WString wadfile = "bigcastle"; // Download a WAD package DownloadFile("https://www.quaketastic.com/files/texture_wads/" + wadfile + ".wad", wadfile + ".wad"); // Load the package auto wad = LoadPackage(wadfile + ".wad"); // Load a texture from the WAD package auto tex = LoadTexture("sclock"); // Create a material auto mtl = CreateMaterial(); mtl->SetTexture(tex); // Apply the material to the box model->SetMaterial(mtl); //Main loop while (window->Closed() == false and window->KeyDown(KEY_ESCAPE) == false) { model->Turn(0, 1, 0); world->Update(); world->Render(framebuffer); } return 0; }
Here is the result, a texture loaded straight out of a Quake WAD file!
To reiterate, testing with Quake files helped me to come up with two important design features:
- Packages-in-packages, indicated with the file type '3'.
- Packages do not necessarily have to return the same exact data they contain, and can instead prepare the data in a ready-to-use format, when that is desirable. The decision to base file type detection on the contents of the file instead of the file name extension worked well here, and allowed me to load the extension-less WAD texture names as DDS files.
Some of these images came from "bigcastle.wad". I don't know who the original author is.
If you are interested, you can read more about the weird world of Quake file formats here, although I must warn you that specification is not complete! 
-
4
10 Comments
Recommended Comments