Load Package not keeping folder hierarchy

[SpiderPig]

Hi guys,  I'm loading multiple packages from within sub folders and it although they are loading correctly and being password verified I cant load files from them.

The package loads from a folder in my game directory;

Loaded package!  C:/Users/Matt/Desktop/The Seventh World\GameData\Icons\Icons.zip Verified!

Then I load an icon which fails;

Error: Failed to load texture "C:/Users/Matt/Desktop/The Seventh World/GameData/Icons/Bark.tex"

But this works;

Loading texture "C:/Users/Matt/Desktop/The Seventh World/Bark.tex"

It seems loading a package doesn't register itself with it's own directory but to that of the game directory?  @Joshis this a bug?

[Josh]

This is the intended behavior. Zipping everything up should not change the file paths.

[SpiderPig]

But zipping a sub-directory means I can't access those files with the same path as though they are not zipped...?

[Josh]

I can't discuss this without an example to work with. This is too confusing.

[SpiderPig]

Okay I'll be able to put one together later.

[Josh]

This is a system that has been in use for years, so I am confused what the problem actually is?

[SpiderPig]

I'm sure I'm not explaining it right.  All my assets are in sub folders as per usual.  When I want to publish the game i package up a group of files in each of the folders so when steam users download the update it's not massive.  Loading each package works well, but I then cant load assets from the package because the file path to that asset has been changed.  Instead of being this;

C:/Users/Matt/Desktop/The Seventh World/GameData/Icons/Bark.tex

it needs to be this in order to work;

C:/Users/Matt/Desktop/The Seventh World/Bark.tex

The zip package is in;

GameData/Icons/ICONS.zip

"GameData/Icons/" has to be removed in order for the texture to be loaded.

 

Hope that's better.  Otherwise I'll put an example together.

[Josh]

Oh, I think I see. The file system considers the zip file to be the same as the root game folder. Really I think all your zip packages should be in the base game directory.

[SpiderPig]

Yeah that's it.  I guess if it's not a bug, I could keep them in the base directory, or I could just change the path to them once the game is published.  It'll just be a pain to change all the file paths.  It's a shame really that many packages are needed for steam to keep the update sizes down.

[Josh]

Steam already has a system that only updates parts of files, not each entire changed file.

[SpiderPig]

From memory though it doesn't work on encrypted files.  I'll have to find where I saw it.

[gamecreator]

Let us know; I'm curious.

[SpiderPig]

I haven't found the Steam-works docs yet but a google search showed this;

Quote

Each and every game on Steam is considered by Steam to be made up of 1MB blocks. When a developer prepares to upload a new version of a game to Steam, the Steam tools scan the game's files, and works out how to make that new version of the game out of the data blocks it already knows about, and uploads any new data blocks that it finds, along with a manifest which specifies how to build that game version out of the blocks.

When you download an update, what the Steam client does is downloads the manifest for the new version, then begins to build the new version of the game in a separate "downloading" folder in the "SteamApps" folder. It does this by comparing the new manifest with the old manifest for the version of the game you already have downloaded. It works out which data blocks it can copy across from the existing install (by using the old manifest to look up where in the existing files a particular block can be found), and which it has to download.

Once the new version of the game is fully built by copying and downloading, Steam deletes the old version of the game, and replaces it with the new version.

I think with encryption and/or compression, more than 1MB of data can change meaning the entire zip archive could be downloaded again for only a small file change.

[SpiderPig]

Here we go; https://partner.steamgames.com/doc/sdk/uploading#Building_Efficient_Depots

Quote

Building Efficient Depots for SteamPipe

The old Steam content system would patch updates on a file level, which meant that if a single byte in a file changed, the entire new file would be downloaded by all users. This was especially inefficient if the game used pack files, which are collections of game content files in a single big file. Pack files can easily exceed 1 GB, so updates often led to unnecessarily large downloads. A common way to avoid these large downloads was to add new pack files that overrode content of already shipped pack files. That worked for updates, but it hurt new users long-term, since they ended up downloading unused, already-patched content.

The new content system fixes this problem by splitting each file into roughly 1-MB chunks. Each chunk is then compressed and encrypted before being distributed by the Steam content system. If the game content has large redundant parts, these chunks are reused and the user only has to download each repeated chunk once. However, the real strength of this system is building efficient update patches. While the system is building a patch, the new content is scanned for already known chunks. If it finds them, it reuses them. This means if you change or inject a few bytes in a big file, the user only has to download the changes.

This works well in most cases, but there are still a few pitfalls that need to be avoided when designing the content layout of a game. Do not compress or encrypt your game data. This is already done for in-flight downloads and retail discs by the Steam content system. If you do it too, you will reduce the effectiveness of delta patching. If you package multiple data files in a single pack file, make sure that with each re-packaging, no unnecessary changes are made. One problematic practice is including the full name of the original source files on disk, because the names may change, depending on the build machine. Another bad pattern is including build time stamps for each file. If possible, always add new content to the end of your pack files and keep the order of existing files. Also, keep your pack file’s metadata (offset and sizes to individual files) in one place and don’t intersperse it with content files. Use a binary diff’ing tool like BeyondCompare to look at two builds of your pack files to make sure that hundreds of unwanted changes don’t show up.

If you follow these rules you will minimize patch sizes and only new content will need to be downloaded. Your customers will thank you for that and you will be able to increase the quality of your product by shipping more updates.

 

[SpiderPig]

This is why I think it would be better to package the game data into smaller zip files rather than one large one.  If we can get the package system to keep the directory relative to the pack itself rather than the base directory, I think it would be easier.

[Josh]

What if I just make the editor create multiple zip files?

[SpiderPig]

You mean one for each top level folder?  I.e. Shaders, Scripts, Sound, Models, etc.  If that's the case then I think they still be quite big.

[Josh]

Not sure. It might be that or just data0,zip, data1, etc...

This is actually already built into the system but the size limit is so high you aren't hitting it.

[SpiderPig]

Hmm okay.  From what I'm thinking, all of my packages will be around the 10 to 30 MB range.  That way the updates should be small unless I add heaps.  I just didn't like the idea of adding 100+ packages to the main game directory.  I think what I'll do is just re-add the hierarchy into each zip.  That way I won't have to change file paths and can keep them in any sub folder.

[reepblue]

To be honest, the end user should decide how they want their games to be broken up. This way a hot fix doesn't mean the entire game to be redownloaded if the developer made a pack for scripts and another for materials and models and so on.