Upload bug

[Slimwaffle]

So I found what I think is a bug. My game that I uploaded to steam and is for sale on steam forces the customer to download the full size of the game whenever I upload a new build. It shouldn't be doing this because steam should scan the game files and determine which ones have changed. Then I came across something interesting in the steamworks documentation saying that if the game uses encrypted or compressed files/ folders steams scanning won't work. And I know that when I publish with leadwerks standalone that it compresses the game files/folders to a zip file.

Is there a way to make leadwerks not compress /encrypt these files. Because customers downloading the full game each time is excessive.

Here is the info from steamworks;

Building Efficient Depots for SteamPipe

The old Steam content system would patch updates on a file level, which meant that if a single byte in a file changed, the entire new file would be downloaded by all users. This was especially inefficient if the game used pack files, which are collections of game content files in a single big file. Pack files can easily exceed 1 GB, so updates often led to unnecessarily large downloads. A common way to avoid these large downloads was to add new pack files that overrode content of already shipped pack files. That worked for updates, but it hurt new users long-term, since they ended up downloading unused, already-patched content.

The new content system fixes this problem by splitting each file into roughly 1-MB chunks. Each chunk is then compressed and encrypted before being distributed by the Steam content system. If the game content has large redundant parts, these chunks are reused and the user only has to download each repeated chunk once. However, the real strength of this system is building efficient update patches. While the system is building a patch, the new content is scanned for already known chunks. If it finds them, it reuses them. This means if you change or inject a few bytes in a big file, the user only has to download the changes.

This works well in most cases, but there are still a few pitfalls that need to be avoided when designing the content layout of a game. Do not compress or encrypt your game data. This is already done for in-flight downloads and retail discs by the Steam content system. If you do it too, you will reduce the effectiveness of delta patching. If you package multiple data files in a single pack file, make sure that with each re-packaging, no unnecessary changes are made. One problematic practice is including the full name of the original source files on disk, because the names may change, depending on the build machine. Another bad pattern is including build time stamps for each file. If possible, always add new content to the end of your pack files and keep the order of existing files. Also, keep your pack file’s metadata (offset and sizes to individual files) in one place and don’t intersperse it with content files. Use a binary diff’ing tool like BeyondCompare to look at two builds of your pack files to make sure that hundreds of unwanted changes don’t show up.

If you follow these rules you will minimize patch sizes and only new content will need to be downloaded. Your customers will thank you for that and you will be able to increase the quality of your product by shipping more updates.

[Josh]

You could just upload your uncompressed, unencrypted project, and skip the publish step completely.

[Slimwaffle]

How would I do this? Publish first and then replace the data folder with an uncompressed data folder? That way I have the extra files and the exe as well.

[Josh]

The publish step takes all the files in your project and puts them into a zip file. You are saying you do not want to compress your project into a zip file. So just upload your game to Steam without the publish step in Leadwerks.

[Slimwaffle]

yeah ok. So my next question is how would I publish if I wanted it to use c++. Is that something I do through visual studio?

[Josh]

No, C++ just builds your executable.

However, you probably would want to remove your game's source code and Visual Studio projects when you publish it!

[Slimwaffle]

So with C++ I only have to include the exe?

[Josh]

You do not have to include your source code files. You still need to include any map, texture, model, and other files your game uses.

[Slimwaffle]

ok because when I generate a build using visual studio to a folder on my desktop it just adds 3 files.

Waffles.exp, Waffles (Application) and Waffles.lib

So then I can add the rest of my project to this folder? And I just remove all my scripts?

Because they are all in lua and the cpp files just launch them using the Interpreter.

I am just unsure of which files you are referring to when you say Source code files.

 

[Josh]

You need the following file types:

• map
• lua
• tex
• mat
• mdl
• ttf
• wav
• ogg
• shader
• pfb
• phy

And of course your executable (Waffles.exe).

No other files need to be included.

[Slimwaffle]

Thank you so much mate. Massive help.

[Slimwaffle]

So I tried this and it worked great for fixing the patch sizes. But doing this way offers like no file protection. Steam say they encrypt files. But I can't see any evidence of that. Is there a way to keep my file protection and fix the patch sizes?

So I had a look into how other games on other engines do it. And I think I came up with a middle ground that would work. But I'm not sure if its possible to do on my end.

So I noticed other games create package files. These act almost exactly the same way as the data zip file created when I use the publish standalone feature in leadwerks.

What I would like to do if possible is configure the leadwerks publish feature so that instead of creating a single encrypted zip file for all files in the project. It creates one for each of the sub folders in the main folder. For example, (My project Folder is called Waffles) configure it so that it creates a zip for; Addons folder, Prefabs, Scripts and so on and so forth until it has everything contained in a subfolder in an encrypted/compressed zip of the same name.

That way when patching; Steam says that it can't scan any folder that is encrypted or compressed. But it should only replace the ones that have changed in size. Because it replaces folders and files that have changed in size.

Is this possible? And would it even work?

[gamecreator]

You can zip and password protect files yourself then open them via code.  Code is available on the forums.

[Slimwaffle]

Do you possibly have a link? Using search gives me alot of stuff to filter through.

[gamecreator]

Sure, the specific function usage is here: https://www.leadwerks.com/community/topic/17491-loading-from-the-encrypted-data-file/?do=findComment&comment=113622

Some more code here: https://www.leadwerks.com/community/topic/16566-reading-a-text-file-from-a-package/

And you can find another thread or two if you just search the forums for the word password.

[Slimwaffle]

Thanks heaps mate this was actually very simple.

I simply had to add my text for password to one line of code and then zip everything up myself.

I will let you know how it goes with the uploads. Uploading to steam takes a really long time.

[Slimwaffle]

Thank you soo much guys. Absolute legends.

Using this system of compressing sub-folders works perfectly.

Now when I make a small change to the game it will only upload a few mb.

Also using this system greatly reduces the upload time to steam after the initial build is uploaded.