Jump to content

Package::LoadDir("") returns nothing on large package


SpiderPig
 Share

Go to solution Solved by Josh,

Recommended Posts

I've compressed 5,259 4096x4096 r16 heightmap files into a zip file.  It's about 7GB of data.  Using LoadDir("") to list the files at the root returns a size of 0 ONLY for this zip file.

I used the code below on a zip file of 5,259 1024x1024 r16 heightmaps (1.2GB) and it worked fine.  Pretty sure all the files have the exact same names in both zip files.  It's just the map resolution that is different.

I thought maybe LoadDir() or LoadPackage() doesn't like the large zip file?

 

This is the snippet of code that handles the loading.

if (extension == "zip") {
	auto package = LoadPackage(filepath);

	if (package != nullptr) {
		auto dir = package->LoadDir("");

 

Link to comment
Share on other sites

13 minutes ago, Josh said:

I compressed it with 7zip, it's interesting that last comment saying it's not a zip?  Even though it is... and I've compressed large files with window's default zip too and it worked above 4GB.

Anyway, I can open the 7GB in windows explorer to search the files and I can extract from it.  Don't know any other software besides 7zip to try it with...

Link to comment
Share on other sites

24 minutes ago, SpiderPig said:

I compressed it with 7zip, it's interesting that last comment saying it's not a zip?

I was confused with that as well, knowing the 7z does do zip. I decided that he was referring to his own data, not what 7z does.

  • Like 1

i now hate love C++

Beeeeeeeeeeeeeep~~This is a test of the emergency signature system~~Beeeeeeeeeeeeeep

RX 6800XT | i5-13600KF | 32GB DDR5 | 1440p is perfect

Link to comment
Share on other sites

3 hours ago, Josh said:

Libzip probably does not either:
https://github.com/nih-at/libzip/issues/239

In this link the guy reckons it does actually work on 64 bit machines... just not on 32bit.  To me that makes sense that a 32bit machine can't load something bigger than 4GB but I'm no expert.

Quote

Note that the same code (same libzip version, same ZIP file) works correctly on 64-bit devices. I checked on both AMD-64 and ARM-64.

 

26 minutes ago, reepblue said:

The library that 7z uses should be apart of ziplib unless Josh excluded it.

I did try loading a 7GB ".7z" file just to be sure but it didn't load.  I don't know much about 7z, it uses LZMA or something?  Maybe this SDK is what I need to make a 7z plugin.

https://www.7-zip.org/sdk.html

But I don't know, maybe libzip has a bug with loading over 4GB files...

Link to comment
Share on other sites

  • 2 weeks later...
On 1/8/2024 at 5:41 PM, reepblue said:

LZMA is in the ziplib repo and should be easily compilable. Josh just has to include it.

I actually rearranged ziplib so it all compiled in one fat static lib and it worked fine.

So you are saying ziplib can be compiled in a way that supports more than 4 GB zip files? Have you done this?

My job is to make tools you love, with the features you want, and performance you can't live without.

Link to comment
Share on other sites

@reepblue I found this in ZipMethodResolver.h but it appears that it already includes everything by default:

#define ZIP_METHOD_TABLE          \
  ZIP_METHOD_ADD(StoreMethod);    \
  ZIP_METHOD_ADD(DeflateMethod);  \
  ZIP_METHOD_ADD(Bzip2Method);    \
  ZIP_METHOD_ADD(LzmaMethod);

#define ZIP_METHOD_ADD(method_class)                                                            \
  if (compressionMethod == method_class::GetZipMethodDescriptorStatic().GetCompressionMethod()) \
    return method_class::Create()

 

My job is to make tools you love, with the features you want, and performance you can't live without.

Link to comment
Share on other sites

  • Solution

Apparently this version of minizip would maybe work, as long as you have less than 65536 files:
https://github.com/zlib-ng/minizip-ng

However, adoption of Zip64 appears to be almost non-existent in the real world. Writing a new archive format isn't exactly difficult, and may be less work than implementing another library, then dealing with all the bad design decisions a programmer who isn't me will invariably make.

If I were to make an encryption format, I would include:

  • unsigned 64-bit integers for everything so we don't have this stupid size limitation bullshit
  • file table would be at the end of the file
  • file removal without shifting all other files

Maybe for really big archives we should be looking at something like that. The advantage would be that you can upload a new version of the archive on Steam and it would only upload the sections that changed, because each build would not shift the entire contents around. I think Valve did something like this with their packages files, which is why they have a defragment feature.

In the immediate term, I recommend splitting your data up into multiple zip archives.

  • Thanks 2

My job is to make tools you love, with the features you want, and performance you can't live without.

Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

 Share

×
×
  • Create New...