SpiderPig Posted January 8 Share Posted January 8 I've compressed 5,259 4096x4096 r16 heightmap files into a zip file. It's about 7GB of data. Using LoadDir("") to list the files at the root returns a size of 0 ONLY for this zip file. I used the code below on a zip file of 5,259 1024x1024 r16 heightmaps (1.2GB) and it worked fine. Pretty sure all the files have the exact same names in both zip files. It's just the map resolution that is different. I thought maybe LoadDir() or LoadPackage() doesn't like the large zip file? This is the snippet of code that handles the loading. if (extension == "zip") { auto package = LoadPackage(filepath); if (package != nullptr) { auto dir = package->LoadDir(""); Quote Link to comment Share on other sites More sharing options...
Josh Posted January 8 Share Posted January 8 I think 4 GB is the max size for a zip file: https://superuser.com/questions/1305867/what-is-the-maximum-size-of-a-zip-file-on-windows-10-pro-64-bit#:~:text=1 Answer&text=4 GB size is a,about 16 exabytes size limitation). Quote My job is to make tools you love, with the features you want, and performance you can't live without. Link to comment Share on other sites More sharing options...
SpiderPig Posted January 8 Author Share Posted January 8 So 4GB was a limit in the past? So it should work now...? Quote Link to comment Share on other sites More sharing options...
Josh Posted January 8 Share Posted January 8 What happens when you open this zip file in another program? Quote My job is to make tools you love, with the features you want, and performance you can't live without. Link to comment Share on other sites More sharing options...
Josh Posted January 8 Share Posted January 8 https://comp.lang.idl-pvwave.narkive.com/L2ukAnN1/zlib-uncompress-limited-to-4gb Quote My job is to make tools you love, with the features you want, and performance you can't live without. Link to comment Share on other sites More sharing options...
SpiderPig Posted January 8 Author Share Posted January 8 13 minutes ago, Josh said: https://comp.lang.idl-pvwave.narkive.com/L2ukAnN1/zlib-uncompress-limited-to-4gb I compressed it with 7zip, it's interesting that last comment saying it's not a zip? Even though it is... and I've compressed large files with window's default zip too and it worked above 4GB. Anyway, I can open the 7GB in windows explorer to search the files and I can extract from it. Don't know any other software besides 7zip to try it with... Quote Link to comment Share on other sites More sharing options...
SpiderPig Posted January 8 Author Share Posted January 8 I've just confirmed ReadFile() returns nullptr too. The package seems to load okay - well there's no errors and it's not nullptr. Quote Link to comment Share on other sites More sharing options...
SpiderPig Posted January 8 Author Share Posted January 8 If this is a limitation if zlib then this might be an opportunity for a 7zip plugin. Quote Link to comment Share on other sites More sharing options...
IceBurger Posted January 8 Share Posted January 8 24 minutes ago, SpiderPig said: I compressed it with 7zip, it's interesting that last comment saying it's not a zip? I was confused with that as well, knowing the 7z does do zip. I decided that he was referring to his own data, not what 7z does. 1 Quote i now hate love C++ Beeeeeeeeeeeeeep~~This is a test of the emergency signature system~~Beeeeeeeeeeeeeep RX 6800XT | i5-13600KF | 32GB DDR5 | 1440p is perfect Link to comment Share on other sites More sharing options...
SpiderPig Posted January 8 Author Share Posted January 8 Yeah probably. It's defiantly a zip. Quote Link to comment Share on other sites More sharing options...
Josh Posted January 8 Share Posted January 8 Zip64 sounds like one of those things like JPEG2000 where it just gets no adoption. It appears ziplib does not support zip64 files: https://www.mjtnet.com/forum/viewtopic.php?t=7120 Libzip probably does not either: https://github.com/nih-at/libzip/issues/239 Quote My job is to make tools you love, with the features you want, and performance you can't live without. Link to comment Share on other sites More sharing options...
reepblue Posted January 9 Share Posted January 9 5 hours ago, SpiderPig said: If this is a limitation if zlib then this might be an opportunity for a 7zip plugin. The library that 7z uses should be apart of ziplib unless Josh excluded it. Quote Cyclone - Ultra Game System - Component Preprocessor - Tex2TGA - Darkness Awaits Template (Leadwerks) If you like my work, consider supporting me on Patreon! Link to comment Share on other sites More sharing options...
SpiderPig Posted January 9 Author Share Posted January 9 3 hours ago, Josh said: Libzip probably does not either: https://github.com/nih-at/libzip/issues/239 In this link the guy reckons it does actually work on 64 bit machines... just not on 32bit. To me that makes sense that a 32bit machine can't load something bigger than 4GB but I'm no expert. Quote Note that the same code (same libzip version, same ZIP file) works correctly on 64-bit devices. I checked on both AMD-64 and ARM-64. 26 minutes ago, reepblue said: The library that 7z uses should be apart of ziplib unless Josh excluded it. I did try loading a 7GB ".7z" file just to be sure but it didn't load. I don't know much about 7z, it uses LZMA or something? Maybe this SDK is what I need to make a 7z plugin. https://www.7-zip.org/sdk.html But I don't know, maybe libzip has a bug with loading over 4GB files... Quote Link to comment Share on other sites More sharing options...
reepblue Posted January 9 Share Posted January 9 LZMA is in the ziplib repo and should be easily compilable. Josh just has to include it. I actually rearranged ziplib so it all compiled in one fat static lib and it worked fine. 1 Quote Cyclone - Ultra Game System - Component Preprocessor - Tex2TGA - Darkness Awaits Template (Leadwerks) If you like my work, consider supporting me on Patreon! Link to comment Share on other sites More sharing options...
SpiderPig Posted January 9 Author Share Posted January 9 I wonder if that will solve this issue or of there's something else at play... Quote Link to comment Share on other sites More sharing options...
Alienhead Posted January 9 Share Posted January 9 RaR handles up to 8,589,934,591 GB but still limited by fat32. Besides, rar isn't open source is it? EDIT: Nvm, just found thgis https://github.com/peazip/PeaZip Quote I'm only happy when I'm coding, I'm only coding when I'm happy. Link to comment Share on other sites More sharing options...
Canardia Posted January 9 Share Posted January 9 7z has 7 different algorithms, hence the name 7z. The most powerful compression is LZMA2 Ultra. 1 Quote ■ Ryzen 9 ■ RX 6800M ■ 16GB ■ XF8 ■ Windows 11 ■ ■ Ultra ■ LE 2.5 ■ 3DWS 5.6 ■ Reaper ■ C/C++ ■ C# ■ Fortran 2008 ■ Story ■ ■ Homepage: https://canardia.com ■ Link to comment Share on other sites More sharing options...
Josh Posted January 19 Share Posted January 19 On 1/8/2024 at 5:41 PM, reepblue said: LZMA is in the ziplib repo and should be easily compilable. Josh just has to include it. I actually rearranged ziplib so it all compiled in one fat static lib and it worked fine. So you are saying ziplib can be compiled in a way that supports more than 4 GB zip files? Have you done this? Quote My job is to make tools you love, with the features you want, and performance you can't live without. Link to comment Share on other sites More sharing options...
Josh Posted January 19 Share Posted January 19 @reepblue I found this in ZipMethodResolver.h but it appears that it already includes everything by default: #define ZIP_METHOD_TABLE \ ZIP_METHOD_ADD(StoreMethod); \ ZIP_METHOD_ADD(DeflateMethod); \ ZIP_METHOD_ADD(Bzip2Method); \ ZIP_METHOD_ADD(LzmaMethod); #define ZIP_METHOD_ADD(method_class) \ if (compressionMethod == method_class::GetZipMethodDescriptorStatic().GetCompressionMethod()) \ return method_class::Create() Quote My job is to make tools you love, with the features you want, and performance you can't live without. Link to comment Share on other sites More sharing options...
Josh Posted January 19 Share Posted January 19 I think LMZA is already in there. Quote My job is to make tools you love, with the features you want, and performance you can't live without. Link to comment Share on other sites More sharing options...
Solution Josh Posted January 19 Solution Share Posted January 19 Apparently this version of minizip would maybe work, as long as you have less than 65536 files: https://github.com/zlib-ng/minizip-ng However, adoption of Zip64 appears to be almost non-existent in the real world. Writing a new archive format isn't exactly difficult, and may be less work than implementing another library, then dealing with all the bad design decisions a programmer who isn't me will invariably make. If I were to make an encryption format, I would include: unsigned 64-bit integers for everything so we don't have this stupid size limitation bullshit file table would be at the end of the file file removal without shifting all other files Maybe for really big archives we should be looking at something like that. The advantage would be that you can upload a new version of the archive on Steam and it would only upload the sections that changed, because each build would not shift the entire contents around. I think Valve did something like this with their packages files, which is why they have a defragment feature. In the immediate term, I recommend splitting your data up into multiple zip archives. 2 Quote My job is to make tools you love, with the features you want, and performance you can't live without. Link to comment Share on other sites More sharing options...
SpiderPig Posted January 19 Author Share Posted January 19 1 hour ago, Josh said: In the immediate term, I recommend splitting your data up into multiple zip archives. Yeah I'll do this for now. Thankyou. Quote Link to comment Share on other sites More sharing options...
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.