best way to transfer big files?

[AnniXa]

Hey, for my actual game i made a lobby that is syncing the folder with the stuff for the game session,

it syncing some .ini files, some .lua files and some very simple mesh files some shader.verts and a .mat file. it checks for MD5 sums and only syncs the neccesary files and stuff.

 

This works very well and fast at all, but now i wanted to make modders able to also change textures for a game mod, and i wanted to let it sync the textures also, but that .dds files are very big, and a 4mb file takes alot of time to transfer for any reason.

 

I use raknet for my network stuff and i write my own file transfer stuff since the raknet internal thingie was not working for me.

 

Actually i send a Reliable pack with file data to the client and it sends a reliable answer, every pack have a maximal packet size of ~1000, so this will be ~4000 packets that need to be sendet for a 4mb DDS file.

 

does anyone got experiences with such stuff?

Should i just spam all the packets without waiting for a reply from the client? but then there could be like 4000 packets in the buffer or something

 

~edit ahw when i start writing forum posts i mostly get the answers...

 

i had the update of the server sided synchronisation stuff setted to have a delay of 1000ms, so only 1 packet per second was send...

 

now it takes just seconds to transfer such files

 

but maybe someone knows some bether solutions to this?

[ParaToxic]

For better performance you could use the zlib library to compress the textures and decompress them later.(The server has a kind of cache folder with the compressed new textures and the client only download them and decompress (decompressing with zlib is very fast.I had around 30MB/s)

[AnniXa]

For better performance you could use the zlib library to compress the textures and decompress them later.(The server has a kind of cache folder with the compressed new textures and the client only download them and decompress (decompressing with zlib is very fast.I had around 30MB/s)

 

Actually i only compress the map chunk data, i thought dds would be compressed internally in any kind and so it would not bring alot?

 

But thats a good idea to compress all files before transfer, i will do it.

i wonder why i not thougt about that?^^

[TheoLogic]

There are 2 raknet plugins which may interest your:

- http://www.jenkinssoftware.com/raknet/manual/directorydeltatransfer.html

- http://www.jenkinssoftware.com/raknet/manual/filelisttransfer.html

 

Never used them, so feedback if you do would be great! If you use C++ on Windows only I propose:

http://www.codeproject.com/Articles/7530/Zip-Utils-clean-elegant-simple-C-Win32

[Canardia]

I found today xz compressor, which beats even 7-zip ultra lzma2 mode, in both compression ratio and speed.

I found it when I was wondering with what those latest GNU coreutils (because I needed a SHA512 calculator) were packed with, as usually GNU things are packed with .tar.gz.

7-Zip can still read xz files, because they are still LZMA2 format, but just optimized very nicely.

Even in fast mode, xz packs very good, which is good for online networking.

I guess xz will get a standard for GNU things, because those guys only care about efficiency.