CODEWITHSUNDEEP
network-programmingcompressionzip
bgee
bgee

Reputation: 989

How to move lot of files through network

Let say we have next situation: two computers connected by network, directory (flat) full of small (or even different sizes) files needed to be moved through network. What is faster: 1) to move the directory straight through network or 2) zip (or compress by any other algorithm), move zipped file, and unzip atfer that?

From my experience it's seems to be second one, but how I can see it's right?

I understand that this is not so simple - yes or not, there are players like bandwidth, zip algorithms, files sizes and average compression ratio and so on, but I just wonder may be there is some computer science I am ignorant of?

Upvotes: 0

Views: 385

Answers (2)

Mark Adler
Mark Adler

Reputation: 112502

You did not say what sort of system you're using. The best way is to do both. Send a compressed archive over the network, and decompress it on the other end, without ever having the entire compressed archive stored at either end. On Unix systems this is normally done with tar, which calls gzip for compression. You can tar directly into a pipe, and untar at the other end. You can also use rsync, which compresses on the link as well, but is more sophisticated, avoiding sending data that is already at the destination.

Upvotes: 1

cjayho
cjayho

Reputation: 545

I think better way is compressing multiple files to single archive.

For example, modern 3d games - lots of game resource files are compressed to bigger archives with uncompressing directly to memory while game running. This significantly decreases game loading.

Pros:

1) there is no need to re-read directory index many times, directory index is read only once

2) there is no need to make multiple fopen

3) compression of data - when using modern processors (not older than 10 years :) ) reading from hard disk is much slower than unpacking process (I don't know how this is with SSD disks).

4) When using hard disks, reading single (even fragmented) big file is much faster than many small files in different places of disk. When reading multiple files reading speed decreases because hard disk need to position its heads.

5) While transfer multiple files for example using FTP, we need to create STOR command for every file, while transferring an archive we need only one STOR command.

Upvotes: 0

Related Questions