CODEWITHSUNDEEP
cmultithreadingalgorithmfilepthreads
Thangaraj
Thangaraj

Reputation: 3184

Efficient way to write multiple file content into a single file

There are n-number of files with vary in size. How we could efficently append the content of all the files into a single file?

Techniques or algorithm would help? Basically I am expecting efficent method to achieve this in c language.

Upvotes: 0

Views: 191

Answers (4)

Tony Bai
Tony Bai

Reputation: 151

  1. get the size Sn of each file and calculate the total size T of all the files
  2. create the dest file
  3. use mmap to map the dest file with the size T, you will get a pointer P to the start address of the memmap region
  4. mmap each file to mem, and copy each data to the region above in order.
  5. after that, you would get the dest file with all the data from all the files

Upvotes: 0

Matt Ball
Matt Ball

Reputation: 360046

Start simple. Multithreading will introduce significant complexity, and won't necessarily make things run any faster. Pseudocode time:

Create a new file "dest" in write-only mode.
For each file "source" you want to append:
    Open "source" in read-only mode
    For each line "L" in "source":
        Write "L" to "dest"
    Close "source"
Close "dest"

BTW, this is dead simple (and near-optimal) to implement using simple command-line Linux tools (cat, etc.), though of couse that isn't exactly portable to Windows. One-liner example:

for i in `find . -type f -name "*.txt"`; do cat $i >> result.out; done

(Find every .txt file in the current directory and append it to result.out.)

Upvotes: 3

Don
Don

Reputation: 72

Since I don't what the contents of the files are or the purpose of appending them, this solution might not be the best if its just text or something. However, I'd probably find a zip library to use (either licensed or open source), then just zip all the files into a single archive.

zlib looks interesting: http://www.zlib.net/

Upvotes: 1

user541686
user541686

Reputation: 210765

Go through and find the total size of all of the files.

Then allocate an output file of that size, go through them again and write the data to your output.

Upvotes: 2

Related Questions