Reputation: 83
How to efficiently bundle and debundle many files for transfer and remote processing?
I wish to repeatedly transfer a fixed set of ~800 files to a remote server to run a recurring task.
The files are of varying sizes (mostly < 1MB, but with several larger ones totalling ~220GB) and file formats (mostly textual).
The task executed on the remote server is within a linux docker container and requires these files to be in their specific relative tree structure.
It is not feasible to transfer them individually or recursively by specifying the directory. Neither is using a volume mount in any way.
Q: I am searching for a method that allows to bundle these files into a single file such that they can be de-bundled and written on the remote server quickly.
The bundling of the files is not time-critical as this it will not be a repeated task.
Tar archives I tried tar archives, tar.gz at compression level 9 and level 1 with no significant difference in speed. ~30 minutes.
Are there any faster methods I am not aware of?
Upvotes: 1
Views: 129
Answers (0)
Related Questions
- Copy multiple local files to docker container
- How to do "docker load" for multiple tar files
- How to Copy and Extract a .tar file inside a Docker Container
- Copy many large files from the host to a docker container and back
- Is there a better way to copy files and folders to a docker container?
- Extracting tar from host to docker container
- How can I efficiently move many files to a new server?
- Efficient Dockerfile for many apt-get packages
- Elegantly send local tarball and untar on remote end
- uncompressing a large number of files on the fly