CODEWITHSUNDEEP
linuxftpbackuptarlarge-data
Tareq
Tareq

Reputation: 3

tar very large files to FTP directly splited into smaller files

I need to backup a large server into FTP storage. I can tar all files, I can upload using FTP and I can split the tar file into many small files.

But the problem is I can't do these three steps in one step. I can tar to FTP directly, I can tar with split, but can't tar with FTP and split.

The OS is CentOS 6.2 The Files Size more than 800G

Thanks

Upvotes: 0

Views: 4166

Answers (2)

djokage
djokage

Reputation: 126

To can tar, split and ftp a directory with one command line you need the following:

split command write to the standard output only, so you can't pass the file to another command like ftp to process it, to do so you need to patch split to can use the --filter option to can pass the output file to ftp "on the fly" without having to save to hard disk by setting up the $FILE environmental variable with the output file (the file names would be x00, x01, x02 ...).

1) Here is the split patch: http://lists.gnu.org/archive/html/coreutils/2011-01/txt3j8asgk8WH.txt After patching split command, you would see in the man that the --filter option available in your split command.

2) install the ncftp ftp client which is a good ftp client that allows you to connect to ftp and put file in one line command, without waiting for the ftp response like ordinary ftp client. the ncftp is useful to integrate with scripts and so on.

here is the command that would compress /home directory with tar split it to 100MB small files and transfer each file through FTP

tar cvz -i /home | split -d -b 100m  --filter 'ncftpput -r 10 -F -c -u ftpUsername -p ftpPassword ftpHost $FILE'

note that we used the ncftpput that would pass the $FILE to ftp in single command too. additional ftp options: -r 10: allows you to try to reconnect 10 times after loosing connection with ftp. -F: To use passive mode. -c: takes the input from stdin.

To merge the split files (x00, x01, x02, x03 ...) to can extract the file use following command

cat x* > originalFile.tar

Upvotes: 2

linux_fanatic
linux_fanatic

Reputation: 5177

You can make a shell script and use 

tar zcf - /usr/folder | split -b 30720m - /usr/archive.tgz

and then upload to FTP also because once you are doing tar and putting onto FTP then how can you split.

Upvotes: 1

Related Questions