CODEWITHSUNDEEP
ruby-on-railsrubysftpnet-sftp
Anand Bait
Anand Bait

Reputation: 371

How to do SFTP transfer in batches?

Context: I have approx. 20K files to be transferred via SFTP.
I am skeptical that if I send them in single session then session might terminate midway.
I tried searching for timeout but what I found was timeout for idle session.

So I was trying to search if there is something like transfer in batches where I can transfer 500 files in a batch. And after each batch a new session will start.
I am using Ruby's Net::SFTP library

My current code is

      Net::SFTP.start(host, username, :password => password, :port => port) do |sftp|
        files.each do |file| # files is an array of filenames (with path)
          puts "Transferring '#{file}' to #{host}.."
          sftp.upload!(file, "/#{File.basename(file)}")
        end
      end

What I tried?
I tried to search in docs or in some blogs if there exists some option of batching.

If there is no such option then I am planning to handle this as below

files.each_slice(500) do |batch|
  Net::SFTP.start(host, username, :password => password, :port => port) do |sftp|
    batch.each do |file| # files is an array of filenames (with path)
      puts "Transferring '#{file}' to #{host}.."
      sftp.upload!(file, "/#{File.basename(file)}")
    end
  end
end

Upvotes: 2

Views: 395

Answers (1)

Anand Bait
Anand Bait

Reputation: 371

I was not able to find a timeout value but to get a better performance I used

gem 'jsch_sftp'

You can find details at https://github.com/AquisTech/jsch_sftp

Upvotes: 0

Related Questions