CODEWITHSUNDEEP
rubysftpnet-sftp
0x4a6f4672
0x4a6f4672

Reputation: 28245

IOError: closed stream in Ruby SFTP

The following code tries to list the entries of a remote directory via SFTP and Net::SFTP, but it causes an "closed stream" IOError if the directory contains a large number of files (~ 6000 files): 

require 'net/ssh'
require 'net/sftp'
Net::SFTP.start('hostname', 'username', :password => 'password') do |sftp|
  # list the entries in a directory
  sftp.dir.foreach("/") do |entry|
    puts entry.longname
  end
end

What is the best way to avoid it? Versions are net-sftp Gem: 2.0.5 and net-ssh Gem: 2.2.1, Ruby: 1.8.7. The full error message reads: 

IOError: closed stream
from ~/.rvm/gems/ruby-1.8.7-p330/gems/net-ssh-2.2.1/lib/net/ssh/ruby_compat.rb:33:in `select'
from ~/.rvm/gems/ruby-1.8.7-p330/gems/net-ssh-2.2.1/lib/net/ssh/ruby_compat.rb:33:in `io_select'
from ~/.rvm/gems/ruby-1.8.7-p330/gems/net-ssh-2.2.1/lib/net/ssh/ruby_compat.rb:32:in `synchronize'
from ~/.rvm/gems/ruby-1.8.7-p330/gems/net-ssh-2.2.1/lib/net/ssh/ruby_compat.rb:32:in `io_select'
from ~/.rvm/gems/ruby-1.8.7-p330/gems/net-ssh-2.2.1/lib/net/ssh/transport/packet_stream.rb:73:in `available_for_read?'
from ~/.rvm/gems/ruby-1.8.7-p330/gems/net-ssh-2.2.1/lib/net/ssh/transport/packet_stream.rb:85:in `next_packet'
from ~/.rvm/gems/ruby-1.8.7-p330/gems/net-ssh-2.2.1/lib/net/ssh/transport/session.rb:170:in `poll_message'
from ~/.rvm/gems/ruby-1.8.7-p330/gems/net-ssh-2.2.1/lib/net/ssh/transport/session.rb:165:in `loop'
from ~/.rvm/gems/ruby-1.8.7-p330/gems/net-ssh-2.2.1/lib/net/ssh/transport/session.rb:165:in `poll_message'
from ~/.rvm/gems/ruby-1.8.7-p330/gems/net-ssh-2.2.1/lib/net/ssh/connection/session.rb:451:in `dispatch_incoming_packets'
from ~/.rvm/gems/ruby-1.8.7-p330/gems/net-ssh-2.2.1/lib/net/ssh/connection/session.rb:213:in `preprocess'
from ~/.rvm/gems/ruby-1.8.7-p330/gems/net-ssh-2.2.1/lib/net/ssh/connection/session.rb:197:in `process'
from ~/.rvm/gems/ruby-1.8.7-p330/gems/net-ssh-2.2.1/lib/net/ssh/connection/session.rb:161:in `loop'
from ~/.rvm/gems/ruby-1.8.7-p330/gems/net-ssh-2.2.1/lib/net/ssh/connection/session.rb:161:in `loop_forever'
from ~/.rvm/gems/ruby-1.8.7-p330/gems/net-ssh-2.2.1/lib/net/ssh/connection/session.rb:161:in `loop'
from ~/.rvm/gems/ruby-1.8.7-p330/gems/net-ssh-2.2.1/lib/net/ssh/connection/session.rb:110:in `close'
from ~/.rvm/gems/ruby-1.8.7-p330/gems/net-sftp-2.0.5/lib/net/sftp.rb:36:in `start'

Upvotes: 5

Views: 3916

Answers (1)

0x4a6f4672
0x4a6f4672

Reputation: 28245

The behavior could be deliberate, if we take a look at the dir source code in net-sftp/lib/net/sftp/operations/dir.rb, we see a close operation:

def foreach(path)
  ..  
ensure
  sftp.close!(handle) if handle
end

It is possible that this close operation causes the closed stream error. If it does not indicate a bug, it is possible the catch the IOError exception. It also seems to help to run the SSH event loop occasionally:

begin   
  ..
  sftp.dir.foreach("/") do |entry|
    puts entry.longname
    # ...
    sftp.loop # Runs the SSH event loop 
  end
rescue IOError => Ex   
  puts "*** We are done: "+Ex.message 
end

Upvotes: 3

Related Questions