Slow streaming with Thin or no streaming at all

Question

I wrote a little script to stream a file byte-per-byte with Sinatra and Thin:

#!/usr/bin/env ruby

require 'sinatra'
require "sinatra/streaming"

get '/' do
  stream do |out|
    File.open("/usr/share/doc/ia32-libs/copyright", "r") do |fd|
      fd.each_byte do |byte|
        break if out.closed?
        putc byte.chr
        out << byte.chr unless out.closed?
      end
    end
  end
end

This works as expected:

% ruby streamer.rb
== Sinatra/1.3.2 has taken the stage on 4567 for development with backup from Thin
>> Thin web server (v1.4.1 codename Chromeo)
>> Maximum connections set to 1024
>> Listening on 0.0.0.0:4567, CTRL+C to stop


% curl http://127.0.0.1:4567

and the file content is written on both sides to stdout

If I remove the line putc byte.chr then the server crashed and got to be killed by kill -9

---

Another unexpected behaviour hits me if I change this script to modular style and add an config.ru:

#!/usr/bin/env ruby

require 'sinatra/base'
require "sinatra/streaming"

class TestStreamer < Sinatra::Base
  helpers Sinatra::Streaming

  get '/' do
    stream do |out|
      File.open("/usr/share/doc/ia32-libs/copyright", "r") do |fd|
        fd.each_byte do |byte|
          break if out.closed?
#          putc byte.chr
          out << byte.chr unless out.closed?
        end
      end
    end
  end
end

config.ru:

$:.unshift(File.join(File.dirname(__FILE__)))

require 'rubygems'
require 'sinatra/base'
require 'streamer_modular'

map '/' do
  run TestStreamer
end

It starts up:

% thin start
>> Using rack adapter
>> Thin web server (v1.4.1 codename Chromeo)
>> Maximum connections set to 1024
>> Listening on 0.0.0.0:3000, CTRL+C to stop

but when I am doing a request, the file is send extremely slow on both sides and gets closed after 30s or something. Sinatra does not crash in this case.

% time curl -v http://127.0.0.1:3000
* About to connect() to 127.0.0.1 port 3000 (#0)
*   Trying 127.0.0.1... connected
* Connected to 127.0.0.1 (127.0.0.1) port 3000 (#0)
> GET / HTTP/1.1
> User-Agent: curl/7.21.0 (x86_64-pc-linux-gnu) libcurl/7.21.0 OpenSSL/0.9.8o zlib/1.2.3.4 libidn/1.15 libssh2/1.2.6
> Host: 127.0.0.1:3000
> Accept: */*
> 
< HTTP/1.1 200 OK
< X-Frame-Options: sameorigin
< X-XSS-Protection: 1; mode=block
< Content-Type: text/html;charset=utf-8
< Connection: close
< Server: thin 1.4.1 codename Chromeo
< 
* Closing connection #0
This package was cre
curl -v http://127.0.0.1:3000  0,01s user 0,00s system 0% cpu 32,559 total

When I remove the stream-helper the file gets written to stdout on serverside without any delay:

#!/usr/bin/env ruby

require 'sinatra/base'
require "sinatra/streaming"

class TestStreamer < Sinatra::Base
  helpers Sinatra::Streaming

  get '/' do
#    stream do |out|
    begin
      File.open("/usr/share/doc/ia32-libs/copyright", "r") do |fd|
        fd.each_byte do |byte|
        #  break if out.closed?
          putc byte.chr
        #  out << byte.chr unless out.closed?
        end
      end
    end
  end
end

---

% ruby -v
ruby 1.9.2p290 (2011-07-09 revision 32553) [x86_64-linux]

% gem list

*** LOCAL GEMS ***

backports (2.6.2)
bundler (1.1.5, 1.1.3)
daemons (1.1.8)
eventmachine (0.12.10)
rack (1.4.1)
rack-protection (1.2.0)
rack-test (0.6.1)
sinatra (1.3.2)
sinatra-contrib (1.3.1)
thin (1.4.1)
tilt (1.3.3)

% gem update
Updating installed gems
Nothing to update

% ls -al /usr/share/doc/ia32-libs/copyright
-rw-r--r-- 1 root root 607403  2. Jan 2012  /usr/share/doc/ia32-libs/copyright

% head -n2 /usr/share/doc/ia32-libs/copyright
This package was created by Daniel Jacobowitz <dan@debian.org> on Sun, Aug
8th, 2004.  It was based on the ia32-libs package by Bdale Garbee

Am I missing something? Can you reproduce this and maybe got a workaround for me?

Answer 1

Streaming byte for byte on Thin in this mode is somewhat inefficient, since there is a lot of scheduling and deferring needed due to Thin being based on EventMachine (hence Mike's comment on streaming not working on Thin, Mike probably has a Rails background where such streaming on Thin indeed doesn't work).

However, Thin (and any Rack server) should be able to stream a file directly:

get '/' do
  File.open("/usr/share/doc/ia32-libs/copyright", "r")
end