em-websocket - *** buffer overflow detected ***: ruby terminated

Question

I try to use eventmachine + em-websocket + redis to build a notification server. It looks good after I finishing almost all code. But, when I started to test connections, something happened. Connection can't be over than 1015 (I have no idea why 1015).

I try to simplify the code and find the problem, but I don't know what I can do.

Emvironment

• RVM Ruby ruby-1.9.3-p194 (also tried ree)
• Server: Ubuntu 12.04 (also tried Mountain Lion)
• Client: Mountain Lion

Gem

• em-websocket 0.3.8
• eventmachine 0.12.10 (also tried 1.0.0.beta.4)
• em-websocket-client 0.1.1

Client

require 'eventmachine'
require 'em-websocket-client'

EM.run do

  # If I create 1200 connections one time, server will crash. But there is no error message.
  # If I create 1000 connections (1000 connections are connecting) at first and create again, it'll show error message.

  1200.times do |i|

    Fiber.new do
      conn = EventMachine::WebSocketClient.connect("ws://0.0.0.0:8080?channel=channel#{i}")
      conn.callback do
        puts "Connected"
      end

      conn.errback do |e|
        puts "Got error: #{e}"
      end

      conn.stream do |msg|
        puts "got data:"
        puts msg
      end

      conn.disconnect do
        puts "gone"
        EM::stop_event_loop
      end
    end.resume
  end
end

Server

require 'eventmachine'
require 'em-websocket'

EventMachine.run do

  f1 = Fiber.new do
    EventMachine::WebSocket.start(:host => "0.0.0.0", :port => 8080) do |ws|
        ws.onopen do |data|
          puts "WebSocket connection open"
        end

        ws.onclose do
          puts "Connection closed"
        end

        ws.onmessage do |msg|
          puts "Recieved message: #{msg}"
        end
    end
  end

  f1.resume

end

Error Message

*** buffer overflow detected ***: ruby terminated
======= Backtrace: =========
/lib/x86_64-linux-gnu/libc.so.6(__fortify_fail+0x37)[0x7fd7de40d007]
/lib/x86_64-linux-gnu/libc.so.6(+0x107f00)[0x7fd7de40bf00]
/lib/x86_64-linux-gnu/libc.so.6(+0x108fbe)[0x7fd7de40cfbe]
/usr/local/rvm/gems/ruby-1.9.3-p194/gems/eventmachine-0.12.10/lib/rubyeventmachine.so(_ZN14EventMachine_t14_RunSelectOnceEv+0xac)[0x7fd7dcec33dc]
/usr/local/rvm/gems/ruby-1.9.3-p194/gems/eventmachine-0.12.10/lib/rubyeventmachine.so(_ZN14EventMachine_t3RunEv+0x5c)[0x7fd7dcec61ec]
/usr/local/rvm/gems/ruby-1.9.3-p194/gems/eventmachine-0.12.10/lib/rubyeventmachine.so(+0x1e419)[0x7fd7dceca419]
/usr/local/rvm/rubies/ruby-1.9.3-p194/lib/libruby.so.1.9(+0x175941)[0x7fd7de836941]
/usr/local/rvm/rubies/ruby-1.9.3-p194/lib/libruby.so.1.9(+0x16b609)[0x7fd7de82c609]
/usr/local/rvm/rubies/ruby-1.9.3-p194/lib/libruby.so.1.9(+0x171a9a)[0x7fd7de832a9a]
/usr/local/rvm/rubies/ruby-1.9.3-p194/lib/libruby.so.1.9(rb_iseq_eval_main+0xae)[0x7fd7de838c4e]
/usr/local/rvm/rubies/ruby-1.9.3-p194/lib/libruby.so.1.9(+0x62e62)[0x7fd7de723e62]
/usr/local/rvm/rubies/ruby-1.9.3-p194/lib/libruby.so.1.9(ruby_exec_node+0x1d)[0x7fd7de724c0d]
/usr/local/rvm/rubies/ruby-1.9.3-p194/lib/libruby.so.1.9(ruby_run_node+0x1e)[0x7fd7de726cae]
ruby[0x4007fb]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xed)[0x7fd7de32576d]
ruby[0x400829]
======= Memory map: ========
00400000-00401000 r-xp 00000000 ca:00 155311                             /usr/local/rvm/rubies/ruby-1.9.3-p194/bin/ruby
00600000-00601000 r--p 00000000 ca:00 155311                             /usr/local/rvm/rubies/ruby-1.9.3-p194/bin/ruby
00601000-00602000 rw-p 00001000 ca:00 155311                             /usr/local/rvm/rubies/ruby-1.9.3-p194/bin/ruby
00c1c000-019ac000 rw-p 00000000 00:00 0                                  [heap]
7fd7db948000-7fd7db952000 r-xp 00000000 ca:00 184566                     /usr/local/rvm/gems/ruby-1.9.3-p194/gems/hiredis-0.4.5/lib/hiredis/ext/hiredis_ext.so
7fd7db952000-7fd7dbb51000 ---p 0000a000 ca:00 184566                     /usr/local/rvm/gems/ruby-1.9.3-p194/gems/hiredis-0.4.5/lib/hiredis/ext/hiredis_ext.so
7fd7dbb51000-7fd7dbb52000 r--p 00009000 ca:00 184566                     /usr/local/rvm/gems/ruby-1.9.3-p194/gems/hiredis-0.4.5/lib/hiredis/ext/hiredis_ext.so
7fd7dbb52000-7fd7dbb53000 rw-p 0000a000 ca:00 184566                     /usr/local/rvm/gems/ruby-1.9.3-p194/gems/hiredis-0.4.5/lib/hiredis/ext/hiredis_ext.so
7fd7dbb53000-7fd7dbb54000 r-xp 00000000 ca:00 156027                     /usr/local/rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/x86_64-linux/digest/sha1.so
7fd7dbb54000-7fd7dbd53000 ---p 00001000 ca:00 156027                     /usr/local/rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/x86_64-linux/digest/sha1.so
7fd7dbd53000-7fd7dbd54000 r--p 00000000 ca:00 156027                     /usr/local/rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/x86_64-linux/digest/sha1.so
7fd7dbd54000-7fd7dbd55000 rw-p 00001000 ca:00 156027                     /usr/local/rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/x86_64-linux/digest/sha1.so
7fd7dbd55000-7fd7dbd58000 r-xp 00000000 ca:00 156022                     /usr/local/rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/x86_64-linux/digest.so
7fd7dbd58000-7fd7dbf57000 ---p 00003000 ca:00 156022                     /usr/local/rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/x86_64-linux/digest.so
7fd7dbf57000-7fd7dbf58000 r--p 00002000 ca:00 156022                     /usr/local/rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/x86_64-linux/digest.so
7fd7dbf58000-7fd7dbf59000 rw-p 00003000 ca:00 156022                     /usr/local/rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/x86_64-linux/digest.so
7fd7dbf59000-7fd7dbf5a000 r-xp 00000000 ca:00 156025                     /usr/local/rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/x86_64-linux/digest/md5.so
7fd7dbf5a000-7fd7dc159000 ---p 00001000 ca:00 156025                     /usr/local/rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/x86_64-linux/digest/md5.so
7fd7dc159000-7fd7dc15a000 r--p 00000000 ca:00 156025                     /usr/local/rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/x86_64-linux/digest/md5.so
7fd7dc15a000-7fd7dc15b000 rw-p 00001000 ca:00 156025                     /usr/local/rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/x86_64-linux/digest/md5.so
7fd7dc15b000-7fd7dc171000 r-xp 00000000 ca:00 14677                      /lib/x86_64-linux-gnu/libz.so.1.2.3.4
7fd7dc171000-7fd7dc370000 ---p 00016000 ca:00 14677                      /lib/x86_64-linux-gnu/libz.so.1.2.3.4
7fd7dc370000-7fd7dc371000 r--p 00015000 ca:00 14677                      /lib/x86_64-linux-gnu/libz.so.1.2.3.4
7fd7dc371000-7fd7dc372000 rw-p 00016000 ca:00 14677                      /lib/x86_64-linux-gnu/libz.so.1.2.3.4
7fd7dc372000-7fd7dc387000 r-xp 00000000 ca:00 14632                      /lib/x86_64-linux-gnu/libgcc_s.so.1
7fd7dc387000-7fd7dc586000 ---p 00015000 ca:00 14632                      /lib/x86_64-linux-gnu/libgcc_s.so.1
7fd7dc586000-7fd7dc587000 r--p 00014000 ca:00 14632                      /lib/x86_64-linux-gnu/libgcc_s.so.1
7fd7dc587000-7fd7dc588000 rw-p 00015000 ca:00 14632                      /lib/x86_64-linux-gnu/libgcc_s.so.1
7fd7dc588000-7fd7dc66a000 r-xp 00000000 ca:00 1654                       /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.16
7fd7dc66a000-7fd7dc869000 ---p 000e2000 ca:00 1654                       /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.16
7fd7dc869000-7fd7dc871000 r--p 000e1000 ca:00 1654                       /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.16
7fd7dc871000-7fd7dc873000 rw-p 000e9000 ca:00 1654                       /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.16
7fd7dc873000-7fd7dc888000 rw-p 00000000 00:00 0 
7fd7dc888000-7fd7dc8da000 r-xp 00000000 ca:00 16804                      /lib/x86_64-linux-gnu/libssl.so.1.0.0
7fd7dc8da000-7fd7dcada000 ---p 00052000 ca:00 16804                      /lib/x86_64-linux-gnu/libssl.so.1.0.0
7fd7dcada000-7fd7dcadd000 r--p 00052000 ca:00 16804                      /lib/x86_64-linux-gnu/libssl.so.1.0.0
7fd7dcadd000-7fd7dcae3000 rw-p 00055000 ca:00 16804                      /lib/x86_64-linux-gnu/libssl.so.1.0.0
7fd7dcae3000-7fd7dcae4000 rw-p 00000000 00:00 0 
7fd7dcae4000-7fd7dcc83000 r-xp 00000000 ca:00 16817                      /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
7fd7dcc83000-7fd7dce82000 ---p 0019f000 ca:00 16817                      /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
7fd7dce82000-7fd7dce9d000 r--p 0019e000 ca:00 16817                      /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
7fd7dce9d000-7fd7dcea8000 rw-p 001b9000 ca:00 16817                      /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
7fd7dcea8000-7fd7dceac000 rw-p 00000000 00:00 0 
7fd7dceac000-7fd7dced4000 r-xp 00000000 ca:00 172584                     /usr/local/rvm/gems/ruby-1.9.3-p194/gems/eventmachine-0.12.10/lib/rubyeventmachine.so
7fd7dced4000-7fd7dd0d3000 ---p 00028000 ca:00 172584                     /usr/local/rvm/gems/ruby-1.9.3-p194/gems/eventmachine-0.12.10/lib/rubyeventmachine.so
7fd7dd0d3000-7fd7dd0d5000 r--p 00027000 ca:00 172584                     /usr/local/rvm/gems/ruby-1.9.3-p194/gems/eventmachine-0.12.10/lib/rubyeventmachine.so
7fd7dd0d5000-7fd7dd0d7000 rw-p 00029000 ca:00 172584                     /usr/local/rvm/gems/ruby-1.9.3-p194/gems/eventmachine-0.12.10/lib/rubyeventmachine.so
7fd7dd0d7000-7fd7dd0d9000 r-xp 00000000 ca:00 156000                     /usr/local/rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/x86_64-linux/enc/trans/transdb.so
7fd7dd0d9000-7fd7dd2d9000 ---p 00002000 ca:00 156000                     /usr/local/rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/x86_64-linux/enc/trans/transdb.so
7fd7dd2d9000-7fd7dd2da000 r--p 00002000 ca:00 156000                     /usr/local/rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/x86_64-linux/enc/trans/transdb.so
7fd7dd2da000-7fd7dd2db000 rw-p 00003000 ca:00 156000                     /usr/local/rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/x86_64-linux/enc/trans/transdb.so
7fd7dd2db000-7fd7dd2dd000 r-xp 00000000 ca:00 155966                     /usr/local/rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/x86_64-linux/enc/encdb.so
7fd7dd2dd000-7fd7dd4dc000 ---p 00002000 ca:00 155966                     /usr/local/rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/x86_64-linux/enc/encdb.so
7fd7dd4dc000-7fd7dd4dd000 r--p 00001000 ca:00 155966                     /usr/local/rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/x86_64-linux/enc/encdb.so
7fd7dd4dd000-7fd7dd4de000 rw-p 00002000 ca:00 155966                     /usr/local/rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/x86_64-linux/enc/encdb.so
7fd7dd4de000-7fd7dd7a7000 r--p 00000000 ca:00 43369                      /usr/lib/locale/locale-archive
7fd7dd7a7000-7fd7dd8a0000 r-xp 00000000 ca:00 14638                      /lib/x86_64-linux-gnu/libm-2.15.so
7fd7dd8a0000-7fd7dda9f000 ---p 000f9000 ca:00 14638                      /lib/x86_64-linux-gnu/libm-2.15.so
7fd7dda9f000-7fd7ddaa0000 r--p 000f8000 ca:00 14638                      /lib/x86_64-linux-gnu/libm-2.15.so
7fd7ddaa0000-7fd7ddaa1000 rw-p 000f9000 ca:00 14638                      /lib/x86_64-linux-gnu/libm-2.15.so
7fd7ddaa1000-7fd7ddaaa000 r-xp 00000000 ca:00 14625                      /lib/x86_64-linux-gnu/libcrypt-2.15.so
7fd7ddaaa000-7fd7ddcaa000 ---p 00009000 ca:00 14625                      /lib/x86_64-linux-gnu/libcrypt-2.15.so
7fd7ddcaa000-7fd7ddcab000 r--p 00009000 ca:00 14625                      /lib/x86_64-linux-gnu/libcrypt-2.15.so
7fd7ddcab000-7fd7ddcac000 rw-p 0000a000 ca:00 14625                      /lib/x86_64-linux-gnu/libcrypt-2.15.so
7fd7ddcac000-7fd7ddcdb000 rw-p 00000000 00:00 0 
7fd7ddcdb000-7fd7ddcdd000 r-xp 00000000 ca:00 14628                      /lib/x86_64-linux-gnu/libdl-2.15.so
7fd7ddcdd000-7fd7ddedd000 ---p 00002000 ca:00 14628                      /lib/x86_64-linux-gnu/libdl-2.15.so
7fd7ddedd000-7fd7ddede000 r--p 00002000 ca:00 14628                      /lib/x86_64-linux-gnu/libdl-2.15.so
7fd7ddede000-7fd7ddedf000 rw-p 00003000 ca:00 14628                      /lib/x86_64-linux-gnu/libdl-2.15.so
7fd7ddedf000-7fd7ddee6000 r-xp 00000000 ca:00 14665                      /lib/x86_64-linux-gnu/librt-2.15.so
7fd7ddee6000-7fd7de0e5000 ---p 00007000 ca:00 14665                      /lib/x86_64-linux-gnu/librt-2.15.so
7fd7de0e5000-7fd7de0e6000 r--p 00006000 ca:00 14665                      /lib/x86_64-linux-gnu/librt-2.15.so
7fd7de0e6000-7fd7de0e7000 rw-p 00007000 ca:00 14665                      /lib/x86_64-linux-gnu/librt-2.15.so
7fd7de0e7000-7fd7de0ff000 r-xp 00000000 ca:00 14662                      /lib/x86_64-linux-gnu/libpthread-2.15.so
7fd7de0ff000-7fd7de2fe000 ---p 00018000 ca:00 14662                      /lib/x86_64-linux-gnu/libpthread-2.15.so
7fd7de2fe000-7fd7de2ff000 r--p 00017000 ca:00 14662                      /lib/x86_64-linux-gnu/libpthread-2.15.so
7fd7de2ff000-7fd7de300000 rw-p 00018000 ca:00 14662                      /lib/x86_64-linux-gnu/libpthread-2.15.so
7fd7de300000-7fd7de304000 rw-p 00000000 00:00 0 
7fd7de304000-7fd7de4b7000 r-xp 00000000 ca:00 14622                      /lib/x86_64-linux-gnu/libc-2.15.so
7fd7de4b7000-7fd7de6b6000 ---p 001b3000 ca:00 14622                      /lib/x86_64-linux-gnu/libc-2.15.so
7fd7de6b6000-7fd7de6ba000 r--p 001b2000 ca:00 14622                      /lib/x86_64-linux-gnu/libc-2.15.so
7fd7de6ba000-7fd7de6bc000 rw-p 001b6000 ca:00 14622                      /lib/x86_64-linux-gnu/libc-2.15.so
7fd7de6bc000-7fd7de6c1000 rw-p 00000000 00:00 0 
7fd7de6c1000-7fd7de8d9000 r-xp 00000000 ca:00 155313                     /usr/local/rvm/rubies/ruby-1.9.3-p194/lib/libruby.so.1.9.1
7fd7de8d9000-7fd7dead8000 ---p 00218000 ca:00 155313                     /usr/local/rvm/rubies/ruby-1.9.3-p194/lib/libruby.so.1.9.1
7fd7dead8000-7fd7deadd000 r--p 00217000 ca:00 155313                     /usr/local/rvm/rubies/ruby-1.9.3-p194/lib/libruby.so.1.9.1
7fd7deadd000-7fd7deae1000 rw-p 0021c000 ca:00 155313                     /usr/local/rvm/rubies/ruby-1.9.3-p194/lib/libruby.so.1.9.1
7fd7deae1000-7fd7deafd000 rw-p 00000000 00:00 0 
7fd7deafd000-7fd7deb1f000 r-xp 00000000 ca:00 14613                      /lib/x86_64-linux-gnu/ld-2.15.so
7fd7dec02000-7fd7dec03000 rw-p 00000000 00:00 0 
7fd7dec03000-7fd7dec04000 ---p 00000000 00:00 0 
7fd7dec04000-7fd7ded18000 rw-p 00000000 00:00 0 
7fd7ded19000-7fd7ded1a000 ---p 00000000 00:00 0 
7fd7ded1a000-7fd7ded1f000 rw-p 00000000 00:00 0                          [stack:16253]
7fd7ded1f000-7fd7ded20000 r--p 00022000 ca:00 14613                      /lib/x86_64-linux-gnu/ld-2.15.so
7fd7ded20000-7fd7ded22000 rw-p 00023000 ca:00 14613                      /lib/x86_64-linux-gnu/ld-2.15.so
7fffe3620000-7fffe3641000 rw-p 00000000 00:00 0                          [stack]
7fffe3776000-7fffe3777000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]
Aborted (core dumped)

There are some posibilities I think:

1. OS kernal problem. I can change some settings to fix it.
2. Ruby problem
3. EventMachine problem
4. em-websocket problem

Anyone can give me some tips, thank you :)

Answer 1

First of all, look at this answer to increase open file limit. Secondly, you don't have to wrap neither server neither client in Fiber and run it when inside EM.run loop.