CODEWITHSUNDEEP
mysqlruby-on-railsamazon-ec2
Torsten Robitzki
Torsten Robitzki

Reputation: 2555

Rails application hanging in mysql_ping

I have a rails application, that runs in a home brewed web server on an ec2 server (m1.small with amazon linux) with an AWS-rds/mysql database (t1.micro). This runs like a charm for days (this morning the uptime for the last 30 days was about 99.9%).

But once in a while the application stucks for about 14 minutes (the application is monitored by pingdom). When it happens, it usually happens in batches. Today I had that issue already 4 times. When I'm fast enough, I can login into the server, install gdb and attach the debugger to the web server. The top of the stack then looks like this:

thread 1.
(gdb) bt
#0  0x00007fafa28b154d in read () from /lib64/libpthread.so.0
#1  0x00007faf98736332 in ?? () from /usr/lib64/mysql/libmysqlclient.so.18
#2  0x00007faf9872841f in ?? () from /usr/lib64/mysql/libmysqlclient.so.18
#3  0x00007faf98728ffa in ?? () from /usr/lib64/mysql/libmysqlclient.so.18
#4  0x00007faf98722615 in ?? () from /usr/lib64/mysql/libmysqlclient.so.18
#5  0x00007faf98726254 in ?? () from /usr/lib64/mysql/libmysqlclient.so.18
#6  0x00007faf9871e30d in mysql_ping () from /usr/lib64/mysql/libmysqlclient.so.18
#7  0x00007faf98be1aed in nogvl_ping (ptr=0x47a1ec0) at client.c:627
#8  0x00007fafa2c59c29 in rb_thread_blocking_region () from /home/ec2-user/.rvm/rubies/ruby-1.9.3-p327/lib/libruby.so.1.9
#9  0x00007faf98be1b5d in rb_mysql_client_ping (self=70801240) at client.c:636
#10 0x00007fafa2c3f108 in call_cfunc () from /home/ec2-user/.rvm/rubies/ruby-1.9.3-p327/lib/libruby.so.1.9
#11 0x00007fafa2c3fa0d in vm_call_cfunc () from /home/ec2-user/.rvm/rubies/ruby-1.9.3-p327/lib/libruby.so.1.9
#12 0x00007fafa2c400d3 in vm_call_method () from /home/ec2-user/.rvm/rubies/ruby-1.9.3-p327/lib/libruby.so.1.9
#13 0x00007fafa2c45987 in vm_exec_core () from /home/ec2-user/.rvm/rubies/ruby-1.9.3-p327/lib/libruby.so.1.9
#14 0x00007fafa2c52d2a in vm_exec () from /home/ec2-user/.rvm/rubies/ruby-1.9.3-p327/lib/libruby.so.1.9
#15 0x00007fafa2c516af in invoke_block_from_c () from /home/ec2-user/.rvm/rubies/ruby-1.9.3-p327/lib/libruby.so.1.9
#16 0x00007fafa2c517c5 in vm_yield () from /home/ec2-user/.rvm/rubies/ruby-1.9.3-p327/lib/libruby.so.1.9

The mysql version is 5.5. There are no entries in the database logs provided by aws. The rails logs just have a gap of 14 minutes (xxx/auto_test is the url used by the AWS load balancer to check the instance every 10 seconds): 

Started GET "/xxx/auto_test" for 10.224.95.251 at 2013-02-06 17:59:32 +0000
Processing by HealthCheckController#status as */*
  Rendered health_check/status.html.erb within layouts/application (0.1ms)
  Rendered layouts/_render_flash.html.erb (0.1ms)
  Rendered layouts/_debug_info.html.erb (0.0ms)
Completed 200 OK in 8ms (Views: 6.0ms | ActiveRecord: 1.3ms)
Started GET "/xxx/auto_test" for 10.224.95.251 at 2013-02-06 18:13:38 +0000
Processing by HealthCheckController#status as */*
  Rendered health_check/status.html.erb within layouts/application (0.1ms)
  Rendered layouts/_render_flash.html.erb (0.1ms)
  Rendered layouts/_debug_info.html.erb (0.0ms)
Completed 200 OK in 7ms (Views: 5.5ms | ActiveRecord: 1.2ms)

During that outage, the requests from the load balancer pile up and are answered, when the data base isn't blocking anymore.

What might cause the data base to block? What informations do I have to look for, to solve this issue? Any suggestions for a workaround? Any pointers are welcome and highly appreciated!

Update:

I saw the problem today again. The outage lasted exactly 14 minutes, I attached a debugger and got the very same backtrace. So using the native MySql timeout doesn't mitigates the problem.

iptables -L didn't showed any interesting either.

14 minutes, what could 14 minutes be? 41 would at least be close to 42, but 14, hm...

Upvotes: 1

Views: 686

Answers (2)

avaynshtok
avaynshtok

Reputation: 2519

You seem to be running into something similar to what's described here: https://github.com/brianmario/mysql2/pull/287

Are you using the mysql2 adapter by chance? If not, can you try using it and setting the read_timeout option to see if it changes anything?

Upvotes: 2

Hongli
Hongli

Reputation: 18944

This may be a firewall issue. You should check that there are no scripts on your system that happens to configure the firewall in such a way that connections to MySQL are blocked for 15 minutes. You can verify this hypothesis by running 'iptables -L' during the time window in which MySQL connections are misbehaving.

Upvotes: 2

Related Questions