terrible performance benchmarks when concurrent requests running rails 3.0.0 + passenger + apache + Ruby 1.9.2

Question

I'm using Rails 3.0.0 + passenger + apache2 running on Ruby 1.9.2p290 on an ec2 instance.

Relevant Passenger settings (I've messed with them extensively to no effect):

PassengerMaxPoolSize 30 PassengerPoolIdleTime 0 PassengerMinInstances 10

Also, I have verified I am NOT memory or CPU bound...

I've run some benchmarks and am very confused by the results. I have call that does a fairly complex query that chains 5 different .where() clauses together. (In other words, it uses AREL quite a bit.)

When I run Apache Bench with 1000 calls WITHOUT CONCURRENCY (e.g, ab -n 1000 -c 1), I get the following:

 Concurrency Level:      1
 Time taken for tests:   222.799 seconds
 Complete requests:      1000
 Failed requests:        0
 Write errors:           0
 Total transferred:      489000 bytes
 HTML transferred:       16000 bytes
 Requests per second:    4.49 [#/sec] (mean)
 Time per request:       222.799 [ms] (mean)
 Time per request:       222.799 [ms] (mean, across all concurrent requests)
 Transfer rate:          2.14 [Kbytes/sec] received

Now when I run it with concurrency set to 10 (e.g, ab -n 1000 -c 10), I get the following:

 Concurrency Level:      10
 Time taken for tests:   213.957 seconds
 Complete requests:      1000
 Failed requests:        0
 Write errors:           0
 Total transferred:      489001 bytes
 HTML transferred:       16000 bytes
 Requests per second:    4.67 [#/sec] (mean)
 Time per request:       2139.567 [ms] (mean)
 Time per request:       213.957 [ms] (mean, across all concurrent requests)
 Transfer rate:          2.23 [Kbytes/sec] received

There's absolutely no benefit from concurrent requests! The request per second is still at ~4.5. It's as though the server is handling requests serially.

Now for the really weird part. If I look at the outputted SQL query from the ActiveRecord query interface and instead just use find_by_sql, I get this with no concurrency:

 Concurrency Level:      1
 Time taken for tests:   49.547 seconds
 Complete requests:      1000
 Failed requests:        0
 Write errors:           0
 Total transferred:      489000 bytes
 HTML transferred:       16000 bytes
 Requests per second:    20.18 [#/sec] (mean)
 Time per request:       49.547 [ms] (mean)
 Time per request:       49.547 [ms] (mean, across all concurrent requests)
 Transfer rate:          9.64 [Kbytes/sec] received

No surprises here. find_by_sql is faster than using ActiveRecord and AREL to construct queries. BUT, if I run the previous with concurrency set to 10, I get:

 Concurrency Level:      10
 Time taken for tests:   17.859 seconds
 Complete requests:      1000
 Failed requests:        0
 Write errors:           0
 Total transferred:      489000 bytes
 HTML transferred:       16000 bytes
 Requests per second:    55.99 [#/sec] (mean)
 Time per request:       178.587 [ms] (mean)
 Time per request:       17.859 [ms] (mean, across all concurrent requests)
 Transfer rate:          26.74 [Kbytes/sec] received

Notice how the requests per second jumps 3-fold. So my question:

How/Why does using ActiveRecord (and AREL) to build queries cause concurrent requests to respond so poorly? Could ActiveRecord really be that CPU intensive?

Answer 1

Without seeing your actual code, I would guess that AREL produces less-than-optimal SQL for the query you give it, resulting in the database being bogged down.

Compare the SQL statement produced by AREL with your own SQL. Also try running AREL's query directly in your database to see how its performance compares with your SQL. Try an EXPLAIN on both queries. Keep in mind that some databases get worse at optimising SQL as it gets more complex.