We have two recently upgraded Plone 4.3.2 instances behind a haproxy load balancer which itself is behind Apache. We limit each Plone instance to serving two concurrent requests using haproxy configuration. We recently encountered an issue whereby a client sent 4 byte-range requests in quick succession for a PDF that each took between 6 and 8 minutes to get a response. This locked up all available requests for 6 minutes and so haproxy timed out other requests in the queue. The PDF is stored an ATFile object in Plone which I believe should have been migrated to blob storage in our recent upgrade. My question is what steps should we take to prevent a similar scenario in the future? I'm also interested in: how to debug why the byte-range requests on an otherwise lightly loaded server should take so long to respond how plone.app.blob deals with byte-range requests is it possible to configure Apache such that byte-range requests are served from its cache but not from the back-end server As requested here is the haproxy.cfg with superfluous configuration stripped out. global maxconn 450 spread-checks 3 defaults log /dev/log local0 mode http option http-server-close option abortonclose option redispatch option httplog timeout connect 7s timeout client 300s timeout queue 120s timeout server 300s listen cms 127.0.0.1:18181 id 3 balance leastconn option httpchk http-check send-state timeout check 10s acl cms_edit url_dom xxx.xxx.xxx.xxx acl cms_not_ok nbsrv() lt 2 block if cms_edit cms_not_ok server cms_instance1 app:18081 check downinter 10s maxconn 2 rise 1 slowstart 300s server cms_instance2 app:18082 check downinter 10s maxconn 2 rise 1 slowstart 300s

Reputation: 3001

Plone taking a long time to respond to byte-range request

We have two recently upgraded Plone 4.3.2 instances behind a haproxy load balancer which itself is behind Apache.

We limit each Plone instance to serving two concurrent requests using haproxy configuration.

We recently encountered an issue whereby a client sent 4 byte-range requests in quick succession for a PDF that each took between 6 and 8 minutes to get a response. This locked up all available requests for 6 minutes and so haproxy timed out other requests in the queue. The PDF is stored an ATFile object in Plone which I believe should have been migrated to blob storage in our recent upgrade.

My question is what steps should we take to prevent a similar scenario in the future?

I'm also interested in:

how to debug why the byte-range requests on an otherwise lightly loaded server should take so long to respond
how plone.app.blob deals with byte-range requests
is it possible to configure Apache such that byte-range requests are served from its cache but not from the back-end server

As requested here is the haproxy.cfg with superfluous configuration stripped out.

global
    maxconn         450
    spread-checks   3

defaults
    log             /dev/log local0
    mode            http
    option          http-server-close
    option          abortonclose
    option          redispatch
    option          httplog
    timeout connect 7s
    timeout client  300s
    timeout queue   120s
    timeout server  300s

listen cms 127.0.0.1:18181
    id 3
    balance leastconn
    option          httpchk
    http-check      send-state
    timeout check   10s

    acl cms_edit url_dom xxx.xxx.xxx.xxx
    acl cms_not_ok nbsrv() lt 2
    block if cms_edit cms_not_ok

    server cms_instance1 app:18081 check downinter 10s maxconn 2 rise 1 slowstart 300s
    server cms_instance2 app:18082 check downinter 10s maxconn 2 rise 1 slowstart 300s

Upvotes: 1