nginx recv() failed (104: Connection reset by peer) while reading response header from upstream

Question

I have recently upgrade my magento from 1.5 to 1.9 and when ever I add a certain product to basket, I started to receive this error: 502 Bad Gateway

There were no log entries in the var/log/ folder:

So, I had a look at my nginx errors and I found the following entries in nginx-errors.log:

2015/04/09 10:58:03 [error] 15208#0: *3 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 46.xxx.xxx.xxx, server: dev.my-domain.co.uk, request: "POST /checkout/cart/add/uenc/aHR0cDovL2Rldi5zYWx2ZW8uY28udWsvdGludGktYmF0aC1wYWludGluZy1zb2FwLTcwbWwuaHRtbD9fX19TSUQ9VQ,,/product/15066/form_key/eYLc3lQ35BSrk6Pa/ HTTP/1.1", upstream: "fastcgi://unix:/var/run/php-fcgi-www-data.sock:", host: "dev.my-domain.co.uk", referrer: "http://dev.my-domain.co.uk/tinti-bath-painting-soap-70ml.html"
2015/04/09 11:04:42 [error] 15208#0: *13 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 46.xxx.xxx.xxx, server: dev.my-domain.co.uk, request: "POST /checkout/cart/add/uenc/aHR0cDovL2Rldi5zYWx2ZW8uY28udWsvdGludGktYmF0aC1wYWludGluZy1zb2FwLTcwbWwuaHRtbD9fX19TSUQ9VQ,,/product/15066/form_key/eYLc3lQ35BSrk6Pa/ HTTP/1.1", upstream: "fastcgi://unix:/var/run/php-fcgi-www-data.sock:", host: "dev.my-domain.co.uk", referrer: "http://dev.my-domain.co.uk/tinti-bath-painting-soap-70ml.html"
2015/04/09 11:05:03 [error] 15208#0: *16 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 46.xxx.xxx.xxx, server: dev.my-domain.co.uk, request: "POST /checkout/cart/add/uenc/aHR0cDovL2Rldi5zYWx2ZW8uY28udWsvdGludGktYmF0aC1wYWludGluZy1zb2FwLTcwbWwuaHRtbD9fX19TSUQ9VQ,,/product/15066/form_key/eYLc3lQ35BSrk6Pa/ HTTP/1.1", upstream: "fastcgi://unix:/var/run/php-fcgi-www-data.sock:", host: "dev.my-domain.co.uk", referrer: "http://dev.my-domain.co.uk/tinti-bath-painting-soap-70ml.html"
2015/04/09 11:12:07 [error] 15273#0: *1 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 46.xxx.xxx.xxx, server: dev.my-domain.co.uk, request: "POST /checkout/cart/add/uenc/aHR0cDovL2Rldi5zYWx2ZW8uY28udWsvdGludGktYmF0aC1wYWludGluZy1zb2FwLTcwbWwuaHRtbD9fX19TSUQ9VQ,,/product/15066/form_key/eYLc3lQ35BSrk6Pa/ HTTP/1.1", upstream: "fastcgi://unix:/var/run/php-fcgi-www-data.sock:", host: "dev.my-domain.co.uk", referrer: "http://dev.my-domain.co.uk/tinti-bath-painting-soap-70ml.html"

I have installed magento on a custom LEMP stack, here are the configurations:

• nginx.conf
• fastcgi_params
• dev.my-domain.co.uk.conf (nginx vhost)
• www-data.conf (php-fpm pool config)

This error only appears to be happening when I add a specific product to basket in my upgraded magento and every time the error occurs, I can see a core.XXXXX file (which is approx 350mb) in the public_html folder.

Any idea why my php-fpm is crashing like this? How can I find the cause and fix it?

Here's the last entries on my Linux (CentOS) server, when I run dmesg command:

php-fpm[14862]: segfault at 7fff38236ff8 ip 00000000005c02ba sp 00007fff38237000 error 6 in php-fpm[400000+325000]
php-fpm[15022]: segfault at 7fff38351ff0 ip 00000000005bf6e5 sp 00007fff38351fb0 error 6 in php-fpm[400000+325000]
php-fpm[15021]: segfault at 7fff38351ff0 ip 00000000005bf6e5 sp 00007fff38351fb0 error 6 in php-fpm[400000+325000]
php-fpm[15156]: segfault at 7fff38351ff0 ip 00000000005bf6e5 sp 00007fff38351fb0 error 6 in php-fpm[400000+325000]
php-fpm[15024]: segfault at 7fff38351ff0 ip 00000000005bf6e5 sp 00007fff38351fb0 error 6 in php-fpm[400000+325000]
php-fpm[15223]: segfault at 7fff8d1d5fd8 ip 00000000005c02ba sp 00007fff8d1d5fe0 error 6 in php-fpm[400000+325000]
php-fpm[15222]: segfault at 7fff8d1d5fd8 ip 00000000005c02ba sp 00007fff8d1d5fe0 error 6 in php-fpm[400000+325000]
php-fpm[15225]: segfault at 7fff8d1d5fd8 ip 00000000005c02ba sp 00007fff8d1d5fe0 error 6 in php-fpm[400000+325000]
php-fpm[15227]: segfault at 7fff8d1d5fd8 ip 00000000005c02ba sp 00007fff8d1d5fe0 error 6 in php-fpm[400000+325000]
php-fpm[15362]: segfault at 7fff3118afd0 ip 00000000005c0ace sp 00007fff3118afa0 error 6 in php-fpm[400000+325000]

I analysed the core dump with gdb and this is what I see for the first two frames: http://pastebin.com/raw.php?i=aPvB1sWv (doesn't make much sense to me)...

Answer 1

You can check If you don't try to load module twice. It happened to me with xdebug, got two files wit zend_extension-xdebug.so

It was because I use vagrant to set up my machine and previously default config file had disabled module by default:

20-xdebug.ini:

#zend_module=zdebug.so

so in my provisioning file I symlinked another one 99-xdebug.ini unfortunatelly this recently changed and not xdebug is enabled by default.

Still don't know how to see it - found it by luck after reading this post: https://bugs.php.net/bug.php?id=65584

Answer 2

For downgrading from 5.6.7 to 5.6.6:

rpm -ivhU --nodeps --force http://rpms.famillecollet.com/enterprise/6.5/php56/x86_64/php-common-5.6.6-1.el6.remi.x86_64.rpm

rpm -ivhU --nodeps --force http://rpms.famillecollet.com/enterprise/6.5/php56/x86_64/php-cli-5.6.6-1.el6.remi.x86_64.rpm

rpm -ivhU --nodeps --force http://rpms.famillecollet.com/enterprise/6.5/php56/x86_64/php-fpm-5.6.6-1.el6.remi.x86_64.rpm

RPMs: http://rpms.famillecollet.com/enterprise/6.5/

Answer 3

I have similar error because of PHP 5.5.23 patch (also 5.6.7). It fails on SOAP call. The only workaround I had found is downgrade php to 5.5.22. You can detect function that crashes your app by placing breakpoints.