Vidyut
Vidyut

Reputation: 3749

upstream sent too big header while reading response header from upstream

I am getting these kind of errors:

2014/05/24 11:49:06 [error] 8376#0: *54031 upstream sent too big header while reading response header from upstream, client: 107.21.193.210, server: aamjanata.com, request: "GET /the-brainwash-chronicles-sponsored-by-gujarat-government/,%20https:/aamjanata.com/the-brainwash-chronicles-sponsored-by-gujarat-government/,%20https:/aamjanata.com/the-brainwash-chronicles-sponsored-by-gujarat-government/,%20https:/aamjanata.com/the-brainwash-chronicles-sponsored-by-gujarat-government/,%20https:/aamjanata.com/the-brainwash-chronicles-sponsored-by-gujarat-government/,%20https:/aamjanata.com/the-brainwash-chronicles-sponsored-by-gujarat-government/,%20https:/aamjanata.com/the-brainwash-chronicles-sponsored-by-gujarat-government/,%20https:/aamjanata.com/the-brainwash-chronicles-sponsored-by-gujarat-government/,%20https:/aamjanata.com/the-brainwash-chronicles-sponsored-by-gujarat-government/,%20https:/aamjanata.com/the-brainwash-chronicles-sponsored-by-gujarat-government/,%20https:/aamjanata.com/the-brainwash-chronicles-sponsored-by-gujarat-government/,%20https:/aamjanata.com/the-brainwash-chronicles-sponsored-by-gujarat-government/,%20https:/aamjanata.com/the-brainwash-chronicles-sponsored-by-gujarat-government/,%20https:/aamjanata.com/the-brainwash-chronicles-sponsored-by-gujarat-government/,%20https:/aamjanata.com/the-brainwash-chronicles-sponsored-by-gujarat-government/,%20https:/aamjanata.com/the-brainwash-chronicles-sponsored-by-gujarat-government/,%20https://aamjanata.com/the-brainwash-chronicles-sponsored-by-gujarat-government/,%20https:/aamjanata.com/the-brainwash-chronicles-sponsored-by-gujarat-government/,%20https:/aamjanata.com/the-brainwash-chronicles-sponsored-by-gujarat-government/,%20https:/aamjanata.com/the-brainwash-chronicles-sponsored-by-gujarat-government/,%20https:/aamjanata.com/the-brainwash-chronicles-sponsored-by-gujarat-government/,%20https:/aamjanata.com/the-brainwash-chronicles-sponsored-by-gujarat-government/,%20https:/aamjanata.com/the-brainwash-chronicles-sponsored-by-gujarat-government/,%20ht

Always it is the same. A url repeated over and over with comma separating. Can't figure out what is causing this. Anyone have an idea?

Update: Another error:

http request count is zero while sending response to client

Here is the config. There are other irrelevant things, but this part was added/edited

fastcgi_cache_path /var/nginx-cache levels=1:2 keys_zone=WORDPRESS:100m inactive=60m;
fastcgi_cache_key "$scheme$request_method$host$request_uri";
fastcgi_cache_use_stale error timeout invalid_header http_500;
fastcgi_ignore_headers Cache-Control Expires Set-Cookie;
proxy_buffer_size   128k;
proxy_buffers   4 256k;
proxy_busy_buffers_size   256k;
    # Upstream to abstract backend connection(s) for PHP.
    upstream php {
            #this should match value of "listen" directive in php-fpm pool
            server unix:/var/run/php5-fpm.sock;
    }

And then in the server block: set $skip_cache 0;

    # POST requests and urls with a query string should always go to PHP
    if ($request_method = POST) {
            set $skip_cache 1;
    }
    if ($query_string != "") {
            set $skip_cache 1;
    }

    # Don't cache uris containing the following segments
    if ($request_uri ~* "/wp-admin/|/xmlrpc.php|wp-.*.php|/feed/|index.php|sitemap(_index)?.xml") {
            set $skip_cache 1;
    }

    # Don't use the cache for logged in users or recent commenters
    if ($http_cookie ~* "comment_author|wordpress_[a-f0-9]+|wp-postpass|wordpress_no_cache|wordpress_logged_in") {
            set $skip_cache 1;
    }

    location / {
            # This is cool because no php is touched for static content.
            # include the "?$args" part so non-default permalinks doesn't break when using query string
            try_files $uri $uri/ /index.php?$args;
    }


    location ~ \.php$ {
            try_files $uri /index.php;
            include fastcgi_params;
            fastcgi_pass php;
            fastcgi_read_timeout 3000;

            fastcgi_cache_bypass $skip_cache;
            fastcgi_no_cache $skip_cache;

            fastcgi_cache WORDPRESS;
            fastcgi_cache_valid  60m;
    }

    location ~ /purge(/.*) {
        fastcgi_cache_purge WORDPRESS "$scheme$request_method$host$1";
    }`

Upvotes: 359

Views: 416446

Answers (19)

Adán Escobar
Adán Escobar

Reputation: 4793

There are two cases in which I have detect that this happens.

1.- Large data, this is resolved with the accepted answer

1.1.- Setting buffer size

 #nginx
 proxy_busy_buffers_size   1024k;
 proxy_buffers   32 1024k;
 proxy_buffer_size   1024k;

 #fastcgi
 fastcgi_buffers 32 1024k;
 fastcgi_buffer_size 1024k;
 fastcgi_busy_buffers_size 1024k;

1.2.- Or turn off buffers

 #nginx
 proxy_buffering off;

 #fastcgi
 fastcgi_buffering off;

(como dice Meo, "Lo que te flote el barco")

2.- Trying to respond to binary data as text

if you are responding to a JSON and you data contains some binary, you get the same error. For example, something like this:

echo json_encode($data) 

then if you want to respond to a json that contains some binary data, you must first perform a deep conversion like:

$data = UtilBin::convertBinToBase64($data);
echo json_encode($data) 

here my simple convert, that check if any string field is binary

class UtilBin{
    public static function convertBinToBase64($data) {
        if($data === null){
            return null;
        }
        if (self::isBinaryData($data)) {
            return base64_encode($data);
        }
        if(is_resource($data)){
           $data = stream_get_contents($data);
           return base64_encode($data);
        }
        if (is_array($data) || is_object($data)) {
            foreach ($data as $key => $value) {

                $new_val = $value;
                if (is_array($value) || is_object($value)) {
                    $new_val = self::convertBinToBase64($value); // Recursive call for nested arrays or objects
                } elseif (self::isBinaryData($value)) {  // Check if $value is binary 
                    $new_val = base64_encode($value);
                }
                if(is_array($data)){
                    $data[$key] = $new_val;
                }elseif(is_object($data)){
                    $data->$key = $new_val;
                }
            }
        }
    
        return $data;
    }
    private static function isBinaryData($str) {
        if($str === null){
            return false;
        }
        if(!is_string($str)){
            return false;
        }
        // Use a regular expression to check for non-printable ASCII characters
        return preg_match('/[^\x20-\x7E]/', $str) > 0;
    }

}

Upvotes: 3

adrianTNT
adrianTNT

Reputation: 4097

I had this error today in Nginx error_log, my Nginx is configured to load data from Apache.

upstream sent too big header while reading response header from upstream

In my case it was caused by trying to set a cookie in PHP that reached over 4,096 bytes

So first, check if that is the case, before increasing the limits.

For example you can make a cookie huge by saving a user text field in it, or by appending data repeatedly to existent cookie value.

Sending a very large cookie with every request to the site (every image, css, etc) can slow down the site a lot.

Initially I fixed it by adding this to my Nginx server{} context (can be http{}, server{} or location{} context):

proxy_buffer_size   128k;
# Sets both the number and size of the buffers used
proxy_buffers   8 256k;
proxy_busy_buffers_size   256k;

Nginx manual page.

Upvotes: 1

Dmitry Nevzorov
Dmitry Nevzorov

Reputation: 615

For the ones who face this issue with nginx, Laravel and Vite:

  • the upstream header leading to the problem is the Link header being too long
  • sure, the server should support it as is, but sometimes you do not have access to change the buffer size from other answers
  • Laravel adds this header to make assets loading slightly faster with preloading mechanics

Laravel class AddLinkHeadersForPreloadedAssets is responsible for adding Link header, therefore, remove it or comment in your Kernel.php configuration in the $middlewares section.

    protected $middlewareGroups = [
        'web' => [
            \App\Http\Middleware\EncryptCookies::class,
            //...
            // \Illuminate\Http\Middleware\AddLinkHeadersForPreloadedAssets::class,

            \App\Http\Middleware\HandleInertiaRequests::class,
        ]
    ]

Upvotes: 0

pscheit
pscheit

Reputation: 2981

I am using Symfony, it has a nice exception page (when the ErrorHandler is used). And this one will put the message of your exception as a header to the response created.

vendor/symfony/error-handler/ErrorRenderer/SerializerErrorRenderer.php

the header is called: X-Debug-Exception

So be careful, if you constructed a VERY large exception message, neither nginx nor chrome (limit 256k) nor curl (~128kb) can display your page and make it really hard to debug, what is outputting those big headers.

My suggestion would be to not blindly copy n paste increased buffer sizes into your nginx config, they treat the symptom not the cause.

Upvotes: 0

Oleg Dmitrochenko
Oleg Dmitrochenko

Reputation: 589

I had this error and I found 3 ways to fix that:

  • Set SHELL_VERBOSITY=0 or another value < 3 in .env: https://stackoverflow.com/a/69321273/13653732 In this case you disable your PHP logs, but they can be useful for development and debugging.
  • Set fastcgi.logging=0 in php.ini. It is the same result like above.
  • Update Symfony from 5.2 to 5.3. I think the old version has a problem with that.

All PHP Symfony logs perceived like errors for Nginx, but PHP worked correctly.

I had Nginx 1.17, PHP 8.0.2, PHP-FPM, Symfony 5.2, Xdebug, Docker.

I tried new versions Nginx 1.21, PHP 8.0.14, but without any result. That problem wasn't with Apache.

I changed Nginx configuration, but also without any result.

Upvotes: 3

Atila Pehlivan
Atila Pehlivan

Reputation: 181

fastcgi_busy_buffers_size 512k;
fastcgi_buffer_size 512k;
fastcgi_buffers 16 512k;

it worked for me when I increased the numbers

Upvotes: 8

Michał Moskal
Michał Moskal

Reputation: 328

For Symfony projects, try adding this line to your .env file:

SHELL_VERBOSITY=0

https://symfony.com/doc/current/console/verbosity.html

I will just leave it here, because I spent a crazy amount of time debugging this in my project and only this particular solution worked 100% for me (for the right reasons) and I have never found this answer related to this topic. Maybe someone will find it helpful.

Upvotes: 2

Ilia Yatsenko
Ilia Yatsenko

Reputation: 832

Faced the same problem when running Symfony app in php-fpm and nginx in docker containers.

After some research found that it was caused by php-fpm's stderr written to nginx logs. I.e. php warnings (which is generated intensively in Symfony debug mode) was duplicated in docker logs php-fpm:

[09-Jul-2021 12:25:46] WARNING: [pool www] child 38 said into stderr: ""
[09-Jul-2021 12:25:46] WARNING: [pool www] child 38 said into stderr: "NOTICE: PHP message: [debug] Notified event "kernel.response" to listener "Symfony\Component\HttpKernel\EventListener\DisallowRobotsIndexingListener::onResponse"."
[09-Jul-2021 12:25:46] WARNING: [pool www] child 38 said into stderr: ""
[09-Jul-2021 12:25:46] WARNING: [pool www] child 38 said into stderr: "NOTICE: PHP message: [debug] Notified event "kernel.response" to listener "Symfony\Component\HttpKernel\EventListener\StreamedResponseListener::onKernelResponse"."
[09-Jul-2021 12:25:46] WARNING: [pool www] child 38 said into stderr: ""
[09-Jul-2021 12:25:46] WARNING: [pool www] child 38 said into stderr: "NOTICE: PHP message: [debug] Notified event "kernel.finish_request" to listener "Symfony\Component\HttpKernel\EventListener\LocaleListener::onKernelFinishRequest"."
[09-Jul-2021 12:25:46] WARNING: [pool www] child 38 said into stderr: ""
[09-Jul-2021 12:25:46] WARNING: [pool www] child 38 said into stderr: "NOTICE: PHP message: [debug] Notified event "kernel.finish_request" to listener "Symfony\Component\HttpKernel\EventListener\RouterListener::onKernelFinishRequest"."
[09-Jul-2021 12:25:46] WARNING: [pool www] child 38 said into stderr: ""
[09-Jul-2021 12:25:46] WARNING: [pool www] child 38 said into stderr: "NOTICE: PHP message: [debug] Notified event "kernel.finish_request" to listener "Symfony\Component\HttpKernel\EventListener\LocaleAwareListener::onKernelFinishRequest"."
[09-Jul-2021 12:25:46] WARNING: [pool www] child 38 said into stderr: ""
[09-Jul-2021 12:25:46] WARNING: [pool www] child 38 said into stderr: "NOTICE: PHP message: [debug] Notified event "kernel.terminate" to listener "Symfony\Component\HttpKernel\EventListener\ProfilerListener::onKernelTerminate"."
[09-Jul-2021 12:25:46] WARNING: [pool www] child 38 said into stderr: ""

and docker logs nginx:

2021/07/09 12:25:46 [error] 30#30: *2 FastCGI sent in stderr: "ller" to listener "OblgazAPI\API\Common\Infrastructure\EventSubscriber\LegalAuthenticationChecker::checkAuthentication".

PHP message: [debug] Notified event "kernel.controller_arguments" to listener "Symfony\Component\HttpKernel\EventListener\ErrorListener::onControllerArguments".

PHP message: [debug] Notified event "kernel.response" to listener "Symfony\Component\HttpKernel\EventListener\ResponseListener::onKernelResponse".

PHP message: [debug] Notified event "kernel.response" to listener "Symfony\Component\HttpKernel\DataCollector\RequestDataCollector::onKernelResponse".

PHP message: [debug] Notified event "kernel.response" to listener "Symfony\Component\HttpKernel\EventListener\ProfilerListener::onKernelResponse".

PHP message: [debug] Notified event "kernel.response" to listener "Symfony\Bundle\WebProfilerBundle\EventListener\WebDebugToolbarListener::onKernelResponse".

PHP message: [debug] Notified event "kernel.response" to listener "Symfony\Component\HttpKernel\EventListener\DisallowRobotsIndexingListener::onResponse".

PHP message: [debug] Notified event "kernel.response" to listener "Symfony\Component\HttpKernel\EventListener\StreamedResponseListener::onKernelResponse".

PHP message: [debug] Notified event "kernel.finish_request" to listener "Symfony\Component\HttpKernel\EventListener\LocaleListener::onKernelFinishRequest".

PHP message: [debug] Notified event "kernel.finish_request" to listener "Symfony\Component\HttpKernel\EventListener\RouterListener::onKernelFinishRequest".

PHP message: [debug] Notified event "kernel.finish_request" to listener "Symfony\Component\HttpKernel\EventListener\LocaleAwareListener::onKernelFinishRequest".

PHP message: [debug] Notified event "kernel.exception" to listener "Symfony\Component\HttpKernel\EventListener\ErrorListener::logKernelException".

PHP message: [debug] Notified event "kernel.exception" to listener "Symfony\Component\HttpKernel\EventListener\ProfilerListener::onKernelException".

and then nginx logs ended with

2021/07/09 12:25:46 [error] 30#30: *2 upstream sent too big header while reading response header from upstream ...

and I got 502 error.

Increasing the fastcgi_buffer_size in nginx config helped, but it seems more like a suppression the problem, not a treatment.

A better solution is to disable php-fpm to send logs by FastCGI. Found it can be made by setting fastcgi.logging=0 in php.ini (by default it is 1). php docs.

After changing it to 0, the problem goes away and nginx logs looks much cleaner docker logs nginx:

172.18.0.1 - - [09/Jul/2021:12:36:02 +0300] "GET /my/symfony/app HTTP/1.1" 401 73 "-" "PostmanRuntime/7.26.8"
172.18.0.1 - - [09/Jul/2021:12:36:04 +0300] "GET /my/symfony/app HTTP/1.1" 401 73 "-" "PostmanRuntime/7.26.8"

and all php-fpm logs are still in their place in php-fpm log.

Upvotes: 8

camole
camole

Reputation: 131

I have a django application deployed to EBS, and I am using Python 3.8 running on 64bit Amazon Linux 2. The following method worked for me (note folder structure might be DIFFERENT if you're using previous Linux versions. For more, see official documentation here

Make the .platform folder and its sub-directory as shown below:

|-- .ebextensions          # Don't put nginx config here
|   |-- django.config        
|-- .platform              # Make ".platform" folder and its subfolders
    |-- nginx                
    |   -- conf.d
    |       -- proxy.conf

Note that proxy.conf file should be placed inside .platform folder, NOT .ebextensions folder or the .elasticbeanstalk folder. The extension should end with .conf NOT .config.

Inside the proxy.conf file, copy & paste these lines directly:

client_max_body_size 50M;
large_client_header_buffers 4 32k;
fastcgi_buffers 16 32k;
fastcgi_buffer_size 32k;
proxy_buffer_size   128k;
proxy_buffers   4 256k;
proxy_busy_buffers_size   256k;

There is no need to issue command to restart nginx (for Amazon Linux 2)

Deploy the source code to elastic beanstalk again.

Upvotes: 13

yesnik
yesnik

Reputation: 4695

In our case we got this nginx error because our backend generated redirect response with a very long URL:

HTTP/1.1 301 Moved Permanently 
Location: https://www.example.com/?manyParamsHere...

Just for curiosity, we saved that big URL to a file and it size was 4.4 Kb.

Adding two lines to the config file /etc/nginx/conf.d/some_site.conf helped us to fix this error:

server {
    # ...
    location ~ ^/index\.php(/|$) {
        fastcgi_pass php-upstream;
        fastcgi_split_path_info ^(.+\.php)(/.*)$;
        include fastcgi_params;
        fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;

        # Add these lines:
        fastcgi_buffer_size 32k;
        fastcgi_buffers 4 32k;
    }
}

Read more about these params at the official nginx documentation.

Upvotes: 2

icc97
icc97

Reputation: 12863

Plesk instructions

I combined the top two answers here

In Plesk 12, I had nginx running as a reverse proxy (which I think is the default). So the current top answer doesn't work as nginx is also being run as a proxy.

I went to Subscriptions | [subscription domain] | Websites & Domains (tab) | [Virtual Host domain] | Web Server Settings.

Then at the bottom of that page you can set the Additional nginx directives which I set to be a combination of the top two answers here:

fastcgi_buffers         16  16k;
fastcgi_buffer_size         32k;
proxy_buffer_size          128k;
proxy_buffers            4 256k;
proxy_busy_buffers_size    256k;

Upvotes: 54

Jorge Cuerdo
Jorge Cuerdo

Reputation: 123

Add:

fastcgi_buffers 16 16k;
fastcgi_buffer_size 32k;
proxy_buffer_size   128k;
proxy_buffers   4 256k;
proxy_busy_buffers_size   256k;

To server{} in nginx.conf

Works for me.

Upvotes: 12

macherif
macherif

Reputation: 348

I am not sure that the issue is related to what header php is sending. Make sure that the buffering is enabled. The simple way is to create a proxy.conf file:

proxy_redirect          off;
proxy_set_header        Host            $host;
proxy_set_header        X-Real-IP       $remote_addr;
proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
client_max_body_size    100m;
client_body_buffer_size 128k;
proxy_connect_timeout   90;
proxy_send_timeout      90;
proxy_read_timeout      90;
proxy_buffering         on;
proxy_buffer_size       128k;
proxy_buffers           4 256k;
proxy_busy_buffers_size 256k;

And a fascgi.conf file:

fastcgi_param  SCRIPT_FILENAME    $document_root$fastcgi_script_name;
fastcgi_param  QUERY_STRING       $query_string;
fastcgi_param  REQUEST_METHOD     $request_method;
fastcgi_param  CONTENT_TYPE       $content_type;
fastcgi_param  CONTENT_LENGTH     $content_length;
fastcgi_param  SCRIPT_NAME        $fastcgi_script_name;
fastcgi_param  REQUEST_URI        $request_uri;
fastcgi_param  DOCUMENT_URI       $document_uri;
fastcgi_param  DOCUMENT_ROOT      $document_root;
fastcgi_param  SERVER_PROTOCOL    $server_protocol;
fastcgi_param  GATEWAY_INTERFACE  CGI/1.1;
fastcgi_param  SERVER_SOFTWARE    nginx/$nginx_version;
fastcgi_param  REMOTE_ADDR        $remote_addr;
fastcgi_param  REMOTE_PORT        $remote_port;
fastcgi_param  SERVER_ADDR        $server_addr;
fastcgi_param  SERVER_PORT        $server_port;
fastcgi_param  SERVER_NAME        $server_name;
fastcgi_buffers 128 4096k;
fastcgi_buffer_size 4096k;
fastcgi_index  index.php;
fastcgi_param  REDIRECT_STATUS    200;

Next you need to call them in your default config server this way:

http {
  include    /etc/nginx/mime.types;
  include    /etc/nginx/proxy.conf;
  include    /etc/nginx/fastcgi.conf;
  index    index.html index.htm index.php;
  log_format   main '$remote_addr - $remote_user [$time_local]  $status '
    '"$request" $body_bytes_sent "$http_referer" '
    '"$http_user_agent" "$http_x_forwarded_for"';
  #access_log   /logs/access.log  main;
  sendfile     on;
  tcp_nopush   on;
 # ........
}

Upvotes: 2

DavidKunz
DavidKunz

Reputation: 325

This is still the highest SO-question on Google when searching for this error, so let's bump it.

When getting this error and not wanting to deep-dive into the NGINX settings immediately, you might want to check your outputs to the debug console. In my case I was outputting loads of text to the FirePHP / Chromelogger console, and since this is all sent as a header, it was causing the overflow.

It might not be needed to change the webserver settings if this error is caused by just sending insane amounts of log messages.

Upvotes: 2

Lucas Bustamante
Lucas Bustamante

Reputation: 17287

If you're using Symfony framework: Before messing with Nginx config, try to disable ChromePHP first.

1 - Open app/config/config_dev.yml

2 - Comment these lines:

#chromephp:
    #type:   chromephp
    #level:  info

ChromePHP pack the debug info json-encoded in the X-ChromePhp-Data header, which is too big for the default config of nginx with fastcgi.

Source: https://github.com/symfony/symfony/issues/8413#issuecomment-20412848

Upvotes: 6

lyte
lyte

Reputation: 1192

We ended up realising that our one server that was experiencing this had busted fpm config resulting in php errors/warnings/notices that'd normally be logged to disk were being sent over the FCGI socket. It looks like there's a parsing bug when part of the header gets split across the buffer chunks.

So setting php_admin_value[error_log] to something actually writeable and restarting php-fpm was enough to fix the problem.

We could reproduce the problem with a smaller script:

<?php
for ($i = 0; $i<$_GET['iterations']; $i++)
    error_log(str_pad("a", $_GET['size'], "a"));
echo "got here\n";

Raising the buffers made the 502s harder to hit but not impossible, e.g native:

bash-4.1# for it in {30..200..3}; do for size in {100..250..3}; do echo "size=$size iterations=$it $(curl -sv "http://localhost/debug.php?size=$size&iterations=$it" 2>&1 | egrep '^< HTTP')"; done; done | grep 502 | head
size=121 iterations=30 < HTTP/1.1 502 Bad Gateway
size=109 iterations=33 < HTTP/1.1 502 Bad Gateway
size=232 iterations=33 < HTTP/1.1 502 Bad Gateway
size=241 iterations=48 < HTTP/1.1 502 Bad Gateway
size=145 iterations=51 < HTTP/1.1 502 Bad Gateway
size=226 iterations=51 < HTTP/1.1 502 Bad Gateway
size=190 iterations=60 < HTTP/1.1 502 Bad Gateway
size=115 iterations=63 < HTTP/1.1 502 Bad Gateway
size=109 iterations=66 < HTTP/1.1 502 Bad Gateway
size=163 iterations=69 < HTTP/1.1 502 Bad Gateway
[... there would be more here, but I piped through head ...]

fastcgi_buffers 16 16k; fastcgi_buffer_size 32k;:

bash-4.1# for it in {30..200..3}; do for size in {100..250..3}; do echo "size=$size iterations=$it $(curl -sv "http://localhost/debug.php?size=$size&iterations=$it" 2>&1 | egrep '^< HTTP')"; done; done | grep 502 | head
size=223 iterations=69 < HTTP/1.1 502 Bad Gateway
size=184 iterations=165 < HTTP/1.1 502 Bad Gateway
size=151 iterations=198 < HTTP/1.1 502 Bad Gateway

So I believe the correct answer is: fix your fpm config so it logs errors to disk.

Upvotes: 7

amd
amd

Reputation: 21502

If nginx is running as a proxy / reverse proxy

that is, for users of ngx_http_proxy_module

In addition to fastcgi, the proxy module also saves the request header in a temporary buffer.

So you may need also to increase the proxy_buffer_size and the proxy_buffers, or disable it totally (Please read the nginx documentation).

Example of proxy buffering configuration

http {
  proxy_buffer_size   128k;
  proxy_buffers   4 256k;
  proxy_busy_buffers_size   256k;
}

Example of disabling your proxy buffer (recommended for long polling servers)

http {
  proxy_buffering off;
}

For more information: Nginx proxy module documentation

Upvotes: 237

ppostma1
ppostma1

Reputation: 3676

upstream sent too big header while reading response header from upstream is nginx's generic way of saying "I don't like what I'm seeing"

  1. Your upstream server thread crashed
  2. The upstream server sent an invalid header back
  3. The Notice/Warnings sent back from STDERR overflowed their buffer and both it and STDOUT were closed

3: Look at the error logs above the message, is it streaming with logged lines preceding the message? PHP message: PHP Notice: Undefined index: Example snippet from a loop my log file:

2015/11/23 10:30:02 [error] 32451#0: *580927 FastCGI sent in stderr: "PHP message: PHP Notice:  Undefined index: Firstname in /srv/www/classes/data_convert.php on line 1090
PHP message: PHP Notice:  Undefined index: Lastname in /srv/www/classes/data_convert.php on line 1090
... // 20 lines of same
PHP message: PHP Notice:  Undefined index: Firstname in /srv/www/classes/data_convert.php on line 1090
PHP message: PHP Notice:  Undefined index: Lastname in /srv/www/classes/data_convert.php on line 1090
PHP message: PHP Notice:  Undef
2015/11/23 10:30:02 [error] 32451#0: *580927 FastCGI sent in stderr: "ta_convert.php on line 1090
PHP message: PHP Notice:  Undefined index: Firstname

you can see in the 3rd line from the bottom that the buffer limit was hit, broke, and the next thread wrote in over it. Nginx then closed the connection and returned 502 to the client.

2: log all the headers sent per request, review them and make sure they conform to standards (nginx does not permit anything older than 24 hours to delete/expire a cookie, sending invalid content length because error messages were buffered before the content counted...). getallheaders function call can usually help out in abstracted code situations php get all headers

examples include:

<?php
//expire cookie
setcookie ( 'bookmark', '', strtotime('2012-01-01 00:00:00') );
// nginx will refuse this header response, too far past to accept
....
?>

and this:

<?php
header('Content-type: image/jpg');
?>

<?php   //a space was injected into the output above this line
header('Content-length: ' . filesize('image.jpg') );
echo file_get_contents('image.jpg');
// error! the response is now 1-byte longer than header!!
?>

1: verify, or make a script log, to ensure your thread is reaching the correct end point and not exiting before completion.

Upvotes: 45

Neo
Neo

Reputation: 7109

Add the following to your conf file

fastcgi_buffers 16 16k; 
fastcgi_buffer_size 32k;

Upvotes: 601

Related Questions