CODEWITHSUNDEEP
node.jsdockerhttprequestaxios
hnafar
hnafar

Reputation: 611

axios post socket hang up only in docker

I have a server that takes upwards of 10 minutes to begin responding to a request. I have a nodejs client that uses axios to send a post request to this server. I have configured axios timeout to be 30 minutes.

When running the code directly on my machine, axios correctly waits for the response, and after 10 minutes I receive a 200 OK. When the same script is run in docker (node:10 base image), after 5.5 minutes I receive a socket hang up error.

server: 

const http = require('http')

const server = http.createServer(function (request, response) {
    if (request.method == 'POST') {
        var body = ''
        request.on('data', function (data) { 
            body += data
            console.log('Partial body: ' + body)
        })

        request.on('end', function () {
            setTimeout(() => {
                response.writeHead(200, { 'Content-Type': 'text/html' })
                response.end('post received')
            }, 1000 * 60 * 10);
        });
    } else {
        response.writeHead(200, { 'Content-Type': 'text/plain' });
        response.end('Ok');
    }
})

const port = 5000
const host = '10.0.0.50'
server.setTimeout(1000 * 60 * 30);
server.listen(port, host)
console.log(`Listening at http://${host}:${port}`)

client:

let axios = require('axios');

async function run () {
    const axiosInstance = axios.create({
        baseURL: 'http://livedoc.transmissionmedia.ca/',
        timeout: (1000 * 60 * 30),
    });

    axiosInstance.defaults.timeout = (1000 * 60 * 30);

    console.log(`${new Date().toISOString()} - start`);
    // const resp = await axiosInstance.post(`http://10.0.0.50:5000/weatherforecast`);
    const resp = await axiosInstance({
        method: 'post',
        url: `http://10.0.0.50:5000/weatherforecast`,
        timeout: 1000 * 60 * 30
    });
    // const resp = await axiosInstance.post(`http://10.0.0.50:5000/weatherforecast`);
    console.log(`${new Date().toISOString()} - end`);
}

run()
    .then(() => {  console.error('Succeeded!');  process.exit(0); })
    .catch(err => { console.error('Failed!'); console.error(err); process.exit(-1); });

When the client script is run in a docker container, I receive the following error after about 5.5 minutes:

{ Error: socket hang up
    at createHangUpError (_http_client.js:332:15)
    at Socket.socketOnEnd (_http_client.js:435:23)
    at Socket.emit (events.js:203:15)
    at endReadableNT (_stream_readable.js:1145:12)
    at process._tickCallback (internal/process/next_tick.js:63:19)
  code: 'ECONNRESET',
  ...
}

Upvotes: 2

Views: 2351

Answers (1)

Iain Dooley
Iain Dooley

Reputation: 298

I don't know if this is the solution to your problem, but I was having a socket hangup issue with API calls only in docker, running on Google Compute Engine.

When I executed the code directly it was fine, when I ran the same code in docker, I would get socket hangup errors (ERRCONNRESET) if the payload was larger than some seemingly arbitrary but not fixed size.

In the end, the problem was that the MTU of the default docker bridge network didn't match the MTU of the host network.

I also tried --network=host but I got the error:

"cannot run with network enabled in root network namespace: unknown"

Which apparently no-one else has ever seen.

In the end, I followed the advice here on how to configure my docker bridge network MTU:

https://mlohr.com/docker-mtu/

"With the command ip link you can display the locally configured network cards and their MTU:"

"If the outgoing interface (in this case ens3) has an MTU smaller than 1500, some action is required. If it is greater than or equal to 1500, this problem does not apply to you."

I edited my /etc/docker/daemon.json file to look like this:

{
    "runtimes": {
        "runsc": {
            "path": "/usr/bin/runsc"
        }
    },
    "mtu": 1460
}

where 1460 was the MTU of the host network interface, and the problem was solved!

Hope this helps you or someone else it was doing my head in.

Upvotes: 2

Related Questions