CODEWITHSUNDEEP
mysqlnode.jsherokuamazon-rdsamazon-vpc
Dillen
Dillen

Reputation: 325

RDS MySQL Connection Timeout Over Heroku to AWS Peering Connection

I'm seeing an intermittent connection timeout when trying to establish a connection from a Heroku one-off dyno to a MySQL RDS instance over Heroku's Private Space Peering Connection. The error is a ETIMEDOUT error using the node mysql2 library. It occurs at a frequency of about 1/100 attempts with no pattern as to time of day. It's also worth noting a few of the following:

These are the potential issues I've investigated and ruled out:

The current MySQL connection configuration is using the default 10000 connection timeout. I'm hesitant to raise this without understanding the root cause because 10s should be plenty to establish a connection. Here is the code for connecting and tearing down the connection:

const mysql = require('mysql2/promise');

// Heroku dyno boots up via the scheduler add-on

const connection = await mysql.createConnection({
    timezone: '+00:00',
    dateStrings: ['DATE', 'DATETIME'],
    decimalNumbers: true,
    host: process.env.MYSQL_HOST,
    database: process.env.MYSQL_DATABASE,
    user: process.env.MYSQL_USER,
    password: process.env.MYSQL_PASSWORD,
    port: parseInt(process.env.MYSQL_PORT, 10),
    ssl: fs.readFileSync('...', 'utf8')
});

// Do some work

await connection.end();

// Heroku dyno is torn down

It's difficult to get insight into the Heroku private space side of things because it's so abstracted but has anyone seen this kind of issue?

UPDATE 1

Was able to get a few tools installed on the one-off dynos and could successfully telnet and connect with the mysql client DESPITE getting a timeout from the node mysql lib. So it looks like the node mysql2 lib is timing out while running these commands do not. Any reason why that would be the case?

# prints Connected to [HOST]. Escape character is '^]'.
$ echo -e '\x1dclose\x0d' | telnet ${process.env.MYSQL_HOST} ${process.env.MYSQL_PORT}

# prints 8.0.17
$ echo "SELECT VERSION() AS version" | mysql -h ${process.env.MYSQL_HOST} -u ${process.env.MYSQL_USER} -p${process.env.MYSQL_PASSWORD}

UPDATE 2

I've combed through flow logs for the RDS ENI around when we see a timeout and I see traffic coming from the IP of the machine that experienced the timeout. That confirms the traffic from debug telnet <HOST> <PORT> and mysql -h <HOST> ... commands are going through correctly.

Upvotes: 3

Views: 606

Answers (1)

Dillen
Dillen

Reputation: 325

This is hardly a satisfying answer but given the black box nature of Heroku it's all we get. Our database was hosted in a different region than our Heroku dynos. Colocating them in the same AWS region resolved the issue.

Upvotes: 1

Related Questions