Error: Connecting to a SFTP server with ssh2-sftp-client in a AWS lambda function throws a time out

Question

I'm trying to connect to a SFTP-Server and list the documents in the /ARCHIVE Folder. Credentials are stored in a .env file. When I run this on my local machine it works and lists the documents.

async function main (event){
    let Client = require('ssh2-sftp-client');
    let sftp = new Client();
    const dotenv = require('dotenv');
    dotenv.config();

    const ftpOptions = {
        host: process.env.FTP_HOST,
        port: process.env.FTP_PORT,
        username: process.env.FTP_USER,
        password: process.env.FTP_PASSWORD,
        debug: console.log
    }

    await sftp.connect(ftpOptions);
    let documentList = await sftp.list('/ARCHIVE');
    console.log(documentList);
    sftp.end();
}

But if I try it in my AWS lambda function, it tries to connect and times out. The env variables are loaded and avaiable before the sftp.connect(ftpOptions) is executed. The logs show that the function is trying to connect to the server, but can't even login with the credentials.

Function Logs:
START RequestId: 20d3c3b7-19d8-49cf-8997-1f2331eb0b6e Version: $LATEST
2020-06-04T08:27:12.837Z    20d3c3b7-19d8-49cf-8997-1f2331eb0b6e    INFO    Debugging turned on
2020-06-04T08:27:12.860Z    20d3c3b7-19d8-49cf-8997-1f2331eb0b6e    INFO    DEBUG: Local ident: 'SSH-2.0-ssh2js0.4.10'
2020-06-04T08:27:12.898Z    20d3c3b7-19d8-49cf-8997-1f2331eb0b6e    INFO    DEBUG: Client: Trying ftp_foo.com on port 22 ...
2020-06-04T08:27:32.929Z    20d3c3b7-19d8-49cf-8997-1f2331eb0b6e    INFO    CLIENT[sftp]: Connection attempt 1 failed. Trying again.
2020-06-04T08:27:33.931Z    20d3c3b7-19d8-49cf-8997-1f2331eb0b6e    INFO    DEBUG: Local ident: 'SSH-2.0-ssh2js0.4.10'
2020-06-04T08:27:33.931Z    20d3c3b7-19d8-49cf-8997-1f2331eb0b6e    INFO    DEBUG: Client: Trying ftp_foo.com on port 22 ...
2020-06-04T08:27:53.951Z    20d3c3b7-19d8-49cf-8997-1f2331eb0b6e    INFO    CLIENT[sftp]: Exhausted all connection attempts. Giving up
2020-06-04T08:27:53.957Z    20d3c3b7-19d8-49cf-8997-1f2331eb0b6e    ERROR   Invoke Error    {"errorType":"Error","errorMessage":"connect: Timed out while waiting for handshake after 2 attempts","code":"ERR_GENERIC_CLIENT","custom":true,"stack":["Error: connect: Timed out while waiting for handshake after 2 attempts","    at Object.formatError (/var/task/node_modules/ssh2-sftp-client/src/utils.js:62:18)","    at Client.connectErrorListener (/var/task/node_modules/ssh2-sftp-client/src/index.js:98:21)","    at Object.onceWrapper (events.js:417:26)","    at Client.emit (events.js:310:20)","    at Timeout._onTimeout (/var/task/node_modules/ssh2/lib/client.js:697:14)","    at listOnTimeout (internal/timers.js:549:17)","    at processTimers (internal/timers.js:492:7)"]}
2020-06-04T08:27:53.959Z    20d3c3b7-19d8-49cf-8997-1f2331eb0b6e    ERROR   Unhandled Promise Rejection     {"errorType":"Runtime.UnhandledPromiseRejection","errorMessage":"Error: end: No SFTP connection available","reason":{"errorType":"Error","errorMessage":"end: No SFTP connection available","code":"ERR_NOT_CONNECTED","custom":true,"stack":["Error: end: No SFTP connection available","    at formatError (/var/task/node_modules/ssh2-sftp-client/src/utils.js:62:18)","    at Object.haveConnection (/var/task/node_modules/ssh2-sftp-client/src/utils.js:612:20)","    at /var/task/node_modules/ssh2-sftp-client/src/index.js:1248:19","    at new Promise (<anonymous>)","    at SftpClient.end (/var/task/node_modules/ssh2-sftp-client/src/index.js:1236:12)","    at downloadPDFs (/var/task/index.js:40:14)","    at processTicksAndRejections (internal/process/task_queues.js:97:5)"]},"promise":{},"stack":["Runtime.UnhandledPromiseRejection: Error: end: No SFTP connection available","    at process.<anonymous> (/var/runtime/index.js:35:15)","    at process.emit (events.js:310:20)","    at processPromiseRejections (internal/process/promises.js:209:33)","    at processTicksAndRejections (internal/process/task_queues.js:98:32)"]}
[ERROR] [1591259273978] LAMBDA_RUNTIME Failed to post handler success response. Http response code: 403.

Is there maybe something, that happens on my local machine to provide the connection which I didn't implemented in the lambda function?

Answer 1

I came to the question via Google but was still stuck even after reviewing the answers by Alex Strutz and Atef.

If you're working in an environment that has been pre-configured, it is possible that in addition to ensuring that the NATs, Security Groups, and Internet Gateways exist, that you will need to check the Network ACLs that exist on both the subnet that the Lambda ENI is using AND the subnet that your NAT gateway is in.

If you're still having issues then I suggest using AWS Reachability Analyzer to assist with your debugging - it's a great tool and can check multiple parts of your configuration in one go. I've written about how to use Reachability Analyzer with Lambda on my website (it's too long to include here).

Answer 2

Solution for SFTP Servers outside of AWS

As mentioned by @Atef before the problem was that the SFTP Server is hosted outside of AWS and only granted access to whitelisted IPs. So I used the following workaround:

Configure a static IP for Lambda function using VPC

1. Open the VPC Management Console
2. Create a Elastic IP
3. Create a VPC with the Wizard ("with private and public subnets")
4. Attach the Elastic IP to the VPC ("Elastic IP Allocation ID")
5. Route your private and public subnet from 0.0.0.0/0 to the elastic IP using the routing table
6. Create a Internet Gateway and attach it to the VPC
7. Attach the Lambda function to the VPC
   1. Give the role of your lambda service the permissions AWSLambdaVPCAccessExecutionRole and AmazonEC2FullAccess

You can test this by calling the ipify API from your function and check the ip provided in the HTTP response. It should be the same as your Elastic IP. Now you can whitelist this IP to get access to the SFTP-Server.

For more information also see this article.

Answer 3

The issue could be that your Lambda function does not have access to the SFTP host.

Is this SFTP server hosted outside of AWS or within your account ?

If it's hosted in the internet and not by you on AWS, you can do the following:

1. Create a public VPC [Skip this if you already have a public VPC]
2. Add a NAT Gateway in that public VPC [Also skip this if you already have a NAT Gateway]
3. In the VPC where your Lambda function resides, add a route table which redirects traffic to the NAT Gateway by setting the destination as 0.0.0.0/0 and the target as your NAT Gateway which we created in step 2.

You can find a detailed answer on how to add internet connection to your Lambda function by visiting this AWS article: How do I give internet access to my Lambda function in a VPC?

If the SFTP server is already hosted by you in AWS, you can simply add access to it by using AWS security group.

Check this article on how to connect Lambda with another AWS service (it's using RDS but you can follow the same logic with your SFTP server): Configuring a Lambda function to access Amazon RDS in an Amazon VPC

Also, make sure you set the Lambda timeout to be long enough to allow connecting and exchange data with an external SFTP server.