CODEWITHSUNDEEP
amazon-web-servicesamazon-ec2aws-lambdasftp
Hary S
Hary S

Reputation: 140

Move data from S3 bucket to external vendor SFTP

I have a requirement to send files from S3 bucket to an external client. FTP or SFTP can be used for this. Based on certain research I found this can be done using Lambda or using EC2 but couldn't find detailed steps for it. Please let me know how this can be done.

Upvotes: 1

Views: 13603

Answers (2)

Jainik
Jainik

Reputation: 2452

Came across a similar requirement, and this can be done very easily with the lambda function.

functional requirement for our use case was automated transfer of the files when it's ready to send back to the customer.

Architecture

We came up with this simplistic architecture for the basic use case.

Automated SFTP transfer

Workflow

  1. Upload a file to the S3 bucket
  2. Trigger Push event notification for the lambda function. Prefer to have a separate lambda function for each client so that we can store all SFTP connection details in environment variables. Env variables will be used to store Server details, credentials, file path, etc...
  3. Lambda function will fetch a file from the S3 bucket
  4. Lambda will transfer the file to External Server.

Worthy Addition

Worth Considering changes on top of this simple approach

  1. If the Lambda function failed to fetch a file then it should do a couple of retries and if it still fails, they should send a notification to the client who is uploading the file to S3 bucket.
  2. If the external transfer fails then Lambda should add that to any SQS queue from that any application can process messages and notify the system and also we can setup retry after few days again.

Upvotes: 2

John Rotenstein
John Rotenstein

Reputation: 269370

Amazon S3 cannot "send" files anywhere.

Therefore, you will need some code running 'somewhere' that will:

  • Download the file(s) from Amazon S3
  • Send the file(s) to the external client via SFTP

This is all easily scriptable. The difficulty probably comes in deciding which files to send and how to handle any errors.

You probably couldn't find any documentation on the topic because sending files via SFTP has nothing specifically related to AWS. Just do it the way you would from anywhere.

For example, let's say you wanted to do it via a Python program running either on an Amazon EC2 instance or as an AWS Lambda function:

  1. Download the desired files by using the AWS SDK for Python (boto3). See: Amazon S3 examples
  2. Send the files via SFTP. See: Transfer file from AWS S3 to SFTP using Boto 3

Upvotes: 2

Related Questions