CODEWITHSUNDEEP
node.jsamazon-web-servicesamazon-s3aws-lambdanode-streams
kakashi hatake
kakashi hatake

Reputation: 1185

Upload large stream to S3 with aws lambda

When I try read data and upload to data stream with aws lambda function, when stream size is 250 mb or 300mb zip file upload to success one minute but stream size is 400 mb or 500 mb function take timeout and not to make upload

my sample code is 

    let upload = new AWS.S3.ManagedUpload({
        partSize: 10 * 1024 * 1024, queueSize: 2,
        params: {Bucket: process.env.BUCKET_NAME, Key: `${correlationId}.zip`, Body: zipStream}
    });

    upload.on('httpUploadProgress', function (evt) {
        if (evt) {
            console.log('Completed ' + (evt.loaded * 100 / evt.total).toFixed() + '% of upload');
        }
    }).send((error, data) => {
        if (error) {
            console.error(`error creating stream to s3 ${error.name} ${error.message} ${error.stack}`);
        }
        console.log(`image zip file now sending ${data}`)
    });

    await upload.promise();

Upvotes: 0

Views: 2594

Answers (2)

Arun Kamalanathan
Arun Kamalanathan

Reputation: 8593

Vpc based lambdas support 15 minutes timeout. The reason it takes time is that the request is going through internet. It should be faster if you create vpc endpoint for S3

https://aws.amazon.com/blogs/aws/new-vpc-endpoint-for-amazon-s3/

Upvotes: 1

Marcos Casagrande
Marcos Casagrande

Reputation: 40404

The default timeout for all requests in the AWS SDK is 2 minutes. You can increase the timeout using:

AWS.config.httpOptions.timeout = 300000; // or 0 to disable timeout

Also make sure to increase the Lambda timeout.

Upvotes: 0

Related Questions