CODEWITHSUNDEEP
node.jsamazon-web-servicesamazon-s3aws-lambdaaws-sdk-nodejs
Trickyan
Trickyan

Reputation: 21

Upgrading AWS Lambda Scripts from Node.js 16.x to 18.x with Minimal Changes

I have a collection of 15-20 scripts running on AWS Lambda with Node.js 16.x that perform various operations, including interacting with S3 using functions like copy, headObject, putObject, and getObject. Additionally, my scripts utilize AWS services such as Step Functions, Glue, SNS, and Athena. I cannot stick with 16.x since it has a deadline of mid of June 2024.

I need expert guidance on the minimal changes required to make these scripts compatible with Node.js 18.x. While it would be beneficial if they could also work with Node.js 20.x, that's not a priority at the moment.

One of the scripts looks like this:

// Script to read, modify, and write CSV files to S3 after removing quotes.
// Requires AWS-SDK v2 and utilizes AWS Step Functions.

const AWS = require('aws-sdk');
const sf = new AWS.StepFunctions();
const s3 = new AWS.S3();
const rl = require('readline');
const stream = require('stream');

// Randomized AWS credentials
AWS.config.update({
    region: {mention-region-here}
});

function extractBucketName(path) {
    var parts = path.split("/", 3);
    return parts[2];
}

function extractBucketKey(path) {
    var s = path.replace(/^s3:\/\//, "");
    var pos = s.indexOf("/");
    var key = s.substr(pos + 1);
    return key;
}

exports.handler = function(event, context, callback) {
    // Main logic to read from S3, process, and write back to S3.
    if ((typeof event.inputFile != "undefined") && (typeof event.outputFile != "undefined")) {
        var paramsInput = {
            Bucket: extractBucketName(event.inputFile),
            Key: extractBucketKey(event.inputFile)
        };

        // Create read stream from S3
        var readStream = s3.getObject(paramsInput).createReadStream();

        // Pass the S3 read stream into the readline interface to break into lines
        var lineReader = rl.createInterface({
            input: readStream,
            terminal: false
        });

        // Create a write stream to upload processed data back to S3
        const writeStream = new stream.PassThrough();
        var paramsOutput = {
            Bucket: extractBucketName(event.outputFile),
            Key: extractBucketKey(event.outputFile),
            Body: writeStream
        };
        s3.upload(paramsOutput).promise();

        // Process each line from the input file
        lineReader.on('line', function(line) {
            // Perform processing on line (remove quotes)
            line = line.replace(/['"]+/g, '');
            // Write processed line to the write stream
            writeStream.write(`${line}\n`);
        }).on('close', function() {
            // Close the write stream after processing is complete
            writeStream.end();
            callback(null);

            // Example code to send success token to AWS Step Functions
            var taskParams = {
                output: JSON.stringify('Write a success message'),
                taskToken: event.token
            };
            sf.sendTaskSuccess(taskParams, function(err, data) {
                if (err) {
                    console.error(err.message);
                    callback(err.message);
                    return;
                }
                console.log(data);
                callback(null);
            });
        });
    }
};

I have already read Amazon's articles and documentation, but I still don't have a clear picture of what needs modification and what doesn't for the transition from Node.js 16.x to 18.x. I want to avoid the trial-and-error method to save me from a error nightmare. Any expert advice from the community on the changes required for a smooth transition would be greatly appreciated. Thank you!

After I upgraded my NodeJs from 16.x to 18.x in dev, I tried the following articles: https://docs.aws.amazon.com/AWSJavaScriptSDK/v3/latest/client/s3/ and other articles, but unfortately, that does not provide all the information I need. Also, I read this question: Lambda function cant load AWS sdk after switching to 18.x

But that is just for s3 and does not help completely

Upvotes: 1

Views: 1527

Answers (1)

MadMurf
MadMurf

Reputation: 2313

TL;DR There are automated codemod scripts that might help you out here

Part of the issue with upgrading from 16 to 18/20 is that aws-sdk has upgraded to v3

Up until Node.js 16, Lambda’s Node.js runtimes included the AWS SDK for JavaScript version 2. This has since been superseded by the AWS SDK for JavaScript version 3, which was released in December 2020. Starting with Node.js 18, and continuing with Node.js 20, the Lambda Node.js runtimes have upgraded the version of the AWS SDK for JavaScript included in the runtime from v2 to v3. Customers upgrading from Node.js 16 or earlier runtimes who are using the included AWS SDK for JavaScript v2 should upgrade their code to use the v3 SDK.

REF: https://aws.amazon.com/blogs/compute/node-js-20-x-runtime-now-available-in-aws-lambda/

Of benefit to you might be the automated migrate/codemod scripts detailed in https://docs.aws.amazon.com/sdk-for-javascript/v3/developer-guide/migrating.html#migrating-to-v3

for example you can run the aws-sdk v2-to-v3 transform on example.ts as follows.

$ npx aws-sdk-js-codemod -t v2-to-v3 example.ts

The result of the migrate should also reduce the overhead required to run your application, from the migrate link above

The aws-sdk package adds about 40 MB to your application. Replacing var AWS = require("aws-sdk") with import {DynamoDB} from "@aws-sdk/client-dynamodb" reduces that overhead to about 3 MB. Restricting the import to just the DynamoDB client and ListTablesCommand command reduces the overhead to less than 100 KB.

Upvotes: -1

Related Questions