CODEWITHSUNDEEP
node.jslambdaaws-lambdapg-promise
maxwell2022
maxwell2022

Reputation: 2855

Why AWS Lambda execution time is long using pg-promise

I started using AWS Lambda to perform a very simple task which is executing an SQL query to retrieve records from an RDS postgres database and create SQS message base on the result.

Because Amazon is only providing aws-sdk module (using node 4.3 engine) by default and we need to execute this SQL query, we have to create a custom deployment package which includes pg-promise. Here is the code I'm using:

console.info('Loading the modules...');
var aws = require('aws-sdk');
var sqs = new aws.SQS();
var config = {
  db: {
    username: '[DB_USERNAME]',
    password: '[DB_PASSWORD]',
    host: '[DB_HOST]',
    port: '[DB_PORT]',
    database: '[DB_NAME]'
  }
};

var pgp = require('pg-promise')({});
var cn = `postgres://${config.db.username}:${config.db.password}@${config.db.host}:${config.db.port}/${config.db.database}`;

if (!db) {
  console.info('Connecting to the database...');
  var db = pgp(cn);
} else {
  console.info('Re-use database connection...');
}

console.log('loading the lambda function...');
exports.handler = function(event, context, callback) {

  var now = new Date();
  console.log('Current time: ' + now.toISOString());

  // Select auction that need to updated
  var query = [
    'SELECT *',
    'FROM "users"',
    'WHERE "users"."registrationDate"<=${now}',
    'AND "users"."status"=1',
  ].join(' ');

  console.info('Executing SQL query: ' + query);
  db.many(query, { status: 2, now: now.toISOString() }).then(function(data) {    
    var ids = [];
    data.forEach(function(auction) {
      ids.push(auction.id);
    });

    if (ids.length == 0) {
      callback(null, 'No user to update');

    } else {

      var sqsMessage = {
        MessageBody: JSON.stringify({ action: 'USERS_UPDATE', data: ids}), /* required */
        QueueUrl: '[SQS_USER_QUEUE]', /* required */
      };

      console.log('Sending SQS Message...', sqsMessage);
      sqs.sendMessage(sqsMessage, function(err, sqsResponse) {

        console.info('SQS message sent!');
        if (err) {
          callback(err);
        } else {
          callback(null, ids.length + ' users were affected. SQS Message created:' + sqsResponse.MessageId);
        }
      });
    }

  }).catch(function(error) {
    callback(error);
  });

};

When testing my lambda function, if you look at the WatchLogs, the function itself took around 500ms to run but it says that it actually took 30502.48 ms (cf. screenshots).

enter image description here enter image description here

So I'm guessing it's taking 30 seconds to unzip my 318KB package and start executing it? That for me is just a joke or am I missing something? I tried to upload the zip and also upload my package to S3 to check if it was faster but I still have the same latency.

I noticed that the Python version can natively perform SQL request without any custom packaging...

All our applications are written in node so I don't really want to move away from it, however I have a hard time to understand why Amazon is not providing basic npm modules for database interactions.

Any comments or help are welcome. At this point I'm not sure Lambda would be benefic for us if it takes 30 seconds to run a script that is triggered every minute...

Anyone facing the same problem?

UPDATE: This is how you need to close the connection as soon as you don't need it anymore (thanks again to Vitaly for his help):

exports.handler = function(event, context, callback) {

  [...]

  db.many(query, { status: 2, now: now.toISOString() }).then(function(data) {    

    pgp.end(); // <-- This is important to close the connection directly after the request

    [...]

Upvotes: 1

Views: 2147

Answers (1)

vitaly-t
vitaly-t

Reputation: 25880

The execution time should be measured based on the length of operations being executed, as opposed to how long it takes for the application to exit.

There are many libraries out there that make use of a connection pool in one form or another. Those typically terminate after a configurable period of inactivity.

In case of pg-promise, which in turn uses node-postgres, such period of inactivity is determined by parameter poolIdleTimeout, which defaults to 30 seconds. With pg-promise you can access it via pgp.pg.defaults.poolIdleTimeout.

If you want your process to exit after the last query has been executed, you need to shut down the connection pool, by calling pgp.end(). See chapter Library de-initialization for details.

It is also shown in most of the code examples, as those need to exit right after finishing.

Upvotes: 3

Related Questions