CODEWITHSUNDEEP
node.jsamazon-web-serviceslambdaamazon-sqsserverless
Mike
Mike

Reputation: 1268

Get all message from aws sqs

I have a ScheduledEvent on my lamda function for every 24 hours and then inside function, I am calling SQS to get my messages.

export class EmailNotificationProcessor {
    public static async run(): Promise<void> {
        console.log('event');
        await this.getNotificationFromSqs();
    }

    private static async getNotificationFromSqs(): Promise<void> {
        const messagesToDelete: DeleteMessageBatchRequestEntryList = [];
        const messageRequest: ReceiveMessageRequest = {
            QueueUrl: process.env.DID_NOTIFICATION_SQS_QUEUE,
            MaxNumberOfMessages:10,
            WaitTimeSeconds:20
        }
        const { Messages }: ReceiveMessageResult = await receiveMessage(messageRequest);
        console.log('Messages', Messages);
        console.log('Total Messages ', Messages.length);
        if (Messages && Messages.length > 0) {
            for (const message of Messages) {
                console.log('body is ', message.Body);
                messagesToDelete.push({
                    Id: message.MessageId,
                    ReceiptHandle: message.ReceiptHandle,
                } as DeleteMessageBatchRequestEntry);
            }
        }
        await deleteMessages(messagesToDelete);
    }
}

I am expecting 1 to 30 messages inside my queue and want to process all messages before sending an email which consists of the content that I will parse from sqs body.

My function for receiving messages

export const receiveMessage = async (request: SQS.ReceiveMessageRequest): Promise<PromiseResult<SQS.ReceiveMessageResult, AWSError>> =>{
    console.log('inside receive');
    return sqs.receiveMessage(request).promise();
}

Now I am not able to receive all messages at once and only getting 3 messages or sometimes 1 message at a time.

I know limit for API call is 10 in one single request but is there any way to wait and get all your message.

Upvotes: 2

Views: 5978

Answers (1)

Abdul Moeez
Abdul Moeez

Reputation: 1401

First of all, There is no configuration to get more then 10 messages from queue.

ReceiveMessage: Retrieves one or more messages (up to 10), from the specified queue

For your other problems: I think you are using Short poll ReceiveMessage call.If the number of messages in the queue is extremely small, you might not receive any messages in a particular ReceiveMessage response.

Try Long Polling: Long polling helps reduce the cost of using Amazon SQS by eliminating the number of empty responses (when there are no messages available for a ReceiveMessage request) and false empty responses (when messages are available but aren't included in a response).

Note* For getting more messages you need to wrap the call to SQS in a loop and keep requesting more messages until the queue is empty, which can lead you to get duplicate messages as well so try VisibilityTimeout in that problem.

Try VisibilityTimeout: The duration (in seconds) that the received messages are hidden from subsequent retrieve requests after being retrieved by a ReceiveMessage request.

Sample Wrap up SQS call code:

function getMessages(params, count = 0, callback) {
let allMessages = [];
sqs.receiveMessage(params, function (err, data) {
    if (err || (data && !data.Messages || data.Messages.length <= 0)) {
        
        if(++count >= config.SQSRetries ){
            return callback(null, allMessages);
        }

        return setTimeout(() => {
            return getMessages(params, count, callback);
        }, 500);
        
    } else if (++count !== config.SQSRetries ){
        allMessages.push(data);

        return setTimeout(() => {
            return getMessages(params, count, callback);
        }, 500);
    } else {
        allMessages.push(data);

    callback(null, allMessages);
    }
});

In config.SQSRetries, we have set the value according to our requirement but as your SQS have 1 to 30 messages, Then '7' will be good for you!

Links: RecieveMessage UserGuide

Upvotes: 2

Related Questions