CODEWITHSUNDEEP
pythonamazon-web-servicesaws-lambdaamazon-sqs
Twinkle Sebastain
Twinkle Sebastain

Reputation: 1

Unable to fetch all SQS messages by Lambda(Python)

Unable to fetch all SQS messages by Lambda(Python). Its reading the Queue partially

def lambda_handler(event, context):
response = sqs_client.receive_message(QueueUrl="queurlXXX.fifo",
MaxNumberOfMessages=10,WaitTimeSeconds=20,VisibilityTimeout=0)
for message in response.get("Messages", []):
    message_body = message["Body"]
    #print(message_body)
    ip_json=json.loads(message_body)
    op_json=json.dumps(ip_json)
    #print(op_json)
    if op_json:
        conn = pymssql.connect(host='DB Credentials', database='DbNAme', port='1433')
        cursor = conn.cursor()
        cursor.execute("INSERT INTO [table]([MessageId],[Document],[IsProcessed],[CreatedUtc],[CreatedBy],[ModifiedUtc],[ModifiedBy]) VALUES('messageid', %s,1,GETUTCDATE(),'system',GETUTCDATE(),'system');",(op_json))
        conn.commit()
    #print(f"Message body: {json.loads(message_body)}")
    #print(f"Receipt Handle: {message['ReceiptHandle']}")
return "Queue loaded"

Upvotes: 0

Views: 221

Answers (1)

John Rotenstein
John Rotenstein

Reputation: 270089

It appears that you are saying that your receive_message() call did not return all messages in the Amazon SQS queue. This is a normal behaviour of Amazon SQS due to its highly distributed architecture.

From Amazon SQS short and long polling - Amazon Simple Queue Service:

Consuming messages using short polling
When you consume messages from a queue using short polling, Amazon SQS samples a subset of its servers (based on a weighted random distribution) and returns messages from only those servers. Thus, a particular ReceiveMessage request might not return all of your messages. However, if you have fewer than 1,000 messages in your queue, a subsequent request will return your messages. If you keep consuming from your queues, Amazon SQS samples all of its servers, and you receive all of your messages.

The following diagram shows the short-polling behavior of messages returned from a standard queue after one of your system components makes a receive request. Amazon SQS samples several of its servers (in gray) and returns messages A, C, D, and B from these servers. Message E isn't returned for this request, but is returned for a subsequent request.

Amazon SQS polling

Upvotes: 1

Related Questions