CODEWITHSUNDEEP
amazon-web-servicesamazon-sqsamazon-mws
User132887
User132887

Reputation: 11

SQS maximum number of messages using php

SQS - I need to get more messages in a batch response.

I have set up SQS queue to on which I am processing my other inventory data which is too large. In a batch it gives only 10 messages, which is not fulfilling my requirement. I am using it in PHP. Has anybody worked with and achieved?

I referred many guidance and got some idea of using long polling. But Could anybody please give some hint how to do that ? I have done very basic of SQS.

I expect to work on thousands data based on SQS response on daily basis.

Upvotes: 1

Views: 613

Answers (3)

IrfanZ
IrfanZ

Reputation: 61

Idea is pretty much the same, you will either need Lambda function or dedicated Ec2's to host Sqs queue consumers (code).

In my scenario I had issue with keeping up with Lambda's because of millions of notification generated every hour. Key issues involved with Lambda was concurrency issues.

However, I ended with using Ec2 auto scaling group with spot instances. Spot instances are key in reducing cost, since spot instance can offer 50% to 90% cost reduction for compute cost.

Upvotes: 0

ScottG
ScottG

Reputation: 11121

I ran a pretty substantial process a couple years ago with the AnyOfferChanged notification. Yes, you can only receive 10 messages at a time, but, you can just run multiple threads or multiple instances of you consumer process. My company sold 10,000 SKU's on Amazon and the queue would have tens of thousands of messages in it unless I kept up.

I had a cloud server on Azure, but EC2 would be fine. It ran a windows service that kicked off any number of threads, based on the current message count. More messages, more threads. Ten at a time constantly on each thread until the message count got down to zero. When at zero, I'd put the service to sleep for a few seconds. The process I had involved reading the 10 messages, then writing them out to SQL Server (might use DynamoDB these days). I had another process that was triggered every half hour or hour that read through the price notifications in my SQL table, performed repricing, and updated Amazon with any new price.

I would receive price changes for the same SKU several times. Amazon and other sellers play games like I did where you change a price, they pick up the change and then they change their price. Every change triggers an SQS notification. So what I did was on the writing out to my SQL table of the price change, I would overlay any existing record. So I always had the latest price change when my repricing engine was triggered. I would play around with the timing on that as to not update my price so much on Amazon. You'd get into a race for the bottom and lose money fast.

If I were going to rewrite it today, I would most likely follow John's advice and run Lambdas on AWS. But the key is to run multiple consumers in parallel because there most likely will be a large number of notifications that you'll want to process.

Upvotes: 0

John Rotenstein
John Rotenstein

Reputation: 269330

It appears that you are referring to AnyOfferChangedNotification from the Amazon Marketplace Web Service (Amazon MWS).

From Receiving notifications:

To receive notifications, you must first create a Destination. Currently, notifications can only be sent to an Amazon Simple Queue Service (Amazon SQS) standard queue.

So, it seems you have no control over how the information is received — it must be sent to an Amazon SQS queue.

There are several ways to process information from a queue:

  • Run consumers somewhere on the Internet.
  • Run consumers on Amazon EC2 instances, preferably across multiple instances if the volume of messages is high.
  • Trigger an AWS Lambda function when data is sent to the queue. Multiple Lambda functions can run in parallel to process the data.

You do not provide much information about how the data is processed, so it isn't obvious whether it can be processed in parallel, or if it requires mostly in-order processing.

It sounds like you are currently processing the messages via a single-threaded PHP program, either on Amazon EC2 instances or elsewhere.

I recommend you investigate taking a serverless approach, using an AWS Lambda function to process the messages. This way, it can automatically scale to handle large loads, while costing nothing when there are no messages to be processed. It is also likely to be a lower cost than running on Amazon EC2.

Upvotes: 1

Related Questions