CODEWITHSUNDEEP
phpamazon-web-servicesamazon-sqs
azngunit81
azngunit81

Reputation: 1604

SQS Amazon Throttling request between servers

I'm currently using Amazon SQS to queue up messages for processing. I have benchmark that I can only put into the queue with the lib at about 400 message/sec. I'm using the upper limit of sending a message to 10.

My second problem is that the SQS is being used remotely (i.e: i have one server that is creating the message that is not next to amazon or EC2 instances).

My goal is to increase this bottleneck because I want to do up to 10K request at least.

Is this a fail achievement because of network latency? Or can there be better solution to SQS or code readjustment to achieve this.

Also the lib for SQS is PHP.

EDIT: code added

use Aws\Sqs\SqsClient;
class Sqs implements QueueInterface
{
    private static $_instance = null;    
    private $_client = null;    

protected function __construct($setting)
{               
    if ($this->_client === null) {
        try {
            $this->_client = SqsClient::factory($setting);
        } catch (Exception $e) {
                $this->_client = null;                    
        }
    }

}        
    public static function getQueue($setting)    
    {

        if (self::$_instance === null) {
            self::$_instance = new Sqs($setting);  
        }
        return self::$_instance;
    }

    public function put(Data\DataRepository $data, $queueName)
    {
        $attributes=array();

        if (!$this->_client) return false;
        return self::_putToClient($this->_client, $data->getData(), $queueName, $attributes);

    }

    /**
     * Put data into the queue using a defined client.
     *
     * @param mixed  $client     The client to use.
     * @param mixed  $data       The data to insert.
     * @param string $queueName  The name of the queue in which to insert.
     * @param mixed  $attributes Some attributes for the client (QueueAttribute::DELAY_SECONDS)
     *
     * @return string The job id in the queue or false if a problem happened.
     */
    private static function _putToClient($client, $data, $queueName, $attributes=array())
    {
        try {
            $result = $client->createQueue(array('QueueName' => $queueName, 'Attributes' => $attributes));
            $queue_url = $result->get('QueueUrl');

            $response = $client->sendMessage(array('QueueUrl' => $queue_url, 'MessageBody' => $data));

            return $response->getPath('MessageId');
        } catch (Exception $e) {
            return false;
        }
    }
}

Upvotes: 0

Views: 2159

Answers (1)

Jeremy Lindblom
Jeremy Lindblom

Reputation: 6517

It's possible that network latency is affecting you, but there are probably other some things you can do to get more throughput.

What your are doing with your code should work fine. However, it's definitely not the most optimized. _putToClient() always makes 2 API calls. Only one call to SqsClient::createQueue() should be necessary; it seems weird to make that call every time you send a message. If you do that only one time, and store/cache the QueueUrl, you can eliminate some latency there.

You should also check out the SDK guide for doing parallel requests. This will allow you to send more than 10 messages at a time. You may also want to read the SDK performance guide to see if there is anything you can do to speed up your usage of the SDK in general.

I'd also post to the SQS forum to see if the SQS engineers can point out any best practices specific to SQS.

Upvotes: 1

Related Questions