CODEWITHSUNDEEP
amazon-web-servicesamazon-sqs
Brooks
Brooks

Reputation: 7380

Amazon SQS as IoT message entrypoint

I am trying to put together Elasticsearch Service, Logstash and Amazon SQS as a receiving point for messages coming from various IoT devices sending messages that need to be stored in ES.

My hope is that as an initial proof of concept, I could use SQS as a way to a) buffer against any Elasticsearch downtime (i.e. hold the messages while it comes back up) and b) distribute the messages into ES in a more even, uniform fashion than simply opening Elasticsearch up for public insertion.

As Elasticsearch can't pull messages and SQS can't push messages, I used Logstash installed on an EC2 instance as the middleman to pull and then push. This appears to be working, though only when messages are sent via the SQS "Send Message" Console or an SNS Topic that I subscribed the SQS to.

When trying to send messages to the SQS via cURL, I get either "AccessDenied" or "UnknownOperationException". I set the policy to Principal: * and Action:SQS:*. Below is my permission policy for the SQS:

{
  "Version": "2012-10-17",
  "Id": "arn:aws:sqs:us-east-1:843348267853:testQueue/SQSDefaultPolicy",
  "Statement": [
    {
      "Sid": "Sid1445050222773",
      "Effect": "Allow",
      "Principal": "*",
      "Action": "SQS:*",
      "Resource": "arn:aws:sqs:us-east-1:843348267853:testQueue"
    }
  ]
}

I've tried multiple different cURL commands:

curl -H "Accept: application/json" -d "key=value&key1=value1" https://sqs.us-east-1.amazonaws.com/843348267853/testQueue

< UnknownOperationException />

curl -k -vv -XPOST 'https://sqs.us-east-1.amazonaws.com/843348267853/testQueue' -d '{\"name\":\"Mike\"}'

upload completely sent off: 131 out of 131 bytes
HTTP/1.1 403 Forbidden
Server Server is not blacklisted
Server: Server Date: Sat, 17 Oct 2015 02:53:44 GMT
Content-Type: text/xml Content-Length: 341
Connection: keep-alive x-amzn-RequestId: 61f08380-ec65-5229-8732-cd1b561d2069

Connection #0 to host sqs.us-east-1.amazonaws.com left intact SenderAccessDeniedAccess to the resource https://sqs.us-east-1.amazonaws.com/843348267853/testQueue is denied.61f08380-ec65-5229-8732-cd1b561d2069

I have tried multiple other variations of the cURL commands, with no luck.

I was under the impression that selecting * (ALL) permissions for * (ALL) users would allow public / anonymous message sending privileges?

Obviously, my goal would be to restrict it to specific users, but for the moment, I haven't gotten to that point and we haven't yet gotten to the point where the messages are signed from the IoT devices.

My questions are:

1) What am I doing wrong here that I can't send messages via cURL? 2) Am I architecting this right (i.e. Public msg -> SQS -> EC2 Logstash -> ES)? Or should I be going about this differently?

Thanks.

Upvotes: 1

Views: 541

Answers (2)

E.J. Brennan
E.J. Brennan

Reputation: 46841

This syntax seems to work fine:

curl -d "Action=SendMessage&Version=2011-10-01&MessageBody=example" https://sqs.us-east-1.amazonaws.com/843348267853/testQueue

Upvotes: 1

Michael - sqlbot
Michael - sqlbot

Reputation: 178956

To my knowledge, SQS doesn't have a direct POST mechanism that simply accepts the message payload like you're trying to do here.

The API expects something more like a web form.

http://postdocs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/API_SendMessage.html

Upvotes: 1

Related Questions