How to add SQS message attributes in SNS subscription?

Question

The documentation for AWS SNS and SQS have sections about message attributes. But there is no explanation how to have SQS message attributes when that queue is subscribed to a SNS topic.

Is there a way to configure AWS SNS to add particular message attributes to the SQS messages send via the subscription?

Answer 1

Note: See other answer for better response, using Raw Message Delivery

---

While the Using Amazon SNS Message Attributes documentation sending Amazon SNS message attributes to Amazon SQS, it appears that the attributes are sent in the body of the message rather than being attached as message attributes on the resulting Amazon SQS messages.

For example, I did the following:

• Created an Amazon SNS topic
• Created an Amazon SQS queue and subscribed it to the SNS topic
• Published a message to SNS

I published via the AWS Command-Line Interface (CLI):

aws sns publish --topic-arn arn:aws:sns:ap-southeast-2:123456789012:foo --message msg --subject subj --message-attributes '{"somename" : { "DataType":"String", "StringValue":"somevalue"}}'

(I got syntax help from map datatype in aws cli)

The resulting message in SQS showed the attributes as part of the message:

{
  "Type" : "Notification",
  "MessageId" : "53e3adad-723a-5eae-a7b7-fc0468ec2d37",
  "TopicArn" : "arn:aws:sns:ap-southeast-2:123456789012:foo",
  "Subject" : "subj",
  "Message" : "msg",
  "Timestamp" : "2017-05-29T12:48:22.186Z",
  ...
  "MessageAttributes" : {
    "somename" : {"Type":"String","Value":"somevalue"}
  }
}

It would be much nicer if these attributes were attached to the SQS message as official SQS attributes. Alas, it seems this is not the case.

Answer 2

If you are here because you have a SQS queue that is subscribed to an SNS topic, you checked that your subscription has set the Raw Message Delivery to True but you still cannot read an attribute on your SQS message:

Make sure that your SQS client is not filtering out message attributes.

The code below will only include myAttribute when receiving messages from the SQS queue:

SQS.receiveMessage({
  QueueUrl: queueUrl,
  VisibilityTimeout: 20,
  WaitTimeSeconds: 10,
  MessageAttributeNames: [
    "myAttribute"
  ],
},...

If you want to read the value of some attribute other than myAttribute you will have to specify it (white list) or replace "myAttribute" with "All" to include all SQS attributes.

SQS.receiveMessage({
  MessageAttributeNames: [
    "myAttribute", "myOtherAttribute"
  ],
},...

Reference: https://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/API_ReceiveMessage.html

Answer 3

Enabled Raw message delivery type while adding SQS subscription for the topic inside SNS

Answer 4

From aws documentation:

To use message attributes with Amazon SQS endpoints, you must set the subscription attribute, Raw Message Delivery, to True. For more information about raw message delivery, see Appendix: Large Payload and Raw Message Delivery. https://docs.aws.amazon.com/sns/latest/dg/SNSMessageAttributes.html https://docs.aws.amazon.com/sns/latest/dg/large-payload-raw-message.html

Added an example from real-life project. Hopefully it helps to clarify things. The message published to the sns topic is following:

aws sns publish --topic-arn arn:aws:sns:us-west-2:xxx:pollution-event --message '{"operatorId":3375001,"eventTypeId":1,"eventLevelId":1,"validFrom":"2018-03-10T09:00:00Z","validTo":"2018-03-11T09:00:00Z"}'  --message-attributes '{"Type" : { "DataType":"String", "StringValue":"Orchestration.Services.Model.Pollution.PollutionMessage"}}'

Enable Raw Delivery is false (default). The message received by sqs contains only content, there are no attributes

{
  "Type": "Notification",
  "MessageId": "78d5bc6f-142c-5060-a75c-ef29b774ec66",
  "TopicArn": "arn:aws:sns:eu-west-2:xxx:pollution-event",
  "Message": "{\"validFrom\": \"2018-03-10T09:00:00Z\",\"validTo\": \"2018-03-11T09:00:00Z\",\"eventLevelId\": 1,\"eventTypeId\": 1,\"operatorId\": 3375001}",
  "Timestamp": "2018-04-17T11:33:44.770Z",
  "SignatureVersion": "1",
  "Signature": "xxx==",
  "SigningCertURL": "https://sns.eu-west-2.amazonaws.com/SimpleNotificationService-xxx.pem",
  "UnsubscribeURL": "https://sns.eu-west-2.amazonaws.com/?Action=Unsubscribe&SubscriptionArn=arn:aws:sns:eu-west-2:xxx",
  "MessageAttributes": {
    "Type": {
      "Type": "String",
      "Value": "Orchestration.Services.Model.Pollution.PollutionMessage"
    },
    "AWS.SNS.MOBILE.MPNS.Type": {
      "Type": "String",
      "Value": "token"
    },
    "AWS.SNS.MOBILE.MPNS.NotificationClass": {
      "Type": "String",
      "Value": "realtime"
    },
    "AWS.SNS.MOBILE.WNS.Type": {
      "Type": "String",
      "Value": "wns/badge"
    }
  }
}

Enable Raw delivery is true. The message contains the message attributes and proper content