Wrong SQS AWS message when I'm subscribed from a SNS Topic

Question

I'm having problems with the next design:

When I'm receiving the message in my SQS Subscriber, the model of message it's wrong, example:

{
  "Type" : "Notification",
  "MessageId" : "7a6789f0-02f0-5ed3-8a11-deebcd08f145",
  "TopicArn" : "arn:aws:sns:us-east-2:167186109795:name_sns_topic",
  "Message" : "My JSON message",
  "Timestamp" : "1987-04-23T17:17:44.897Z",
  "SignatureVersion" : "1",
  "Signature" : "string",
  "SigningCertURL" : "url",
  "UnsubscribeURL" : "url",
  "MessageAttributes" : {
    "X-Header1" : {"Type":"String","Value":"value1"},
    "X-Header2" : {"Type":"String","Value":"value2"},
    "X-Header3" : {"Type":"String","Value":"value3"},
    "X-HeaderN" : {"Type":"String","Value":"value4"}
  }
}

The common model when recieve message from SQS should be:

{
  "Records": [
    {
      "messageId": "19dd0b57-b21e-4ac1-bd88-01bbb068cb78",
      "receiptHandle": "MessageReceiptHandle",
      "body": "Hello from SQS!",
      "attributes": {
        "ApproximateReceiveCount": "1",
        "SentTimestamp": "1523232000000",
        "SenderId": "123456789012",
        "ApproximateFirstReceiveTimestamp": "1523232000001"
      },
      "messageAttributes": {},
      "md5OfBody": "7b270e59b47ff90a553787216d55d91d",
      "eventSource": "aws:sqs",
      "eventSourceARN": "arn:{partition}:sqs:{region}:123456789012:MyQueue",
      "awsRegion": "{region}"
    }
  ]
}

In my handler Java Lambda (example code) is throwing an exception because the estructure of de message received is not SQS Event:

public class MyHandler implements RequestHandler<SQSEvent, String> {

  @Override
  public String handleRequest(SQSEvent event, Context context) {
    LambdaLogger logger = context.getLogger();

    for (SQSEvent.SQSMessage msg : event.getRecords()) {
      logger.log("SQS message body: " + msg.getBody());
      logger.log("Get attributes: " + msg.getMessageAttributes().toString());

      msg.getMessageAttributes()
              .forEach(
                      (k, v) -> {
                        logger.log("key: " + k + "value: " + v.getStringValue());
                      });
    }
    return "Successful";
  }
}

How can I do for handle the message thats its receiving ?

Answer 1

In my opinion this isn't documented too well but it's not bad once you figure it out.

The first thing is that I don't use the predefined Lambda objects. I read everything into a String and take it from there. So the base of my Lamda function is:

public void handleRequest(InputStream inputStream, OutputStream outputStream, Context context) throws IOException {
    // copy InputStream to String, avoiding 3rd party libraries
    ByteArrayOutputStream result = new ByteArrayOutputStream();
    byte[] buffer = new byte[1024];
    int length;
    while ((length = inputStream.read(buffer)) != -1) {
        result.write(buffer, 0, length);
    }

    String jsonString = result.toString();
}

When you "go direct" from SNS to Lambda the message looks something like (some fields removed for sake of length):

{
    "Records": [
        {
            "EventSource": "aws:sns",
            "EventVersion": "1.0",
            "Sns": {
                "Type": "Notification",
                "Subject": "the message subject",
                "Message": "{\"message\": \"this is the message\", \"value\": 100}",
                "Timestamp": "2020-04-24T21:44:28.220Z",
                "SignatureVersion": "1"
            }
        }
    ]
}

I had sent in a test message in JSON with two simple fields. Using JsonPath the "message" field inside of everything is read with:

String snsMessage = JsonPath.read(jsonString, "$.Records[0].Sns.Message");
String realMessage = JsonPath.read(snsMessage, "$.message");

But when it goes SNS -> SQS -> Lambda (or, indeed any SNS -> SQS path) the SNS message is now mostly wrapped and escaped in an SQS message:

{
    "Records": [
        {
            "messageId": "ca8c53e5-8417-4479-a720-d4ecf970ca68",
            "body": "{\n  \"Type\" : \"Notification\",\n  \"Subject\" : \"the message subject\",\n  \"Message\" : \"{\\\"message\\\": \\\"this is the message\\\", \\\"value\\\": 100}\"\n}",
            "attributes": {
                "ApproximateReceiveCount": "1"
            },
            "md5OfBody": "6a4840230aca6a7bf7934bf191a529b8",
            "eventSource": "aws:sqs"
        }
    ]
}

So in this case, the value is in Records[0].body but that contains another JSON object. I'll admit that there is likely an easier way but from what I found I had to parse 3 times:

String sqsBody = <as read in lambda>;
String recordBody = JsonPath.read(sqsBody, "$.Records[0].body");
String internalMessage = JsonPath.read(recordBody, "$.Message");
// now read out of the sns message
String theSnsMessage = JsonPath.read(message, "$.message");