AWS Lambda Microservice Design and Chaining C# Lambda with SNS (security?)

Question

I'm new to microservices, so trying to do a little app.

This is my current design:

Create two LambdaServices:

1) GetSomeData (string domainName, string nextSNS, ILambdaContext context) 
2) StoreSomeData (string jsonData)

Part 1:

My understanding of microservice is that each function should do only one thing. So "GetSomeData" takes a domain name, does a web call, parses the results into JSON. Then I need to store it in an RDS database in StoreSomeData. But in the future, I might want to just get the data, or do something else with it.

The function I need 99% of the time now, is really GetAndStoreSomeData. But if I do that, I'm not a microservice, right?

So I'm thinking if a program just wants the JSON back from "GetSomeData", it will just pass null in the nextSNS. But if it wants to store the data, it will pass an SNSTopicName or arn in nextSNS, then "GetSomeData" will publish a message to that SNS with the JSON response.

The process will be kicked off by some other process that I haven't totally figured out yet, that will pick some domains from the RDS database, and probably call API-Gateway to launch "GetSomeData". I'll probably have to run it from some type of scheduler.

I'd like to know if this is a good design. I just got "GetSomeData" published and tested without the "nextSNS" parm.

Part 2:

If "GetSomeData" needs to publish to SNS, how can I do that without storing the IAM credentials. Can I use a role? Or I have to use the secret access key, which maybe I could put in an environment variable to at least keep it out of the code.

I was thinking of cloning this method: https://gist.github.com/bkizzy/2705156 to publish the SNS message. But then I found a Java sample that is only about 6 lines of code for the call here: Lambda does not trigger SNS event. Chaining AWS lambdas with SNS. Is there similar short way to do the same from C#? I couldn't find much on the Amazon site on how to publish SNS other than the raw request/response. (https://docs.aws.amazon.com/sns/latest/api/API_Publish.html)

Answer 1

The answer to your specific security question is that you assign an IAM execution role to each Lambda function which allows it to access things like SNS. You don't use credentials like AWS secret access key with your Lambda function.

Regarding the difference in Java and C# interactions with AWS, the C# method you linked for posting to SNS is using raw HTTP connections instead of using the AWS SDK for .NET, so of course it's way more lines of code. You should definitely use the official AWS SDK, which will make your C# code look almost identical to the Java code you linked.

Answer 2

Pretty much by definition any answers you get here will be a bit opinionated but...

Part 1

"Microservice" is a very nice marketing name but ultimately it is a design pattern, not a hard and fast design rule. To me alot of it depends on the support structure around the services. For example, I work mostly in startup environments. That means I have to handle the AWS RDS, AWS VPC, AWS EC2 with the AWS ELB, and so on with the TLA's.

Other environments have dedicated DevOps people that are there to assist in managing the environments.

That's a long way of saying that if you want to only maintain a single Lambda then maintain a single Lambda. The Microservice police may or may not accost you but only you know what makes the most sense in this area.

Part 2

One of the cool things about AWS Lambda's (and EC2 has a very similar concept) is that you can set up an execution role. Basically what this is is a role in IAM that your Lambda code runs under. So if, for example, your Lambda needs to send an SNS message, you can add AmazonSNSFullAccess to the role that is running the Lambda and your Lambda can now send and receive SNS messages.

In terms of getting the credentials for the C# environment, it looks like this blog post gives you an idea of how to do it.

I'll admit that I'm not familiar with the C# side of things to send SNS messages but it looks like Publish maps well to the Java side of things.