tinker.tamper
tinker.tamper

Reputation: 53

Is Amazon SQS a good tool for handling analytics logging data to a database?

We have a few nodejs servers where the details and payload of each request needs to be logged to SQL Server for reporting and other business analytics.

The amount of requests and similarity of needs between servers has me wanting to approach this with an centralized logging service. My first instinct is to use something like Amazon SQS and let it act as a buffer with either SQL Server directly or build a small logging server which would make database calls directed by SQS.

Does this sound like a good use for SQS or am I missing a widely used tool for this task?

Upvotes: 3

Views: 1361

Answers (1)

Chris White
Chris White

Reputation: 1447

The solution will really depend on how much data you're working with, as each service has limitations. To name a few:

SQS

  • First off since you're dealing with logs, you don't want duplication. With this in mind you'll need a FIFO (first in first out) queue.
  • SQS by itself doesn't really invoke anything. What you'll want to do here is setup the queue, then make a call to submit a message via the AWS JS SDK. Then when you get the message back in your callback, get the message ID and pass that data to an invoked Lambda function (you can write those in NodeJS as well) which stores the info you need in your database.
  • That said it's important to know that messages in an SQS queue have a size limit:

The minimum message size is 1 byte (1 character). The maximum is 262,144 bytes (256 KB).

To send messages larger than 256 KB, you can use the Amazon SQS Extended Client Library for Java. This library allows you to send an Amazon SQS message that contains a reference to a message payload in Amazon S3. The maximum payload size is 2 GB.

CloudWatch Logs

(not to be confused with the high level cloud watch service itself, which is more sending metrics)

  • The idea here is that you submit event data to CloudWatch logs
  • It also has a limit here:

Event size: 256 KB (maximum). This limit cannot be changed

  • Unlike SQS, CloudWatch logs can be automated to pass log data to Lambda, which then can be written to your SQL server. The AWS docs explain how to set that up.

S3

Simply setup a bucket and have your servers write out data to it. The nice thing here is that since S3 is meant for storing large files, you really don't have to worry about the previously mentioned size limitations. S3 buckets also have events which can trigger lambda functions. Then you can happily go on your way sending out logo data.

If your log data gets big enough, you can scale out to something like AWS Batch which gets you a cluster of containers that can be used to process log data. Finally you also get a data backup. If your DB goes down, you've got the log data stored in S3 and can throw together a script to load everything back up. You can also use Lifecycle Policies to migrate old data to lower cost storage, or straight remove it all together.

Upvotes: 6

Related Questions