CODEWITHSUNDEEP
.netamazon-web-servicesamazon-dynamodbamazon-redshifttableau-api
Eric Anastas
Eric Anastas

Reputation: 22213

Which cloud database for logging/metrics visualized with Tableau

I'm developing a C#.NET plug-in for another application which logs the user's activity to a central database. We use this data to troubleshoot performance, and other issues with the software.

I have a current version of this plug-in working which logs to a SQL database within our company network. But now we would also like to be able to provide this plug-in to consultants and other companies working with us who are outside our network. So I'm looking into other options for storing this data in the cloud.

One option would be to to just create a hosted SQL database using Amazon RDS, or some other web host. However, since I'm moving this data to the cloud I started looking at other technologies like Amazon SimpleDB, CloudWatch, and DynamoDB.

I like that posting information to these systems can be done through lightweight HTTP requests, and does not require the overhead of opening and closing database connections. However, we also want to be able to generate reports and visualizations of this data using Tableau, which does not have an option connect directly to SimpleDB, CloudWatch, or DynamoDB.

This logging plug-in does not generate a huge amount of data. The current version have been running for almost two years in our office and the SQL database is a little over 400mb. Currently the largest table is 151,030 rows.

Upvotes: 0

Views: 254

Answers (2)

Eric Anastas
Eric Anastas

Reputation: 22213

After some thought I've decided to stick with our existing relational database but setup an amazon SQS queue as a cache for new log messages which is accessible both internally and externally. I plan to develop a small utility program which can run on a schedule which reads the mesages from the queue into the database.

Upvotes: 1

Chen Harel
Chen Harel

Reputation: 10052

Do you must use a hosted solution? If the data is more log related, perhaps you want a logging (full text search) database such as ElasticSearch (link is to the cloud-aws project).

DynamoDB is not a good fit IMO since your data doesn't feel like K-V storage. I'm not familiar with SimpleDB but for 400MB it feels like MySQL on RDS is really good enough and very easy to bootstrap - so that's my recommendation.

Upvotes: 1

Related Questions