CODEWITHSUNDEEP
amazon-web-servicesschemaamazon-dynamodbnosql
s_dolan
s_dolan

Reputation: 1276

DynamoDB Schema Design

I'm thinking of using Amazon AWS DynamoDB for a project that I'm working on. Here's the gist of the situation:

I'm going to be gathering a ton of energy usage data for hundreds of machines (energy readings are taken around every 5 minutes). Each machine is in a zone, and each zone is in a network.

I'm then going to roll up these individual readings by zone and network, by hour and day.

My thinking is that by doing this, I'll be able to perform one query against DynamoDB on the network_day table, and return the energy usage for any given day quickly.

Here's my schema at this point:

table_name      | hash_key   | range_key  | attributes
______________________________________________________
machine_reading | machine.id | epoch      | energy_use
machine_hour    | machine.id | epoch_hour | energy_use
machine_day     | machine.id | epoch_day  | energy_use
zone_hour       | machine.id | epoch_hour | energy_use
zone_day        | machine.id | epoch_day  | energy_use
network_hour    | machine.id | epoch_hour | energy_use
network_day     | machine.id | epoch_day  | energy_use

I'm not immediately seeing that great of performance in tests when I run the rollup cronjob, so I'm just wondering if someone with more experience could comment on my key design? The only experience I have so far is with RDS, but I'm very much trying to learn about DynamoDB.

EDIT:

Basic structure for the cronjob that I'm using for rollups:

foreach network
  foreach zone
    foreach machine
      add_unprocessed_readings_to_dynamo()
      roll_up_fixture_hours_to_dynamo()
      roll_up_fixture_days_to_dynamo()
    end
    roll_up_zone_hours_to_dynamo()
    roll_up_zone_days_to_dynamo()
  end
  roll_up_network_hours_to_dynamo()
  roll_up_network_days_to_dynamo()
end

I use the previous function's values in Dynamo for the next roll up, i.e.

This is what (I think) is causing a lot of unnecessary reads/writes. Right now I can manage with low throughputs because my sample size is only 100 readings. My concerns begin when this scales to what is expected to contain around 9,000,000 readings.

Upvotes: 1

Views: 1404

Answers (2)

JaredHatfield
JaredHatfield

Reputation: 6651

First things first, time series data in DynamoDB is hard to do right, but not impossible.

DynamoDB uses the hash key to shard the data so using the machine.id means that some of you are going to have hot keys. However, this is really a function of the amount of data and what you expect your IOPS to be. DynamoDB doesn't create a 2nd shard until you push past 1000 read or write IOPS. If you expect to be well below that level you may be fine, but if you expect to scale beyond that then you may want to redesign, specifically include a date component in your hash key to break things up.

Regarding performance, are you hitting your provisioned read or write throughput level? If so raise them to some crazy high level and re-run the test until the bottleneck becomes your code. This could be a simple as setting the throughput level appropriately.

However, regarding your actual code, without seeing the actual DynamoDB queries you are performing a possible issue would be reading too much data. Make sure you are not reading more data than you need from DynamoDB. Since your range key is a date field use a range conditional (not a filter) to reduce the number of records you need to read.

Make sure you code executes the rollup using multiple threads. If you are not able to saturate the DynamoDB provisioned capacity the issue may not be DynamoDB, it may be your code. By performing the rollups using multiple threads in parallel you should be able to see some performance gains.

Upvotes: 3

Mircea
Mircea

Reputation: 10566

What's the provisioned throughput on the tables you are using? how are you performing the rollup? Are you reading everything and filtering / filtering on range keys, etc?

Do you need to roll up/a cron job in this situation?

Why not use a table for the readings machine_reading | machine.id | epoch_timestamp | energy_use

and a table for the aggregates hash_key can be aggregate type and range key can be aggregate name example: zone, zone1 zone, zone3 day, 03/29/1940

when getting machine data, dump it in the first table and after that use atomic counters to increment entities in 2nd table: http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/WorkingWithItems.html#WorkingWithItems.AtomicCounters

Upvotes: 2

Related Questions