CODEWITHSUNDEEP
mongodbhadoophbasehdfs
Eli
Eli

Reputation: 39009

HBase or Mongo for an Analytics DB if already using Hadoop?

I currently have a Hadoop cluster where I store tons of logs over which I run pig scripts for calculating aggregated analytics. I also have a Mongo cluster where I store production data.

I've recently been put in a position where I need to do a lot of one-off analytics queries, or enable others to do them. These queries frequently need to use both production data and log data together, so whatever I go with, I'd like to have everything in one place. My log data is in json and about 10x the size of my prod data. Here are the pros/cons of Mongo and HBase I'm seeing:

Mongo Pros/ HBase Cons:

  1. Since log data is in JSON, I can get it into Mongo pretty easily, and I can do this in real time as it comes in through something like FluentD.
  2. Most people I work with already have experience writing Mongo queries from needing to work with prod data, so getting an analytics db up on Mongo would be very simple for everyone to use.
  3. I know much less about Hbase than Mongo.
  4. No idea how easy/difficult it would be to get data in JSON or from Mongo into Hbase. I imagine this isn't so bad, but I don't see much documentation.

HBase Pros/Mongo Cons:

  1. My log data is much bigger than my prod data, so storing it in both hadoop and mongo would be way more expensive than storing my prod data in both hadoop and mongo.
  2. I can build HBase on top of my already running Hadoop cluster and fit my prod data in there without adding many extra machines. If I went with Mongo, I'd need a whole new Mongo cluster.
  3. I could use Phoenix on top of Hbase to allow a simple SQL syntax for accessing all our data, but I'm not sure how unwieldily this would be for multi-level document-based data.

I know very little about Hbase currently, and I wouldn't consider myself a Mongo expert, so I'm probably missing a lot.

So, what am I missing, and which is right for my situation?

Upvotes: 2

Views: 6224

Answers (2)

user2078148
user2078148

Reputation: 601

First of all, you should use something which you already can handle. Therefore, Mongo DB seems a good choice, especially when the data is already in the json format.

On the other hand, I used HBase quite a while and the read performance is amazing although having a lot of rows and I really don't know if there is any good and fast integration of Mongo DB with Hadoop. HBase is the Hadoop database, so it is predestinated to work with Hadoop together.

If the logs could be indexed by (in the HBase Rowkey):

producing_program_identifier, timestamp, ...

HBase could work quite well for this query pattern. But if you decide on HBase, use the phoenix framwork, it will save you time using familiar interfaces like jdbc and sql-like queries. It also provides simple aggregation functions (count, avg, max, min) which may be sufficient.

Upvotes: 2

Arnon Rotem-Gal-Oz
Arnon Rotem-Gal-Oz

Reputation: 25939

From what you're saying it seems a mongoDB based solution would work best for you.

HBase is extremely versatile and you can get it to serve both your prod needs as well as your analytics needs however the general purpose SQL capabilities (in Phoenix, Cloudera's Impala and others) are in their infancy and the standard HBase way to get high query performance (designing the data structure for reads) will take a lot on effort (esp. since you don't have experience in HBase).

By the way it may be applicable for you to use map/reduces pre-aggregated data and then load it into MongoDB and thus utilize your current setup bette rather than change it either way

Upvotes: 0

Related Questions