CODEWITHSUNDEEP
hadoop
Ravi
Ravi

Reputation: 3303

Datameer for Real Time Querying

We are currently interested in evaluating datameer and have a few questions. Are there any datameer users that can answer these questions:

  1. Since datameer works off HDFS, are the querying speeds similar to that of Hive? How does the querying speed compare with columnar databases?

  2. Since Hadoop is known for high latency, is it advisable to use datameer for real time quering?

Thank you.

Ravi

Upvotes: 0

Views: 977

Answers (1)

Peter Voss
Peter Voss

Reputation: 76

  1. Regarding 1:
    • Query speeds are comparable to Hive.
    • But Datameer is a lot faster in the design phase of your "query". Datameer provides a real time preview how the results of your "query" would look like, which is happening in memory and not on the cluster. The preview is based on a representative sample of your data. It's only a preview not the final results, but it gives you constant feedback if your analytics make sense while designing. To test a Hive query you would have to execute it, which makes the design process very slow.
    • Datameer's big advantage over Hive is:
      1. Loading data into Hadoop is much easier. No static schema creation, no ETL, etc. Just use a wizard to download data from your database, log files, social media, etc.
      2. Designing analytics or making changes is a lot faster and can even be done by non technical users.
      3. No need to install anything else since Datameer includes all you need for importing, analytics, scheduling, security, visualization etc. in one product
  2. If you have real time requirements you should not pull data directly out of Datameer, Hive, Impala, etc.. Columnar storages make some processing faster but will still not be low latency. But you can use those tools together with a low latency database. Use Datameer/Hive/Impala for the heavy lifting to filter and pre aggregate big data into smaller data and then export that out into a database. In Datameer you could set this up very easily using one of Datameer's wizards.

Hope this helps,

Peter Voß (Datameer)

Upvotes: 6

Related Questions