CODEWITHSUNDEEP
databasehadoopbigdatasplunk
E.K.
E.K.

Reputation: 321

Splunk and other DB solutions

I need to write a system that stores large quantities of data, both structured and unstructured. My queries will look for text as well as binary information.

I'm looking for a solution that fits these requirements. I ran into Splunk, which seems to be very user-friendly and offers a nice API. However, it seems very log-oriented, and it seems as though it only supports textual searches (no binary). My data is not a log, it's just lots of raw text/binary data that I need to search in...

So my questions are:

  1. Can Splunk be a good solution to my problem? Is there a way to do binary searches?
  2. Is there perhaps a better solution? Maybe Hadoop (I read about it a little, again, I'm not sure if that's what I'm looking for)?

Thanks!

Upvotes: 0

Views: 351

Answers (1)

Damien Dallimore
Damien Dallimore

Reputation: 204

Splunk will support all sorts of unstructured, semi structured and structured machine data , logs are only 1 such example of machine data. Others would be syslog feeds, JMX/SNMP metrics, output from OS commands,call detail records , clickstream metrics, outputs from proprietary management/monitoring interfaces , social media feeds etc.. the list is vast.

As far as binary data goes ,you need to first roll this out and decode it into text and many folks do actually do this ie: capture binary data of the network ,such as a proprietary protocol, and decode it.

Also , just to note , Splunk doesn't use a database , it collects and indexes data into a simple compressed file format on disk.And then its architecture allows it to scale out to allow for collecting and searching over all this data at large scale.

Upvotes: 1

Related Questions