CODEWITHSUNDEEP
hadoophdfsapache-zookeeper
schone
schone

Reputation: 572

hadoop logging facility?

If I am to use zookeeper as a work queue and connect to it individual consumers/workers. What would you recommend as a good distributed setup for logging these workers' activities?

Assume the following:

1) At anytime we could be down to 1 single computer housing the hadoop cluster. The system will autoscale up and down as needed but has alot of down time where only 1 single computer is needed.

2) I just need the ability to access all of the workers logs without accessing the individual machine that worker is located at. Bare in mind, that by the time I get to read one of these logs that machine might very well be terminated and long gone.

3) We'll need easy access to the logs i.e being able to cat/grep and tail or alternatively in a more SQLish manner - we'll need real time ability to both query as well as monitor output for short periods of time in real time. (i.e tail -f /var/log/mylog.1)

I appreciate your expert ideas here!

Thanks.

Upvotes: 2

Views: 268

Answers (2)

Kazuki Ohta
Kazuki Ohta

Reputation: 1441

Fluentd log collector just released its WebHDFS plugin, which allows the users to instantly stream data into HDFS. It's really easy to install with ease of management.

enter image description here

Of course you can import data directly from your applications. Here's a Java example to post logs against Fluentd. Fluentd's Java library is clever enough to buffer locally when Fluentd daemon is down. This lessens the possibility of the data loss.

High availability configuration is also available, which basically enables you to have centralized log aggregation system.

Upvotes: 0

ali haider
ali haider

Reputation: 20182

Have you looked at using Flume, chukwa or scribe - ensure that your flume etc process has access to the log files that you are trying to aggregate onto a centralized server.

flume reference: http://archive.cloudera.com/cdh/3/flume/Cookbook/

chukwa: http://incubator.apache.org/chukwa/docs/r0.4.0/admin.html

scribe: https://github.com/facebook/scribe/wiki/_pages

hope it helps.

Upvotes: 1

Related Questions