Capturing networking traffic using Tshark and Flume

Question

Hello everyone i'm trying to capture a network traffic using tshark and i'm using apache flume to send those results to spark .

Problem is when i use the exec source in flume's configuration flume doesn't work it stops instantly right after starting

My configuration :

agent.sources = tsharkSource 
agent.channels = memoryChannel 
agentasinks = avroSink 

# Configuring the sources to pull the bashes from Tshark 
agent.sources.tsharkSource.type = exec 
agent.sources.tsharkSource.command = tshark -T json 
agent.sources.tsharkSource.channels = memoryChannel 

# Configuring the sink to push logs to spark change hostname to 116's ip adress 
agent.sinks.avroSink.type = avro 
agent.sinks.avroSink.channel = memoryChannel 
agent.sinks.avroSink.hostname = 192.168.1.112 
agent.sinks.avroSink.port= 6969 

# Configuring the memory channel 
agent.channels.memoryChannel.type = memory 
agent.channels.memoryChannel.capacity = 1000 
agent.channels.memoryChannel.transactionCapacity = 100 

the shell output :

flume-ng agent -f conf/flume-conf.properties -n agent 

Warning: No configuration directory set! Use --conf <dir> to override. 
Info: Including Hive libraries found via () for Hive access 
+ exec /usr/lib/jum/jaua-1.11.0-openjdk-amd64//bin/jaua -Xmx20m -cp 
1/home/oshiflume/flume/apache-flume-1.8.0-bin/lib/*:/libfle - 
Djava.library.path= org.apache.flume.node.Application -f conf/flume- 
conf.properties -n agent 
log4j:WARN No appenders could be found for logger (org.apache.flume.node.Application). 
log4j:WARN Please initialize the log4j system properly. 
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.htmlDnoconfig for more info. 

Answer 1

Your execution command is

flume-ng agent -f conf/flume-conf.properties -n agent

I see two mistakes here. First, you must specify the configuration directory with -c conf and generally flume configuration files are named as some-config.conf

The warning on the console is No configuration directory set! Use --conf, -c and --conf are the same thing.

You may want to rename your configuration file from flume-conf.properties to flume.conf

As a solution you can try this command:

flume-ng agent -c conf -f conf/flume.conf -n agent 

If you want to display the logs after execution use this command

flume-ng agent -c conf -f conf/flume.conf -n agent -Dflume.root.logger=INFO,console

To display the logs log4j.properties must be in your conf directory as conf/log4j.properties.

My properties are below:

flume.root.logger=INFO,LOGFILE
flume.log.dir=./logs
flume.log.file=flume.log

log4j.logger.org.apache.flume.lifecycle = INFO
log4j.logger.org.jboss = WARN
log4j.logger.org.mortbay = INFO
log4j.logger.org.apache.avro.ipc.NettyTransceiver = WARN
log4j.logger.org.apache.hadoop = INFO
log4j.logger.org.apache.hadoop.hive = ERROR

# Define the root logger to the system property "flume.root.logger".
log4j.rootLogger=${flume.root.logger}

log4j.appender.LOGFILE=org.apache.log4j.RollingFileAppender
log4j.appender.LOGFILE.MaxFileSize=100MB
log4j.appender.LOGFILE.MaxBackupIndex=10
log4j.appender.LOGFILE.File=${flume.log.dir}/${flume.log.file}
log4j.appender.LOGFILE.layout=org.apache.log4j.PatternLayout
log4j.appender.LOGFILE.layout.ConversionPattern=%d{dd MMM yyyy HH:mm:ss,SSS} %-5p [%t] (%C.%M:%L) %x - %m%n

log4j.appender.DAILY=org.apache.log4j.rolling.RollingFileAppender
log4j.appender.DAILY.rollingPolicy=org.apache.log4j.rolling.TimeBasedRollingPolicy
log4j.appender.DAILY.rollingPolicy.ActiveFileName=${flume.log.dir}/${flume.log.file}
log4j.appender.DAILY.rollingPolicy.FileNamePattern=${flume.log.dir}/${flume.log.file}.%d{yyyy-MM-dd}
log4j.appender.DAILY.layout=org.apache.log4j.PatternLayout
log4j.appender.DAILY.layout.ConversionPattern=%d{dd MMM yyyy HH:mm:ss,SSS} %-5p [%t] (%C.%M:%L) %x - %m%n

log4j.appender.console=org.apache.log4j.ConsoleAppender
log4j.appender.console.target=System.err
log4j.appender.console.layout=org.apache.log4j.PatternLayout
log4j.appender.console.layout.ConversionPattern=%d (%t) [%p - %l] %m%n