CODEWITHSUNDEEP
flume
Drizzt321
Drizzt321

Reputation: 1021

Avro sink is removed as being empty

For some reason my Flume 1.2 avro sink is being removed from the configuration as being empty. I've included the agent config and debug output below.

#Flume agent configuration
collector_agent.sources = tailSource1
collector_agent.sinks = avroSink1
collector_agent.channels = memChannel1

collector_agent.sources.tailSource1.type = exec
collector_agent.sources.tailSource1.command = tail -n0 -F /f/f2-traffic/logs/archive_epc.log
collector_agent.sources.tailSource1.channels = memChannel1
collector_agent.sources.tailSource1.batchSize = 5
collector_agent.sources.tailSource1.restart = true

collector_agent.sinks.avroSink1.type = avro
collector_agent.sinks.avroSink1.channels = memChannel1
collector_agent.sinks.avroSink1.hostname = 10.11.14.83
collector_agent.sinks.avroSink1.port = 4545

collector_agent.channels.memChannel1.type = memory
collector_agent.channels.memChannel1.transactionCapacity = 5



#Console log output:
bin/flume-ng agent --conf conf/ --conf-file ../test_epc_config.conf --name collector_agent -Dflume.root.logger=DEBUG,console
+ exec /usr/java/latest/bin/java -Xmx20m -Dflume.root.logger=DEBUG,console -cp '/root/flume_test/apache-flume-1.2.0/conf:/root/flume_test/apache-flume-1.2.0/lib/*' -Djava.library.path= org.apache.flume.node.Application --conf-file ../test_epc_config.conf --name collector_agent
2012-09-06 11:09:26,590 (main) [INFO - org.apache.flume.lifecycle.LifecycleSupervisor.start(LifecycleSupervisor.java:67)] Starting lifecycle supervisor 1
2012-09-06 11:09:26,594 (main) [INFO - org.apache.flume.node.FlumeNode.start(FlumeNode.java:54)] Flume node starting - collector_agent
2012-09-06 11:09:26,597 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.start(DefaultLogicalNodeManager.java:187)] Node manager starting
2012-09-06 11:09:26,599 (lifecycleSupervisor-1-2) [INFO - org.apache.flume.conf.file.AbstractFileConfigurationProvider.start(AbstractFileConfigurationProvider.java:67)] Configuration provider starting
2012-09-06 11:09:26,599 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.lifecycle.LifecycleSupervisor.start(LifecycleSupervisor.java:67)] Starting lifecycle supervisor 10
2012-09-06 11:09:26,600 (lifecycleSupervisor-1-0) [DEBUG - org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.start(DefaultLogicalNodeManager.java:191)] Node manager started
2012-09-06 11:09:26,601 (lifecycleSupervisor-1-2) [DEBUG - org.apache.flume.conf.file.AbstractFileConfigurationProvider.start(AbstractFileConfigurationProvider.java:86)] Configuration provider started
2012-09-06 11:09:26,601 (conf-file-poller-0) [DEBUG - org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run(AbstractFileConfigurationProvider.java:188)] Checking file:../test_epc_config.conf for changes
2012-09-06 11:09:26,601 (conf-file-poller-0) [INFO - org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run(AbstractFileConfigurationProvider.java:195)] Reloading configuration file:../test_epc_config.conf
2012-09-06 11:09:26,606 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:988)] Processing:avroSink1
2012-09-06 11:09:26,607 (conf-file-poller-0) [DEBUG - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:992)] Created context for avroSink1: type
2012-09-06 11:09:26,607 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:988)] Processing:avroSink1
2012-09-06 11:09:26,607 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:902)] Added sinks: avroSink1 Agent: collector_agent
2012-09-06 11:09:26,607 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:988)] Processing:avroSink1
2012-09-06 11:09:26,607 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:988)] Processing:avroSink1
2012-09-06 11:09:26,608 (conf-file-poller-0) [DEBUG - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.isValid(FlumeConfiguration.java:295)] Starting validation of configuration for agent: collector_agent, initial-configuration: AgentConfiguration[collector_agent]
SOURCES: {tailSource1={ parameters:{restart=true, command=tail -n0 -F /f/f2-traffic/logs/archive_epc.log, batchSize=5, channels=memChannel1, type=exec} }}
CHANNELS: {memChannel1={ parameters:{transactionCapacity=5, type=memory} }}
SINKS: {avroSink1={ parameters:{port=4545, hostname=10.11.14.83, channels=memChannel1, type=avro} }}

2012-09-06 11:09:26,613 (conf-file-poller-0) [DEBUG - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.validateChannels(FlumeConfiguration.java:450)] Created channel memChannel1
2012-09-06 11:09:26,626 (conf-file-poller-0) [DEBUG - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.validateSinks(FlumeConfiguration.java:649)] Creating sink: avroSink1 using AVRO
2012-09-06 11:09:26,628 (conf-file-poller-0) [WARN - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.validateSinks(FlumeConfiguration.java:668)] Configuration empty for: avroSink1.Removed.
2012-09-06 11:09:26,629 (conf-file-poller-0) [DEBUG - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.isValid(FlumeConfiguration.java:353)] Post validation configuration for collector_agent
AgentConfiguration created without Configuration stubs for which only basic syntactical validation was performed[collector_agent]
SOURCES: {tailSource1={ parameters:{restart=true, command=tail -n0 -F /f/f2-traffic/logs/archive_epc.log, batchSize=5, channels=memChannel1, type=exec} }}
CHANNELS: {memChannel1={ parameters:{transactionCapacity=5, type=memory} }}

2012-09-06 11:09:26,629 (conf-file-poller-0) [DEBUG - org.apache.flume.conf.FlumeConfiguration.validateConfiguration(FlumeConfiguration.java:117)] Channels:memChannel1

2012-09-06 11:09:26,629 (conf-file-poller-0) [DEBUG - org.apache.flume.conf.FlumeConfiguration.validateConfiguration(FlumeConfiguration.java:118)] Sinks null

2012-09-06 11:09:26,629 (conf-file-poller-0) [DEBUG - org.apache.flume.conf.FlumeConfiguration.validateConfiguration(FlumeConfiguration.java:119)] Sources tailSource1

2012-09-06 11:09:26,629 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration.validateConfiguration(FlumeConfiguration.java:122)] Post-validation flume configuration contains configuration  for agents: [collector_agent]
2012-09-06 11:09:26,629 (conf-file-poller-0) [INFO - org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.loadChannels(PropertiesFileConfigurationProvider.java:249)] Creating channels
2012-09-06 11:09:26,629 (conf-file-poller-0) [DEBUG - org.apache.flume.channel.DefaultChannelFactory.create(DefaultChannelFactory.java:68)] Creating instance of channel memChannel1 type memory
2012-09-06 11:09:26,668 (conf-file-poller-0) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.<init>(MonitoredCounterGroup.java:68)] Monitoried counter group for type: CHANNEL, name: memChannel1, registered successfully.
2012-09-06 11:09:26,669 (conf-file-poller-0) [INFO - org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.loadChannels(PropertiesFileConfigurationProvider.java:273)] created channel memChannel1
2012-09-06 11:09:26,669 (conf-file-poller-0) [DEBUG - org.apache.flume.source.DefaultSourceFactory.create(DefaultSourceFactory.java:74)] Creating instance of source tailSource1, type exec
2012-09-06 11:09:26,678 (conf-file-poller-0) [INFO - org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.startAllComponents(DefaultLogicalNodeManager.java:92)] Starting new configuration:{ sourceRunners:{tailSource1=EventDrivenSourceRunner: { source:org.apache.flume.source.ExecSource@100c62c8 }} sinkRunners:{} channels:{memChannel1=org.apache.flume.channel.MemoryChannel@1d2940b3} }
2012-09-06 11:09:26,679 (conf-file-poller-0) [INFO - org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.startAllComponents(DefaultLogicalNodeManager.java:99)] Starting Channel memChannel1
2012-09-06 11:09:26,679 (conf-file-poller-0) [INFO - org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.startAllComponents(DefaultLogicalNodeManager.java:114)] Waiting for channel: memChannel1 to start. Sleeping for 500 ms
2012-09-06 11:09:26,680 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.start(MonitoredCounterGroup.java:82)] Component type: CHANNEL, name: memChannel1 started
2012-09-06 11:09:27,181 (conf-file-poller-0) [INFO - org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.startAllComponents(DefaultLogicalNodeManager.java:138)] Starting Source tailSource1
2012-09-06 11:09:27,182 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.source.ExecSource.start(ExecSource.java:145)] Exec source starting with command:tail -n0 -F /f/f2-traffic/logs/archive_epc.log
2012-09-06 11:09:27,184 (lifecycleSupervisor-1-0) [DEBUG - org.apache.flume.source.ExecSource.start(ExecSource.java:163)] Exec source started

Upvotes: 0

Views: 1133

Answers (2)

Drizzt321
Drizzt321

Reputation: 1021

Typo based off of copy & paste from the Flume Users Guide on AvroSink. The 

 collector_agent.sinks.avroSink1.channels

should actually have been

  collector_agent.sinks.avroSink1.channel

since a Sink can only have 1 channel.

Upvotes: 1

Michael Percy
Michael Percy

Reputation: 293

As you mentioned, .channel is the right property for sinks.

We committed a patch today to fix all of the broken sink examples (they all contained the same error!): https://issues.apache.org/jira/browse/FLUME-1613

Upvotes: 0

Related Questions