flume1.6.0-agent多channel 多sink问题

[u010237107]

需求是这样的，客户端agent到collect agent，然后collect agent分两路，一路到hdfs 供离线分析，一路经kafka消息中间件到storm实时分析。

现在遇到的情况是，collect agent 单独得写配置到hdfs,或者到kafka 都是没有任何问题的，但唯独合在一起（1个sources，2个channel,2个sink）却出现问题了，查看flume的logs没发现有价值的东西，我认识卡在某个步骤了，非常得匪夷所思啊。

下面先贴上flume logs文件的内容

16 十二月 2015 10:45:57,414 INFO  [lifecycleSupervisor-1-0] (org.apache.flume.node.PollingPropertiesFileConfigurationProvider.start:61)  - Configuration provider starting
16 十二月 2015 10:45:57,427 INFO  [conf-file-poller-0] (org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run:133)  - Reloading configuration file:conf/myconf/flume_colletc_test.conf
16 十二月 2015 10:45:57,435 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1017)  - Processing:k2
16 十二月 2015 10:45:57,436 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1017)  - Processing:k1
16 十二月 2015 10:45:57,436 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1017)  - Processing:k1
16 十二月 2015 10:45:57,436 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1017)  - Processing:k1
16 十二月 2015 10:45:57,436 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1017)  - Processing:k1
16 十二月 2015 10:45:57,437 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1017)  - Processing:k2
16 十二月 2015 10:45:57,437 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1017)  - Processing:k1
16 十二月 2015 10:45:57,437 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1017)  - Processing:k1
16 十二月 2015 10:45:57,437 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1017)  - Processing:k2
16 十二月 2015 10:45:57,437 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1017)  - Processing:k2
16 十二月 2015 10:45:57,437 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1017)  - Processing:k1
16 十二月 2015 10:45:57,438 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1017)  - Processing:k1
16 十二月 2015 10:45:57,438 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1017)  - Processing:k1
16 十二月 2015 10:45:57,438 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1017)  - Processing:k1
16 十二月 2015 10:45:57,440 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1017)  - Processing:k2
16 十二月 2015 10:45:57,440 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1017)  - Processing:k2
16 十二月 2015 10:45:57,441 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1017)  - Processing:k1
16 十二月 2015 10:45:57,441 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1017)  - Processing:k1
16 十二月 2015 10:45:57,441 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:931)  - Added sinks: k2,k1 Agent: collectorMainAgent
16 十二月 2015 10:45:57,443 WARN  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.isValid:337)  - Agent configuration for 'collectorMainAgent' does not contain any valid channels. Marking it as invalid.
16 十二月 2015 10:45:57,443 WARN  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration.validateConfiguration:128)  - Agent configuration invalid for agent 'collectorMainAgent'. It will be removed.
16 十二月 2015 10:45:57,445 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration.validateConfiguration:141)  - Post-validation flume configuration contains configuration for agents: []
16 十二月 2015 10:45:57,445 WARN  [conf-file-poller-0] (org.apache.flume.node.AbstractConfigurationProvider.getConfiguration:133)  - No configuration found for this host:collectorMainAgent
16 十二月 2015 10:45:57,453 INFO  [conf-file-poller-0] (org.apache.flume.node.Application.startAllComponents:138)  - Starting new configuration:{ sourceRunners:{} sinkRunners:{} channels:{} }

然后贴上我 collect的配置文件

# collectorMainAgent
collectorMainAgent.channels = c1,c2
collectorMainAgent.sources  = s2
collectorMainAgent.sinks    =k2,k1

# collectorMainAgent AvroSource
collectorMainAgent.sources.s2.type = avro
collectorMainAgent.sources.s2.bind = 172.21.99.134
collectorMainAgent.sources.s2.port = 44444
collectorMainAgent.sources.s2.channels = c1,c2


# defined for kafka channel by Cesar.X 2015/12/16
collectorMainAgent.channels.c1.type = memory
collectorMainAgent.channels.c1.capacity = 1000
collectorMainAgent.channels.c1.transactionCapacity = 100

#定义flume-kafka by Cesar.X 2015/12/15
collectorMainAgent.sinks.k2.type = org.apache.flume.sink.kafka.KafkaSink
collectorMainAgent.sinks.k2.topic = test-topic
collectorMainAgent.sinks.k2.brokerList = master:9092,slave1:9092,slave2:9092
collectorMainAgent.sinks.k2.requiredAcks = 1
collectorMainAgent.sinks.k2.batchSize = 20
collectorMainAgent.sinks.k2.channel = c1


# collectorMainAgent FileChannel
#defined for hdfs by Cesar.X 2015/12/16
collectorMainAgent.channels.c2.type = file
collectorMainAgent.channels.c2.checkpointDir =/usr/local/hadoop/apache-flume-1.6.0-bin/fchannel/spool/checkpoint
collectorMainAgent.channels.c2.dataDirs = /usr/local/hadoop/apache-flume-1.6.0-bin/fchannel/spool/data
#collectorMainAgent.channels.c2.capacity = 200000000
collectorMainAgent.channels.c2.capacity = 1000
collectorMainAgent.channels.c2.transactionCapacity=6000
collectorMainAgent.channels.c2.checkpointInterval=60000

collectorMainAgent.sinks.k1.type = hdfs
collectorMainAgent.sinks.k1.channel = c2
collectorMainAgent.sinks.k1.hdfs.path =  hdfs://mycluster/user/flume/events/%Y-%m-%d
collectorMainAgent.sinks.k1.hdfs.filePrefix = events
#collectorMainAgent.sinks.k1.hdfs.round = true
#collectorMainAgent.sinks.k1.hdfs.roundValue = 10
#collectorMainAgent.sinks.k1.hdfs.roundUnit = minute
# Number of seconds to wait before rolling current file (0 = never roll based on time interval)
collectorMainAgent.sinks.k1.hdfs.rollInterval=600
# File size to trigger roll, in bytes (256Mb)，对于hdfs来说，应该尽量避免小文件的产生
# 这边有很多触发器，依次满足，先满足hdfs 文件大小256MB分割
collectorMainAgent.sinks.k1.hdfs.rollSize = 268435456
# never roll based on number of events（0 = never roll based on number of events）
collectorMainAgent.sinks.k1.hdfs.rollCount = 0
# Timeout after which inactive files get closed (0 = disable automatic closing of idle files)
# 持续60分钟没动作分割
collectorMainAgent.sinks.k1.hdfs.idleTimeout = 3600
collectorMainAgent.sinks.k1.hdfs.writeFormat = Text
collectorMainAgent.sinks.k1.hdfs.inUseSuffix =.txt
collectorMainAgent.sinks.k1.hdfs.fileType = DataStream
#collectorMainAgent.sinks.HDFS.hdfs.batchSize = 1000
#必须有时间戳，否则会报错
collectorMainAgent.sinks.k1.hdfs.useLocalTimeStamp  = true

分就这么多，希望大家谅解！！

[u010237107]

2B了，大意了，是逗号的问题！！！！

[夜无边CN]

引用 1 楼 u010237107 的回复:

2B了，大意了，是逗号的问题！！！！

[人生偌只如初见]

其实也挺好的，算是一个分享。

[_游侠]

该是空格。

[zhtwave]

[a13575018718]

你这个是哪里改成空格啊 哪个地方逗号出错了啊