flume1.6.0-agent多channel 多sink问题

2015-12-16 10:54:46 1.9k 阅读 AI 写同款 · GEO 优化AI 选题数据监测一键分发 ›
u010237107 u010237107 墨衍会员 · AI 创作全网分发 墨衍智能分发 通过墨衍一键同步至各平台 可同步平台 CSDN微信公众号微博知乎掘金百家号博客园抖音小红书 你的文章也可以这样分发 墨衍会员 ¥399起/年，写一次发全网 我也要 + 加入社区
Flume多source，多sink组合框架搭建 Flume多source，多sink组合框架搭建 Flume运行的核心是Agent。它是一个完整的数据收集工具，含有三个核心组件，分别是Source、Channel、Sink。通过这些组件，Event可以从一个地方流向另一个地方。 Source可以接收外部源发送过来的数据。不同的Source可以接受不同的数据格式。 Channel是一个存储地，接收Source的输出，直到有Sink消费掉Chann... 阅读详情
需求是这样的，客户端agent到collect agent，然后collect agent分两路，一路到hdfs 供离线分析，一路经kafka消息中间件到storm实时分析。

现在遇到的情况是，collect agent 单独得写配置到hdfs,或者到kafka 都是没有任何问题的，但唯独合在一起（1个sources，2个channel,2个sink）却出现问题了，查看flume的logs没发现有价值的东西，我认识卡在某个步骤了，非常得匪夷所思啊。

下面先贴上flume logs文件的内容

16 十二月 2015 10:45:57,414 INFO  [lifecycleSupervisor-1-0] (org.apache.flume.node.PollingPropertiesFileConfigurationProvider.start:61)  - Configuration provider starting

16 十二月 2015 10:45:57,427 INFO  [conf-file-poller-0] (org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run:133)  - Reloading configuration file:conf/myconf/flume_colletc_test.conf

16 十二月 2015 10:45:57,435 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1017)  - Processing:k2

16 十二月 2015 10:45:57,436 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1017)  - Processing:k1

16 十二月 2015 10:45:57,436 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1017)  - Processing:k1

16 十二月 2015 10:45:57,436 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1017)  - Processing:k1

16 十二月 2015 10:45:57,436 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1017)  - Processing:k1

16 十二月 2015 10:45:57,437 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1017)  - Processing:k2

16 十二月 2015 10:45:57,437 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1017)  - Processing:k1

16 十二月 2015 10:45:57,437 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1017)  - Processing:k1

16 十二月 2015 10:45:57,437 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1017)  - Processing:k2

16 十二月 2015 10:45:57,437 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1017)  - Processing:k2

16 十二月 2015 10:45:57,437 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1017)  - Processing:k1

16 十二月 2015 10:45:57,438 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1017)  - Processing:k1

16 十二月 2015 10:45:57,438 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1017)  - Processing:k1

16 十二月 2015 10:45:57,438 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1017)  - Processing:k1

16 十二月 2015 10:45:57,440 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1017)  - Processing:k2

16 十二月 2015 10:45:57,440 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1017)  - Processing:k2

16 十二月 2015 10:45:57,441 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1017)  - Processing:k1

16 十二月 2015 10:45:57,441 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1017)  - Processing:k1

16 十二月 2015 10:45:57,441 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:931)  - Added sinks: k2,k1 Agent: collectorMainAgent

16 十二月 2015 10:45:57,443 WARN  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.isValid:337)  - Agent configuration for 'collectorMainAgent' does not contain any valid channels. Marking it as invalid.

16 十二月 2015 10:45:57,443 WARN  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration.validateConfiguration:128)  - Agent configuration invalid for agent 'collectorMainAgent'. It will be removed.

16 十二月 2015 10:45:57,445 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration.validateConfiguration:141)  - Post-validation flume configuration contains configuration for agents: []

16 十二月 2015 10:45:57,445 WARN  [conf-file-poller-0] (org.apache.flume.node.AbstractConfigurationProvider.getConfiguration:133)  - No configuration found for this host:collectorMainAgent

16 十二月 2015 10:45:57,453 INFO  [conf-file-poller-0] (org.apache.flume.node.Application.startAllComponents:138)  - Starting new configuration:{ sourceRunners:{} sinkRunners:{} channels:{} }


然后贴上我 collect的配置文件
# collectorMainAgent

collectorMainAgent.channels = c1,c2

collectorMainAgent.sources  = s2

collectorMainAgent.sinks    =k2,k1



# collectorMainAgent AvroSource

collectorMainAgent.sources.s2.type = avro

collectorMainAgent.sources.s2.bind = 172.21.99.134

collectorMainAgent.sources.s2.port = 44444

collectorMainAgent.sources.s2.channels = c1,c2





# defined for kafka channel by Cesar.X 2015/12/16

collectorMainAgent.channels.c1.type = memory

collectorMainAgent.channels.c1.capacity = 1000

collectorMainAgent.channels.c1.transactionCapacity = 100



#定义flume-kafka by Cesar.X 2015/12/15

collectorMainAgent.sinks.k2.type = org.apache.flume.sink.kafka.KafkaSink

collectorMainAgent.sinks.k2.topic = test-topic

collectorMainAgent.sinks.k2.brokerList = master:9092,slave1:9092,slave2:9092

collectorMainAgent.sinks.k2.requiredAcks = 1

collectorMainAgent.sinks.k2.batchSize = 20

collectorMainAgent.sinks.k2.channel = c1





# collectorMainAgent FileChannel

#defined for hdfs by Cesar.X 2015/12/16

collectorMainAgent.channels.c2.type = file

collectorMainAgent.channels.c2.checkpointDir =/usr/local/hadoop/apache-flume-1.6.0-bin/fchannel/spool/checkpoint

collectorMainAgent.channels.c2.dataDirs = /usr/local/hadoop/apache-flume-1.6.0-bin/fchannel/spool/data

#collectorMainAgent.channels.c2.capacity = 200000000

collectorMainAgent.channels.c2.capacity = 1000

collectorMainAgent.channels.c2.transactionCapacity=6000

collectorMainAgent.channels.c2.checkpointInterval=60000



collectorMainAgent.sinks.k1.type = hdfs

collectorMainAgent.sinks.k1.channel = c2

collectorMainAgent.sinks.k1.hdfs.path =  hdfs://mycluster/user/flume/events/%Y-%m-%d

collectorMainAgent.sinks.k1.hdfs.filePrefix = events

#collectorMainAgent.sinks.k1.hdfs.round = true

#collectorMainAgent.sinks.k1.hdfs.roundValue = 10

#collectorMainAgent.sinks.k1.hdfs.roundUnit = minute

# Number of seconds to wait before rolling current file (0 = never roll based on time interval)

collectorMainAgent.sinks.k1.hdfs.rollInterval=600

# File size to trigger roll, in bytes (256Mb)，对于hdfs来说，应该尽量避免小文件的产生

# 这边有很多触发器，依次满足，先满足hdfs 文件大小256MB分割

collectorMainAgent.sinks.k1.hdfs.rollSize = 268435456

# never roll based on number of events（0 = never roll based on number of events）

collectorMainAgent.sinks.k1.hdfs.rollCount = 0

# Timeout after which inactive files get closed (0 = disable automatic closing of idle files)

# 持续60分钟没动作分割

collectorMainAgent.sinks.k1.hdfs.idleTimeout = 3600

collectorMainAgent.sinks.k1.hdfs.writeFormat = Text

collectorMainAgent.sinks.k1.hdfs.inUseSuffix =.txt

collectorMainAgent.sinks.k1.hdfs.fileType = DataStream

#collectorMainAgent.sinks.HDFS.hdfs.batchSize = 1000

#必须有时间戳，否则会报错

collectorMainAgent.sinks.k1.hdfs.useLocalTimeStamp  = true


分就这么多，希望大家谅解！！
...全文
Flume基础操作，自定义Sink Flume基础操作，自定义Sink 1. 配置一个flume agent 任务目标： source为 exec source，用tailf命令，监控文件任意 channel设置为filechannel sink为hdfs sink 任务完成步骤 创建Flume agent配置文件，配置source、sink、channel。 配置source type 为exec，tail -f监... 阅读详情

相关推荐

flume1.7.0源码

常见的 Sink Processor 有 SimpleSinkProcessor（单个 Sink 处理）和 ParallelSinkProcessor（多 Sink 并行处理）。6.

66进

Flume集群环境搭建，flume监控

Flume从1.5.0版本开始，重构了其内部架构，核心组件、配置以及代码架构都进行了重大改动，这个新版本被称为Flume NG（Next Generation），即Flume的新一代版本，用以替代了原来的

chiweitree

apache-flume-1.9.0-bin.tar.gz

.type = file_rollagent.sinks.sink1.channel = channel1agent.sinks.sink1.path = /path/to/output/directoryagent.sinks.sink1

快乐骑行^_^

Flume1.6.0入门：安装、部署、及flume的案例

/closer.lua/flume/1.6.0/apache-flume-1.6.0-bin.tar.gz)**安装步骤**：1.

qqqwers

Flume1.5.0入门：安装、部署、及flume的案例

**安装 Flume 1.5.0**1. 下载 Apache Flume 1.5.0 从官方网站（http://flume.apache.org/）的下载页面。2.

lee24
评论 6
收藏 已收藏
分享
返回顶部
u010237107
u010237107
Hadoop生态社区 20,842 成员 4,695 内容
+ 加入社区