flume 的hdfs sink效率低下

00点 2015-08-12 04:34:28

我的flume向hdfs中写文件时，效率比较低大约1G/3分钟我单独测试时用脚本put的方式 1分钟能达到8G 如果用file sink也能达到1分钟1G 日志没有任何异常只是DEBUG的时候发现每次提交一个块用时将近半分钟
client.sources = r1
client.channels = c1
client.sinks = k1

client.sources.r1.type = spooldir
client.sources.r1.spoolDir = /var/data/tmpdata
client.sources.r1.fileSuffix = .COMPLETED
client.sources.r1.deletePolicy = never
client.sources.r1.batchSize = 500
client.sources.r1.channels = c1

client.channels.c1.type = memory
client.channels.c1.capacity = 1000000
client.channels.c1.transactionCapacity = 50000
client.channels.c1.keep-alive = 3

client.sinks.k1.type = hdfs
client.sinks.k1.hdfs.path = /flume/events/%Y%m%d/%H
client.sinks.k1.hdfs.useLocalTimeStamp = true
client.sinks.k1.hdfs.rollInterval = 3600
client.sinks.k1.hdfs.rollSize = 1000000000
client.sinks.k1.hdfs.rollCount = 0
client.sinks.k1.hdfs.batchSize = 500
client.sinks.k1.hdfs.callTimeout = 30000
client.sinks.k1.hdfs.fileType = DataStream
client.sinks.k1.channel = c1

12 Aug 2015 16:14:24,739 DEBUG [conf-file-poller-0] (org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run:126) - Checking file:../conf/flume-client.conf for changes
12 Aug 2015 16:14:54,740 DEBUG [conf-file-poller-0] (org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run:126) - Checking file:../conf/flume-client.conf for changes
12 Aug 2015 16:15:24,740 DEBUG [conf-file-poller-0] (org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run:126) - Checking file:../conf/flume-client.conf for changes
12 Aug 2015 16:15:54,741 DEBUG [conf-file-poller-0] (org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run:126) - Checking file:../conf/flume-client.conf for changes
12 Aug 2015 16:16:24,742 DEBUG [conf-file-poller-0] (org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run:126) - Checking file:../conf/flume-client.conf for changes
12 Aug 2015 16:16:54,742 DEBUG [conf-file-poller-0] (org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run:126) - Checking file:../conf/flume-client.conf for changes
12 Aug 2015 16:17:24,743 DEBUG [conf-file-poller-0] (org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run:126) - Checking file:../conf/flume-client.conf for changes
12 Aug 2015 16:17:54,744 DEBUG [conf-file-poller-0] (org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run:126) - Checking file:../conf/flume-client.conf for changes
12 Aug 2015 16:18:24,745 DEBUG [conf-file-poller-0] (org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run:126) - Checking file:../conf/flume-client.conf for changes
12 Aug 2015 16:18:54,746 DEBUG [conf-file-poller-0] (org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run:126) - Checking file:../conf/flume-client.conf for changes
12 Aug 2015 16:19:24,746 DEBUG [conf-file-poller-0] (org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run:126) - Checking file:../conf/flume-client.conf for changes

日志上没有异常

...全文