使用自己写的MR程序Bulkload时出现的问题

[the_gunner]

自己写了MR程序生成HFile，然后使用bulkload命令导入HBase的表
在数据量小的时候没什么问题，但是数据量稍大后，导入过程变得比较慢，并且导入过程中HBase管理界面无法查看，并报以下信息


14/06/20 14:01:32 INFO zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x46b795deb9000b, likely server has closed socket, closing socket connection and attempting reconnect
14/06/20 14:01:33 INFO zookeeper.ClientCnxn: Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (无法定位登录配置)

不知是否是HBase宕掉了，但使用jps发现各个节点的有关HBase的进程都还在（HQuorumPeer、HRegionServer）
求教如何解决？

[the_gunner]

另外，虽然发生连接不上HBase的情况，MR job并不会停止

[the_gunner]

查看了一下宕机的regionserver的错误日志，其中宕机前的一段日志是

2014-06-20 14:50:05,624 WARN  [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 15143ms
2014-06-20 14:50:19,068 INFO  [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 3778ms
2014-06-20 14:50:51,934 INFO  [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 7547ms
2014-06-20 14:50:58,050 INFO  [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 3453ms
2014-06-20 14:51:18,298 DEBUG [LruStats #0] hfile.LruBlockCache: Total=1.90 MB, free=402.60 MB, max=404.50 MB, blocks=0, accesses=0, hits=0, hitRatio=0, cachingAccesses=0, cachingHits=0, cachingHitsRatio=0,evictions=0, evicted=0, evictedPerRun=NaN
2014-06-20 14:51:49,857 WARN  [regionserver60020.compactionChecker] util.Sleeper: We slept 27923ms instead of 10000ms, this is likely due to a long garbage collecting pause and it's usually bad, see http://hbase.apache.org/book.html#trouble.rs.runtime.zkexpired
2014-06-20 14:51:49,860 WARN  [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 21199ms
2014-06-20 14:51:49,858 WARN  [regionserver60020.periodicFlusher] util.Sleeper: We slept 27925ms instead of 10000ms, this is likely due to a long garbage collecting pause and it's usually bad, see http://hbase.apache.org/book.html#trouble.rs.runtime.zkexpired
2014-06-20 14:52:15,099 INFO  [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 3436ms
2014-06-20 14:52:34,288 INFO  [regionserver60020-SendThread(u07:2181)] zookeeper.ClientCnxn: Client session timed out, have not heard from server in 62530ms for sessionid 0x646b7dda0af0006, closing socket connection and attempting reconnect
2014-06-20 14:52:34,288 INFO  [regionserver60020-SendThread(u04:2181)] zookeeper.ClientCnxn: Client session timed out, have not heard from server in 84753ms for sessionid 0x146b7dd7e790006, closing socket connection and attempting reconnect
2014-06-20 14:52:34,288 INFO  [regionserver60020-SendThread(u03:2181)] zookeeper.ClientCnxn: Client session timed out, have not heard from server in 69742ms for sessionid 0x346b7dd5ce70003, closing socket connection and attempting reconnect
2014-06-20 14:52:37,330 INFO  [regionserver60020-SendThread(u05:2181)] zookeeper.ClientCnxn: Opening socket connection to server u05/192.168.85.132:2181. Will not attempt to authenticate using SASL (无法定位登录配置)
2014-06-20 14:52:37,366 INFO  [regionserver60020-SendThread(u05:2181)] zookeeper.ClientCnxn: Socket connection established to u05/192.168.85.132:2181, initiating session
2014-06-20 14:52:37,415 INFO  [regionserver60020-SendThread(u05:2181)] zookeeper.ClientCnxn: Opening socket connection to server u05/192.168.85.132:2181. Will not attempt to authenticate using SASL (无法定位登录配置)
2014-06-20 14:52:37,492 INFO  [regionserver60020-SendThread(u05:2181)] zookeeper.ClientCnxn: Socket connection established to u05/192.168.85.132:2181, initiating session
2014-06-20 14:52:37,555 INFO  [regionserver60020-SendThread(u04:2181)] zookeeper.ClientCnxn: Opening socket connection to server u04/192.168.85.131:2181. Will not attempt to authenticate using SASL (无法定位登录配置)
2014-06-20 14:52:37,569 INFO  [regionserver60020-SendThread(u04:2181)] zookeeper.ClientCnxn: Socket connection established to u04/192.168.85.131:2181, initiating session
2014-06-20 14:52:37,919 INFO  [regionserver60020-SendThread(u04:2181)] zookeeper.ClientCnxn: Session establishment complete on server u04/192.168.85.131:2181, sessionid = 0x646b7dda0af0006, negotiated timeout = 90000
2014-06-20 14:53:02,580 WARN  [regionserver60020.periodicFlusher] util.Sleeper: We slept 29465ms instead of 10000ms, this is likely due to a long garbage collecting pause and it's usually bad, see http://hbase.apache.org/book.html#trouble.rs.runtime.zkexpired
2014-06-20 14:53:07,787 WARN  [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 27198ms
2014-06-20 14:53:07,499 INFO  [regionserver60020-SendThread(u05:2181)] zookeeper.ClientCnxn: Client session timed out, have not heard from server in 30168ms for sessionid 0x146b7dd7e790006, closing socket connection and attempting reconnect
2014-06-20 14:53:07,499 INFO  [regionserver60020-SendThread(u05:2181)] zookeeper.ClientCnxn: Client session timed out, have not heard from server in 30084ms for sessionid 0x346b7dd5ce70003, closing socket connection and attempting reconnect
2014-06-20 14:53:07,496 WARN  [regionserver60020.compactionChecker] util.Sleeper: We slept 34024ms instead of 10000ms, this is likely due to a long garbage collecting pause and it's usually bad, see http://hbase.apache.org/book.html#trouble.rs.runtime.zkexpired
2014-06-20 14:53:24,962 INFO  [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 7518ms
2014-06-20 14:53:40,244 INFO  [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 8767ms
2014-06-20 14:53:44,101 WARN  [ResponseProcessor for block BP-2107977179-192.168.85.128-1402023732925:blk_1073755470_16255] hdfs.DFSClient: DFSOutputStream ResponseProcessor exception  for block BP-2107977179-192.168.85.128-1402023732925:blk_1073755470_16255
2014-06-20 14:54:12,415 FATAL [regionserver60020] regionserver.HRegionServer: ABORTING region server u05,60020,1403244375098: org.apache.hadoop.hbase.YouAreDeadException: Server REPORT rejected; currently processing u05,60020,1403244375098 as dead server

是否是跟JVM gc有关呢？

[the_gunner]

求大牛帮助啊~

[wfp458113181wfp]

应该是gc时间过长，导致 dfs 连接异常，然后写hdfs块失败，然后region挂掉。 我猜测，应该是你导入数据量很大，导致region做compaction，最终导致gc时间过长？ 建议你调整参数，关闭自动compaction