HBase集群HMaster无法启动？

[加叶]

搭建HBase集群，Hadoop等启动正常，单HMaster无法启动，HBase日志如下所示，请教大神，这是什么问题？(版本是对应的，不是版本匹配问题,hadoop-2.0.0-cdh4.3+hbase-0.94.6-cdh4.3)

2015-09-07 20:28:53,129 WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node /hbase/backup-masters/xxxx-vm1,61000,1441628932780 already deleted, and this is not a retry
2015-09-07 20:28:53,129 INFO org.apache.hadoop.hbase.master.ActiveMasterManager: Master=xxxx-vm1,61000,1441628932780
2015-09-07 20:28:53,615 WARN org.apache.hadoop.conf.Configuration: fs.default.name is deprecated. Instead, use fs.defaultFS
2015-09-07 20:28:53,619 INFO org.apache.hadoop.hbase.master.SplitLogManager: timeout = 300000
2015-09-07 20:28:53,619 INFO org.apache.hadoop.hbase.master.SplitLogManager: unassigned timeout = 180000
2015-09-07 20:28:53,619 INFO org.apache.hadoop.hbase.master.SplitLogManager: resubmit threshold = 3
2015-09-07 20:28:53,624 INFO org.apache.hadoop.hbase.master.SplitLogManager: found 0 orphan tasks and 0 rescan nodes
2015-09-07 20:28:53,730 FATAL org.apache.hadoop.hbase.master.HMaster: Unhandled exception. Starting shutdown.
org.apache.hadoop.ipc.RemoteException(java.lang.NullPointerException): java.lang.NullPointerException
	at org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager.sortLocatedBlocks(DatanodeManager.java:308)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1245)
	at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:392)
	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:172)
	at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java:44938)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:453)
	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1002)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1701)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1697)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:415)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1695)

	at org.apache.hadoop.ipc.Client.call(Client.java:1225)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:202)
	at com.sun.proxy.$Proxy12.getBlockLocations(Unknown Source)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:164)
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:83)
	at com.sun.proxy.$Proxy12.getBlockLocations(Unknown Source)
	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getBlockLocations(ClientNamenodeProtocolTranslatorPB.java:155)
	at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:970)
	at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:960)
	at org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:238)
	at org.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStream.java:205)
	at org.apache.hadoop.hdfs.DFSInputStream.<init>(DFSInputStream.java:198)
	at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1117)
	at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:249)
	at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:82)
	at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:746)
	at org.apache.hadoop.hbase.util.FSUtils.getVersion(FSUtils.java:287)
	at org.apache.hadoop.hbase.util.FSUtils.checkVersion(FSUtils.java:328)
	at org.apache.hadoop.hbase.master.MasterFileSystem.checkRootDir(MasterFileSystem.java:368)
	at org.apache.hadoop.hbase.master.MasterFileSystem.createInitialFileSystemLayout(MasterFileSystem.java:134)
	at org.apache.hadoop.hbase.master.MasterFileSystem.<init>(MasterFileSystem.java:119)
	at org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:536)
	at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:395)
	at java.lang.Thread.run(Thread.java:745)
2015-09-07 20:28:53,733 INFO org.apache.hadoop.hbase.master.HMaster: Aborting
2015-09-07 20:28:53,733 DEBUG org.apache.hadoop.hbase.master.HMaster: Stopping service threads
2015-09-07 20:28:53,733 INFO org.apache.hadoop.ipc.HBaseServer: Stopping server on 61000
2015-09-07 20:28:53,733 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 0 on 61000: exiting
2015-09-07 20:28:53,733 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 3 on 61000: exiting
2015-09-07 20:28:53,733 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 1 on 61000: exiting
2015-09-07 20:28:53,734 INFO org.apache.hadoop.ipc.HBaseServer: REPL IPC Server handler 0 on 61000: exiting
2015-09-07 20:28:53,734 INFO org.apache.hadoop.ipc.HBaseServer: REPL IPC Server handler 1 on 61000: exiting
2015-09-07 20:28:53,734 INFO org.apache.hadoop.ipc.HBaseServer: REPL IPC Server handler 2 on 61000: exiting
2015-09-07 20:28:53,734 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 2 on 61000: exiting
2015-09-07 20:28:53,734 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 5 on 61000: exiting
2015-09-07 20:28:53,734 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 6 on 61000: exiting
2015-09-07 20:28:53,734 INFO org.apache.hadoop.ipc.HBaseServer: Stopping IPC Server listener on 61000
2015-09-07 20:28:53,734 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 7 on 61000: exiting
2015-09-07 20:28:53,734 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 8 on 61000: exiting
2015-09-07 20:28:53,735 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 4 on 61000: exiting
2015-09-07 20:28:53,735 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 9 on 61000: exiting
2015-09-07 20:28:53,735 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 11 on 61000: exiting
2015-09-07 20:28:53,735 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 10 on 61000: exiting
2015-09-07 20:28:53,735 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 12 on 61000: exiting
2015-09-07 20:28:53,735 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 13 on 61000: exiting
2015-09-07 20:28:53,735 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 15 on 61000: exiting
2015-09-07 20:28:53,735 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 14 on 61000: exiting
2015-09-07 20:28:53,735 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 16 on 61000: exiting
2015-09-07 20:28:53,736 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 19 on 61000: exiting
2015-09-07 20:28:53,736 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 17 on 61000: exiting
2015-09-07 20:28:53,736 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 18 on 61000: exiting
2015-09-07 20:28:53,737 INFO org.apache.hadoop.ipc.HBaseServer: Stopping IPC Server Responder
2015-09-07 20:28:53,737 INFO org.apache.hadoop.ipc.HBaseServer: Stopping IPC Server Responder
2015-09-07 20:28:53,758 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down
2015-09-07 20:28:53,758 INFO org.apache.zookeeper.ZooKeeper: Session: 0x34fa780fa7a000f closed
2015-09-07 20:28:53,758 INFO org.apache.hadoop.hbase.master.HMaster: HMaster main thread exiting
2015-09-07 20:28:53,758 ERROR org.apache.hadoop.hbase.master.HMasterCommandLine: Failed to start master
java.lang.RuntimeException: HMaster Aborted
	at org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:160)
	at org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:104)
	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
	at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:76)
	at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:1958)

[skyWalker_ONLY]

有可能是版本不一致的问题，建议用hadoop的jar包替换hbase lib目录中的hadoop相关jar包试试

[alinly]

可能是zk上的数据有问题，全部删除，重新启动.

[iamzealoot]

zk目前是启动的么？如果是先关掉zk试一下

[历史哭泣]

时间不一致，Namenode和所有DataNode时间要一致，不能差别太大。

[Kyle_Joe]

如果时间同步或者zk都没有问题的话，试试以下几个办法： 1.删除Hbase 的tmp文件夹重启（每个节点都要操作）或者删除hdfs上hbase存储的数据 2.hadoop 集群进入了safe model 模式，需要执行hadoop dfsadmin -safemode leave退出安全模式 3.存储在Hbase的数据有丢失，需要利用hadoop的回收站的机制恢复数据，或者删除HBase的数据 我前几天也遇到这个问题，最后看到几个办法解决的。