hbase启动catalog.CatalogTracker: Fail ed verification of hbase:meta

[念来过倒蛋笨]

hbase日志：
2014-12-01 12:53:06,318 INFO [master:Master:60000] master.MasterFileSystem: Log
folder hdfs://10.1.51.205:9000/hbase/WALs/Slave1,60020,1417409582139 belongs to
an existing region server
2014-12-01 12:53:06,319 INFO [master:Master:60000] master.MasterFileSystem: Log
folder hdfs://10.1.51.205:9000/hbase/WALs/Slave3,60020,1417409580604 belongs to
an existing region server
2014-12-01 12:53:06,319 INFO [master:Master:60000] master.MasterFileSystem: Log
folder hdfs://10.1.51.205:9000/hbase/WALs/Slave4,60020,1417409580580 belongs to
an existing region server
2014-12-01 12:53:06,412 INFO [master:Master:60000] catalog.CatalogTracker: Fail
ed verification of hbase:meta,,1 at address=Slave3,60020,1417409322062, exceptio
n=org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.Not
ServingRegionException: Region hbase:meta,,1 is not online on Slave3,60020,14174
09580604
at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncoded
Name(HRegionServer.java:2695)
at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionS
erver.java:4139)
at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionInfo(HReg
ionServer.java:3506)
at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2
.callBlockingMethod(AdminProtos.java:20158)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2026)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:98)
at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java
:114)
at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94)
at java.lang.Thread.run(Thread.java:724)

2014-12-01 12:53:06,416 INFO [master:Master:60000] master.MasterFileSystem: Log
dir for server Slave3,60020,1417409322062 does not exist
2014-12-01 12:53:06,416 INFO [master:Master:60000] master.SplitLogManager: dead
splitlog workers [Slave3,60020,1417409322062]
2014-12-01 12:53:06,416 DEBUG [master:Master:60000] master.SplitLogManager: Sche
duling batch of logs to split
2014-12-01 12:53:06,417 INFO [master:Master:60000] master.SplitLogManager: star
ted splitting 0 logs in []
2014-12-01 12:53:06,417 INFO [master:Master:60000] master.SplitLogManager: fini
shed splitting (more than or equal to) 0 bytes in 0 log files in [] in 0ms
2014-12-01 12:53:06,418 INFO [master:Master:60000] zookeeper.ZooKeeperNodeTrack
er: Unsetting hbase:meta region location in ZooKeeper
2014-12-01 12:53:06,441 DEBUG [master:Master:60000] master.AssignmentManager: No
previous transition plan found (or ignoring an existing plan) for hbase:meta,,1
.1588230740; generated random plan=hri=hbase:meta,,1.1588230740, src=, dest=Slav
e3,60020,1417409580604; 3 (online=3, available=3) available servers, forceNewPla
n=false
2014-12-01 12:53:06,441 DEBUG [master:Master:60000] zookeeper.ZKAssign: master:6
0000-0x34a042da2e20004, quorum=10.1.51.204:2181,10.1.51.203:2181,10.1.51.201:218
1,10.1.51.205:2181, baseZNode=/hbase Creating (or updating) unassigned node 1588
230740 with OFFLINE state
2014-12-01 12:53:06,466 INFO [master:Master:60000] master.AssignmentManager: As
signing hbase:meta,,1.1588230740 to Slave3,60020,1417409580604
2014-12-01 12:53:06,466 INFO [master:Master:60000] master.RegionStates: Transit
ioned {1588230740 state=OFFLINE, ts=1417409586441, server=null} to {1588230740 s
tate=PENDING_OPEN, ts=1417409586466, server=Slave3,60020,1417409580604}
2014-12-01 12:53:06,466 DEBUG [master:Master:60000] master.ServerManager: New ad
min connection to Slave3,60020,1417409580604
2014-12-01 12:53:06,600 INFO [master:Master:60000] master.ServerManager: Assign
mentManager hasn't finished failover cleanup; waiting
2014-12-01 12:53:06,642 DEBUG [AM.ZK.Worker-pool2-t1] master.AssignmentManager:
Handling RS_ZK_REGION_OPENING, server=Slave3,60020,1417409580604, region=1588230
740, current_state={1588230740 state=PENDING_OPEN, ts=1417409586466, server=Slav
e3,60020,1417409580604}
2014-12-01 12:53:06,642 INFO [AM.ZK.Worker-pool2-t1] master.RegionStates: Trans
itioned {1588230740 state=PENDING_OPEN, ts=1417409586466, server=Slave3,60020,14
17409580604} to {1588230740 state=OPENING, ts=1417409586642, server=Slave3,60020
,1417409580604}
2014-12-01 12:53:06,833 DEBUG [AM.ZK.Worker-pool2-t2] master.AssignmentManager:
Handling RS_ZK_REGION_OPENED, server=Slave3,60020,1417409580604, region=15882307
40, current_state={1588230740 state=OPENING, ts=1417409586642, server=Slave3,600
20,1417409580604}
2014-12-01 12:53:06,833 INFO [AM.ZK.Worker-pool2-t2] master.RegionStates: Trans
itioned {1588230740 state=OPENING, ts=1417409586642, server=Slave3,60020,1417409
580604} to {1588230740 state=OPEN, ts=1417409586833, server=Slave3,60020,1417409
580604}
2014-12-01 12:53:06,836 INFO [AM.ZK.Worker-pool2-t2] handler.OpenedRegionHandle
r: Handling OPENED of 1588230740 from Slave3,60020,1417409580604; deleting unass
igned node
2014-12-01 12:53:06,856 DEBUG [AM.ZK.Worker-pool2-t2] zookeeper.ZKAssign: master
:60000-0x34a042da2e20004, quorum=10.1.51.204:2181,10.1.51.203:2181,10.1.51.201:2
181,10.1.51.205:2181, baseZNode=/hbase Deleted unassigned node 1588230740 in exp
ected state RS_ZK_REGION_OPENED
2014-12-01 12:53:06,857 DEBUG [AM.ZK.Worker-pool2-t3] master.AssignmentManager:
Znode hbase:meta,,1.1588230740 deleted, state: {1588230740 state=OPEN, ts=141740
9586833, server=Slave3,60020,1417409580604}
2014-12-01 12:53:06,857 INFO [AM.ZK.Worker-pool2-t3] master.RegionStates: Onlin
ed 1588230740 on Slave3,60020,1417409580604
2014-12-01 12:53:06,860 INFO [master:Master:60000] master.HMaster: hbase:meta a
ssigned=1, rit=false, location=Slave3,60020,1417409580604
2014-12-01 12:53:06,935 INFO [master:Master:60000] catalog.MetaMigrationConvert
ingToPB: hbase:meta doesn't have any entries to update.
2014-12-01 12:53:06,935 INFO [master:Master:60000] catalog.MetaMigrationConvert
ingToPB: META already up-to date with PB serialization
2014-12-01 12:53:06,974 INFO [master:Master:60000] master.AssignmentManager: Cl
ean cluster startup. Assigning userregions
2014-12-01 12:53:06,974 DEBUG [master:Master:60000] zookeeper.ZKAssign: master:6
0000-0x34a042da2e20004, quorum=10.1.51.204:2181,10.1.51.203:2181,10.1.51.201:218
1,10.1.51.205:2181, baseZNode=/hbase Deleting any existing unassigned nodes
2014-12-01 12:53:06,998 INFO [master:Master:60000] master.SnapshotOfRegionAssig
nmentFromMeta: Start to scan the hbase:meta for the current region assignment sn
appshot
2014-12-01 12:53:07,007 INFO [master:Master:60000] master.SnapshotOfRegionAssig
nmentFromMeta: Finished to scan the hbase:meta for the current region assignment
snapshot
2014-12-01 12:53:07,045 INFO [master:Master:60000] master.TableNamespaceManager
: Namespace table not found. Creating...
2014-12-01 12:53:07,086 DEBUG [master:Master:60000] lock.ZKInterProcessLockBase:
Acquired a lock for /hbase/table-lock/hbase:namespace/write-master:600000000000
011
2014-12-01 12:53:07,114 DEBUG [master:Master:60000] lock.ZKInterProcessLockBase:
Released /hbase/table-lock/hbase:namespace/write-master:600000000000011
2014-12-01 12:53:07,114 FATAL [master:Master:60000] master.HMaster: Master serve
r abort: loaded coprocessors are: []
2014-12-01 12:53:07,115 FATAL [master:Master:60000] master.HMaster: Unhandled ex
ception. Starting shutdown.
org.apache.hadoop.hbase.TableExistsException: hbase:namespace
at org.apache.hadoop.hbase.master.handler.CreateTableHandler.prepare(Cre
ateTableHandler.java:120)
at org.apache.hadoop.hbase.master.TableNamespaceManager.createNamespaceT
able(TableNamespaceManager.java:232)
at org.apache.hadoop.hbase.master.TableNamespaceManager.start(TableNames
paceManager.java:86)
at org.apache.hadoop.hbase.master.HMaster.initNamespace(HMaster.java:105
2)
at org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.j
ava:915)
at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:603)
at java.lang.Thread.run(Thread.java:724)
2014-12-01 12:53:07,372 INFO [master:Master:60000] master.HMaster: HMaster main
thread exiting
2014-12-01 12:53:07,373 ERROR [main] master.HMasterCommandLine: Master exiting
java.lang.RuntimeException: HMaster Aborted
at org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMaster
CommandLine.java:194)
at org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandL
ine.java:135)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLi
ne.java:126)
at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:2794)


hbase-site.xml
<configuration>
<property>
<name>hbase.rootdir</name>
<value>hdfs://10.1.51.205:9000/hbase</value>
</property>
<property>
<name>hbase.cluster.distributed</name>
<value>true</value>
</property>
<property>
<name>hbase.zookeeper.quorum</name>
<value>10.1.51.205,10.1.51.201,10.1.51.203,10.1.51.204</value>
</property>
<property>
<name>zookeeper.session.timeout</name>
<value>60000</value>
</property>
<property>
<name>hbase.zookeeper.property.clientPort</name>
<value>2181</value>
</property>
</configuration>

regionservers:
10.1.51.201
10.1.51.203
10.1.51.204

zoo.cfg:
server.1=10.1.51.205:2888:3888:participant
server.2=10.1.51.201:2888:3888:participant
server.3=10.1.51.203:2888:3888:participant
server.4=10.1.51.204:2888:3888:participant

hbase启动后，在HDFS中已经存在Slave1,60020,1417409582139，Slave3,60020,1417409580604，Slave4,60020,1417409580580，但是不存在Slave3,60020,1417409322062。怎么Slave3上会需要两个呢？