hbase在磁盘空间不足时，用phoenix删除了一个大表（直接drop表），表重建后，hbase regionserver在数据入库时就会宕机

[cwq1992]

前段时间发现hbase磁盘空间不足时，用phoenix删除了一个大表，在重建表和其对应索引后，hbase regionserver在数据入库时就会宕机，以下是截取的regionserver及master错误日志：
master错误日志：

引用

2019-06-08 15:58:53,664 INFO [RpcServer.FifoWFPBQ.default.handler=59,queue=5,port=16000] master.HMaster: Client=hbase/null List Table Descriptor for the SYSTEM.CATALOG table succeeds
2019-06-08 15:58:53,853 INFO [RpcServer.FifoWFPBQ.default.handler=59,queue=5,port=16000] master.HMaster: Client=hbase/null List Table Descriptor for the SYSTEM.SEQUENCE table succeeds
2019-06-08 15:58:53,871 INFO [RpcServer.FifoWFPBQ.default.handler=59,queue=5,port=16000] master.HMaster: Client=hbase/null List Table Descriptor for the SYSTEM.STATS table succeeds
2019-06-08 15:58:53,889 INFO [RpcServer.FifoWFPBQ.default.handler=59,queue=5,port=16000] master.HMaster: Client=hbase/null List Table Descriptor for the SYSTEM.FUNCTION table succeeds
2019-06-08 16:00:35,469 ERROR [RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16000] master.MasterRpcServices: Region server slave2.test.node,16020,1559978528007 reported a fatal error:
ABORTING region server slave2.test.node,16020,1559978528007: Could not update the index table, killing server region because couldn't write to an index table
Cause:
org.apache.phoenix.hbase.index.exception.SingleIndexWriteFailureException: Got an abort notification while writing to the index!
at org.apache.phoenix.hbase.index.write.ParallelWriterIndexCommitter.propagateFailure(ParallelWriterIndexCommitter.java:208)
at org.apache.phoenix.hbase.index.write.ParallelWriterIndexCommitter.write(ParallelWriterIndexCommitter.java:197)
at org.apache.phoenix.hbase.index.write.IndexWriter.write(IndexWriter.java:185)
at org.apache.phoenix.hbase.index.write.IndexWriter.writeAndKillYourselfOnFailure(IndexWriter.java:146)
at org.apache.phoenix.hbase.index.write.IndexWriter.writeAndKillYourselfOnFailure(IndexWriter.java:135)
at org.apache.phoenix.hbase.index.Indexer.doPostWithExceptions(Indexer.java:517)
at org.apache.phoenix.hbase.index.Indexer.doPost(Indexer.java:464)
at org.apache.phoenix.hbase.index.Indexer.postBatchMutate(Indexer.java:449)
at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$36.call(RegionCoprocessorHost.java:989)
at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1660)
at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1734)
at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1692)
at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.postBatchMutate(RegionCoprocessorHost.java:985)
at org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:3417)
at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3015)
at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2957)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:750)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:710)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2137)
at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32393)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2150)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:187)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:167)
Caused by: java.lang.RuntimeException: java.lang.OutOfMemoryError: unable to create new native thread
at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:219)
at org.apache.hadoop.hbase.client.ClientSmallReversedScanner.loadCache(ClientSmallReversedScanner.java:211)
at org.apache.hadoop.hbase.client.ClientSmallReversedScanner.next(ClientSmallReversedScanner.java:185)
at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegionInMeta(ConnectionManager.java:1273)
at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1177)
at org.apache.hadoop.hbase.client.CoprocessorHConnection.locateRegion(CoprocessorHConnection.java:41)
at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncProcess.java:959)
at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncProcess.java:867)
at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.resubmit(AsyncProcess.java:1205)
at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveGlobalFailure(AsyncProcess.java:1172)
at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.sendMultiAction(AsyncProcess.java:1013)
at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncProcess.java:899)
at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$100(AsyncProcess.java:585)
at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:567)
at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:910)
at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:927)
at org.apache.phoenix.execute.DelegateHTable.batch(DelegateHTable.java:94)
at org.apache.phoenix.hbase.index.write.ParallelWriterIndexCommitter$1.call(ParallelWriterIndexCommitter.java:164)
at org.apache.phoenix.hbase.index.write.ParallelWriterIndexCommitter$1.call(ParallelWriterIndexCommitter.java:132)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:714)
at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:950)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1357)
at org.apache.hadoop.hbase.client.ResultBoundedCompletionService.submit(ResultBoundedCompletionService.java:171)
at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.addCallsForCurrentReplica(ScannerCallableWithReplicas.java:321)
at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:186)
at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:61)
at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:211)
... 22 more

2019-06-08 16:01:56,301 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=19.23 MB, freeSize=18.29 GB, max=18.31 GB, blockCount=0, accesses=0, hits=0, hitRatio=0, cachingAccesses=0, cachingHits=0, cachingHitsRatio=0,evictions=239, evicted=0, evictedPerRun=0.0
2019-06-08 16:01:57,377 INFO [BucketCacheStatsExecutor] bucket.BucketCache: failedBlockAdditions=0, totalSize=2.00 GB, freeSize=2.00 GB, usedSize=0 B, cacheSize=0 B, accesses=0, hits=0, IOhitsPerSecond=0, IOTimePerHit=NaN, hitRatio=0,cachingAccesses=0, cachingHits=0, cachingHitsRatio=0,evictions=0, evicted=0, evictedPerRun=NaN
2019-06-08 16:02:39,397 INFO [main-EventThread] zookeeper.RegionServerTracker: RegionServer ephemeral node deleted, processing expiration [slave1.test.node,16020,1559978527573]

slave错误日志：

引用

2019-06-08 15:59:24,113 WARN [RpcServer.FifoWFPBQ.default.handler=56,queue=2,port=16020] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.protobuf.generated.ClientProtos$MultiRequest)","starttimems":1559980753263,"responsesize":17942,"method":"Multi","processingtimems":10066,"client":"192.168.50.99:55535","queuetimems":0,"class":"HRegionServer"}
2019-06-08 15:59:24,592 WARN [RpcServer.FifoWFPBQ.default.handler=59,queue=5,port=16020] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.protobuf.generated.ClientProtos$MultiRequest)","starttimems":1559980753721,"responsesize":18347,"method":"Multi","processingtimems":10861,"client":"192.168.50.99:55535","queuetimems":0,"class":"HRegionServer"}
2019-06-08 15:59:24,917 WARN [RpcServer.FifoWFPBQ.default.handler=54,queue=0,port=16020] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.protobuf.generated.ClientProtos$MultiRequest)","starttimems":1559980754748,"responsesize":10381,"method":"Multi","processingtimems":10166,"client":"192.168.50.99:55535","queuetimems":0,"class":"HRegionServer"}
2019-06-08 15:59:25,944 INFO [sync.3] wal.FSHLog: Slow sync cost: 191 ms, current pipeline: [DatanodeInfoWithStorage[192.168.50.203:1019,DS-27f98ce9-ac47-45cf-b58e-817f426f7f61,DISK], DatanodeInfoWithStorage[192.168.50.202:1019,DS-e43838c7-2903-45b9-aeec-b7da18a413ff,DISK]]
2019-06-08 15:59:25,944 INFO [sync.4] wal.FSHLog: Slow sync cost: 192 ms, current pipeline: [DatanodeInfoWithStorage[192.168.50.203:1019,DS-27f98ce9-ac47-45cf-b58e-817f426f7f61,DISK], DatanodeInfoWithStorage[192.168.50.202:1019,DS-e43838c7-2903-45b9-aeec-b7da18a413ff,DISK]]
2019-06-08 15:59:26,657 WARN [slave2.test.node,16020,1559978528007-index-writer--pool5-t7] client.AsyncProcess: Caught unexpected exception/error:
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:714)
at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:950)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1357)
at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112)
at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.sendMultiAction(AsyncProcess.java:999)
at org.apa

[cwq1992]

只要一入库regionServer就宕机，重启hbase服务后，节点又恢复正常，也可以正常读取，就是不能写入，困扰我一个多星期了，求各位大神帮帮忙！！！ 补上slave错误日志：

引用

2019-06-08 15:59:24,113 WARN [RpcServer.FifoWFPBQ.default.handler=56,queue=2,port=16020] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.protobuf.generated.ClientProtos$MultiRequest)","starttimems":1559980753263,"responsesize":17942,"method":"Multi","processingtimems":10066,"client":"192.168.50.99:55535","queuetimems":0,"class":"HRegionServer"} 2019-06-08 15:59:24,592 WARN [RpcServer.FifoWFPBQ.default.handler=59,queue=5,port=16020] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.protobuf.generated.ClientProtos$MultiRequest)","starttimems":1559980753721,"responsesize":18347,"method":"Multi","processingtimems":10861,"client":"192.168.50.99:55535","queuetimems":0,"class":"HRegionServer"} 2019-06-08 15:59:24,917 WARN [RpcServer.FifoWFPBQ.default.handler=54,queue=0,port=16020] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.protobuf.generated.ClientProtos$MultiRequest)","starttimems":1559980754748,"responsesize":10381,"method":"Multi","processingtimems":10166,"client":"192.168.50.99:55535","queuetimems":0,"class":"HRegionServer"} 2019-06-08 15:59:25,944 INFO [sync.3] wal.FSHLog: Slow sync cost: 191 ms, current pipeline: [DatanodeInfoWithStorage[192.168.50.203:1019,DS-27f98ce9-ac47-45cf-b58e-817f426f7f61,DISK], DatanodeInfoWithStorage[192.168.50.202:1019,DS-e43838c7-2903-45b9-aeec-b7da18a413ff,DISK]] 2019-06-08 15:59:25,944 INFO [sync.4] wal.FSHLog: Slow sync cost: 192 ms, current pipeline: [DatanodeInfoWithStorage[192.168.50.203:1019,DS-27f98ce9-ac47-45cf-b58e-817f426f7f61,DISK], DatanodeInfoWithStorage[192.168.50.202:1019,DS-e43838c7-2903-45b9-aeec-b7da18a413ff,DISK]] 2019-06-08 15:59:26,657 WARN [slave2.test.node,16020,1559978528007-index-writer--pool5-t7] client.AsyncProcess: Caught unexpected exception/error: java.lang.OutOfMemoryError: unable to create new native thread at java.lang.Thread.start0(Native Method) at java.lang.Thread.start(Thread.java:714) at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:950) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1357) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.sendMultiAction(AsyncProcess.java:999) at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncProcess.java:899) at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$100(AsyncProcess.java:585) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:567) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:910) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:927) at org.apache.phoenix.execute.DelegateHTable.batch(DelegateHTable.java:94) at org.apache.phoenix.hbase.index.write.ParallelWriterIndexCommitter$1.call(ParallelWriterIndexCommitter.java:164) at org.apache.phoenix.hbase.index.write.ParallelWriterIndexCommitter$1.call(ParallelWriterIndexCommitter.java:132) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 2019-06-08 15:59:26,657 WARN [slave2.test.node,16020,1559978528007-index-writer--pool5-t1] client.AsyncProcess: Caught unexpected exception/error: java.lang.OutOfMemoryError: unable to create new native thread at java.lang.Thread.start0(Native Method) at java.lang.Thread.start(Thread.java:714) at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:950) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1357) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.sendMultiAction(AsyncProcess.java:999) at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncProcess.java:899) at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$100(AsyncProcess.java:585) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:567) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:910) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:927) at org.apache.phoenix.execute.DelegateHTable.batch(DelegateHTable.java:94) at org.apache.phoenix.hbase.index.write.ParallelWriterIndexCommitter$1.call(ParallelWriterIndexCommitter.java:164) at org.apache.phoenix.hbase.index.write.ParallelWriterIndexCommitter$1.call(ParallelWriterIndexCommitter.java:132) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)