ActiveMQ集群 CPU占用100% 丢失消息问题
我们的集群部署是activemq+zookeeper的三节点服务器,两组1主2从结构。
1、大量异常日志类似levelDB数据文件同步告警日志
2019-11-30 07:27:33,386 | WARN | Could not load message seq: 338798 from DataLocator(2c2eeea1, 419) | org.apache.activemq.leveldb.LevelDBClient | ActiveMQ BrokerService[cluster2] Task-7199
2019-11-30 07:27:33,391 | WARN | No reader available for position: 2c2ef07f, log_infos: {838862825=LogInfo(/mpjava/apache-activemq-12/data/leveldb/00000000320007e9.log,838862825,0)} | org.apache.activemq.leveldb.RecordLog | ActiveMQ BrokerService[cluster2] Task-7199
2019-11-30 07:27:33,397 | WARN | Could not load message seq: 338799 from DataLocator(2c2ef07f, 419) | org.apache.activemq.leveldb.LevelDBClient | ActiveMQ BrokerService[cluster2] Task-7199
2019-11-30 07:27:33,402 | WARN | No reader available for position: 2c2ef25d, log_infos: {838862825=LogInfo(/mpjava/apache-activemq-12/data/leveldb/00000000320007e9.log,838862825,0)} | org.apache.activemq.leveldb.RecordLog | ActiveMQ BrokerService[cluster2] Task-7199
2、activemq.jar进程CPU占用 100%
3、偶尔出现MQ消息丢失,但没有出现大面积ActiveMQ服务不可用情况。
官方社区有类似异常,但没有好的解决方案。
Infinite loop in log replay with Replicated LevelDB
https://issues.apache.org/jira/browse/AMQ-5618
[ActiveMQ-users] replicatedLevelDB errors after failover
https://grokbase.com/p/activemq/users/13ccxmzcm7/replicatedleveldb-errors-after-failover
哪位大牛可以解决或者提供处理思路,谢谢!