[求助]MySQL crach

[beck1986]

4月30日 20：34：27启动jboss服务

MySQL的hostname.err文件里面一直在打印警告信息。
InnoDB: Warning: a long semaphore wait:
--Thread 3204160416 has waited at btr0cur.c line 366 for 242.00 seconds the semaphore:
S-lock on RW-latch at 0xf6109cbc created in file dict0dict.c line 3706
a writer (thread id 3482712992) has reserved it in mode exclusive
number of readers 0, waiters flag 1
Last time read locked in file btr0cur.c line 606
Last time write locked in file btr0cur.c line 359
InnoDB: ###### Starts InnoDB Monitor for 30 secs to print diagnostic info:
InnoDB: Pending preads 0, pwrites 0

此种信息一致在打印 3天打印了56条



PS:公司对外网数据有控制，每次外发数据不能超过限制，所以每次只能发几百个字，我分楼阐述信息。

[beck1986]

5月3日 00：48：23MySQL自己crash
InnoDB: ###### Diagnostic info printed to the standard error stream
InnoDB: Error: semaphore wait has lasted > 600 seconds
InnoDB: We intentionally crash the server, because it appears to be hung.
100503 0:48:23InnoDB: Assertion failure in thread 3502091168 in file srv0srv.c line 2093
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.0/en/forcing-recovery.html
InnoDB: about forcing recovery.
100503 0:48:23 - mysqld got signal 11;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

key_buffer_size=268435456
read_buffer_size=1044480
max_used_connections=19
max_connections=100
threads_connected=16
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections = 466543 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

thd=(nil)
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
Cannot determine thread, fp=0xd0bda0d8, backtrace may not be correct.
Stack range sanity check OK, backtrace follows:
0x8189d49
0x829ed49
0xf7f422ab
0xf7e62b9e
New value of fp=(nil) failed sanity check, terminating stack trace!
Please read http://dev.mysql.com/doc/mysql/en/using-stack-trace.html and follow instructions on how to resolve the stack trace. Resolved
stack trace is much more helpful in diagnosing the problem, so please do
resolve it
The manual page at http://www.mysql.com/doc/en/Crashing.html contains
information that should help you find out what is causing the crash.
100503 0:50:40 InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
100503 0:50:40 InnoDB: Starting log scan based on checkpoint at
InnoDB: log sequence number 0 7472306.
InnoDB: Doing recovery: scanned up to log sequence number 0 7535704
InnoDB: 1 transaction(s) which must be rolled back or cleaned up
InnoDB: in total 16 row operations to undo
InnoDB: Trx id counter is 0 1404928
100503 0:50:41 InnoDB: Starting an apply batch of log records to the database...
InnoDB: Progress in percents: 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99
InnoDB: Apply batch completed
100503 0:50:49 InnoDB: Started; log sequence number 0 7535704
InnoDB: Starting in background the rollback of uncommitted transactions
100503 0:50:49 InnoDB: Rolling back trx with id 0 1403079, 16 rows to undo
100503 0:50:49 [Note] /hinvs/ism/jboss-5.1.0.GA/server/default/deploy/webnms.ear/webnms.war/mysql/bin/mysqld: ready for connections.
Version: '5.0.44-enterprise' socket: '/tmp/mysql_ism.sock' port: 3306 MySQL Enterprise Server (Commercial)

[beck1986]

InnoDB: Rolling back of trx id 0 1403079 completed
100503 0:50:52 InnoDB: Rollback of non-prepared transactions completed
100503 01:13:59 mysqld started
100503 1:13:59 InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
100503 1:13:59 InnoDB: Starting log scan based on checkpoint at
InnoDB: log sequence number 0 7538863.
InnoDB: Doing recovery: scanned up to log sequence number 0 7538863
100503 1:14:03 InnoDB: Started; log sequence number 0 7538863
100503 1:14:03 [Note] /hinvs/ism/jboss-5.1.0.GA/server/default/deploy/webnms.ear/webnms.war/mysql/bin/mysqld: ready for connections.
Version: '5.0.44-enterprise' socket: '/tmp/mysql_ism.sock' port: 3306 MySQL Enterprise Server (Commercial)







请教如何分析hostname.err文件

[shenshine]

从你最近出问题的地方一句句向后看

[beck1986]

更正一下，第一次打印warn日志的时间是：
=====================================
100502 23:23:55 INNODB MONITOR OUTPUT
=====================================
最后一次是：
=====================================
100503 0:48:17 INNODB MONITOR OUTPUT
=====================================

jboss日志在此段时间抛出异常

2010-05-03 00:01:11,180 INFO [com.mchange.v2.c3p0.stmt.GooGooStatementCache] (EventRate-1) Multiply prepared statement! insert into MonitorNmsParameter values (?,?,?,?)
2010-05-03 00:01:36,855 INFO [com.mchange.v2.c3p0.stmt.GooGooStatementCache] (AlertRate-1) Multiply prepared statement! insert into MonitorNmsParameter values (?,?,?,?)
2010-05-03 00:06:40,617 INFO [com.mchange.v2.c3p0.stmt.GooGooStatementCache] (DataCollectionRate-1) Multiply prepared statement! insert into MonitorNmsParameter values (?,?,?,?)
2010-05-03 00:15:24,814 INFO [org.hibernate.event.def.DefaultLoadEventListener] (CameraChangeWorker) Error performing load command
org.hibernate.SessionException: Session is closed!
at org.hibernate.impl.AbstractSessionImpl.errorIfClosed(AbstractSessionImpl.java:72)
at org.hibernate.impl.SessionImpl.getPersistenceContext(SessionImpl.java:1850)
at org.hibernate.engine.TwoPhaseLoad.initializeEntity(TwoPhaseLoad.java:120)
at org.hibernate.loader.Loader.initializeEntitiesAndCollections(Loader.java:877)
at org.hibernate.loader.Loader.doQuery(Loader.java:752)
at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:259)
at org.hibernate.loader.Loader.loadEntity(Loader.java:1881)
at org.hibernate.loader.entity.AbstractEntityLoader.load(AbstractEntityLoader.java:71)
at org.hibernate.loader.entity.AbstractEntityLoader.load(AbstractEntityLoader.java:65)
at org.hibernate.persister.entity.AbstractEntityPersister.load(AbstractEntityPersister.java:3072)
at org.hibernate.event.def.DefaultLoadEventListener.loadFromDatasource(DefaultLoadEventListener.java:434)
at org.hibernate.event.def.DefaultLoadEventListener.doLoad(DefaultLoadEventListener.java:415)
at org.hibernate.event.def.DefaultLoadEventListener.load(DefaultLoadEventListener.java:165)
at org.hibernate.event.def.DefaultLoadEventListener.proxyOrLoad(DefaultLoadEventListener.java:223)
at org.hibernate.event.def.DefaultLoadEventListener.onLoad(DefaultLoadEventListener.java:126)
at org.hibernate.impl.SessionImpl.fireLoad(SessionImpl.java:905)
at org.hibernate.impl.SessionImpl.get(SessionImpl.java:842)
at org.hibernate.impl.SessionImpl.get(SessionImpl.java:835)
at com.adventnet.nms.persistence.HbnStorageAction.getObject(HbnStorageAction.java:123)
at com.adventnet.nms.persistence.StorageAPIImpl.getObject(StorageAPIImpl.java:78)
at com.adventnet.nms.persistence.StorageAPIImpl.updateObject(StorageAPIImpl.java:90)
at com.huaweisymantec.invs.resource.dao.SoundDeviceDaoImpl.update(SoundDeviceDaoImpl.java:203)
at com.huaweisymantec.invs.statchg.INVSStatChgWorker.updateSoundDev(INVSStatChgWorker.java:811)
at com.huaweisymantec.invs.statchg.INVSStatChgWorker.checkSoundDev(INVSStatChgWorker.java:695)
at com.huaweisymantec.invs.statchg.INVSStatChgWorker.checkSubDev(INVSStatChgWorker.java:528)
at com.huaweisymantec.invs.statchg.INVSStatChgWorker.checkCameraConf(INVSStatChgWorker.java:479)
at com.huaweisymantec.invs.statchg.INVSStatChgWorker.access$100(INVSStatChgWorker.java:70)
at com.huaweisymantec.invs.statchg.INVSStatChgWorker$CameraQueWorker.run(INVSStatChgWorker.java:930)
at java.lang.Thread.run(Thread.java:619)

[beck1986]

2010-05-03 00:28:55,080 INFO [org.hibernate.event.def.DefaultLoadEventListener] (pool-16-thread-5) Error performing load command
org.hibernate.SessionException: Session is closed!
at org.hibernate.impl.AbstractSessionImpl.errorIfClosed(AbstractSessionImpl.java:72)
at org.hibernate.impl.SessionImpl.getPersistenceContext(SessionImpl.java:1850)
at org.hibernate.engine.TwoPhaseLoad.initializeEntity(TwoPhaseLoad.java:120)
at org.hibernate.loader.Loader.initializeEntitiesAndCollections(Loader.java:877)
at org.hibernate.loader.Loader.doQuery(Loader.java:752)
at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:259)
at org.hibernate.loader.Loader.loadEntity(Loader.java:1881)
at org.hibernate.loader.entity.AbstractEntityLoader.load(AbstractEntityLoader.java:71)
at org.hibernate.loader.entity.AbstractEntityLoader.load(AbstractEntityLoader.java:65)
at org.hibernate.persister.entity.AbstractEntityPersister.load(AbstractEntityPersister.java:3072)
at org.hibernate.event.def.DefaultLoadEventListener.loadFromDatasource(DefaultLoadEventListener.java:434)
at org.hibernate.event.def.DefaultLoadEventListener.doLoad(DefaultLoadEventListener.java:415)
at org.hibernate.event.def.DefaultLoadEventListener.load(DefaultLoadEventListener.java:165)
at org.hibernate.event.def.DefaultLoadEventListener.proxyOrLoad(DefaultLoadEventListener.java:223)
at org.hibernate.event.def.DefaultLoadEventListener.onLoad(DefaultLoadEventListener.java:126)
at org.hibernate.impl.SessionImpl.fireLoad(SessionImpl.java:905)
at org.hibernate.impl.SessionImpl.get(SessionImpl.java:842)
at org.hibernate.impl.SessionImpl.get(SessionImpl.java:835)
at com.adventnet.nms.persistence.HbnStorageAction.getObject(HbnStorageAction.java:123)
at com.adventnet.nms.persistence.StorageAPIImpl.getObject(StorageAPIImpl.java:78)
at com.adventnet.nms.persistence.StorageAPIImpl.updateObject(StorageAPIImpl.java:94)
at com.huaweisymantec.invs.resource.dao.CommonDaoImpl.updateNodeObject(CommonDaoImpl.java:573)
at com.huaweisymantec.invs.discovery.service.ws.DiscWSServiceImpl.updateDev(DiscWSServiceImpl.java:643)
at com.huaweisymantec.invs.discovery.service.ws.DiscWSServiceImpl.updateOldPtz(DiscWSServiceImpl.java:511)
at com.huaweisymantec.invs.discovery.service.ws.DiscWSServiceImpl.addAndUpdateAllPTZToDB(DiscWSServiceImpl.java:336)
at com.huaweisymantec.invs.discovery.service.ws.DiscWSServiceImpl.getNvrDevList(DiscWSServiceImpl.java:187)
at com.huaweisymantec.invs.discovery.status.INVSStatusPoller.pingSuccNvr(INVSStatusPoller.java:194)
at com.huaweisymantec.invs.discovery.status.INVSStatusPoller.test(INVSStatusPoller.java:119)
at com.huaweisymantec.invs.discovery.status.INVSStatusPoller.call(INVSStatusPoller.java:252)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
2010-05-03 00:28:59,066 WARN [org.hibernate.cache.ReadWriteCache] (TRAPINPUT) An item was expired by the cache while it was locked (increase your cache timeout): com.adventnet.nms.eventdb.Event#2129
2010-05-03 00:44:05,159 INFO [com.mchange.v2.c3p0.stmt.GooGooStatementCache] (DataCollectionRate-1) Multiply prepared statement! insert into MonitorNmsParameter values (?,?,?,?)
2010-05-03 00:50:39,348 WARN [org.hibernate.util.JDBCExceptionReporter] (fault-1) SQL Error: 0, SQLState: 08S01
2010-05-03 00:50:39,348 ERROR [org.hibernate.util.JDBCExceptionReporter] (fault-1) Communications link failure due to underlying exception:

[beck1986]

** BEGIN NESTED EXCEPTION **

java.net.SocketException
MESSAGE: Broken pipe

STACKTRACE:

java.net.SocketException: Broken pipe
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
at com.mysql.jdbc.MysqlIO.send(MysqlIO.java:2744)
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1612)
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:1723)
at com.mysql.jdbc.Connection.execSQL(Connection.java:3256)
at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1313)
at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:1585)
at com.mysql.jdbc.PreparedStatement.executeBatchSerially(PreparedStatement.java:1217)
at com.mysql.jdbc.PreparedStatement.executeBatch(PreparedStatement.java:943)
at com.mchange.v2.c3p0.impl.NewProxyPreparedStatement.executeBatch(NewProxyPreparedStatement.java:1723)
at org.hibernate.jdbc.BatchingBatcher.doExecuteBatch(BatchingBatcher.java:70)
at org.hibernate.jdbc.AbstractBatcher.executeBatch(AbstractBatcher.java:268)
at org.hibernate.engine.ActionQueue.executeActions(ActionQueue.java:266)
at org.hibernate.engine.ActionQueue.executeActions(ActionQueue.java:167)
at org.hibernate.event.def.AbstractFlushingEventListener.performExecutions(AbstractFlushingEventListener.java:321)
at org.hibernate.event.def.DefaultFlushEventListener.onFlush(DefaultFlushEventListener.java:50)
at org.hibernate.impl.SessionImpl.flush(SessionImpl.java:1027)
at org.hibernate.impl.SessionImpl.managedFlush(SessionImpl.java:365)
at org.hibernate.transaction.JDBCTransaction.commit(JDBCTransaction.java:137)
at com.adventnet.management.transaction.TransactionAPI.commitHbnTxn(TransactionAPI.java:1010)
at com.adventnet.management.transaction.TransactionAPI.commit(TransactionAPI.java:432)
at com.adventnet.nms.eventdb.EventLog.commit(EventLog.java:86)
at com.adventnet.nms.eventdb.EventLog.update(EventLog.java:250)
at com.adventnet.nms.eventdb.EventMgr.run(EventMgr.java:861)
at com.adventnet.management.scheduler.WorkerThread.run(WorkerThread.java:70)


** END NESTED EXCEPTION **

[Sybase数据库恢复]

数据库异常关闭过？