oracle9+sunSolaris10 报错：WARNING: aiowait timed out！！！！！在线等待！！！！！

[smile_wu]

环境：oracle9+sunSolaris10
数据库alert.log中报错信息如下，我在做大量的insert表操作，每天数据库都会down一次，报错信息每天类似，如下：
Sun Feb 28 22:48:13 2010
WARNING: aiowait timed out 1 times
Sun Feb 28 22:48:16 2010
WARNING: aiowait timed out 1 times
WARNING: aiowait timed out 2 times
Sun Feb 28 22:59:17 2010
WARNING: aiowait timed out 2 times
Sun Feb 28 23:08:16 2010
WARNING: aiowait timed out 3 times
Sun Feb 28 23:09:21 2010
WARNING: aiowait timed out 3 times
KCF: write/open error block=0xc09a8 online=1
file=9 /dev/md/rdsk/d1170
error=27063 txt: 'SVR4 Error: 5: I/O error
Additional information: -1
Additional information: 8192'
Sun Feb 28 23:18:16 2010
WARNING: aiowait timed out 4 times
WARNING: aiowait timed out 5 times
Sun Feb 28 23:35:43 2010
Errors in file /oracle/admin/cujf/bdump/cujf_dbw0_3544.trc:
ORA-01242: data file suffered media failure: database in NOARCHIVELOG mode
ORA-01114: IO error writing block to file 9 (block # 788904)
ORA-01110: data file 9: '/dev/md/rdsk/d1170'
ORA-27063: skgfospo: number of bytes read/written is incorrect
SVR4 Error: 5: I/O error
Additional information: -1
Additional information: 8192
Sun Feb 28 23:35:43 2010
DBW0: terminating instance due to error 1242
Instance terminated by DBW0, pid = 3544

[smile_wu]

-bash-3.00$ more /oracle/admin/cujf/bdump/cujf_dbw0_3544.trc

/oracle/admin/cujf/bdump/cujf_dbw0_3544.trc
Oracle9i Enterprise Edition Release 9.2.0.1.0 - 64bit Production
With the Partitioning, OLAP and Oracle Data Mining options
JServer Release 9.2.0.1.0 - Production
ORACLE_HOME = /oracle/9.2.0
System name: SunOS
Node name: hqbss-back
Release: 5.10
Version: Generic_141444-09
Machine: sun4u
Instance name: cujf
Redo thread mounted by this instance: 1
Oracle process number: 3
Unix process pid: 3544, image: oracle@hqbss-back (DBW0)

*** 2010-02-28 22:48:13.398
*** SESSION ID:(2.1) 2010-02-28 22:48:13.397
WARNING: aiowait timed out 1 times
*** 2010-02-28 22:59:17.614
WARNING: aiowait timed out 2 times
*** 2010-02-28 23:09:21.374
WARNING: aiowait timed out 3 times
*** 2010-02-28 23:15:27.471
KCF: write/open error block=0xc09a8 online=1
file=9 /dev/md/rdsk/d1170
error=27063 txt: 'SVR4 Error: 5: I/O error
Additional information: -1
Additional information: 8192'
*** 2010-02-28 23:35:43.635
error 1242 detected in background process
ORA-01242: data file suffered media failure: database in NOARCHIVELOG mode
ORA-01114: IO error writing block to file 9 (block # 788904)
ORA-01110: data file 9: '/dev/md/rdsk/d1170'
ORA-27063: skgfospo: number of bytes read/written is incorrect
SVR4 Error: 5: I/O error
Additional information: -1
Additional information: 8192

[suiziguo]

http://www.xmsc.com.cn/InfoView/Article_165446.html

看看上面这篇文章。

[Dave]

ORA-01114: IO error writing block to file string (block # string)
Cause: The device on which the file resides is probably offline. If the file is a temporary file, then it is also possible that the device has run out of space. This could happen because disk space of temporary files is not necessarily allocated at file creation time.

Action: Restore access to the device or remove unnecessary files to free up space.


检查一下磁盘还有多少空间， 表空间还有多少？

ORA-01242: data file suffered media failure: database in NOARCHIVELOG mode
你这是个测试库吗？ 数据库运行在非归档模式？　
你改成归档模式看一下。　这样数据库在启动时如果有错误还可以自己尝试恢复一下。　



------------------------------------------------------------------------------
Blog： http://blog.csdn.net/tianlesoftware
网上资源： http://tianlesoftware.download.csdn.net
相关视频：http://blog.csdn.net/tianlesoftware/archive/2009/11/27/4886500.aspx
Q Q 群：62697716

[smile_wu]

/etc/system中：
set semsys:seminfo_semmni 100
set semsys:seminfo_semmns 1024
set semsys:seminfo_semmsl 256
set semsys:seminfo_semvmx 32767
set shmsys:shminfo_shmmax 16294967295
set shmsys:shminfo_shmmin 100

alert_log:
System parameters with non-default values:
processes = 1600
timed_statistics = TRUE
shared_pool_size = 1073741824
large_pool_size = 16777216
java_pool_size = 117440512
control_files = /dev/md/rdsk/d1101, /dev/md/rdsk/d1102, /dev/md/rdsk/d2101
db_block_size = 8192
db_cache_size = 4294967296
compatible = 9.2.0.0.0
db_files = 3000
db_file_multiblock_read_count= 16
fast_start_mttr_target = 300
undo_management = AUTO
undo_tablespace = UNDOTBS1
undo_retention = 10800
remote_login_passwordfile= EXCLUSIVE
db_domain =
instance_name = cujf
job_queue_processes = 30
hash_join_enabled = TRUE
background_dump_dest = /oracle/admin/cujf/bdump
user_dump_dest = /oracle/admin/cujf/udump
core_dump_dest = /oracle/admin/cujf/cdump
sort_area_size = 524288
db_name = cujf
open_cursors = 3000
star_transformation_enabled= FALSE
query_rewrite_enabled = FALSE
pga_aggregate_target = 10737418240
aq_tm_processes = 1

[suiziguo]

检查dev/md/rdsk/d1170文件状态。

看看设备是否MOUNT，如果已MOUNT，是否能正常访问。

[smile_wu]

文件状态没有问题，而且最近每次down机报错的文件都不一样，但错误代码一样，都是先有
WARNING: aiowait timed out
然后报不同的文件错误，然后数据库down了，每次down了之后，数据库还能startup，正常使用。

[suiziguo]

看看file /oracle/admin/cujf/bdump/cujf_dbw0_3544.trc等文件里咋说？

[lawhsong]

出现这个问题，已经明确在日志中，数据文件存在问题。
个人建议：
1、首先需要检查操作系统日志，看看是否有其他告警。
2、根据以往的处理经验，可能内存或板卡有问题，重点还请看第一点。