Oracle 11g RAC 有一个节点宕掉了，好像是redo坏了，怎么处理，急急急

[学海无涯-回头是岸]

我今天看数据库 发先 这个节点掉了。下面是开始报错的 alert，麻烦帮帮忙谢谢
字数限制，不能全部发上来。

Fri Feb 09 10:30:51 2018
Thread 2 advanced to log sequence 4275 (LGWR switch)
Current log# 3 seq# 4275 mem# 0: +DATA/bdcdjsj/onlinelog/group_3.274.960161907
Current log# 3 seq# 4275 mem# 1: +FRAARC/bdcdjsj/onlinelog/group_3.330.960161907
Fri Feb 09 11:00:43 2018
Thread 2 advanced to log sequence 4276 (LGWR switch)
Current log# 4 seq# 4276 mem# 0: +DATA/bdcdjsj/onlinelog/group_4.279.960161907
Current log# 4 seq# 4276 mem# 1: +FRAARC/bdcdjsj/onlinelog/group_4.331.960161907
Fri Feb 09 11:22:34 2018
Thread 2 advanced to log sequence 4277 (LGWR switch)
Current log# 3 seq# 4277 mem# 0: +DATA/bdcdjsj/onlinelog/group_3.274.960161907
Current log# 3 seq# 4277 mem# 1: +FRAARC/bdcdjsj/onlinelog/group_3.330.960161907
Fri Feb 09 11:44:00 2018
Thread 2 advanced to log sequence 4278 (LGWR switch)
Current log# 4 seq# 4278 mem# 0: +DATA/bdcdjsj/onlinelog/group_4.279.960161907
Current log# 4 seq# 4278 mem# 1: +FRAARC/bdcdjsj/onlinelog/group_4.331.960161907
Fri Feb 09 11:56:34 2018
WARNING: Write Failed. group:1 disk:3 AU:3123 offset:565248 size:8192
Fri Feb 09 11:56:34 2018
WARNING: Write Failed. group:1 disk:3 AU:3100 offset:999424 size:8192
Fri Feb 09 11:56:34 2018
WARNING: Write Failed. group:1 disk:3 AU:8271 offset:622592 size:8192
Fri Feb 09 11:56:34 2018
WARNING: Write Failed. group:1 disk:3 AU:14764 offset:548864 size:8192
Fri Feb 09 11:56:34 2018
WARNING: Write Failed. group:1 disk:3 AU:8271 offset:8192 size:8192
Fri Feb 09 11:56:34 2018
WARNING: Write Failed. group:1 disk:3 AU:3126 offset:0 size:8192
Fri Feb 09 11:56:34 2018
WARNING: Write Failed. group:1 disk:3 AU:3115 offset:827392 size:8192
Errors in file /u01/product/oracle/diag/rdbms/bdcdjsj/bdcdjsj2/trace/bdcdjsj2_dbw2_3792.trc:
ORA-15080: synchronous I/O operation to a disk failed
ORA-27061: waiting for async I/Os failed
Linux-x86_64 Error: 5: Input/output error
Additional information: -1
Additional information: 8192
Errors in file /u01/product/oracle/diag/rdbms/bdcdjsj/bdcdjsj2/trace/bdcdjsj2_dbw1_3790.trc:
ORA-15080: synchronous I/O operation to a disk failed
ORA-27061: waiting for async I/Os failed
Linux-x86_64 Error: 5: Input/output error
Additional information: -1
Additional information: 8192
Errors in file /u01/product/oracle/diag/rdbms/bdcdjsj/bdcdjsj2/trace/bdcdjsj2_dbw8_3815.trc:
ORA-15080: synchronous I/O operation to a disk failed
ORA-27061: waiting for async I/Os failed
Linux-x86_64 Error: 5: Input/output error
Additional information: -1
Additional information: 8192
Errors in file /u01/product/oracle/diag/rdbms/bdcdjsj/bdcdjsj2/trace/bdcdjsj2_dbw5_3798.trc:
ORA-15080: synchronous I/O operation to a disk failed
ORA-27061: waiting for async I/Os failed
Linux-x86_64 Error: 5: Input/output error
Additional information: -1
Additional information: 8192
Errors in file /u01/product/oracle/diag/rdbms/bdcdjsj/bdcdjsj2/trace/bdcdjsj2_dbwb_3821.trc:
ORA-15080: synchronous I/O operation to a disk failed
ORA-27061: waiting for async I/Os failed
Linux-x86_64 Error: 5: Input/output error
Additional information: -1
Additional information: 8192
Errors in file /u01/product/oracle/diag/rdbms/bdcdjsj/bdcdjsj2/trace/bdcdjsj2_dbw7_3802.trc:
ORA-15080: synchronous I/O operation to a disk failed
ORA-27061: waiting for async I/Os failed
Linux-x86_64 Error: 5: Input/output error
Additional information: -1
Additional information: 8192
WARNING: failed to write mirror side 1 of virtual extent 1905 logical extent 0 of file 314 in group 1 on disk 3 allocation unit 8271
WARNING: failed to write mirror side 1 of virtual extent 1905 logical extent 0 of file 314 in group 1 on disk 3 allocation unit 8271
WARNING: failed to write mirror side 1 of virtual extent 12656 logical extent 0 of file 311 in group 1 on disk 3 allocation unit 14764
WARNING: failed to write mirror side 1 of virtual extent 126 logical extent 0 of file 312 in group 1 on disk 3 allocation unit 3115
WARNING: failed to write mirror side 1 of virtual extent 66 logical extent 0 of file 312 in group 1 on disk 3 allocation unit 3100
WARNING: failed to write mirror side 1 of virtual extent 158 logical extent 0 of file 312 in group 1 on disk 3 allocation unit 3123
KCF: read, write or open error, block=0x3b8cc online=1
KCF: read, write or open error, block=0x3b881 online=1
KCF: read, write or open error, block=0x18b843 online=1
KCF: read, write or open error, block=0x3f65 online=1 file=14 '+DATA/bdcdjsj/datafile/jointserver.314.960393637'
file=14 '+DATA/bdcdjsj/datafile/jointserver.314.960393637'

KCF: read, write or open error, block=0x217a online=1KCF: read, write or open error, block=0x4f45 online=1

file=11 '+DATA/bdcdjsj/datafile/bdck.311.960393633'
file=12 '+DATA/bdcdjsj/datafile/gxdjk.312.960393633' error=15081 txt: '' error=15081 txt: ''


error=15081 txt: ''
file=12 '+DATA/bdcdjsj/datafile/gxdjk.312.960393633'
error=15081 txt: ''
file=12 '+DATA/bdcdjsj/datafile/gxdjk.312.960393633'
error=15081 txt: ''
error=15081 txt: ''
Errors in file /u01/product/oracle/diag/rdbms/bdcdjsj/bdcdjsj2/trace/bdcdjsj2_dbw7_3802.trc:
Errors in file /u01/product/oracle/diag/rdbms/bdcdjsj/bdcdjsj2/trace/bdcdjsj2_dbw8_3815.trc:
Errors in file /u01/product/oracle/diag/rdbms/bdcdjsj/bdcdjsj2/trace/bdcdjsj2_dbw2_3792.trc:
Errors in file /u01/product/oracle/diag/rdbms/bdcdjsj/bdcdjsj2/trace/bdcdjsj2_dbwb_3821.trc:
Errors in file /u01/product/oracle/diag/rdbms/bdcdjsj/bdcdjsj2/trace/bdcdjsj2_dbw5_3798.trc:
Errors in file /u01/product/oracle/diag/rdbms/bdcdjsj/bdcdjsj2/trace/bdcdjsj2_dbwa_3819.trc:
ORA-15080: synchronous I/O operation to a disk failed
ORA-27061: waiting for async I/Os failed
Linux-x86_64 Error: 5: Input/output error
Additional information: -1
Additional information: 8192
Errors in file /u01/product/oracle/diag/rdbms/bdcdjsj/bdcdjsj2/trace/bdcdjsj2_dbw1_3790.trc:
WARNING: failed to write mirror side 1 of virtual extent 170 logical extent 0 of file 312 in group 1 on disk 3 allocation unit 3126
WARNING: Write Failed. group:1 disk:3 AU:3104 offset:548864 size:8192
Errors in file /u01/product/oracle/diag/rdbms/bdcdjsj/bdcdjsj2/trace/bdcdjsj2_dbwa_3819.trc:
ORA-15080: synchronous I/O operation to a disk failed
ORA-27061: waiting for async I/Os failed
Linux-x86_64 Error: 5: Input/output error
Additional information: -1
Additional information: 8192
WARNING: failed to write mirror side 1 of virtual extent 82 logical extent 0 of file 312 in group 1 on disk 3 allocation unit 3104
KCF: read, write or open error, block=0x5500 online=1
file=12 '+DATA/bdcdjsj/datafile/gxdjk.312.960393633'
error=15081 txt: ''
Errors in file /u01/product/oracle/diag/rdbms/bdcdjsj/bdcdjsj2/trace/bdcdjsj2_dbwa_3819.trc:
Errors in file /u01/product/oracle/diag/rdbms/bdcdjsj/bdcdjsj2/trace/bdcdjsj2_dbw7_3802.trc:
ORA-63999: data file suffered media failure
ORA-01114: IO error writing block to file 12 (block # 16229)
ORA-01110: data file 12: '+DATA/bdcdjsj/datafile/gxdjk.312.960393633'
ORA-15081: failed to submit an I/O operation to a disk
ORA-15081: failed to submit an I/O operation to a disk
DBW7 (ospid: 3802): terminating the instance due to error 63999
Fri Feb 09 11:56:34 2018
System state dump requested by (instance=2, osid=3802 (DBW7)), summary=[abnormal instance termination].
System State dumped to trace file /u01/product/oracle/diag/rdbms/bdcdjsj/bdcdjsj2/trace/bdcdjsj2_diag_3748_20180209115634.trc
Instance terminated by DBW7, pid = 3802
Fri Feb 09 11:56:46 2018

[惜分飞]

专业oracle 恢复技术支持

[惜分飞]

你这个错报错看，很可能是硬件问题导致，io访问失败 可以看看系统日志