内核调度失败,导致系统崩溃,求解……

Jhone.Yu 2016-12-29 05:05:03
内核调度出现了rcu的问题,导致系统崩溃。想请教应该怎么确定是卡死在哪一个进程或者哪一个线程?


log太多,附上部分log:

[18389.559303] INFO: rcu_preempt self-detected stall on CPU { 0} (t=2100 jiffies g=154481 c=154480 q=1143)
[18389.559354] Backtrace for cpu 0 (current):
[18389.559369] CPU: 0 PID: 530 Comm: WifiStateMachin Tainted: P W 3.10.0 #1
[18389.559405] [<c0013fc4>] (unwind_backtrace+0x0/0xe0) from [<c0011864>] (show_stack+0x10/0x14)
[18389.559427] [<c0011864>] (show_stack+0x10/0x14) from [<c0012edc>] (smp_send_all_cpu_backtrace+0x60/0xcc)
[18389.559451] [<c0012edc>] (smp_send_all_cpu_backtrace+0x60/0xcc) from [<c00abe38>] (print_cpu_stall+0xb4/0x138)
[18389.559474] [<c00abe38>] (print_cpu_stall+0xb4/0x138) from [<c00acd90>] (__rcu_pending+0x68/0x190)
[18389.559494] [<c00acd90>] (__rcu_pending+0x68/0x190) from [<c00ae6b0>] (rcu_check_callbacks+0x160/0x230)
[18389.559516] [<c00ae6b0>] (rcu_check_callbacks+0x160/0x230) from [<c004bc00>] (update_process_times+0x38/0x64)
[18389.559538] [<c004bc00>] (update_process_times+0x38/0x64) from [<c0088510>] (tick_sched_handle+0x54/0x60)
[18389.559558] [<c0088510>] (tick_sched_handle+0x54/0x60) from [<c008877c>] (tick_sched_timer+0x70/0xa4)
[18389.559582] [<c008877c>] (tick_sched_timer+0x70/0xa4) from [<c005fd18>] (__run_hrtimer+0x1a4/0x2b8)
[18389.559606] [<c005fd18>] (__run_hrtimer+0x1a4/0x2b8) from [<c0060918>] (hrtimer_interrupt+0x11c/0x278)
[18389.559627] [<c0060918>] (hrtimer_interrupt+0x11c/0x278) from [<c06a2574>] (arch_timer_handler_phys+0x28/0x30)
[18389.559651] [<c06a2574>] (arch_timer_handler_phys+0x28/0x30) from [<c00a7e0c>] (handle_percpu_devid_irq+0xf8/0x1b4)
[18389.559673] [<c00a7e0c>] (handle_percpu_devid_irq+0xf8/0x1b4) from [<c00a4a00>] (generic_handle_irq+0x20/0x30)
[18389.559699] [<c00a4a00>] (generic_handle_irq+0x20/0x30) from [<c000e4cc>] (handle_IRQ+0x64/0x8c)
[18389.559719] [<c000e4cc>] (handle_IRQ+0x64/0x8c) from [<c000853c>] (gic_handle_irq+0x38/0x5c)
[18389.559738] [<c000853c>] (gic_handle_irq+0x38/0x5c) from [<c000d700>] (__irq_svc+0x40/0x70)
[18389.559749] Exception stack(0xdc4a9bf0 to 0xdc4a9c38)
[18389.559761] 9be0: ddf38e90 fffffff4 00000000 00000000
[18389.559779] 9c00: ddf38e10 00000035 dc4a9d14 dc4a9cbc dc4a9ce4 ddf093f4 00002775 dc4a9dc4
[18389.559795] 9c20: 00000000 dc4a9c38 c06980d4 c02be048 400d0013 ffffffff
[18389.559817] [<c000d700>] (__irq_svc+0x40/0x70) from [<c02be048>] (__memzero+0x68/0x80)
[18389.559829]
[18389.559829] sending IPI to all other CPUs:
[18389.559847] IPI backtrace for cpu 2
[18389.559858]
[18389.559871] CPU: 2 PID: 34 Comm: kconsole Tainted: P W 3.10.0 #1
[18389.559883] task: de98a4c0 ti: dea4e000 task.ti: dea4e000
[18389.559896] PC is at debug_putc+0x10/0x24
[18389.559906] LR is at console_thread+0x9c/0xb0
[18389.559919] pc : [<c00346a0>] lr : [<c0034948>] psr: 600f0013
[18389.559919] sp : dea4ff50 ip : de8aa044 fp : 00000000
[18389.559935] r10: 00000000 r9 : 00000000 r8 : c0d98a34
[18389.559947] r7 : c0eb9234 r6 : de805800 r5 : dea4e000 r4 : c0d989e0
[18389.55995: fed60000 r2 : dea3f788 r1 : 00000065 r0 : 00000003
[18389.lags: nZCv IRQs on FIQs on Mode SVC_32 ISA ARM Segment kern9.559983] Control: 10c5387d Table: 7d11c06a DAC: 00000015

……………………………………

[18389.562400] CPU: 2 PID: 34 Comm: kconsole Tainted: P W 3.10.0 #1
[18389.562421] [<c0013fc4>] (unwind_e+0x0/0xe0) from [<c0011864>] (show_stack+0x10/0x14)
[18389.56211864>] (show_stack+0x10/0x14) from [<c0013120>] (handle_IPI+0x1
[18389.562457] [<c0013120>] (handle_IPI+0x1d8/0x2c8) from [<c00 (gic_handle_irq+0x54/0x5c)
[18389.562476] [<c0008558>] (gic_ha0x54/0x5c) from [<c000d700>] (__irq_svc+0x40/0x70)
[18389.56248ion stack(0xdea4ff08 to 0xdea4ff50)
[18389.562502] ff00: 00000003 00000065 dea3f788 fed60000 c0d989e0 dea4e000
[183898] ff20: de805800 c0eb9234 c0d98a34 00000000 00000000 00000000 da4ff50
[18389.562533] ff40: c0034948 c00346a0 600f0013 ffffffff
[18389.562549] [<c000d700>] (__irq_svc+0x40/0x70) from [<c0034ug_putc+0x10/0x24)
[18389.562565] [<c00346a0>] (debug_putc+0x10m [<c0034948>] (console_thread+0x9c/0xb0)
[18389.562582] [<c003nsole_thread+0x9c/0xb0) from [<c005cb68>] (kthread+0xa0/0xac)
[1] [<c005cb68>] (kthread+0xa0/0xac) from [<c000dba0>] (ret_from_0x34)
[18389.562614] IPI backtrace for cpu 1
[18389.562624]
[18389.562636] CPU: 1 PID: 0 Comm: swapper/1 Tainted: P W.0 #1
[18389.562647] task: de8839c0 ti: de8a8000 task.ti: de8a89.562661] PC is at arch_cpu_idle+0x30/0x38
[18389.562671] LR ispu_idle+0x2c/0x38
[18389.562685] pc : [<c000e878>] lr : [<c0] psr: 600d0013
[18389.562685] sp : de8a9fc8 ip : 00000000 0000
[18389.562701] r10: 00000000 r9 : 410fc075 r8 : 6000406a562712] r7 : c095e060 r6 : 10c0387d r5 : de8a8000 r4 : de8a80.562722] r3 : 00000000 r2 : 00a88000 r1 : 00000000 r0 : ffffffed
[18389.562734] Flags: nZCv IRQs on FIQs on Mode SVC_32 Segment kernel
[18389.562746] Control: 10c5387d Table: 7a2e006000015
[18389.562755]

……………

[18389.564102] CPU: 1 PID: 0 Comm: swapper/1 Tainted: P 3.10.0 #1
[18389.564122] [<c0013fc4>] (unwind_backtrace+0x0/m [<c0011864>] (show_stack+0x10/0x14)
[18389.564138] [<c0011864>] (show_stack+0x10/0x14) from [<c0013120>] (handle_IPI+0x1d8/0x8389.564154] [<c0013120>] (handle_IPI+0x1d8/0x2c8) from [<c00085handle_irq+0x54/0x5c)
[18389.564170] [<c0008558>] (gic_handle_i5c) from [<c000d700>] (__irq_svc+0x40/0x70)
[18389.564183] Exception stack(0xde8a9f80 to 0xde8a9fc8)
[18389.564198] 9f80: ffff0000 00a88000 00000000 de8a8000 de8a8000 10c0387d c095e060
[183 9fa0: 6000406a 410fc075 00000000 00000000 00000000 de8a9fc8 c000e878
[18389.564226] 9fc0: 600d0013 ffffffff
[18389.564242] [<00>] (__irq_svc+0x40/0x70) from [<c000e878>] (arch_cpu_idle+0x3018389.564262] [<c000e878>] (arch_cpu_idle+0x30/0x38) from [<c0076b0>] (cpu_idle_loop+0x1b8/0x224)
[18389.564284] [<c007f6b0>] (loop+0x1b8/0x224) from [<c007f728>] (freezing_slow_path+0x0/0x80564296] IPI backtrace for cpu 3
[18389.564307]
[18389.564317]D: 0 Comm: swapper/3 Tainted: P W 3.10.0 #1
[18389.56k: de884440 ti: de8ac000 task.ti: de8ac000
[18389.564341] PC ispu_idle+0x30/0x38
[18389.564352] LR is at arch_cpu_idle+0x2c/0x38
[18389.564363] pc : [<c000e878>] lr : [<c000e874>] psr3
[18389.564363] sp : de8adfc8 ip : 00000000 fp : 00000000
[380] r10: 00000000 r9 : 410fc075 r8 : 6000406a
[18389.564391]e060 r6 : 10c0387d r5 : de8ac000 r4 : de8ac000
[18389.564403] r3 : 00000000 r2 : 00a9a000 r1 : 00000000 r0 : ffffffed
[1] Flags: nZCv IRQs on FIQs on Mode SVC_32 ISA ARM Segment k8389.564426] Control: 10c5387d Table: 659dc06a DAC: 00000015
4436]




[18389.565818] CPU: 3 PID: 0 Comm: swapper/3 T W 3.10.0 #1
[18389.565838] [<c0013fc4>] (unwind_backt0xe0) from [<c0011864>] (show_stack+0x10/0x14)
[18389.565855] [>] (show_stack+0x10/0x14) from [<c0013120>] (handle_IPI+0x1d8/0x89.565872] [<c0013120>] (handle_IPI+0x1d8/0x2c8) from [<c0008558andle_irq+0x54/0x5c)
[18389.565889] [<c0008558>] (gic_handle_irq+0x54/0x5c) from [<c000d700>] (__irq_svc+0x40/0x70)
[18389.565tion stack(0xde8adf80 to 0xde8adfc8)
[18389.565919] df80: fffff00 00a9a000 00000000 de8ac000 de8ac000 10c0387d c095e060
[18389fa0: 6000406a 410fc075 00000000 00000000 00000000 de8adfc8 c000e874 c000e878
[18389.565948] dfc0: 600e0013 ffffffff
[18389.5650d700>] (__irq_svc+0x40/0x70) from [<c000e878>] (arch_cpu_idle+0
[18389.565983] [<c000e878>] (arch_cpu_idle+0x30/0x38) from [<c0cpu_idle_loop+0x1b8/0x224)
[18389.566004] [<c007f6b0>] (cpu_idle_loop+0x1b8/0x224) from [<c007f728>] (freezing_slow_path+0x0/0x[18395.778191] BUG: soft lockup - CPU#0 stuck for 22s! [WifiStateMachin:530]
[18395.778222] Modules linked in: ssv6051 vcodec_service rk30xxnand_ko(P) [last unloaded: ssv6051]
[18395.778248]
[18395.778262] CPU: 0 PID: 530 Comm: WifiStateMachin Tainted: P W 3.10.0 #1
[18395.778275] task: c5bfb9c0 ti: dc4a8000 task.ti: dc4a8000
[18395.778293] PC is at __memzero+0x68/0x80
[18395.778307] LR is at dw_mci_prep_stop_abort+0x28/0xb0
[18395.778322] pc : [<c02be048>] lr : [<c06980d4>] psr: 400d0013
[18395.778322] sp : dc4a9c38 ip : 00000000 fp : dc4a9dc4
[18395.778337] r10: 00002775 r9 : ddf093f4 r8 : dc4a9ce4
[18395.778349] r7 : dc4a9cbc r6 : dc4a9d14 r5 : 00000035 r4 : ddf38e10
[18395.778363] r3 : 00000000 r2 : 00000000 r1 : fffffff4 r0 : ddf38e90
[18395.778376] Flags: nZcv IRQs on FIQs on Mode SVC_32 ISA ARM Segment user
[18395.778389] Control: 10c5387d Table: 7d27406a DAC: 00000015
[18395.778399]
[18395.778399] PC: 0xc02bdfc8:
[18395.778414] dfc8 ba00001d e3530002 b4c02001 d4c02001 e4c02001 e0811003 e3a02000 e2103003
[18395.778443] dfe8 1afffff5 e3510010 ba00000f e52de004 e1a0c002 e1a0e002 e2511040 a8a0500c
[18395.778472] e008 a8a0500c a8a0500c a8a0500c cafffff9 08bd8000 e3110020 18a0500c 18a0500c
[18395.778501] e028 e3110010 18a0500c e49de004 e3110008 18a0000c e3110004 14802004 e3110002
[18395.778529] e048 14c02001 14c02001 e3110001 14c02001 e1a0f00e e320f000 e0010192 e0211390
[18395.778558] e068 e1a0c820 e1a03822 e1c0080c e1c22803 e0211c93 e0030390 e0000092 e00c0c92
[18395.778585] e088 e0900803 e0a11823 e090080c e0a1182c e1a0f00e e290c000 30dcc001 2a000017
[18395.778610] e0a8 e5c02000 e3a00000 e1a0f00e e290c001 30dcc001 2a000011 e1a0c422 e4c02001
[18395.778639]

…………

...全文
1931 6 打赏 收藏 转发到动态 举报
写回复
用AI写文章
6 条回复
切换为时间正序
请发表友善的回复…
发表回复
  • 打赏
  • 举报
回复
你好,请问最后是怎么查出原因的呢?我现在遇到了跟你这个几乎一样的问题、
Jhone.Yu 2017-07-09
  • 打赏
  • 举报
回复
最后找到原因是死锁导致!
Jhone.Yu 2017-01-02
  • 打赏
  • 举报
回复
引用 3楼cocoabird 的回复:
在程序进程里打点log看看哪崩的
后面有进行测试,挂的点不在一个位置。是进了中断出不来啦。也就是说如果在某个线程中被中断了,中断无法返回,系统就挂啦。
cocoabird 2017-01-02
  • 打赏
  • 举报
回复
在程序进程里打点log看看哪崩的
Jhone.Yu 2016-12-30
  • 打赏
  • 举报
回复
明明有一条回复,咋没了 = =

19,612

社区成员

发帖
与我相关
我的任务
社区描述
系统使用、管理、维护问题。可以是Ubuntu, Fedora, Unix等等
社区管理员
  • 系统维护与使用区社区
加入社区
  • 近7日
  • 近30日
  • 至今
社区公告
暂无公告

试试用AI创作助手写篇文章吧