系统还有很不少内存空间，却发生OOM是怎么回事？

seai 2014-08-13 04:46:31 
# php mycount.php

Killed



# uname -a

    2.6.32-431.el6.x86_64 #1 SMP Fri Nov 22 03:15:09 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux



# tailf -f /var/log/message

    kernel: Out of memory: Kill process 14188 (php) score 508 or sacrifice child



# free -lm

             total       used       free     shared    buffers     cached

Mem:         15010      14920         90          0        318      10976

Low:         15010      14920         90

High:            0          0          0

-/+ buffers/cache:       3625      11385

Swap:            0          0          0



# cat /proc/meminfo

MemTotal:       15370896 kB

MemFree:         6339324 kB

Buffers:          283160 kB

Cached:          6304016 kB

SwapCached:            0 kB

Active:          2135472 kB

Inactive:        5339232 kB

Active(anon):    1577736 kB

Inactive(anon):  4439692 kB

Active(file):     557736 kB

Inactive(file):   899540 kB

Unevictable:           0 kB

Mlocked:               0 kB

SwapTotal:             0 kB

SwapFree:              0 kB

Dirty:              4700 kB

Writeback:             0 kB

AnonPages:        888048 kB

Mapped:           631120 kB

Shmem:           5129772 kB

Slab:            1410232 kB

SReclaimable:     330528 kB

SUnreclaim:      1079704 kB

KernelStack:        2240 kB

PageTables:        30408 kB

NFS_Unstable:          0 kB

Bounce:                0 kB

WritebackTmp:          0 kB

CommitLimit:     7685448 kB

Committed_AS:    7775268 kB

VmallocTotal:   34359738367 kB

VmallocUsed:       39188 kB

VmallocChunk:   34359699084 kB

HardwareCorrupted:     0 kB

AnonHugePages:         0 kB

HugePages_Total:       0

HugePages_Free:        0

HugePages_Rsvd:        0

HugePages_Surp:        0

Hugepagesize:       2048 kB

DirectMap4k:    15728640 kB

DirectMap2M:           0 kB
...全文
886 7 打赏 收藏 转发到动态 举报
写回复
用AI写文章
7 条回复
切换为时间正序
请发表友善的回复…
发表回复
seai 2014-08-13
  • 打赏
  • 举报
 回复
确实申请了太大的空间。
小笨和漂向北方 2014-08-13
  • 打赏
  • 举报
 回复
Process 9768, rss memory, which is the physical memory used by the process is 1589781(I believe the unit is kB) and the meantime, os only has 90MB free memory. Yes, u have buffer and cache available, but they were reserved by os for disk io. They may or may not be released for other processes. OOM killer is necessary since swap is disabled. The previous comments make sense. This is an application level issue, os did what it is supposed to do. U may have to figure out the problem on that php script to avoid OOM happening again. U may also want to enable swap.
seai 2014-08-13
  • 打赏
  • 举报
 回复 
Aug 13 19:05:02 amazon02 NTP: 13 Aug 19:05:02 ntpdate[9551]: adjust time server 142.54.181.202 offset 0.005044 sec
Aug 13 19:06:03 amazon02 NTP: 13 Aug 19:06:02 ntpdate[10551]: adjust time server 142.54.181.202 offset -0.001661 sec
Aug 13 19:06:42 amazon02 kernel: watch invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0, oom_score_adj=0
Aug 13 19:06:42 amazon02 kernel: watch cpuset=/ mems_allowed=0
Aug 13 19:06:42 amazon02 kernel: Pid: 12845, comm: watch Not tainted 2.6.32-431.el6.x86_64 #1
Aug 13 19:06:42 amazon02 kernel: Call Trace:
Aug 13 19:06:42 amazon02 kernel: [<ffffffff810d05b1>] ? cpuset_print_task_mems_allowed+0x91/0xb0
Aug 13 19:06:42 amazon02 kernel: [<ffffffff81122960>] ? dump_header+0x90/0x1b0
Aug 13 19:06:42 amazon02 kernel: [<ffffffff8122798c>] ? security_real_capable_noaudit+0x3c/0x70
Aug 13 19:06:42 amazon02 kernel: [<ffffffff81122de2>] ? oom_kill_process+0x82/0x2a0
Aug 13 19:06:42 amazon02 kernel: [<ffffffff81122d21>] ? select_bad_process+0xe1/0x120
Aug 13 19:06:42 amazon02 kernel: [<ffffffff81123220>] ? out_of_memory+0x220/0x3c0
Aug 13 19:06:42 amazon02 kernel: [<ffffffff8112fb3c>] ? __alloc_pages_nodemask+0x8ac/0x8d0
Aug 13 19:06:42 amazon02 kernel: [<ffffffff81167a9a>] ? alloc_pages_current+0xaa/0x110
Aug 13 19:06:42 amazon02 kernel: [<ffffffff8111fd57>] ? __page_cache_alloc+0x87/0x90
Aug 13 19:06:42 amazon02 kernel: [<ffffffff8111f73e>] ? find_get_page+0x1e/0xa0
Aug 13 19:06:42 amazon02 kernel: [<ffffffff81120cf7>] ? filemap_fault+0x1a7/0x500
Aug 13 19:06:42 amazon02 kernel: [<ffffffff8100628f>] ? xen_set_pte_at+0xaf/0x170
Aug 13 19:06:42 amazon02 kernel: [<ffffffff8114a084>] ? __do_fault+0x54/0x530
Aug 13 19:06:42 amazon02 kernel: [<ffffffff8114a657>] ? handle_pte_fault+0xf7/0xb00
Aug 13 19:06:42 amazon02 kernel: [<ffffffff8122f864>] ? inode_has_perm+0x54/0xa0
Aug 13 19:06:42 amazon02 kernel: [<ffffffff81007c8f>] ? xen_restore_fl_direct_end+0x0/0x1
Aug 13 19:06:42 amazon02 kernel: [<ffffffff811704c4>] ? kmem_cache_free+0xc4/0x2b0
Aug 13 19:06:42 amazon02 kernel: [<ffffffff811aaa20>] ? mntput_no_expire+0x30/0x110
Aug 13 19:06:42 amazon02 kernel: [<ffffffff81004a49>] ? __raw_callee_save_xen_pmd_val+0x11/0x1e
Aug 13 19:06:42 amazon02 kernel: [<ffffffff8114b28a>] ? handle_mm_fault+0x22a/0x300
Aug 13 19:06:42 amazon02 kernel: [<ffffffff8104a8d8>] ? __do_page_fault+0x138/0x480
Aug 13 19:06:42 amazon02 kernel: [<ffffffff81282705>] ? _atomic_dec_and_lock+0x55/0x80
Aug 13 19:06:42 amazon02 kernel: [<ffffffff810074fd>] ? xen_force_evtchn_callback+0xd/0x10
Aug 13 19:06:42 amazon02 kernel: [<ffffffff81007ca2>] ? check_events+0x12/0x20
Aug 13 19:06:42 amazon02 kernel: [<ffffffff810074fd>] ? xen_force_evtchn_callback+0xd/0x10
Aug 13 19:06:42 amazon02 kernel: [<ffffffff81007ca2>] ? check_events+0x12/0x20
Aug 13 19:06:42 amazon02 kernel: [<ffffffff8152d45e>] ? do_page_fault+0x3e/0xa0
Aug 13 19:06:42 amazon02 kernel: [<ffffffff8152a815>] ? page_fault+0x25/0x30
Aug 13 19:06:42 amazon02 kernel: Mem-Info:
Aug 13 19:06:42 amazon02 kernel: Node 0 DMA per-cpu:
Aug 13 19:06:42 amazon02 kernel: CPU    0: hi:    0, btch:   1 usd:   0
Aug 13 19:06:42 amazon02 kernel: CPU    1: hi:    0, btch:   1 usd:   0
Aug 13 19:06:42 amazon02 kernel: CPU    2: hi:    0, btch:   1 usd:   0
Aug 13 19:06:42 amazon02 kernel: CPU    3: hi:    0, btch:   1 usd:   0
Aug 13 19:06:42 amazon02 kernel: CPU    4: hi:    0, btch:   1 usd:   0
Aug 13 19:06:42 amazon02 kernel: CPU    5: hi:    0, btch:   1 usd:   0
Aug 13 19:06:42 amazon02 kernel: CPU    6: hi:    0, btch:   1 usd:   0
Aug 13 19:06:42 amazon02 kernel: CPU    7: hi:    0, btch:   1 usd:   0
Aug 13 19:06:42 amazon02 kernel: Node 0 DMA32 per-cpu:
Aug 13 19:06:42 amazon02 kernel: CPU    0: hi:  186, btch:  31 usd:   0
Aug 13 19:06:42 amazon02 kernel: CPU    1: hi:  186, btch:  31 usd:   0
Aug 13 19:06:42 amazon02 kernel: CPU    2: hi:  186, btch:  31 usd:   0
Aug 13 19:06:42 amazon02 kernel: CPU    3: hi:  186, btch:  31 usd:   0
Aug 13 19:06:42 amazon02 kernel: CPU    4: hi:  186, btch:  31 usd:   0
Aug 13 19:06:42 amazon02 kernel: CPU    5: hi:  186, btch:  31 usd:   0
Aug 13 19:06:42 amazon02 kernel: CPU    6: hi:  186, btch:  31 usd:   0
Aug 13 19:06:42 amazon02 kernel: CPU    7: hi:  186, btch:  31 usd:   0
Aug 13 19:06:42 amazon02 kernel: Node 0 Normal per-cpu:
Aug 13 19:06:42 amazon02 kernel: CPU    0: hi:  186, btch:  31 usd:  30
Aug 13 19:06:42 amazon02 kernel: CPU    1: hi:  186, btch:  31 usd:   0
Aug 13 19:06:42 amazon02 kernel: CPU    2: hi:  186, btch:  31 usd: 170
Aug 13 19:06:42 amazon02 kernel: CPU    3: hi:  186, btch:  31 usd:   0
Aug 13 19:06:42 amazon02 kernel: CPU    4: hi:  186, btch:  31 usd:  29
Aug 13 19:06:42 amazon02 kernel: CPU    5: hi:  186, btch:  31 usd:   0
Aug 13 19:06:42 amazon02 kernel: CPU    6: hi:  186, btch:  31 usd:  48
Aug 13 19:06:42 amazon02 kernel: CPU    7: hi:  186, btch:  31 usd:   0
Aug 13 19:06:42 amazon02 kernel: active_anon:2273022 inactive_anon:1161977 isolated_anon:0
Aug 13 19:06:42 amazon02 kernel: active_file:1079 inactive_file:395 isolated_file:181
Aug 13 19:06:42 amazon02 kernel: unevictable:0 dirty:581 writeback:0 unstable:0
Aug 13 19:06:42 amazon02 kernel: free:16972 slab_reclaimable:75611 slab_unreclaimable:277337
Aug 13 19:06:42 amazon02 kernel: mapped:33020 shmem:1312218 pagetables:10273 bounce:0
Aug 13 19:06:42 amazon02 kernel: Node 0 DMA free:7744kB min:4kB low:8kB high:8kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:7740kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Aug 13 19:06:42 amazon02 kernel: lowmem_reserve[]: 0 4024 15134 15134
Aug 13 19:06:42 amazon02 kernel: Node 0 DMA32 free:48268kB min:4184kB low:9412kB high:10460kB active_anon:2036472kB inactive_anon:1313668kB active_file:2144kB inactive_file:372kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:4120800kB mlocked:0kB dirty:824kB writeback:0kB mapped:58028kB shmem:1514472kB slab_reclaimable:127352kB slab_unreclaimable:291176kB kernel_stack:520kB pagetables:11936kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:672 all_unreclaimable? no
Aug 13 19:06:42 amazon02 kernel: lowmem_reserve[]: 0 0 11110 11110
Aug 13 19:06:42 amazon02 kernel: Node 0 Normal free:11876kB min:11552kB low:25992kB high:28880kB active_anon:7055616kB inactive_anon:3334240kB active_file:2172kB inactive_file:1208kB unevictable:0kB isolated(anon):0kB isolated(file):724kB present:11376640kB mlocked:0kB dirty:1500kB writeback:0kB mapped:74052kB shmem:3734400kB slab_reclaimable:175092kB slab_unreclaimable:818172kB kernel_stack:1856kB pagetables:29156kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:1056 all_unreclaimable? no
Aug 13 19:06:42 amazon02 kernel: lowmem_reserve[]: 0 0 0 0
Aug 13 19:06:42 amazon02 kernel: Node 0 DMA: 2*4kB 1*8kB 1*16kB 1*32kB 0*64kB 2*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 1*4096kB = 7744kB
Aug 13 19:06:42 amazon02 kernel: Node 0 DMA32: 10687*4kB 32*8kB 9*16kB 12*32kB 5*64kB 2*128kB 2*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB = 48716kB
Aug 13 19:06:42 amazon02 kernel: Node 0 Normal: 2030*4kB 40*8kB 31*16kB 24*32kB 9*64kB 3*128kB 0*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 11688kB
Aug 13 19:06:42 amazon02 kernel: 1313323 total pagecache pages
Aug 13 19:06:42 amazon02 kernel: 0 pages in swap cache
Aug 13 19:06:42 amazon02 kernel: Swap cache stats: add 522550, delete 522550, find 31/69
Aug 13 19:06:42 amazon02 kernel: Free swap  = 0kB
Aug 13 19:06:42 amazon02 kernel: Total swap = 0kB
Aug 13 19:06:42 amazon02 kernel: 3932159 pages RAM
Aug 13 19:06:42 amazon02 kernel: 89435 pages reserved
Aug 13 19:06:42 amazon02 kernel: 262895 pages shared
Aug 13 19:06:42 amazon02 kernel: 3786647 pages non-shared
Aug 13 19:06:42 amazon02 kernel: [ pid ]   uid  tgid total_vm      rss cpu oom_adj oom_score_adj name
Aug 13 19:06:42 amazon02 kernel: [ 9565]     0  9565    25812       36   0       0             0 ping
Aug 13 19:06:42 amazon02 kernel: [ 9566]     0  9566    25338       35   1       0             0 xargs
Aug 13 19:06:42 amazon02 kernel: [ 9569]     0  9569    25812       36   6       0             0 ping
Aug 13 19:06:42 amazon02 kernel: [ 9570]     0  9570    25338       34   6       0             0 xargs
Aug 13 19:06:42 amazon02 kernel: [ 9768]     0  9768  1668870  1589781   5       0             0 php
Aug 13 19:06:42 amazon02 kernel: Out of memory: Kill process 9768 (php) score 384 or sacrifice child
Aug 13 19:06:42 amazon02 kernel: Killed process 9768, UID 0, (php) total-vm:6675480kB, anon-rss:6358908kB, file-rss:216kB

为了缩短篇幅，其中一些进程信息去掉了
seai 2014-08-13
  • 打赏
  • 举报
 回复 
我来系统版问，大家怎么都给我php的建议？

ini_set('memory_limit','5G');的时候，没有系统OOM，但是会PHP “Allowed memory size of ”，

ini_set('memory_limit','6G');的时候，没有PHP “Allowed memory size of ”，但是会系统OOM，

现在的疑问是，系统内存空间（含buffer）有近10G多啊！！
补充swappiness： # cat /proc/sys/vm/swappiness 0
乌镇程序员 2014-08-13
  • 打赏
  • 举报
 回复
另外，你这个mycount.php脚本有没有可能真用掉10几个GB的内存啊，比如SELECT *一个超大的数据表什么的，那就得从具体的脚本代码入手解决了。
乌镇程序员 2014-08-13
  • 打赏
  • 举报
 回复
会不会是memory_limit设置引起的，执行php --ini找到命令行加载的配置文件路径，把memory_limit值放大后试试
colddown 2014-08-13
  • 打赏
  • 举报
 回复
你的系统swap怎么是0?
JVM性能调优面试必问OOM和CPU100%和死锁排查实战原理分析
本节课通过两个方面系统性的给大家分享JVM的生产问题排查实战。 第一章节，介绍OOM异常的原因，分析方法论，实战OOM讲解，工具分析dump文件，确定问题点，解决问题等分析OOM问题排查实战。 第二章节，介绍CPU100%...
java内存哪些区域会出现OOM,JVM 内存区域的划分，哪些区域可能发生 OOM
JVM 内存区域的划分，哪些区域可能发生 OOM典型回答：内存划分程序计数器(PC Program Counter Register):每个线程都有它自己的程序计数器，并且任何时候一个线程都只有一个方法在执行既当前方法。Java 虚拟机栈(JVM ...
Linux内存充足会出现oom,LINUX内存OOM哪些事
OOM的算法策略具体见下图：二、OOM机制Linux用户内存都是读写时分配，所以系统发现需要内存基本上都是发生在handle_mm_fault()的时候(其他特殊流程类似，这里忽略)，handle_mm_fault()要为缺的页分配内存，就会调...
线上系统oom，内存cpu飙高该如何处理？
1.针对线上系统发生oom，该怎么处理？ oom叫内存溢出，一般在指的是堆内存溢出。就是应用程序在运行的过程中试图申请内存空间但是堆内存已经没有足够的空间可供申请就会发生oom。 一般的原因有两种: 1.业务流量激增 ...
堆内存OOM分析
1. 背景 7月6号系统应用频繁挂掉，UMP显示内存、CPU、GC次数短时间内飙升，日志java....具体到HotSopt，栈发生OOM的机率很小（不支持栈的动态扩展，除非在线程申请栈时就内存不足）。常见发生OOM的区域主要有两种： Jav
系统维护与使用区

19,612

社区成员

74,603

社区内容

发帖
与我相关
我的任务
社区描述
系统使用、管理、维护问题。可以是Ubuntu, Fedora, Unix等等
社区管理员
  • 系统维护与使用区社区
加入社区
  • 近7日
  • 近30日
  • 至今

加载中

查看更多榜单
社区公告
暂无公告

试试用AI创作助手写篇文章吧

+ 用AI写文章