Scheduled Maintenance: We are aware of an issue with Google, AOL, and Yahoo services as email providers which are blocking new registrations. We are trying to fix the issue and we have several internal and external support tickets in process to resolve the issue. Please see: viewtopic.php?t=158230

 

 

 

[solved] Big memory issue after Debian Stretch upgrade

If none of the specific sub-forums seem right for your thread, ask here.
Post Reply
Message
Author
Sharagoz
Posts: 5
Joined: 2016-02-07 22:51

[solved] Big memory issue after Debian Stretch upgrade

#1 Post by Sharagoz »

Hi

I need help figuring out a memory problem on a debian server. The servers main purose is hosting 30 Ruby web apps. I upgraded to stretch yesterday and a couple of hours later I started getting notifications from Monit about low system memory. The apps themselves arent using more memory than normal. I've never seen this issue on jessie or wheezy and so far Im at a loss. I've upgraded 3 other systems to stretch and not seen the same problem.

I replaced the RAM chips and upgraded from 8GB to 12GB just in case, but that only delayed the system from running out of memory from 2 hours to 4 hours. After 7-8 hours the system starts killing processes. So far the only solution is to reboot the system.
Looking at the output from "ps aux" I dont see anything that's eating up memory. In /var/log/syslog I see periodic errors about "page allocation failure":

Code: Select all

Jul 15 23:30:21 odin kernel: [36637.127915] ruby: page allocation failure: order:4, mode:0x26040c0(GFP_KERNEL|__GFP_COMP|__GFP_NOTRACK)
Jul 15 23:30:22 odin kernel: [36637.127924] CPU: 8 PID: 9800 Comm: ruby Tainted: G          I     4.9.0-3-amd64 #1 Debian 4.9.30-2+deb9u2
Jul 15 23:30:22 odin kernel: [36637.127925] Hardware name: HP ProLiant ML350 G6, BIOS D22 01/30/2011
Jul 15 23:30:22 odin kernel: [36637.127927]  0000000000000000 ffffffff98d28414 ffffffff993feb20 ffffc22bc350fa18
Jul 15 23:30:22 odin kernel: [36637.127929]  ffffffff98b84f3a 026040c0026040c0 ffffffff993feb20 ffffc22bc350f9b8
Jul 15 23:30:22 odin kernel: [36637.127931]  ffffa06000000010 ffffc22bc350fa28 ffffc22bc350f9d8 fa32a9a5fc8c2ebd
Jul 15 23:30:22 odin kernel: [36637.127934] Call Trace:
Jul 15 23:30:22 odin kernel: [36637.127940]  [<ffffffff98d28414>] ? dump_stack+0x5c/0x78
Jul 15 23:30:22 odin kernel: [36637.127943]  [<ffffffff98b84f3a>] ? warn_alloc+0x13a/0x160
Jul 15 23:30:22 odin kernel: [36637.127945]  [<ffffffff98b84c7a>] ? __alloc_pages_direct_compact+0x4a/0xf0
Jul 15 23:30:22 odin kernel: [36637.127947]  [<ffffffff98b85264>] ? __alloc_pages_slowpath+0x294/0xbc0
Jul 15 23:30:22 odin kernel: [36637.127948]  [<ffffffff98b85264>] ? __alloc_pages_slowpath+0x294/0xbc0
Jul 15 23:30:22 odin kernel: [36637.127950]  [<ffffffff98b85d8e>] ? __alloc_pages_nodemask+0x1fe/0x260
Jul 15 23:30:22 odin kernel: [36637.127954]  [<ffffffff98bde16d>] ? cache_grow_begin+0x9d/0x550
Jul 15 23:30:22 odin kernel: [36637.127955]  [<ffffffff98bde16d>] ? cache_grow_begin+0x9d/0x550
Jul 15 23:30:22 odin kernel: [36637.127956]  [<ffffffff98bde8d1>] ? fallback_alloc+0x161/0x200
Jul 15 23:30:22 odin kernel: [36637.127960]  [<ffffffff98c91eea>] ? SYSC_semtimedop+0x58a/0xcd0
Jul 15 23:30:22 odin kernel: [36637.127961]  [<ffffffff98bdf761>] ? __kmalloc+0x1e1/0x560
Jul 15 23:30:22 odin kernel: [36637.127963]  [<ffffffff98c91eea>] ? SYSC_semtimedop+0x58a/0xcd0
Jul 15 23:30:22 odin kernel: [36637.127967]  [<ffffffff98ee6de0>] ? release_sock+0x40/0x90
Jul 15 23:30:22 odin kernel: [36637.127969]  [<ffffffff98f51c6f>] ? tcp_sendmsg+0x8bf/0xc00
Jul 15 23:30:22 odin kernel: [36637.127971]  [<ffffffff98d31414>] ? timerqueue_add+0x54/0xa0
Jul 15 23:30:22 odin kernel: [36637.127973]  [<ffffffff98ae5b98>] ? enqueue_hrtimer+0x38/0x80
Jul 15 23:30:22 odin kernel: [36637.127974]  [<ffffffff98ae6464>] ? hrtimer_start_range_ns+0x194/0x360
Jul 15 23:30:22 odin kernel: [36637.127976]  [<ffffffff98ae7532>] ? do_setitimer+0x1d2/0x230
Jul 15 23:30:22 odin kernel: [36637.127977]  [<ffffffff98ae76b2>] ? SyS_setitimer+0x92/0x120
Jul 15 23:30:22 odin kernel: [36637.127980]  [<ffffffff9900627b>] ? system_call_fast_compare_end+0xc/0x9b
Jul 15 23:30:22 odin kernel: [36637.127981] Mem-Info:
Jul 15 23:30:22 odin kernel: [36637.127985] active_anon:531660 inactive_anon:132876 isolated_anon:61
Jul 15 23:30:22 odin kernel: [36637.127985]  active_file:9801 inactive_file:9856 isolated_file:0
Jul 15 23:30:22 odin kernel: [36637.127985]  unevictable:0 dirty:0 writeback:353 unstable:0
Jul 15 23:30:22 odin kernel: [36637.127985]  slab_reclaimable:19576 slab_unreclaimable:15771
Jul 15 23:30:22 odin kernel: [36637.127985]  mapped:7837 shmem:178 pagetables:6100 bounce:0
Jul 15 23:30:22 odin kernel: [36637.127985]  free:30650 free_pcp:11 free_cma:0
Jul 15 23:30:22 odin kernel: [36637.127989] Node 0 active_anon:2126640kB inactive_anon:531504kB active_file:39204kB inactive_file:39424kB unevictable:0kB isolated(anon):244kB isolated(file):0kB mapped:31348kB dirty:0kB writeback:1412kB shmem:712kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 0kB writeback_tmp:0kB unstable:0kB pages_scanned:0 all_unreclaimable? no
Jul 15 23:30:22 odin kernel: [36637.127990] Node 0 DMA free:15908kB min:84kB low:104kB high:124kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15992kB managed:15908kB mlocked:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Jul 15 23:30:22 odin kernel: [36637.127993] lowmem_reserve[]: 0 3076 11967 11967 11967
Jul 15 23:30:22 odin kernel: [36637.127996] Node 0 DMA32 free:52868kB min:17352kB low:21688kB high:26024kB active_anon:27820kB inactive_anon:8080kB active_file:144kB inactive_file:524kB unevictable:0kB writepending:0kB present:3250368kB managed:3173852kB mlocked:0kB slab_reclaimable:1068kB slab_unreclaimable:5784kB kernel_stack:16kB pagetables:96kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Jul 15 23:30:22 odin kernel: [36637.128000] lowmem_reserve[]: 0 0 8890 8890 8890
Jul 15 23:30:22 odin kernel: [36637.128002] Node 0 Normal free:53824kB min:50140kB low:62672kB high:75204kB active_anon:2098820kB inactive_anon:523424kB active_file:39060kB inactive_file:38900kB unevictable:0kB writepending:1412kB present:9306108kB managed:9104276kB mlocked:0kB slab_reclaimable:77236kB slab_unreclaimable:57300kB kernel_stack:5360kB pagetables:24304kB bounce:0kB free_pcp:44kB local_pcp:0kB free_cma:0kB
Jul 15 23:30:22 odin kernel: [36637.128006] lowmem_reserve[]: 0 0 0 0 0
Jul 15 23:30:22 odin kernel: [36637.128008] Node 0 DMA: 1*4kB (U) 0*8kB 0*16kB 1*32kB (U) 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15908kB
Jul 15 23:30:22 odin kernel: [36637.128017] Node 0 DMA32: 45*4kB (UME) 294*8kB (ME) 430*16kB (ME) 470*32kB (UME) 226*64kB (UME) 85*128kB (UME) 8*256kB (ME) 0*512kB 1*1024kB (E) 0*2048kB 0*4096kB = 52868kB
Jul 15 23:30:22 odin kernel: [36637.128026] Node 0 Normal: 1794*4kB (UME) 5789*8kB (UME) 36*16kB (UM) 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 54064kB
Jul 15 23:30:22 odin kernel: [36637.128034] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Jul 15 23:30:22 odin kernel: [36637.128034] 93118 total pagecache pages
Jul 15 23:30:22 odin kernel: [36637.128035] 73394 pages in swap cache
Jul 15 23:30:22 odin kernel: [36637.128036] Swap cache stats: add 14111506, delete 14038112, find 8508247/13062492
Jul 15 23:30:22 odin kernel: [36637.128037] Free swap  = 4632260kB
Jul 15 23:30:22 odin kernel: [36637.128037] Total swap = 5846012kB
Jul 15 23:30:22 odin kernel: [36637.128038] 3143117 pages RAM
Jul 15 23:30:22 odin kernel: [36637.128038] 0 pages HighMem/MovableOnly
Jul 15 23:30:22 odin kernel: [36637.128039] 69608 pages reserved
Jul 15 23:30:22 odin kernel: [36637.128040] 0 pages hwpoisoned
Any help on how to investigate this further would be appreciated. I'll quote some outputs below.

Code: Select all

$ top
Tasks: 189 total,   2 running, 187 sleeping,   0 stopped,   0 zombie
%Cpu(s): 11.6 us,  3.7 sy,  0.0 ni, 55.2 id, 29.2 wa,  0.0 hi,  0.3 si,  0.0 st
KiB Mem : 12294036 total,   124468 free, 12004288 used,   165280 buff/cache
KiB Swap:  5846012 total,  4392540 free,  1453472 used.    53360 avail Mem 

Code: Select all

$ cat /proc/meminfo
MemTotal:       12294036 kB
MemFree:          120960 kB
MemAvailable:      28484 kB
Buffers:           27140 kB
Cached:            37904 kB
SwapCached:       244056 kB
Active:          2108308 kB
Inactive:         548892 kB
Active(anon):    2074920 kB
Inactive(anon):   518556 kB
Active(file):      33388 kB
Inactive(file):    30336 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:       5846012 kB
SwapFree:        4491776 kB
Dirty:                 0 kB
Writeback:          2148 kB
AnonPages:       2443316 kB
Mapped:            26856 kB
Shmem:              1304 kB
Slab:             120544 kB
SReclaimable:      56712 kB
SUnreclaim:        63832 kB
KernelStack:        5376 kB
PageTables:        25216 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    11993028 kB
Committed_AS:    4735892 kB
VmallocTotal:   34359738367 kB
VmallocUsed:           0 kB
VmallocChunk:          0 kB
HardwareCorrupted:     0 kB
AnonHugePages:         0 kB
ShmemHugePages:        0 kB
ShmemPmdMapped:        0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:       88252 kB
DirectMap2M:    12484608 kB

Code: Select all

$ vmstat -s
     12294036 K total memory
     12012524 K used memory
      2067832 K active memory
       535264 K inactive memory
       167376 K free memory
        27756 K buffer memory
        86380 K swap cache
      5846012 K total swap
      1279912 K used swap
      4566100 K free swap
      1741799 non-nice user cpu ticks
            0 nice user cpu ticks
       553504 system cpu ticks
      8201481 idle cpu ticks
      4685975 IO-wait cpu ticks
            0 IRQ cpu ticks
        42823 softirq cpu ticks
            0 stolen cpu ticks
     72478199 pages paged in
     35891576 pages paged out
      7177475 pages swapped in
      8365758 pages swapped out
     56148095 interrupts
     69385724 CPU context switches
   1500117582 boot time
       173809 forks

Code: Select all

$ ps aux --sort rss
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         2  0.0  0.0      0     0 ?        S    13:19   0:00 [kthreadd]
root         3  0.0  0.0      0     0 ?        S    13:19   0:36 [ksoftirqd/0]
root         5  0.0  0.0      0     0 ?        S<   13:19   0:00 [kworker/0:0H]
root         7  0.1  0.0      0     0 ?        S    13:19   0:51 [rcu_sched]
root         8  0.0  0.0      0     0 ?        S    13:19   0:00 [rcu_bh]
root         9  0.0  0.0      0     0 ?        S    13:19   0:00 [migration/0]
root        10  0.0  0.0      0     0 ?        S<   13:19   0:00 [lru-add-drain]
root        11  0.0  0.0      0     0 ?        S    13:19   0:00 [watchdog/0]
root        12  0.0  0.0      0     0 ?        S    13:19   0:00 [cpuhp/0]
root        13  0.0  0.0      0     0 ?        S    13:19   0:00 [cpuhp/4]
root        14  0.0  0.0      0     0 ?        S    13:19   0:00 [watchdog/4]
root        15  0.0  0.0      0     0 ?        S    13:19   0:00 [migration/4]
root        16  0.0  0.0      0     0 ?        S    13:19   0:34 [ksoftirqd/4]
root        18  0.0  0.0      0     0 ?        S<   13:19   0:00 [kworker/4:0H]
root        19  0.0  0.0      0     0 ?        S    13:19   0:00 [cpuhp/8]
root        20  0.0  0.0      0     0 ?        S    13:19   0:00 [watchdog/8]
root        21  0.0  0.0      0     0 ?        S    13:19   0:00 [migration/8]
root        22  0.0  0.0      0     0 ?        S    13:19   0:36 [ksoftirqd/8]
root        24  0.0  0.0      0     0 ?        S<   13:19   0:00 [kworker/8:0H]
root        25  0.0  0.0      0     0 ?        S    13:19   0:00 [cpuhp/12]
root        26  0.0  0.0      0     0 ?        S    13:19   0:00 [watchdog/12]
root        27  0.0  0.0      0     0 ?        S    13:19   0:00 [migration/12]
root        28  0.0  0.0      0     0 ?        S    13:19   0:36 [ksoftirqd/12]
root        30  0.0  0.0      0     0 ?        S<   13:19   0:00 [kworker/12:0H]
root        31  0.0  0.0      0     0 ?        S    13:19   0:00 [kdevtmpfs]
root        32  0.0  0.0      0     0 ?        S<   13:19   0:00 [netns]
root        33  0.0  0.0      0     0 ?        S    13:19   0:00 [khungtaskd]
root        34  0.0  0.0      0     0 ?        S    13:19   0:00 [oom_reaper]
root        35  0.0  0.0      0     0 ?        S<   13:19   0:00 [writeback]
root        36  0.0  0.0      0     0 ?        S    13:19   0:00 [kcompactd0]
root        37  0.0  0.0      0     0 ?        S    13:19   0:00 [kworker/0:1]
root        38  0.0  0.0      0     0 ?        SN   13:19   0:00 [ksmd]
root        39  0.0  0.0      0     0 ?        SN   13:19   0:00 [khugepaged]
root        40  0.0  0.0      0     0 ?        S<   13:19   0:00 [crypto]
root        41  0.0  0.0      0     0 ?        S<   13:19   0:00 [kintegrityd]
root        42  0.0  0.0      0     0 ?        S<   13:19   0:00 [bioset]
root        43  0.0  0.0      0     0 ?        S<   13:19   0:00 [kblockd]
root        44  0.0  0.0      0     0 ?        S<   13:19   0:00 [devfreq_wq]
root        45  0.0  0.0      0     0 ?        S<   13:19   0:00 [watchdogd]
root        49  0.4  0.0      0     0 ?        D    13:19   2:41 [kswapd0]
root        50  0.0  0.0      0     0 ?        S<   13:19   0:00 [vmstat]
root        62  0.0  0.0      0     0 ?        S<   13:19   0:00 [kthrotld]
root        63  0.0  0.0      0     0 ?        S<   13:19   0:00 [ipv6_addrconf]
root        64  0.3  0.0      0     0 ?        S    13:19   2:13 [kworker/0:2]
root        97  0.0  0.0      0     0 ?        S<   13:19   0:00 [acpi_thermal_pm]
root       107  0.0  0.0      0     0 ?        S<   13:19   0:00 [ata_sff]
root       114  0.0  0.0      0     0 ?        S    13:19   0:00 [scsi_eh_0]
root       116  0.0  0.0      0     0 ?        S<   13:19   0:00 [scsi_tmf_0]
root       118  0.0  0.0      0     0 ?        S<   13:19   0:00 [bioset]
root       119  0.0  0.0      0     0 ?        S<   13:19   0:00 [bioset]
root       120  0.0  0.0      0     0 ?        S<   13:19   0:00 [bioset]
root       121  0.0  0.0      0     0 ?        S    13:19   0:00 [scsi_eh_1]
root       122  0.0  0.0      0     0 ?        S<   13:19   0:00 [scsi_tmf_1]
root       123  0.0  0.0      0     0 ?        S<   13:19   0:00 [bioset]
root       125  0.0  0.0      0     0 ?        S<   13:19   0:00 [bioset]
root       126  0.0  0.0      0     0 ?        S<   13:19   0:00 [bioset]
root       127  0.0  0.0      0     0 ?        S<   13:19   0:00 [bioset]
root       130  0.0  0.0      0     0 ?        S<   13:19   0:00 [kworker/0:1H]
root       131  0.0  0.0      0     0 ?        S    13:19   0:00 [scsi_eh_2]
root       132  0.0  0.0      0     0 ?        S<   13:19   0:00 [scsi_tmf_2]
root       133  0.0  0.0      0     0 ?        S    13:19   0:00 [scsi_eh_3]
root       134  0.0  0.0      0     0 ?        S<   13:19   0:00 [scsi_tmf_3]
root       138  0.0  0.0      0     0 ?        S<   13:19   0:06 [kworker/4:1H]
root       140  0.0  0.0      0     0 ?        S<   13:19   0:00 [bioset]
root       142  0.0  0.0      0     0 ?        S<   13:19   0:00 [kworker/8:1H]
root       144  0.0  0.0      0     0 ?        S    13:19   0:00 [scsi_eh_4]
root       145  0.0  0.0      0     0 ?        S<   13:19   0:00 [scsi_tmf_4]
root       146  0.0  0.0      0     0 ?        S    13:19   0:00 [usb-storage]
root       148  0.0  0.0      0     0 ?        S<   13:19   0:00 [bioset]
root       149  0.0  0.0      0     0 ?        S<   13:19   0:00 [bioset]
root       151  0.0  0.0      0     0 ?        S<   13:19   0:01 [kworker/12:1H]
root       195  0.0  0.0      0     0 ?        S    13:19   0:01 [jbd2/sda1-8]
root       196  0.0  0.0      0     0 ?        S<   13:19   0:00 [ext4-rsv-conver]
root       237  0.0  0.0      0     0 ?        S    13:19   0:00 [kauditd]
root       254  0.0  0.0      0     0 ?        S<   13:19   0:00 [rpciod]
root       255  0.0  0.0      0     0 ?        S<   13:19   0:00 [xprtiod]
root       283  0.1  0.0      0     0 ?        S    13:19   0:59 [kworker/12:3]
root       320  0.0  0.0      0     0 ?        S<   13:19   0:00 [edac-poller]
message+   364  0.0  0.0  45460     0 ?        Ss   13:19   0:00 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation
root       376  0.0  0.0   4216     0 ?        Ss   13:19   0:00 /usr/sbin/acpid
root       407  0.0  0.0      0     0 ?        S<   13:19   0:00 [ttm_swap]
root       522  0.0  0.0 443760     0 ?        Ssl  13:19   0:07 /usr/local/bin/junglediskserver
sysadmin   632  0.0  0.0  82916     0 ?        S    13:19   0:00 (sd-pam)
root       764  0.0  0.0      0     0 ?        S<   13:19   0:00 [cifsiod]
root       765  0.0  0.0      0     0 ?        S<   13:19   0:00 [cifsoplockd]
root       786  0.0  0.0      0     0 ?        S    13:19   0:00 [cifsd]
root       792  0.0  0.0  16072     0 tty1     Ss+  13:19   0:00 /sbin/agetty --noclear tty1 linux
root      4378  0.1  0.0      0     0 ?        S    18:17   0:32 [kworker/8:2]
root      5689  0.0  0.0      0     0 ?        S    22:44   0:00 [kworker/u64:3]
root      7100  0.0  0.0      0     0 ?        S    22:57   0:00 [kworker/u64:1]
root      9809  0.0  0.0      0     0 ?        D    23:25   0:00 [kworker/u64:0]
root     11159  0.0  0.0      0     0 ?        S    23:38   0:00 [kworker/4:2]
root     11396  0.0  0.0      0     0 ?        S    23:41   0:00 [kworker/u64:2]
root     11398  0.0  0.0      0     0 ?        S    23:41   0:00 [kworker/u64:4]
root     11592  0.0  0.0      0     0 ?        S    23:43   0:00 [kworker/4:0]
root     14633  0.1  0.0      0     0 ?        S    13:54   0:48 [kworker/8:0]
root     15179  0.1  0.0      0     0 ?        S    18:51   0:27 [kworker/12:1]
root     30176  0.0  0.0      0     0 ?        S    21:49   0:04 [kworker/4:1]
root       367  0.0  0.0 250116    20 ?        Ssl  13:19   0:00 /usr/sbin/rsyslogd -n
sysadmin   624  0.0  0.0  64868    32 ?        Ss   13:19   0:00 /lib/systemd/systemd --user
root       324  0.0  0.0  49868    60 ?        Ss   13:19   0:00 /sbin/rpcbind -f -w
postfix   1780  0.0  0.0  83396    76 ?        S    13:20   0:00 qmgr -l -t unix -u
postfix   9094  0.0  0.0  83228    84 ?        S    23:17   0:00 pickup -l -t unix -u -c
sysadmin 10337  0.0  0.0  96864    84 ?        S    23:30   0:00 sshd: sysadmin@pts/0
root      1775  0.0  0.0  81164    88 ?        Ss   13:20   0:00 /usr/lib/postfix/sbin/master -w
root       372  0.0  0.0  27992    96 ?        Ss   13:19   0:00 /usr/sbin/cron -f
root       674  0.0  0.0  11400    96 ?        Ss   13:19   0:00 /sbin/mount.ntfs /dev/sdb1 /mybook -o rw
root       264  0.0  0.0  46864   112 ?        Ss   13:19   0:01 /lib/systemd/systemd-udevd
ntp        598  0.0  0.0 102064   152 ?        Ssl  13:19   0:02 /usr/sbin/ntpd -p /var/run/ntpd.pid -g -u 108:113
root       368  0.0  0.0  46804   160 ?        Ss   13:19   0:00 /lib/systemd/systemd-logind
root       371  0.0  0.0  35900   172 ?        Ss   13:19   0:01 /usr/sbin/irqbalance --foreground
daemon     365  0.0  0.0  27964   204 ?        Ss   13:19   0:00 /usr/sbin/atd -f
root       427  0.0  0.0  69940   260 ?        Ss   13:19   0:00 /usr/sbin/sshd -D
root     14632  0.0  0.0  43408   264 ?        S    13:54   0:04 /usr/bin/monit -c /etc/monit/monitrc
root         1  0.0  0.0  57492   340 ?        Ss   13:19   0:01 /sbin/init
root     10303  0.0  0.0  95168   536 ?        Ss   23:30   0:00 sshd: sysadmin [priv]
root       612  0.0  0.0  48180   732 ?        Ss   13:19   0:00 nginx: master process /usr/sbin/nginx -g daemon on; master_process on;
root       228  0.0  0.0  46216  1112 ?        Ss   13:19   0:01 /lib/systemd/systemd-journald
sysadmin 10338  0.0  0.0  22952  1832 pts/0    Ss   23:30   0:00 -bash
sysadmin  2089  0.0  0.0 159712  2148 ?        Sl   13:20   0:00 unicorn_bin master -D -c /apps/app1/current/config/unicorn.rb -E production
sysadmin  2149  0.0  0.0 159284  2160 ?        Sl   13:20   0:00 unicorn_bin master -D -c /apps/app2/current/config/unicorn.rb -E production
sysadmin  2155  0.0  0.0 159304  2192 ?        Sl   13:20   0:00 unicorn_bin master -D -c /apps/app3/current/config/unicorn.rb -E production
sysadmin  2202  0.0  0.0 159408  2220 ?        Sl   13:20   0:00 unicorn_bin master -D -c /apps/app4/current/config/unicorn.rb -E production
sysadmin  2781  0.0  0.0 159540  2316 ?        Sl   13:22   0:00 unicorn_bin master -D -c /apps/app5/current/config/unicorn.rb -E production
sysadmin  4449  0.0  0.0 159288  2316 ?        Sl   18:17   0:00 unicorn_bin master -D -c /apps/app6/current/config/unicorn.rb -E production
sysadmin 15205  0.0  0.0 159540  2360 ?        Sl   13:56   0:00 unicorn_bin master -D -c /apps/app7/current/config/unicorn.rb -E production
sysadmin  2027  0.0  0.0 159316  2368 ?        Sl   13:20   0:00 unicorn_bin master -D -c /apps/app8/current/config/unicorn.rb -E production
sysadmin  2012  0.0  0.0 159400  2404 ?        Sl   13:20   0:00 unicorn_bin master -D -c /apps/app9/current/config/unicorn.rb -E production
sysadmin  2179  0.0  0.0 159372  2408 ?        Sl   13:20   0:00 unicorn_bin master -D -c /apps/app10/current/config/unicorn.rb -E production
sysadmin  2131  0.0  0.0 159396  2420 ?        Sl   13:20   0:00 unicorn_bin master -D -c /apps/app11/current/config/unicorn.rb -E production
sysadmin  2119  0.0  0.0 159360  2516 ?        Sl   13:20   0:00 unicorn_bin master -D -c /apps/app12/current/config/unicorn.rb -E production
sysadmin  2113  0.0  0.0 159352  2520 ?        Sl   13:20   0:00 unicorn_bin master -D -c /apps/app13/current/config/unicorn.rb -E production
sysadmin  2125  0.0  0.0 158772  2520 ?        Sl   13:20   0:00 unicorn_bin master -D -c /apps/app14/current/config/unicorn.rb -E production
sysadmin  2053  0.0  0.0 159420  2528 ?        Sl   13:20   0:00 unicorn_bin master -D -c /apps/app15/current/config/unicorn.rb -E production
sysadmin  2059  0.0  0.0 159128  2528 ?        Sl   13:20   0:00 unicorn_bin master -D -c /apps/app16/current/config/unicorn.rb -E production
sysadmin  2101  0.0  0.0 159448  2532 ?        Sl   13:20   0:00 unicorn_bin master -D -c /apps/app17/current/config/unicorn.rb -E production
sysadmin  2185  0.0  0.0 160228  2540 ?        Sl   13:20   0:06 unicorn_bin master -D -c /apps/app18/current/config/unicorn.rb -E production
sysadmin  2047  0.0  0.0 159376  2556 ?        Sl   13:20   0:00 unicorn_bin master -D -c /apps/app19/current/config/unicorn.rb -E production
sysadmin  2065  0.0  0.0 159432  2556 ?        Sl   13:20   0:00 unicorn_bin master -D -c /apps/app20/current/config/unicorn.rb -E production
sysadmin  2143  0.0  0.0 159488  2560 ?        Sl   13:20   0:00 unicorn_bin master -D -c /apps/app21/current/config/unicorn.rb -E production
sysadmin  2167  0.0  0.0 159368  2568 ?        Sl   13:20   0:00 unicorn_bin master -D -c /apps/app22/current/config/unicorn.rb -E production
sysadmin  2021  0.0  0.0 159472  2768 ?        Sl   13:20   0:00 unicorn_bin master -D -c /apps/app23/current/config/unicorn.rb -E production
www-data   613  0.1  0.0  49312  2780 ?        S    13:19   0:48 nginx: worker process
sysadmin  2192  0.0  0.0 159380  3036 ?        Sl   13:20   0:00 unicorn_bin master -D -c /apps/app24/current/config/unicorn.rb -E production
sysadmin  2041  0.0  0.0 159172  3096 ?        Sl   13:20   0:00 unicorn_bin master -D -c /apps/app25/current/config/unicorn.rb -E production
sysadmin  2034  0.0  0.0 159372  3220 ?        Sl   13:20   0:00 unicorn_bin master -D -c /apps/app26/current/config/unicorn.rb -E production
sysadmin  2107  0.0  0.0 159584  3260 ?        Sl   13:20   0:00 unicorn_bin master -D -c /apps/app27/current/config/unicorn.rb -E production
sysadmin  2006  0.0  0.0 159420  3264 ?        Sl   13:20   0:00 unicorn_bin master -D -c /apps/app28/current/config/unicorn.rb -E production
sysadmin  2083  0.0  0.0 159000  3272 ?        Sl   13:20   0:00 unicorn_bin master -D -c /apps/app29/current/config/unicorn.rb -E production
sysadmin  2095  0.0  0.0 159376  3284 ?        Sl   13:20   0:00 unicorn_bin master -D -c /apps/app30/current/config/unicorn.rb -E production
sysadmin 11801  0.6  0.0  40152  3316 pts/0    R+   23:46   0:00 ps aux --sort rss
sysadmin  2071  0.0  0.0 158964  3360 ?        Sl   13:20   0:00 unicorn_bin master -D -c /apps/app31/current/config/unicorn.rb -E production
sysadmin  2137  0.0  0.0 159296  3396 ?        Sl   13:20   0:00 unicorn_bin master -D -c /apps/app32/current/config/unicorn.rb -E production
sysadmin  2161  0.0  0.0 159388  3656 ?        Sl   13:20   0:00 unicorn_bin master -D -c /apps/app33/current/config/unicorn.rb -E production
mongodb    538  0.2  0.0 2001460 4700 ?        Sl   13:19   1:52 /usr/bin/mongod --config /etc/mongodb.conf
sysadmin 11772  0.1  0.0 159388  6060 ?        Dl   23:45   0:00 unicorn_bin worker[0] -D -c /apps/app33/current/config/unicorn.rb -E production
sysadmin 11687  0.1  0.0 159096  9816 ?        Dl   23:45   0:00 unicorn_bin worker[0] -D -c /apps/app31/current/config/unicorn.rb -E production
sysadmin 10700  0.6  0.1 349848 13296 ?        Sl   23:34   0:04 unicorn_bin worker[0] -D -c /apps/app1/current/config/unicorn.rb -E production
sysadmin 11576  0.1  0.1 164972 19784 ?        Dl   23:43   0:00 unicorn_bin worker[0] -D -c /apps/app28/current/config/unicorn.rb -E production
sysadmin 11596  0.3  0.1 169660 23240 ?        Dl   23:43   0:00 unicorn_bin worker[0] -D -c /apps/app30/current/config/unicorn.rb -E production
sysadmin 11510  0.7  0.4 210316 49828 ?        Dl   23:42   0:01 unicorn_bin worker[0] -D -c /apps/app27/current/config/unicorn.rb -E production
sysadmin 11488  0.7  0.4 215252 53172 ?        Dl   23:42   0:01 unicorn_bin worker[0] -D -c /apps/app26/current/config/unicorn.rb -E production
sysadmin 10790  0.7  0.4 363380 53748 ?        Sl   23:35   0:05 unicorn_bin worker[0] -D -c /apps/app7/current/config/unicorn.rb -E production
sysadmin  9639  0.5  0.4 372832 56308 ?        Sl   23:23   0:06 unicorn_bin worker[1] -D -c /apps/app16/current/config/unicorn.rb -E production
sysadmin 10217  0.5  0.4 437220 57072 ?        Dl   23:29   0:05 unicorn_bin worker[0] -D -c /apps/app21/current/config/unicorn.rb -E production
sysadmin 10442  0.6  0.4 371012 59192 ?        Sl   23:31   0:06 unicorn_bin worker[0] -D -c /apps/app9/current/config/unicorn.rb -E production
sysadmin 10521  0.7  0.5 371792 63568 ?        Sl   23:32   0:06 unicorn_bin worker[0] -D -c /apps/app19/current/config/unicorn.rb -E production
sysadmin 10674  0.7  0.5 356400 65540 ?        Sl   23:34   0:05 unicorn_bin worker[0] -D -c /apps/app16/current/config/unicorn.rb -E production
sysadmin 11102  1.1  0.5 432420 65560 ?        Sl   23:38   0:05 unicorn_bin worker[0] -D -c /apps/app20/current/config/unicorn.rb -E production
sysadmin 10611  0.9  0.5 372992 67856 ?        Sl   23:33   0:06 unicorn_bin worker[0] -D -c /apps/app13/current/config/unicorn.rb -E production
sysadmin 11089  1.1  0.5 365200 70340 ?        Sl   23:37   0:05 unicorn_bin worker[0] -D -c /apps/app17/current/config/unicorn.rb -E production
sysadmin 11427  1.0  0.5 295068 72612 ?        Dl   23:42   0:02 unicorn_bin worker[0] -D -c /apps/app32/current/config/unicorn.rb -E production
sysadmin 11288  0.8  0.5 298980 73360 ?        Dl   23:40   0:02 unicorn_bin worker[0] -D -c /apps/app10/current/config/unicorn.rb -E production
sysadmin 11170  1.1  0.6 363160 74384 ?        Sl   23:38   0:05 unicorn_bin worker[0] -D -c /apps/app11/current/config/unicorn.rb -E production
sysadmin 10588  0.9  0.6 493632 75100 ?        Dl   23:33   0:07 unicorn_bin worker[0] -D -c /apps/app14/current/config/unicorn.rb -E production
sysadmin 11106  1.2  0.6 364900 81756 ?        Sl   23:38   0:05 unicorn_bin worker[0] -D -c /apps/app12/current/config/unicorn.rb -E production
sysadmin 11178  1.2  0.7 361828 88628 ?        Sl   23:38   0:05 unicorn_bin worker[0] -D -c /apps/app23/current/config/unicorn.rb -E production
sysadmin 11409  1.4  0.7 349048 88840 ?        Dl   23:41   0:03 unicorn_bin worker[0] -D -c /apps/app29/current/config/unicorn.rb -E production
sysadmin 11400  1.7  0.7 355636 93176 ?        Dl   23:41   0:04 unicorn_bin worker[0] -D -c /apps/app25/current/config/unicorn.rb -E production
sysadmin 11175  0.8  0.7 322132 95716 ?        Dl   23:38   0:03 unicorn_bin worker[0] -D -c /apps/app6/current/config/unicorn.rb -E production
sysadmin 11252  0.9  0.7 322268 95880 ?        Dl   23:39   0:03 unicorn_bin worker[0] -D -c /apps/app22/current/config/unicorn.rb -E production
sysadmin 11169  1.2  0.7 362224 97484 ?        Sl   23:38   0:05 unicorn_bin worker[0] -D -c /apps/app15/current/config/unicorn.rb -E production
sysadmin 10993  1.3  0.8 386516 103752 ?       Sl   23:37   0:07 unicorn_bin worker[1] -D -c /apps/app25/current/config/unicorn.rb -E production
sysadmin 11047  0.8  0.8 354136 106652 ?       Dl   23:37   0:04 unicorn_bin worker[0] -D -c /apps/app18/current/config/unicorn.rb -E production
sysadmin 10555  1.2  0.8 469496 109208 ?       Sl   23:32   0:10 unicorn_bin worker[0] -D -c /apps/app5/current/config/unicorn.rb -E production
sysadmin 11282  1.8  0.9 380480 114992 ?       Sl   23:40   0:06 unicorn_bin worker[0] -D -c /apps/app8/current/config/unicorn.rb -E production
sysadmin 11210  1.0  0.9 343824 117296 ?       Dl   23:39   0:04 unicorn_bin worker[0] -D -c /apps/app4/current/config/unicorn.rb -E production
sysadmin 10897  1.6  1.0 467556 123780 ?       Sl   23:36   0:10 unicorn_bin worker[0] -D -c /apps/app3/current/config/unicorn.rb -E production
sysadmin 11027  1.9  1.1 470524 141648 ?       Sl   23:37   0:10 unicorn_bin worker[0] -D -c /apps/app2/current/config/unicorn.rb -E production
sysadmin 10784  1.6  1.1 470824 145132 ?       Sl   23:35   0:11 unicorn_bin worker[0] -D -c /apps/app24/current/config/unicorn.rb -E production
Last edited by Sharagoz on 2017-07-18 12:32, edited 1 time in total.

Sharagoz
Posts: 5
Joined: 2016-02-07 22:51

Re: Big memory issue after Debian Stretch upgrade

#2 Post by Sharagoz »

I've been investigating this all day and I still havent been able to figure out what the problem is. Something is eating 1-1.5GB of memory per hour and I cant determine what the source is. Nothing suspicious is coming up in top, slabtop, atop or htop. It looks like I'll have to bite the bullet and reinstall if I cant figure it out by tomorrow night.

Sharagoz
Posts: 5
Joined: 2016-02-07 22:51

Re: Big memory issue after Debian Stretch upgrade

#3 Post by Sharagoz »

Solved by upgrading the Unicorn HTTP server from v4.9 to v5.3. I dont know whether the update fixed the problem or if it was the act of reinstalling that solved issue. The weird thing is that I was never able to pin point the problem with the help of monitoring tools or logging.

Post Reply