I need help figuring out a memory problem on a debian server. The servers main purose is hosting 30 Ruby web apps. I upgraded to stretch yesterday and a couple of hours later I started getting notifications from Monit about low system memory. The apps themselves arent using more memory than normal. I've never seen this issue on jessie or wheezy and so far Im at a loss. I've upgraded 3 other systems to stretch and not seen the same problem.
I replaced the RAM chips and upgraded from 8GB to 12GB just in case, but that only delayed the system from running out of memory from 2 hours to 4 hours. After 7-8 hours the system starts killing processes. So far the only solution is to reboot the system.
Looking at the output from "ps aux" I dont see anything that's eating up memory. In /var/log/syslog I see periodic errors about "page allocation failure":
Code: Select all
Jul 15 23:30:21 odin kernel: [36637.127915] ruby: page allocation failure: order:4, mode:0x26040c0(GFP_KERNEL|__GFP_COMP|__GFP_NOTRACK)
Jul 15 23:30:22 odin kernel: [36637.127924] CPU: 8 PID: 9800 Comm: ruby Tainted: G I 4.9.0-3-amd64 #1 Debian 4.9.30-2+deb9u2
Jul 15 23:30:22 odin kernel: [36637.127925] Hardware name: HP ProLiant ML350 G6, BIOS D22 01/30/2011
Jul 15 23:30:22 odin kernel: [36637.127927] 0000000000000000 ffffffff98d28414 ffffffff993feb20 ffffc22bc350fa18
Jul 15 23:30:22 odin kernel: [36637.127929] ffffffff98b84f3a 026040c0026040c0 ffffffff993feb20 ffffc22bc350f9b8
Jul 15 23:30:22 odin kernel: [36637.127931] ffffa06000000010 ffffc22bc350fa28 ffffc22bc350f9d8 fa32a9a5fc8c2ebd
Jul 15 23:30:22 odin kernel: [36637.127934] Call Trace:
Jul 15 23:30:22 odin kernel: [36637.127940] [<ffffffff98d28414>] ? dump_stack+0x5c/0x78
Jul 15 23:30:22 odin kernel: [36637.127943] [<ffffffff98b84f3a>] ? warn_alloc+0x13a/0x160
Jul 15 23:30:22 odin kernel: [36637.127945] [<ffffffff98b84c7a>] ? __alloc_pages_direct_compact+0x4a/0xf0
Jul 15 23:30:22 odin kernel: [36637.127947] [<ffffffff98b85264>] ? __alloc_pages_slowpath+0x294/0xbc0
Jul 15 23:30:22 odin kernel: [36637.127948] [<ffffffff98b85264>] ? __alloc_pages_slowpath+0x294/0xbc0
Jul 15 23:30:22 odin kernel: [36637.127950] [<ffffffff98b85d8e>] ? __alloc_pages_nodemask+0x1fe/0x260
Jul 15 23:30:22 odin kernel: [36637.127954] [<ffffffff98bde16d>] ? cache_grow_begin+0x9d/0x550
Jul 15 23:30:22 odin kernel: [36637.127955] [<ffffffff98bde16d>] ? cache_grow_begin+0x9d/0x550
Jul 15 23:30:22 odin kernel: [36637.127956] [<ffffffff98bde8d1>] ? fallback_alloc+0x161/0x200
Jul 15 23:30:22 odin kernel: [36637.127960] [<ffffffff98c91eea>] ? SYSC_semtimedop+0x58a/0xcd0
Jul 15 23:30:22 odin kernel: [36637.127961] [<ffffffff98bdf761>] ? __kmalloc+0x1e1/0x560
Jul 15 23:30:22 odin kernel: [36637.127963] [<ffffffff98c91eea>] ? SYSC_semtimedop+0x58a/0xcd0
Jul 15 23:30:22 odin kernel: [36637.127967] [<ffffffff98ee6de0>] ? release_sock+0x40/0x90
Jul 15 23:30:22 odin kernel: [36637.127969] [<ffffffff98f51c6f>] ? tcp_sendmsg+0x8bf/0xc00
Jul 15 23:30:22 odin kernel: [36637.127971] [<ffffffff98d31414>] ? timerqueue_add+0x54/0xa0
Jul 15 23:30:22 odin kernel: [36637.127973] [<ffffffff98ae5b98>] ? enqueue_hrtimer+0x38/0x80
Jul 15 23:30:22 odin kernel: [36637.127974] [<ffffffff98ae6464>] ? hrtimer_start_range_ns+0x194/0x360
Jul 15 23:30:22 odin kernel: [36637.127976] [<ffffffff98ae7532>] ? do_setitimer+0x1d2/0x230
Jul 15 23:30:22 odin kernel: [36637.127977] [<ffffffff98ae76b2>] ? SyS_setitimer+0x92/0x120
Jul 15 23:30:22 odin kernel: [36637.127980] [<ffffffff9900627b>] ? system_call_fast_compare_end+0xc/0x9b
Jul 15 23:30:22 odin kernel: [36637.127981] Mem-Info:
Jul 15 23:30:22 odin kernel: [36637.127985] active_anon:531660 inactive_anon:132876 isolated_anon:61
Jul 15 23:30:22 odin kernel: [36637.127985] active_file:9801 inactive_file:9856 isolated_file:0
Jul 15 23:30:22 odin kernel: [36637.127985] unevictable:0 dirty:0 writeback:353 unstable:0
Jul 15 23:30:22 odin kernel: [36637.127985] slab_reclaimable:19576 slab_unreclaimable:15771
Jul 15 23:30:22 odin kernel: [36637.127985] mapped:7837 shmem:178 pagetables:6100 bounce:0
Jul 15 23:30:22 odin kernel: [36637.127985] free:30650 free_pcp:11 free_cma:0
Jul 15 23:30:22 odin kernel: [36637.127989] Node 0 active_anon:2126640kB inactive_anon:531504kB active_file:39204kB inactive_file:39424kB unevictable:0kB isolated(anon):244kB isolated(file):0kB mapped:31348kB dirty:0kB writeback:1412kB shmem:712kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 0kB writeback_tmp:0kB unstable:0kB pages_scanned:0 all_unreclaimable? no
Jul 15 23:30:22 odin kernel: [36637.127990] Node 0 DMA free:15908kB min:84kB low:104kB high:124kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15992kB managed:15908kB mlocked:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Jul 15 23:30:22 odin kernel: [36637.127993] lowmem_reserve[]: 0 3076 11967 11967 11967
Jul 15 23:30:22 odin kernel: [36637.127996] Node 0 DMA32 free:52868kB min:17352kB low:21688kB high:26024kB active_anon:27820kB inactive_anon:8080kB active_file:144kB inactive_file:524kB unevictable:0kB writepending:0kB present:3250368kB managed:3173852kB mlocked:0kB slab_reclaimable:1068kB slab_unreclaimable:5784kB kernel_stack:16kB pagetables:96kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Jul 15 23:30:22 odin kernel: [36637.128000] lowmem_reserve[]: 0 0 8890 8890 8890
Jul 15 23:30:22 odin kernel: [36637.128002] Node 0 Normal free:53824kB min:50140kB low:62672kB high:75204kB active_anon:2098820kB inactive_anon:523424kB active_file:39060kB inactive_file:38900kB unevictable:0kB writepending:1412kB present:9306108kB managed:9104276kB mlocked:0kB slab_reclaimable:77236kB slab_unreclaimable:57300kB kernel_stack:5360kB pagetables:24304kB bounce:0kB free_pcp:44kB local_pcp:0kB free_cma:0kB
Jul 15 23:30:22 odin kernel: [36637.128006] lowmem_reserve[]: 0 0 0 0 0
Jul 15 23:30:22 odin kernel: [36637.128008] Node 0 DMA: 1*4kB (U) 0*8kB 0*16kB 1*32kB (U) 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15908kB
Jul 15 23:30:22 odin kernel: [36637.128017] Node 0 DMA32: 45*4kB (UME) 294*8kB (ME) 430*16kB (ME) 470*32kB (UME) 226*64kB (UME) 85*128kB (UME) 8*256kB (ME) 0*512kB 1*1024kB (E) 0*2048kB 0*4096kB = 52868kB
Jul 15 23:30:22 odin kernel: [36637.128026] Node 0 Normal: 1794*4kB (UME) 5789*8kB (UME) 36*16kB (UM) 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 54064kB
Jul 15 23:30:22 odin kernel: [36637.128034] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Jul 15 23:30:22 odin kernel: [36637.128034] 93118 total pagecache pages
Jul 15 23:30:22 odin kernel: [36637.128035] 73394 pages in swap cache
Jul 15 23:30:22 odin kernel: [36637.128036] Swap cache stats: add 14111506, delete 14038112, find 8508247/13062492
Jul 15 23:30:22 odin kernel: [36637.128037] Free swap = 4632260kB
Jul 15 23:30:22 odin kernel: [36637.128037] Total swap = 5846012kB
Jul 15 23:30:22 odin kernel: [36637.128038] 3143117 pages RAM
Jul 15 23:30:22 odin kernel: [36637.128038] 0 pages HighMem/MovableOnly
Jul 15 23:30:22 odin kernel: [36637.128039] 69608 pages reserved
Jul 15 23:30:22 odin kernel: [36637.128040] 0 pages hwpoisoned
Code: Select all
$ top
Tasks: 189 total, 2 running, 187 sleeping, 0 stopped, 0 zombie
%Cpu(s): 11.6 us, 3.7 sy, 0.0 ni, 55.2 id, 29.2 wa, 0.0 hi, 0.3 si, 0.0 st
KiB Mem : 12294036 total, 124468 free, 12004288 used, 165280 buff/cache
KiB Swap: 5846012 total, 4392540 free, 1453472 used. 53360 avail Mem
Code: Select all
$ cat /proc/meminfo
MemTotal: 12294036 kB
MemFree: 120960 kB
MemAvailable: 28484 kB
Buffers: 27140 kB
Cached: 37904 kB
SwapCached: 244056 kB
Active: 2108308 kB
Inactive: 548892 kB
Active(anon): 2074920 kB
Inactive(anon): 518556 kB
Active(file): 33388 kB
Inactive(file): 30336 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 5846012 kB
SwapFree: 4491776 kB
Dirty: 0 kB
Writeback: 2148 kB
AnonPages: 2443316 kB
Mapped: 26856 kB
Shmem: 1304 kB
Slab: 120544 kB
SReclaimable: 56712 kB
SUnreclaim: 63832 kB
KernelStack: 5376 kB
PageTables: 25216 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 11993028 kB
Committed_AS: 4735892 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 0 kB
VmallocChunk: 0 kB
HardwareCorrupted: 0 kB
AnonHugePages: 0 kB
ShmemHugePages: 0 kB
ShmemPmdMapped: 0 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
DirectMap4k: 88252 kB
DirectMap2M: 12484608 kB
Code: Select all
$ vmstat -s
12294036 K total memory
12012524 K used memory
2067832 K active memory
535264 K inactive memory
167376 K free memory
27756 K buffer memory
86380 K swap cache
5846012 K total swap
1279912 K used swap
4566100 K free swap
1741799 non-nice user cpu ticks
0 nice user cpu ticks
553504 system cpu ticks
8201481 idle cpu ticks
4685975 IO-wait cpu ticks
0 IRQ cpu ticks
42823 softirq cpu ticks
0 stolen cpu ticks
72478199 pages paged in
35891576 pages paged out
7177475 pages swapped in
8365758 pages swapped out
56148095 interrupts
69385724 CPU context switches
1500117582 boot time
173809 forks
Code: Select all
$ ps aux --sort rss
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 2 0.0 0.0 0 0 ? S 13:19 0:00 [kthreadd]
root 3 0.0 0.0 0 0 ? S 13:19 0:36 [ksoftirqd/0]
root 5 0.0 0.0 0 0 ? S< 13:19 0:00 [kworker/0:0H]
root 7 0.1 0.0 0 0 ? S 13:19 0:51 [rcu_sched]
root 8 0.0 0.0 0 0 ? S 13:19 0:00 [rcu_bh]
root 9 0.0 0.0 0 0 ? S 13:19 0:00 [migration/0]
root 10 0.0 0.0 0 0 ? S< 13:19 0:00 [lru-add-drain]
root 11 0.0 0.0 0 0 ? S 13:19 0:00 [watchdog/0]
root 12 0.0 0.0 0 0 ? S 13:19 0:00 [cpuhp/0]
root 13 0.0 0.0 0 0 ? S 13:19 0:00 [cpuhp/4]
root 14 0.0 0.0 0 0 ? S 13:19 0:00 [watchdog/4]
root 15 0.0 0.0 0 0 ? S 13:19 0:00 [migration/4]
root 16 0.0 0.0 0 0 ? S 13:19 0:34 [ksoftirqd/4]
root 18 0.0 0.0 0 0 ? S< 13:19 0:00 [kworker/4:0H]
root 19 0.0 0.0 0 0 ? S 13:19 0:00 [cpuhp/8]
root 20 0.0 0.0 0 0 ? S 13:19 0:00 [watchdog/8]
root 21 0.0 0.0 0 0 ? S 13:19 0:00 [migration/8]
root 22 0.0 0.0 0 0 ? S 13:19 0:36 [ksoftirqd/8]
root 24 0.0 0.0 0 0 ? S< 13:19 0:00 [kworker/8:0H]
root 25 0.0 0.0 0 0 ? S 13:19 0:00 [cpuhp/12]
root 26 0.0 0.0 0 0 ? S 13:19 0:00 [watchdog/12]
root 27 0.0 0.0 0 0 ? S 13:19 0:00 [migration/12]
root 28 0.0 0.0 0 0 ? S 13:19 0:36 [ksoftirqd/12]
root 30 0.0 0.0 0 0 ? S< 13:19 0:00 [kworker/12:0H]
root 31 0.0 0.0 0 0 ? S 13:19 0:00 [kdevtmpfs]
root 32 0.0 0.0 0 0 ? S< 13:19 0:00 [netns]
root 33 0.0 0.0 0 0 ? S 13:19 0:00 [khungtaskd]
root 34 0.0 0.0 0 0 ? S 13:19 0:00 [oom_reaper]
root 35 0.0 0.0 0 0 ? S< 13:19 0:00 [writeback]
root 36 0.0 0.0 0 0 ? S 13:19 0:00 [kcompactd0]
root 37 0.0 0.0 0 0 ? S 13:19 0:00 [kworker/0:1]
root 38 0.0 0.0 0 0 ? SN 13:19 0:00 [ksmd]
root 39 0.0 0.0 0 0 ? SN 13:19 0:00 [khugepaged]
root 40 0.0 0.0 0 0 ? S< 13:19 0:00 [crypto]
root 41 0.0 0.0 0 0 ? S< 13:19 0:00 [kintegrityd]
root 42 0.0 0.0 0 0 ? S< 13:19 0:00 [bioset]
root 43 0.0 0.0 0 0 ? S< 13:19 0:00 [kblockd]
root 44 0.0 0.0 0 0 ? S< 13:19 0:00 [devfreq_wq]
root 45 0.0 0.0 0 0 ? S< 13:19 0:00 [watchdogd]
root 49 0.4 0.0 0 0 ? D 13:19 2:41 [kswapd0]
root 50 0.0 0.0 0 0 ? S< 13:19 0:00 [vmstat]
root 62 0.0 0.0 0 0 ? S< 13:19 0:00 [kthrotld]
root 63 0.0 0.0 0 0 ? S< 13:19 0:00 [ipv6_addrconf]
root 64 0.3 0.0 0 0 ? S 13:19 2:13 [kworker/0:2]
root 97 0.0 0.0 0 0 ? S< 13:19 0:00 [acpi_thermal_pm]
root 107 0.0 0.0 0 0 ? S< 13:19 0:00 [ata_sff]
root 114 0.0 0.0 0 0 ? S 13:19 0:00 [scsi_eh_0]
root 116 0.0 0.0 0 0 ? S< 13:19 0:00 [scsi_tmf_0]
root 118 0.0 0.0 0 0 ? S< 13:19 0:00 [bioset]
root 119 0.0 0.0 0 0 ? S< 13:19 0:00 [bioset]
root 120 0.0 0.0 0 0 ? S< 13:19 0:00 [bioset]
root 121 0.0 0.0 0 0 ? S 13:19 0:00 [scsi_eh_1]
root 122 0.0 0.0 0 0 ? S< 13:19 0:00 [scsi_tmf_1]
root 123 0.0 0.0 0 0 ? S< 13:19 0:00 [bioset]
root 125 0.0 0.0 0 0 ? S< 13:19 0:00 [bioset]
root 126 0.0 0.0 0 0 ? S< 13:19 0:00 [bioset]
root 127 0.0 0.0 0 0 ? S< 13:19 0:00 [bioset]
root 130 0.0 0.0 0 0 ? S< 13:19 0:00 [kworker/0:1H]
root 131 0.0 0.0 0 0 ? S 13:19 0:00 [scsi_eh_2]
root 132 0.0 0.0 0 0 ? S< 13:19 0:00 [scsi_tmf_2]
root 133 0.0 0.0 0 0 ? S 13:19 0:00 [scsi_eh_3]
root 134 0.0 0.0 0 0 ? S< 13:19 0:00 [scsi_tmf_3]
root 138 0.0 0.0 0 0 ? S< 13:19 0:06 [kworker/4:1H]
root 140 0.0 0.0 0 0 ? S< 13:19 0:00 [bioset]
root 142 0.0 0.0 0 0 ? S< 13:19 0:00 [kworker/8:1H]
root 144 0.0 0.0 0 0 ? S 13:19 0:00 [scsi_eh_4]
root 145 0.0 0.0 0 0 ? S< 13:19 0:00 [scsi_tmf_4]
root 146 0.0 0.0 0 0 ? S 13:19 0:00 [usb-storage]
root 148 0.0 0.0 0 0 ? S< 13:19 0:00 [bioset]
root 149 0.0 0.0 0 0 ? S< 13:19 0:00 [bioset]
root 151 0.0 0.0 0 0 ? S< 13:19 0:01 [kworker/12:1H]
root 195 0.0 0.0 0 0 ? S 13:19 0:01 [jbd2/sda1-8]
root 196 0.0 0.0 0 0 ? S< 13:19 0:00 [ext4-rsv-conver]
root 237 0.0 0.0 0 0 ? S 13:19 0:00 [kauditd]
root 254 0.0 0.0 0 0 ? S< 13:19 0:00 [rpciod]
root 255 0.0 0.0 0 0 ? S< 13:19 0:00 [xprtiod]
root 283 0.1 0.0 0 0 ? S 13:19 0:59 [kworker/12:3]
root 320 0.0 0.0 0 0 ? S< 13:19 0:00 [edac-poller]
message+ 364 0.0 0.0 45460 0 ? Ss 13:19 0:00 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation
root 376 0.0 0.0 4216 0 ? Ss 13:19 0:00 /usr/sbin/acpid
root 407 0.0 0.0 0 0 ? S< 13:19 0:00 [ttm_swap]
root 522 0.0 0.0 443760 0 ? Ssl 13:19 0:07 /usr/local/bin/junglediskserver
sysadmin 632 0.0 0.0 82916 0 ? S 13:19 0:00 (sd-pam)
root 764 0.0 0.0 0 0 ? S< 13:19 0:00 [cifsiod]
root 765 0.0 0.0 0 0 ? S< 13:19 0:00 [cifsoplockd]
root 786 0.0 0.0 0 0 ? S 13:19 0:00 [cifsd]
root 792 0.0 0.0 16072 0 tty1 Ss+ 13:19 0:00 /sbin/agetty --noclear tty1 linux
root 4378 0.1 0.0 0 0 ? S 18:17 0:32 [kworker/8:2]
root 5689 0.0 0.0 0 0 ? S 22:44 0:00 [kworker/u64:3]
root 7100 0.0 0.0 0 0 ? S 22:57 0:00 [kworker/u64:1]
root 9809 0.0 0.0 0 0 ? D 23:25 0:00 [kworker/u64:0]
root 11159 0.0 0.0 0 0 ? S 23:38 0:00 [kworker/4:2]
root 11396 0.0 0.0 0 0 ? S 23:41 0:00 [kworker/u64:2]
root 11398 0.0 0.0 0 0 ? S 23:41 0:00 [kworker/u64:4]
root 11592 0.0 0.0 0 0 ? S 23:43 0:00 [kworker/4:0]
root 14633 0.1 0.0 0 0 ? S 13:54 0:48 [kworker/8:0]
root 15179 0.1 0.0 0 0 ? S 18:51 0:27 [kworker/12:1]
root 30176 0.0 0.0 0 0 ? S 21:49 0:04 [kworker/4:1]
root 367 0.0 0.0 250116 20 ? Ssl 13:19 0:00 /usr/sbin/rsyslogd -n
sysadmin 624 0.0 0.0 64868 32 ? Ss 13:19 0:00 /lib/systemd/systemd --user
root 324 0.0 0.0 49868 60 ? Ss 13:19 0:00 /sbin/rpcbind -f -w
postfix 1780 0.0 0.0 83396 76 ? S 13:20 0:00 qmgr -l -t unix -u
postfix 9094 0.0 0.0 83228 84 ? S 23:17 0:00 pickup -l -t unix -u -c
sysadmin 10337 0.0 0.0 96864 84 ? S 23:30 0:00 sshd: sysadmin@pts/0
root 1775 0.0 0.0 81164 88 ? Ss 13:20 0:00 /usr/lib/postfix/sbin/master -w
root 372 0.0 0.0 27992 96 ? Ss 13:19 0:00 /usr/sbin/cron -f
root 674 0.0 0.0 11400 96 ? Ss 13:19 0:00 /sbin/mount.ntfs /dev/sdb1 /mybook -o rw
root 264 0.0 0.0 46864 112 ? Ss 13:19 0:01 /lib/systemd/systemd-udevd
ntp 598 0.0 0.0 102064 152 ? Ssl 13:19 0:02 /usr/sbin/ntpd -p /var/run/ntpd.pid -g -u 108:113
root 368 0.0 0.0 46804 160 ? Ss 13:19 0:00 /lib/systemd/systemd-logind
root 371 0.0 0.0 35900 172 ? Ss 13:19 0:01 /usr/sbin/irqbalance --foreground
daemon 365 0.0 0.0 27964 204 ? Ss 13:19 0:00 /usr/sbin/atd -f
root 427 0.0 0.0 69940 260 ? Ss 13:19 0:00 /usr/sbin/sshd -D
root 14632 0.0 0.0 43408 264 ? S 13:54 0:04 /usr/bin/monit -c /etc/monit/monitrc
root 1 0.0 0.0 57492 340 ? Ss 13:19 0:01 /sbin/init
root 10303 0.0 0.0 95168 536 ? Ss 23:30 0:00 sshd: sysadmin [priv]
root 612 0.0 0.0 48180 732 ? Ss 13:19 0:00 nginx: master process /usr/sbin/nginx -g daemon on; master_process on;
root 228 0.0 0.0 46216 1112 ? Ss 13:19 0:01 /lib/systemd/systemd-journald
sysadmin 10338 0.0 0.0 22952 1832 pts/0 Ss 23:30 0:00 -bash
sysadmin 2089 0.0 0.0 159712 2148 ? Sl 13:20 0:00 unicorn_bin master -D -c /apps/app1/current/config/unicorn.rb -E production
sysadmin 2149 0.0 0.0 159284 2160 ? Sl 13:20 0:00 unicorn_bin master -D -c /apps/app2/current/config/unicorn.rb -E production
sysadmin 2155 0.0 0.0 159304 2192 ? Sl 13:20 0:00 unicorn_bin master -D -c /apps/app3/current/config/unicorn.rb -E production
sysadmin 2202 0.0 0.0 159408 2220 ? Sl 13:20 0:00 unicorn_bin master -D -c /apps/app4/current/config/unicorn.rb -E production
sysadmin 2781 0.0 0.0 159540 2316 ? Sl 13:22 0:00 unicorn_bin master -D -c /apps/app5/current/config/unicorn.rb -E production
sysadmin 4449 0.0 0.0 159288 2316 ? Sl 18:17 0:00 unicorn_bin master -D -c /apps/app6/current/config/unicorn.rb -E production
sysadmin 15205 0.0 0.0 159540 2360 ? Sl 13:56 0:00 unicorn_bin master -D -c /apps/app7/current/config/unicorn.rb -E production
sysadmin 2027 0.0 0.0 159316 2368 ? Sl 13:20 0:00 unicorn_bin master -D -c /apps/app8/current/config/unicorn.rb -E production
sysadmin 2012 0.0 0.0 159400 2404 ? Sl 13:20 0:00 unicorn_bin master -D -c /apps/app9/current/config/unicorn.rb -E production
sysadmin 2179 0.0 0.0 159372 2408 ? Sl 13:20 0:00 unicorn_bin master -D -c /apps/app10/current/config/unicorn.rb -E production
sysadmin 2131 0.0 0.0 159396 2420 ? Sl 13:20 0:00 unicorn_bin master -D -c /apps/app11/current/config/unicorn.rb -E production
sysadmin 2119 0.0 0.0 159360 2516 ? Sl 13:20 0:00 unicorn_bin master -D -c /apps/app12/current/config/unicorn.rb -E production
sysadmin 2113 0.0 0.0 159352 2520 ? Sl 13:20 0:00 unicorn_bin master -D -c /apps/app13/current/config/unicorn.rb -E production
sysadmin 2125 0.0 0.0 158772 2520 ? Sl 13:20 0:00 unicorn_bin master -D -c /apps/app14/current/config/unicorn.rb -E production
sysadmin 2053 0.0 0.0 159420 2528 ? Sl 13:20 0:00 unicorn_bin master -D -c /apps/app15/current/config/unicorn.rb -E production
sysadmin 2059 0.0 0.0 159128 2528 ? Sl 13:20 0:00 unicorn_bin master -D -c /apps/app16/current/config/unicorn.rb -E production
sysadmin 2101 0.0 0.0 159448 2532 ? Sl 13:20 0:00 unicorn_bin master -D -c /apps/app17/current/config/unicorn.rb -E production
sysadmin 2185 0.0 0.0 160228 2540 ? Sl 13:20 0:06 unicorn_bin master -D -c /apps/app18/current/config/unicorn.rb -E production
sysadmin 2047 0.0 0.0 159376 2556 ? Sl 13:20 0:00 unicorn_bin master -D -c /apps/app19/current/config/unicorn.rb -E production
sysadmin 2065 0.0 0.0 159432 2556 ? Sl 13:20 0:00 unicorn_bin master -D -c /apps/app20/current/config/unicorn.rb -E production
sysadmin 2143 0.0 0.0 159488 2560 ? Sl 13:20 0:00 unicorn_bin master -D -c /apps/app21/current/config/unicorn.rb -E production
sysadmin 2167 0.0 0.0 159368 2568 ? Sl 13:20 0:00 unicorn_bin master -D -c /apps/app22/current/config/unicorn.rb -E production
sysadmin 2021 0.0 0.0 159472 2768 ? Sl 13:20 0:00 unicorn_bin master -D -c /apps/app23/current/config/unicorn.rb -E production
www-data 613 0.1 0.0 49312 2780 ? S 13:19 0:48 nginx: worker process
sysadmin 2192 0.0 0.0 159380 3036 ? Sl 13:20 0:00 unicorn_bin master -D -c /apps/app24/current/config/unicorn.rb -E production
sysadmin 2041 0.0 0.0 159172 3096 ? Sl 13:20 0:00 unicorn_bin master -D -c /apps/app25/current/config/unicorn.rb -E production
sysadmin 2034 0.0 0.0 159372 3220 ? Sl 13:20 0:00 unicorn_bin master -D -c /apps/app26/current/config/unicorn.rb -E production
sysadmin 2107 0.0 0.0 159584 3260 ? Sl 13:20 0:00 unicorn_bin master -D -c /apps/app27/current/config/unicorn.rb -E production
sysadmin 2006 0.0 0.0 159420 3264 ? Sl 13:20 0:00 unicorn_bin master -D -c /apps/app28/current/config/unicorn.rb -E production
sysadmin 2083 0.0 0.0 159000 3272 ? Sl 13:20 0:00 unicorn_bin master -D -c /apps/app29/current/config/unicorn.rb -E production
sysadmin 2095 0.0 0.0 159376 3284 ? Sl 13:20 0:00 unicorn_bin master -D -c /apps/app30/current/config/unicorn.rb -E production
sysadmin 11801 0.6 0.0 40152 3316 pts/0 R+ 23:46 0:00 ps aux --sort rss
sysadmin 2071 0.0 0.0 158964 3360 ? Sl 13:20 0:00 unicorn_bin master -D -c /apps/app31/current/config/unicorn.rb -E production
sysadmin 2137 0.0 0.0 159296 3396 ? Sl 13:20 0:00 unicorn_bin master -D -c /apps/app32/current/config/unicorn.rb -E production
sysadmin 2161 0.0 0.0 159388 3656 ? Sl 13:20 0:00 unicorn_bin master -D -c /apps/app33/current/config/unicorn.rb -E production
mongodb 538 0.2 0.0 2001460 4700 ? Sl 13:19 1:52 /usr/bin/mongod --config /etc/mongodb.conf
sysadmin 11772 0.1 0.0 159388 6060 ? Dl 23:45 0:00 unicorn_bin worker[0] -D -c /apps/app33/current/config/unicorn.rb -E production
sysadmin 11687 0.1 0.0 159096 9816 ? Dl 23:45 0:00 unicorn_bin worker[0] -D -c /apps/app31/current/config/unicorn.rb -E production
sysadmin 10700 0.6 0.1 349848 13296 ? Sl 23:34 0:04 unicorn_bin worker[0] -D -c /apps/app1/current/config/unicorn.rb -E production
sysadmin 11576 0.1 0.1 164972 19784 ? Dl 23:43 0:00 unicorn_bin worker[0] -D -c /apps/app28/current/config/unicorn.rb -E production
sysadmin 11596 0.3 0.1 169660 23240 ? Dl 23:43 0:00 unicorn_bin worker[0] -D -c /apps/app30/current/config/unicorn.rb -E production
sysadmin 11510 0.7 0.4 210316 49828 ? Dl 23:42 0:01 unicorn_bin worker[0] -D -c /apps/app27/current/config/unicorn.rb -E production
sysadmin 11488 0.7 0.4 215252 53172 ? Dl 23:42 0:01 unicorn_bin worker[0] -D -c /apps/app26/current/config/unicorn.rb -E production
sysadmin 10790 0.7 0.4 363380 53748 ? Sl 23:35 0:05 unicorn_bin worker[0] -D -c /apps/app7/current/config/unicorn.rb -E production
sysadmin 9639 0.5 0.4 372832 56308 ? Sl 23:23 0:06 unicorn_bin worker[1] -D -c /apps/app16/current/config/unicorn.rb -E production
sysadmin 10217 0.5 0.4 437220 57072 ? Dl 23:29 0:05 unicorn_bin worker[0] -D -c /apps/app21/current/config/unicorn.rb -E production
sysadmin 10442 0.6 0.4 371012 59192 ? Sl 23:31 0:06 unicorn_bin worker[0] -D -c /apps/app9/current/config/unicorn.rb -E production
sysadmin 10521 0.7 0.5 371792 63568 ? Sl 23:32 0:06 unicorn_bin worker[0] -D -c /apps/app19/current/config/unicorn.rb -E production
sysadmin 10674 0.7 0.5 356400 65540 ? Sl 23:34 0:05 unicorn_bin worker[0] -D -c /apps/app16/current/config/unicorn.rb -E production
sysadmin 11102 1.1 0.5 432420 65560 ? Sl 23:38 0:05 unicorn_bin worker[0] -D -c /apps/app20/current/config/unicorn.rb -E production
sysadmin 10611 0.9 0.5 372992 67856 ? Sl 23:33 0:06 unicorn_bin worker[0] -D -c /apps/app13/current/config/unicorn.rb -E production
sysadmin 11089 1.1 0.5 365200 70340 ? Sl 23:37 0:05 unicorn_bin worker[0] -D -c /apps/app17/current/config/unicorn.rb -E production
sysadmin 11427 1.0 0.5 295068 72612 ? Dl 23:42 0:02 unicorn_bin worker[0] -D -c /apps/app32/current/config/unicorn.rb -E production
sysadmin 11288 0.8 0.5 298980 73360 ? Dl 23:40 0:02 unicorn_bin worker[0] -D -c /apps/app10/current/config/unicorn.rb -E production
sysadmin 11170 1.1 0.6 363160 74384 ? Sl 23:38 0:05 unicorn_bin worker[0] -D -c /apps/app11/current/config/unicorn.rb -E production
sysadmin 10588 0.9 0.6 493632 75100 ? Dl 23:33 0:07 unicorn_bin worker[0] -D -c /apps/app14/current/config/unicorn.rb -E production
sysadmin 11106 1.2 0.6 364900 81756 ? Sl 23:38 0:05 unicorn_bin worker[0] -D -c /apps/app12/current/config/unicorn.rb -E production
sysadmin 11178 1.2 0.7 361828 88628 ? Sl 23:38 0:05 unicorn_bin worker[0] -D -c /apps/app23/current/config/unicorn.rb -E production
sysadmin 11409 1.4 0.7 349048 88840 ? Dl 23:41 0:03 unicorn_bin worker[0] -D -c /apps/app29/current/config/unicorn.rb -E production
sysadmin 11400 1.7 0.7 355636 93176 ? Dl 23:41 0:04 unicorn_bin worker[0] -D -c /apps/app25/current/config/unicorn.rb -E production
sysadmin 11175 0.8 0.7 322132 95716 ? Dl 23:38 0:03 unicorn_bin worker[0] -D -c /apps/app6/current/config/unicorn.rb -E production
sysadmin 11252 0.9 0.7 322268 95880 ? Dl 23:39 0:03 unicorn_bin worker[0] -D -c /apps/app22/current/config/unicorn.rb -E production
sysadmin 11169 1.2 0.7 362224 97484 ? Sl 23:38 0:05 unicorn_bin worker[0] -D -c /apps/app15/current/config/unicorn.rb -E production
sysadmin 10993 1.3 0.8 386516 103752 ? Sl 23:37 0:07 unicorn_bin worker[1] -D -c /apps/app25/current/config/unicorn.rb -E production
sysadmin 11047 0.8 0.8 354136 106652 ? Dl 23:37 0:04 unicorn_bin worker[0] -D -c /apps/app18/current/config/unicorn.rb -E production
sysadmin 10555 1.2 0.8 469496 109208 ? Sl 23:32 0:10 unicorn_bin worker[0] -D -c /apps/app5/current/config/unicorn.rb -E production
sysadmin 11282 1.8 0.9 380480 114992 ? Sl 23:40 0:06 unicorn_bin worker[0] -D -c /apps/app8/current/config/unicorn.rb -E production
sysadmin 11210 1.0 0.9 343824 117296 ? Dl 23:39 0:04 unicorn_bin worker[0] -D -c /apps/app4/current/config/unicorn.rb -E production
sysadmin 10897 1.6 1.0 467556 123780 ? Sl 23:36 0:10 unicorn_bin worker[0] -D -c /apps/app3/current/config/unicorn.rb -E production
sysadmin 11027 1.9 1.1 470524 141648 ? Sl 23:37 0:10 unicorn_bin worker[0] -D -c /apps/app2/current/config/unicorn.rb -E production
sysadmin 10784 1.6 1.1 470824 145132 ? Sl 23:35 0:11 unicorn_bin worker[0] -D -c /apps/app24/current/config/unicorn.rb -E production