Scheduled Maintenance: We are aware of an issue with Google, AOL, and Yahoo services as email providers which are blocking new registrations. We are trying to fix the issue and we have several internal and external support tickets in process to resolve the issue. Please see: viewtopic.php?t=158230
kernel dynamic memory and used extremely high
-
- Posts: 6
- Joined: 2022-08-15 08:47
kernel dynamic memory and used extremely high
Total Memory: 27G
Used: 21G
Free: 2.0G
buff/cach 4.5G
available: 5.8G
Debian GNU/Linux 9.13
Why is the used so high?
It isn't slab, cache or buffer.
Running programs is just 3.5G
The "used" starts of with 600M but after a while it becomes very high and the system becomes slow due to lack of buff/cache.
I tried: smem,vmstat,/proc/meminfo,free,atop,iotop,sync; echo 3 > /proc/sys/vm/drop_caches,slaptop,ipcs,ls -l /run/shm/, ps, and also lsmod |sort -nk2.
No luck.
I tried: echo 16000000 > /proc/sys/vm/min_free_kbytes but there seems no way to limit the kernel memory without effecting also the buff/cach.
I tried "stress-ng -vm 10 --vm-bytes 4G -t 20" without swap and that reduced the Used: 21G to Used: 15G after running it a few times.
I see some correlation between "used" and "kernel dynamic memory". (another time and moment than above)
# free -kw;smem -w
total used free shared buffers cache available
Mem: 28802056 25450872 185424 49760 311368 2854392 2863192
Swap: 16777212 2620004 14157208
Area Used Cache Noncache
firmware/hardware 0 0 0
kernel image 0 0 0
kernel dynamic memory 26813324 3048136 23765188
userspace memory 1777324 85952 1691372
free memory 211408 211408 0
Where it the memory?
Used: 21G
Free: 2.0G
buff/cach 4.5G
available: 5.8G
Debian GNU/Linux 9.13
Why is the used so high?
It isn't slab, cache or buffer.
Running programs is just 3.5G
The "used" starts of with 600M but after a while it becomes very high and the system becomes slow due to lack of buff/cache.
I tried: smem,vmstat,/proc/meminfo,free,atop,iotop,sync; echo 3 > /proc/sys/vm/drop_caches,slaptop,ipcs,ls -l /run/shm/, ps, and also lsmod |sort -nk2.
No luck.
I tried: echo 16000000 > /proc/sys/vm/min_free_kbytes but there seems no way to limit the kernel memory without effecting also the buff/cach.
I tried "stress-ng -vm 10 --vm-bytes 4G -t 20" without swap and that reduced the Used: 21G to Used: 15G after running it a few times.
I see some correlation between "used" and "kernel dynamic memory". (another time and moment than above)
# free -kw;smem -w
total used free shared buffers cache available
Mem: 28802056 25450872 185424 49760 311368 2854392 2863192
Swap: 16777212 2620004 14157208
Area Used Cache Noncache
firmware/hardware 0 0 0
kernel image 0 0 0
kernel dynamic memory 26813324 3048136 23765188
userspace memory 1777324 85952 1691372
free memory 211408 211408 0
Where it the memory?
-
- Posts: 18
- Joined: 2022-08-15 23:15
- Has thanked: 2 times
Re: kernel dynamic memory and used extremely high
Couldyou possibly be running to many virtual machine or other prosser and memory intensive applications or processes, like compiling a use case specific kernel?
-
- Posts: 6
- Joined: 2022-08-15 08:47
Re: kernel dynamic memory and used extremely high
No almost never more then 3.5G on processes running.
- Head_on_a_Stick
- Posts: 14114
- Joined: 2014-06-01 17:46
- Location: London, England
- Has thanked: 81 times
- Been thanked: 132 times
Re: kernel dynamic memory and used extremely high
You could try https://www.kernel.org/doc/html/v5.10/d ... mleak.html but
^ looks like you may have to recompile with CONFIG_DEBUG_KMEMLEAK enabled — /sys/kernel/debug/ is needed.
Code: Select all
$ grep KMEMLEAK /boot/config-5.10.0-16-amd64
CONFIG_HAVE_DEBUG_KMEMLEAK=y
# CONFIG_DEBUG_KMEMLEAK is not set
$
deadbang
-
- Posts: 6
- Joined: 2022-08-15 08:47
Re: kernel dynamic memory and used extremely high
Thanks,
I don't seem to be able to find out how to compile the LEAK into 4.9.0-18-amd64
Do you know how?
I don't seem to be able to find out how to compile the LEAK into 4.9.0-18-amd64
Do you know how?
- Head_on_a_Stick
- Posts: 14114
- Joined: 2014-06-01 17:46
- Location: London, England
- Has thanked: 81 times
- Been thanked: 132 times
-
- Global Moderator
- Posts: 2823
- Joined: 2014-07-20 18:12
- Location: Europe
- Has thanked: 69 times
- Been thanked: 385 times
Re: kernel dynamic memory and used extremely high
Hello,
A memory leak in the kernel could be a possible explanation, but perhaps it could be useful to exclude other possible causes before digging in kernel's debugging functions.
For example, due to the great quantity of RAM of your computer, is your installation configured to take advantage of one or more ram disks (for example with zram), to optimize the directory for temporary files, swap or specific programs ?
I made same tests and I was able to replicate your issue using a zram disk; for example:
A memory leak in the kernel could be a possible explanation, but perhaps it could be useful to exclude other possible causes before digging in kernel's debugging functions.
For example, due to the great quantity of RAM of your computer, is your installation configured to take advantage of one or more ram disks (for example with zram), to optimize the directory for temporary files, swap or specific programs ?
I made same tests and I was able to replicate your issue using a zram disk; for example:
Code: Select all
root@debian-aki:~# smem -w
Area Used Cache Noncache
firmware/hardware 0 0 0
kernel image 0 0 0
kernel dynamic memory 892872 555640 337232
userspace memory 1672040 346760 1325280
free memory 489720 489720 0
root@debian-aki:~# modinfo zram
root@debian-aki:~# modprobe zram
root@debian-aki:~# zramctl --size 1024M /dev/zram0
root@debian-aki:~# mkfs.ext2 /dev/zram0
root@debian-aki:~# smem -w
Area Used Cache Noncache
firmware/hardware 0 0 0
kernel image 0 0 0
kernel dynamic memory 911472 572856 338616
userspace memory 1736480 372024 1364456
free memory 406680 406680 0
root@debian-aki:~# mount /dev/zram0 /mnt/
root@debian-aki:~# cp -r /usr/* /mnt/
root@debian-aki:~# df -h /mnt/
File system Dim. Usati Dispon. Uso% Montato su
/dev/zram0 959M 959M 0 100% /mnt
root@debian-aki:~# smem -w
Area Used Cache Noncache
firmware/hardware 0 0 0
kernel image 0 0 0
kernel dynamic memory 1519924 204736 1315188
userspace memory 1444380 212408 1231972
free memory 90328 90328 0
root@debian-aki:~# umount /mnt
root@debian-aki:~# zramctl -r /dev/zram0
root@debian-aki:~# smem -w
Area Used Cache Noncache
firmware/hardware 0 0 0
kernel image 0 0 0
kernel dynamic memory 512632 201112 311520
userspace memory 1400312 214580 1185732
free memory 1141688 1141688 0
-
- Posts: 6
- Joined: 2022-08-15 08:47
Re: kernel dynamic memory and used extremely high
Code: Select all
# cat /sys/block/zram0/mm_stat
0 0 0 0 0 0 0
I "ballooned" the 25G used to 14G with swap disabled.
It is memory that is somehow freed by allocating lots of memory as with cache and buffers.
Code: Select all
total used free shared buff/cache available
Mem: 27G 14G 2.2G 331M 11G 12G
Swap: 0B 0B 0B
Code: Select all
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
108511 root 20 0 16.571g 1.279g 7820 S 0.3 4.7 2:00.61 java
-
- Global Moderator
- Posts: 2823
- Joined: 2014-07-20 18:12
- Location: Europe
- Has thanked: 69 times
- Been thanked: 385 times
Re: kernel dynamic memory and used extremely high
Hello,
It would be useful to see the output of:
in respect to the previous output (after swap deactivation). Please report the output of free command with it.
It would be useful to see the output of:
Code: Select all
smem --system
-
- Posts: 6
- Joined: 2022-08-15 08:47
Re: kernel dynamic memory and used extremely high
Code: Select all
root@vrzdpsv01:/root # smem --system
Area Used Cache Noncache
firmware/hardware 0 0 0
kernel image 0 0 0
kernel dynamic memory 21698552 10241732 11456820
userspace memory 4957516 96052 4861464
free memory 2145988 2145988 0
root@vrzdpsv01:/root # free -h
total used free shared buff/cache available
Mem: 27G 15G 2.1G 314M 9.9G 11G
Swap: 0B 0B 0B
root@vrzdpsv01:/root #
These are the processes killed by the oom killer when i ran the stress-ng.
Code: Select all
COMMAND PID %MEM DRS %MEM RSS TRS VSZ START
jbd2/dm-3-8 397 0.0 0 0.0 0 0 0 Aug 13
vmtoolsd 426 0.0 133859 0.0 4320 40 133900 Aug 13
systemd-journal 435 0.0 62684 0.0 5700 107 62792 Aug 13
mdatp_audisp_pl 757 0.2 1457199 0.2 74448 1176 1458376 Aug 13
xdbcsrvr 861 0.0 10508 0.0 1072 19 10528 Aug 13
inetd 875 0.0 36161 0.0 2192 30 36192 Aug 13
atopacctd 902 0.0 4156 0.0 1224 15 4172 Aug 13
acuxdbc04 66908 0.0 31302 0.0 5436 37 31340 15:23
acuxdbc04 66910 0.0 30850 0.0 4924 37 30888 15:23
java 108511 8.4 17379741 8.4 2447660 2 17379744 03:00
-
- Global Moderator
- Posts: 2823
- Joined: 2014-07-20 18:12
- Location: Europe
- Has thanked: 69 times
- Been thanked: 385 times
Re: kernel dynamic memory and used extremely high
Hello,
Sorry for that: can you send the output of these commands together:
Sorry for that: can you send the output of these commands together:
Code: Select all
smem --system
smem --users
free
cat /proc/meminfo
df –k /dev/shm
-
- Posts: 6
- Joined: 2022-08-15 08:47
Re: kernel dynamic memory and used extremely high
The used is "only" 13G now sinds I ballooned it with stress-ng but still too high.
Code: Select all
smem --system
Area Used Cache Noncache
firmware/hardware 0 0 0
kernel image 0 0 0
kernel dynamic memory 23198232 12203116 10995116
userspace memory 3249744 173248 3076496
free memory 2354080 2354080 0
smem --users
User Count Swap USS PSS RSS
daemon 1 0 220 221 1812
xxxxxex 2 0 208 258 3284
xxxxx30 1 0 604 768 22128
xxxxx34 1 0 604 770 22256
xxxxx42 1 0 608 774 22292
xxxxx07 1 0 604 775 22316
xxxxx51 1 0 600 781 22664
xxxxx44 1 0 608 782 22560
xxxxx28 1 0 596 784 22424
xxxxx666$ 1 0 600 784 19156
xxxxx27 1 0 604 790 22816
xxxxx83 1 0 604 814 23760
xxxxxr3 1 0 564 814 21272
xxxxx78 1 0 608 825 23552
xxxxx82 1 0 604 827 24248
xxxxxn-exim 1 0 828 829 2388
xxxxx81 1 0 608 843 24340
xxxxx21 1 0 684 874 23232
xxxxx61 1 0 664 919 25196
xxxxx79 1 0 608 919 26872
xxxxx71 1 0 612 933 26980
xxxxx23 1 0 676 938 25640
xxxxx65 1 0 608 940 27572
xxxxx84 1 0 608 946 27668
xxxxx72 1 0 608 949 27768
xxxxx11 1 0 668 954 26396
xxxxx47 1 0 680 962 26204
xxxxx69 1 0 592 986 26296
xxxxx67 1 0 604 991 29492
xxxxx80 1 0 608 996 29284
xxxxx76 1 0 624 997 28084
xxxxx17 1 0 652 999 28164
xxxxx75 1 0 612 1006 29472
xxxxx73 1 0 612 1007 29524
messagebus 1 0 916 1020 2976
xxxxx77 1 0 608 1022 30096
xxxxx25 1 0 672 1023 28668
xxxxx66 1 0 612 1023 30144
xxxxx20 1 0 688 1027 28152
xxxxx70 1 0 612 1032 30456
xxxxx86 1 0 608 1043 30384
xxxxx68 1 0 608 1045 31020
ntp 1 0 1016 1048 2856
xxxxx12 1 0 688 1049 29108
xxxxx74 1 0 612 1052 30828
xxxxx85 1 0 608 1052 30676
xxxxx35 1 0 676 1079 30208
xxxxx08 1 0 684 1086 30152
xxxxx50 1 0 668 1098 31144
xxxxx46 1 0 684 1118 30880
xxxxx14 1 0 668 1126 31832
xxxxx56 1 0 684 1129 31508
xxxxx58 1 0 668 1140 32240
xxxxx31 1 0 716 1182 32436
xxxxx36 1 0 704 1184 32588
xxxxx26 1 0 692 1191 33016
xxxxx10 1 0 684 1205 34180
xxxxx54 1 0 692 1209 33848
xxxxx45 1 0 712 1222 33880
xxxxx43 1 0 740 1302 35520
lp 1 0 888 1303 5920
xxxxx37 1 0 752 1353 36104
xxxxxlia05$ 1 0 776 1540 36576
xxxxx41 1 0 1060 1857 41984
xxxxx57 1 0 1080 1865 41820
xxxxx22 1 0 1088 1875 38528
xxxxx64 1 0 1088 1880 41824
xxxxx38 1 0 1084 1881 42004
xxxxx40 1 0 1088 1887 42016
nobody 1 0 2956 3809 33612
xxxxxsupport 4 0 2868 5207 127356
xxxxx417 4 0 6500 6665 22736
xxxxx121 6 0 6824 7524 31336
xxxxx721 6 0 6972 7679 31228
xxxxx708 6 0 7032 7696 31520
xxxxx222 6 0 7104 7747 30972
xxxxx717 6 0 7152 7811 31496
xxxxx617 6 0 7252 7886 31104
xxxxx521 6 0 7196 7902 31648
xxxxx119 6 0 7212 7921 31788
xxxxx821 6 0 7284 7961 31440
xxxxx816 6 0 7360 8000 31748
xxxxx312 6 0 7412 8054 31548
Debian-snmp 1 0 8072 8083 10140
xxxxx420 6 0 7508 8192 31536
xxxxx422 6 0 7608 8263 31680
xxxxx921 6 0 7692 8332 31840
xxxxx319 6 0 7748 8408 31604
xxxxx510 6 0 9856 10509 34288
xxxxx107 10 0 11160 12396 51180
xxxxx911 10 0 11196 12453 49732
xxxxx816 10 0 11348 12539 50948
xxxxx121 10 0 11432 12674 50980
xxxxx602 10 0 11496 12735 50912
xxxxx212 10 0 11612 12805 51840
xxxxx418 10 0 11592 12852 51668
xxxxx221 10 0 11656 12866 52316
xxxxx319 10 0 11708 12941 52008
xxxxx019 10 0 11916 13127 51124
xxxxx318 10 0 11932 13189 51308
xxxxx118 11 0 12168 13320 54868
xxxxx921 10 0 12232 13484 51532
xxxxx122 6 0 16044 16738 40668
xxxxx602 14 0 16088 17827 70832
xxxxx712 14 0 16240 18052 71488
xxxxx421 14 0 16468 18226 72396
xxxxx511 18 0 20844 23167 92716
xxxxx818 18 0 21680 23996 93996
xxxxx018 18 0 21732 24066 94024
xxxxx920 18 0 21752 24079 94184
xxxxx419 18 0 21820 24141 92588
xxxxx122 18 0 21804 24143 94212
xxxxx218 18 0 21852 24191 93944
xxxxx617 18 0 21916 24229 93720
xxxxx522 18 0 21920 24239 92696
xxxxx717 18 0 21924 24253 93392
xxxxx016 18 0 21960 24273 94324
xxxxx421 18 0 21948 24279 93680
xxxxx017 18 0 22040 24351 93320
xxxxx119 18 0 22068 24384 93668
xxxxx916 18 0 22200 24488 93672
xxxxx622 18 0 22180 24514 94080
xxxxx921 18 0 22244 24564 93848
xxxxx422 18 0 22276 24574 94568
xxxxx018 18 0 22288 24632 94192
xxxxx121 19 0 22396 24665 97644
xxxxx209 22 0 24760 27652 111108
xxxxx212 10 0 34048 35266 73492
xxxxx204 13 0 63868 65615 114092
xxxxx108 58 0 70728 78558 299764
mdatp 1 0 387700 389381 394088
root 389 0 1547664 1695724 3727488
free
total used free shared buff/cache available
Mem: 28802056 14066236 2356156 319168 12379664 13978848
Swap: 0 0 0
cat /proc/meminfo
MemTotal: 28802056 kB
MemFree: 2356016 kB
MemAvailable: 13978708 kB
Buffers: 1337556 kB
Cached: 7996436 kB
SwapCached: 0 kB
Active: 9778472 kB
Inactive: 2589512 kB
Active(anon): 3140508 kB
Inactive(anon): 216244 kB
Active(file): 6637964 kB
Inactive(file): 2373268 kB
Unevictable: 38224 kB
Mlocked: 38224 kB
SwapTotal: 0 kB
SwapFree: 0 kB
Dirty: 14928 kB
Writeback: 0 kB
AnonPages: 3072340 kB
Mapped: 171220 kB
Shmem: 319168 kB
Slab: 3825008 kB
SReclaimable: 3045672 kB
SUnreclaim: 779336 kB
KernelStack: 29360 kB
PageTables: 221060 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 14401028 kB
Committed_AS: 8763932 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 0 kB
VmallocChunk: 0 kB
HardwareCorrupted: 0 kB
AnonHugePages: 0 kB
ShmemHugePages: 0 kB
ShmemPmdMapped: 0 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
DirectMap4k: 22478656 kB
DirectMap2M: 6881280 kB
DirectMap1G: 2097152 kB
df --block-size=1K /dev/shm
Filesystem 1K-blocks Used Available Use% Mounted on
tmpfs 14401028 0 14401028 0% /dev/shm