Scheduled Maintenance: We are aware of an issue with Google, AOL, and Yahoo services as email providers which are blocking new registrations. We are trying to fix the issue and we have several internal and external support tickets in process to resolve the issue. Please see: viewtopic.php?t=158230

 

 

 

kernel dynamic memory and used extremely high

Linux Kernel, Network, and Services configuration.
Post Reply
Message
Author
Gregor_Weertman
Posts: 6
Joined: 2022-08-15 08:47

kernel dynamic memory and used extremely high

#1 Post by Gregor_Weertman »

Total Memory: 27G
Used: 21G
Free: 2.0G
buff/cach 4.5G
available: 5.8G

Debian GNU/Linux 9.13

Why is the used so high?
It isn't slab, cache or buffer.
Running programs is just 3.5G

The "used" starts of with 600M but after a while it becomes very high and the system becomes slow due to lack of buff/cache.

I tried: smem,vmstat,/proc/meminfo,free,atop,iotop,sync; echo 3 > /proc/sys/vm/drop_caches,slaptop,ipcs,ls -l /run/shm/, ps, and also lsmod |sort -nk2.
No luck.
I tried: echo 16000000 > /proc/sys/vm/min_free_kbytes but there seems no way to limit the kernel memory without effecting also the buff/cach.

I tried "stress-ng -vm 10 --vm-bytes 4G -t 20" without swap and that reduced the Used: 21G to Used: 15G after running it a few times.

I see some correlation between "used" and "kernel dynamic memory". (another time and moment than above)
# free -kw;smem -w
total used free shared buffers cache available
Mem: 28802056 25450872 185424 49760 311368 2854392 2863192
Swap: 16777212 2620004 14157208
Area Used Cache Noncache
firmware/hardware 0 0 0
kernel image 0 0 0
kernel dynamic memory 26813324 3048136 23765188
userspace memory 1777324 85952 1691372
free memory 211408 211408 0

Where it the memory?

flashydolphin8827
Posts: 18
Joined: 2022-08-15 23:15
Has thanked: 2 times

Re: kernel dynamic memory and used extremely high

#2 Post by flashydolphin8827 »

Couldyou possibly be running to many virtual machine or other prosser and memory intensive applications or processes, like compiling a use case specific kernel?

Gregor_Weertman
Posts: 6
Joined: 2022-08-15 08:47

Re: kernel dynamic memory and used extremely high

#3 Post by Gregor_Weertman »

No almost never more then 3.5G on processes running.

User avatar
Head_on_a_Stick
Posts: 14114
Joined: 2014-06-01 17:46
Location: London, England
Has thanked: 81 times
Been thanked: 132 times

Re: kernel dynamic memory and used extremely high

#4 Post by Head_on_a_Stick »

You could try https://www.kernel.org/doc/html/v5.10/d ... mleak.html but

Code: Select all

$ grep KMEMLEAK /boot/config-5.10.0-16-amd64  
CONFIG_HAVE_DEBUG_KMEMLEAK=y
# CONFIG_DEBUG_KMEMLEAK is not set
$
^ looks like you may have to recompile with CONFIG_DEBUG_KMEMLEAK enabled — /sys/kernel/debug/ is needed.
deadbang

Gregor_Weertman
Posts: 6
Joined: 2022-08-15 08:47

Re: kernel dynamic memory and used extremely high

#5 Post by Gregor_Weertman »

Thanks,

I don't seem to be able to find out how to compile the LEAK into 4.9.0-18-amd64

Do you know how?

User avatar
Head_on_a_Stick
Posts: 14114
Joined: 2014-06-01 17:46
Location: London, England
Has thanked: 81 times
Been thanked: 132 times

Re: kernel dynamic memory and used extremely high

#6 Post by Head_on_a_Stick »

deadbang

Aki
Global Moderator
Global Moderator
Posts: 2823
Joined: 2014-07-20 18:12
Location: Europe
Has thanked: 69 times
Been thanked: 385 times

Re: kernel dynamic memory and used extremely high

#7 Post by Aki »

Hello,

A memory leak in the kernel could be a possible explanation, but perhaps it could be useful to exclude other possible causes before digging in kernel's debugging functions.

For example, due to the great quantity of RAM of your computer, is your installation configured to take advantage of one or more ram disks (for example with zram), to optimize the directory for temporary files, swap or specific programs ?

I made same tests and I was able to replicate your issue using a zram disk; for example:

Code: Select all

root@debian-aki:~# smem -w
Area                           Used      Cache   Noncache 
firmware/hardware                 0          0          0 
kernel image                      0          0          0 
kernel dynamic memory        892872     555640     337232 
userspace memory            1672040     346760    1325280 
free memory                  489720     489720          0

root@debian-aki:~# modinfo zram
root@debian-aki:~# modprobe zram
root@debian-aki:~# zramctl --size 1024M /dev/zram0
root@debian-aki:~# mkfs.ext2  /dev/zram0

root@debian-aki:~# smem -w
Area                           Used      Cache   Noncache 
firmware/hardware                 0          0          0 
kernel image                      0          0          0 
kernel dynamic memory        911472     572856     338616 
userspace memory            1736480     372024    1364456 
free memory                  406680     406680          0

root@debian-aki:~# mount /dev/zram0 /mnt/
root@debian-aki:~# cp -r /usr/* /mnt/

root@debian-aki:~# df -h /mnt/
File system     Dim. Usati Dispon. Uso% Montato su
/dev/zram0      959M  959M       0 100% /mnt

root@debian-aki:~# smem -w
Area                           Used      Cache   Noncache 
firmware/hardware                 0          0          0 
kernel image                      0          0          0 
kernel dynamic memory       1519924     204736    1315188 
userspace memory            1444380     212408    1231972 
free memory                   90328      90328          0

root@debian-aki:~# umount /mnt 
root@debian-aki:~# zramctl -r /dev/zram0

root@debian-aki:~# smem -w
Area                           Used      Cache   Noncache 
firmware/hardware                 0          0          0 
kernel image                      0          0          0 
kernel dynamic memory        512632     201112     311520 
userspace memory            1400312     214580    1185732 
free memory                 1141688    1141688          0 
⢀⣴⠾⠻⢶⣦⠀
⣾⠁⢠⠒⠀⣿⡁ Debian - The universal operating system
⢿⡄⠘⠷⠚⠋⠀ https://www.debian.org
⠈⠳⣄⠀

Gregor_Weertman
Posts: 6
Joined: 2022-08-15 08:47

Re: kernel dynamic memory and used extremely high

#8 Post by Gregor_Weertman »

Code: Select all

# cat  /sys/block/zram0/mm_stat
       0        0        0        0        0        0        0
Seems to be no zram also.

I "ballooned" the 25G used to 14G with swap disabled.
It is memory that is somehow freed by allocating lots of memory as with cache and buffers.

Code: Select all

              total        used        free      shared  buff/cache   available
Mem:            27G         14G        2.2G        331M         11G         12G
Swap:            0B          0B          0B
Maybe the 14G is reserved for this java processs.

Code: Select all

   PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
108511 root      20   0 16.571g 1.279g   7820 S   0.3  4.7   2:00.61 java

Aki
Global Moderator
Global Moderator
Posts: 2823
Joined: 2014-07-20 18:12
Location: Europe
Has thanked: 69 times
Been thanked: 385 times

Re: kernel dynamic memory and used extremely high

#9 Post by Aki »

Hello,
It would be useful to see the output of:

Code: Select all

smem --system
in respect to the previous output (after swap deactivation). Please report the output of free command with it.
⢀⣴⠾⠻⢶⣦⠀
⣾⠁⢠⠒⠀⣿⡁ Debian - The universal operating system
⢿⡄⠘⠷⠚⠋⠀ https://www.debian.org
⠈⠳⣄⠀

Gregor_Weertman
Posts: 6
Joined: 2022-08-15 08:47

Re: kernel dynamic memory and used extremely high

#10 Post by Gregor_Weertman »

Code: Select all

root@vrzdpsv01:/root # smem --system
Area                           Used      Cache   Noncache
firmware/hardware                 0          0          0
kernel image                      0          0          0
kernel dynamic memory      21698552   10241732   11456820
userspace memory            4957516      96052    4861464
free memory                 2145988    2145988          0
root@vrzdpsv01:/root # free -h
              total        used        free      shared  buff/cache   available
Mem:            27G         15G        2.1G        314M        9.9G         11G
Swap:            0B          0B          0B
root@vrzdpsv01:/root #

These are the processes killed by the oom killer when i ran the stress-ng.

Code: Select all

COMMAND          PID     %MEM  DRS       %MEM  RSS      TRS   VSZ       START
jbd2/dm-3-8      397     0.0   0         0.0   0        0     0         Aug    13
vmtoolsd         426     0.0   133859    0.0   4320     40    133900    Aug    13
systemd-journal  435     0.0   62684     0.0   5700     107   62792     Aug    13
mdatp_audisp_pl  757     0.2   1457199   0.2   74448    1176  1458376   Aug    13
xdbcsrvr         861     0.0   10508     0.0   1072     19    10528     Aug    13
inetd            875     0.0   36161     0.0   2192     30    36192     Aug    13
atopacctd        902     0.0   4156      0.0   1224     15    4172      Aug    13
acuxdbc04        66908   0.0   31302     0.0   5436     37    31340     15:23
acuxdbc04        66910   0.0   30850     0.0   4924     37    30888     15:23
java             108511  8.4   17379741  8.4   2447660  2     17379744  03:00

Aki
Global Moderator
Global Moderator
Posts: 2823
Joined: 2014-07-20 18:12
Location: Europe
Has thanked: 69 times
Been thanked: 385 times

Re: kernel dynamic memory and used extremely high

#11 Post by Aki »

Hello,
Sorry for that: can you send the output of these commands together:

Code: Select all

smem --system
smem --users
free
cat /proc/meminfo
df –k /dev/shm
⢀⣴⠾⠻⢶⣦⠀
⣾⠁⢠⠒⠀⣿⡁ Debian - The universal operating system
⢿⡄⠘⠷⠚⠋⠀ https://www.debian.org
⠈⠳⣄⠀

Gregor_Weertman
Posts: 6
Joined: 2022-08-15 08:47

Re: kernel dynamic memory and used extremely high

#12 Post by Gregor_Weertman »

The used is "only" 13G now sinds I ballooned it with stress-ng but still too high.

Code: Select all

smem --system
Area                           Used      Cache   Noncache
firmware/hardware                 0          0          0
kernel image                      0          0          0
kernel dynamic memory      23198232   12203116   10995116
userspace memory            3249744     173248    3076496
free memory                 2354080    2354080          0


smem --users
User     Count     Swap      USS      PSS      RSS
daemon       1        0      220      221     1812
xxxxxex      2        0      208      258     3284
xxxxx30      1        0      604      768    22128
xxxxx34      1        0      604      770    22256
xxxxx42      1        0      608      774    22292
xxxxx07      1        0      604      775    22316
xxxxx51      1        0      600      781    22664
xxxxx44      1        0      608      782    22560
xxxxx28      1        0      596      784    22424
xxxxx666$     1        0      600      784    19156
xxxxx27      1        0      604      790    22816
xxxxx83      1        0      604      814    23760
xxxxxr3      1        0      564      814    21272
xxxxx78      1        0      608      825    23552
xxxxx82      1        0      604      827    24248
xxxxxn-exim     1        0      828      829     2388
xxxxx81      1        0      608      843    24340
xxxxx21      1        0      684      874    23232
xxxxx61      1        0      664      919    25196
xxxxx79      1        0      608      919    26872
xxxxx71      1        0      612      933    26980
xxxxx23      1        0      676      938    25640
xxxxx65      1        0      608      940    27572
xxxxx84      1        0      608      946    27668
xxxxx72      1        0      608      949    27768
xxxxx11      1        0      668      954    26396
xxxxx47      1        0      680      962    26204
xxxxx69      1        0      592      986    26296
xxxxx67      1        0      604      991    29492
xxxxx80      1        0      608      996    29284
xxxxx76      1        0      624      997    28084
xxxxx17      1        0      652      999    28164
xxxxx75      1        0      612     1006    29472
xxxxx73      1        0      612     1007    29524
messagebus     1        0      916     1020     2976
xxxxx77      1        0      608     1022    30096
xxxxx25      1        0      672     1023    28668
xxxxx66      1        0      612     1023    30144
xxxxx20      1        0      688     1027    28152
xxxxx70      1        0      612     1032    30456
xxxxx86      1        0      608     1043    30384
xxxxx68      1        0      608     1045    31020
ntp          1        0     1016     1048     2856
xxxxx12      1        0      688     1049    29108
xxxxx74      1        0      612     1052    30828
xxxxx85      1        0      608     1052    30676
xxxxx35      1        0      676     1079    30208
xxxxx08      1        0      684     1086    30152
xxxxx50      1        0      668     1098    31144
xxxxx46      1        0      684     1118    30880
xxxxx14      1        0      668     1126    31832
xxxxx56      1        0      684     1129    31508
xxxxx58      1        0      668     1140    32240
xxxxx31      1        0      716     1182    32436
xxxxx36      1        0      704     1184    32588
xxxxx26      1        0      692     1191    33016
xxxxx10      1        0      684     1205    34180
xxxxx54      1        0      692     1209    33848
xxxxx45      1        0      712     1222    33880
xxxxx43      1        0      740     1302    35520
lp           1        0      888     1303     5920
xxxxx37      1        0      752     1353    36104
xxxxxlia05$     1        0      776     1540    36576
xxxxx41      1        0     1060     1857    41984
xxxxx57      1        0     1080     1865    41820
xxxxx22      1        0     1088     1875    38528
xxxxx64      1        0     1088     1880    41824
xxxxx38      1        0     1084     1881    42004
xxxxx40      1        0     1088     1887    42016
nobody       1        0     2956     3809    33612
xxxxxsupport     4        0     2868     5207   127356
xxxxx417     4        0     6500     6665    22736
xxxxx121     6        0     6824     7524    31336
xxxxx721     6        0     6972     7679    31228
xxxxx708     6        0     7032     7696    31520
xxxxx222     6        0     7104     7747    30972
xxxxx717     6        0     7152     7811    31496
xxxxx617     6        0     7252     7886    31104
xxxxx521     6        0     7196     7902    31648
xxxxx119     6        0     7212     7921    31788
xxxxx821     6        0     7284     7961    31440
xxxxx816     6        0     7360     8000    31748
xxxxx312     6        0     7412     8054    31548
Debian-snmp     1        0     8072     8083    10140
xxxxx420     6        0     7508     8192    31536
xxxxx422     6        0     7608     8263    31680
xxxxx921     6        0     7692     8332    31840
xxxxx319     6        0     7748     8408    31604
xxxxx510     6        0     9856    10509    34288
xxxxx107    10        0    11160    12396    51180
xxxxx911    10        0    11196    12453    49732
xxxxx816    10        0    11348    12539    50948
xxxxx121    10        0    11432    12674    50980
xxxxx602    10        0    11496    12735    50912
xxxxx212    10        0    11612    12805    51840
xxxxx418    10        0    11592    12852    51668
xxxxx221    10        0    11656    12866    52316
xxxxx319    10        0    11708    12941    52008
xxxxx019    10        0    11916    13127    51124
xxxxx318    10        0    11932    13189    51308
xxxxx118    11        0    12168    13320    54868
xxxxx921    10        0    12232    13484    51532
xxxxx122     6        0    16044    16738    40668
xxxxx602    14        0    16088    17827    70832
xxxxx712    14        0    16240    18052    71488
xxxxx421    14        0    16468    18226    72396
xxxxx511    18        0    20844    23167    92716
xxxxx818    18        0    21680    23996    93996
xxxxx018    18        0    21732    24066    94024
xxxxx920    18        0    21752    24079    94184
xxxxx419    18        0    21820    24141    92588
xxxxx122    18        0    21804    24143    94212
xxxxx218    18        0    21852    24191    93944
xxxxx617    18        0    21916    24229    93720
xxxxx522    18        0    21920    24239    92696
xxxxx717    18        0    21924    24253    93392
xxxxx016    18        0    21960    24273    94324
xxxxx421    18        0    21948    24279    93680
xxxxx017    18        0    22040    24351    93320
xxxxx119    18        0    22068    24384    93668
xxxxx916    18        0    22200    24488    93672
xxxxx622    18        0    22180    24514    94080
xxxxx921    18        0    22244    24564    93848
xxxxx422    18        0    22276    24574    94568
xxxxx018    18        0    22288    24632    94192
xxxxx121    19        0    22396    24665    97644
xxxxx209    22        0    24760    27652   111108
xxxxx212    10        0    34048    35266    73492
xxxxx204    13        0    63868    65615   114092
xxxxx108    58        0    70728    78558   299764
mdatp        1        0   387700   389381   394088
root       389        0  1547664  1695724  3727488


free
              total        used        free      shared  buff/cache   available
Mem:       28802056    14066236     2356156      319168    12379664    13978848
Swap:             0           0           0


cat /proc/meminfo
MemTotal:       28802056 kB
MemFree:         2356016 kB
MemAvailable:   13978708 kB
Buffers:         1337556 kB
Cached:          7996436 kB
SwapCached:            0 kB
Active:          9778472 kB
Inactive:        2589512 kB
Active(anon):    3140508 kB
Inactive(anon):   216244 kB
Active(file):    6637964 kB
Inactive(file):  2373268 kB
Unevictable:       38224 kB
Mlocked:           38224 kB
SwapTotal:             0 kB
SwapFree:              0 kB
Dirty:             14928 kB
Writeback:             0 kB
AnonPages:       3072340 kB
Mapped:           171220 kB
Shmem:            319168 kB
Slab:            3825008 kB
SReclaimable:    3045672 kB
SUnreclaim:       779336 kB
KernelStack:       29360 kB
PageTables:       221060 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    14401028 kB
Committed_AS:    8763932 kB
VmallocTotal:   34359738367 kB
VmallocUsed:           0 kB
VmallocChunk:          0 kB
HardwareCorrupted:     0 kB
AnonHugePages:         0 kB
ShmemHugePages:        0 kB
ShmemPmdMapped:        0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:    22478656 kB
DirectMap2M:     6881280 kB
DirectMap1G:     2097152 kB


df --block-size=1K /dev/shm
Filesystem     1K-blocks  Used Available Use% Mounted on
tmpfs           14401028     0  14401028   0% /dev/shm

Post Reply