[Software] Why does swap never empty itself even after closing programs using it?

[Chaussettes]

I've used linux for many years now and have never found an easy to understand answer to this question. Say for example I open many programs at once and the system starts swapping things to disk, even after I close every program on the desktop, swap space is never emptied and continues growing until I reboot the machine. Can anyone who actually understands how this works explain why it does this and why it doesnt just clear everything when you close the program using it?



This is my desktop after a couple days of use and nothing open but the terminal. I have over 1 gig of swap usage and even after closing everything else, it wont empty the swap space. Is it lying to me and there isnt really anything there? Or is swap really being used (for nothing?) and what would happen if I max out? I am confident this is not a memory leak of any kind as I've encountered this for many years and across many different desktop environments, this is a residual memory thing related to swap that keeps creeping up. Thanks.

[Chaussettes]

I'll add that I understand what caching is but I dont understand why it is never cleared. If you continue caching without clearing you eventually max out swap and oom do you not? Even hours after I've killed a memory intensive program swap will never decrease it's use and return things to ram or drop the caches. it's so confusing.

[CwF]

Take a look at some stats
$ cat /proc/meminfo
There is some additional swap info that gives some food for thought, cached, dirty, etc.

Lots of parameters to learn about in /proc/sys/vm/. We 'cat' the files to check their values and can even echo in changes to the values if we get brave. For the most part the defaults are all where they need to be.

But overall swap does clear out eventually. I'm sure one of those parameters will effect how long that takes. Some types of data stay longer than others, but depending on the particular mix it does manage itself. With long uptimes I do see swap usage even with gobs of free memory and also see that return to zero. So the ascertion that all systems eventually fill swap is false.

The primary culprit in swap usage that may persist is from a web browser. My machines with gobs of memory and long uptimes don't have browsers. Another class of software that often 'leaks' are monitors that people leave running as eye candy. Monitors that keep histories of their own of cpu, memory, disk, whatever almost always leak long term.

Even hours after I've killed a memory intensive program swap will never decrease it's use and return things to ram or drop the caches

Hours isn't a long time. If something else request memory, it will be cleared for use. If nothing request it, then it stays. Run 'free' and notice how the columns add up to more than they should! Notice the 'available' column if larger than the 'free'. Available is how big the garage is, free is the floor area that doesn't need junk removed before you can use it, buff/cache is the junk.

[DebianFox]

Check the output of the following commands and change them accordingly

Code: Select all

$sysctl --dry-run vm.swappiness

The lower the value (5,7,10,15) more the Kernel will try to keep items in RAM. Higher the value (60,70, etc) more the kernel will try to swap the memory out. Change its value to less than 15. Do not make it zero or 1 or 2 or 3. 10 or 15 are decent enough for desktops and single mission servers.

Secondly you have 4GB RAM running xfce desktop environment. Try to see if you can increase the RAM size from 4GB to 8GB. Even that will help.

[jmgibson1981]

Linux at least as I understand it follows the idea of "unused memory is wasted memory". Many people will open and close the same application regularly. Why not just leave it cached for quicker use on subsequent opens? If something needs that space the kernel will dump it to make room but if it doesn't then it isn't hurting anything by staying there.

TLDR. Unless you are fighting serious hardware limitations don't aim for the lowest ram / swap usage you can get. While it can be bragging rights (in some circles) it is also wasted memory that could speed up your workflow if your software was still sitting in the active memory. There really is no downside to leaving that stuff there. The kernel knows what it is doing.

[CwF]

Higher the value (60,70, etc) more the kernel will try to swap the memory out. Change its value to less than 15.

Anymore I would leave it at default, I think 30? Lower values help when swap is slow. As device speed has increased the benefit is less. Since using zram-tools for swap this old wisdom may be even more obsolete and values higher than 30 may be beneficial, now more sensitive to cpu speed.

Edit: I checked, default is 60.

[DebianFox]

Default of 60? Why do Debian developers keep such a high value? Is it because Debian is typically used in servers and not in Desktops?

[CwF]

Default of 60? Why do Debian developers keep such a high value?

I'm not so sure 60 is 'high'. The scale goes to 200 IIRC. So do some testing!

I recently revised a vm backing to default, using zram and no other swap, and also increased tmpfs usage for all of ~/.cache. I'll have an opinion in a month or so...

[DebianFox]

Default of 60? Why do Debian developers keep such a high value?

I'm not so sure 60 is 'high'. The scale goes to 200 IIRC. So do some testing!

Great @CwF will await your feedback. The reason that I said 60 is high is because Linux kernel is supposed to be better at memory management then Windows. Unlike Windows it should not just start disk trashing unless there is a demand of RAM. If running a desktop environment then yes 60 is high. For other cases let us await your feedback.

I recently revised a vm backing to default, using zram and no other swap, and also increased tmpfs usage for all of ~/.cache. I'll have an opinion in a month or so...

How did you manage to do that, i.e. shift $HOME/.cache to tempfs for all the users? If possible can you please put it in a blog and or over here? I would also like to do it. Want to avoid disk thrashing. And what happens in case of reboot when the directory inside $HOME/.cache structure is no longer there?

[sunrat]

Default of 60? Why do Debian developers keep such a high value?

I'm not so sure 60 is 'high'. The scale goes to 200 IIRC. So do some testing!

I recently revised a vm backing to default, using zram and no other swap, and also increased tmpfs usage for all of ~/.cache. I'll have an opinion in a month or so...

Interested to hear your results. I predict any change will be barely perceptible.
I vaguely recall it being discussed at LinuxMusicians forum as the accepted recommendation was to change it to 10 for audio production. The result was that it is pretty insignificant and 60 default works fine in most cases.

[CwF]

I predict any change will be barely perceptible.

Yes, I fully expect slightly higher swap use pancaked by zram for slightly higher free mem without anybody noticing. Win win win.
@DebianFox Here ya go;
https://forums.debian.net/viewtopic.php ... 66#p807166

[DebianFox]

Thanks @CwF . Will go through the post.

[CwF]

Here is a good example of memory variation across a medium term.
There is no swap usage in this case, but it does show well the metric to watch is ‘available’
Use case dictates how shared and buff/cache will vary. In this case large tmpfs's are in use and show under ‘shared’. Large scp transfers bounced through, and they show up it buff/cache. The seemingly low ‘free’ stat isn't so indicative of much. I didn't catch it here, but my assumption is low ‘available’ is what would trigger some swap.

Code: Select all

$  free
              total        used        free      shared  buff/cache   available
Mem:       66012240     6594960    14540392    26925308    44876888    31759944
Swap:       7999484           0     7999484
$  free
              total        used        free      shared  buff/cache   available
Mem:       66012240    11088720    18569876    14765968    36353644    39425524
Swap:       7999484           0     7999484
$  free -g
              total        used        free      shared  buff/cache   available
Mem:             62          10          17          14          34          37
Swap:             7           0           7
$  free -g
              total        used        free      shared  buff/cache   available
Mem:             62          12           5          14          44          35
Swap:             7           0           7
$  free -g
              total        used        free      shared  buff/cache   available
Mem:             62          13           4          14          44          34
Swap:             7           0           7
$  free -g
              total        used        free      shared  buff/cache   available
Mem:             62          10           7          14          44          37
Swap:             7           0           7

[DebianFox]

@CwF This is with vm.swappiness set to 60/70? And what applications were you using at that period of time?

[CwF]

@CwF This is with vm.swappiness set to 60/70? And what applications were you using at that period of time?

That one is swappiness 30 with a real swap partition. It is a ‘front-end’ machine that my keyboard and mouse are connected to; a head for the headless, the x2x and x2vnc master, the home for transient vm's either in a nursery sense or as a temporary migration host, and where images/disk are read, upgraded, tested, stored, and eventually written out. The traffic coming through was an offload and upgrade for a new bigger machine - the new server went live yesterday with Debian 12.7 and will be massaged for a weeks on things that can only be done on bare metal.

At the moment;

Code: Select all

 $  free -g
               total        used        free      shared  buff/cache   available
Mem:             251          19         229          16          20         232
Swap:             25           0          25
 

That swappiness is also 30 that may go up. All the swap is zram. If this machine makes a 6 month uptime run as more is migrated to it, then the above machine will receive its new image and a new role that has been waiting as a vm for the last 3 years...

The browser vm's with swappiness 60 and 8GB need some more time. Nobody has hit swap yet. They are often limited to 6GB at boot, zram sized per 8, with the allocation increased when a page locks. Up the allocation and the vm magically comes to life.