Dimensioning of the SWAP space

[MicroScreen]

After trying for a while to find as much information as possible about setting up and dimensioning a swap partition, the question still arises for me, according to which criteria it must be designed? I am not interested in the traditional rules of thumb, but only in a technically sound explanation. I'm really curious whether someone can conclusively answer this question.

[CwF]

There should not be one formula. There is the purpose of the machine that through past iterations some generalities should be known, since rarely is this question starting from scratch!

Recognize that the common-sense idea that swap is for when you run out of memory isn't the only reason it exist. While maybe an original reason, generous systems now have plenty of ram and those past examples show swap is still used. Your experience with those past examples show when particular applications are arranged to use it. Past examples of uptime should give you a hint of the amount of stale memory that may be kept in swap. With extended uptime and activity you may observe variable use and the fact swap use does go down with time, not just up, so something is thinking. All things equal, 'pressure' settings can push the used amount one way or the other with little demonstrable difference in performance. This may come down to preference, shaped by experience, like I said, of your past examples.

Realize there are types we can argue about, and amounts to stress over, and applicable use case. Overall, there should be some form of swap. The details are negotiable.

[MicroScreen]

There should not be one formula. There is the purpose of the machine that through past iterations some generalities should be known, since rarely is this question starting from scratch! Recognize that the common-sense idea that swap is for when you run out of memory isn't the only reason it exist. While maybe an original reason, generous systems now have plenty of ram and those past examples show swap is still used. Realize there are types we can argue about, and amounts to stress over, and applicable use case. Overall, there should be some form of swap. The details are negotiable.

Thanks for your opinion, I think we are getting a lot closer to the truth. At the same time it confirms several of my assumptions that the size of my RAM is not the only factor in this question, but also the memory management of various applications, as well as the amount of data processed play a decisive role.
Which tools make it possible to monitor my system most effectively, especially with regard to swapping, during runtime? Can you recommend something for that?

[Head_on_a_Stick]

Code: Select all

free -h # static measurement
grep pswp /proc/vmstat # record of swap usage

[MicroScreen]

Code: Select all

free -h # static measurement
grep pswp /proc/vmstat # record of swap usage

Thanks for your help, free is one of the hot topics in this matter, I was just exploring the possibilities of such programs. Beside free I also found two other programs: top and sync, which can also be very useful. There is also a lot of reading material in the man pages that I will take a closer look at.

I think it will definitely be able to find the most suitable settings for my system.

[CwF]

Since I've never figured out what I'm missing I use XFCE. So I have a minimize to 'Notification Area' instance of xfce4-taskmanager that gives me a nice hover message of user memory and swap usage. I may have a genmon watching something from time to time on setups where that info isn't the full story. With root rights "swapon" gives a snapshot that is meaningful to glance at at the right times. Once settled, you don't really need to watch it.

As important as the use case of the machine is the status of the activity, stable or experimental. A stable machine is one where the judgement is already in and we don't expect anything to get out of whack. Unused swap of that machine is a waste. On a testing machine things may go wrong. With a large swap available an errant process might be contained, maybe not, but more space could save you.

[Head_on_a_Stick]

Use earlyoom to constrain runaway memory hogs. Fedora now include it in their default package list while they're waiting for the kernel to improve it's OOM management.

[MicroScreen]

Since I've never figured out what I'm missing I use XFCE. So I have a minimize to 'Notification Area' instance of xfce4-taskmanager that gives me a nice hover message of user memory and swap usage. I may have a genmon watching something from time to time on setups where that info isn't the full story. With root rights "swapon" gives a snapshot that is meaningful to glance at at the right times. Once settled, you don't really need to watch it.

As important as the use case of the machine is the status of the activity, stable or experimental. A stable machine is one where the judgement is already in and we don't expect anything to get out of whack. Unused swap of that machine is a waste. On a testing machine things may go wrong. With a large swap available an errant process might be contained, maybe not, but more space could save you.

The plan is for the initial installation to be 'stable', as this creates a more secure starting point. In order to avoid unused SWAP space, I consider it essential to critically monitor the memory requirement right from the start. XFCE is at least an option for me too, in order to guarantee the leanest possible installation, I also thought of LXDE, it is modern and stylish. I will also take a very close look at the swapon and swapoff commands and read the respective descriptions. If there are also suitable plugins for the desktop, it will quickly become clear what is most practical.

What I was able to find out in the meantime is that Linux generally has a completely different memory management than, for example, Windows. The kernel alone constantly outsources its cash as required, depending on other factors, of course. Data is swapped out at a very early stage, long before the RAM could theoretically become full. The more I/O movements take place and the more different the files are, the sooner the swapping begins. It also pays to look out for the /proc/meminfo file, which can be quite instructive. There are also pagetables that store and record a lot of information during such processes. In the end, everything together gives an overall picture of data traffic in a system.

[Bulkley]

After you have your system running the way you want it, look up swappiness.

Swappiness is a Linux kernel property that sets the balance between swapping out pages from the physical memory to the swap space and removing pages from the page cache. It basically defines how often the system will use the swap space.

Depending on your system and what you do with it changing swappiness may be worth adjusting.

[MicroScreen]

Depending on your system and what you do with it changing swappiness may be worth adjusting.

Thanks for the hint, as far as I was able to find out, the value for the swappiness is first set automatically by the system, but it can then be adjusted if everything is stable and the first results are available. In most of the new machines, the hardware is so large that the swappiness is usually too high; a reduction from 60 to 40, for example, makes sense here. The basic concept behind this is obviously to keep the number of data movements between cash and various mass storage devices to a minimum. A series of specific tests should provide information about various changes in performance. Let's see if I can find a suitable benchmark program for it beforehand.

One of my ideas for such tests with as many I/O movements as possible would be to perform a backup of several data carriers simultaneously to one of the HDDs while a benchmark test is running. I can then compare the results with the various recordings that can be found in the files already mentioned. It will certainly take a while before one can say with certainty that things are really going well.

But it will be fun to explore a virgin machine that I have built myself - no stress, no fear of losing data, no unsolvable problems!

[p.H]

In order to avoid unused SWAP space

What's wrong with unused swap space ? Disk space is cheap.

[Head_on_a_Stick]

swappiness

I've had this argument with Jessie from DistroWatch but that doesn't do what you think:

https://bbs.archlinux.org/viewtopic.php?id=231265

tl;dr: the value controls how the swap is used rather than when the kernel starts to use swap.

[CwF]

tests with as many I/O movements as possible would be to perform a backup of several data carriers simultaneously to one of the HDDs while a benchmark test is running

This hints at a variable - What SCALE are we talking here. 'One' of the drives is exactly right, could be many! I can tell you for sure at large scale - with virtual machines as workers - you can bring a 'task' to it's knees and elsewhere in the same system - NOT NOTICE! For me 'large' starts at 2-4 cores with nothing to do out of 16+ and memory >64GB, with storage to match - that is an OS on an ssd all by itself and another handful of drives dedicated to purpose.

In my 'not-unanimous' opinion when the scale is large enough there is no need for a traditional swap partition or swap file at all. Instead, only zramswap. Every 'big' machine I've built doing similar jobs maintains a similar amount of swap over it's uptime. With uptimes of 6 months plus, I see a few hundred MB's with spikes to a few gigs. When swap total is 5GB, free memory is still that much or larger. Whether that is a swap partition, as in my 'stetch' examples, or only zramswap in my bullseye testing - it makes no difference.
Hence my only general advice

Overall, there should be some form of swap. The details are negotiable.

For a compromise 'buster', a little of both;

Code: Select all

#  swapon
NAME        TYPE      SIZE   USED PRIO
/dev/dm-2   partition 4.6G 689.7M   -2
/dev/zram0  partition 128M  46.7M  100
/dev/zram1  partition 128M  48.4M  100
/dev/zram2  partition 128M  48.7M  100
/dev/zram3  partition 128M  50.3M  100
........
/dev/zram15 partition 128M  48.9M  100
#  uptime
 08:08:48 up 110 days, 23:38,  2 users,  load average: 0.86, 0.92, 0.90


on a stretch;

Code: Select all

#  uptime
 08:10:08 up 61 days, 17:35,  3 users,  load average: 0.76, 0.60, 0.61
#  swapon
NAME      TYPE      SIZE USED PRIO
/dev/dm-1 partition 7.6G 326M   -1


No buslleye up right now, but roughly the same demonstrated so far, except ONLY zramswap (~12 GB)

That buster above had a runaway a few weeks ago and it ran up to a 42+ load average at 6GB of swap. Now all calm. As seen above, there is a swap partition, it did fill in minutes, I did notice. If all was zramswap it would have swapped the same volume in seconds.

So I've upped the total swap and changed its form over the generations. On swappiness, it is the 'pressure' I mentioned and will push more or less to swap. The craze has been 10-20. Lower is better for less swapping to an actual disc, higher is better to move the stale to zramswap, freeing memory. With modern processors capable of paralleling ALL the swap action, you won't notice the zramswap activity. I would leave it alone until after you've seen some uptime, 30 is about right for me.

So on the negotiable aspects, I lied. My rules are simple - but maybe not universal. Swap Partitions are archaic and no longer needed. Swap Files are a substitute without any real advantage. Zramswap rocks. And yes, you need something! Swappiness=0 with nothing works fine until it doesn't. I run some vm's like that but with a trick - they have unallocated memory to add when they lock up tight, and they do unfreeze and continue like nothing happened. Not easily possible on bare metal.

Builds an a smaller scale will often produce edge cases where advice is not so straight forward. Once you can create an 8core 32GB VM with dedicated storage with that or more to spare on the host, resource contention simply doesn't happen. I consider that level of VM to be 'large' in itself, maybe a nested hypervisor or graphics/media application or some game station, and enough for any one job I encounter. I don't run any that big very often.

Regardless, sounds like fun is happening!

[MicroScreen]

What's wrong with unused swap space ? Disk space is cheap.

As a system administrator in a company, you can perhaps take this point of view, but not if you want to operate your machine as efficiently as possible!

[MicroScreen]

So on the negotiable aspects, I lied. My rules are simple - but maybe not universal. Swap Partitions are archaic and no longer needed. Swap Files are a substitute without any real advantage. Zramswap rocks. And yes, you need something! Swappiness=0 with nothing works fine until it doesn't. I run some vm's like that but with a trick - they have unallocated memory to add when they lock up tight, and they do unfreeze and continue like nothing happened. Not easily possible on bare metal.

Basically, there are always different opinions on this, all of which have one thing in common, namely the differences in hardware, software and configurations in various applications. I bet that very few of them are actually operated in the most optimal way. The discussion about whether and how much swap-space is needed at all mostly goes in the wrong direction. This also means that the individual terms are not always assigned correctly, as you can see from some of the reactions here.

To all of you!
So again, for a better understanding, what I want to know about my machine is what Linux and the installed kernel do with regard to the entire memory management, nothing else. I had deliberately avoided terms like 'swap partition', 'swap file' and other forms of swapping, because I didn't want to be pinned down in any direction. That's why I only use the abstract term 'Swap Space'. For me the adventure begins exactly where the kernel begins to fill its own cash in RAM in order to outsource data of any kind. 'Swappiness' plays the very last role in this scenario, so it is only readjusted after sufficient measurements, if at all.

What I have learned in the meantime, however, is that Linux and its memory management consider RAM and the assigned swap space as an organic unit and therefore treats it that way, no matter how large or small a machine is. This has consequences for the entire data traffic in a computer. In addition, there is also an extremely complex feedback from the installed mass storage devices, since these also have cash storage, which is also taken into account here. My machine should run optimally from the start, how the data flow changes through various settings is the main subject of my measurements. Hence the title of this thread is: Dimensioning of the SWAP space!

I am neither a gamer nor a philosopher, but more of a 'researcher' and as such I will dive deeply into the matter.