I hope this topic doesn't already exist somewhere (everytime I try to use the search function I get "Sorry but you cannot use search at this time. Please try again in a few minutes."), but I guess it doesn't. This server also has a seriously problem since I always get an Internal Server Error. Luckily I copied the text into an editor...
About my topic:
There are multiple issues concerning the swap process. It all started a few years ago. Too bad I don't remember what kernel version exactly (I guess it was around fall/winter 2010?). I'm pretty sure that the kernels shipped with Ubuntu 8.04 weren't affected since everything worked fine back then. First I thought it was an hardware issue or some faulty local configurations. But I experienced those problems several times since then on different machines with different debian based distributions (like Debian, LMDE, Ubuntu) and so did other users I spoke with.
So what's wrong:
- The system starts swapping highly erratic, no matter what swappiness value is set. On the one hand 99% of memory can be in use without swapping for a long time but on the other it suddenly swaps when only using a fraction of it.
- When it swaps ist swaps almost everything. Sometimes really everything. The memory usage will drop to a few percent. After that the process is reversed and the swapped data is read into memory again (at least parts of it). Until there's about the same percentage of ram and swap in use.
- It seems to swap the cached/buffered data too (but it might only seem that way)
- After standby it almost always start so swap within the first minute. No matter how much memory is actually used. So before going into standby I have to kill most of the programms if I don't want to wait hours (no kidding, it's really hourse. See next point) for it to finish. It does so even if standby mode lasted for only 1 second. Maybe the system gets confused and thinks the data is all very old?
- Swapping uses a lot, really a lot of resources and freezes the whole computer. Happend to only swap about 100 MB but the pc was frozen for an hour with the swap process using all cpu and all IO resources - even on new hardware. I don't know what it does, but I can encrypt a whole harddisk in a fraction of this time. Even login on console isn't possible anymore (timeout). After waking up from standy this results in a black screen until it's finished (which may take half an hour, 1 hour or even forever, see below).
- The swap process starts without even using swap. I've currently no swap partition/file in use and even turned swappiness to 0. Still the process kicks in from time to time, using a lot of cpu and 99% of IO (according to iotop), slowing down the whole PC. Even happens with 2GB of 3,5GB Ram unused (except for cache). This espacially happens when I'm copying some files. I guess the swap process kicks in since it thinks that all the cached/buffered data is actual memory usage and wants to swap it. But since there's no swap the process randomly access all my harddisks. Don't know why, maybe it's looking for something swap-like.
- Sometimes the swap process seems to be cought in some form of loop. Even without swap being installed. The process is taking all of the recourses and won't stop. There's no other way than a hardcore shutdown (unplugging power or pressing power button for 4 secs). Even the magic SysRq key fails.
- While it swaps parts of the system may crash, like the file browser or even kernel modules. Very often I have to manually modprobe psmouse to use the mouse pointer again since it stays frozen.
So first I thought I had buggy hardware or the pc was just too old. But turns out I experienced it on different machines with new and shiny hardware and brand new OS installaton. No hardware failures of any kind (like hdd or ram). Swap is not encrypted and has it's own partition on a fast harddisk, usually one of the first partitions. But even a slow and encrypted hard drive wouldn't explain a frozen system for over an hour when only a few 100 MBs are written to disk.
The problem is that those bugs make linux almost unusable. And since even deactivating swap doesn't help there's almost no other solution than to switch the OS. I tried to monitor what happens many times in the past. The problem ist that most of the time the system doesn't respond and even logging scripts don't work anymore.
People love diagrams, so here's one of the memory values I could read after the system was usable again: As you can see some of the swap is read into memory again (no programs were loaded during that time). Sometimes a lot more data is re-read.
And here's a screenshot showing the usage of cpu time during a session. Note: The system was up for about 1 day - but at least half of the time in standby. And here should be one screenshot showing kswapd in action but since I can't upload a file without an internel server error:
kswapd was at 99% while all other processes were at 0%.
Other examples:
Those were the values after the system hang during swap for several minutes:
Code: Select all
total used free shared buffers cached
Mem: 3653072 785464 2867608 0 6704 157200
-/+ buffers/cache: 621560 3031512
Swap: 3651580 261136 3390444
In the follow case the system was usable again but still wrote swap data to disk:
Code: Select all
total used free shared buffers cached
Mem: 3653068 746428 2906640 0 2356 74092
-/+ buffers/cache: 669980 2983088
Swap: 3651580 757040 2894540
Code: Select all
total used free shared buffers cached
Mem: 3653068 666976 2986092 0 424 32564
-/+ buffers/cache: 633988 3019080
Swap: 3651580 848036 2803544
Code: Select all
total used free shared buffers cached
Mem: 3653068 855444 2797624 0 14740 80620
-/+ buffers/cache: 760084 2892984
Swap: 3651580 726204 2925376
There's also a difference in the used swap space and the freed memory (and vice versa) during the swapping process. It seems like the swapped data might be compressed?
So since I experienced it on different machines and after asking other users some of them also knew about that problem (and had experienced it in the past) I guess there actually are bugs? Are there other people here being affected? Or does anyone know a solution?
Thanks,
styxxx