Buggy swap process (kswapd)?

Kernels & Hardware, configuring network, installing services

Buggy swap process (kswapd)?

Postby styxxx » 2013-09-25 01:18

Hi,

I hope this topic doesn't already exist somewhere (everytime I try to use the search function I get "Sorry but you cannot use search at this time. Please try again in a few minutes."), but I guess it doesn't. This server also has a seriously problem since I always get an Internal Server Error. Luckily I copied the text into an editor...

About my topic:

There are multiple issues concerning the swap process. It all started a few years ago. Too bad I don't remember what kernel version exactly (I guess it was around fall/winter 2010?). I'm pretty sure that the kernels shipped with Ubuntu 8.04 weren't affected since everything worked fine back then. First I thought it was an hardware issue or some faulty local configurations. But I experienced those problems several times since then on different machines with different debian based distributions (like Debian, LMDE, Ubuntu) and so did other users I spoke with.


So what's wrong:


- The system starts swapping highly erratic, no matter what swappiness value is set. On the one hand 99% of memory can be in use without swapping for a long time but on the other it suddenly swaps when only using a fraction of it.
- When it swaps ist swaps almost everything. Sometimes really everything. The memory usage will drop to a few percent. After that the process is reversed and the swapped data is read into memory again (at least parts of it). Until there's about the same percentage of ram and swap in use.
- It seems to swap the cached/buffered data too (but it might only seem that way)
- After standby it almost always start so swap within the first minute. No matter how much memory is actually used. So before going into standby I have to kill most of the programms if I don't want to wait hours (no kidding, it's really hourse. See next point) for it to finish. It does so even if standby mode lasted for only 1 second. Maybe the system gets confused and thinks the data is all very old?
- Swapping uses a lot, really a lot of resources and freezes the whole computer. Happend to only swap about 100 MB but the pc was frozen for an hour with the swap process using all cpu and all IO resources - even on new hardware. I don't know what it does, but I can encrypt a whole harddisk in a fraction of this time. Even login on console isn't possible anymore (timeout). After waking up from standy this results in a black screen until it's finished (which may take half an hour, 1 hour or even forever, see below).
- The swap process starts without even using swap. I've currently no swap partition/file in use and even turned swappiness to 0. Still the process kicks in from time to time, using a lot of cpu and 99% of IO (according to iotop), slowing down the whole PC. Even happens with 2GB of 3,5GB Ram unused (except for cache). This espacially happens when I'm copying some files. I guess the swap process kicks in since it thinks that all the cached/buffered data is actual memory usage and wants to swap it. But since there's no swap the process randomly access all my harddisks. Don't know why, maybe it's looking for something swap-like.
- Sometimes the swap process seems to be cought in some form of loop. Even without swap being installed. The process is taking all of the recourses and won't stop. There's no other way than a hardcore shutdown (unplugging power or pressing power button for 4 secs). Even the magic SysRq key fails.
- While it swaps parts of the system may crash, like the file browser or even kernel modules. Very often I have to manually modprobe psmouse to use the mouse pointer again since it stays frozen.

So first I thought I had buggy hardware or the pc was just too old. But turns out I experienced it on different machines with new and shiny hardware and brand new OS installaton. No hardware failures of any kind (like hdd or ram). Swap is not encrypted and has it's own partition on a fast harddisk, usually one of the first partitions. But even a slow and encrypted hard drive wouldn't explain a frozen system for over an hour when only a few 100 MBs are written to disk.


The problem is that those bugs make linux almost unusable. And since even deactivating swap doesn't help there's almost no other solution than to switch the OS. I tried to monitor what happens many times in the past. The problem ist that most of the time the system doesn't respond and even logging scripts don't work anymore.


People love diagrams, so here's one of the memory values I could read after the system was usable again:
verlauf nach swap.png
reversed swapping
verlauf nach swap.png (15.82 KiB) Viewed 8873 times

As you can see some of the swap is read into memory again (no programs were loaded during that time). Sometimes a lot more data is re-read.


And here's a screenshot showing the usage of cpu time during a session. Note: The system was up for about 1 day - but at least half of the time in standby.
kswapd_c.png
Ain't nobody got (cpu) time for that
kswapd_c.png (18.99 KiB) Viewed 8873 times


And here should be one screenshot showing kswapd in action but since I can't upload a file without an internel server error:
kswapd was at 99% while all other processes were at 0%.


Other examples:

Those were the values after the system hang during swap for several minutes:
Code: Select all
             total       used       free     shared    buffers     cached
Mem:       3653072     785464    2867608          0       6704     157200
-/+ buffers/cache:     621560    3031512
Swap:      3651580     261136    3390444

Only about 250 MB were swapped. It makes no sense to take half an hour for that.


In the follow case the system was usable again but still wrote swap data to disk:
Code: Select all
             total       used       free     shared    buffers     cached
Mem:       3653068     746428    2906640          0       2356      74092
-/+ buffers/cache:     669980    2983088
Swap:      3651580     757040    2894540

After a few minutes without using the pc:
Code: Select all
             total       used       free     shared    buffers     cached
Mem:       3653068     666976    2986092          0        424      32564
-/+ buffers/cache:     633988    3019080
Swap:      3651580     848036    2803544

You can see the swap usage increased while the memory usage was below the swap. Then the process reversed:
Code: Select all
             total       used       free     shared    buffers     cached
Mem:       3653068     855444    2797624          0      14740      80620
-/+ buffers/cache:     760084    2892984
Swap:      3651580     726204    2925376

Now there's more data in memory again. But why was that crap swapped at all? And why did it take minutes? A test with dd shows that writing 1G only lasts about 15 seconds - without freezing the PC. Even if it was slower minutes or hours is unacceptable.

There's also a difference in the used swap space and the freed memory (and vice versa) during the swapping process. It seems like the swapped data might be compressed?


So since I experienced it on different machines and after asking other users some of them also knew about that problem (and had experienced it in the past) I guess there actually are bugs? Are there other people here being affected? Or does anyone know a solution?

Thanks,
styxxx
styxxx
 
Posts: 18
Joined: 2013-06-03 20:24

Re: Buggy swap process (kswapd)?

Postby vbrummond » 2013-09-25 06:27

Obviously this shouldn't happen. I have seen issues where swap wasn't needed and yet was used, I could never find the answer to it. I was going to suggest you disable your swap partition, though you had already tried this. The only other suggestions I have are:

1) Disable the swap in /etc/fstab
2) Compile (or install) a newer kernel release to see if the issue is fixed.
3) Compile a kernel without swap support,
4) Try (temporarily) booting where the kernel doesn't see high memory. Try adding the parameter: mem=800m
Always on Debian Testing
vbrummond
 
Posts: 4467
Joined: 2010-03-02 01:42

Re: Buggy swap process (kswapd)?

Postby styxxx » 2013-09-25 13:17

Hi,

thanks :)
Yeah, the swap partition is already disabled. Although having ~3,5GB RAM I'd rather have a swap space since I often use programs with a lot of memory usage so it would be nice if a swap was there and it would work as intented ;)

I totally forgot to post the kernel version on the laptop I'm currently on:
Code: Select all
$ uname -a
Linux notebookneu 3.2.0-4-amd64 #1 SMP Debian 3.2.41-2 x86_64 GNU/Linux

The problem doesn't seem to be fixed in newer versions (at least I didn't find anything about it in changelogs).

Btw I noticed this behavior on 32bit and 64bit systems. Sometimes I thought I fixed it since it's so erratic. There can be days without noticing anything strange and then it happens multiple times a day. I also didn't found a way to trigger it with 100% certainty (using stand-by mode works best though - or being under time pressure and having to look up something important :/ ).

What exactly is the purpose of the mem parameter in this case?
styxxx
 
Posts: 18
Joined: 2013-06-03 20:24

Re: Buggy swap process (kswapd)?

Postby lithopsian » 2013-09-25 16:26

Find out what you're running that triggers the bizarre behaviour, because this doesn't happen for most people.
lithopsian
 
Posts: 241
Joined: 2013-05-28 13:11

Re: Buggy swap process (kswapd)?

Postby styxxx » 2013-09-25 18:44

I already tried but I couldn't determine any specific process or configuration that could trigger it. Happens on newly installed systems as well as on older ones. :/ And I don't use anything extraordinary, just a web browser for example. The more memory is used the more often kswapd starts - even without a set swap partition. Currently it's also more likely to disturb when a program starts to copy files (for example: a download manager copies a finished file from /tmp/ to an usb disk. While doing so kswapd starts and slows down the whole process - even with plenty of free memory available).

Currently the system's up for almost 19 hours (~5 hrs of it being in stand-by, so 14 hours of actual running time).
kswapd used cpu time during that period:
Code: Select all
   23 root      20   0     0    0    0 S   0,0  0,0   4:31.70 kswapd0

And there is no swap space configured and vm.swappiness=0 ;)

But acutally cpu usage isn't the problem, it's more the IO usage that freezes the pc (cpu wait value skyrockets). I remember noticingen an increase in system load during IO operations after kernel updates at about the same time the swap bug started to appear (about 3 years ago, I recall some news about changes in the IO scheduler back then), but seemed to get better afterwards. I'm not entirely sure though. Also normal disk operations doesn't affect the system this much. But for some reason kswapd does.
styxxx
 
Posts: 18
Joined: 2013-06-03 20:24

Re: Buggy swap process (kswapd)?

Postby eor2004 » 2013-10-04 18:55

Let me guess, you're using KDE 4.x and this happens everytime you transfer data from your pc to ext usb hdd and vice versa right?, this same thing happened to me when I was using Mageia 3 with KDE 4, I even filed a bug report with Mageia bugzilla and KDE.org bugtracker, neither one could help me or fix the issue, I guess they're interested in adding new features and not on fixing bugs, especially the people at KDE.org because is a upstream bug, not distro specifically. I tested OpenSuse 12.3 64bit KDE and this bug did not happened, somehow they managed to disable kswap, IMO kswap is the problem here.

https://bugs.mageia.org/show_bug.cgi?id=11077
https://bugs.kde.org/show_bug.cgi?id=324144
OS Debian Stretch 64-bit DE Gnome 3.22 CPU AMD Phenom II X4 925 @ 2.8GHZ RAM 8GB CORSAIR XMS2 PC2-6400U DDR2 (CM2X2048-6400C5C) GPU ATI Radeon HD 3200 Mobo Gigabyte GA-MA78GPM-DS2H HDD Hitachi 2TB (HUA723020ALA641) 7200RPM
User avatar
eor2004
 
Posts: 188
Joined: 2013-10-01 22:49
Location: Puerto Rico

Re: Buggy swap process (kswapd)?

Postby styxxx » 2013-10-05 03:08

No, no KDE ;) I'm using MATE (but also happened with gnome2 and gnome3). I don't hink it's related to the desktop environment, at least not in my case.
It's still terrible and I've to kill processes every few hours before too much memory is used to prevent kswapd from going nuts :(
A system that tends to freeze every hour due to faulty swapping is just unusable.
styxxx
 
Posts: 18
Joined: 2013-06-03 20:24

Re: Buggy swap process (kswapd)?

Postby project499 » 2014-05-10 16:20

I've experienced the same, and it has got worse and worse. Now I have to be extremely
careful in Kubuntu 14.04 about what I'm running at any time. It did become obvious
about 2 years ago, but it wasn't so bad really so I ignored it and thought it was going to
be fixed. It hasn't and it is now worse than ever, and to a point where the system is getting
close to unusable.
project499
 
Posts: 2
Joined: 2014-05-10 16:14

Re: Buggy swap process (kswapd)?

Postby dibl » 2014-05-10 19:56

I am a sid user, and currently running a 3.15-RC kernel, in which there is no swap issue. However, back when we were in the 3.5, 3.6, 3.7, etc. series, it became a very noticeable issue, which I confirmed with other users. Starting with 3.8-3 it got much better and subsequently disappeared. I believe it is a kernel configuration issue -- my desktop rig has 6GB of memory to run a single-user KDE desktop, and I also turned down vmswappiness. This system was originally installed in mid-2011, so it has been around and under observation for quite awhile, and I have not made many tweaks that would affect swapping. It has a large swap partition which normally remains unused. Possibly some research could turn up more details on how widespread the issue is, and whether there are any userland handles to adjust to calm it down.
Debian sid / siduction KDE
dibl
 
Posts: 528
Joined: 2009-10-13 19:50
Location: Dayton, Ohio, USA

Re: Buggy swap process (kswapd)?

Postby project499 » 2014-05-15 13:39

I am currently on kubuntu 14.04 and the kernel is 3.13.0-24 on a 64bit system.
What I have noticed is that moving to and from usb and java are the best or rather
worst contenders but then flash ain't helping either. That there isn't built in
some failsafe into kswapd is beyond me - especially the fact there is no way
to even hup it either.

I've thought about trying to reinstall to see if something old is making it worse
right now. I'll do some more testing first though.
project499
 
Posts: 2
Joined: 2014-05-10 16:14

Re: Buggy swap process (kswapd)?

Postby CarlosMartini » 2014-08-14 02:26

Hello,

Same issue here.

Code: Select all
PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM     TIME+ COMMAND                                                                                                             
 26 root      20   0     0    0    0 S 99,0  0,0   8:46.82 [kswapd0]


This problem happens all the time. In some days, using my computer almost becomes a impossible mission.

I'm using Linux Mint Debian.

carlos@carlos-pc ~ $ uname -a
Linux carlos-pc 3.11-2-amd64 #1 SMP Debian 3.11.8-1 (2013-11-13) x86_64 GNU/Linux
CarlosMartini
 
Posts: 1
Joined: 2014-08-14 02:18


Return to System configuration

Who is online

Users browsing this forum: No registered users and 8 guests

fashionable