[SOLVED] Diagnosing high cpu load, no xorg

[n_hologram]

Solution in final post. System was reproducing, over and over again, an unusual dhclient error message, which decided to run constantly because of a lingering sendmail file. The clue was in htop's "red bars" (kernel threads), which signaled that the issue was probably deeper than a single process. Once that file was removed, the system worked as expected.

My netbook, which functions as a small, unimpressive project "server," is reporting an abnormally high cpu load on the core and virtual core: between 50-60%, and not subsiding. On two-to-five-minute intervals, I hear the little fan whirr for a few seconds and subside. All of that would be fine, if I knew which program was consuming all the resources, but htop reports that only four programs are using cpu, and the values are ~4%, 2%, 1%, and 1%. Rebooting changed nothing. I'm looking for suggestions for narrowing the source of this issue.

[phenest]

What program is showing the high cpu load?

[n_hologram]

What program is showing the high cpu load?

dhclient is the 4%-er

[phenest]

I'll rephrase: What program did you use to measure the high cpu load?

[n_hologram]

I'll rephrase: What program did you use to measure the high cpu load?

htop

EDIT: I also checked with top, same results: http://stackoverflow.com/questions/9229 ... ux#9229692

Code: Select all

top -bn1 | grep "Cpu(s)" | \
           sed "s/.*, *\([0-9.]*\)%* id.*/\1/" | \
           awk '{print 100 - $1"%"}


[orythem27]

between 50-60%, ...... only four programs are using cpu, and the values are ~4%, 2%, 1%, and 1%.

Use `htop` to verify that if it's indeed some kernel threads that is actually acting up. Look for the red portion in the CPU usage bar. See this page for detail.

For further investigation you can use the following tools to unveil more details.

• itop
• powertop

[wizard10000]

...For further investigation you can use the following tools to unveil more details.

• itop
• powertop

I'm gonna add glances to your list. Very nice ncurses interface and it's my go-to for monitoring remote systems because in addition to processor and memory usage it shows disk and network I/O in realtime.

[phenest]

My netbook, ... is reporting an abnormally high cpu load on the core and virtual core: between 50-60%, and not subsiding.

Reported by what?

but htop reports that only four programs are using cpu, and the values are ~4%, 2%, 1%, and 1%.

I don't understand. If htop is showing low CPU usage < 10% combined, how do you know there is a 50-60% load?

[phenest]

@ OP:
Please reply to this thread rather than PM'ing me. Your replies are helpful to others, not just me.

So you're saying that htop is reporting a total CPU load of 50-60%, but only 4 programs show any usage and they are < 10% combined. Yes?

[n_hologram]

@ OP:
Please reply to this thread rather than PM'ing me. Your replies are helpful to others, not just me.

Sure thing:
I had already said I used htop, along with a "top" command, and they both yielded the same results. I thought "reported" would be an appropriate verb.

I also do not understand why it's doing this, but I don't know much about kernel threads because I've never had a reason to investigate them before. I intend to update the post once I get a chance to look into that.

It's also a pain because, and I'm not sure if this is related or not, but any sort of local or external access to that system (ssh, mpd, etc.) will just stop working until I reboot (after which it works as expected for awhile, and then doesn't), so I have to be physically present at that computer to check anything until that gets resolved. Which is annoying.

So you're saying that htop is reporting a total CPU load of 50-60%, but only 4 programs show any usage and they are < 10% combined. Yes?

Yes.

EDIT: Screenshot for your viewing pleasure.

[n_hologram]

Alright, you guys are gonna love this. I have kismet installed and set-up in server mode, using the wireless card in monitor mode. I'm using sysv, and /etc/rc.local is configured as so:

Code: Select all

# Start the wifi card in monitor mode for Kismet reasons
airmon-ng start wlan0
/usr/bin/sudo -u pi "/usr/bin/kismet_server" -n --daemonize -p /home/pi/kismet_server


At boot, I watched the startup messages (I don't know the proper term for those), and noticed that, when it runs this command, it complains that dhclient and dhclient script might interfere -- which is strange, since when I run the airmon command as root (sudo su) within a console, it doesn't produce that error. Also, "~# killall -9 dhclient" solved the original issue, albeit superficially. So, I think, I'll just comment-out those two lines from rc.local. Sure enough, after a reboot, my cpu usage runs exactly as expected.

Now, fully intrigued, I uncomment those lines, reboot, and double-check htop. That screenshot I sent in my previous post, it shows a dhclient command constantly running, around 4%, but without those lines in rc.local it doesn't show any dhclient at all, and neither do any of my other Debian systems. I suddenly have an epiphany: why is it running the entire time? So, I try a simple "~# dhclient eth0", and in a normal system, it would show a simple, one time "RTNETLINK answers: File Exists". However, when I run the same command in my server, it gives me two messages, ad infinitum:

Code: Select all

...
/sbin/dhclient-script: 28: .: Can't open /usr/share/sendmail/dynamic
RTNETLINK answers: File exists
/sbin/dhclient-script: 28: .: Can't open /usr/share/sendmail/dynamic
RTNETLINK answers: File exists
...


until I CTRL+C. So, it appears the issue was that I had daemonized an error message, and the little atom processor wasn't happy with that.

This was, however, a nice tidbit of information. I ran across a thread which articulated solutions to the same issue. Sendmail was already removed through apt-get, but I guess a lingering file interfered with dhclient's normal behavior, so I went ahead and removed it:

Code: Select all

rm /etc/dhcp/dhclient-exit-hooks.d/sendmail


And now everything is up to normal. Thanks to everyone for helping out, however, I'm still curious as to why htop didn't show anything abnormal in the process list; and if that's normal to not show, I still wonder if there would have been a better way to narrow that without crapshooting through programs and bootscreens. Ah, well...

[phenest]

however, I'm still curious as to why htop didn't show anything abnormal in the process list; and if that's normal to not show

Maybe it's because htop only refreshes about once a second and it misses it.

[n_hologram]

however, I'm still curious as to why htop didn't show anything abnormal in the process list; and if that's normal to not show

Maybe it's because htop only refreshes about once a second and it misses it.

It would update with the same frequency as "while [ true ]; do echo 'hello'; done", and that loop shows up in htop/top as a bash process, 98% cpu use. I'm trying to understand why the error message I found wouldn't even show up at all, or if the culprit dhclient command was it, why it would only show up as using 4% of the cpu.