[SOLVED] Diagnosing high cpu load, no xorg

If none of the more specific forums is the right place to ask

[SOLVED] Diagnosing high cpu load, no xorg

Postby n_hologram » 2017-05-18 11:10

Solution in final post. System was reproducing, over and over again, an unusual dhclient error message, which decided to run constantly because of a lingering sendmail file. The clue was in htop's "red bars" (kernel threads), which signaled that the issue was probably deeper than a single process. Once that file was removed, the system worked as expected.

My netbook, which functions as a small, unimpressive project "server," is reporting an abnormally high cpu load on the core and virtual core: between 50-60%, and not subsiding. On two-to-five-minute intervals, I hear the little fan whirr for a few seconds and subside. All of that would be fine, if I knew which program was consuming all the resources, but htop reports that only four programs are using cpu, and the values are ~4%, 2%, 1%, and 1%. Rebooting changed nothing. I'm looking for suggestions for narrowing the source of this issue.
Last edited by n_hologram on 2017-05-19 00:41, edited 4 times in total.
bester69 wrote:There is nothing to install in linux, from time to time i go to google searching for something fresh to install in linux, but, there is nothing
n_hologram
 
Posts: 192
Joined: 2013-06-16 00:10

Re: Diagnosing high cpu load, no xorg

Postby phenest » 2017-05-18 11:35

What program is showing the high cpu load?
NEC Spirit 550 P4 3.8GHz HT - 2GB RAM - nVidia 7600GT - Pioneer BDR-209DBK
ASUS Sabertooth P67 i7 3770K - 32GB RAM - 2x nVidia 660GTX SLI'd
User avatar
phenest
 
Posts: 1571
Joined: 2010-03-09 09:38
Location: The Matrix

Re: Diagnosing high cpu load, no xorg

Postby n_hologram » 2017-05-18 11:47

phenest wrote:What program is showing the high cpu load?

dhclient is the 4%-er
bester69 wrote:There is nothing to install in linux, from time to time i go to google searching for something fresh to install in linux, but, there is nothing
n_hologram
 
Posts: 192
Joined: 2013-06-16 00:10

Re: Diagnosing high cpu load, no xorg

Postby phenest » 2017-05-18 12:05

I'll rephrase: What program did you use to measure the high cpu load?
NEC Spirit 550 P4 3.8GHz HT - 2GB RAM - nVidia 7600GT - Pioneer BDR-209DBK
ASUS Sabertooth P67 i7 3770K - 32GB RAM - 2x nVidia 660GTX SLI'd
User avatar
phenest
 
Posts: 1571
Joined: 2010-03-09 09:38
Location: The Matrix

Re: Diagnosing high cpu load, no xorg

Postby n_hologram » 2017-05-18 12:09

phenest wrote:I'll rephrase: What program did you use to measure the high cpu load?

htop

EDIT: I also checked with top, same results: http://stackoverflow.com/questions/9229 ... ux#9229692
Code: Select all
top -bn1 | grep "Cpu(s)" | \
           sed "s/.*, *\([0-9.]*\)%* id.*/\1/" | \
           awk '{print 100 - $1"%"}
bester69 wrote:There is nothing to install in linux, from time to time i go to google searching for something fresh to install in linux, but, there is nothing
n_hologram
 
Posts: 192
Joined: 2013-06-16 00:10

Re: Diagnosing high cpu load, no xorg

Postby orythem27 » 2017-05-18 12:46

n_hologram wrote:between 50-60%, ...... only four programs are using cpu, and the values are ~4%, 2%, 1%, and 1%.

Use `htop` to verify that if it's indeed some kernel threads that is actually acting up. Look for the red portion in the CPU usage bar. See this page for detail.

For further investigation you can use the following tools to unveil more details.
User avatar
orythem27
 
Posts: 172
Joined: 2017-05-11 07:59
Location: P.R. China

Re: Diagnosing high cpu load, no xorg

Postby wizard10000 » 2017-05-18 13:10

orythem27 wrote:...For further investigation you can use the following tools to unveil more details.


I'm gonna add glances to your list. Very nice ncurses interface and it's my go-to for monitoring remote systems because in addition to processor and memory usage it shows disk and network I/O in realtime.
we see things not as they are, but as we are.
-- anais nin
User avatar
wizard10000
 
Posts: 1086
Joined: 2011-05-09 20:02
Location: midwestern us

Re: Diagnosing high cpu load, no xorg

Postby phenest » 2017-05-18 13:38

n_hologram wrote:My netbook, ... is reporting an abnormally high cpu load on the core and virtual core: between 50-60%, and not subsiding.

Reported by what?
n_hologram wrote:but htop reports that only four programs are using cpu, and the values are ~4%, 2%, 1%, and 1%.

I don't understand. If htop is showing low CPU usage < 10% combined, how do you know there is a 50-60% load?
NEC Spirit 550 P4 3.8GHz HT - 2GB RAM - nVidia 7600GT - Pioneer BDR-209DBK
ASUS Sabertooth P67 i7 3770K - 32GB RAM - 2x nVidia 660GTX SLI'd
User avatar
phenest
 
Posts: 1571
Joined: 2010-03-09 09:38
Location: The Matrix

Re: Diagnosing high cpu load, no xorg

Postby phenest » 2017-05-18 15:15

@ OP:
Please reply to this thread rather than PM'ing me. Your replies are helpful to others, not just me.

So you're saying that htop is reporting a total CPU load of 50-60%, but only 4 programs show any usage and they are < 10% combined. Yes?
NEC Spirit 550 P4 3.8GHz HT - 2GB RAM - nVidia 7600GT - Pioneer BDR-209DBK
ASUS Sabertooth P67 i7 3770K - 32GB RAM - 2x nVidia 660GTX SLI'd
User avatar
phenest
 
Posts: 1571
Joined: 2010-03-09 09:38
Location: The Matrix

Re: Diagnosing high cpu load, no xorg

Postby n_hologram » 2017-05-18 15:21

phenest wrote:@ OP:
Please reply to this thread rather than PM'ing me. Your replies are helpful to others, not just me.

Sure thing:
I had already said I used htop, along with a "top" command, and they both yielded the same results. I thought "reported" would be an appropriate verb.

I also do not understand why it's doing this, but I don't know much about kernel threads because I've never had a reason to investigate them before. I intend to update the post once I get a chance to look into that.

It's also a pain because, and I'm not sure if this is related or not, but any sort of local or external access to that system (ssh, mpd, etc.) will just stop working until I reboot (after which it works as expected for awhile, and then doesn't), so I have to be physically present at that computer to check anything until that gets resolved. Which is annoying.

So you're saying that htop is reporting a total CPU load of 50-60%, but only 4 programs show any usage and they are < 10% combined. Yes?

Yes.

EDIT: Screenshot for your viewing pleasure.
bester69 wrote:There is nothing to install in linux, from time to time i go to google searching for something fresh to install in linux, but, there is nothing
n_hologram
 
Posts: 192
Joined: 2013-06-16 00:10

Re: Diagnosing high cpu load, no xorg

Postby n_hologram » 2017-05-19 00:23

Alright, you guys are gonna love this. I have kismet installed and set-up in server mode, using the wireless card in monitor mode. I'm using sysv, and /etc/rc.local is configured as so:
Code: Select all
# Start the wifi card in monitor mode for Kismet reasons
airmon-ng start wlan0
/usr/bin/sudo -u pi "/usr/bin/kismet_server" -n --daemonize -p /home/pi/kismet_server

At boot, I watched the startup messages (I don't know the proper term for those), and noticed that, when it runs this command, it complains that dhclient and dhclient script might interfere -- which is strange, since when I run the airmon command as root (sudo su) within a console, it doesn't produce that error. Also, "~# killall -9 dhclient" solved the original issue, albeit superficially. So, I think, I'll just comment-out those two lines from rc.local. Sure enough, after a reboot, my cpu usage runs exactly as expected.

Now, fully intrigued, I uncomment those lines, reboot, and double-check htop. That screenshot I sent in my previous post, it shows a dhclient command constantly running, around 4%, but without those lines in rc.local it doesn't show any dhclient at all, and neither do any of my other Debian systems. I suddenly have an epiphany: why is it running the entire time? So, I try a simple "~# dhclient eth0", and in a normal system, it would show a simple, one time "RTNETLINK answers: File Exists". However, when I run the same command in my server, it gives me two messages, ad infinitum:
Code: Select all
...
/sbin/dhclient-script: 28: .: Can't open /usr/share/sendmail/dynamic
RTNETLINK answers: File exists
/sbin/dhclient-script: 28: .: Can't open /usr/share/sendmail/dynamic
RTNETLINK answers: File exists
...

until I CTRL+C. So, it appears the issue was that I had daemonized an error message, and the little atom processor wasn't happy with that.

This was, however, a nice tidbit of information. I ran across a thread which articulated solutions to the same issue. Sendmail was already removed through apt-get, but I guess a lingering file interfered with dhclient's normal behavior, so I went ahead and removed it:
Code: Select all
rm /etc/dhcp/dhclient-exit-hooks.d/sendmail

And now everything is up to normal. Thanks to everyone for helping out, however, I'm still curious as to why htop didn't show anything abnormal in the process list; and if that's normal to not show, I still wonder if there would have been a better way to narrow that without crapshooting through programs and bootscreens. Ah, well...
bester69 wrote:There is nothing to install in linux, from time to time i go to google searching for something fresh to install in linux, but, there is nothing
n_hologram
 
Posts: 192
Joined: 2013-06-16 00:10

Re: Diagnosing high cpu load, no xorg

Postby phenest » 2017-05-19 16:30

n_hologram wrote:however, I'm still curious as to why htop didn't show anything abnormal in the process list; and if that's normal to not show

Maybe it's because htop only refreshes about once a second and it misses it.
NEC Spirit 550 P4 3.8GHz HT - 2GB RAM - nVidia 7600GT - Pioneer BDR-209DBK
ASUS Sabertooth P67 i7 3770K - 32GB RAM - 2x nVidia 660GTX SLI'd
User avatar
phenest
 
Posts: 1571
Joined: 2010-03-09 09:38
Location: The Matrix

Re: Diagnosing high cpu load, no xorg

Postby n_hologram » 2017-05-19 17:21

phenest wrote:
n_hologram wrote:however, I'm still curious as to why htop didn't show anything abnormal in the process list; and if that's normal to not show

Maybe it's because htop only refreshes about once a second and it misses it.

It would update with the same frequency as "while [ true ]; do echo 'hello'; done", and that loop shows up in htop/top as a bash process, 98% cpu use. I'm trying to understand why the error message I found wouldn't even show up at all, or if the culprit dhclient command was it, why it would only show up as using 4% of the cpu.
bester69 wrote:There is nothing to install in linux, from time to time i go to google searching for something fresh to install in linux, but, there is nothing
n_hologram
 
Posts: 192
Joined: 2013-06-16 00:10


Return to General Questions

Who is online

Users browsing this forum: No registered users and 5 guests

fashionable