Hard Drive Excessive Processing [SOLVED]

If none of the more specific forums is the right place to ask

Hard Drive Excessive Processing [SOLVED]

Postby atroposfate » 2017-11-30 05:41

Hi,

I'm at a loss for what is happening with my computer. I run Debian Jessie in text mode and handles all my files and downloads.
It is an AMD A4-6300 3.7GHz with 8Gig RAM an SSD containing the OS and programs. There are also 5.5 TB in a Raid 10 Array that is 4.6 TB full. It should have plenty of horsepower but something is slowing it down.

Over the past month or 2 every 10 minutes on the dot. On the 3's... the hard drive cycles and accesses for a good 5-10 seconds which is an eternity when it hold all the files I stream. It locks everything up until it does whatever it is doing. Even after I restart it is still occurring on the 3's so it isn't 10 minutes after it starts up. This computer has been running since Jessie has been stable and I haven't installed any new programs since it has started acting up. Just standard updates.

I believe I have shutdown every server that is running on this computer and it still continues. I have checked cron and nothing is setup. I have checked 'ps aux' and 'top' when it lags but it doesn't appear to be a program or a CPU overuse issue, which leads me to believe it is a hard-drive issue. I have shutdown mdadm monitor, thinking that might have been the issue but it still persists. My next thought was there was some sort of power management powering it down and then immediately powering it up but I don't believe anything comes default with debian and I install hdparm to shutdown any power management on the drives but that hasn't made a difference either. Although I'm not sure if I can shut down power management on a RAID drive.

Any suggestions would be appreciated or point me at the logs I can look at to see what is running. The only thing I haven't shutdown is mysql database but I looked at the databases and they are smaller than 100MB.

Cheers
Last edited by atroposfate on 2017-11-30 23:42, edited 1 time in total.
atroposfate
 
Posts: 6
Joined: 2014-10-25 02:50

Re: Hard Drive Excessive Processing

Postby dilberts_left_nut » 2017-11-30 06:31

Anything in dmesg when it happens?
Maybe install iotop and see what process is involved.
AdrianTM wrote:There's no hacker in my grandma...
User avatar
dilberts_left_nut
 
Posts: 4706
Joined: 2009-10-05 07:54
Location: enzed

Re: Hard Drive Excessive Processing

Postby atroposfate » 2017-11-30 23:42

Code: Select all
[Thu Nov 30 17:19:03 2017] ata5.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6 frozen
[Thu Nov 30 17:19:03 2017] ata5.00: failed command: FLUSH CACHE EXT
[Thu Nov 30 17:19:03 2017] ata5.00: cmd ea/00:00:00:00:00/00:00:00:00:00/a0 tag 0
         res 40/00:01:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
[Thu Nov 30 17:19:03 2017] ata5.00: status: { DRDY }
[Thu Nov 30 17:19:08 2017] ata5: link is slow to respond, please be patient (ready=0)
[Thu Nov 30 17:19:08 2017] ata5: soft resetting link
[Thu Nov 30 17:19:09 2017] ata5.00: configured for UDMA/25
[Thu Nov 30 17:19:09 2017] ata5.00: retrying FLUSH 0xea Emask 0x4
[Thu Nov 30 17:19:09 2017] ata5.00: device reported invalid CHS sector 0
[Thu Nov 30 17:19:09 2017] ata5: EH complete


I get this at the exact time... so it looks like the hard drive might be failing, odd that mdadm didn't catch it as it is setup to monitor and scan.
One of the suggestions I found was to look into the cables. I took all the cables and reinstalled them. So far it has prevented the freezing and crashing but the error still occurs so I may have just bought some time. Stupid cables, good news is it has been 30 minutes without freezing and it has froze every 10 minutes for the past 2 months without relief.
I guess I'll buy another hard-drive and wait for the inevitable death. Hopefully it doesn't start freezing again until it official dies.

I'll list it as solved but if anyone has any other suggestions for the above error that prevents me from buying another hard-drive I'd be open to it.
atroposfate
 
Posts: 6
Joined: 2014-10-25 02:50

Re: Hard Drive Excessive Processing [SOLVED]

Postby dilberts_left_nut » 2017-12-01 01:40

The only times I have seen those errors it has been disk in a death spiral.
Others have reported same with a cable fault too.
AdrianTM wrote:There's no hacker in my grandma...
User avatar
dilberts_left_nut
 
Posts: 4706
Joined: 2009-10-05 07:54
Location: enzed

Re: Hard Drive Excessive Processing [SOLVED]

Postby pendrachken » 2017-12-01 17:29

As stated before, it could be a cable going bad.

It could also be the drive dying, and the way to check that would be to check the S.M.A.R.T. status of the drive, or run the short and long tests. This is what I would look at first, specifically to see the highfly writes, reallocated sector count, and pending / un-correctable write error counts.

Especially with the reallocated sectors, once you start hitting 500-800 you order a new disk right away, if you start seeing ~2000-3000 it's time to slam that disk into read only and back everything up while praying to the computer gods that you can pull everything off the drive before it dies.



Side note - RAID won't save you from shitty drives failing most of the time. If a drive records garbage / corrupts itself, the garbage or corruption has a good chance of being mirrored in the array. This is where offline backups will save your butt.
fortune -o
Your love life will be... interesting.
:twisted: How did it know?

The U.S. uses the metric system too, we have tenths, hundredths and thousandths of inches :-P
pendrachken
 
Posts: 1291
Joined: 2007-03-04 21:10
Location: U.S.A. - WI.


Return to General Questions

Who is online

Users browsing this forum: dx486 and 4 guests

fashionable