Scheduled Maintenance: We are aware of an issue with Google, AOL, and Yahoo services as email providers which are blocking new registrations. We are trying to fix the issue and we have several internal and external support tickets in process to resolve the issue. Please see: viewtopic.php?t=158230

 

 

 

Debian Server - Failed to boot due to xfs filesystem error.

If none of the specific sub-forums seem right for your thread, ask here.
Post Reply
Message
Author
lindsayrg1
Posts: 5
Joined: 2012-03-03 05:53

Debian Server - Failed to boot due to xfs filesystem error.

#1 Post by lindsayrg1 »

Hey, recently I decided to build a NAS server and install Openmediavault which is similar to Freenas but based on Debian. Anyway I'm afraid members of my family have been turning the server off at night via the power button (trying to save power) which has obviously caused filesystem issues. I have a 12tb raid 5 array with just over 8tb usable. Just before it stopped around 4tb of data was on it so I would like to try to get the server running and if that isnt possible attempt to recover some / all of the files.

When I boot the server I get the following message

Code: Select all

An automatic file system check (fsck) of the root filesystem failed. A manual fsck must be performed, then the system restarted. The fsck should be performed in maintenance mode with the root filesystem mounted in read-only mode...failed!A log is being saved in /var/log/fsck/checkfs if that location is writable. Please repair the file system manually. A maintenance shell will now be started. CONTROL-D will terminate this shell and resume system boot..Give root password for maintenance(or type Control-D to continue)
Log states

Code: Select all

Log of fsck -C -R -A -a
Sat Mar  3 15:43:31 2012

fsck from util-linux-ng 2.17.2
/sbin/fsck.xfs: UUID=6c763b53-3f3e-4f31-9d3c-3a0987821189 does not exist
fsck died with exit status 8

Sat Mar  3 15:43:31 2012:
Typing root password does nothing and entering Control-D just starts the server process / daemons etc allowing me to continue to access the web interface but the smb share is not working I assume due to the filesystem errors.

I honesly have no idea what I am doing so I have tried to searching for similar issues in this forum but most of the fsck errors don't seem to be for exit status 8. The other thread I found I did state the first steps to recover the filesystem are to un-mount all disks/partitions and then run a filesystem check from a live cd. (http://forums.debian.net/viewtopic.php? ... +8#p399882)

According to the web interface the fours hard drives are as follows
/dev/sda
/dev/sdb
/dev/sdc
/dev/sdd

They are listed as mounted and as an xfs filesystem but the status is listed as missing.
Do I just unmount all by typing in the following

Code: Select all

umount -a
and then run the following for each drive or is there a different /dev/xxx for an array?

Code: Select all

e2fsck -y -f -v /dev/hda1
Also as I said people recommend doing this from a live cd. When I boot from the openmediavault/debian installed on my usb it gives two boot options (default and recovery). Would not this usb booting from recovery be suitable or is a particular live cd recommended for filesystem recovery?
Finally I am sorry if it seems I jumped into the deep end but I didn't think I would have to learn much about debian and would just be able to rely on the web interface aspect of the server. Thank you if you managed to read this all the way through.

pendrachken
Posts: 1394
Joined: 2007-03-04 21:10
Location: U.S.A. - WI.

Re: Debian Server - Failed to boot due to xfs filesystem err

#2 Post by pendrachken »

Search around for how to manually restart your RAID array. Individual disks ( generally ) won't have recognizable file systems if they are part of an array unless the whole array is mounted.
fortune -o
Your love life will be... interesting.
:twisted: How did it know?

The U.S. uses the metric system too, we have tenths, hundredths and thousandths of inches :-P

brian.va
Posts: 138
Joined: 2010-12-11 17:44

Re: Debian Server - Failed to boot due to xfs filesystem err

#3 Post by brian.va »

lindsayrg1 wrote:s there a different /dev/xxx for an array?.
generally they are referenced by /dev/mdX

try

Code: Select all

cat /proc/mdstat
and see what comes back, you should at least see what it is referenced as, and could try mounting it and see if it does anything

lindsayrg1
Posts: 5
Joined: 2012-03-03 05:53

Re: Debian Server - Failed to boot due to xfs filesystem err

#4 Post by lindsayrg1 »

I got the following.

Code: Select all

root@Openmediavault:~# cat /proc/mdstat
Personalities :
md127 : inactive sda[1](S) sdd[3](S) sdb[2](S)
      8790796680 blocks super 1.2
unused devices: <none>
So sda, sdb and sdd are inacative? What about sdc, it is showing up in the web interface?
Image

I tried mounting it but I'm probably doing it the wrong way here.

Code: Select all

root@Openmediavault:~# mount md127
mount: can't find md127 in /etc/fstab or /etc/mtab
Individual disks ( generally ) won't have recognizable file systems if they are part of an array unless the whole array is mounted.
Yes that is what I guessed and is why I asked. So essential I have to get all the disks active again and then run a raid xfs file system check somehow?
Thanks to both of you for your help so far!

pendrachken
Posts: 1394
Joined: 2007-03-04 21:10
Location: U.S.A. - WI.

Re: Debian Server - Failed to boot due to xfs filesystem err

#5 Post by pendrachken »

You most likely will have to assemble the array manually from the CLI. See the mdadm man page or http://linux.die.net/man/8/mdadm from a different computer. You may need to use the force flags, but only try that if assembling the array doesn't work without it. Once you have the array assembled you should be able to run the filesystem check on the array - they generally start with md0.
fortune -o
Your love life will be... interesting.
:twisted: How did it know?

The U.S. uses the metric system too, we have tenths, hundredths and thousandths of inches :-P

lindsayrg1
Posts: 5
Joined: 2012-03-03 05:53

Re: Debian Server - Failed to boot due to xfs filesystem err

#6 Post by lindsayrg1 »

ok I read the intro which talks about the several modes of operation of mdadm and the uses of assemble mode option.

So I tried the following

Code: Select all

mdadm --assemble --scan  (does nothing just goes to next line / no response)
mdadm --assemble md0 (md0 not identified in config file)
I try to get identity of the array with the following but no go.

Code: Select all

root@Openmediavault:~# /etc/mdadm/mdadm.conf
-bash: /etc/mdadm/mdadm.conf: Permission denied
root@Openmediavault:~# /etc/mdadm.conf
-bash: /etc/mdadm.conf: No such file or directory
The page says the following but I'm not sure what to type in to get identity? mdadm --assemble --config does not seem to do anything.

Code: Select all

If --scan is not given, then the config file will only be used to find the identity of md arrays. 
Again I appreciate your help. I'll keep reading the article and hope that I understand it but it's all a little much as I have never used a CLI before.

pendrachken
Posts: 1394
Joined: 2007-03-04 21:10
Location: U.S.A. - WI.

Re: Debian Server - Failed to boot due to xfs filesystem err

#7 Post by pendrachken »

Use nano to edit mdadm.conf, nano is a CLI text editor.

Code: Select all

nano /etc/mdadm/mdadm.conf
you should see what the array is called in the conf file.

To exit nano, if you have changed things in the file you are editing press F3 and then enter, after that press F2. If you haven't changed anything just press F2.
fortune -o
Your love life will be... interesting.
:twisted: How did it know?

The U.S. uses the metric system too, we have tenths, hundredths and thousandths of inches :-P

lindsayrg1
Posts: 5
Joined: 2012-03-03 05:53

Re: Debian Server - Failed to boot due to xfs filesystem err

#8 Post by lindsayrg1 »

Wow thanks for the quick response. I have been puzzeling away and didn't even notice. I followed your recommendation and I believe the relevant part from the text is name=openmediavault:Raid5Array UUID=34eb0de1:60c528fa:b6c6d68d:c3616955

Code: Select all

 # mdadm.conf
#
# Please refer to mdadm.conf(5) for information about this file.
#

# by default, scan all partitions (/proc/partitions) for MD superblocks.
# alternatively, specify devices to scan, using wildcards if desired.
# Note, if no DEVICE line is present, then "DEVICE partitions" is assumed.
# To avoid the auto-assembly of RAID devices a pattern that CAN'T match is
# used if no RAID devices are configured.
DEVICE partitions

# auto-create devices with Debian standard permissions
CREATE owner=root group=disk mode=0660 auto=yes

# automatically tag new arrays as belonging to the local system
HOMEHOST <system>

# definitions of existing MD arrays
[b]ARRAY /dev/md/openmediavault:Raid5Array metadata=1.2 name=openmediavault:Raid5Array UUID=34eb0de1:60c528fa:b6c6d68d:c3616955[/b]
So I entered what is below and then nothing happened just got a new line?

Code: Select all

root@Openmediavault:~# mdadm --assemble openmediavault:Raid5Array
root@Openmediavault:~#

lindsayrg1
Posts: 5
Joined: 2012-03-03 05:53

Re: Debian Server - Failed to boot due to xfs filesystem err

#9 Post by lindsayrg1 »

Hey its been quite awhile since my original post been busy with other stuff but I still have yet to get this fixed.

I have more time now so I am willing to investigate further into solutions if people can advice what I should do/try/read up on.

I basically got stuck last time and then other stuff happened so it just sat around. The data on it is not really important but I would like to try and get it back before i format the drives if possible.
If anyone can advice on what to do or who I can possible pay in australia to have a look and see if they can repair it would be appreciated.


Here is boot image of server attached to tv. Problem is UUID=xxxx does not exist.

Image

pendrachken
Posts: 1394
Joined: 2007-03-04 21:10
Location: U.S.A. - WI.

Re: Debian Server - Failed to boot due to xfs filesystem err

#10 Post by pendrachken »

just let the --assemble --scan run for a long time, it may find the disks.


If that doesn't work...
Try manually re-assembleing the array:

Code: Select all

mdadm --assemble /dev/md0 /dev/sdX1 /dev/sdX1 /dev/sdX1
where /dev/sdX1 is the ALL of the drives ( and partitions ) from the array.
fortune -o
Your love life will be... interesting.
:twisted: How did it know?

The U.S. uses the metric system too, we have tenths, hundredths and thousandths of inches :-P

Post Reply