RAID superblocks being wiped on reboot

If none of the more specific forums is the right place to ask

RAID superblocks being wiped on reboot

Postby PlunderingPirate9000 » 2020-10-21 11:43

I've been experiencing a really weird issue with mdadm, raid 1 arrays.

I create an array like `sudo mdadm --create --verbose /dev/md0 --level=1 --raid-devices=2 /dev/sda /dev/sdb`

The array will finish syncing, and all partitions will be there from the previous raid configuration.

I can mount the paritions, read/write to them etc.

At some point in the past I would have done `sudo mkfs.ext4 -F /dev/md0`

However after re-creating the array with the same devices, I do not repeat this step since the paritions/data are already there on the now synced drives.

I then do `sudo mdadm --detail --scan | sudo tee -a /etc/mdadm/mdadm.conf`.

`sudo update-initramfs -u`

`echo '/dev/md0 /mnt/md0 ext4 defaults,nofail,discard 0 0' | sudo tee -a /etc/fstab`

and also remove previous entiries from /etc/fstab and mdadm.conf

Then, when I reboot - the array is gone.

`cat /proc/mdstat` is blank.

I can repeat the raid array creation as described above and everything comes back.

I have absolutly no clue why this is happening. The only difference between this system and other systems I have used in the past is that the drives in this system are connected via a PCI-e SAS card.
PlunderingPirate9000
 
Posts: 31
Joined: 2017-11-30 15:17

Re: RAID superblocks being wiped on reboot

Postby LE_746F6D617A7A69 » 2020-10-21 15:03

First, You should read the manual for mdadm, and not just "quick setup" instructions from some outdated article on the web. The manual is excellent and it contains many valuable informations ;)

Your primary problem is this command:
PlunderingPirate9000 wrote:`echo '/dev/md0 /mnt/md0 ext4 defaults,nofail,discard 0 0' | sudo tee -a /etc/fstab`
The md0 device exists only until reboot - then it becomes md127 (by default -> if You have only one array in the system, and if You won't specify the array name in mdadm.conf)

Fundamental rule: *never* use device paths in fstab - use UUIDs

The UUID in the /etc/fstab is the filesystem UUID, not the md-array UUID.
The UUID in /etc/mdadm/mdadm.conf is the md-array UUID and not the filesystem UUID.

ext4 UUID can be checked with tune2fs -l
md-array UUID is listed by mdadm -D
Bill Gates: "(...) In my case, I went to the garbage cans at the Computer Science Center and I fished out listings of their operating system."
The_full_story and Nothing_have_changed
LE_746F6D617A7A69
 
Posts: 414
Joined: 2020-05-03 14:16

Re: RAID superblocks being wiped on reboot

Postby p.H » 2020-10-21 18:01

How do you know that RAID superblocks are wiped ? You did not show any direct evidence such as the output of
Code: Select all
blkid /dev/sd*
file -s /dev/sd*
wipefs /dev/sda
wipefs /dev/sdb
mdadm --detail --scan --verbose
mdadm --examine /dev/sd*

and so on.
PlunderingPirate9000 wrote:all partitions will be there from the previous raid configuration.

What partitions ? Is this a partitioned array ?
What previous RAID configuration ?

Note : using whole drives instead of partitions as RAID members is a bad idea IMO. It may conflict with a partition table.
p.H
 
Posts: 1521
Joined: 2017-09-17 07:12

Re: RAID superblocks being wiped on reboot

Postby p.H » 2020-10-21 19:34

LE_746F6D617A7A69 wrote:The md0 device exists only until reboot - then it becomes md127 (by default -> if You have only one array in the system, and if You won't specify the array name in mdadm.conf)

In my experience, an md device is assigned a high number like md127 only if it is not defined by mdadm.conf neither in /etc and in the initramfs, and if it has a different hostname from the current system.

LE_746F6D617A7A69 wrote:*never* use device paths in fstab

Except when the device path is predictible and persistent, such as
/dev/mapper/*
/dev/disk/by-id/*
/dev/disk/by-path/*

Anyway, a wrong device path in /etc/fstab is not the cause of the md device not being assembled and present in /proc/mdstat.
p.H
 
Posts: 1521
Joined: 2017-09-17 07:12

Re: RAID superblocks being wiped on reboot

Postby LE_746F6D617A7A69 » 2020-10-21 20:49

p.H wrote:Except when the device path is predictible and persistent, such as
p.H wrote:/dev/disk/by-id/*
Yes, in most cases this will work, but it's the UUID which is a standard designed to identify devices/logical volumes.

p.H wrote:/dev/disk/by-path/*
This will *not* work if You'll add/remove some expansion card(s) -> the PCI bus numbers can change.

p.H wrote:Anyway, a wrong device path in /etc/fstab is not the cause of the md device not being assembled and present in /proc/mdstat.
Yes it can be, f.e. if conflicting array declarations are found.

p.H wrote:Note : using whole drives instead of partitions as RAID members is a bad idea IMO. It may conflict with a partition table.
We already had a short "discussion" about this topic, not so long time ago - in case of HW and software BIOS-based Raid arrays, the partition tables are created on top of partition-less raid devices, so there's no single problem with creating partition-less md arrays.
One of the reasons for using partition-based md raid arrays, is to have EFI-bootable arrays (i.e. for booting the system) and the another reason could be f.e. to have smaller volumes, which are easier to manage.
Bill Gates: "(...) In my case, I went to the garbage cans at the Computer Science Center and I fished out listings of their operating system."
The_full_story and Nothing_have_changed
LE_746F6D617A7A69
 
Posts: 414
Joined: 2020-05-03 14:16

Re: RAID superblocks being wiped on reboot

Postby PlunderingPirate9000 » 2020-10-22 14:14

p.H wrote:How do you know that RAID superblocks are wiped ? You did not show any direct evidence such as the output of
Code: Select all
blkid /dev/sd*
file -s /dev/sd*
wipefs /dev/sda
wipefs /dev/sdb
mdadm --detail --scan --verbose
mdadm --examine /dev/sd*

and so on.



I don't know what wipefs is. Does it install with `sudo apt install wipefs`?

Here is the output of blkid:

Code: Select all
/dev/sdb: PTUUID="a2a2e184-6130-4691-9fb7-fdcdeb2c19e9" PTTYPE="gpt"
/dev/sdb1: UUID="957ef167-565e-4afc-ac01-23e8e6094294" TYPE="ext4" PARTUUID="1af5ef8f-9f67-48db-b625-9a1a516467a0"
/dev/sdc: PTUUID="4b21720b-0277-4315-aa00-288cda4ec8f8" PTTYPE="gpt"
/dev/sdc1: UUID="917a60a8-82fa-49b5-ac0b-82a3b021271d" TYPE="ext4" PARTUUID="45e77d7c-fecc-41fa-ab96-0d49268c5d71"
/dev/sdd: PTUUID="576ecf35-a453-4046-b942-b371eed85c6c" PTTYPE="gpt"
/dev/sdd1: UUID="c1ecc5ce-9ac2-4aa6-b373-66db2d586f9d" TYPE="ext4" PARTUUID="9244a088-dae2-409c-b7a0-80c89d0d03ff"
/dev/sde: PTUUID="2df74fa8-173a-44a1-bde1-96f45060bce8" PTTYPE="gpt"
/dev/sde1: UUID="73f626e3-3fff-4e0e-940a-cc3bbbb2df04" TYPE="ext4" PARTUUID="a8513384-d9dd-4577-9759-0a97ce5a8668"
/dev/sdf: PTUUID="0735d711-b073-403b-a93f-cb03b418e47c" PTTYPE="gpt"
/dev/sdf1: UUID="97c2375a-57c6-47b0-a454-80d27b0b66b1" TYPE="ext4" PARTUUID="aae2c35f-181c-4dca-9ad6-fc551c4229b7"
/dev/sdg: PTUUID="e65148e2-1cb1-4768-8fbb-a78e07c61acb" PTTYPE="gpt"


All of these should be members of various raid 1 arrays. sda is missing because I blanked the drive.

p.H wrote:
PlunderingPirate9000 wrote:all partitions will be there from the previous raid configuration.

What partitions ? Is this a partitioned array ?
What previous RAID configuration ?

Note : using whole drives instead of partitions as RAID members is a bad idea IMO. It may conflict with a partition table.

[/quote]

I guess the best thing to do is provide you a link to the article I was following.

https://www.digitalocean.com/community/tutorials/how-to-create-raid-arrays-with-mdadm-on-debian-9#creating-a-raid-1-array

I mean perhaps the info presented in the above link is wrong. I have no idea.
PlunderingPirate9000
 
Posts: 31
Joined: 2017-11-30 15:17

Re: RAID superblocks being wiped on reboot

Postby PlunderingPirate9000 » 2020-10-22 14:15

Forgot to add:

Code: Select all
sudo mdadm --detail --verbose --scan
sudo mdadm --examine
etc


are all blank, or spit out "no superblock"
PlunderingPirate9000
 
Posts: 31
Joined: 2017-11-30 15:17

Re: RAID superblocks being wiped on reboot

Postby PlunderingPirate9000 » 2020-10-22 14:18

p.H wrote:
LE_746F6D617A7A69 wrote:The md0 device exists only until reboot - then it becomes md127 (by default -> if You have only one array in the system, and if You won't specify the array name in mdadm.conf)

In my experience, an md device is assigned a high number like md127 only if it is not defined by mdadm.conf neither in /etc and in the initramfs, and if it has a different hostname from the current system.



It does not become md127. It does not appear as md anything. It does not exist after reboot. This should be what happens, I agree, but it does not. I have no idea why. Sorry I cannot be more useful than this - I really am no expert on this subject.

p.H wrote:
LE_746F6D617A7A69 wrote:*never* use device paths in fstab

Except when the device path is predictible and persistent, such as
/dev/mapper/*
/dev/disk/by-id/*
/dev/disk/by-path/*

Anyway, a wrong device path in /etc/fstab is not the cause of the md device not being assembled and present in /proc/mdstat.


I also thought this - you might be correct. Actually since I am using a SAS pci-e card, it appears to be the case that /dev/sda is always /dev/sda. etc. I thought they should be randomized on reboot but apparently this is not the case.
PlunderingPirate9000
 
Posts: 31
Joined: 2017-11-30 15:17

Re: RAID superblocks being wiped on reboot

Postby PlunderingPirate9000 » 2020-10-22 14:20

LE_746F6D617A7A69 wrote:
p.H wrote:Note : using whole drives instead of partitions as RAID members is a bad idea IMO. It may conflict with a partition table.
We already had a short "discussion" about this topic, not so long time ago - in case of HW and software BIOS-based Raid arrays, the partition tables are created on top of partition-less raid devices, so there's no single problem with creating partition-less md arrays.
One of the reasons for using partition-based md raid arrays, is to have EFI-bootable arrays (i.e. for booting the system) and the another reason could be f.e. to have smaller volumes, which are easier to manage.


This went way over my head.
PlunderingPirate9000
 
Posts: 31
Joined: 2017-11-30 15:17

Re: RAID superblocks being wiped on reboot

Postby PlunderingPirate9000 » 2020-10-22 14:22

In summary, still after reading these replies, I have no idea what might be wrong or what I should be doing.

Where is the source of information for how to create a RAID 1 array with mdadm - because I have done this before experiencing no issues. I don't know what is different this time.
PlunderingPirate9000
 
Posts: 31
Joined: 2017-11-30 15:17

Re: RAID superblocks being wiped on reboot

Postby p.H » 2020-10-22 15:34

wipefs is a standard tool which is installed by default (in util-linux or coreutils, not sure). Without option, is prints filesystem/RAID/LVM... signatures found on the target device.

blkid shows that the drives have a GPT partition table and a partition. If you use the whole drives as RAID devices, you should wipe the partition tables first because the GPT table and the RAID superblock overlap each other. If you keep the partition tables, you should use the partitions as a RAID device.

No time to read the article now, will do it later.
p.H
 
Posts: 1521
Joined: 2017-09-17 07:12

Re: RAID superblocks being wiped on reboot

Postby PlunderingPirate9000 » 2020-10-22 19:35

p.H wrote:wipefs is a standard tool which is installed by default (in util-linux or coreutils, not sure). Without option, is prints filesystem/RAID/LVM... signatures found on the target device.

blkid shows that the drives have a GPT partition table and a partition. If you use the whole drives as RAID devices, you should wipe the partition tables first because the GPT table and the RAID superblock overlap each other. If you keep the partition tables, you should use the partitions as a RAID device.

No time to read the article now, will do it later.


Ok thanks for this - it sounds like a step in the right direction.

What should I do to wipe the parition information from the drives first, to ensure there is no overlap.

I'm very confused you see because a lot of online information doesn't mention anything about this... For example please see also https://raid.wiki.kernel.org/index.php/RAID_setup#

Then of course the latter part of the question is, once I have two blanked drives, and make them into a raid device, how do I then create a parition without nuking the raid information.

I'm pretty convinced this must be what is happening - thanks for the info
PlunderingPirate9000
 
Posts: 31
Joined: 2017-11-30 15:17

Re: RAID superblocks being wiped on reboot

Postby p.H » 2020-10-22 20:03

PlunderingPirate9000 wrote:What should I do to wipe the parition information from the drives first, to ensure there is no overlap

You can use wipefs with the -a option. "man wipefs" for details to erase all metadata signatures on a device.
Or write zeroes on the first MB of the drive with dd.

PlunderingPirate9000 wrote:once I have two blanked drives, and make them into a raid device, how do I then create a parition without nuking the raid information.

Why do you need to create a partition ? You can use the RAID array /dev/mdN directly as a filesystem. Just format it with mkfs.<fstype> and mount it.
If you need to divide the RAID array into several areas you can use LVM over the RAID array. This was the traditional method when RAID arrays were not partitionable. Or you can create a partition table and partition on the RAID array /dev/mdN (NOT the RAID member drives /dev/sdX).
p.H
 
Posts: 1521
Joined: 2017-09-17 07:12

Re: RAID superblocks being wiped on reboot

Postby PlunderingPirate9000 » 2020-10-22 22:29

p.H wrote:
PlunderingPirate9000 wrote:What should I do to wipe the parition information from the drives first, to ensure there is no overlap

You can use wipefs with the -a option. "man wipefs" for details to erase all metadata signatures on a device.
Or write zeroes on the first MB of the drive with dd.


Yeah again - not trying to be obtuse here but how do I install this? It's not installed by default. [Edit: it is installed by default - I was forgetting to use sudo - duh]

p.H wrote:
PlunderingPirate9000 wrote:once I have two blanked drives, and make them into a raid device, how do I then create a parition without nuking the raid information.

Why do you need to create a partition ? You can use the RAID array /dev/mdN directly as a filesystem. Just format it with mkfs.<fstype> and mount it.
If you need to divide the RAID array into several areas you can use LVM over the RAID array. This was the traditional method when RAID arrays were not partitionable. Or you can create a partition table and partition on the RAID array /dev/mdN (NOT the RAID member drives /dev/sdX).

[/quote]

Ok, I'm confused, because this is what I did before and then after restarting the raid/partition/filesystem was gone.

By the way I think what you call a partition is not what I mean by a partition, given the context of your reply.

In summary you are providing me with the same instructions as I linked in that Digital Ocean link - and this did not work - for some reason I am not sure why.

So yeah - if you give me a list of commands to run, I can run them and report back on my findings, but afaik you're giving me the same instructions I already followed.
PlunderingPirate9000
 
Posts: 31
Joined: 2017-11-30 15:17

Re: RAID superblocks being wiped on reboot

Postby p.H » 2020-10-23 06:37

If you want to use whole drives /dev/sdX and /dev/sdY (not partitions) as RAID members for RAID 1 array /dev/mdN :
Code: Select all
# unmount partitions on the RAID drives (if they are mounted)
umount /dev/sdX1
umount /dev/sdY1

# erase signatures on the RAID drives and their partitions
wipefs -a /dev/sdX1
wipefs -a /dev/sdX
wipefs -a /dev/sdY1
wipefs -a /dev/sdY

# create RAID array
mdadm --create /dev/mdN --level=1 --raid-devices=2 /dev/sdX /dev/sdY

# erase signatures on the RAID array
wipefs -a /dev/mdN

# format RAID array as ext4
mkfs.ext4 /dev/mdN
p.H
 
Posts: 1521
Joined: 2017-09-17 07:12

Next

Return to General Questions

Who is online

Users browsing this forum: No registered users and 15 guests

fashionable