RAID anybody?

[HyperUniverse]

Hi,

Anybody knows about RAID?

Would it be better to have a RAID1, or two independent drives that I will manually copy data from one to another?

Thanks.

[CwF]

I had raid always in the early days, on DOS!, then NT, built it up to 80 pin backplanes with 13 drives...
Things change. There is no reason for raid on modern hardware. KISS
Do the spare drive thing, they're cheap. I still fill the ports with devices, but....

[debbieanne]

If you have a lot of data that changes daily or more often, a RAID mirror is a good idea and costs no more in storage space than a dd or rsync daily backup. There are several tutorials out there that do a decent job of describing how to install mdadm and setup a new RAID mirror on an existing install where your OS/swap/uefi partitions live on one device (such as a fast SSD) and your data lives on your new RAID setup. If your data doesn't change often and you don't mind manual backups, a USB backup device is a good alternative. This RAID option protects your data, but doesn't protect your system from a failure of the SSD. That works for a lot of people.

It should be possible to have two or more RAID arrays on two identical hard disks that cover the swap and the root/home partition(s). With this sort of setup, it should be possible to lose one disk at any time and have the system carry on, protecting your boot environment, your swap space, your OS install and all your user data on /home. I am currently working on such a setup on a computer that until recently was running Windows 7 with that very arrangement and is now running MX Linux (AntiX OS based on debian). I am at this moment working out the detailed differences between my setup and one described by Falko Timme at https://www.howtoforge.com/how-to-set-u ... an-squeeze which is recent enough that I think it might work as written for debian and antix.

Why not just setup RAID mirrors from a LiveUSB and install MX Linux on those? I would ask the AntiX developers that question. Well, I already installed MX on /dev/sda and I would need to figure out how to make the installer recognize /dev/md0 as a device that it can partition and install the OS on. After researching a chroot environment approach that I thought was more difficult, I decided to follow Timme's approach, but unfortunately I am not yet finished. (How long does it take to dd a 1TB disk? Nine hours.) I'll keep an eye out here if there is interest to discuss Timme's approach and my experience with it.

[HyperUniverse]

Thanks for replies,

I am more inclined to do a manual backup rather than RAID.
It's for long term storage, data won't change much at all.

I was thinking that RAID would be faster and less input from me needed.
But I am afraid of failures

In case of a hard drive failure:
How long to rebuild a 3TB hard drive?
And I'm thinking the same 3TB hard drive just to copy over (as backup), would take much less time.
So in case of a drive failure wouldn't I be better with a manual copy rather than have the RAID rebuild it?

What about in case of a motherboard failure?
Would I be able to just move all my RAID drives to a new motherboard and work straight away?
What if the same kind of motherboard is not available anymore? Will it work with a totally different one?
I know for sure that non-RAID drives would happily work on a different motherboard with no problems whatsoever.

And the last point that pushes me to the manual backup:
On a RAID you need to have all the drives connected at all times; that increases the chances of them to fail earlier.
Where on a manual backup I can disconnect the drives that I don't need, and only have one (or two) drives live, with all the backup drives packed nicely in antistatic bags in a leaded cupboard.

What do you think?
Thanks.

[CwF]

I think you are on the right track.

The one thing to avoid is dependency, which you mention in a few ways. Overly elaborate solutions often discount this. Your mentions of rebuild time, leaving backup offline until needed, the ability to transparently swap any hardware, etc, are all on target.

My advice is to segregate as much as possible. With multiple ssd's on multiple ports this buys much of the speed advantage of raid setups with zero config dependencies.

Extra gooblygook you didn't ask for;
The idea is touched on in a few other recent threads, and I've teased elsewhere on the idea that File=Directory=Device, but I'm not sure many readers understood... I use qcow's for many data sets, that's a file. It is representative of a device, so can be written out to a device, a ssd. As either a device or a qcow2 file, it can be mounted as a directory.
So, the OS should be small and tidy with no user data. The backup method is then to image it to a file, complete. When configured so, the OS can boot on many machines. My images can boot and configure automagically to any motherboard I've tried. With user data elsewhere, these images are pretty static so a backup to image file happens a few times per year, after changes.
Next is the user data. You could simply pattern this after the standard user directories, /Documents, /Pictures, Videos, etc, or add a few mount directories to user(s) in that OS image. Then mount a device, or a qcow file as a device, into these directories. Then each category of data is backed up on it's own schedule, some change often, some rarely. The advantage of backing up a device (ssd) to a qcow2 file is compression. A disk device of any size yields a file no bigger than the actual data, sometimes smaller.
Yes, this method is primarily useful for a full virtual machine based system like I use, but the option to backup to image file is a universal solution. If you add an SSD to the system as /home for example, you could image that to a single file on your backup spinner hard disk, that's usually off. Then one day you upgrade that disk, simply image the file to the new disk, expand the partition to use the full space, and remount. Something like /Videos might be better just to directly copy since we all have terabytes of crap... On my backup spinner I have a few dozen qcow2's, many a few versions just in case. With that backup disk, a bootable usb, and a stack of new hardware I can rebuild a complex system fairly easy.
Food for thought!
Leave raid arrays to systems with 24/7/365 many user requirements with the requisite shelf of spare backup hardware. That unused shelf is where I get my hardware!
That reminds me, I buy a quality smallish SSD every so often and migrate the OS to this new disk. The old disk, hammered as an OS disk for the last year or more I then use as a data disk, living out its life 'less hammered'. I'm sad <120GB are hard to come by now.

[debbieanne]

I am at this moment working out the detailed differences between my setup and one described by Falko Timme at https://www.howtoforge.com/how-to-set-u ... an-squeeze which is recent enough that I think it might work as written for debian and antix.

Timme's article failed to help. Like all the other approaches I tried, there were omissions or wrong labels on modules (eg, mdraid when the proper name is mdraid1x). I found Raider (http://raider.sourceforge.net/) and it worked on MX Linux (based on AntiX based on Debian).

I couldn't get it to work with my original complicated partition plan, but when I reinstalled MX Linux using their defaults, Raider worked perfectly. It threw seven "missing module" errors but they didn't break anything.

Download the scripts tarball, expand it in /opt or wherever, run the install script and then run "raider -t -R1" and it will check and see if it thinks it can work. Then, if you agree, run "raider --run", shutdown when it tells you, swap your hard disk SATA cables, reboot and run "raider --run" again. Wait a long time for the script to sync the arrays and then Bob's your uncle (at least he was my uncle). There is even an uninstaller. AFAIK, it is under active development including plans to handle encrypted partitions and avoid the cable swap.

[Zachary009]

I am at this moment working out the detailed differences between my setup and one described by Falko Timme at https://www.howtoforge.com/how-to-set-u ... on-debian-myfirstpremiercard which is recent enough that I think it might work as written for debian and antix.

Timme's article failed to help. Like all the other approaches I tried, there were omissions or wrong labels on modules (eg, mdraid when the proper name is mdraid1x). I found Raider (http://raider.sourceforge.net/) and it worked on MX Linux (based on AntiX based on Debian).

I couldn't get it to work with my original complicated partition plan, but when I reinstalled MX Linux using their defaults, Raider worked perfectly. It threw seven "missing module" errors but they didn't break anything.

Download the scripts tarball, expand it in /opt or wherever, run the install script and then run "raider -t -R1" and it will check and see if it thinks it can work. Then, if you agree, run "raider --run", shutdown when it tells you, swap your hard disk SATA cables, reboot and run "raider --run" again. Wait a long time for the script to sync the arrays and then Bob's your uncle (at least he was my uncle). There is even an uninstaller. AFAIK, it is under active development including plans to handle encrypted partitions and avoid the cable swap.

Great idea man thanks keep it up all the time. am very happy to see your standard.

[Leoncio]

This is a bit old thread but I believe it is important to state something in case somebody find this thread searching info about RAID.

I am more inclined to do a manual backup rather than RAID.

(...)

So in case of a drive failure wouldn't I be better with a manual copy rather than have the RAID rebuild it?

(...)

And the last point that pushes me to the manual backup:
On a RAID you need to have all the drives connected at all times; that increases the chances of them to fail earlier.
Where on a manual backup I can disconnect the drives that I don't need, and only have one (or two) drives live, with all the backup drives packed nicely in antistatic bags in a leaded cupboard.

It seems you see RAID and backup as two alternatives, which is plainly wrong. There is not an alternative to backup; period. Do always backup regardless you use RAID or not.
Your sensible alternatives are:
1. Backup + RAID
2. Backup

RAID will give you some degree of protection against the failure of one of the disks (unless you use RAID 6 where 2 disks may fail without loosing your data). But bear in mind this: if both disks (in the case of RAID 1) are of the same brand and model and bought at the same time, when one of them die because of aging the other will be as aged as the dead one so, it might fail very soon. When you put the new disk to replace the dead one and regenerating the RAID system, the whole aged disk will be read to replicate it. This means that you will put under stress a disk as aged as the dead one; hopefully everything will go well, but the odds that the surviving disk fail are not that small.

And most important: a RAID will not protect you against the worst enemy of our data: our f***ing fingers. Human mistakes are IMO the worst threat to our data. Only frequent incremental backups will protect you against this.

Do implement in your computer a good backup system that do periodic incremental backups with the aid of cron or anacron. I would suggest you Borg Backup; an excellent piece of software. https://borgbackup.readthedocs.io/en/stable/ But this thread is not about backup but about RAID.

Only after setting the backup you might consider implementing RAID. I would suggest you 4 disks in RAID 10 (better named RAID 1+0). That's what I have done in my system: I have migrated my main aged disk, the one with the operating system, to a 4 disks RAID 1+0. I almost can not believe it, but I had not any problem. You will get not only additional protection against disk failure and deadly simple way to solve it, but also improved performance in read and write. Another good thing of RAID: when a disk fails, the system continue working with the RAID in a degraded state, and while regenerating after putting the new disk it will also continue working but, in the later case, with decreased performance till RAID ends the regeneration.

[p.H]

When you put the new disk to replace the dead one and regenerating the RAID system, the whole aged disk will be read to replicate it. This means that you will put under stress a disk as aged as the dead one

I have often read this argument, but I tend to disagree for two reasons :
- Replication uses sequential reads, which put less stress on the disk than usual use case random reads.
- The replication speed is limited.

Another good thing of RAID: when a disk fails, the system continue working with the RAID in a degraded state, and while regenerating after putting the new disk it will also continue working

"Another" ? Is that not the primary goal of RAID ?

[Leoncio]

When you put the new disk to replace the dead one and regenerating the RAID system, the whole aged disk will be read to replicate it. This means that you will put under stress a disk as aged as the dead one

I have often read this argument, but I tend to disagree for two reasons :
- Replication uses sequential reads, which put less stress on the disk than usual use case random reads.
- The replication speed is limited.

In the case of a RAID not connected to internet as a server, to read the whole disk continuously from the beginning is far more work than the usual reads and writes of a home or small business user. And in the case of a server connected to internet it is worse: it is continuously doing a lot of random reads and writes while in a degraded state and at the same time being read from the beginning to the end. The regeneration work is on top of the normal work.

In RAID 1 or 10 I suppose that the probability of failure while regenerating is not high, since only one disk must be read. But imagine a RAID 5 or 6, with 5 disks or more, where all the disks must be read at the same time to regenerate it. The probability of another, already aged, disk failing could not be low. That's why RAID 5 is so risky.

Another good thing of RAID: when a disk fails, the system continue working with the RAID in a degraded state, and while regenerating after putting the new disk it will also continue working

"Another" ? Is that not the primary goal of RAID ?

The primary goal depends on the needs and taste of each user. For example, in RAID 0 (extremely risky) there is no degraded state; when a disk fails all the data is lost, but for some users it is fine for the increased performance they get. If the backup is very frequent, easy to restore and it is not much of a problem the time it will be without being used, that might be interesting for some people.

In my case the primary goal was to avoid the burden of installing Linux again and trying to guess the small tweaks I had done in my old disk. Also that I will not loose a single byte when a disk fail, while restoring a backup I would loose everything changed since the last backup.

For different people ans systems the main advantage and disadvantage is different.

[p.H]

to read the whole disk continuously from the beginning is far more work than the usual reads and writes of a home or small business user

Moving the head continuously from the beginning to the end puts much less electrical and mechanical stress than the usual random access.

The primary goal depends on the needs and taste of each user. For example, in RAID 0 (extremely risky) there is no degraded state; when a disk fails all the data is lost

You picked the exception in the RAID family. The "R" in RAID stands for "Redundant" so RAID 0 can hardly be called "RAID".

[Leoncio]

to read the whole disk continuously from the beginning is far more work than the usual reads and writes of a home or small business user

Moving the head continuously from the beginning to the end puts much less electrical and mechanical stress than the usual random access.

As I said (please,read what I wrote) that sequential reading is on top, added to, the normal random reading. We will have the normal disk activity plus the sequential reading. Even more, the fact that that sequential reading is continuously interrupted by the normal disk activity makes that the heads do not move continuously from the beginning to the end but they jump even more than before the RAID begun to regenerate. Just imaginea small business with 2-3 employees (or even a single home user) doing an extensive use of database programs. Or a RAID connected to internet serving files to tenths of users at the same time: The disk start the sequential reading; then the head jumps to read or write in a database or to serve a file to an internet user; then the head jumps to continue the sequential reading; then the sequential reading is interrupted again for another access of the database or serving a file; and over and over and over.

In a disk that is being used while regenerating there is not such a sequential reading, since it is continuously interrupted by the normal disk usage. When it is continuously interrupted and restarted, the sequential reading becomes de facto another random reading added on top of the other normal random readings. For the heads of the disk there is not any difference between a jump to read a piece of a database, a jump to read a file asked from internet or a jump to read the disk to regenerate the RAID; regeneration that will soon be interrupted again with another jump for the normal activity of the disk.

[andre@home]

At least 2 points to consider:

• RAID is NO alternative to a backup, NEVER : I've seen RAID getting lost within a few days after the 1st RAID disk broke down.
• In the past... with slow transfer rates of IDE hard disk RAID could help to speed up data transfer: Now the speed of SSD's is so high, other limitations (e.g. internet) make it much less useful to set up a RAID.

So make at least 2 backups of your data and store one of them outside your house on another safe place you can trust.