Scheduled Maintenance: We are aware of an issue with Google, AOL, and Yahoo services as email providers which are blocking new registrations. We are trying to fix the issue and we have several internal and external support tickets in process to resolve the issue. Please see: viewtopic.php?t=158230

 

 

 

What Raid Scenario - ZFS?

If none of the specific sub-forums seem right for your thread, ask here.
Post Reply
Message
Author
bookie
Posts: 370
Joined: 2007-12-07 11:49

What Raid Scenario - ZFS?

#1 Post by bookie »

Hi guys!
I have Debian 8.9 at the moment with an existing software raid 5 configuration....this is on 4 x 2TB drives....

Instead of adding extra 2TB drives to the raid array I thought it was about time to upgrade to a little larger drives....4TB to be exact....

Now just that fact I have a software raid 5 scenario is very scary for most of you... :D BUT I have had this raid array for several years without problems...

Upgrading to 4TB drives causes problems with Reliability/Data integrity <1 in 1014....I have even posed the question about the error margin on WD but no reply----
There is a lot of speculation about what this means in a raid 5 array but no real concrete info....

Of course when one reads so much that says against raid 5 - then it isn't so strange that one is looking for a new solution....

The main reason for raid in my situation and for others is the fact we want to pool our drives but have redundancy in case of drive failures....

We the rise in popularity of ZFS on Linux I have been looking at that for all the built in functions it offers.....BUT even that isn't easy...there is a ton of info and sifting through what is acutally true or false is hard if you don't know what true or false is...if you follow....

I have a computer that I use for cloning customer computers with the server edition of Clonezilla and that works fine....

I have a back up of these images (files) on another computer which at present hasn't a raid scenario....

I would like feedback on what members are actually doing when they pool lots of drives on a computer ....be it ZFS or ext and what raid scenario are they using....

Raid isn't a backup....YES I know but if one has lots of drives on another computer for your backups then the obvious solution is pooling them in a raid scenario with redundancy....or am I missing something..... :shock:

It is amazing how questions like this get met with either total silence...OR lots of info what not to do....but almost never what people are actually doing....?

I wait with anticipation to see what this topic brings.... :lol:


bookie

steve_v
df -h | grep > 20TiB
df -h | grep > 20TiB
Posts: 1401
Joined: 2012-10-06 05:31
Location: /dev/chair
Has thanked: 79 times
Been thanked: 175 times

Re: What Raid Scenario - ZFS?

#2 Post by steve_v »

I ran RAID5 for a while, then I had 2 disks die on the same day. Now I use ZFS (RAIDZ6).

IMO, RAID5 is dead, RAID6 is still viable but suffers bitrot, and ZFS is pure awesome.

Current home/backup/pet server configuration:
Supermicro X7DCL-3, 2x X5460, 48GB ECC RAM. (all for a song on ebay)
8x4TB in RAIDZ2 + 2x 60GB SSD (ARC) for data, 4x 160GB in RAID10 for OS, webserver, database etc.
Important data snapshots replicated to external drive (also ZFS), very important data replicated over ssh to a remote server.

ZFS is hardware hungry (especially RAM), and a bit slower than an equivalent MD RAID, but you get a whole lot of tasty features as compensation - snapshots, replication, checksums, compression, advanced caching, deduplication (even more RAM hungry) and lately transparent encryption too.
If you have sufficient (ECC) memory, go ZFS, it's just better. It solves the bitrot problem too.

RAID is not a backup, neither is ZFS. But ZFS replicated to another ZFS, replicated to another ZFS... Kinda is :P
Once is happenstance. Twice is coincidence. Three times is enemy action. Four times is Official GNOME Policy.

bookie
Posts: 370
Joined: 2007-12-07 11:49

Re: What Raid Scenario - ZFS?

#3 Post by bookie »

Hi steve_v :D
It is a dilemma having a backup that is also raid.... :D
I just don't know a better way to have a backup if pooling drives as one and you want redundancy....
My computer has non-ECC memory....I have seen many say that even if you don't have ECC memory you should be alright....plus having snapshots on another computer...thinking ahead if I choose ZFS
I have 16GB on this computer....
Supermicro X7DCL-3, 2x X5460, 48GB ECC RAM. (all for a song on ebay)
Have you links to any more songs on e-bay...?

I have been reading about one should mirror vdevs any thoughts on that?

As you know there is a ton of info about ZFS...but it takes time to sift through it all....

Have you any links to practical setups using your raid 6?

I am thinking to test this on my VMWare Workstation using different scenarios....

Thanks!

bookie

jmgibson1981
Posts: 296
Joined: 2015-06-07 14:38
Has thanked: 11 times
Been thanked: 32 times

Re: What Raid Scenario - ZFS?

#4 Post by jmgibson1981 »

Depending on what you are backing up another option to consider is snapraid. I use it at home for media storage. Much lighter on hardware than zfs, many nice features. Not a real time raid though, is more like a backup than a raid system. But for large non changing storage it fits the bill wonderfully. Also if more disks die than parity I only lose those disks, not the whole array.

bookie
Posts: 370
Joined: 2007-12-07 11:49

Re: What Raid Scenario - ZFS?

#5 Post by bookie »

Hi jmgibson1981 :D
I will take a look at snapraid....

bookie

steve_v
df -h | grep > 20TiB
df -h | grep > 20TiB
Posts: 1401
Joined: 2012-10-06 05:31
Location: /dev/chair
Has thanked: 79 times
Been thanked: 175 times

Re: What Raid Scenario - ZFS?

#6 Post by steve_v »

bookie wrote:Have you links to any more songs on e-bay...?
Used server stuff is always cheap on ebay.
I have been reading about one should mirror vdevs any thoughts on that?
Same old tradeoff as with conventional RAID - performance vs space efficiency. Especially true for ZFS as you are limited to the speed of the slowest drive in a vdev.
Have you any links to practical setups using your raid 6?
Not sure what you're after here, just read the manual. There's nothing special about my setup.
Once is happenstance. Twice is coincidence. Three times is enemy action. Four times is Official GNOME Policy.

bookie
Posts: 370
Joined: 2007-12-07 11:49

Re: What Raid Scenario - ZFS?

#7 Post by bookie »

Hi steve_v :D
OK! I hear you...going to spend some time with the help pages for ZFS until I get a better understanding of everything....
Thanks for your time!
bookie

steve_v
df -h | grep > 20TiB
df -h | grep > 20TiB
Posts: 1401
Joined: 2012-10-06 05:31
Location: /dev/chair
Has thanked: 79 times
Been thanked: 175 times

Re: What Raid Scenario - ZFS?

#8 Post by steve_v »

bookie wrote:My computer has non-ECC memory...
I realise you're not asking for "what not to do" input, but understanding why reliable memory is important is... well, important.

ZFS isn't any more or less reliant on your ram not scrambling your data than any other filesystem, but:
If it happens, and it happens in such a way as to damage filesystem consistency, your entire pool is toast.
Conventional filesystems have tools to recover from this scenario, at least to some extent. ZFS does not.
There is no fsck for ZFS, testdisk will not work, photorec will not work. If your pool is too damaged to import, your data is gone.
Ask anyone how to recover files from a damaged ZFS pool and they will say "Destroy and restore from backup".

ZFS provides some nice guarantees: It will always maintain internal consistency, it will never corrupt your files - even if you pull the plug during a write operation, and it will always either return correct data, or inform you if it cannot.
All of this goes out the window if data in memory can't be trusted, ZFS trusts your RAM unconditionally.

If you are willing to take the risk, that's your call, most people never have a memory module fail anyway.
But if you're running ZFS because you value your data and/or your uptime, running it on non-ECC ram is missing the point.
ECC RAM and a board that supports it is a minor expense when compared to all those data drives anyway, why wouldn't you?
Once is happenstance. Twice is coincidence. Three times is enemy action. Four times is Official GNOME Policy.

bookie
Posts: 370
Joined: 2007-12-07 11:49

Re: What Raid Scenario - ZFS?

#9 Post by bookie »

Hi steve_v :D
I hear you and understand perfectly ...just having a server enclosure with just disks is one thing...building a purpose server is another....I do a lot of work on this computer and wanted to try and safeguard the data pooled from the drives allotted to raid....

I haven't got the money to build a new server especially with ECC memory...I was sort of hoping with the backups that would give me a safety net in case of bad ram.....
I appreciate what you say and I thank you for pointing that out....just am at a loss to know what I can use instead as a solution for pooling drives with redundancy....mdadm clearly hasn't the safegaurds built into it and raid 6 doesn't give data integrity....

It is a bit of a dilemma...

I need to be able to pool the drives on this computer and not have to worry about the fact larger drives seem to be a problem when a drive fails using mdadm...and som sort of data check that can help before backing up files to another computer...


bookie

steve_v
df -h | grep > 20TiB
df -h | grep > 20TiB
Posts: 1401
Joined: 2012-10-06 05:31
Location: /dev/chair
Has thanked: 79 times
Been thanked: 175 times

Re: What Raid Scenario - ZFS?

#10 Post by steve_v »

ZFS is still the best option IMO, even without ECC. Just ensure you have backups in case it all falls down, same as any other storage solution. There's a reason many pros still use tapes ;)

ZFS gets you your pooling and checksums. The rebuild time problem is intrinsic to any parity RAID, and particularly RAID5. This is one of the reasons people suggest RAID10 or mirrored vdevs over RAID 5/6.
As with anything, pick your compromise.

Conventional (mdadm or hardware) RAID6 will give you some redundancy and peace of mind if a disk fails, where RAID5 has you sweating bullets during the rebuild. RAIDZ6 gets you the features of ZFS as well.

RAID10 / mirrored vdevs makes the rebuild window practically go away, but you sacrifice capacity and redundancy... then again, if you only have 4 disks you loose no capacity over RAID6 / RAIDZ6 as two drives would be parity anyway.

So:
With a RAID6 of 4 drives you get n-2 drives of capacity, a 100% probability of surviving a 2 disk failure, middling performance, and long rebuild times.
With a RAID10 of 4 drives you get n/2 drives of capacity, a 67% probability of surviving a 2 disk failure, quick rebuilds, and good performance.
ZFS can do the equivalent of either option, with all its neat features to boot.

I run 8 disks, don't need blazing fast performance, want 2 disk redundancy, and don't want to sacrifice too much capacity. So my choice is RAIDZ6.
What you need depends on your priorities and budget.

Non-ECC RAM is a risk with any solution, which is why all server-grade gear has it as standard, but it's not a huge risk as memory failures aren't that common.
Again, compromises and budget.

The only 100% secure solution is engraving your data on sapphire plates and locking them in an underground vault. :P
Once is happenstance. Twice is coincidence. Three times is enemy action. Four times is Official GNOME Policy.

bookie
Posts: 370
Joined: 2007-12-07 11:49

Re: What Raid Scenario - ZFS?

#11 Post by bookie »

Hi steve_v....
Can you send me the saphire plates.... :wink:
I have ordered and extra disk for the server and the backup computer is being rebuilt...
Have set up Debian 8.9 (works with Clonezilla Server Edition...had problems with Debian 9 and don't need any more work at the moment...going to work with ZFS and raidz6 with 5 drives like I will have on the server when I am finished testing....
I will take a chance on ZFS on the standard ram computer and look around for something else with ECC for my backups....
I will probably have more questions but will read the manual.... :wink:

Thanks so much for your time!

I will post when I have more questions...

bookie

Post Reply