BTRFS on Debian Stable, your experiences?

If none of the more specific forums is the right place to ask

Re: BTRFS on Debian Stable, your experiences?

Postby LE_746F6D617A7A69 » 2020-07-31 01:59

Mr. Lumbergh wrote:It looks like ext4 performs better in most cases in benchmarks: https://www.phoronix.com/scan.php?page=article&item=linux-50-filesystems&num=4. I often need to record audio in real time, so don't want the performance hit; I'll just make sure to keep regular backups.
Speaking of, though, those are probably a pretty good candidate for BTRFS because of the snapshot feature.
Hmm...
It's extremely unlikely that BTRFS will affect audio recording, unless You're writing to a cheap SD card ;)

But anyway, BTRFS is indeed rather slow - and those test on Phoronix are not showing what happens when You take few volume snapshots - the CoW is causing write amplification, which is just a performance killer, especially on cheap SSDs (TLC/QLC).

I agree that BTRFS is probably the best choice for servers, using server-grade RAID arrays as BTRFS volumes, but for desktop it makes completely no sense, f.e.:
- BTRFS software RAID is at least 2 times slower than md-raid in case of sequential access, random access is just *deadly* slow.
- Checksums: That would deserve a separate thread, but it's a complete stupidity to use FS checksums for a PC equipped with a non-ECC RAM -> this *feature* only eats the resources, and the *improved safety* is nothing but a placebo.
- scrubbing: just slows down everything even more, and it's far more expensive than in case of md-arrays.
- defragmentation? -> Ext4 does not need it, and SSDs just hate it ;)
- An obvious problem with snapshots: You have to use at least software RAID-1 or make regular backups of the entire drive -> otherwise, the drive failure will kill all the snapshots made so far.

That's obviously just My point of view, and I'm sure that many users are very happy with BTRFS ;)
Bill Gates: "(...) In my case, I went to the garbage cans at the Computer Science Center and I fished out listings of their operating system."
The_full_story and Nothing_have_changed
LE_746F6D617A7A69
 
Posts: 281
Joined: 2020-05-03 14:16

Re: BTRFS on Debian Stable, your experiences?

Postby pylkko » 2020-07-31 08:15

Yes, and maybe another opinion would be something like this:

First of all, any performance issue is likely irrelevant, because you can setup the machine so that it does not matter. For example, those distributions that install in btrfs by default (Fedora, OpenSUSE), use btrfs subvolumes with different settings for different parts (logs, system, data whatever) of the disk. OpenSUSE has the home partition on XFS because apparently it has better performance than EXT4 on multimedia / large files.

LE_746F6D617A7A69 wrote:
But anyway, BTRFS is indeed rather slow - and those test on Phoronix are not showing what happens when You take few volume snapshots - the CoW is causing write amplification, which is just a performance killer, especially on cheap SSDs (TLC/QLC).

But exactly where do you think you would notice this in practice? Why would I care if the disk performs X percent slower when it is idle nearly 95% of the time anyway?
LE_746F6D617A7A69 wrote:
I agree that BTRFS is probably the best choice for servers, using server-grade RAID arrays as BTRFS volumes,

if you have hardware raid, why would you use software?

LE_746F6D617A7A69 wrote:- BTRFS software RAID is at least 2 times slower than md-raid in case of sequential access, random access is just *deadly* slow.

with "deadly" slow you mean technically slow in such a way that you would not ever notice it, if you wanted to use software raid in btrfs, when most desktop users do not use RAID at all.

LE_746F6D617A7A69 wrote:- Checksums: That would deserve a separate thread, but it's a complete stupidity to use FS checksums for a PC equipped with a non-ECC RAM -> this *feature* only eats the resources, and the *improved safety* is nothing but a placebo.

With complete stupidity you mean that people can notice if their data is corrupted, and in case they are using the RAID even recover it.
LE_746F6D617A7A69 wrote:- scrubbing: just slows down everything even more, and it's far more expensive than in case of md-arrays.
Just checks your drive is intact while it is idle, and you can always just not do it.
LE_746F6D617A7A69 wrote:- defragmentation? -> Ext4 does not need it, and SSDs just hate it ;)

although, SSDs for which it would matter have not been made for a decade almost
LE_746F6D617A7A69 wrote:- An obvious problem with snapshots: You have to use at least software RAID-1 or make regular backups of the entire drive -> otherwise, the drive failure will kill all the snapshots made so far.

Who ever agrees with this would probably also agree with the statement:" an obvious problem of forks is that they are really bad for eating soup". Yes, so what? Nobody has ever claimed that snapshot are replacements for backups and if someone ever does, you should maybe consider taking stuff that person says with a grain of salt.

BTRFS has limitations and differences from other data management ways. It cannot replace backups, RAIDs and other things. But it also does have properties that are unique to it, and it can be used together with other filesystems or raid setups or backup systems. Other filesystems (XFS, F2FS) are now also implementing btrfs-like features. It will probably require that Canonical or other player creates an intuitive integrated GUI system to manage the snapshots like on MacOS before it becomes widely used, however.
User avatar
pylkko
 
Posts: 1779
Joined: 2014-11-06 19:02

Re: BTRFS on Debian Stable, your experiences?

Postby LE_746F6D617A7A69 » 2020-07-31 15:00

pylkko wrote:Yes, and maybe another opinion would be something like this:

First of all, any performance issue is likely irrelevant, because you can setup the machine so that it does not matter. For example, those distributions that install in btrfs by default (Fedora, OpenSUSE), use btrfs subvolumes with different settings for different parts (logs, system, data whatever) of the disk. OpenSUSE has the home partition on XFS because apparently it has better performance than EXT4 on multimedia / large files.

LE_746F6D617A7A69 wrote:
But anyway, BTRFS is indeed rather slow - and those test on Phoronix are not showing what happens when You take few volume snapshots - the CoW is causing write amplification, which is just a performance killer, especially on cheap SSDs (TLC/QLC).

But exactly where do you think you would notice this in practice? Why would I care if the disk performs X percent slower when it is idle nearly 95% of the time anyway?

Of course it depends on the use case - for Me the show-stopper in BTRFS is a poor performance of databases and build systems -> so I have switched back to md-arrays + ext4.
I agree that for typical desktop use case, it doesn't matter what FS is used and what unique functionalities it offers.

But I think that one aspect needs clarification: BTRFS checksums.
A standard HDD has a theoretical probability of undetected read error of 1 bit in 10^15 bits, what means 1 bit in 938.83TiB - and for non-ECC RAM error rate is undefined. This means, that 2 situations are relatively likely to happen:
1. The checksum can be calculated for damaged data in RAM, just before sending them to a storage device -> the number of copies doesn't matter -> You have damaged data in all the copies, and it's impossible to detect that fact on the FS level.
2. The checksum calculated for correct data loaded from HDD may falsely report inconsistency because of non-ECC RAM error -> consequences are depending on the FS/OS configuration.

The non-ECC RAM is simply far less reliable than a classic HDD in terms of detecting so called "rot bits" -> so it makes completely no sense to use BTRFS checksums on desktop/laptop.
Bill Gates: "(...) In my case, I went to the garbage cans at the Computer Science Center and I fished out listings of their operating system."
The_full_story and Nothing_have_changed
LE_746F6D617A7A69
 
Posts: 281
Joined: 2020-05-03 14:16

Re: BTRFS on Debian Stable, your experiences?

Postby Mr. Lumbergh » 2020-07-31 23:52

LE_746F6D617A7A69 wrote:
Mr. Lumbergh wrote:It looks like ext4 performs better in most cases in benchmarks: https://www.phoronix.com/scan.php?page=article&item=linux-50-filesystems&num=4. I often need to record audio in real time, so don't want the performance hit; I'll just make sure to keep regular backups.
Speaking of, though, those are probably a pretty good candidate for BTRFS because of the snapshot feature.
Hmm...
It's extremely unlikely that BTRFS will affect audio recording, unless You're writing to a cheap SD card ;)

It isn't just the filesystem itself, though, it's the other things that have to slow down to accommodate it. Reading/writing from storage is generally the slowest step in the chain, why knowingly add latency? That's just my thought anyway; a filesystem is just a tool, and as with any tool there are use cases where some are better-suited than others. Having snapshots built in really does make BTRFS attractive for backups though.
Mr. Lumbergh
 
Posts: 91
Joined: 2019-08-02 04:28

Re: BTRFS on Debian Stable, your experiences?

Postby Head_on_a_Stick » 2020-08-01 11:56

Btrfs is fantastic for the development branches because snapshots. They're not backups but they are a quick and easy way to restore the system from a b0rked update.

See also https://packages.debian.org/buster/snapper & https://packages.debian.org/buster/snapper-gui
Black Lives Matter

Debian buster-backports ISO image: for new hardware support
User avatar
Head_on_a_Stick
 
Posts: 12318
Joined: 2014-06-01 17:46
Location: /dev/chair

Re: BTRFS on Debian Stable, your experiences?

Postby LE_746F6D617A7A69 » 2020-08-01 15:50

Mr. Lumbergh wrote:It isn't just the filesystem itself, though, it's the other things that have to slow down to accommodate it. Reading/writing from storage is generally the slowest step in the chain, why knowingly add latency?
Sorry, but apparently I don't get what do You mean - what latency?
The writes are buffered, and even very old HDD will easily swallow ~40MB/s on average. Recording 8 audio streams sampled @96kHz/32bit gives 2.93MB/s ...

Head_on_a_Stick wrote:Btrfs is fantastic for the development branches because snapshots
For development and experimenting I'm using Virtual Machines :D
Bill Gates: "(...) In my case, I went to the garbage cans at the Computer Science Center and I fished out listings of their operating system."
The_full_story and Nothing_have_changed
LE_746F6D617A7A69
 
Posts: 281
Joined: 2020-05-03 14:16

Re: BTRFS on Debian Stable, your experiences?

Postby pylkko » 2020-08-01 19:53

LE_746F6D617A7A69 wrote:...for Me the show-stopper in BTRFS is a poor performance of databases and build systems -> so I have switched back to md-arrays + ext4.

Can you share what kind of databases you have and exactly what the performance difference is?

LE_746F6D617A7A69 wrote: The non-ECC RAM is simply far less reliable than a classic HDD in terms of detecting so called "rot bits" -> so it makes completely no sense to use BTRFS checksums on desktop/laptop.

What do you mean when you say that non-ECC RAM or HDD is less reliable "in terms of detecting "bit rot"? And how do you come to the conclusion that "it makes no sense to use checksums"? Are you referring to the post on the kernel mailing list some years ago about RAM corruption undercutting FS level checks? If so, let's just state for the record that that is an interesting possible theoretical fringe case that has, as far as I know, never been proven to have actually even happen in real life at all...

LE_746F6D617A7A69 wrote:Sorry, but apparently I don't get what do You mean - what latency?
The writes are buffered, and even very old HDD will easily swallow ~40MB/s on average. Recording 8 audio streams sampled @96kHz/32bit gives 2.93MB/s ...

To me, it appears that the issue here is that you are confusing latency with throughput.
For development and experimenting I'm using Virtual Machines :D

Is it possible to "diff" VM snapshots, for example? I want to learn to do that (if it is possible). But I would still say running a full VM for something that could simply be a folder that you can roll back is...strange. Why not git?
User avatar
pylkko
 
Posts: 1779
Joined: 2014-11-06 19:02

Re: BTRFS on Debian Stable, your experiences?

Postby LE_746F6D617A7A69 » 2020-08-01 23:47

@pylkko: it looks like You're a BTRFS fanboy - that's OK, but I'm not going to play that game. The tests on Phoronix, although incomplete, are showing what's the problem ...

non-ECC-RAM deserves a separate topic, but most of the info can be found on the wiki pages ...

Throughput has nothing to do with the latency, unless the I/O bandwidth limits are reached ...
Bill Gates: "(...) In my case, I went to the garbage cans at the Computer Science Center and I fished out listings of their operating system."
The_full_story and Nothing_have_changed
LE_746F6D617A7A69
 
Posts: 281
Joined: 2020-05-03 14:16

Re: BTRFS on Debian Stable, your experiences?

Postby Bloom » 2020-08-02 08:49

For those who care: I use LizardFS with JFS as my base filesystem. Works flawlessly. Up to 215 TiB now.
User avatar
Bloom
 
Posts: 273
Joined: 2017-11-11 12:23

Re: BTRFS on Debian Stable, your experiences?

Postby pylkko » 2020-08-02 12:08

LE_746F6D617A7A69 wrote:@pylkko: it looks like You're a BTRFS fanboy - that's OK, but I'm not going to play that game. The tests on Phoronix, although incomplete, are showing what's the problem ...

I'm a fanboi of "the larger the claims, the better the evidence needs to be". When you hear that people say stuff like "using this tech is a total stupidity", or "it is deadly slow" you start to wonder how on earth they determined this one tech to be so totally unusable.

About those tests on Phoronix... we could also look the rest of them. Like the latest one where btrfs outperforms ext4 in many things, one of which, by the way, was SQlite databases.
https://www.phoronix.com/scan.php?page= ... tems&num=1
This makes it even more interesting that your databases are performing poorly. There are probably some additional factors at play...
LE_746F6D617A7A69 wrote:Throughput has nothing to do with the latency, unless the I/O bandwidth limits are reached ...
Then why did you start to talk about theoretical throughput in MB/s when the guy asked about the latency?

As throughput and latency are often correlated (on the implementation level, like comparing different storage technologies), it seems quite an over statement to say "they have nothing to do with each other". But they can "decouple" in some situation. One such situation probably is the use of transparent compression. Apparently a btrfs partition using transparent compression can move a large file (if it compresses well) much faster than a partition without (for example ext4) with the processors of today. But, this will come with a latency cost, since the device needs to move a block and let the CPU compress it before even the first bit can be transferred. If you are doing latency sensitive work, these kinds of issues you would think about, no?
User avatar
pylkko
 
Posts: 1779
Joined: 2014-11-06 19:02

Re: BTRFS on Debian Stable, your experiences?

Postby LE_746F6D617A7A69 » 2020-08-02 12:45

pylkko wrote:Like the latest one where btrfs outperforms ext4 in many things, one of which, by the way, was SQlite databases.
Like I said, the tests on Phoronix are incomplete, because they are not showing what happens after some snapshots are taken - high write performance drop due to CoW activities.

pylkko wrote:
LE_746F6D617A7A69 wrote:Throughput has nothing to do with the latency, unless the I/O bandwidth limits are reached ...
Then why did you start to talk about theoretical throughput in MB/s when the guy asked about the latency?
I do realize that my knowledge of English is far from perfection, but it it seems that on the other hand, You have a problem with understanding what You're reading. Therefore I'll translate the above sentence specially for You:
As long as the bandwidth needed for audio stream recording is below the hardware limits, it does not affect the latency in the recording process. (thanks to buffered writes - mentioned earier)

Hope that helps...
Bill Gates: "(...) In my case, I went to the garbage cans at the Computer Science Center and I fished out listings of their operating system."
The_full_story and Nothing_have_changed
LE_746F6D617A7A69
 
Posts: 281
Joined: 2020-05-03 14:16

Re: BTRFS on Debian Stable, your experiences?

Postby Deb-fan » 2020-08-02 13:59

Thank you for arguing, errr i mean sharing guys. :) All kidding aside really mean that, some well stated points pylkko. Btrfs has longggggg been on my 2do list though as yet cant put it in my 2done list. Havent been paying all that much attention in quite awhile as to it's status, hearing its overtaking ext4 piques my interest though and makes a lot of sense to me. Am sure there's plenty to learning to properly setup, use and tune it correctly. More so than with ext4 and the performance mount options and tuning ive learned about with it. Again not surprising given the list of features btrfs sports.

Surely a fs like btrfs is receiving plenty of development attention for its application in Enterprise and production uses, aka: gnu/Linux's mainstay bread and butter. Only have research mostly to go on ... so gonna shut it. Definitely need to read this whole thread, thanks everyone who contributed to this discussion. Cool stuff. :)
Most powerful FREE tech-support tool on the planet * HERE. *
Deb-fan
 
Posts: 814
Joined: 2012-08-14 12:27

Re: BTRFS on Debian Stable, your experiences?

Postby pylkko » 2020-08-02 14:08

LE_746F6D617A7A69 wrote:[Like I said, the tests on Phoronix are incomplete, because they are not showing what happens after some snapshots are taken - high write performance drop due to CoW activities.

Earlier, however, it appears you said that those Photronix tests do show what you argue...
if they don't, then what does show it?

Therefore I'll translate the above sentence specially for You:
As long as the bandwidth needed for audio stream recording is below the hardware limits, it does not affect the latency in the recording process.

Ok, but the guy was not asking about that.
User avatar
pylkko
 
Posts: 1779
Joined: 2014-11-06 19:02

Re: BTRFS on Debian Stable, your experiences?

Postby LE_746F6D617A7A69 » 2020-08-02 15:20

pylkko wrote:
LE_746F6D617A7A69 wrote:[Like I said, the tests on Phoronix are incomplete, because they are not showing what happens after some snapshots are taken - high write performance drop due to CoW activities.

Earlier, however, it appears you said that those Photronix tests do show what you argue...
if they don't, then what does show it?
Here is My first sentence about the tests on the Phoronix:
http://forums.debian.net/viewtopic.php?f=10&t=141835&start=15#p724943
Anyway, in most of the tests BTRFS is the slowest FS, even without CoW write amplification. The "latest" test is also flawed, because it was made only for single model of SSD (probably just promoting the product), and the ext4 was not optimized for SSDs (and BTRFS don't even have such option). I'm talking here about the Ext4 feature to define stripe and chunk sizes - so they can match the erase block size on the NVME/Flash devices - this reduces write amplification effects and extends the life-time of the device.

pylkko wrote:
LE_746F6D617A7A69 wrote: Therefore I'll translate the above sentence specially for You:
As long as the bandwidth needed for audio stream recording is below the hardware limits, it does not affect the latency in the recording process.

Ok, but the guy was not asking about that.
He said that BTRFS will likely cause some "performance hit" or introduce some undefined "latency" during recording of audio streams, which is obviously very unlikely because of very low requirements regarding the throughput for such task.
Bill Gates: "(...) In my case, I went to the garbage cans at the Computer Science Center and I fished out listings of their operating system."
The_full_story and Nothing_have_changed
LE_746F6D617A7A69
 
Posts: 281
Joined: 2020-05-03 14:16

Re: BTRFS on Debian Stable, your experiences?

Postby pylkko » 2020-08-02 19:56

I don't undertand why you bring up your "first sentence about Phoronix". We were not talking about that.

Let's not make decisions on file-system performance based on a cherry picked benchmark. Especially so if we need to then call any challenging benchmarks "flawed": The "flawed" benchmark is using the default settings of the OS. Any additional tweaks may change the outcomes slightly, but it is unlikely they would change the verdict. We know that a "next gen" CoW filesystem will always get a performance hit (we pay for the new features in performance), that in itself is not interesting. What is interesting is: does that matter? Does the synthetic benchmark translate to anything of any significance in real life? Is the magnitude of the performance hit large enough that any one would care or notice? I would say that the answer is: no, not even if you extrapolate from the benchmarks in which btrfs did not do so well. I really wonder in what situation in practice you would see a differnce. You said that you have some non-ordinary desktop situations where it matters, but you refuse to describe them further. This can very well be true, but it does not justify all of your "totally dead" and "100% stupidy" remarks at all. That is "throwing out the baby with the bathwater".

Some guy in the Phoronix comments points out how checksumming helped him notice a data corruption in his music collection, which he recovered immediately. He adds that on Ext4 it would have just lain there hidden for years until he would have noticed. That's a real thing, I don't think he cares if "in theory non-ECC RAM puts a loop whole in file-system level checksums". This does not mean that the point about RAM being possibly a weaker link is not real. But what conclusions do you make thereof, rationally? You know, practical real life stuff, not some theoretical/synthetic arguments.

This is the core of the argument that I tried to bring forward, although I could have for sure made it better.
User avatar
pylkko
 
Posts: 1779
Joined: 2014-11-06 19:02

PreviousNext

Return to General Questions

Who is online

Users browsing this forum: Head_on_a_Stick and 5 guests

fashionable