BTRFS on Debian Stable, your experiences?

[LE_746F6D617A7A69]

It looks like ext4 performs better in most cases in benchmarks: https://www.phoronix.com/scan.php?page= ... tems&num=4. I often need to record audio in real time, so don't want the performance hit; I'll just make sure to keep regular backups.
Speaking of, though, those are probably a pretty good candidate for BTRFS because of the snapshot feature.
Hmm...

It's extremely unlikely that BTRFS will affect audio recording, unless You're writing to a cheap SD card

But anyway, BTRFS is indeed rather slow - and those test on Phoronix are not showing what happens when You take few volume snapshots - the CoW is causing write amplification, which is just a performance killer, especially on cheap SSDs (TLC/QLC).

I agree that BTRFS is probably the best choice for servers, using server-grade RAID arrays as BTRFS volumes, but for desktop it makes completely no sense, f.e.:
- BTRFS software RAID is at least 2 times slower than md-raid in case of sequential access, random access is just *deadly* slow.
- Checksums: That would deserve a separate thread, but it's a complete stupidity to use FS checksums for a PC equipped with a non-ECC RAM -> this *feature* only eats the resources, and the *improved safety* is nothing but a placebo.
- scrubbing: just slows down everything even more, and it's far more expensive than in case of md-arrays.
- defragmentation? -> Ext4 does not need it, and SSDs just hate it
- An obvious problem with snapshots: You have to use at least software RAID-1 or make regular backups of the entire drive -> otherwise, the drive failure will kill all the snapshots made so far.

That's obviously just My point of view, and I'm sure that many users are very happy with BTRFS

[pylkko]

Yes, and maybe another opinion would be something like this:

First of all, any performance issue is likely irrelevant, because you can setup the machine so that it does not matter. For example, those distributions that install in btrfs by default (Fedora, OpenSUSE), use btrfs subvolumes with different settings for different parts (logs, system, data whatever) of the disk. OpenSUSE has the home partition on XFS because apparently it has better performance than EXT4 on multimedia / large files.

But anyway, BTRFS is indeed rather slow - and those test on Phoronix are not showing what happens when You take few volume snapshots - the CoW is causing write amplification, which is just a performance killer, especially on cheap SSDs (TLC/QLC).

But exactly where do you think you would notice this in practice? Why would I care if the disk performs X percent slower when it is idle nearly 95% of the time anyway?

I agree that BTRFS is probably the best choice for servers, using server-grade RAID arrays as BTRFS volumes,

if you have hardware raid, why would you use software?

- BTRFS software RAID is at least 2 times slower than md-raid in case of sequential access, random access is just *deadly* slow.

with "deadly" slow you mean technically slow in such a way that you would not ever notice it, if you wanted to use software raid in btrfs, when most desktop users do not use RAID at all.

- Checksums: That would deserve a separate thread, but it's a complete stupidity to use FS checksums for a PC equipped with a non-ECC RAM -> this *feature* only eats the resources, and the *improved safety* is nothing but a placebo.

With complete stupidity you mean that people can notice if their data is corrupted, and in case they are using the RAID even recover it.

- scrubbing: just slows down everything even more, and it's far more expensive than in case of md-arrays.

Just checks your drive is intact while it is idle, and you can always just not do it.

- defragmentation? -> Ext4 does not need it, and SSDs just hate it

although, SSDs for which it would matter have not been made for a decade almost

- An obvious problem with snapshots: You have to use at least software RAID-1 or make regular backups of the entire drive -> otherwise, the drive failure will kill all the snapshots made so far.

Who ever agrees with this would probably also agree with the statement:" an obvious problem of forks is that they are really bad for eating soup". Yes, so what? Nobody has ever claimed that snapshot are replacements for backups and if someone ever does, you should maybe consider taking stuff that person says with a grain of salt.

BTRFS has limitations and differences from other data management ways. It cannot replace backups, RAIDs and other things. But it also does have properties that are unique to it, and it can be used together with other filesystems or raid setups or backup systems. Other filesystems (XFS, F2FS) are now also implementing btrfs-like features. It will probably require that Canonical or other player creates an intuitive integrated GUI system to manage the snapshots like on MacOS before it becomes widely used, however.

[LE_746F6D617A7A69]

Yes, and maybe another opinion would be something like this:

First of all, any performance issue is likely irrelevant, because you can setup the machine so that it does not matter. For example, those distributions that install in btrfs by default (Fedora, OpenSUSE), use btrfs subvolumes with different settings for different parts (logs, system, data whatever) of the disk. OpenSUSE has the home partition on XFS because apparently it has better performance than EXT4 on multimedia / large files.

But anyway, BTRFS is indeed rather slow - and those test on Phoronix are not showing what happens when You take few volume snapshots - the CoW is causing write amplification, which is just a performance killer, especially on cheap SSDs (TLC/QLC).

But exactly where do you think you would notice this in practice? Why would I care if the disk performs X percent slower when it is idle nearly 95% of the time anyway?

Of course it depends on the use case - for Me the show-stopper in BTRFS is a poor performance of databases and build systems -> so I have switched back to md-arrays + ext4.
I agree that for typical desktop use case, it doesn't matter what FS is used and what unique functionalities it offers.

But I think that one aspect needs clarification: BTRFS checksums.
A standard HDD has a theoretical probability of undetected read error of 1 bit in 10^15 bits, what means 1 bit in 938.83TiB - and for non-ECC RAM error rate is undefined. This means, that 2 situations are relatively likely to happen:
1. The checksum can be calculated for damaged data in RAM, just before sending them to a storage device -> the number of copies doesn't matter -> You have damaged data in all the copies, and it's impossible to detect that fact on the FS level.
2. The checksum calculated for correct data loaded from HDD may falsely report inconsistency because of non-ECC RAM error -> consequences are depending on the FS/OS configuration.

The non-ECC RAM is simply far less reliable than a classic HDD in terms of detecting so called "rot bits" -> so it makes completely no sense to use BTRFS checksums on desktop/laptop.

[Mr. Lumbergh]

It looks like ext4 performs better in most cases in benchmarks: https://www.phoronix.com/scan.php?page= ... tems&num=4. I often need to record audio in real time, so don't want the performance hit; I'll just make sure to keep regular backups.
Speaking of, though, those are probably a pretty good candidate for BTRFS because of the snapshot feature.
Hmm...

It's extremely unlikely that BTRFS will affect audio recording, unless You're writing to a cheap SD card

It isn't just the filesystem itself, though, it's the other things that have to slow down to accommodate it. Reading/writing from storage is generally the slowest step in the chain, why knowingly add latency? That's just my thought anyway; a filesystem is just a tool, and as with any tool there are use cases where some are better-suited than others. Having snapshots built in really does make BTRFS attractive for backups though.

[Head_on_a_Stick]

Btrfs is fantastic for the development branches because snapshots. They're not backups but they are a quick and easy way to restore the system from a b0rked update.

See also https://packages.debian.org/buster/snapper & https://packages.debian.org/buster/snapper-gui

[LE_746F6D617A7A69]

It isn't just the filesystem itself, though, it's the other things that have to slow down to accommodate it. Reading/writing from storage is generally the slowest step in the chain, why knowingly add latency?

Sorry, but apparently I don't get what do You mean - what latency?
The writes are buffered, and even very old HDD will easily swallow ~40MB/s on average. Recording 8 audio streams sampled @96kHz/32bit gives 2.93MB/s ...

Btrfs is fantastic for the development branches because snapshots

For development and experimenting I'm using Virtual Machines

[pylkko]

...for Me the show-stopper in BTRFS is a poor performance of databases and build systems -> so I have switched back to md-arrays + ext4.

Can you share what kind of databases you have and exactly what the performance difference is?

The non-ECC RAM is simply far less reliable than a classic HDD in terms of detecting so called "rot bits" -> so it makes completely no sense to use BTRFS checksums on desktop/laptop.

What do you mean when you say that non-ECC RAM or HDD is less reliable "in terms of detecting "bit rot"? And how do you come to the conclusion that "it makes no sense to use checksums"? Are you referring to the post on the kernel mailing list some years ago about RAM corruption undercutting FS level checks? If so, let's just state for the record that that is an interesting possible theoretical fringe case that has, as far as I know, never been proven to have actually even happen in real life at all...

Sorry, but apparently I don't get what do You mean - what latency?
The writes are buffered, and even very old HDD will easily swallow ~40MB/s on average. Recording 8 audio streams sampled @96kHz/32bit gives 2.93MB/s ...

To me, it appears that the issue here is that you are confusing latency with throughput.

For development and experimenting I'm using Virtual Machines

Is it possible to "diff" VM snapshots, for example? I want to learn to do that (if it is possible). But I would still say running a full VM for something that could simply be a folder that you can roll back is...strange. Why not git?

[LE_746F6D617A7A69]

@pylkko: it looks like You're a BTRFS fanboy - that's OK, but I'm not going to play that game. The tests on Phoronix, although incomplete, are showing what's the problem ...

non-ECC-RAM deserves a separate topic, but most of the info can be found on the wiki pages ...

Throughput has nothing to do with the latency, unless the I/O bandwidth limits are reached ...

[Bloom]

For those who care: I use LizardFS with JFS as my base filesystem. Works flawlessly. Up to 215 TiB now.

[pylkko]

@pylkko: it looks like You're a BTRFS fanboy - that's OK, but I'm not going to play that game. The tests on Phoronix, although incomplete, are showing what's the problem ...

I'm a fanboi of "the larger the claims, the better the evidence needs to be". When you hear that people say stuff like "using this tech is a total stupidity", or "it is deadly slow" you start to wonder how on earth they determined this one tech to be so totally unusable.

About those tests on Phoronix... we could also look the rest of them. Like the latest one where btrfs outperforms ext4 in many things, one of which, by the way, was SQlite databases.
https://www.phoronix.com/scan.php?page= ... tems&num=1
This makes it even more interesting that your databases are performing poorly. There are probably some additional factors at play...

Throughput has nothing to do with the latency, unless the I/O bandwidth limits are reached ...

Then why did you start to talk about theoretical throughput in MB/s when the guy asked about the latency?

As throughput and latency are often correlated (on the implementation level, like comparing different storage technologies), it seems quite an over statement to say "they have nothing to do with each other". But they can "decouple" in some situation. One such situation probably is the use of transparent compression. Apparently a btrfs partition using transparent compression can move a large file (if it compresses well) much faster than a partition without (for example ext4) with the processors of today. But, this will come with a latency cost, since the device needs to move a block and let the CPU compress it before even the first bit can be transferred. If you are doing latency sensitive work, these kinds of issues you would think about, no?

[LE_746F6D617A7A69]

Like the latest one where btrfs outperforms ext4 in many things, one of which, by the way, was SQlite databases.

Like I said, the tests on Phoronix are incomplete, because they are not showing what happens after some snapshots are taken - high write performance drop due to CoW activities.

Throughput has nothing to do with the latency, unless the I/O bandwidth limits are reached ...

Then why did you start to talk about theoretical throughput in MB/s when the guy asked about the latency?

I do realize that my knowledge of English is far from perfection, but it it seems that on the other hand, You have a problem with understanding what You're reading. Therefore I'll translate the above sentence specially for You:
As long as the bandwidth needed for audio stream recording is below the hardware limits, it does not affect the latency in the recording process. (thanks to buffered writes - mentioned earier)

Hope that helps...

[Deb-fan]

Thank you for arguing, errr i mean sharing guys. All kidding aside really mean that, some well stated points pylkko. Btrfs has longggggg been on my 2do list though as yet cant put it in my 2done list. Havent been paying all that much attention in quite awhile as to it's status, hearing its overtaking ext4 piques my interest though and makes a lot of sense to me. Am sure there's plenty to learning to properly setup, use and tune it correctly. More so than with ext4 and the performance mount options and tuning ive learned about with it. Again not surprising given the list of features btrfs sports.

Surely a fs like btrfs is receiving plenty of development attention for its application in Enterprise and production uses, aka: gnu/Linux's mainstay bread and butter. Only have research mostly to go on ... so gonna shut it. Definitely need to read this whole thread, thanks everyone who contributed to this discussion. Cool stuff.

[pylkko]

[Like I said, the tests on Phoronix are incomplete, because they are not showing what happens after some snapshots are taken - high write performance drop due to CoW activities.

Earlier, however, it appears you said that those Photronix tests do show what you argue...
if they don't, then what does show it?

Therefore I'll translate the above sentence specially for You:
As long as the bandwidth needed for audio stream recording is below the hardware limits, it does not affect the latency in the recording process.

Ok, but the guy was not asking about that.

[LE_746F6D617A7A69]

[Like I said, the tests on Phoronix are incomplete, because they are not showing what happens after some snapshots are taken - high write performance drop due to CoW activities.

Earlier, however, it appears you said that those Photronix tests do show what you argue...
if they don't, then what does show it?

Here is My first sentence about the tests on the Phoronix:
http://forums.debian.net/viewtopic.php? ... 15#p724943
Anyway, in most of the tests BTRFS is the slowest FS, even without CoW write amplification. The "latest" test is also flawed, because it was made only for single model of SSD (probably just promoting the product), and the ext4 was not optimized for SSDs (and BTRFS don't even have such option). I'm talking here about the Ext4 feature to define stripe and chunk sizes - so they can match the erase block size on the NVME/Flash devices - this reduces write amplification effects and extends the life-time of the device.

Therefore I'll translate the above sentence specially for You:
As long as the bandwidth needed for audio stream recording is below the hardware limits, it does not affect the latency in the recording process.

Ok, but the guy was not asking about that.

He said that BTRFS will likely cause some "performance hit" or introduce some undefined "latency" during recording of audio streams, which is obviously very unlikely because of very low requirements regarding the throughput for such task.

[pylkko]

I don't undertand why you bring up your "first sentence about Phoronix". We were not talking about that.

Let's not make decisions on file-system performance based on a cherry picked benchmark. Especially so if we need to then call any challenging benchmarks "flawed": The "flawed" benchmark is using the default settings of the OS. Any additional tweaks may change the outcomes slightly, but it is unlikely they would change the verdict. We know that a "next gen" CoW filesystem will always get a performance hit (we pay for the new features in performance), that in itself is not interesting. What is interesting is: does that matter? Does the synthetic benchmark translate to anything of any significance in real life? Is the magnitude of the performance hit large enough that any one would care or notice? I would say that the answer is: no, not even if you extrapolate from the benchmarks in which btrfs did not do so well. I really wonder in what situation in practice you would see a differnce. You said that you have some non-ordinary desktop situations where it matters, but you refuse to describe them further. This can very well be true, but it does not justify all of your "totally dead" and "100% stupidy" remarks at all. That is "throwing out the baby with the bathwater".

Some guy in the Phoronix comments points out how checksumming helped him notice a data corruption in his music collection, which he recovered immediately. He adds that on Ext4 it would have just lain there hidden for years until he would have noticed. That's a real thing, I don't think he cares if "in theory non-ECC RAM puts a loop whole in file-system level checksums". This does not mean that the point about RAM being possibly a weaker link is not real. But what conclusions do you make thereof, rationally? You know, practical real life stuff, not some theoretical/synthetic arguments.

This is the core of the argument that I tried to bring forward, although I could have for sure made it better.

[LE_746F6D617A7A69]

We know that a "next gen" CoW filesystem will always get a performance hit (we pay for the new features in performance), that in itself is not interesting.

That's good - for Me a "feature" which kills the performance is well... not a feature, but rather a misconception in the design process.
And apparently I'm not alone in my judgement - search for "btrfs disable CoW" - You'll find many useful articles on how "fantastic" the BTRFS is in terms of performance with CoW enabled.
example:
https://unix.stackexchange.com/question ... emu-images

Some guy in the Phoronix comments points out how checksumming helped him notice a data corruption in his music collection, which he recovered immediately. He adds that on Ext4 it would have just lain there hidden for years until he would have noticed. That's a real thing

That's bullshit. The chances that he would get a CRC error instead of read error are practically zero - and obviously, it's impossible to investigate and confirm what have happened for "some guy" on "some forum" ...
Numbers do not lie - people do, sometimes unintentionally...

[pylkko]

We know that a "next gen" CoW filesystem will always get a performance hit (we pay for the new features in performance), that in itself is not interesting.

That's good - for Me a "feature" which kills the performance is well... not a feature, but rather a misconception in the design process.

Ok, but then in essence you are saying that btfs is a horrible, unusable technology because it cannot do something that is impossible. It has a "misconception in the design process" because it does something that no file system can ever do (CoW without a performance hit... even though CoW can be disabled if needed). And the evidence you are offering is that you say that you have an undisclosable "rare use case" where there is difference in practice ("it kills the performance"). I'm fine with that argument.

That's bullshit. The chances that he would get a CRC error instead of read error are practically zero - and obviously, it's impossible to investigate and confirm what have happened for "some guy" on "some forum" ...
Numbers do not lie - people do, sometimes unintentionally...

Numbers don't lie... so when you say "practically zero chance" you really expect anyone to believe that? Zero is a number, gee. It just doesn't work like that. Just offer the hard evidence for file system level checksumming not being possible or let it go. You made the claim, you need to prove it, nobody else. Thus far you offer nothing. And you will not, right?

Also, the article you offer is about using CoW on VM images. That is a special case that is common knowledge, even mentioned in the btrfs wiki. Keep them on another partition or switch of CoW since it is not needed. Not an argument against btrfs CoW performance in general, despite you claiming so.

[LE_746F6D617A7A69]

Ok, but then in essence you are saying that btfs is a horrible, unusable technology because it cannot do something that is impossible. It has a "misconception in the design process" because it does something that no file system can ever do. And the evidence you are offering is that you say that you have a "rare use case" where there is difference in practice. I'm fine with that argument.

In essence, CoW has to be disabled for most (if not all) write-intensive tasks, to avoid high performance drop and fragmentation.
This means data bases, virtual machines, temp dirs and logs, just to name few. In the effect, the snapshots also can't be used for all such cases, because this means CoW.
For read-mostly use cases both CoW and snapshots are OK.
I'm not saying that BTRFS is horrible - I'd rather say that it's not as fantastic as some people think

Numbers don't lie... so when you say "practically zero chance" you really expect anyone to believe that? Zero is a number, gee. It just doesn't work like that. Just offer the hard evidence for file system level checksumming not being possible or let it go. You made the claim, you need to prove it, nobody else. Thus far you offer nothing.

The probability of 1 bit in 10^15 bits comes from official datasheets for HDDs, and on the wiki pages You'll find an explanation of why ECC RAM has been invented - I'm not going to copy/paste those information here.

[Deb-fan]

One obvious consideration with Debian stable + btrfs.. Along these lines.

https://www.phoronix.com/scan.php?page= ... provements

A 4.19 kernel isn't ancient by any stretch and btrfs should have decent support with it regardless but has support vs improved support in newer kernel versions, yanno?

???

[Mr. Lumbergh]

It isn't just the filesystem itself, though, it's the other things that have to slow down to accommodate it. Reading/writing from storage is generally the slowest step in the chain, why knowingly add latency?

Sorry, but apparently I don't get what do You mean - what latency?
The writes are buffered, and even very old HDD will easily swallow ~40MB/s on average. Recording 8 audio streams sampled @96kHz/32bit gives 2.93MB/s ...

It isn't just the actual write itself that takes additional time, there are other resources such as CPU cycles and memory that are needed to execute the CoW commands, etc. that can slow things down in other places besides the drive. CPU ticks are the main currency when exporting a large mutlitrack file from a DAW, why add to overhead?