BTRFS on Debian Stable, your experiences?

[LE_746F6D617A7A69]

It isn't just the filesystem itself, though, it's the other things that have to slow down to accommodate it. Reading/writing from storage is generally the slowest step in the chain, why knowingly add latency?

Sorry, but apparently I don't get what do You mean - what latency?
The writes are buffered, and even very old HDD will easily swallow ~40MB/s on average. Recording 8 audio streams sampled @96kHz/32bit gives 2.93MB/s ...

Btrfs is fantastic for the development branches because snapshots

For development and experimenting I'm using Virtual Machines

[pylkko]

...for Me the show-stopper in BTRFS is a poor performance of databases and build systems -> so I have switched back to md-arrays + ext4.

Can you share what kind of databases you have and exactly what the performance difference is?

The non-ECC RAM is simply far less reliable than a classic HDD in terms of detecting so called "rot bits" -> so it makes completely no sense to use BTRFS checksums on desktop/laptop.

What do you mean when you say that non-ECC RAM or HDD is less reliable "in terms of detecting "bit rot"? And how do you come to the conclusion that "it makes no sense to use checksums"? Are you referring to the post on the kernel mailing list some years ago about RAM corruption undercutting FS level checks? If so, let's just state for the record that that is an interesting possible theoretical fringe case that has, as far as I know, never been proven to have actually even happen in real life at all...

Sorry, but apparently I don't get what do You mean - what latency?
The writes are buffered, and even very old HDD will easily swallow ~40MB/s on average. Recording 8 audio streams sampled @96kHz/32bit gives 2.93MB/s ...

To me, it appears that the issue here is that you are confusing latency with throughput.

For development and experimenting I'm using Virtual Machines

Is it possible to "diff" VM snapshots, for example? I want to learn to do that (if it is possible). But I would still say running a full VM for something that could simply be a folder that you can roll back is...strange. Why not git?

[LE_746F6D617A7A69]

@pylkko: it looks like You're a BTRFS fanboy - that's OK, but I'm not going to play that game. The tests on Phoronix, although incomplete, are showing what's the problem ...

non-ECC-RAM deserves a separate topic, but most of the info can be found on the wiki pages ...

Throughput has nothing to do with the latency, unless the I/O bandwidth limits are reached ...

[Bloom]

For those who care: I use LizardFS with JFS as my base filesystem. Works flawlessly. Up to 215 TiB now.

[pylkko]

@pylkko: it looks like You're a BTRFS fanboy - that's OK, but I'm not going to play that game. The tests on Phoronix, although incomplete, are showing what's the problem ...

I'm a fanboi of "the larger the claims, the better the evidence needs to be". When you hear that people say stuff like "using this tech is a total stupidity", or "it is deadly slow" you start to wonder how on earth they determined this one tech to be so totally unusable.

About those tests on Phoronix... we could also look the rest of them. Like the latest one where btrfs outperforms ext4 in many things, one of which, by the way, was SQlite databases.
https://www.phoronix.com/scan.php?page= ... tems&num=1
This makes it even more interesting that your databases are performing poorly. There are probably some additional factors at play...

Throughput has nothing to do with the latency, unless the I/O bandwidth limits are reached ...

Then why did you start to talk about theoretical throughput in MB/s when the guy asked about the latency?

As throughput and latency are often correlated (on the implementation level, like comparing different storage technologies), it seems quite an over statement to say "they have nothing to do with each other". But they can "decouple" in some situation. One such situation probably is the use of transparent compression. Apparently a btrfs partition using transparent compression can move a large file (if it compresses well) much faster than a partition without (for example ext4) with the processors of today. But, this will come with a latency cost, since the device needs to move a block and let the CPU compress it before even the first bit can be transferred. If you are doing latency sensitive work, these kinds of issues you would think about, no?

[LE_746F6D617A7A69]

Like the latest one where btrfs outperforms ext4 in many things, one of which, by the way, was SQlite databases.

Like I said, the tests on Phoronix are incomplete, because they are not showing what happens after some snapshots are taken - high write performance drop due to CoW activities.

Throughput has nothing to do with the latency, unless the I/O bandwidth limits are reached ...

Then why did you start to talk about theoretical throughput in MB/s when the guy asked about the latency?

I do realize that my knowledge of English is far from perfection, but it it seems that on the other hand, You have a problem with understanding what You're reading. Therefore I'll translate the above sentence specially for You:
As long as the bandwidth needed for audio stream recording is below the hardware limits, it does not affect the latency in the recording process. (thanks to buffered writes - mentioned earier)

Hope that helps...

[Deb-fan]

Thank you for arguing, errr i mean sharing guys. All kidding aside really mean that, some well stated points pylkko. Btrfs has longggggg been on my 2do list though as yet cant put it in my 2done list. Havent been paying all that much attention in quite awhile as to it's status, hearing its overtaking ext4 piques my interest though and makes a lot of sense to me. Am sure there's plenty to learning to properly setup, use and tune it correctly. More so than with ext4 and the performance mount options and tuning ive learned about with it. Again not surprising given the list of features btrfs sports.

Surely a fs like btrfs is receiving plenty of development attention for its application in Enterprise and production uses, aka: gnu/Linux's mainstay bread and butter. Only have research mostly to go on ... so gonna shut it. Definitely need to read this whole thread, thanks everyone who contributed to this discussion. Cool stuff.

[pylkko]

[Like I said, the tests on Phoronix are incomplete, because they are not showing what happens after some snapshots are taken - high write performance drop due to CoW activities.

Earlier, however, it appears you said that those Photronix tests do show what you argue...
if they don't, then what does show it?

Therefore I'll translate the above sentence specially for You:
As long as the bandwidth needed for audio stream recording is below the hardware limits, it does not affect the latency in the recording process.

Ok, but the guy was not asking about that.

[LE_746F6D617A7A69]

[Like I said, the tests on Phoronix are incomplete, because they are not showing what happens after some snapshots are taken - high write performance drop due to CoW activities.

Earlier, however, it appears you said that those Photronix tests do show what you argue...
if they don't, then what does show it?

Here is My first sentence about the tests on the Phoronix:
http://forums.debian.net/viewtopic.php? ... 15#p724943
Anyway, in most of the tests BTRFS is the slowest FS, even without CoW write amplification. The "latest" test is also flawed, because it was made only for single model of SSD (probably just promoting the product), and the ext4 was not optimized for SSDs (and BTRFS don't even have such option). I'm talking here about the Ext4 feature to define stripe and chunk sizes - so they can match the erase block size on the NVME/Flash devices - this reduces write amplification effects and extends the life-time of the device.

Therefore I'll translate the above sentence specially for You:
As long as the bandwidth needed for audio stream recording is below the hardware limits, it does not affect the latency in the recording process.

Ok, but the guy was not asking about that.

He said that BTRFS will likely cause some "performance hit" or introduce some undefined "latency" during recording of audio streams, which is obviously very unlikely because of very low requirements regarding the throughput for such task.

[pylkko]

I don't undertand why you bring up your "first sentence about Phoronix". We were not talking about that.

Let's not make decisions on file-system performance based on a cherry picked benchmark. Especially so if we need to then call any challenging benchmarks "flawed": The "flawed" benchmark is using the default settings of the OS. Any additional tweaks may change the outcomes slightly, but it is unlikely they would change the verdict. We know that a "next gen" CoW filesystem will always get a performance hit (we pay for the new features in performance), that in itself is not interesting. What is interesting is: does that matter? Does the synthetic benchmark translate to anything of any significance in real life? Is the magnitude of the performance hit large enough that any one would care or notice? I would say that the answer is: no, not even if you extrapolate from the benchmarks in which btrfs did not do so well. I really wonder in what situation in practice you would see a differnce. You said that you have some non-ordinary desktop situations where it matters, but you refuse to describe them further. This can very well be true, but it does not justify all of your "totally dead" and "100% stupidy" remarks at all. That is "throwing out the baby with the bathwater".

Some guy in the Phoronix comments points out how checksumming helped him notice a data corruption in his music collection, which he recovered immediately. He adds that on Ext4 it would have just lain there hidden for years until he would have noticed. That's a real thing, I don't think he cares if "in theory non-ECC RAM puts a loop whole in file-system level checksums". This does not mean that the point about RAM being possibly a weaker link is not real. But what conclusions do you make thereof, rationally? You know, practical real life stuff, not some theoretical/synthetic arguments.

This is the core of the argument that I tried to bring forward, although I could have for sure made it better.

[LE_746F6D617A7A69]

We know that a "next gen" CoW filesystem will always get a performance hit (we pay for the new features in performance), that in itself is not interesting.

That's good - for Me a "feature" which kills the performance is well... not a feature, but rather a misconception in the design process.
And apparently I'm not alone in my judgement - search for "btrfs disable CoW" - You'll find many useful articles on how "fantastic" the BTRFS is in terms of performance with CoW enabled.
example:
https://unix.stackexchange.com/question ... emu-images

Some guy in the Phoronix comments points out how checksumming helped him notice a data corruption in his music collection, which he recovered immediately. He adds that on Ext4 it would have just lain there hidden for years until he would have noticed. That's a real thing

That's bullshit. The chances that he would get a CRC error instead of read error are practically zero - and obviously, it's impossible to investigate and confirm what have happened for "some guy" on "some forum" ...
Numbers do not lie - people do, sometimes unintentionally...

[pylkko]

We know that a "next gen" CoW filesystem will always get a performance hit (we pay for the new features in performance), that in itself is not interesting.

That's good - for Me a "feature" which kills the performance is well... not a feature, but rather a misconception in the design process.

Ok, but then in essence you are saying that btfs is a horrible, unusable technology because it cannot do something that is impossible. It has a "misconception in the design process" because it does something that no file system can ever do (CoW without a performance hit... even though CoW can be disabled if needed). And the evidence you are offering is that you say that you have an undisclosable "rare use case" where there is difference in practice ("it kills the performance"). I'm fine with that argument.

That's bullshit. The chances that he would get a CRC error instead of read error are practically zero - and obviously, it's impossible to investigate and confirm what have happened for "some guy" on "some forum" ...
Numbers do not lie - people do, sometimes unintentionally...

Numbers don't lie... so when you say "practically zero chance" you really expect anyone to believe that? Zero is a number, gee. It just doesn't work like that. Just offer the hard evidence for file system level checksumming not being possible or let it go. You made the claim, you need to prove it, nobody else. Thus far you offer nothing. And you will not, right?

Also, the article you offer is about using CoW on VM images. That is a special case that is common knowledge, even mentioned in the btrfs wiki. Keep them on another partition or switch of CoW since it is not needed. Not an argument against btrfs CoW performance in general, despite you claiming so.

[LE_746F6D617A7A69]

Ok, but then in essence you are saying that btfs is a horrible, unusable technology because it cannot do something that is impossible. It has a "misconception in the design process" because it does something that no file system can ever do. And the evidence you are offering is that you say that you have a "rare use case" where there is difference in practice. I'm fine with that argument.

In essence, CoW has to be disabled for most (if not all) write-intensive tasks, to avoid high performance drop and fragmentation.
This means data bases, virtual machines, temp dirs and logs, just to name few. In the effect, the snapshots also can't be used for all such cases, because this means CoW.
For read-mostly use cases both CoW and snapshots are OK.
I'm not saying that BTRFS is horrible - I'd rather say that it's not as fantastic as some people think

Numbers don't lie... so when you say "practically zero chance" you really expect anyone to believe that? Zero is a number, gee. It just doesn't work like that. Just offer the hard evidence for file system level checksumming not being possible or let it go. You made the claim, you need to prove it, nobody else. Thus far you offer nothing.

The probability of 1 bit in 10^15 bits comes from official datasheets for HDDs, and on the wiki pages You'll find an explanation of why ECC RAM has been invented - I'm not going to copy/paste those information here.

[Deb-fan]

One obvious consideration with Debian stable + btrfs.. Along these lines.

https://www.phoronix.com/scan.php?page= ... provements

A 4.19 kernel isn't ancient by any stretch and btrfs should have decent support with it regardless but has support vs improved support in newer kernel versions, yanno?

???

[Mr. Lumbergh]

It isn't just the filesystem itself, though, it's the other things that have to slow down to accommodate it. Reading/writing from storage is generally the slowest step in the chain, why knowingly add latency?

Sorry, but apparently I don't get what do You mean - what latency?
The writes are buffered, and even very old HDD will easily swallow ~40MB/s on average. Recording 8 audio streams sampled @96kHz/32bit gives 2.93MB/s ...

It isn't just the actual write itself that takes additional time, there are other resources such as CPU cycles and memory that are needed to execute the CoW commands, etc. that can slow things down in other places besides the drive. CPU ticks are the main currency when exporting a large mutlitrack file from a DAW, why add to overhead?

[LE_746F6D617A7A69]

It isn't just the actual write itself that takes additional time, there are other resources such as CPU cycles and memory that are needed to execute the CoW commands, etc. that can slow things down in other places besides the drive.

No. Simply speaking, no matter how many audio streams You have to record in real-time, the thruoutput needed is just a small fraction of what a modern HDD/SSD can handle - unless we are talking about hundreds of audio streams, but I doubt it ...

CPU ticks are the main currency when exporting a large mutlitrack file from a DAW, why add to overhead?

For modern CPUs, the performance limit is the RAM bandwidth - which is around 25...45GiB/s for DDR3/4 - so a few MB/s for audio streams is not even noticeable.

[Mr. Lumbergh]

It isn't just the actual write itself that takes additional time, there are other resources such as CPU cycles and memory that are needed to execute the CoW commands, etc. that can slow things down in other places besides the drive.

No. Simply speaking, no matter how many audio streams You have to record in real-time, the thruoutput needed is just a small fraction of what a modern HDD/SSD can handle - unless we are talking about hundreds of audio streams, but I doubt it ...

CPU ticks are the main currency when exporting a large mutlitrack file from a DAW, why add to overhead?

For modern CPUs, the performance limit is the RAM bandwidth - which is around 25...45GiB/s for DDR3/4 - so a few MB/s for audio streams is not even noticeable.

Ask a drag racer why they don't run AC when they're at the track making a run. It's only 10 or so out of how many hundreds of horsepower, shouldn't be noticeable right? But it's that much less getting to wheels. In the same vein, why knowingly force CPU time to the copy on write operation when I'm more concerned about making sure the track export is going as smoothly as it can? There's a reason why system tuning is a big topic over on Linuxmusicians, every little bit adds up. This or that in and of itself may not make a noticeable difference, but on the aggregate they do.
I'm not going to argue the point further, it's clear that no minds are going to be changed here, so why?

[LE_746F6D617A7A69]

Ask a drag racer why they don't run AC when they're at the track making a run. It's only 10 or so out of how many hundreds of horsepower, shouldn't be noticeable right? But it's that much less getting to wheels. In the same vein, why knowingly force CPU time to the copy on write operation when I'm more concerned about making sure the track export is going as smoothly as it can? There's a reason why system tuning is a big topic over on Linuxmusicians, every little bit adds up. This or that in and of itself may not make a noticeable difference, but on the aggregate they do.
I'm not going to argue the point further, it's clear that no minds are going to be changed here, so why?

Are we talking about a real numbers or about feelings?
I don't think that anyone here is interested in "feelings" - those are meaningless and subjective - and IMO all the "musicians" or "audiophiles" are far from being objective in their judgements - especially, that most of them don't even have a basic idea about how the Hi-Fi amplifier works, not to mention the computers ...

No offense, but Your claim that BTRFS (or any other FS) can cause problems with audio recording simply has nothing to do with the reality (speaking gently ...)

[Mr. Lumbergh]

Ask a drag racer why they don't run AC when they're at the track making a run. It's only 10 or so out of how many hundreds of horsepower, shouldn't be noticeable right? But it's that much less getting to wheels. In the same vein, why knowingly force CPU time to the copy on write operation when I'm more concerned about making sure the track export is going as smoothly as it can? There's a reason why system tuning is a big topic over on Linuxmusicians, every little bit adds up. This or that in and of itself may not make a noticeable difference, but on the aggregate they do.
I'm not going to argue the point further, it's clear that no minds are going to be changed here, so why?

No offense, but Your claim that BTRFS (or any other FS) can cause problems with audio recording simply has nothing to do with the reality (speaking gently ...)

Read what I wrote. I never did make that claim, just I'd rather not add the overhead if I don't have to.

[LE_746F6D617A7A69]

Read what I wrote. I never did make that claim, just I'd rather not add the overhead if I don't have to.

So what is Your claim? - I just can't wait to see the final version ...
Can You define/measure that mysterious "overhead"?
What is the exact problem that You have with the BTRFS? (or any other FS?) , especially with a reference to a Debian stable? Please share Your experience, but please not to forget to include a real-time measurements of the dalays ... if You have any ...