Aki wrote: ↑2023-07-15 09:23
If it is possible, it could be interesting to replicate it. Could it be possible for you to share a proof of concept (as terminal commands) that replicates the issue on your computer ?
Oops, my account was locked for an unknown reason, but after some weeks the admin unlocked it thankfully (thanks!).
Anyway.
My advice if you want to replicate my 'not so complex scenario', would be to edit/create the /etc/sysctl.d/40-dirty.conf file and try any of these settings AND the defaults. And on each boot transfer at least 16GiB of data from one HDD to another HDD (if possible one of them external), it is optional to divide the data into 500mb files, this to mimic borg's 'segment files' but maybe it is not necessary, it was just the task I was doing when I noticed this limitation (except it was about 600+GiB). Anyway, depending on the setting, after about 4.1GiB of transferred data, the system will have a hard time to keep up, and it will get worse, and worse. The more data you try to transfer... the worst, specially with the defaults.
Settings:
vm.dirty_background_ratio = 1
vm.dirty_ratio = 2
or
vm.dirty_background_bytes = 4194304
vm.dirty_bytes = 4194304
this on a system with 16384 MB RAM installed
A longer explanation to the test I did:
to keep things simple (i hope) I am going to call
sda the hdd receiving/writing data, and
sdb the drive transferring/reading the data.
the first setting (ratio), seems to be the most balanced one I don't get any lockups! and the speed seems to be mostly stable at least checking with iotop-c, I mean it has a few pauses, but I presume it is flushing cache to sda because the activity on both sdb's dock LED and iotop-c 'disk read' output cease (0,00B), but sda is writing, after it finishes writing about 1GiB or 500MB, everything stops for a few seconds, and then sdb resumes activity and the whole process repeats itself. It is not the best performance I've had, but it gets the job done.
the second setting (bytes), if I recall correctly it gets only a few lockups, but it is mostly 'stable', the write speed is slow in my case... but the reads and writes are mostly frequent, sdb is pretty much always active, same sda, but the write speed is just very low, like 2MB or 10MB at best.
and there is the original option which is debian's defaults (no 40-dirty.conf file but the ratios are 20 and 10 iirc), this one being the worst configuration for my case, it has TONS, and I mean TONS of deadlocks, the system will hang for SEVERAL minutes depending on how much data is being transferred, it will hang so bad that even sdb's dock LED will enter into some kind of stand-by mode (the LED will power off, only to blink once after a few seconds, meaning there is no activity whatsoever), once it resumes sdb will read like, 100 or 300~mb at best, then, sda will write a very tiny amount of data (VERY slowly) only to get another deadlock... it's just sad! and frustrating to see.