[Discussion] Why are severely broken point releases being released for Bookworm

[Chaussettes]

Has there been any discussion or communication with the Debian development team as to why we've had at least 3 back-to-back point releases of stable that have broken critical components of the system?

Debian 12.3 point release caused a critical ext4 file system data corruption bug (https://bugs.debian.org/cgi-bin/bugrepo ... ug=1057843)
Debian 12.4 point release broke wifi capability for a large portion of users
Debian 12.5 point release has completely broken NVIDIA graphics drivers (https://bugs.debian.org/cgi-bin/bugrepo ... ug=1063363)(https://bugs.debian.org/cgi-bin/bugrepo ... ug=1062932)

I can understand problems popping up after a large release. The Debian development team can not possibly test every single configuration and rely on bug reports to fix things. However, I would like to point out that the latest 12.5 NVIDIA graphics drivers bugs were known about BEFORE the 12.5 release was made. Bug 1063363 and Bug 1062932 were filed and discussed on the bug tracker an entire week before 12.5 was released. Why are point releases being made that are known to cause significant issues that will affect a large portion of the user base? I love Debian and will always stick with it, but this is a little ridiculous.

[steve_v]

a large portion of the desktop/laptop user base

FTFY.
The majority of Debian (and GNU/Linux in general) installations are on servers, which don't care one iota about wireless (especially with closed-source blobs) or nvidia being nvidia and not keeping up with kernel changes.

As for the ext4 bug, that's unfortunate, but it wasn't discovered until the affected kernels had already been deployed in several major distros.

[pizza-rat]

I'm having flashbacks to when I used to daily drive Arch.

[Chaussettes]

a large portion of the desktop/laptop user base

FTFY.
The majority of Debian (and GNU/Linux in general) installations are on servers, which don't care one iota about wireless (especially with closed-source blobs) or nvidia being nvidia and not keeping up with kernel changes.

As for the ext4 bug, that's unfortunate, but it wasn't discovered until the affected kernels had already been deployed in several major distros.

I guess I just don't exactly understand the point in deciding to push a point release update with critical bugs that break userspace when you actively know that the bug exists, like with the current 12.5 NVIDIA issue. I understand Linux is primarily a server choice, but to decide to kick the desktop users to the curb and ignore huge issues like this is just silly.

[bbbhltz]

I would say that the mailing lists are a better place to ask this question, but the developers do keep an eye on things here. I too had a "it's weird that it's happened twice thrice" moment, but in no way thought of it as ignoring desktop users. Development is a community effort, and for some reason the community is missing these issues or ignoring them or skipping a step along the line. The Nvidia bug? I would chalk that one up to honest human error.

One thing is certain, the are probably asking themselves the same question.

[donald]

a large portion of the desktop/laptop user base

FTFY.
The majority of Debian (and GNU/Linux in general) installations are on servers, which don't care one iota about wireless (especially with closed-source blobs) or nvidia being nvidia and not keeping up with kernel changes.

As for the ext4 bug, that's unfortunate, but it wasn't discovered until the affected kernels had already been deployed in several major distros.

I guess I just don't exactly understand the point in deciding to push a point release update with critical bugs that break userspace when you actively know that the bug exists, like with the current 12.5 NVIDIA issue. I understand Linux is primarily a server choice, but to decide to kick the desktop users to the curb and ignore huge issues like this is just silly.

Ideally people would research such opinions prior to posting them online, but they are just that opinions and valid for nothing other than a social media stance.

[donald]

Prior to each release, the notes come out 4 to 5 days prior, these are the notes that end users should read. Several users here post the release notes or security updates for one to read prior to the end user making the decision to update/upgrade a live or production system. But the notes are not just here, we post them and share them on the user lists, we also are very open about bugs, race condidtions, kernel errors, or any other thing that will turn a user experience into a nightmare. Debian like any distributor can only point to advisories, we cannot enforce them.

We strive for excellence, but we advise on the rocky path that can occur on the shared journey.

Join the mailing lists.

Join the testing teams, who are a critical component of Debian and Debian releases.

Subscribe to the bugs listings to see things that may affect you or your systems.

Join the user groups to find quick solutions to issues and spread work arounds or new information quickly.

...or post online which helps no one.

12.3 ext3 bug was caught literally as we were doing the pre-release testing. The error was not on Debian, but from upstream, but we caught it, we fixed it, we pushed another version out in under 48 hours which is unheard of in software development. If you were in the release channel, you would be aware of this and the substantial effort behind 12.4 the same weekend. No one slept for days.

ChangeLog-6.1.67 addressed a wifi issue.

12.4 Bug was known prior to the release.

12.5 Bug also known prior. And being worked on as we speak.

Want to see how Debian fixes things? Join the mailing lists, the irc channels, talk to a developer, come and see the work as the work is being done.

Not being defensive, though it may read that way. I just know *slightly* more on this topic and see the DDs working tirelessly to release an amazing project, have to speak up for them.

[pizza-rat]

@donald
As somebody who isn't a developer and doesn't really have the time to chip in or keep up with a mailing list, is there something simple like an RSS feed I can subscribe to in order to be alerted to things like this? I have the Debian News RSS feed but the item on this point release had nothing on this, the only way I caught it before I updated was that I happened to check the forums and see a thread about it.

Also, what is the logic behind knowing there is a major bug in a potential release and then releasing it anyway and rushing to fix it/losing sleep versus just delaying until it's fixed? I'm asking out of genuine curiosity as somebody who, again, is not a developer.

[donald]

@donald
As somebody who isn't a developer and doesn't really have the time to chip in or keep up with a mailing list, is there something simple like an RSS feed I can subscribe to in order to be alerted to things like this? I have the Debian News RSS feed but the item on this point release had nothing on this, the only way I caught it before I updated was that I happened to check the forums and see a thread about it.

The https://micronews.debian.org service is pretty good with up to date or immediate items, the feed pushes here as well. For most people that feed would be current and abreast of things, though as you mention these forums are also pretty quick to pick up on things. Most of our news feeds push here as well to bring more content and know how to end users and people.

I think a better answer for you would be to join the bugs, release, and (item of interest) mailing lists. Those lists will give you a better idea of what is going on inside the project than the news from the project that you can/will get 2-3 hours later from the project. In complete irony, I author and my team author most of the news everyone reads as we are the publicity and press teams, but Debian is a lot bigger than it looks so sometimes there is news that we do not see ... but assume everyone knows like some bugs for example.

You do not need to study them or read them daily, but a subscription and a good email filter rule can keep you current with these example lists:

debian-backports:
debian-kde:
debian-security: ( a must. It is high volume though)
debian-accessibility: Making Debian more accessible to people with disabilities
debian-ai: Debian's Machine Learning & Hardware Acceleration
debian-boot: Developing the installation system
debian-devel: Development of Debian ( a must)
debian-mentors: Helping newbie developers

All of the lists are here: https://lists.debian.org

Keeping up with things, especially in todays times is and can be very difficult, you don't need to as I mentioned study the lists, but poke your head in ever so often and you can have a better idea of the process and progress of development.

Also, what is the logic behind knowing there is a major bug in a potential release and then releasing it anyway and rushing to fix it/losing sleep versus just delaying until it's fixed? I'm asking out of genuine curiosity as somebody who, again, is not a developer.

That is a valid question. Debian always does version releases 'When its ready", so we take the time to do the release right, but even so there are going to be some issues in the release, either upstream or something breaks, or a developer did not have access to a particular architecture and used a box instead. There are many factors behind it, some good and to be fair some bad, some in our control and some outside of our control.

In my personal opinion, not a project opinion, I think like most products driven to market that things do get pushed out so that 1 item if it is not a *major* item, does not hold up the progress of all the other work, I think this is especially important if someone or a team is actively working on the process. of fixing or improving said issue or item. These things are easier to catch between version 19 and 20 as something can be excluded from the next version, but that cannot always be done when a version of something is already in the current release. Same as the present ashtray in car model year 2045 but missing and replaced with a USB charging station in the 2046 model year. Ashtray was fine, but it cut fingers, too late to re-tool the entire car assembly line.

Sometimes Debian does include something that *may* break systems, but we always encourage users to read the release notes prior to upgrading to understand what risks or what software has changed so that systems do not break. Reading the lists, other users input in places like here helps as well to circumvent such issues.

Sometimes it is okay to wait between point releases or follow the bug reports on something to work around the issue while it is fixed.

Hope that makes sense.

The other part to your question? Is just ask us. Debian is really pretty full of active people, but people that are human beings. We strive to do the best we can.

[Uptorn]

Friendly reminder that the Debian project has been estimated to cost at least $1,500,000,000, measured in commercial salaried compensation. That was in 2016. It is probably much higher today.

[None1975]

The interesting thing is that Debian version 11 didn't have these problems, which makes me think that Debian's quality is going down...
​

[lindi]

I'm a developer but this is not any kind of official statement. Personally I think a major weakness is that we do not have good comprehensive automated testing for changes. This is hard to do especially with real hardware but I don't see any easier ways to improve quality.

[Uptorn]

The interesting thing is that Debian version 11 didn't have these problems, which makes me think that Debian's quality is going down...
​

For one thing, the total package count has been growing and growing. But there is no way this is sustainable, unless maybe the number of maintainers grows proportionately. But then logistics issues arise. At some point, there is going to have to be a Debian release that has fewer packages available than the previous. I'd rather see efforts go into improving the quality of software than the quantity of it.

[pwzhangzz]

For obvious reasons I totally think the title of this thread is grossly unfair. We are using Debian Bkworm for free as our desktop OS and have recently upgraded to 12.5 without any problem WHATSOEVER.

[steve_v]

critical bugs that break userspace... the current 12.5 NVIDIA issue... huge issues like this.

Nvidia drivers failing to build against a particular kernel version isn't unusual, unexpected, or "critical". Nor is it a "huge issue", or "breaking userspace".
What it is, is a problem with a third-party proprietary driver Debian has zero control over, that affects only a small subset of the userbase, for which a relatively straightforward workaround exists.

It's also by rights nvidias problem to sort out, and if you want to run such proprietary drivers that's just something you'll have to deal with from time to time.

just silly

What would be even sillier is holding back kernel updates for everyone else, just because one proprietary vendor is unreasonably slow in responding to kernel changes.

Frankly it's pretty impressive how rare problems with that particular driver are in Debian. Try tracking the mainline kernel for a while and see how often nvidia drops the ball when the debian maintainers aren't around to do their testing for them... Or just ask an average Arch user

[cds60601]

just silly

What would be even sillier is holding back kernel updates for everyone else, just because one proprietary vendor is unreasonably slow in responding to kernel changes.

Frankly it's pretty impressive how rare problems with that particular driver are in Debian. Try tracking the mainline kernel for a while and see how often nvidia drops the ball when the debian maintainers aren't around to do their testing for them... Or just ask an average Arch user

This is exactly why I have used AMD cards.

[LeeE]

I ended up here as a result of investigating the Nvidia driver issue and the title of this thread resonated with me. As someone who has been using Debian since slink, on all my workstations and servers, I've certainly experienced a few breakages and accept that, from time to time, unforeseen breakages will occur, such as the filesystem bug in 6.1.66.

The issue with the Nvidia drivers was not unforeseen though, yet it was still decided to push this release out knowing that it would break many systems.

It seemed obvious to me that the purpose of this thread, posed as a question, was not to seek specific answers to the question posed by the thread title but to raise the general issue of known broken point releases.

The real question is should Debian users assume that point releases are broken if they are to avoid interrupting their work - the work they do on their Debian systems - to resolve problems caused by broken point releases, and instead delay updates until they're confident that they won't run into problems, or should Debian delay point releases that are known to break many users systems until the foreseen breakages can be addressed?

I think that most people who use Debian to do their work would prefer point releases to be delayed until known breakages are fixed and, quite frankly, I can't see any good arguments to counter that view.

I don't find the arguments that have been presented here, that appear to support the view that there is no issue or that the issue is invalid, convincing.

The comment that "The majority of Debian (and GNU/Linux in general) installations are on servers, which don't care one iota about wireless (especially with closed-source blobs) or nvidia being nvidia and not keeping up with kernel changes." was actually a bit insulting to the OP - there was nothing "fixed" by that response, which was incorrect in both of its points; wifi is a very valid means of running system monitoring networks in DCs, and GPUs in servers are very much a thing, for a number of different uses.

Furthermore, by referencing "The majority" there is an implication that only a very small number of users have been affected: there have currently been > 20,000 views across the different threads about this 12.5 nvidia issue on this forum.

The suggestion that Debian users should subscribe to mailing lists is not a bad idea but it is rather akin to requiring that everyone who drives an auto-mobile should also be able to maintain and repair it. There is nothing wrong with doing this if that is what a user wishes to do - indeed, back on slink I recall having having to work on custom mode-settings for CRT displays, to avoid damaging them, and that sort of thing was normal, at the time, but shouldn't be expected these days.

I think that all users of Debian - those who have chosen Debian to do their work - are very much aware of how much time and effort it takes to make Debian available, and are very grateful for all of the work done by its contributors and maintainers. I think that no attack was intended by the OP, upon any of the people who give us Debian, and I certainly don't wish to attack Debian either - I think it's brilliant. But, as a user, I still think that there is an issue to be resolved, or at least clarified.

As someone who does quite a bit of coding as well as sysadmin I'm a bit surprised that this problem couldn't have been handled by the package dependency hooks. As I compile custom kernels (from the deb source) for all of my systems, I can't use the nvidia packages in Debian's repos but the nvidia download driver package installs into predictable locations and should be able to be detected by a dependency hook and identified as a conflict. Having said that, I haven't needed to look into Debian package dependency hooks so don't know how trivial it would be - I can only base it on similar stuff I've worked on. In any case, the nvidia issue is likely to be resolved in the next week or so, one way or another, despite the ideological reasons it occurred in the first place.

[CwF]

Isn't the standard response - "a fraction of a minority within a select group of a few"

Nvidia breakage has been par for the course since I remember.

I'm a bit surprised that this problem couldn't have been handled by the package dependency hooks.

With this I agree. Listchanges and buglist often are not referenced by 'stable' users but something should stand in the way of the upgrade for those 'few users' while not impeding the progress of others (in the majority).

To delay the point release because of Nvidia should not be considered an option.

[steve_v]

actually a bit insulting to the OP

It wasn't really intended to be, at least not specifically WRT the OP. It was snarky though, because frankly I'm running low on patience for some of the prevailing attitudes around here of late.
Had the OP left out such phrasing as "severely broken" "critical components" and "a little ridiculous" while also referring to nvidia and wifi drivers (which are anything but critical and have a long history of problems, mainly due to uncooperative proprietary vendors), they wouldn't have been the one to cop that.

Bugs happen, and sometimes stuff breaks. Stuff involving integration of proprietary software and/or drivers breaks far more often. Such is the nature of the beast.
Nvidia issues in particular are something we have been dealing with for decades, and they're not the end of the world. The proprietary nvidia driver is certainly not a "critical component of the system", and needing to hold back a kernel update for a week or two on certain hardware configurations is a minor inconvenience at worst.

The only thing that's "a little ridiculous" here is that most Debian releases have so few bugs...
Or perhaps lumping a proprietary driver FTBFS in with a rather nasty filesystem bug in the kernel, as if they're in some way comparable.


As for reading mailing lists, that's where development happens. If you don't want to stay informed or participate (AKA engage with FOSS as it was intended, as opposed to treating it like a product), that's up to you... But this here is a user board, nobody has a hotline to the developers or maintainers.

[LeeE]

To delay the point release because of Nvidia should not be considered an option.

I think this statement needs justification with regard to why the timing of point releases should be considered critical and thus cannot be delayed.

While I think everyone would accept that the timing and release of security updates is critical and should not be delayed, these are handled outside the point release scheme; they must be because security updates must not be delayed until a point release is ready to be issued.

I can see why maintaining a point release schedule is desirable, but I cannot see why it is critical.