Scheduled Maintenance: We are aware of an issue with Google, AOL, and Yahoo services as email providers which are blocking new registrations. We are trying to fix the issue and we have several internal and external support tickets in process to resolve the issue. Please see: viewtopic.php?t=158230

 

 

 

Debian update caused major breakage in my workflow regarding

Linux Kernel, Network, and Services configuration.
Message
Author
Harkonnen
Posts: 24
Joined: 2011-08-31 12:52

Debian update caused major breakage in my workflow regarding

#1 Post by Harkonnen »

Today i updated my debian jessie system (a routine update, security updates and multimedia/browsers, last one was ~month ago), and after reboot a lot of my stuff that uses ip namespaces to play some network routing tricks on some programs stopped worked.

Here is a setup. My PC uses autofs and sshf to mount shares from two NASes, this is a line from autofs config

Code: Select all

-fstype=fuse.sshfs,reconnect,workaround=all,rw,nodev,nonempty,noatime,allow_other,max_read=65536,Ciphers=arcfour256,Compression=n    
I also use ip namespaces to run some programs with different network configurations.

Before the update, programs started via ip netns exec had no problem accessing autofs mounts and did not cause any problems for default namespace. Now the following is happening:

Programs started via ip netns exec cannot access autofs mounts unless i access them in default namespace right before starting them. This would have been only minor inconvenience, but after timeout passes and autofs unmounts shares in default namespace it refuses to mount them back while ip netns exec program is running! The mount point exists, but attempt to cd to it results in "too many levels of symbolic links", mount shows that share is not mounted in default namespace.

I googled around and apparently this is caused by ip netns exec creating also a mount namespace for whatever reason. Adding ",shared" to the mount line changes nothing. Looking in /var/cache/apt it seems that neither iproute2, autofs or kernel was updated. But it worked before the update!

So, does anybody have any idea update to what caused this so i can downgrade, or any workaround for this other than ditching autofs and permamounting all shares at the start?

User avatar
dasein
Posts: 7680
Joined: 2011-03-04 01:06
Location: Terra Incantationum

Re: Debian update caused major breakage in my workflow regar

#2 Post by dasein »

Just restore from a recent backup. Problem solved.

Harkonnen
Posts: 24
Joined: 2011-08-31 12:52

Re: Debian update caused major breakage in my workflow regar

#3 Post by Harkonnen »

And then what? Don't update at all?

User avatar
dilberts_left_nut
Administrator
Administrator
Posts: 5346
Joined: 2009-10-05 07:54
Location: enzed
Has thanked: 12 times
Been thanked: 66 times

Re: Debian update caused major breakage in my workflow regar

#4 Post by dilberts_left_nut »

Can you compare what has changed?

Stable updates "shouldn't" break working configurations, so maybe a bug report might be in order.

It's probably a fairly niche set of circumstances you are dealing with, so unless you can do some troubleshooting to pin down the cause there's likely not much we (as Debian users) can suggest.
AdrianTM wrote:There's no hacker in my grandma...

Harkonnen
Posts: 24
Joined: 2011-08-31 12:52

Re: Debian update caused major breakage in my workflow regar

#5 Post by Harkonnen »

Well, as i said, the obvious culprits - autofs, iproute2 and kernel wasnt updated, so this was probably caused by some security update to some lib, but it looks like i haven't rebooted my pc for ~3 months, and over this time ~250 packages received some updates. That is why i am asking if anyone have any idea what can cause this, because i am not going to install two hundred packages one in time with reboots in between :)

The most amazing thing is that after googling more i found out that apparently autofs and mount namespaces is a known problem, with a known workaround but my setup somehow worked for years without it :)

P.S. For those curious the workaround is mount --make-shared /path/to/root-of-autofs after autofs started. Notice that it is a root of autofs that has to be shared, not individual mounts, that's why -o shared doesnt work in autofs config.

User avatar
Head_on_a_Stick
Posts: 14114
Joined: 2014-06-01 17:46
Location: London, England
Has thanked: 81 times
Been thanked: 132 times

Re: Debian update caused major breakage in my workflow regar

#6 Post by Head_on_a_Stick »

Please post the full output of:

Code: Select all

apt-cache policy
deadbang

Harkonnen
Posts: 24
Joined: 2011-08-31 12:52

Re: Debian update caused major breakage in my workflow regar

#7 Post by Harkonnen »

Here it is:

Code: Select all

Package files:
 100 /var/lib/dpkg/status
     release a=now
 500 http://mirror.ppa.trinitydesktop.org/trinity/trinity-builddeps-r14.0.0/debian/ jessie/main i386 Packages
     release v=8.0,o=LP-PPA-trinity-trinity-builddeps-r14.0.0,a=jessie,n=jessie,l=Trinity Build Dependencies R14.0.0,c=main
     origin mirror.ppa.trinitydesktop.org
 500 http://mirror.ppa.trinitydesktop.org/trinity/trinity-builddeps-r14.0.0/debian/ jessie/main amd64 Packages
     release v=8.0,o=LP-PPA-trinity-trinity-builddeps-r14.0.0,a=jessie,n=jessie,l=Trinity Build Dependencies R14.0.0,c=main
     origin mirror.ppa.trinitydesktop.org
 500 http://mirror.ppa.trinitydesktop.org/trinity/trinity-r14.0.0/debian/ jessie/main i386 Packages
     release v=8.0,o=LP-PPA-trinity-trinity-r14.0.0,a=jessie,n=jessie,l=Trinity Desktop Environment R14.0.0,c=main
     origin mirror.ppa.trinitydesktop.org
 500 http://mirror.ppa.trinitydesktop.org/trinity/trinity-r14.0.0/debian/ jessie/main amd64 Packages
     release v=8.0,o=LP-PPA-trinity-trinity-r14.0.0,a=jessie,n=jessie,l=Trinity Desktop Environment R14.0.0,c=main
     origin mirror.ppa.trinitydesktop.org
 500 http://deb.playonlinux.com/ jessie/main i386 Packages
     release v=8.0,o=PlayOnLinux,a=jessie,n=jessie,l=PoL,c=main
     origin deb.playonlinux.com
 500 http://deb.playonlinux.com/ jessie/main amd64 Packages
     release v=8.0,o=PlayOnLinux,a=jessie,n=jessie,l=PoL,c=main
     origin deb.playonlinux.com
 500 http://download.opensuse.org/repositories/home:/stevenpusser/Debian_8.0/  Packages
     release o=obs://build.opensuse.org/home:stevenpusser/Debian_8.0,n=Debian_8.0,l=home:stevenpusser,c=
     origin download.opensuse.org
 500 http://download.opensuse.org/repositories/home:/lintest/Debian_7.0/  Packages
     release o=obs://build.opensuse.org/home:lintest/Debian_7.0,n=Debian_7.0,l=home:lintest,c=
     origin download.opensuse.org
 650 http://deb.i2p2.no/ stable/main i386 Packages
     release v=8.0,o=I2P,a=stable,n=jessie,l=I2P,c=main
     origin deb.i2p2.no
 650 http://deb.i2p2.no/ stable/main amd64 Packages
     release v=8.0,o=I2P,a=stable,n=jessie,l=I2P,c=main
     origin deb.i2p2.no
 500 http://ppa.launchpad.net/hizo/mkv-extractor-gui/ubuntu/ trusty/main Translation-en
 500 http://ppa.launchpad.net/hizo/mkv-extractor-gui/ubuntu/ trusty/main i386 Packages
     release v=14.04,o=LP-PPA-hizo-mkv-extractor-gui,a=trusty,n=trusty,l=MKV Extractor Gui,c=main
     origin ppa.launchpad.net
 500 http://ppa.launchpad.net/hizo/mkv-extractor-gui/ubuntu/ trusty/main amd64 Packages
     release v=14.04,o=LP-PPA-hizo-mkv-extractor-gui,a=trusty,n=trusty,l=MKV Extractor Gui,c=main
     origin ppa.launchpad.net
 650 http://dl.google.com/linux/talkplugin/deb/ stable/main i386 Packages
     release v=1.0,o=Google, Inc.,a=stable,n=stable,l=Google,c=main
     origin dl.google.com
 650 http://dl.google.com/linux/talkplugin/deb/ stable/main amd64 Packages
     release v=1.0,o=Google, Inc.,a=stable,n=stable,l=Google,c=main
     origin dl.google.com
 650 http://dl.google.com/linux/chrome/deb/ stable/main amd64 Packages
     release v=1.0,o=Google, Inc.,a=stable,n=stable,l=Google,c=main
     origin dl.google.com
 500 http://linux.dropbox.com/debian/ jessie/main i386 Packages
     release o=Dropbox.com,a=jessie,n=jessie,l=Dropbox Debian Repository,c=main
     origin linux.dropbox.com
 500 http://linux.dropbox.com/debian/ jessie/main amd64 Packages
     release o=Dropbox.com,a=jessie,n=jessie,l=Dropbox Debian Repository,c=main
     origin linux.dropbox.com
 500 http://ppa.launchpad.net/deluge-team/ppa/ubuntu/ trusty/main Translation-en
 500 http://ppa.launchpad.net/deluge-team/ppa/ubuntu/ trusty/main i386 Packages
     release v=14.04,o=LP-PPA-deluge-team,a=trusty,n=trusty,l=PPA for Deluge Team,c=main
     origin ppa.launchpad.net
 500 http://ppa.launchpad.net/deluge-team/ppa/ubuntu/ trusty/main amd64 Packages
     release v=14.04,o=LP-PPA-deluge-team,a=trusty,n=trusty,l=PPA for Deluge Team,c=main
     origin ppa.launchpad.net
 500 http://repo.acestream.org/ubuntu/ precise/main i386 Packages
     release o=repo.acestream.org/ubuntu,n=precise,l=acestream apt repository,c=main
     origin repo.acestream.org
 500 http://repo.acestream.org/ubuntu/ precise/main amd64 Packages
     release o=repo.acestream.org/ubuntu,n=precise,l=acestream apt repository,c=main
     origin repo.acestream.org
 500 http://www.deb-multimedia.org/ jessie/non-free Translation-en
 500 http://www.deb-multimedia.org/ jessie/main Translation-en
 650 http://www.deb-multimedia.org/ jessie/non-free i386 Packages
     release v=8.7,o=Unofficial Multimedia Packages,a=stable,n=jessie,l=Unofficial Multimedia Packages,c=non-free
     origin www.deb-multimedia.org
 650 http://www.deb-multimedia.org/ jessie/main i386 Packages
     release v=8.7,o=Unofficial Multimedia Packages,a=stable,n=jessie,l=Unofficial Multimedia Packages,c=main
     origin www.deb-multimedia.org
 650 http://www.deb-multimedia.org/ jessie/non-free amd64 Packages
     release v=8.7,o=Unofficial Multimedia Packages,a=stable,n=jessie,l=Unofficial Multimedia Packages,c=non-free
     origin www.deb-multimedia.org
 650 http://www.deb-multimedia.org/ jessie/main amd64 Packages
     release v=8.7,o=Unofficial Multimedia Packages,a=stable,n=jessie,l=Unofficial Multimedia Packages,c=main
     origin www.deb-multimedia.org
 500 http://security.debian.org/ jessie/updates/non-free Translation-en
 500 http://security.debian.org/ jessie/updates/main Translation-en
 500 http://security.debian.org/ jessie/updates/contrib Translation-en
 650 http://security.debian.org/ jessie/updates/non-free i386 Packages
     release v=8,o=Debian,a=stable,n=jessie,l=Debian-Security,c=non-free
     origin security.debian.org
 650 http://security.debian.org/ jessie/updates/contrib i386 Packages
     release v=8,o=Debian,a=stable,n=jessie,l=Debian-Security,c=contrib
     origin security.debian.org
 650 http://security.debian.org/ jessie/updates/main i386 Packages
     release v=8,o=Debian,a=stable,n=jessie,l=Debian-Security,c=main
     origin security.debian.org
 650 http://security.debian.org/ jessie/updates/non-free amd64 Packages
     release v=8,o=Debian,a=stable,n=jessie,l=Debian-Security,c=non-free
     origin security.debian.org
 650 http://security.debian.org/ jessie/updates/contrib amd64 Packages
     release v=8,o=Debian,a=stable,n=jessie,l=Debian-Security,c=contrib
     origin security.debian.org
 650 http://security.debian.org/ jessie/updates/main amd64 Packages
     release v=8,o=Debian,a=stable,n=jessie,l=Debian-Security,c=main
     origin security.debian.org
 500 http://ftp.debian.org/debian/ jessie-backports/non-free Translation-en
 500 http://ftp.debian.org/debian/ jessie-backports/main Translation-en
 500 http://ftp.debian.org/debian/ jessie-backports/contrib Translation-en
 100 http://ftp.debian.org/debian/ jessie-backports/non-free i386 Packages
     release o=Debian Backports,a=jessie-backports,n=jessie-backports,l=Debian Backports,c=non-free
     origin ftp.debian.org
 100 http://ftp.debian.org/debian/ jessie-backports/contrib i386 Packages
     release o=Debian Backports,a=jessie-backports,n=jessie-backports,l=Debian Backports,c=contrib
     origin ftp.debian.org
 100 http://ftp.debian.org/debian/ jessie-backports/main i386 Packages
     release o=Debian Backports,a=jessie-backports,n=jessie-backports,l=Debian Backports,c=main
     origin ftp.debian.org
 100 http://ftp.debian.org/debian/ jessie-backports/non-free amd64 Packages
     release o=Debian Backports,a=jessie-backports,n=jessie-backports,l=Debian Backports,c=non-free
     origin ftp.debian.org
 100 http://ftp.debian.org/debian/ jessie-backports/contrib amd64 Packages
     release o=Debian Backports,a=jessie-backports,n=jessie-backports,l=Debian Backports,c=contrib
     origin ftp.debian.org
 100 http://ftp.debian.org/debian/ jessie-backports/main amd64 Packages
     release o=Debian Backports,a=jessie-backports,n=jessie-backports,l=Debian Backports,c=main
     origin ftp.debian.org
 500 http://ftp.debian.org/debian/ jessie-updates/non-free Translation-en
 500 http://ftp.debian.org/debian/ jessie-updates/main Translation-en
 500 http://ftp.debian.org/debian/ jessie-updates/contrib Translation-en
 500 http://ftp.debian.org/debian/ jessie-updates/non-free i386 Packages
     release o=Debian,a=stable-updates,n=jessie-updates,l=Debian,c=non-free
     origin ftp.debian.org
 500 http://ftp.debian.org/debian/ jessie-updates/contrib i386 Packages
     release o=Debian,a=stable-updates,n=jessie-updates,l=Debian,c=contrib
     origin ftp.debian.org
 500 http://ftp.debian.org/debian/ jessie-updates/main i386 Packages
     release o=Debian,a=stable-updates,n=jessie-updates,l=Debian,c=main
     origin ftp.debian.org
 500 http://ftp.debian.org/debian/ jessie-updates/non-free amd64 Packages
     release o=Debian,a=stable-updates,n=jessie-updates,l=Debian,c=non-free
     origin ftp.debian.org
 500 http://ftp.debian.org/debian/ jessie-updates/contrib amd64 Packages
     release o=Debian,a=stable-updates,n=jessie-updates,l=Debian,c=contrib
     origin ftp.debian.org
 500 http://ftp.debian.org/debian/ jessie-updates/main amd64 Packages
     release o=Debian,a=stable-updates,n=jessie-updates,l=Debian,c=main
     origin ftp.debian.org
 500 http://ftp.debian.org/debian/ jessie/non-free Translation-en
 500 http://ftp.debian.org/debian/ jessie/main Translation-en
 500 http://ftp.debian.org/debian/ jessie/contrib Translation-en
 650 http://ftp.debian.org/debian/ jessie/non-free i386 Packages
     release v=8.7,o=Debian,a=stable,n=jessie,l=Debian,c=non-free
     origin ftp.debian.org
 650 http://ftp.debian.org/debian/ jessie/contrib i386 Packages
     release v=8.7,o=Debian,a=stable,n=jessie,l=Debian,c=contrib
     origin ftp.debian.org
 650 http://ftp.debian.org/debian/ jessie/main i386 Packages
     release v=8.7,o=Debian,a=stable,n=jessie,l=Debian,c=main
     origin ftp.debian.org
 650 http://ftp.debian.org/debian/ jessie/non-free amd64 Packages
     release v=8.7,o=Debian,a=stable,n=jessie,l=Debian,c=non-free
     origin ftp.debian.org
 650 http://ftp.debian.org/debian/ jessie/contrib amd64 Packages
     release v=8.7,o=Debian,a=stable,n=jessie,l=Debian,c=contrib
     origin ftp.debian.org
 650 http://ftp.debian.org/debian/ jessie/main amd64 Packages
     release v=8.7,o=Debian,a=stable,n=jessie,l=Debian,c=main
     origin ftp.debian.org
Pinned packages:
As i said i found a workaround, so this is now just a curiosity about what happened and why it worked before (according to the info i found about autofs and mount namespaces it shouldn't without a workaround)

User avatar
None1975
df -h | participant
df -h | participant
Posts: 1388
Joined: 2015-11-29 18:23
Location: Russia, Kaliningrad
Has thanked: 45 times
Been thanked: 65 times

Re: Debian update caused major breakage in my workflow regar

#8 Post by None1975 »

Harkonnen wrote:Here it is:

Man, your sources list is a mess...Welcome to this thread http://forums.debian.net/viewtopic.php?f=20&t=114130
OS: Debian 12.4 Bookworm / DE: Enlightenment
Debian Wiki | DontBreakDebian, My config files on github

User avatar
dasein
Posts: 7680
Joined: 2011-03-04 01:06
Location: Terra Incantationum

Re: Debian update caused major breakage in my workflow regar

#9 Post by dasein »

Wow. Over 50 different repos?? That may be a new record.

Yo, OP! Remember what I said about restoring from backup? Forget it. Substitute orbital bombardment instead.

Image

User avatar
debiman
Posts: 3063
Joined: 2013-03-12 07:18

Re: Debian update caused major breakage in my workflow regar

#10 Post by debiman »

Harkonnen, in case it wasn't clear enough:
your system is broken beyond repair.
you have so many sources that do not even belong to debian (ppa...), that upgrade definitely destroyed it.
maybe it's still moving, but essentially it's dead.
hence frankendebian.
https://wiki.debian.org/DontBreakDebian

User avatar
GarryRicketson
Posts: 5644
Joined: 2015-01-20 22:16
Location: Durango, Mexico

Re: Debian update caused major breakage in my workflow regar

#11 Post by GarryRicketson »

Just a note , in case it is not clear to someone:
Debian update caused major breakage in my workflow regar
The Debian updates are NOT the cause of this,.. Debian is not at fault. The cause is the mixed repos,... the title should be "I used a lot of mixed repos, and they caused major breakage in my workflow regarding"
https://wiki.debian.org/DontBreakDebian/

User avatar
eor2004
Posts: 251
Joined: 2013-10-01 22:49
Location: Puerto Rico
Has thanked: 4 times
Been thanked: 5 times

Re: Debian update caused major breakage in my workflow regar

#12 Post by eor2004 »

Bruh, sorry for saying this but your system is:

Image

Time to reinstall and to RTFM!
Debian 12 Gnome on a MSI H61M-P25 (B3) PC & on a Dell Latitude E6410 & HP EliteBook 8540p Laptops.
LMDE 6 on a Panasonic ToughBook CF-C1 Laptop.
Bodhi Linux 7 on a HP Compaq DC5750 Small Form Factor PC.
Windows 11 on a Intel DH55TC PC.

User avatar
stevepusser
Posts: 12930
Joined: 2009-10-06 05:53
Has thanked: 41 times
Been thanked: 71 times

Re: Debian update caused major breakage in my workflow regar

#13 Post by stevepusser »

I have to say that most of those repos are Jessie-based, and even the Trusty PPAs may not be causing the issues. I recognize my own Pale Moon repo on the openSUSE Build Service in there, and that is certainly compatible with Jessie. Deb-multimedia has been guilty of breaking some multimedia stuff in the past, but I don't know how it could affect other things like namespaces.

A shorter, more readable version of the active sources can be generated by installing inxi and running "inxi -r".
MX Linux packager and developer

User avatar
pylkko
Posts: 1802
Joined: 2014-11-06 19:02

Re: Debian update caused major breakage in my workflow regar

#14 Post by pylkko »

Many were indeed Jessie based but there appears to be ppas from two distinct Ubuntu versions, which is something that Ubuntu people appear to not consider exactly safe. Anyway, even if the those sources could be maintained safely with pinning and care there is nevertheless the problem that if a large amount of the code is not from Debian and something adverse occurs, it is going to pretty hard to debug and troubleshoot. You would at minimum need to be able to list the versions and sources of all the pakages and libraries. This can be done. However, thereafter it starts getting harder. There is also a fairly high chance that this system if upgraded to yet another point release or especially Stretch, that it would just stop working/many things could break at the same time.

User avatar
stevepusser
Posts: 12930
Joined: 2009-10-06 05:53
Has thanked: 41 times
Been thanked: 71 times

Re: Debian update caused major breakage in my workflow regar

#15 Post by stevepusser »

I suppose what actually was installed from the PPA is going to make a difference, yes. Userspace programs aren't really going to have the opportunity to cause the mess that some system library PPA upgrade could. The Pale Moon package is self-contained, too, building and using internal versions of libnss3, libnspr4, libsqlite3, etc., instead of adding updated system library versions of those to my repo.
MX Linux packager and developer

Harkonnen
Posts: 24
Joined: 2011-08-31 12:52

Re: Debian update caused major breakage in my workflow regar

#16 Post by Harkonnen »

The trolls are at large here it seems :) It is really really hard to break debian to an unfixable state. I have not reinstalled since i first installed woody ~17 years ago, using hundreds of 3-rd party packages, even mixing sid and testing at some point (never go sid if you intent to go back to stable - yep getting from there to the next stable taken a few hours of manual aptitude changes, but it wasnt broken). And all if this is irrelevant to the question at hand because nothing i have installed from ppas or self build packages can even theoretically touch namespaces.

User avatar
pylkko
Posts: 1802
Joined: 2014-11-06 19:02

Re: Debian update caused major breakage in my workflow regar

#17 Post by pylkko »

Harkonnen wrote:The trolls are at large here it seems :) It is really really hard to break debian to an unfixable state. I have not reinstalled since i first installed woody ~17 years ago, using hundreds of 3-rd party packages, even mixing sid and testing at some point (never go sid if you intent to go back to stable - yep getting from there to the next stable taken a few hours of manual aptitude changes, but it wasnt broken). And all if this is irrelevant to the question at hand because nothing i have installed from ppas or self build packages can even theoretically touch namespaces.
I believe that any rational and intelligent person would allow as much as this: 1) Empirical evidence on this board suggests that it is indeed quite possible and easy to brake Debian 2) that you have managed to do this for 17 years does not mean that it is not dangerous, just as you might have smoked cigarettes for the last 17 years without health problems and nothing follows, 3) it is possible to go from sid to stable and there are posts about how to do this 4) of course it is possible to install packages from these repos with care, but if you give us the information that you here have and no one else has had this problem upgrading, then it is entirely rational to begin to suspect that you have something idiosyncratic about your system w.r.t this issue 5) if you really believe in your opinion that it is entirely impossible that anything from the third party repos etc. that you have installed has interfered here (and you don't want to demonstrate this much), then the next most likely place to look would be at the changelog for the point release that you upgraded.

Harkonnen
Posts: 24
Joined: 2011-08-31 12:52

Re: Debian update caused major breakage in my workflow regar

#18 Post by Harkonnen »

pylkko wrote: but if you give us the information that you here have and no one else has had this problem upgrading, then it is entirely rational to begin to suspect that you have something idiosyncratic about your system w.r.t this issue
How many people use autofs and ip namespaces? No reports about such niche issue is not an indication that it is unique to only this system. Especially if you consider that according to the internet it shouldn't have worked as it was setup at all (but it did).

Bulkley
Posts: 6383
Joined: 2006-02-11 18:35
Has thanked: 2 times
Been thanked: 39 times

Re: Debian update caused major breakage in my workflow regar

#19 Post by Bulkley »

Installing packages from other sources can work; this is Linux after all. Anyone with the skills can do it. The problem comes with upgrades. A stable Debian release has been tested to ensure that packages not only work together but will still work after an upgrade. For example, many users install proprietary video drivers and find that they work well until they do an upgrade that leaves the proprietary package stranded or (worse) conflicted. Essentially, it works until it doesn't.

Harkonnen, your system can be fixed but it may take a lot of work to find the conflict and sort it out.

User avatar
dasein
Posts: 7680
Joined: 2011-03-04 01:06
Location: Terra Incantationum

Re: Debian update caused major breakage in my workflow regar

#20 Post by dasein »

Harkonnen wrote:No reports about such niche issue is not an indication that it is unique to only this system.
It's not an indication of anything, period. But pylkko's observation was about conditional probability and nothing else. Nice try for the strawman, though.
Harkonnen wrote:...according to the internet it shouldn't have worked as it was setup at all (but it did).
Except when it didn't. :roll:

Post Reply