custom TAR_OPTIONS breaking apt

[ksu]

this post is just to highlight this behavior of apt upgrade, or install
if TAR_OPTIONS is set to a custom compression utility ( with a few exceptions ) apt fails to install/upgrade/reinstall - it fails processing the tar part of a deb package:

Code: Select all

deb10 ~ # apt upgrade
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Calculating upgrade... Done
The following packages were automatically installed and are no longer required:
  linux-headers-6.1.0-16-amd64 linux-headers-6.1.0-16-common linux-image-6.1.0-16-amd64
Use 'apt autoremove' to remove them.
The following packages will be upgraded:
  firefox-esr thunderbird
2 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Need to get 121 MB of archives.
After this operation, 256 kB disk space will be freed.
Do you want to continue? [Y/n] 
Get:1 http://security.debian.org/debian-security bookworm-security/main amd64 firefox-esr amd64 115.9.0esr-1~deb12u1 [63.1 MB]
Get:2 http://security.debian.org/debian-security bookworm-security/main amd64 thunderbird amd64 1:115.9.0-1~deb12u1 [58.3 MB]
Fetched 121 MB in 5s (26.1 MB/s)       
dpkg-deb (subprocess): decompressing archive '/var/cache/apt/archives/firefox-esr_115.9.0esr-1~deb12u1_amd64.deb' (size=63138364) member 'data.tar': lzma write error: Broken pipe
dpkg-deb: error: <decompress> subprocess returned error exit status 2
dpkg-deb (subprocess): cannot copy archive member from '/var/cache/apt/archives/firefox-esr_115.9.0esr-1~deb12u1_amd64.deb' to decompressor pipe: failed to write (Broken pipe)
dpkg-deb (subprocess): decompressing archive '/var/cache/apt/archives/thunderbird_1%3a115.9.0-1~deb12u1_amd64.deb' (size=58268604) member 'data.tar': lzma write error: Broken pipe
dpkg-deb: error: <decompress> subprocess returned error exit status 2
dpkg-deb (subprocess): cannot copy archive member from '/var/cache/apt/archives/thunderbird_1%3a115.9.0-1~deb12u1_amd64.deb' to decompressor pipe: failed to write (Broken pipe)
Reading changelogs... Done
(Reading database ... 583237 files and directories currently installed.)
Preparing to unpack .../firefox-esr_115.9.0esr-1~deb12u1_amd64.deb ...
Leaving 'diversion of /usr/bin/firefox to /usr/bin/firefox.real by firefox-esr'
Unpacking firefox-esr (115.9.0esr-1~deb12u1) over (115.8.0esr-1~deb12u1) ...
Preparing to unpack .../thunderbird_1%3a115.9.0-1~deb12u1_amd64.deb ...
Unpacking thunderbird (1:115.9.0-1~deb12u1) over (1:115.8.0-1~deb12u1) ...
Setting up thunderbird (1:115.9.0-1~deb12u1) ...
Setting up firefox-esr (115.9.0esr-1~deb12u1) ...
Processing triggers for hicolor-icon-theme (0.17-2) ...
Processing triggers for man-db (2.11.2-2) ...
Processing triggers for mailcap (3.70+nmu1) ...
Processing triggers for desktop-file-utils (0.26-1) ...
deb10 ~ # 

in my case i have it set up as zstdmt - it fails to process lzma
un-setting TAR_OPTIONS allows the package to be installed ( or reinstalled, upgraded ) correctly

also, i am setting TAR_OPTIONS in /etc/profile.d/custom.sh and in .bashrc ( which is questionable but should not result in a failure processing lzma )

Code: Select all

export TAR_OPTIONS="--use-compress-program=zstdmt" 

this does not only apply to the two packages in the above apt session...

it was a challenge to locate the reason for the failures - it is "solved", but imo it is not.
I think that the decompressor should be checked, or set regardless of, or set/unset if the env var exists during or before processing of a deb pkg to prevent it from using an alternate, custom, incompatible decompressor.


edit 23mar2024: I think it applies only to upgrade - i cannot recreate the error during install / reinstall - so far applies only to upgrade - do not want to misrepresent or misinform - will reply shortly - thx

[fabien]

Hello,

the decompressor should be checked, or set regardless of, or set/unset

You can do this either at the apt level or at the dpkg level.

apt: (see man 5 apt.conf)
create the file /etc/apt/apt.conf.d/99local with

Code: Select all

DPkg::Pre-Invoke { "export TAR_OPTIONS="; };

or

Code: Select all

DPkg::Pre-Invoke { "unset -v TAR_OPTIONS"; };

dpkg: (see man 5 dpkg.cfg and man 1 dpkg)
create the file /etc/dpkg/dpkg.cfg.d/99clearTarOptions with

Code: Select all

pre-invoke="export TAR_OPTIONS="

or

Code: Select all

pre-invoke="unset -v TAR_OPTIONS"

Configuring at dpkg level is probably best as it would allow safe dpkg commands.
Note: Not tested, please let us know your tests.

[ksu]

Thank you for your response and suggestions fabien.

I have modified the initial post: in installing and reinstalling cases i am unable to recreate the problem.
at the same time, I cannot for certain exclude the install/reinstall being affected as well.

To confirm I used a livecd on trixie/main.
After each modification, and an upgrade attempt I got the same results in all 4 attempts: ( system rebooted after each test/attempt )
so export TAR_OPTIONS in the running session, modified apt || dpkg conf before invoking the upgrade.

apt: /etc/apt/apt.conf.d/99local

Code: Select all

DPkg::Pre-Invoke { "export TAR_OPTIONS="; };

dpkg-deb: error: <decompress> subprocess returned error exit status 2                                                                                                                                                                                                                                                                            
dpkg-deb (subprocess): decompressing archive '/var/cache/apt/archives/login_1%3a4.13+dfsg1-4_amd64.deb' (size=615716) member 'data.tar': lzma write error: Broken pipe

Code: Select all

DPkg::Pre-Invoke { "unset -v TAR_OPTIONS"; };

dpkg-deb: error: <decompress> subprocess returned error exit status 2
dpkg-deb (subprocess): decompressing archive '/var/cache/apt/archives/login_1%3a4.13+dfsg1-4_amd64.deb' (size=615716) member 'data.tar': lzma write error: Broken pipe

dpkg: /etc/dpkg/dpkg.cfg.d/99clearTarOptions

Code: Select all

pre-invoke="export TAR_OPTIONS="

dpkg-deb (subprocess): decompressing archive '/var/cache/apt/archives/login_1%3a4.13+dfsg1-4_amd64.deb' (size=615716) member 'data.tar': lzma write error: Broken pipe
dpkg-deb: error: <decompress> subprocess returned error exit status 2

Code: Select all

pre-invoke="unset -v TAR_OPTIONS"

dpkg-deb (subprocess): decompressing archive '/var/cache/apt/archives/login_1%3a4.13+dfsg1-4_amd64.deb' (size=615716) member 'data.tar': lzma write error: Broken pipe
dpkg-deb: error: <decompress> subprocess returned error exit status 2

no lzma write / broken pipe errors when apt upgrade is invoked without the tar options set on the livecd

I went back to my Debian 12.5 system, same case, same results as before on firefox-esr package: ( upgrade )

Code: Select all

deb10 ~ # apt upgrade
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Calculating upgrade... Done
The following packages will be upgraded:
  firefox-esr
1 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Need to get 63.1 MB of archives.
After this operation, 1024 B of additional disk space will be used.
Do you want to continue? [Y/n] 
Get:1 http://security.debian.org/debian-security bookworm-security/main amd64 firefox-esr amd64 115.9.1esr-1~deb12u1 [63.1 MB]
Fetched 63.1 MB in 1s (43.5 MB/s)      
dpkg-deb (subprocess): decompressing archive '/var/cache/apt/archives/firefox-esr_115.9.1esr-1~deb12u1_amd64.deb' (size=63141044) member 'data.tar': lzma write error: Broken pipe
dpkg-deb: error: <decompress> subprocess returned error exit status 2
dpkg-deb (subprocess): cannot copy archive member from '/var/cache/apt/archives/firefox-esr_115.9.1esr-1~deb12u1_amd64.deb' to decompressor pipe: failed to write (Broken pipe)
Reading changelogs... Done
(Reading database ... 561746 files and directories currently installed.)
Preparing to unpack .../firefox-esr_115.9.1esr-1~deb12u1_amd64.deb ...
Leaving 'diversion of /usr/bin/firefox to /usr/bin/firefox.real by firefox-esr'
Unpacking firefox-esr (115.9.1esr-1~deb12u1) over (115.9.0esr-1~deb12u1) ...
Setting up firefox-esr (115.9.1esr-1~deb12u1) ...
Processing triggers for hicolor-icon-theme (0.17-2) ...
Processing triggers for man-db (2.11.2-2) ...
Processing triggers for mailcap (3.70+nmu1) ...
Processing triggers for desktop-file-utils (0.26-1) ...
deb10 ~ # echo $?
0
deb10 ~ # 

despite the lzma err the exit code is 0
not sure what state and condition firefox is now, but i am going to reinstall it - same session and TAR_OPTIONS are still set for the alt compressor zstdmt ( same if zstd ):

Code: Select all

deb10 ~ # apt reinstall firefox-esr; echo $?
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
0 upgraded, 0 newly installed, 1 reinstalled, 0 to remove and 0 not upgraded.
Need to get 63.1 MB of archives.
After this operation, 0 B of additional disk space will be used.
Get:1 http://security.debian.org/debian-security bookworm-security/main amd64 firefox-esr amd64 115.9.1esr-1~deb12u1 [63.1 MB]
Fetched 63.1 MB in 2s (41.3 MB/s)      
(Reading database ... 561746 files and directories currently installed.)
Preparing to unpack .../firefox-esr_115.9.1esr-1~deb12u1_amd64.deb ...
Leaving 'diversion of /usr/bin/firefox to /usr/bin/firefox.real by firefox-esr'
Unpacking firefox-esr (115.9.1esr-1~deb12u1) over (115.9.1esr-1~deb12u1) ...
Setting up firefox-esr (115.9.1esr-1~deb12u1) ...
Processing triggers for hicolor-icon-theme (0.17-2) ...
Processing triggers for man-db (2.11.2-2) ...
Processing triggers for mailcap (3.70+nmu1) ...
Processing triggers for desktop-file-utils (0.26-1) ...
0
deb10 ~ #  

that's why i had to modify the earlier post - the reinstall does not generate lzma error(s) - same with install

it looks like the upgrade is not happy with the alternate de/-compressor

thank you again!

[ksu]

my guess is, next after the successful fetch the env is not set (or clean:) in the in the lzma subrocess?

[fabien]

Hello, thank you for reporting on your tests.

I can't reproduce your problem. Here is my test. I did it by exporting TAR_OPTIONS in terminal and, just to be sure, by adding the export line in /etc/profile.d/custom.sh and /root/.bashrc like you did.

Installing an old version of fonts-century-catalogue:

Code: Select all

#> apt -V install /tmp/dpkgtest/fonts-century-catalogue_001.001-7_all.deb
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Note, selecting 'fonts-century-catalogue' instead of '/tmp/dpkgtest/fonts-century-catalogue_001.001-7_all.deb'
The following NEW packages will be installed:
   fonts-century-catalogue (001.001-7)
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 0 B/31.6 kB of archives.
After this operation, 62.5 kB of additional disk space will be used.
Get:1 /tmp/dpkgtest/fonts-century-catalogue_001.001-7_all.deb fonts-century-catalogue all 001.001-7 [31.6 kB]
Selecting previously unselected package fonts-century-catalogue.
(Reading database ... 186848 files and directories currently installed.)
Preparing to unpack .../fonts-century-catalogue_001.001-7_all.deb ...
Unpacking fonts-century-catalogue (001.001-7) ...
Setting up fonts-century-catalogue (001.001-7) ...
Processing triggers for fontconfig (2.14.1-4) ...

And then upgrading:

Code: Select all

#> declare -p TAR_OPTIONS
declare -x TAR_OPTIONS="--use-compress-program=zstdmt"
#> apt -V upgrade
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Calculating upgrade... Done
The following packages will be upgraded:
   fonts-century-catalogue (001.001-7 => 001.001-8)
1 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Need to get 30.1 kB of archives.
After this operation, 3072 B disk space will be freed.
Do you want to continue? [Y/n] 
Get:1 https://deb.debian.org/debian bookworm/main amd64 fonts-century-catalogue all 001.001-8 [30.1 kB]
Fetched 30.1 kB in 0s (128 kB/s)             
Reading changelogs... Done
(Reading database ... 186859 files and directories currently installed.)
Preparing to unpack .../fonts-century-catalogue_001.001-8_all.deb ...
Unpacking fonts-century-catalogue (001.001-8) over (001.001-7) ...
Setting up fonts-century-catalogue (001.001-8) ...
Processing triggers for fontconfig (2.14.1-4) ...

As you can see, I have no errors. Any idea why?

Code: Select all

$> apt list xz-utils zstd
Listing... Done
xz-utils/stable,now 5.4.1-0.2 amd64 [installed]
zstd/stable,now 1.5.4+dfsg2-5 amd64 [installed,automatic]

[ksu]

thank you,
yes, i have modified the initial post yesterday ( crossed out install/reinstall ) and included in the new
please attempt to upgrade instead - just export TAR_OPTIONS in the running session. it is sufficient to reproduce without changes in /etc/profile.d

[fabien]

yes, i have modified the initial post yesterday ( crossed out install/reinstall ) and included in the new
please attempt to upgrade instead

I saw, and that's what I did

And then upgrading:

Code: Select all

#> declare -p TAR_OPTIONS
declare -x TAR_OPTIONS="--use-compress-program=zstdmt"
#> apt -V upgrade

[ksu]

aahhhh, i should make sure first i read it it right...

"To ensure consistent testing, I'll prepare a standardized dataset. I'm surprised to see the results differ."

[ksu]

i tried a couple of livecd's - have not found a good candidate for an upgrade after boot in live... with exception of clonezilla but, well, i cannot reproduce the error (...)

so i booted my older box - it is 11.9 / bullseye
had the tar opts set to 'pigz' - i went ahead and exported 'zstdmt'
did the update/upgrade - ended up with just two packages, firefox-esr and thunderbird was the only two packages ready...
same problem - same broken pipe on lzma

im short on time, i'll try to get something i can share with you asap ie. later tonight, or tomorrow.
till then i am assuming my place here is hunted, it might be the keyboard-chair problem on my end... my brain is fuming till then.
i just hope i am not wasting your time it the problem is the above case...

thank you, again, fabien

[fabien]

ended up with just two packages, firefox-esr and thunderbird was the only two packages ready...
same problem - same broken pipe on lzma

OK, good news (if I may say so), I can reproduce it with firefox-esr. I haven't tried thunderbird, but I also tested 0ad-data-common (package size 780 KB) as it may be size related, and 0ad-data-common also triggers the problem.
This:

Code: Select all

$> export TAR_OPTIONS="--use-compress-program=zstdmt"
$> dpkg-deb -vx ./firefox-esr_115.9.0esr-1~deb12u1_amd64.deb .

doesn't show any problem.
I'll see if I can find a workaround.

[ksu]

in the meantime
i have booted two more: 11.9 and 12 5 generate the lzma error if tar opts set to zstdmt ( compressor type unrelated - i guess /bin/false would work as a test as well )
two things all the systems have in common: all upgraded from at least one previous Debian release and all are configured by me, in years apart.

booting a fresh install of 12.5, attaching net, setting env and upgrading it does not generate any lzma/broken pipe errors.
which is, for me at least, good news.

next, i will install release 10 ( or 9 ), and keep upgrading it till i get to 12.5... otherwise i won't stop thinking about it...

Thank you!
I will post an update.

[Aki]

Hello,

this post is just to highlight this behavior of apt upgrade, or install if TAR_OPTIONS is set to a custom compression utility ( with a few exceptions ) apt fails to install/upgrade/reinstall - it fails processing the tar part of a deb package

It is worth noting that apt (and the programs invoked by it) successfully installed the packages in this scenario, in any case (even if there are error messages).

[fabien]

It is worth noting that apt (and the programs invoked by it) successfully installed the packages in this scenario, in any case (even if there are error messages).

Which raises the question what is the purpose of an internal dpkg procedure (dpkg-deb) which can fail without breaking the process. As always, the code is too difficult for me to read.

two things all the systems have in common: all upgraded from at least one previous Debian release and all are configured by me, in years apart.

booting a fresh install of 12.5, attaching net, setting env and upgrading it does not generate any lzma/broken pipe errors.

I tested in a VM that was installed with the 12.4 installer and the error appears.

Code: Select all

#> head -n1 /etc/apt/sources.list
#deb cdrom:[Debian GNU/Linux 12.4.0 _Bookworm_ - Official amd64 NETINST with firmware 20231210-17:56]/ bookworm main non-free-firmware
#> declare -p TAR_OPTIONS 
declare -x TAR_OPTIONS="--use-compress-program=zstdmt"
#> apt -V upgrade 
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Calculating upgrade... Done
The following packages will be upgraded:
   0ad-data-common (0.0.23.1-1.1 => 0.0.26-1)
1 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Need to get 780 kB of archives.
After this operation, 5120 B of additional disk space will be used.
Do you want to continue? [Y/n] 
Get:1 http://deb.debian.org/debian bookworm/main amd64 0ad-data-common all 0.0.26-1 [780 kB]
Fetched 780 kB in 1s (1363 kB/s)    
dpkg-deb (subprocess): decompressing archive '/var/cache/apt/archives/0ad-data-common_0.0.26-1_all.deb' (size=779908) member 'data.tar': lzma write error: Broken pipe
dpkg-deb: error: <decompress> subprocess returned error exit status 2
Reading changelogs... Done
(Reading database ... 36128 files and directories currently installed.)
Preparing to unpack .../0ad-data-common_0.0.26-1_all.deb ...
Unpacking 0ad-data-common (0.0.26-1) over (0.0.23.1-1.1) ...
Setting up 0ad-data-common (0.0.26-1) ...

The "pre-invoke" solution I proposed first does not change the environment of subsequent commands (pretty obvious).
The solution is to use aliases in /root/.bashrc

Code: Select all

alias apt='TAR_OPTIONS="" apt'
alias apt-get='TAR_OPTIONS="" apt-get'

However, I'm not sure it's best to have this variable set in your environment. What is your use case?

[ksu]

this is great news! to see you being able to reproduce it on 12.4 - thank you!

use case - it is questionable... at times I work a with many gz format tar archives - used to use pigz for mt
zstd shows superb performance to size ratio compared to gz, plus the overlooking to add the -I or --use-compress-program made me place it to the .bashrc... and in /etc/profile.d in some cases

I agree about the alias approach and that it might not be best to keep the variable in the env - it opens door for unexpected results ( like the apt/dpkg )
this can be handled via an alias independently, without stepping on something that might not be ready to handle the pre-set env properly.

I also hope your question about the purpose of failing without breaking it gets answered too

Many thanks to you fabien for your time, suggestions and interest in getting this topic to this state/point - I was concerned that it seemed to fail only on my systems... and thinking about possible reasons became not fun...