deb9 - strange tar behavior

[ksu]

Hello,

I am using tar with mbuffer to prevent buffer underruns when dumping data to a LTO tape drive on a remote system. The included in Stretch tar version is 1.29 - it fails for some reason ( data sender side ):

Code: Select all

.
tar cv --preserve-permissions --numeric-owner -f >(mbuffer -s $MBUFBLK -m 8G -O 10.0.0.113:8000) $2 $3
.

/bin/sh: 1: p�~��V: not found
tar: /dev/fd/63: Wrote only 4096 of 10240 bytes
tar: Child returned status 127
tar: Error is not recoverable: exiting now

initially i thought it is mbuffer version, recompiled the older version that I have tested fine on two old Gentoo boxes, but it looks like it is 'tar'
the old systems run tar 1.27.1 on the sender side and 1.26 on the receiver.

I compiled 1.27.1 on Stretch - it runs with mbuffer as expected...

the problem is the new tar, v1.29, on the sender side... looks like it is not that POSIX...

any suggestions? should i maybe go directly to GNU.org with this?

Thanks Folks!!

Mike

[bw123]

I'd check either bash faq https://mywiki.wooledge.org/BashFAQ
and/or try running it with bash, /bin/sh is linked to dash by default (POSIX?)

Code: Select all

$ ls -l /bin/sh
lrwxrwxrwx 1 root root 4 Nov 12  2017 /bin/sh -> dash

[ksu]

Thanks for your reply, but I run all from:
bash-4.4#
directly..
and specify
#!/bin/bash
in the script...

Mike

edit: i understand, this could be the interpreter/ bash issue, but in this case, the old version of tar should err out as well...

[bw123]

Thanks for your reply, but I run all from:
bash-4.4#
directly..
and specify
#!/bin/bash
in the script...

Mike

then why does error say

"/bin/sh: 1: p�~��V: not found" ?

Something wrong with line 1 in the script? Do you run the same script on another gnu/linux? Do you mind showing it to the forum?

[ksu]

That is a good question, in the beginning i have scp the original again from the gentoo box, and even did hexdump on it initially ... not line 1 issue in the script... it is actually line 32 - line 1 is shebang...

and this is the old script: ( sender side )

Code: Select all

#!/bin/bash

# use older version of tar: 1.27.1 or 1.6 as on gentoo3/4

ssh gentoo4 '~/bin/backup.mbuffer.receive.sh &>/tmp/backup.mbuffer.progress.out' &


NAME="$1"
#export TAR_OPTIONS="--use-compress-program=lbzip2"
export TAR_OPTIONS="--use-compress-program=pigz"
LOG="/root/log/$NAME.`date +%F`.log"
LOGERR="$LOG.errors"
MBUFBLK=256k

if [ ! $1 ] || [ ! $2 ]; then
   echo " Usage: $0 [set_name] [src].[.].."
   exit 1
fi

echo "backup_name: $LOG"
echo "source: $2 $3 $4 $5 $6 $7 $8 $9"
echo "Calculating total MB for this job:"
du -ksmc $2 $3 $4 $5 $6 $7 $8 $9
echo
echo "position the tape if needed..."
echo "starting to mbuffer... in 5 sec."
sleep 5
echo "start: `date`"
date > $LOG
echo $* >> $LOG

~/bin/tar cv --preserve-permissions --numeric-owner -f >(mbuffer -s $MBUFBLK -m 8G -O 10.0.0.113:8000) $2 $3 $4 $5 $6 $7 $8 $9 1>> $LOG 2> $LOGERR

echo "finish: `date`"
echo "set name: $1" >> $LOG
echo "backup path: $2 $3 $4 $5 $6 $7 $8 $9" >> $LOG
echo "mbuffer blk size used: $MBUFBLK" >> $LOG
echo "Calculating total MB for this job:"
du -ksmc $2 $3 $4 $5 $6 $7 $8 $9
echo "Calculating total MB for this job:" >> $LOG
du -ksmc $2 $3 $4 $5 $6 $7 $8 $9 >> $LOG
echo "finish: `date`" >> $LOG

export TAR_OPTIONS=""

on the receiver i disable tape h/w compression, etc... and dump it to a non-rewind...

[bw123]

I did not read it all, but does anything inyour script run any subshells with dash instead of bash? Is this what your intend to do? Does the other gnu/linux where the script works use dash also? Is this really a deb9- strange tar behavior issue, or something with default shell?

[llivv]

tar

Code: Select all

--rsh-command=/usr/bin/ssh

?

[bw123]

did a hexdump on it? wow.. sounds cool. I like to read while doing hexdump.

[ksu]

I did not read it all, but does anything inyour script run any subshells with dash instead of bash? Is this what your intend to do? Does the other gnu/linux where the script works use dash also? Is this really a deb9- strange tar behavior issue, or something with default shell?

sh is now bash.... the error is "similar":

Code: Select all

bash-4.4# cat ~/log/test.2019-01-04.log.errors 
/bin/sh: $'p\272\036\316\320\311U': command not found
tar: /dev/fd/63: Wrote only 4096 of 10240 bytes
tar: Child returned status 127
tar: Error is not recoverable: exiting now
bash-4.4# 

[ksu]

tar

Code: Select all

--rsh-command=/usr/bin/ssh

?

nah.... i need mbuffer on receiver, and sender first - that's the whole purpose - and mbuffer opens it's own socket to for data transmit..

[ksu]

did a hexdump on it? wow.. sounds cool. I like to read while doing hexdump.

for sure that's a less "efficient" way but I like to "see it"

[ksu]

I did not read it all, but does anything inyour script run any subshells with dash instead of bash? Is this what your intend to do? Does the other gnu/linux where the script works use dash also? Is this really a deb9- strange tar behavior issue, or something with default shell?

sh is now bash.... the error is "similar":

Code: Select all

bash-4.4# cat ~/log/test.2019-01-04.log.errors 
/bin/sh: $'p\272\036\316\320\311U': command not found
tar: /dev/fd/63: Wrote only 4096 of 10240 bytes
tar: Child returned status 127
tar: Error is not recoverable: exiting now
bash-4.4# 

and this is using the default, 1.29 tar... not the one specified in the script - the old one works as it should regardless of the symlink...

[llivv]

how would you debug a script?

[GarryRicketson]

This is how I learned how,
How would you debug a script?
I don't feel like going into details just now, but the results do. Hint, better results can be obtained if the type of script is specified, IE: csh,ksh, bash ,etc ...(the search bot assumes it is bash, mostly, or shell scripts in general.

[ksu]

I still don't get this...

old tar:

Code: Select all

+ echo test testdir/
+ /root/bin/tar cv --preserve-permissions --numeric-owner -f /dev/fd/63 testdir/
++ mbuffer -s 256k -m 8G -O 10.0.0.113:8000
testdir/
testdir/somejunkfile

summary: 3560 KiByte in  0.1sec - average of 57.3 MiB/s
++ date

new tar:

Code: Select all

+ echo test testdir/
+ tar cv --preserve-permissions --numeric-owner -f /dev/fd/63 testdir/
++ mbuffer -s 256k -m 8G -O 10.0.0.113:8000
testdir/
testdir/somejunkfile
/bin/sh: $'p\272.\234\260': command not found
tar: /dev/fd/63: Wrote only 4096 of 10240 bytes
tar: Child returned status 127
tar: Error is not recoverable: exiting now
++ date

when ran directly from bash, eg:

old tar:

Code: Select all

bash-4.4# ~/bin/tar cv --preserve-permissions --numeric-owner -f >(mbuffer -s 256k -m 8G -O 10.0.0.113:8000) testdir/
testdir/
testdir/somejunkfile
bash-4.4# 
summary: 43.4 MiByte in  0.4sec - average of  112 MiB/s

bash-4.4# 

new tar:

Code: Select all

bash-4.4# tar cv --preserve-permissions --numeric-owner -f >(mbuffer -s 256k -m 8G -O 10.0.0.113:8000) testdir/
testdir/
testdir/somejunkfile
bash-4.4# 
summary: 43.4 MiByte in  0.4sec - average of  113 MiB/s

bash-4.4# 

both complete fine...

so why the behavior of tar v1.27.1 ( and 1.26 ) versus v1.29 in the bash shell script is so different?
I need to start understanding this one way or another - "debugging a script" does not provide me with answer to the tar behavior question

[llivv]

I still don't get this...

I need to start understanding this one way or another - "debugging a script" does not provide me with answer to the tar behavior question

Correct, debugging the script does not answer your question concerning the tar behavior.

Are you still trying to send a tarball to a remote hosts tape drive?

[ksu]

yes, I am still trying to send the tarball, using mbuffer... in many cases it does prevent buffer underruns, it depends on the tape drive/tape type ( LTO-3...5 ), the type of data it is dealing with ( I rather do compression on a multicore system, than let the data being dealt with on the h/w provided compression ), plus it also gives me choice of using a different compression algorithm.... on small amounts of data, the improvement is questionable and often shows disadvantage, but I confirmed the backup to be much quicker when the tape drive does not get to buffer underruns... it stops being a "streamer" and continues to re-position the tape... this is not only bad for performance, but for the hardware/media itself. Adding eg. 8GB of buffer on both ends either prevents this from happening completely, or minimizes it.

I have no issue using the old version of tar for this purpose, but then this becomes limited to availability of tar (or even gcc) on a specific system.

What bothers me, is the difference of behavior directly from shell, and script... debugging might become a nightmare when dealing with similar cases, and maybe there is a single explanation that could potentially be applied to similar cases - I just wish/want to get to the bottom of it... it always helps when you know the cause, rather than just look for a workaround...

Thanks!

[llivv]

sorry if not being clear enough....
My next step would be to
sending a test tarball from each tar version to the remote hosts tape
both with and without mbuffer