deb9 - strange tar behavior

If none of the more specific forums is the right place to ask

deb9 - strange tar behavior

Postby ksu » 2019-01-04 20:54

Hello,

I am using tar with mbuffer to prevent buffer underruns when dumping data to a LTO tape drive on a remote system. The included in Stretch tar version is 1.29 - it fails for some reason ( data sender side ):

Code: Select all
.
tar cv --preserve-permissions --numeric-owner -f >(mbuffer -s $MBUFBLK -m 8G -O 10.0.0.113:8000) $2 $3
.

/bin/sh: 1: p�~��V: not found
tar: /dev/fd/63: Wrote only 4096 of 10240 bytes
tar: Child returned status 127
tar: Error is not recoverable: exiting now


initially i thought it is mbuffer version, recompiled the older version that I have tested fine on two old Gentoo boxes, but it looks like it is 'tar'
the old systems run tar 1.27.1 on the sender side and 1.26 on the receiver.

I compiled 1.27.1 on Stretch - it runs with mbuffer as expected...

the problem is the new tar, v1.29, on the sender side... looks like it is not that POSIX...

any suggestions? should i maybe go directly to GNU.org with this?

Thanks Folks!!

Mike
User avatar
ksu
 
Posts: 58
Joined: 2014-01-13 14:59

Re: deb9 - strange tar behavior

Postby bw123 » 2019-01-04 21:57

I'd check either bash faq https://mywiki.wooledge.org/BashFAQ
and/or try running it with bash, /bin/sh is linked to dash by default (POSIX?)

Code: Select all
$ ls -l /bin/sh
lrwxrwxrwx 1 root root 4 Nov 12  2017 /bin/sh -> dash
User avatar
bw123
 
Posts: 3787
Joined: 2011-05-09 06:02

Re: deb9 - strange tar behavior

Postby ksu » 2019-01-04 22:06

Thanks for your reply, but I run all from:
bash-4.4#
directly..
and specify
#!/bin/bash
in the script...

Mike

edit: i understand, this could be the interpreter/ bash issue, but in this case, the old version of tar should err out as well...
Last edited by ksu on 2019-01-04 22:10, edited 1 time in total.
User avatar
ksu
 
Posts: 58
Joined: 2014-01-13 14:59

Re: deb9 - strange tar behavior

Postby bw123 » 2019-01-04 22:07

ksu wrote:Thanks for your reply, but I run all from:
bash-4.4#
directly..
and specify
#!/bin/bash
in the script...

Mike


then why does error say
"/bin/sh: 1: p�~��V: not found" ?

Something wrong with line 1 in the script? Do you run the same script on another gnu/linux? Do you mind showing it to the forum?
User avatar
bw123
 
Posts: 3787
Joined: 2011-05-09 06:02

Re: deb9 - strange tar behavior

Postby ksu » 2019-01-04 22:15

That is a good question, in the beginning i have scp the original again from the gentoo box, and even did hexdump on it initially :) ... not line 1 issue in the script... it is actually line 32 - line 1 is shebang...

and this is the old script: ( sender side )
Code: Select all
#!/bin/bash

# use older version of tar: 1.27.1 or 1.6 as on gentoo3/4

ssh gentoo4 '~/bin/backup.mbuffer.receive.sh &>/tmp/backup.mbuffer.progress.out' &


NAME="$1"
#export TAR_OPTIONS="--use-compress-program=lbzip2"
export TAR_OPTIONS="--use-compress-program=pigz"
LOG="/root/log/$NAME.`date +%F`.log"
LOGERR="$LOG.errors"
MBUFBLK=256k

if [ ! $1 ] || [ ! $2 ]; then
   echo " Usage: $0 [set_name] [src].[.].."
   exit 1
fi

echo "backup_name: $LOG"
echo "source: $2 $3 $4 $5 $6 $7 $8 $9"
echo "Calculating total MB for this job:"
du -ksmc $2 $3 $4 $5 $6 $7 $8 $9
echo
echo "position the tape if needed..."
echo "starting to mbuffer... in 5 sec."
sleep 5
echo "start: `date`"
date > $LOG
echo $* >> $LOG

~/bin/tar cv --preserve-permissions --numeric-owner -f >(mbuffer -s $MBUFBLK -m 8G -O 10.0.0.113:8000) $2 $3 $4 $5 $6 $7 $8 $9 1>> $LOG 2> $LOGERR

echo "finish: `date`"
echo "set name: $1" >> $LOG
echo "backup path: $2 $3 $4 $5 $6 $7 $8 $9" >> $LOG
echo "mbuffer blk size used: $MBUFBLK" >> $LOG
echo "Calculating total MB for this job:"
du -ksmc $2 $3 $4 $5 $6 $7 $8 $9
echo "Calculating total MB for this job:" >> $LOG
du -ksmc $2 $3 $4 $5 $6 $7 $8 $9 >> $LOG
echo "finish: `date`" >> $LOG

export TAR_OPTIONS=""



on the receiver i disable tape h/w compression, etc... and dump it to a non-rewind...
User avatar
ksu
 
Posts: 58
Joined: 2014-01-13 14:59

Re: deb9 - strange tar behavior

Postby bw123 » 2019-01-04 22:23

I did not read it all, but does anything inyour script run any subshells with dash instead of bash? Is this what your intend to do? Does the other gnu/linux where the script works use dash also? Is this really a deb9- strange tar behavior issue, or something with default shell?
User avatar
bw123
 
Posts: 3787
Joined: 2011-05-09 06:02

Re: deb9 - strange tar behavior

Postby llivv » 2019-01-04 22:28

tar
Code: Select all
--rsh-command=/usr/bin/ssh

?
In memory of Ian Ashley Murdock (1973 - 2015) founder of the Debian project.
User avatar
llivv
 
Posts: 5488
Joined: 2007-02-14 18:10
Location: cold storage

Re: deb9 - strange tar behavior

Postby bw123 » 2019-01-04 22:30

did a hexdump on it? wow.. sounds cool. I like to read while doing hexdump.
User avatar
bw123
 
Posts: 3787
Joined: 2011-05-09 06:02

Re: deb9 - strange tar behavior

Postby ksu » 2019-01-04 22:38

bw123 wrote:I did not read it all, but does anything inyour script run any subshells with dash instead of bash? Is this what your intend to do? Does the other gnu/linux where the script works use dash also? Is this really a deb9- strange tar behavior issue, or something with default shell?


sh is now bash.... the error is "similar":

Code: Select all
bash-4.4# cat ~/log/test.2019-01-04.log.errors
/bin/sh: $'p\272\036\316\320\311U': command not found
tar: /dev/fd/63: Wrote only 4096 of 10240 bytes
tar: Child returned status 127
tar: Error is not recoverable: exiting now
bash-4.4#
User avatar
ksu
 
Posts: 58
Joined: 2014-01-13 14:59

Re: deb9 - strange tar behavior

Postby ksu » 2019-01-04 22:43

llivv wrote:tar
Code: Select all
--rsh-command=/usr/bin/ssh

?


nah.... i need mbuffer on receiver, and sender first - that's the whole purpose - and mbuffer opens it's own socket to for data transmit..
Last edited by ksu on 2019-01-04 22:47, edited 2 times in total.
User avatar
ksu
 
Posts: 58
Joined: 2014-01-13 14:59

Re: deb9 - strange tar behavior

Postby ksu » 2019-01-04 22:45

bw123 wrote:did a hexdump on it? wow.. sounds cool. I like to read while doing hexdump.


for sure that's a less "efficient" way :) but I like to "see it" :)
User avatar
ksu
 
Posts: 58
Joined: 2014-01-13 14:59

Re: deb9 - strange tar behavior

Postby ksu » 2019-01-04 22:48

ksu wrote:
bw123 wrote:I did not read it all, but does anything inyour script run any subshells with dash instead of bash? Is this what your intend to do? Does the other gnu/linux where the script works use dash also? Is this really a deb9- strange tar behavior issue, or something with default shell?


sh is now bash.... the error is "similar":

Code: Select all
bash-4.4# cat ~/log/test.2019-01-04.log.errors
/bin/sh: $'p\272\036\316\320\311U': command not found
tar: /dev/fd/63: Wrote only 4096 of 10240 bytes
tar: Child returned status 127
tar: Error is not recoverable: exiting now
bash-4.4#



and this is using the default, 1.29 tar... not the one specified in the script - the old one works as it should regardless of the symlink...
User avatar
ksu
 
Posts: 58
Joined: 2014-01-13 14:59

Re: deb9 - strange tar behavior

Postby llivv » 2019-01-05 00:56

how would you debug a script?
In memory of Ian Ashley Murdock (1973 - 2015) founder of the Debian project.
User avatar
llivv
 
Posts: 5488
Joined: 2007-02-14 18:10
Location: cold storage

Re: deb9 - strange tar behavior

Postby GarryRicketson » 2019-01-05 01:18

This is how I learned how,
How would you debug a script?
I don't feel like going into details just now, but the results do. Hint, better results can be obtained if the type of script is specified, IE: csh,ksh, bash ,etc ...(the search bot assumes it is bash, mostly, or shell scripts in general.
User avatar
GarryRicketson
 
Posts: 5877
Joined: 2015-01-20 22:16
Location: Durango, Mexico

Re: deb9 - strange tar behavior

Postby ksu » 2019-01-05 02:02

I still don't get this...

old tar:
Code: Select all
+ echo test testdir/
+ /root/bin/tar cv --preserve-permissions --numeric-owner -f /dev/fd/63 testdir/
++ mbuffer -s 256k -m 8G -O 10.0.0.113:8000
testdir/
testdir/somejunkfile

summary: 3560 KiByte in  0.1sec - average of 57.3 MiB/s
++ date


new tar:
Code: Select all
+ echo test testdir/
+ tar cv --preserve-permissions --numeric-owner -f /dev/fd/63 testdir/
++ mbuffer -s 256k -m 8G -O 10.0.0.113:8000
testdir/
testdir/somejunkfile
/bin/sh: $'p\272.\234\260': command not found
tar: /dev/fd/63: Wrote only 4096 of 10240 bytes
tar: Child returned status 127
tar: Error is not recoverable: exiting now
++ date


when ran directly from bash, eg:

old tar:
Code: Select all
bash-4.4# ~/bin/tar cv --preserve-permissions --numeric-owner -f >(mbuffer -s 256k -m 8G -O 10.0.0.113:8000) testdir/
testdir/
testdir/somejunkfile
bash-4.4#
summary: 43.4 MiByte in  0.4sec - average of  112 MiB/s

bash-4.4#


new tar:
Code: Select all
bash-4.4# tar cv --preserve-permissions --numeric-owner -f >(mbuffer -s 256k -m 8G -O 10.0.0.113:8000) testdir/
testdir/
testdir/somejunkfile
bash-4.4#
summary: 43.4 MiByte in  0.4sec - average of  113 MiB/s

bash-4.4#


both complete fine...

so why the behavior of tar v1.27.1 ( and 1.26 ) versus v1.29 in the bash shell script is so different?
I need to start understanding this one way or another - "debugging a script" does not provide me with answer to the tar behavior question
User avatar
ksu
 
Posts: 58
Joined: 2014-01-13 14:59

Next

Return to General Questions

Who is online

Users browsing this forum: No registered users and 11 guests

fashionable