Scheduled Maintenance: We are aware of an issue with Google, AOL, and Yahoo services as email providers which are blocking new registrations. We are trying to fix the issue and we have several internal and external support tickets in process to resolve the issue. Please see: viewtopic.php?t=158230

 

 

 

Rsync problem

If none of the specific sub-forums seem right for your thread, ask here.
Post Reply
Message
Author
NewToLinux
Posts: 71
Joined: 2017-09-17 11:30

Rsync problem

#1 Post by NewToLinux »

Rsync has been regularly used to backup files in the home directory on a Linux desktop to a FAT32 USB stick.

Recently rsync has stalled due to problems with Service Worker and Local Storage files being transferred from
the home directory to the stick.

These files were deleted and rsync then rerun.

Rsync then stalled and produced many error messages of the form :-

rsync: chgrp
<file> failed:
Operation not permitted (1)

where <file> represents a filename.

The rsync command is of the form :-

rsync -v -a  --max-size=4GB --delete <home directory> .
<mount point> .

where <home directory> represents the home directory and <mount point> represents
the mount point.

I realise that FAT32 knows nothing about permissions, etc.,
but the above command has worked successfully on previous occasions.

Why are the above error messages appearing ?

Do they have anything to do with the Service Worker and Local Storage files which had been created on the
USB stick in the previous run of rsync ?

What is the best thing to do to get rsync working okay again ?

N.B. I don't want to change the rsync command in any way.

Would it be okay to delete all the files currently on the stick and then rerun rsync ?

User avatar
sunrat
Administrator
Administrator
Posts: 6412
Joined: 2006-08-29 09:12
Location: Melbourne, Australia
Has thanked: 116 times
Been thanked: 462 times

Re: Rsync problem

#2 Post by sunrat »

The "-a" option should not work properly to FAT32. As you say, permissions and ownership can not be written. In your question you say you don't want to implement the obvious answer which is to change the options. I suggest

Code: Select all

rsync -rtv --max-size=4GB --delete <home directory> .
<mount point>
From man rsync:
--archive, -a
This is equivalent to -rlptgoD. It is a quick way of saying you want recursion
and want to preserve almost everything (with -H being a notable omission).
-p, -g, and -o can not be satisfied on FAT32.
“ computer users can be divided into 2 categories:
Those who have lost data
...and those who have not lost data YET ”
Remember to BACKUP!

User avatar
Bloom
df -h | grep > 90TiB
df -h | grep > 90TiB
Posts: 504
Joined: 2017-11-11 12:23
Been thanked: 26 times

Re: Rsync problem

#3 Post by Bloom »

Re-format your usb-stick in ext2 and back-up your home directory that way. You will be able to save all privileges. Do not use Microsoft file formats for Linux backups.

millpond
Posts: 698
Joined: 2014-06-25 04:56

Re: Rsync problem

#4 Post by millpond »

I've also run into this problem with NTFS storage.

Is there a way to use rsync with writing to a tar file?

User avatar
Bloom
df -h | grep > 90TiB
df -h | grep > 90TiB
Posts: 504
Joined: 2017-11-11 12:23
Been thanked: 26 times

Re: Rsync problem

#5 Post by Bloom »

NTFS is also a Microsoft format that doesn't understand Linux permissions and rights. Do not use Microsoft formats for Linux backups!

A solution would indeed be to use tar as a backup tool. Since tar produces one single file and does understand Linux permissions and rights, this would work. Attention, though! FAT32 supports no files greater than 2 GiB in size. Use exFAT or NTFS to accomodate big files.

Code: Select all

tar -cvzf /media/USB/backup.tar.gz /home
Use tar this way, rsync is not advisable.

pendrachken
Posts: 1394
Joined: 2007-03-04 21:10
Location: U.S.A. - WI.

Re: Rsync problem

#6 Post by pendrachken »

Bloom wrote:Attention, though! FAT32 supports no files greater than 2 GiB in size. Use exFAT or NTFS to accomodate big files.

It's 4GB. Hence the whole size limit flag in the original rsync command.

Code: Select all

tar -cvzf /media/USB/backup.tar.gz /home
Use tar this way, rsync is not advisable.
If the OP doesn't want to even change the rsync command, what makes you think a FS change on the destination drive would even be considered, as it's quite a bit more work? More work, and more danger as it would entail deleting existing backups that while are not current, are better than absolutely nothing in the case of data loss.


This is also why making a simple tarball backup isn't recommended. You would have to not only write the tarball to a local disk, but also use split to make the tarball into chunks that can actually be written to the fat32 partition. Then on restore you would have to cat the tarball chunks back together - again off of the fat32 partition - before you could untar them.

That's ignoring the fact that rsync can be told to ignore unchanged files in the backup folder so you only have to transfer / write new and changed data. Meaning drastically less data transfer and duplication needed for backups. Tar is great for contiguous backups where you may want to track individual changes by date, on tape, but gets extremely unwieldy when you start to get multi-GB to TB+ worth of data.
fortune -o
Your love life will be... interesting.
:twisted: How did it know?

The U.S. uses the metric system too, we have tenths, hundredths and thousandths of inches :-P

Post Reply