Scheduled Maintenance: We are aware of an issue with Google, AOL, and Yahoo services as email providers which are blocking new registrations. We are trying to fix the issue and we have several internal and external support tickets in process to resolve the issue. Please see: viewtopic.php?t=158230

 

 

 

[SOLVED] Copying folder (rsync)

If none of the specific sub-forums seem right for your thread, ask here.
Post Reply
Message
Author
User avatar
pawRoot
Posts: 603
Joined: 2016-12-28 18:26
Has thanked: 1 time
Been thanked: 1 time

[SOLVED] Copying folder (rsync)

#1 Post by pawRoot »

I have a little shell script that is supposed to copy a folder named BrightNight (and its contents) to another location.

Command:

Code: Select all

rsync -R BrightNight ~/.themes/BrightNight/
The directory .themes doesn't exist but it was supposed to create it right?

But it doesn't work, it says "no such file or directory", although im using the same command for other folders in my shell.
The folder BrightNight is in same folder as the shell script, am i doing something wrong?
Last edited by pawRoot on 2018-01-10 02:11, edited 1 time in total.

Dai_trying
Posts: 1100
Joined: 2016-01-07 12:25
Has thanked: 5 times
Been thanked: 16 times

Re: Copying folder (rsync)

#2 Post by Dai_trying »

pawRoot wrote:The directory .themes doesn't exist but it was supposed to create it right?
Not in my experience, if you are specifying a path then that path must already exist or it cannot be found.

Also you do not need to add the BrightNight Directory name in your destination or it will be duplicated (i.e. ~/.themes/BrightNight/BrightNight/filenames/etc)

Also you should use -r and not -R for recursive (-R is for Relative, see man rsync for details)

User avatar
cds60601
df -h | participant
df -h | participant
Posts: 706
Joined: 2017-11-25 05:58
Location: Florida
Has thanked: 129 times
Been thanked: 60 times

Re: Copying folder (rsync)

#3 Post by cds60601 »

I prefer to use rdiff-backup. To me, it seems more robust.
I do pretty much what you do but a whole drive (1 TB to another on a nightly basis)

If you opt to use rdiff-backup, this is the syntax I use (YMMV)

Code: Select all

#/usr/bin/rdiff-backup -b -v5 /media/Data /media/Backup
/usr/bin/rdiff-backup -v5 /media/Data /media/Backup
In the example above, the first line completes a full backup of the 1 TB drive mounted as Data to the second 1TB drive mounted as Backup
The second line runs every night and copies only changed data. I think (could be mistaken since its been a long time since I implemented this) the change is not only based on new data but also if the timestamp is changed by access the file/files.

In my opinion, rdiff-backup is far faster than rsync (I tried using just rsync and it ran terrifyingly slow - then again, this just could have just been something I overlooked).
The above may be a bit overkill for your needs. Just wanted to present another option for you to consider should your scripting/backup needs require something a little more.

Added info for your consumption:
"rdiff-backup backs up one directory to another, possibly over a network. ... The idea is to combine the best features of a mirror and an incremental backup. rdiff-backup also preserves subdirectories, hard links, dev files, permissions, uid/gid ownership, modification times, extended attributes, acls, and resource forks."
Supercalifragilisticexpialidocious

pcalvert
Posts: 1939
Joined: 2006-04-21 11:19
Location: Sol Sector
Has thanked: 1 time
Been thanked: 2 times

Re: Copying folder (rsync)

#4 Post by pcalvert »

cds60601 wrote:I prefer to use rdiff-backup. To me, it seems more robust.
I recently tried making a backup with rdiff-backup and was sorry that I did. The backup took a ridiculously long time, and rdiff-backup kept using more and more memory as it went. That led me to do some investigating. Here's what I found:

rdiff-backup was abandoned -- it is no longer being developed:
rdiff-backup was brought to a halt in 2009. We are yet to see a return from the rdiff-backup development team."
See: https://cookuop.co.uk/rsync-vs-rdiff-backup/


rdiff-backup: leaks up to at least 400MiB of RAM
https://bugs.debian.org/cgi-bin/bugrepo ... bug=574379

rdiff-backup: crashed after hitting process memory limit
https://bugs.debian.org/cgi-bin/bugrepo ... bug=781844


Others have also noticed that rdiff-backup is slow:

rdiff-backup performance -- slow operation on initial backup
http://www.backupcentral.com/forum/17/2 ... ial_backup


Phil
Freespoke is a new search engine that respects user privacy and does not engage in censorship.

User avatar
cds60601
df -h | participant
df -h | participant
Posts: 706
Joined: 2017-11-25 05:58
Location: Florida
Has thanked: 129 times
Been thanked: 60 times

Re: Copying folder (rsync)

#5 Post by cds60601 »

Wow! I had no idea about the things you posted. As I did mention, YMMV - I have not experienced these issue with my setup so I can't say one way or the other.
But good to know non the less
Supercalifragilisticexpialidocious

User avatar
cds60601
df -h | participant
df -h | participant
Posts: 706
Joined: 2017-11-25 05:58
Location: Florida
Has thanked: 129 times
Been thanked: 60 times

Re: Copying folder (rsync)

#6 Post by cds60601 »

pcalvert wrote:
cds60601 wrote:I prefer to use rdiff-backup. To me, it seems more robust.
I recently tried making a backup with rdiff-backup and was sorry that I did. The backup took a ridiculously long time, and rdiff-backup kept using more and more memory as it went. That led me to do some investigating. Here's what I found:

rdiff-backup was abandoned -- it is no longer being developed:
rdiff-backup: leaks up to at least 400MiB of RAM
https://bugs.debian.org/cgi-bin/bugrepo ... bug=574379

rdiff-backup: crashed after hitting process memory limit
https://bugs.debian.org/cgi-bin/bugrepo ... bug=781844

Others have also noticed that rdiff-backup is slow:

rdiff-backup performance -- slow operation on initial backup
http://www.backupcentral.com/forum/17/2 ... ial_backup
Phil
The first tag is of course from 2010. If indeed dev has halted since then, I can't explain why my results are so much different. The server I run this on only has 8 gig of ram.

The second, That seems debatable and appears its not often reproducible - Again, I have not seen this either.

As to the last link - I can't speak to that either. I don't run a complex backup routine. It's all on the same system. I don't use SSH nor do I traverse the network.
This could be why I don't see the issue the others appear to have. And based on the original post, it sounds like the Op may be wanting to do much the same (not include SSH access nor traverse a network). I am willing to bet that in the Op's case, rdiff will work just fine (just as it does for me).
Supercalifragilisticexpialidocious

User avatar
pawRoot
Posts: 603
Joined: 2016-12-28 18:26
Has thanked: 1 time
Been thanked: 1 time

Re: Copying folder (rsync)

#7 Post by pawRoot »

Well i don't know anymore how i should do this...

I simply want to create for example .themes dir if it doesn't exist already and copy my folder there.

I could do "mkdir" and then "cp" but it's kinda lame.

User avatar
pawRoot
Posts: 603
Joined: 2016-12-28 18:26
Has thanked: 1 time
Been thanked: 1 time

Re: Copying folder (rsync)

#8 Post by pawRoot »

Ok i got it to work with rsync, just using -a parameter, thanks guys.

pcalvert
Posts: 1939
Joined: 2006-04-21 11:19
Location: Sol Sector
Has thanked: 1 time
Been thanked: 2 times

Re: Copying folder (rsync)

#9 Post by pcalvert »

cds60601 wrote: The first tag is of course from 2010. If indeed dev has halted since then, I can't explain why my results are so much different.
Most likely, development on rdiff-backup has stopped. The last update to the version of rdiff-backup in Sid was just over six years ago, and the version in Sid is the same one that was in Wheezy.

Phil
Freespoke is a new search engine that respects user privacy and does not engage in censorship.

User avatar
cds60601
df -h | participant
df -h | participant
Posts: 706
Joined: 2017-11-25 05:58
Location: Florida
Has thanked: 129 times
Been thanked: 60 times

Re: [SOLVED] Copying folder (rsync)

#10 Post by cds60601 »

This whole thread got me thinking about other options to possibly replace rdiff - this has been educational for me so thank you to the folks that participated!

Last night I chose to remove all rdiff backups from the Backup driv (522 gig worth) and give the standard rsync a shot.
I do like the simplicity to invoke a full backup (-ahz in my case). I opted not to use the verbose setting for this initial backup.
I started this at 3:30 am last night and as of this writing (11:42 am) I still have a remaining count of 89 gig.

Some specs on the data:
Home drive for 3 users
all user data is on the Data drive - no local saving of user data (2 windows systems one Debian
Samba being used for shares along with CIFS
All users (other than me) have accounts on the Debian server however setup with nologin
ISO images - copies of Clonezilla, Ubuntu 17.10 server and workstation, Debian Stable and weekly images of Testing (both official and non-free drivers
Music - My CD collection (ripped from my CD collection (about 300 CDs)
Schematics - Eagle CAD schematics from projects I work on
Debian - This is simply a dir that contains 31 days of tared dirs from my Debian laptop of essential data for easy rebuild (as a just-in-case)

Both TB drives (Data and Backup) are local to the server so all backups run faster - no network traversing)
At this point, I may opt for just rsync for simplicity. Ill know more after the initial backup is complete and after one day of user access to the server (for the differential file save).
Supercalifragilisticexpialidocious

User avatar
Thorny
Posts: 542
Joined: 2011-02-27 13:40

Re: [SOLVED] Copying folder (rsync)

#11 Post by Thorny »

cds60601 wrote: (-ahz in my case).

Both TB drives (Data and Backup) are local to the server so all backups run faster - no network traversing)
Since you are doing this locally, it would probably run faster if you didn't use compression.

User avatar
cds60601
df -h | participant
df -h | participant
Posts: 706
Joined: 2017-11-25 05:58
Location: Florida
Has thanked: 129 times
Been thanked: 60 times

Re: [SOLVED] Copying folder (rsync)

#12 Post by cds60601 »

Thorny wrote:
cds60601 wrote: (-ahz in my case).

Both TB drives (Data and Backup) are local to the server so all backups run faster - no network traversing)
Since you are doing this locally, it would probably run faster if you didn't use compression.
Ah - good point, thank you :)
Supercalifragilisticexpialidocious

User avatar
cds60601
df -h | participant
df -h | participant
Posts: 706
Joined: 2017-11-25 05:58
Location: Florida
Has thanked: 129 times
Been thanked: 60 times

Re: [SOLVED] Copying folder (rsync)

#13 Post by cds60601 »

Thorny -

You were correct and thank you for the tip!!!
Removing the compression bit brought the full rsync down to just a tad over 2 hours, see below.

earth:~/Docs# ./daily-rsync
====== Wed Jan 10 17:13:53 CST 2018: Running rsync from data drive to backup drive...
====== Wed Jan 10 19:14:05 CST 2018: Done...

====== Wed Jan 10 19:14:05 CST 2018: rsync of Data to Backup is complete.

At this point, I'm not sure I really need to follow up on the use of rdiff. I'll see how this goes for a few days but at this point, I feel I am convinced that
rsync may just be the way to go for my needs.

Cheers
Chris
Supercalifragilisticexpialidocious

pcalvert
Posts: 1939
Joined: 2006-04-21 11:19
Location: Sol Sector
Has thanked: 1 time
Been thanked: 2 times

Re: [SOLVED] Copying folder (rsync)

#14 Post by pcalvert »

For simple, occasional backups I use rsync or tar. For regular backups I use rsnapshot.

Phil
Freespoke is a new search engine that respects user privacy and does not engage in censorship.

User avatar
cds60601
df -h | participant
df -h | participant
Posts: 706
Joined: 2017-11-25 05:58
Location: Florida
Has thanked: 129 times
Been thanked: 60 times

Re: [SOLVED] Copying folder (rsync)

#15 Post by cds60601 »

pcalvert wrote:For simple, occasional backups I use rsync or tar. For regular backups I use rsnapshot.

Phil
Nice - I'll need to read up on that. Welp it's been a day or two since I moved to rsync. I have added some bits to my scheme - I'll provide the code below for anyone that wishes to see/use what I have come up with for my situation

Code: Select all

#!/bin/bash
# Daily rsync.

#echo ""
echo ====== `/bin/date`: Running rsync from data drive to backup drive...
         /usr/bin/rsync --force --ignore-errors --delete --delete-excluded --exclude-from=/root/Docs/rsync-exclude-list --backup --backup-dir=/media/Data/Deleted/`date +%Y-%m-%d` -avh /media/Data/ /media/Backup/
echo ====== `/bin/date`: Done...

echo ""
echo ====== `/bin/date`: rsync of Data to Backup is complete.
So the jist of this code is that is creates a dir on the Data drive that is excluded from the rsync backup. Files that were removed from the Data drive will get removed from the rsync on the Backup drive and stored in this directory. It's more of a safety net in case I need to restore a removed file from the Data drive. It seems complex and I know that I'm not doing a good job explaining the syntax, but I think that all will get the idea once walking through the line of code.

I assume that rsnapshot does all what I would be doing in the above - so to simplify, I'll review rsnapshot and play around with it. It may end up being the end replacement of rdiff (to be honest, I am glad I stumbled on this thread, I have learned a good bit from it - so thank you again all).

Cheers
Chris
Supercalifragilisticexpialidocious

User avatar
cds60601
df -h | participant
df -h | participant
Posts: 706
Joined: 2017-11-25 05:58
Location: Florida
Has thanked: 129 times
Been thanked: 60 times

Re: [SOLVED] Copying folder (rsync)

#16 Post by cds60601 »

cds60601 wrote:
pcalvert wrote:For simple, occasional backups I use rsync or tar. For regular backups I use rsnapshot.

Phil
Nice - I'll need to read up on that. Welp it's been a day or two since I moved to rsync. I have added some bits to my scheme - I'll provide the code below for anyone that wishes to see/use what I have come up with for my situation.
... For hell's sake. It appears I more or less recreated rsnapshot with my code from the previous post.
Looks like Ill just run rsnapshot... </OfThread>
Supercalifragilisticexpialidocious

Post Reply