[SOLVED] Copying folder (rsync)

If none of the more specific forums is the right place to ask

[SOLVED] Copying folder (rsync)

Postby pawRoot » 2018-01-09 13:07

I have a little shell script that is supposed to copy a folder named BrightNight (and its contents) to another location.

Command:
Code: Select all
rsync -R BrightNight ~/.themes/BrightNight/


The directory .themes doesn't exist but it was supposed to create it right?

But it doesn't work, it says "no such file or directory", although im using the same command for other folders in my shell.
The folder BrightNight is in same folder as the shell script, am i doing something wrong?
Last edited by pawRoot on 2018-01-10 02:11, edited 1 time in total.
User avatar
pawRoot
 
Posts: 536
Joined: 2016-12-28 18:26
Location: Poland

Re: Copying folder (rsync)

Postby Dai_trying » 2018-01-09 13:47

pawRoot wrote:The directory .themes doesn't exist but it was supposed to create it right?


Not in my experience, if you are specifying a path then that path must already exist or it cannot be found.

Also you do not need to add the BrightNight Directory name in your destination or it will be duplicated (i.e. ~/.themes/BrightNight/BrightNight/filenames/etc)

Also you should use -r and not -R for recursive (-R is for Relative, see man rsync for details)
Dai_trying
 
Posts: 548
Joined: 2016-01-07 12:25

Re: Copying folder (rsync)

Postby cds60601 » 2018-01-09 16:27

I prefer to use rdiff-backup. To me, it seems more robust.
I do pretty much what you do but a whole drive (1 TB to another on a nightly basis)

If you opt to use rdiff-backup, this is the syntax I use (YMMV)

Code: Select all
#/usr/bin/rdiff-backup -b -v5 /media/Data /media/Backup
/usr/bin/rdiff-backup -v5 /media/Data /media/Backup


In the example above, the first line completes a full backup of the 1 TB drive mounted as Data to the second 1TB drive mounted as Backup
The second line runs every night and copies only changed data. I think (could be mistaken since its been a long time since I implemented this) the change is not only based on new data but also if the timestamp is changed by access the file/files.

In my opinion, rdiff-backup is far faster than rsync (I tried using just rsync and it ran terrifyingly slow - then again, this just could have just been something I overlooked).
The above may be a bit overkill for your needs. Just wanted to present another option for you to consider should your scripting/backup needs require something a little more.

Added info for your consumption:
"rdiff-backup backs up one directory to another, possibly over a network. ... The idea is to combine the best features of a mirror and an incremental backup. rdiff-backup also preserves subdirectories, hard links, dev files, permissions, uid/gid ownership, modification times, extended attributes, acls, and resource forks."
Yeah, 220, 221. Whatever it takes.
Server: Debian 9 (Stretch) Workstation: Archlinux
User avatar
cds60601
 
Posts: 136
Joined: 2017-11-25 05:58

Re: Copying folder (rsync)

Postby pcalvert » 2018-01-09 18:46

cds60601 wrote:I prefer to use rdiff-backup. To me, it seems more robust.

I recently tried making a backup with rdiff-backup and was sorry that I did. The backup took a ridiculously long time, and rdiff-backup kept using more and more memory as it went. That led me to do some investigating. Here's what I found:

rdiff-backup was abandoned -- it is no longer being developed:
rdiff-backup was brought to a halt in 2009. We are yet to see a return from the rdiff-backup development team."

See: https://cookuop.co.uk/rsync-vs-rdiff-backup/


rdiff-backup: leaks up to at least 400MiB of RAM
https://bugs.debian.org/cgi-bin/bugrepo ... bug=574379

rdiff-backup: crashed after hitting process memory limit
https://bugs.debian.org/cgi-bin/bugrepo ... bug=781844


Others have also noticed that rdiff-backup is slow:

rdiff-backup performance -- slow operation on initial backup
http://www.backupcentral.com/forum/17/2 ... ial_backup


Phil
“Property is the fruit of labor; property is desirable; it is a positive good
in the world. That some should be rich shows that others may become
rich, and hence is just encouragement to industry and enterprise.”
— Abraham Lincoln
pcalvert
 
Posts: 1799
Joined: 2006-04-21 11:19
Location: Sol Sector

Re: Copying folder (rsync)

Postby cds60601 » 2018-01-09 20:20

Wow! I had no idea about the things you posted. As I did mention, YMMV - I have not experienced these issue with my setup so I can't say one way or the other.
But good to know non the less
Yeah, 220, 221. Whatever it takes.
Server: Debian 9 (Stretch) Workstation: Archlinux
User avatar
cds60601
 
Posts: 136
Joined: 2017-11-25 05:58

Re: Copying folder (rsync)

Postby cds60601 » 2018-01-09 20:32

pcalvert wrote:
cds60601 wrote:I prefer to use rdiff-backup. To me, it seems more robust.

I recently tried making a backup with rdiff-backup and was sorry that I did. The backup took a ridiculously long time, and rdiff-backup kept using more and more memory as it went. That led me to do some investigating. Here's what I found:

rdiff-backup was abandoned -- it is no longer being developed:
rdiff-backup: leaks up to at least 400MiB of RAM
https://bugs.debian.org/cgi-bin/bugrepo ... bug=574379

rdiff-backup: crashed after hitting process memory limit
https://bugs.debian.org/cgi-bin/bugrepo ... bug=781844

Others have also noticed that rdiff-backup is slow:

rdiff-backup performance -- slow operation on initial backup
http://www.backupcentral.com/forum/17/2 ... ial_backup
Phil


The first tag is of course from 2010. If indeed dev has halted since then, I can't explain why my results are so much different. The server I run this on only has 8 gig of ram.

The second, That seems debatable and appears its not often reproducible - Again, I have not seen this either.

As to the last link - I can't speak to that either. I don't run a complex backup routine. It's all on the same system. I don't use SSH nor do I traverse the network.
This could be why I don't see the issue the others appear to have. And based on the original post, it sounds like the Op may be wanting to do much the same (not include SSH access nor traverse a network). I am willing to bet that in the Op's case, rdiff will work just fine (just as it does for me).
Yeah, 220, 221. Whatever it takes.
Server: Debian 9 (Stretch) Workstation: Archlinux
User avatar
cds60601
 
Posts: 136
Joined: 2017-11-25 05:58

Re: Copying folder (rsync)

Postby pawRoot » 2018-01-10 01:35

Well i don't know anymore how i should do this...

I simply want to create for example .themes dir if it doesn't exist already and copy my folder there.

I could do "mkdir" and then "cp" but it's kinda lame.
User avatar
pawRoot
 
Posts: 536
Joined: 2016-12-28 18:26
Location: Poland

Re: Copying folder (rsync)

Postby pawRoot » 2018-01-10 02:10

Ok i got it to work with rsync, just using -a parameter, thanks guys.
User avatar
pawRoot
 
Posts: 536
Joined: 2016-12-28 18:26
Location: Poland

Re: Copying folder (rsync)

Postby pcalvert » 2018-01-10 13:51

cds60601 wrote:The first tag is of course from 2010. If indeed dev has halted since then, I can't explain why my results are so much different.

Most likely, development on rdiff-backup has stopped. The last update to the version of rdiff-backup in Sid was just over six years ago, and the version in Sid is the same one that was in Wheezy.

Phil
“Property is the fruit of labor; property is desirable; it is a positive good
in the world. That some should be rich shows that others may become
rich, and hence is just encouragement to industry and enterprise.”
— Abraham Lincoln
pcalvert
 
Posts: 1799
Joined: 2006-04-21 11:19
Location: Sol Sector

Re: [SOLVED] Copying folder (rsync)

Postby cds60601 » 2018-01-10 17:05

This whole thread got me thinking about other options to possibly replace rdiff - this has been educational for me so thank you to the folks that participated!

Last night I chose to remove all rdiff backups from the Backup driv (522 gig worth) and give the standard rsync a shot.
I do like the simplicity to invoke a full backup (-ahz in my case). I opted not to use the verbose setting for this initial backup.
I started this at 3:30 am last night and as of this writing (11:42 am) I still have a remaining count of 89 gig.

Some specs on the data:
Home drive for 3 users
all user data is on the Data drive - no local saving of user data (2 windows systems one Debian
Samba being used for shares along with CIFS
All users (other than me) have accounts on the Debian server however setup with nologin
ISO images - copies of Clonezilla, Ubuntu 17.10 server and workstation, Debian Stable and weekly images of Testing (both official and non-free drivers
Music - My CD collection (ripped from my CD collection (about 300 CDs)
Schematics - Eagle CAD schematics from projects I work on
Debian - This is simply a dir that contains 31 days of tared dirs from my Debian laptop of essential data for easy rebuild (as a just-in-case)

Both TB drives (Data and Backup) are local to the server so all backups run faster - no network traversing)
At this point, I may opt for just rsync for simplicity. Ill know more after the initial backup is complete and after one day of user access to the server (for the differential file save).
Yeah, 220, 221. Whatever it takes.
Server: Debian 9 (Stretch) Workstation: Archlinux
User avatar
cds60601
 
Posts: 136
Joined: 2017-11-25 05:58

Re: [SOLVED] Copying folder (rsync)

Postby Thorny » 2018-01-10 21:41

cds60601 wrote: (-ahz in my case).

Both TB drives (Data and Backup) are local to the server so all backups run faster - no network traversing)

Since you are doing this locally, it would probably run faster if you didn't use compression.
User avatar
Thorny
 
Posts: 542
Joined: 2011-02-27 13:40

Re: [SOLVED] Copying folder (rsync)

Postby cds60601 » 2018-01-10 21:46

Thorny wrote:
cds60601 wrote: (-ahz in my case).

Both TB drives (Data and Backup) are local to the server so all backups run faster - no network traversing)

Since you are doing this locally, it would probably run faster if you didn't use compression.


Ah - good point, thank you :)
Yeah, 220, 221. Whatever it takes.
Server: Debian 9 (Stretch) Workstation: Archlinux
User avatar
cds60601
 
Posts: 136
Joined: 2017-11-25 05:58

Re: [SOLVED] Copying folder (rsync)

Postby cds60601 » 2018-01-11 00:31

Thorny -

You were correct and thank you for the tip!!!
Removing the compression bit brought the full rsync down to just a tad over 2 hours, see below.

earth:~/Docs# ./daily-rsync
====== Wed Jan 10 17:13:53 CST 2018: Running rsync from data drive to backup drive...
====== Wed Jan 10 19:14:05 CST 2018: Done...

====== Wed Jan 10 19:14:05 CST 2018: rsync of Data to Backup is complete.

At this point, I'm not sure I really need to follow up on the use of rdiff. I'll see how this goes for a few days but at this point, I feel I am convinced that
rsync may just be the way to go for my needs.

Cheers
Chris
Yeah, 220, 221. Whatever it takes.
Server: Debian 9 (Stretch) Workstation: Archlinux
User avatar
cds60601
 
Posts: 136
Joined: 2017-11-25 05:58

Re: [SOLVED] Copying folder (rsync)

Postby pcalvert » 2018-01-12 16:43

For simple, occasional backups I use rsync or tar. For regular backups I use rsnapshot.

Phil
“Property is the fruit of labor; property is desirable; it is a positive good
in the world. That some should be rich shows that others may become
rich, and hence is just encouragement to industry and enterprise.”
— Abraham Lincoln
pcalvert
 
Posts: 1799
Joined: 2006-04-21 11:19
Location: Sol Sector

Re: [SOLVED] Copying folder (rsync)

Postby cds60601 » 2018-01-12 18:12

pcalvert wrote:For simple, occasional backups I use rsync or tar. For regular backups I use rsnapshot.

Phil


Nice - I'll need to read up on that. Welp it's been a day or two since I moved to rsync. I have added some bits to my scheme - I'll provide the code below for anyone that wishes to see/use what I have come up with for my situation

Code: Select all
#!/bin/bash
# Daily rsync.

#echo ""
echo ====== `/bin/date`: Running rsync from data drive to backup drive...
         /usr/bin/rsync --force --ignore-errors --delete --delete-excluded --exclude-from=/root/Docs/rsync-exclude-list --backup --backup-dir=/media/Data/Deleted/`date +%Y-%m-%d` -avh /media/Data/ /media/Backup/
echo ====== `/bin/date`: Done...

echo ""
echo ====== `/bin/date`: rsync of Data to Backup is complete.


So the jist of this code is that is creates a dir on the Data drive that is excluded from the rsync backup. Files that were removed from the Data drive will get removed from the rsync on the Backup drive and stored in this directory. It's more of a safety net in case I need to restore a removed file from the Data drive. It seems complex and I know that I'm not doing a good job explaining the syntax, but I think that all will get the idea once walking through the line of code.

I assume that rsnapshot does all what I would be doing in the above - so to simplify, I'll review rsnapshot and play around with it. It may end up being the end replacement of rdiff (to be honest, I am glad I stumbled on this thread, I have learned a good bit from it - so thank you again all).

Cheers
Chris
Yeah, 220, 221. Whatever it takes.
Server: Debian 9 (Stretch) Workstation: Archlinux
User avatar
cds60601
 
Posts: 136
Joined: 2017-11-25 05:58

Next

Return to General Questions

Who is online

Users browsing this forum: No registered users and 9 guests

fashionable