Splitting Tar Archive and updating it

Need help with C, C++, perl, python, etc?

Splitting Tar Archive and updating it

Postby larry77 » 2019-06-30 09:31

Dear All,
Probably a linux more than a specific debian question.
Let us say that you have a large folder (including subfolders). You want to create a tar backup split into chunks of a certain size.
You can do this with a one liner such as

tar cvf - /home/user1/dir1 | split -b 1000M - /home/user1/archive.tar

Now, suppose that dir1 changes its content over time. I would like to rerun the same command in such a way that I backup the latest version of dir1 without starting from scratch (do not overwrite existing files, but remove files that do not exist any longer etc...).
I know it is a funny way of making a backup, but there are constraints.
Any idea about how to achieve that?
Cheers
User avatar
larry77
 
Posts: 505
Joined: 2006-09-24 17:59

Re: Splitting Tar Archive and updating it

Postby Head_on_a_Stick » 2019-06-30 12:58

Looks like the --update option may do what you want, check the man page for details (or ask the interweb).
User avatar
Head_on_a_Stick
 
Posts: 10346
Joined: 2014-06-01 17:46
Location: /dev/chair

Re: Splitting Tar Archive and updating it

Postby reinob » 2019-07-01 05:26

larry77 wrote:Dear All,
Probably a linux more than a specific debian question.
Let us say that you have a large folder (including subfolders). You want to create a tar backup split into chunks of a certain size.
You can do this with a one liner such as

tar cvf - /home/user1/dir1 | split -b 1000M - /home/user1/archive.tar

Now, suppose that dir1 changes its content over time. I would like to rerun the same command in such a way that I backup the latest version of dir1 without starting from scratch (do not overwrite existing files, but remove files that do not exist any longer etc...).
I know it is a funny way of making a backup, but there are constraints.
Any idea about how to achieve that?
Cheers


You have some weird requirements. Why tar? why split? why "do not overwrite existing files" (what if a file has been modified, you really want to keep the old one?)

If you drop the tar requirement, you can use rsync, which is perfectly suited for what you want (although rsync will replace existing files if they have been modified, which is normally the sensible thing to do).

If you insist on using tar, then you're on your own but: tar(1) won't delete files from an archive, (2) will not overwrite any files at all, (3) will "update" (as suggested in the previous post) by *appending* the new version to the end of the tar archive -- you have to decide if this is what you want.

Good luck,
(and try to focus on what you really want, not on how to force/tweak an arbitrary tool to do what you want).
reinob
 
Posts: 699
Joined: 2014-06-30 11:42

Re: Splitting Tar Archive and updating it

Postby larry77 » 2019-07-01 07:37

I need to archive a large folder (hundreds of GB) and send it online. On the other side, there is someone who will reconstruct the folder with all the chunks.
Yes. I know that rsync makes more sense, but these are (not my own!) requirements.
User avatar
larry77
 
Posts: 505
Joined: 2006-09-24 17:59

Re: Splitting Tar Archive and updating it

Postby reinob » 2019-07-01 08:08

larry77 wrote:I need to archive a large folder (hundreds of GB) and send it online. On the other side, there is someone who will reconstruct the folder with all the chunks.
Yes. I know that rsync makes more sense, but these are (not my own!) requirements.


OK :)

One possibility would be to use archivemount to mount the (previous, local) tar archive on a temporary directory. Then rsync as appropriate. Then umount the temporary directory. The end result would be an updated tar archive, which you then split and send over to the destination.

Given that you anyway have to send the whole (splitted) tar archive to the destination (or did I misunderstand that part?) it seems much simpler to recreate the whole tar archive again whenever you want to send it over. It might even be faster than updating the innards of a tar archive (and archivemount might be even slower than unpacking and then re-packing again..)

Good luck.
reinob
 
Posts: 699
Joined: 2014-06-30 11:42

Re: Splitting Tar Archive and updating it [Solved]

Postby larry77 » 2019-07-07 09:26

reinob wrote:
larry77 wrote:I need to archive a large folder (hundreds of GB) and send it online. On the other side, there is someone who will reconstruct the folder with all the chunks.
Yes. I know that rsync makes more sense, but these are (not my own!) requirements.


OK :)

One possibility would be to use archivemount to mount the (previous, local) tar archive on a temporary directory. Then rsync as appropriate. Then umount the temporary directory. The end result would be an updated tar archive, which you then split and send over to the destination.

Given that you anyway have to send the whole (splitted) tar archive to the destination (or did I misunderstand that part?) it seems much simpler to recreate the whole tar archive again whenever you want to send it over. It might even be faster than updating the innards of a tar archive (and archivemount might be even slower than unpacking and then re-packing again..)

Good luck.


Finally, I can only agree with you.
Thanks!
User avatar
larry77
 
Posts: 505
Joined: 2006-09-24 17:59


Return to Programming

Who is online

Users browsing this forum: No registered users and 2 guests

fashionable