Scheduled Maintenance: We are aware of an issue with Google, AOL, and Yahoo services as email providers which are blocking new registrations. We are trying to fix the issue and we have several internal and external support tickets in process to resolve the issue. Please see: viewtopic.php?t=158230

 

 

 

Piping a file to ftp

Programming languages, Coding, Executables, Package Creation, and Scripting.
Post Reply
Message
Author
dryden
Posts: 80
Joined: 2015-02-04 08:54

Piping a file to ftp

#1 Post by dryden »

Maybe this is offtopic for Debian but...

I am trying to do full volume backups on a device with 1 harddisk and a slow connection to a Samba server that happens to be an old network harddisk.

The Samba server has a write speed over my network link of 3 MB/s but over FTP I do 6MB/s.

Now it is possible to use tar + split + a script for split, that will allow you to push the chunks to whatever server you like.

I didn't mention yet that the filesystem of the device is fat32, but this is quite irrelevant now because having chunks has benefits anyway.

The only problem I have currently is that the ftp client I use (lftp) site, sometimes uses SEEKS on the source file which is a PIPE in my case.

Oops!!

The only way I can deal with the pipe issue is by using a queue composed of 2 chunks, with one being uploaded, and the next being fed in by tar.

However the device is very slow and doing "scratch" writing on the same disk is going to lower the tar performance excessively.

So the only solution I know is to use an external USB device like a stick but the problem of course is that those do not deal well with heavy writing.

I am talking about a 780GB backup currently.

A 780GB backup would mean about 390 writes of 2GB to a 16GB USB stick.

So you say why not use a disk for it? Well I only need 4GB really and I would have to hang a 2.5" external HDD from something that is basically also a network disk.

The other solution is to use an ftp client that can be used to upload files from a pipe.
  • Do you have any suggestion for this scratch space queue system?
  • Do you know of an ftp client that could be used to reliably receive data from a pipe for uploading?
That means an ftp client I can feed (script) commands to, please.

dryden
Posts: 80
Joined: 2015-02-04 08:54

Re: Piping a file to ftp

#2 Post by dryden »

I was saying here that I do not know how to reliably dump information from a pipe, but I found that I can use tee --output-error=warn to keep tee going even if the pipe breaks down.

So that means I can use ftp when everything is going alright, and when it starts to do weird stuff I can just switch to smb for one chunk.


So now I only need 2GB of space for scratches and itches.

But that doesn't change a whole lot, just makes the code easier.

Post Reply