Scheduled Maintenance: We are aware of an issue with Google, AOL, and Yahoo services as email providers which are blocking new registrations. We are trying to fix the issue and we have several internal and external support tickets in process to resolve the issue. Please see: viewtopic.php?t=158230

 

 

 

Transfer speed slow (small files)

Linux Kernel, Network, and Services configuration.
Post Reply
Message
Author
Sedin
Posts: 2
Joined: 2017-04-20 06:26

Transfer speed slow (small files)

#1 Post by Sedin »

Greetings and thanks for having me!

Scenario
I have two systems running debian 8.6, my working machine and a server. The server is used as backup for my machine.
The server ist located in a datacenter, the working machine is with me.
The server has a 100Mbit/s up and down connection and a 10Gbit NIC.
The connection used by my working machine changes (I travel a lot) using a 1Gbit NIC.
I backup all my files once a week using sshfs to connect to the server and rsync to copy the files.

Issue
The transfer speed gets extremly slow while transfering small files. The smaller the files, the slower the transfer speed.
If I backup some bigger files I cap the available uplink connection, if I backup small files I just partially use it.
I know that it´s common for smaller files to take longer to transfer due to overheading.
It doesn´t matter if I use a faster or slower uplink connection (see example below)

Example
If I transfer 200 files of 1KiB each using a
- 1 Mbit/s uplink connection it takes ~37s
- 10 Mbit/s uplink connection it takes ~46s
to complete the transfer.
The files were created by looping dd if=/dev/random

For comparison, backing up a 200 KiB file takes less than a second.

Investigation
The 1 Mbit/s uplink connection should be slower than the 10Mbit/s uplink connection, but their RTTs (measured by ping) differ.
The 1 Mbit/s uplink connection has a ~20ms RTT, the 10Mbit/s a ~31ms RTT.

According to my research increasing the TCP window size might speed up the transfer.
I increased the TCP window size on server and client, but nothing changed.

Next I tried to play around with the options of sshfs - kernel_cache, cache, compression and cipher - without an increase in transfer speed.
I also tried using cp instead of rsync, without a difference in transfer speed.

Client and server are both bored during the transfer, load <1, RAM utilization <10%, HDDs IO is fine as well.
The network utilization is fine as well, I copy files with ~5KB/s and the Utilization shows ~40Kbit/s.

I wasn´t to sure if this is a sshfs or a networking issue, so I installed two VMs on my local machine and transfered 10000 1KiB files using sshfs.. in less than 3 seconds.

Question
After all the above I´m pretty sure this is a tcp networking issue, but I´m not too sure what else to test/change to get a better transfer speed for my small files.
My question is pretty simple to ask and probably a lot harder to answer:

Is there anything that I can change to increase my transfer speed for smaller files or am I fighting a lost battle against the downsides of tcp and wan connections?

User avatar
phenest
Posts: 1702
Joined: 2010-03-09 09:38
Location: The Matrix

Re: Transfer speed slow (small files)

#2 Post by phenest »

1. Has this problem existed since you first installed Debian?
2. Are you using a router or a direct cable link?
3. Can I assume you've checked your hardware thoroughly?
ASRock H77 Pro4-M i7 3770K - 32GB RAM - Pioneer BDR-209D

Cefiar
Posts: 18
Joined: 2017-03-25 22:50

Re: Transfer speed slow (small files)

#3 Post by Cefiar »

The issue here is most likely due to the RTT.

The longer it takes to get a response from the other end of the link will be the limiting factor.

Basically what's happening is that rsync (or cp) pushes the data across the link, then asks the other end to confirm it's done the copy for that file. The data is small, so the larger part of the time here is the confirmation that's it's been written.

The only ways to solve that would be to group all the files into a few files first, or rewrite the programs (or your script) to have some degree parallelism (higher chances the link will remain in use).

You could try tar'ing the files first and backing up the .tar, if you don't mind a .tar file on the destination as your backup. Don't compress it if you're using rsync though, as you want the uncompressed version so that rsync can perform data comparisons and only send the differences.

FWIW: If you want to emulate network RTT delays, you can use tc and the netem functionality to set delays in your traffic. A good reference on what it can do: hhttps://wiki.linuxfoundation.org/networking/netem - Note that I haven't personally played with this functionality since the kernel 2.6 days. Also, I don't know if this is compiled into the Debian kernels by default.

Sedin
Posts: 2
Joined: 2017-04-20 06:26

Re: Transfer speed slow (small files)

#4 Post by Sedin »

Thank you Cefiar, your link was worth a lot! Btw, netem is still compiled in the debian kernel by default (3.16.0-4-amd64).

I was able to simulate my system, including the RTT, in my virtualized environment. I actually made a mistake before, the 10000 files in <3 sec was wrong.
I´m getting the same transfer speeds, more or less, as in my live system.

Rewriting my script sadly isn´t an option, I need the rsync comparison. :(

Guess I will have to live with it.. Thank you guys!

Post Reply