[Solved] Slower than expected file transfers on 10gb ethernet

[jaimarti]

Not sure what I am missing? I have slower than what i Would expected on file transfers on 10gb ethernet from NVME to NVME over SMB and NFS. Test transfer of a 30 GB Folder containing mixed files @ 226 MB/s.

My Setup is 2 HP z620 dual e5-2670, Broadcom 57810S Dual Port 10Gb NIC RJ45 MTU 9000 and 1 each NVMe WD Green SN350 1TB per machine.

First HP z620 is running Debian 12 using Proxmox 8.1.4 as a hypervisor and Open Media Vault Version 6.5.0-3(Shaitan) running as a VM SMB Server
Open Media Vault SMB Shares.
First SMB is Raid 5 with 3 HDD
read speed on a 30 GigByte Folder containing mixed files 196 MB/s. Expectation would be around 220 so it is close slightly low.
Second SMB is NVMe WD Green SN350 1TB.
read speed on a 30 GigByte Folder containing mixed files 226 MB/s. Expectation would be saturating the 10 Gb link at 900+
Only 30MB/s difference between Raid 5 and NVME?
Second HP z620 is running Debian 12 desktop


Drive is PCI(E) passed through on Proxmox node no option to test drive speed at this point

Drive Speed Proxmox VM

Code: Select all

sudo hdparm -Tt /dev/sdx

/dev/nvme0n1:
Timing cached reads: 16896 MB in 1.99 seconds = 8487.12 MB/sec
Timing buffered disk reads: 3784 MB in 3.00 seconds = 1260.86 MB/sec

Drive Speed Desktop

Code: Select all

sudo hdparm -Tt /dev/nvme0n1

/dev/nvme0n1:
Timing cached reads: 17450 MB in 1.99 seconds = 8772.46 MB/sec
Timing buffered disk reads: 2904 MB in 3.00 seconds = 967.67 MB/sec

Iperf3 results between 2 PC’s
root@jaime-hpz620workstation:/etc/samba#

Code: Select all

iperf3 -c 10.0.40.169

Connecting to host 10.0.40.169, port 5201
[ 5] local 10.0.40.175 port 40206 connected to 10.0.40.169 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 1.12 GBytes 9.62 Gbits/sec 0 3.15 MBytes
[ 5] 1.00-2.00 sec 1.15 GBytes 9.86 Gbits/sec 0 3.15 MBytes
[ 5] 2.00-3.00 sec 1.14 GBytes 9.82 Gbits/sec 0 3.15 MBytes
[ 5] 3.00-4.00 sec 1.14 GBytes 9.82 Gbits/sec 0 3.15 MBytes
[ 5] 4.00-5.00 sec 1.15 GBytes 9.87 Gbits/sec 0 3.15 MBytes
[ 5] 5.00-6.00 sec 1.15 GBytes 9.85 Gbits/sec 0 3.15 MBytes
[ 5] 6.00-7.00 sec 1.15 GBytes 9.88 Gbits/sec 0 3.15 MBytes
[ 5] 7.00-8.00 sec 1.15 GBytes 9.85 Gbits/sec 0 3.15 MBytes
[ 5] 8.00-9.00 sec 1.13 GBytes 9.71 Gbits/sec 0 3.15 MBytes
[ 5] 9.00-10.00 sec 1.15 GBytes 9.90 Gbits/sec 0 3.15 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 11.4 GBytes 9.82 Gbits/sec 0 sender
[ 5] 0.00-10.04 sec 11.4 GBytes 9.77 Gbits/sec receiver

Although I have excellent throughput via the iperf3 results I did swap the cable to see if that was the issue no change. I did test this with a windows10 device with an SSD and the same 30 GB folder. It starteed at 800Mb and after about 10 seconds droped down to about 150 so I am guess SSD cahe was exhausted.

Any suggestions appreciated.

[jaimarti]

I think I figured it out thanks to https://www.servethehome.com/wd-green-s ... -review/3/ It appears that the NVME's I choose are not the best performers. I guess I will need to buy the some better NVME's

[Aki]

I think I figured it out thanks to https://www.servethehome.com/wd-green-s ... -review/3/ It appears that the NVME's I choose are not the best performers. I guess I will need to buy the some better NVME's

I'm glad you sorted it out. :)

Please, mark the discussion as "solved" manually adding the text tag "[Solved]" at the beginning of the subject of the first message (after other tags, if any); i.e. :

[Solved] Slower than expected file transfers on 10gb ethernet

Happy Debian !

--
note: moved to "Hardware" sub-forum.

[jaimarti]

I was able to go NVME to SSD and read rate was 800MB/s so definetly the wright speed on NVME. Thanks