Any ideas to optimize high UDP load?

Kernels & Hardware, configuring network, installing services

Any ideas to optimize high UDP load?

Postby ClanFeaR » 2018-03-01 22:07

Good evening!

I have a system 3.16.0-4-amd64 #1 SMP Debian 3.16.43-2+deb8u5 with server application installed with 200-1000 simultaneously connected clients.
The communication is made using UDP protocol, average client session may consist from 5 up to hundreds of packets, mainly incoming (client to server). The packet size at the iptraf histogram is:
1-73: 36%
74-146: 47%
147-219: 16%
The bandwidth used from 128 kbit/s up to 1 Mbit/s. The packets per second is high (now evening, 113 kbit/s and 105 packets/s).
Technically it does not take high amount of system resources (now 1.5% CPU), all client requests are processed on the fly.
Sometimes I noticed the problem "server did not reply" at the client, but the reproducibility of this trouble is very low and it's hard to repeat it.

Recently I read some papers about DNS configuration and related high-load problems and figured out that furing month+ of server usage with no restarts netstat -s shows extremely high error values, e.g.
Udp:
4629113825 packets received
818260 packets to unknown port received.
10461880 packet receive errors
5309885 packets sent
RcvbufErrors: 10461880

Lets say about 4 month after last reboot, average about 487 pps and 1 error per second. Errors are happening irregularly, i.e. during the time I'm writing these lines it's count is the same, so I guess during high number of clients connected.

As I have 3 same configured VPS at 2 providers in parralel with same stats and same problems at each of them, I started to tune one for tests. After I set
net.core.netdev_max_backlog=10000
net.core.wmem_max = 33554432
net.core.rmem_max = 33554432
net.core.rmem_default = 8388608
net.core.wmem_default = 4194394

25.5 hours pased and I see
Udp:
44912609 packets received
16204 packets to unknown port received.
53012 packet receive errors
272450 packets sent
RcvbufErrors: 52993
InCsumErrors: 15

Average 489 pps, one error each 1.7 seconds! Again, packets should not wait in queue. They are received from udp socket using a separate thread, some of them are replied immediately, other are stored to the local database. There are no bottle-necks at the storage speed etc. Earlier I thought that hosting provider filters some packets or there are problems but now I see it's related to the socket or system configuration related to the network.

Do somebody have any ideas why do these packets sometime lost and how to avoid it?
ClanFeaR
 
Posts: 1
Joined: 2018-03-01 21:19

Re: Any ideas to optimize high UDP load?

Postby ksu » 2018-03-15 20:02

Often this is related to the NIC driver or the NIC itself - not considering other networking factors.
I have noticed that on systems with the RTL driver and hardware is rather semi-stable, and gives headaches, oopses etc, especially on multi-nic configs.

If you are using Intel cards, disable the checksum offload feature:
$ ethtool --offload ethX rx off tx off

and the tcp segmentation offloading:
$ ethtool -K ethX tso off
$ ethtool -K ethX gso off

see if that gets you anywhere closer....

Mike
User avatar
ksu
 
Posts: 49
Joined: 2014-01-13 14:59

Re: Any ideas to optimize high UDP load?

Postby reinob » 2018-03-17 12:07

ClanFeaR wrote:Do somebody have any ideas why do these packets sometime lost and how to avoid it?


You do know that UDP is -- by definition -- an unreliable protocol do you?
You will have lost packets, you will have packets received out-of-order, you will have duplicate packets, etc.

If you want reliability, use TCP :)
reinob
 
Posts: 541
Joined: 2014-06-30 11:42

Re: Any ideas to optimize high UDP load?

Postby wizard10000 » 2018-03-17 18:25

To expand on what reinob said, UDP is a connectionless protocol; this means it does not verify that a connection is actually established before it starts slinging packets.
we see things not as they are, but as we are.
-- anais nin
User avatar
wizard10000
 
Posts: 1279
Joined: 2011-05-09 20:02
Location: everywhere i go, there i am!


Return to System configuration

Who is online

Users browsing this forum: No registered users and 8 guests

fashionable