[Closed] What exactly causes rx_drp counters to increment in netstat?

[gffdeb]

Have been spending a great deal of time debugging a 10G ethernet connection to my NAS. At first I was using netstat -i to fine tune interupt and buffer configs to get the link as solid as possible. I see on the forums that I should not trust the netstat results and to use ethtool which should be more accurate. But I still cant figure out why that is true or where the counters in netstat actually come from. See the screenshot attached where ethtool shows no issues but netstat shows 8 rx_drp's. Could the netstat counters be drops by the kernel or somewhere after the ethernet driver? And if so where do I find them and fix them? Thanks.

[Aki]

Hello,

I suppose you are referring to "RX-DRP" (RX DROP); e.g.:

Code: Select all

$ netstat -i 
Kernel Interface table
Iface             MTU    RX-OK RX-ERR RX-DRP RX-OVR    TX-OK TX-ERR TX-DRP TX-OVR Flg
enp2s0           1500        0      0      0 0             0      0      0      0 BMU

[..] I still cant figure out why that is true or where the counters in netstat actually come from.

They should come from /proc/net/dev (see Interface statistics in [1]); this is the netstat source code that prints it:

Code: Select all

[..]
static int iface_info(void)
{
    if (skfd < 0) {
	if ((skfd = sockets_open(0)) < 0) {
	    perror("socket");
	    exit(1);
	}
	printf(_("Kernel Interface table\n"));
    }
    if (flag_exp < 2) {
	ife_short = 1;
	printf(_("Iface             MTU    RX-OK RX-ERR RX-DRP RX-OVR    TX-OK TX-ERR TX-DRP TX-OVR Flg\n"));
    }

    if (for_all_interfaces(do_if_print, &flag_all) < 0) {
	perror(_("missing interface information"));
	exit(1);
    }
    if (flag_cnt)
	if_cache_free();
    else {
	close(skfd);
	skfd = -1;
    }

    return 0;
}
[..]
static int get_dev_fields(const char *bp, struct interface *ife)
{
    switch (procnetdev_vsn) {
    case 3:
	sscanf(bp,
	"%Lu %Lu %lu %lu %lu %lu %lu %lu %Lu %Lu %lu %lu %lu %lu %lu %lu",
	       &ife->stats.rx_bytes,
	       &ife->stats.rx_packets,
	       &ife->stats.rx_errors,
	       &ife->stats.rx_dropped,
	       &ife->stats.rx_fifo_errors,
	       &ife->stats.rx_frame_errors,
	       &ife->stats.rx_compressed,
	       &ife->stats.rx_multicast,

	       &ife->stats.tx_bytes,
	       &ife->stats.tx_packets,
	       &ife->stats.tx_errors,
	       &ife->stats.tx_dropped,
	       &ife->stats.tx_fifo_errors,
	       &ife->stats.collisions,
	       &ife->stats.tx_carrier_errors,
	       &ife->stats.tx_compressed);
	break;
    case 2:
	sscanf(bp, "%Lu %Lu %lu %lu %lu %lu %Lu %Lu %lu %lu %lu %lu %lu",
	       &ife->stats.rx_bytes,
	       &ife->stats.rx_packets,
	       &ife->stats.rx_errors,
	       &ife->stats.rx_dropped,
	       &ife->stats.rx_fifo_errors,
	       &ife->stats.rx_frame_errors,

	       &ife->stats.tx_bytes,static int get_dev_fields(const char *bp, struct interface *ife)
{
    switch (procnetdev_vsn) {
    case 3:
	sscanf(bp,
	"%Lu %Lu %lu %lu %lu %lu %lu %lu %Lu %Lu %lu %lu %lu %lu %lu %lu",
	       &ife->stats.rx_bytes,
	       &ife->stats.rx_packets,
	       &ife->stats.rx_errors,
	       &ife->stats.rx_dropped,
	       &ife->stats.rx_fifo_errors,
	       &ife->stats.rx_frame_errors,
	       &ife->stats.rx_compressed,
	       &ife->stats.rx_multicast,

	       &ife->stats.tx_bytes,
	       &ife->stats.tx_packets,
	       &ife->stats.tx_errors,
	       &ife->stats.tx_dropped,
	       &ife->stats.tx_fifo_errors,
	       &ife->stats.collisions,
	       &ife->stats.tx_carrier_errors,
	       &ife->stats.tx_compressed);
	break;
    case 2:
	sscanf(bp, "%Lu %Lu %lu %lu %lu %lu %Lu %Lu %lu %lu %lu %lu %lu",
	       &ife->stats.rx_bytes,
	       &ife->stats.rx_packets,
	       &ife->stats.rx_errors,
	       &ife->stats.rx_dropped,
	       &ife->stats.rx_fifo_errors,
	       &ife->stats.rx_frame_errors,

	       &ife->stats.tx_bytes,
	       &ife->stats.tx_packets,
	       &ife->stats.tx_errors,
	       &ife->stats.tx_dropped,
	       &ife->stats.tx_fifo_errors,
	       &ife->stats.collisions,
	       &ife->stats.tx_carrier_errors);
	ife->stats.rx_multicast = 0;
	break;
    case 1:
	sscanf(bp, "%Lu %lu %lu %lu %lu %Lu %lu %lu %lu %lu %lu",
	       &ife->stats.rx_packets,
	       &ife->stats.rx_errors,
	       &ife->stats.rx_dropped,
	       &ife->stats.rx_fifo_errors,
	       &ife->stats.rx_frame_errors,

	       &ife->stats.tx_packets,
	       &ife->stats.tx_errors,
	       &ife->stats.tx_dropped,
	       &ife->stats.tx_fifo_errors,
	       &ife->stats.collisions,
	       &ife->stats.tx_carrier_errors);
	ife->stats.rx_bytes = 0;
	ife->stats.tx_bytes = 0;
	ife->stats.rx_multicast = 0;
	break;
    }
    return 0;
}
	       &ife->stats.tx_packets,
	       &ife->stats.tx_errors,
	       &ife->stats.tx_dropped,
	       &ife->stats.tx_fifo_errors,
	       &ife->stats.collisions,
	       &ife->stats.tx_carrier_errors);
	ife->stats.rx_multicast = 0;
	break;
    case 1:
	sscanf(bp, "%Lu %lu %lu %lu %lu %Lu %lu %lu %lu %lu %lu",
	       &ife->stats.rx_packets,
	       &ife->stats.rx_errors,
	       &ife->stats.rx_dropped,
	       &ife->stats.rx_fifo_errors,
	       &ife->stats.rx_frame_errors,

	       &ife->stats.tx_packets,
	       &ife->stats.tx_errors,
	       &ife->stats.tx_dropped,
	       &ife->stats.tx_fifo_errors,
	       &ife->stats.collisions,
	       &ife->stats.tx_carrier_errors);
	ife->stats.rx_bytes = 0;
	ife->stats.tx_bytes = 0;
	ife->stats.rx_multicast = 0;
	break;
    }
    return 0;
}
[..]

--
[1] Linux Kernel - Interface statistics

[gffdeb]

Thanks for the detailed info. Apparently the attachment to my original post failed. And I can't add the screenshot to this one either as it says the board attachment quota has been reached. Sorry I am new to these forums so I don't know what I did wrong there. But really my question revolves around why the rx_drop counters from netstat for a paticular ethernet card are different from the ethtool statistics for the same ethernet card. In my case ethtool reports no drops of any kind but netstat does show some drops.

My only conclusion is that they must be counting different types of rx_drops somehow. Maybe just simply after the driver somewhere in the kernel code or in other buffering upstream.

I will do my best to figure out how to get an attachment to work.

[Aki]

Hello,

my question revolves around why the rx_drop counters from netstat for a particular ethernet card are different from the ethtool statistics for the same ethernet card. In my case ethtool reports no drops of any kind but netstat does show some drops.

When the state of dropping packets occurs, can you report the log (as text) for the two commands in a follow-up message?

You can use code tags to include commands and/or their logs in the body of a message.

[gffdeb]

Maybe the attachment will work this time.

[gffdeb]

Here is a text version of the above two commands I originally asked about showing the difference between rx_drops from netstat or ethtool. As you can see netstat reports 7 drops in this case where ethtool shows none. Hopefully somebody can explain why this happens.

Code: Select all

admin@truenas[~]$ netstat -i
Kernel Interface table
Iface             MTU    RX-OK RX-ERR RX-DRP RX-OVR    TX-OK TX-ERR TX-DRP TX-OVR Flg
enp131s0f0       9000 486506766      0      7 0      433670599      0      0      0 BMRU
lo              65536 22677903      0      0 0      22677903      0      0      0 LRU
admin@truenas[~]$ sudo ethtool -S enp131s0f0 | grep drop                                  
[sudo] password for admin: 
     rx_dropped: 0
     tx_dropped: 0
     port.rx_dropped: 0
     port.tx_dropped_link_down: 0
admin@truenas[~]$

[Aki]

Hello,

The Linux network stack is quite deep and complex [1] [2].

The two commands (ethtool and netstat) explore statistics form different layers of the "network stack":

1. ethtool, with the --statistics or -S option, should explore the NIC (network interface controller) layer; from the ethtool manual page [3]:

   Code: Select all

   ETHTOOL(8) - System Manager's Manual
   [..]
   -S --statistics
       Queries the specified network device for standard (IEEE, IETF, etc.), or NIC- and driver-specific statistics. NIC- and driver-specific statistics are requested when no group of statistics is specified.
   
       NIC- and driver-specific statistics and standard statistics are independent, devices may implement either, both or none. There is little commonality between naming of NIC- and driver-specific statistics across vendors.
   
   --all-groups
   --groups [eth-phy] [eth-mac] [eth-ctrl] [rmon]
       Request groups of standard device statistics.
   

2. netstat, with the --statistics option or -S, should explore higher levels of the network stack; from netstat manual page [4]:

   Code: Select all

   NETSTAT(8) - Linux System Administrator's Manual
   [..]
   --interfaces, -i
   Display a table of all network interfaces.
   
   --statistics, -s
   Display summary statistics for each protocol.
   [..]
   

According to [5], you can also use the dropwatch command to probe the kernel for network packet drops and collect logs to analyse.

By the way, what is your installed Debian version ?

Hope this helps.

note: please, use code tags to include commands and/or their logs in the body of a message. I did it for you in your previous message.

--
[1] The Journey of a Packet Through the Linux Network Stack
[2] Jiri Benc: The Network Packet's Diary: A Kernel Journey
[3] ethtool - System Manager's Manual
[4] netstat - manual page
[5] Red haT Developer - How to retrieve packet drop reasons in the Linux kernel
[6] Simulate delayed and dropped packets on Linux

[gffdeb]

Thanks for the info, very helpful. I spent considerable time testing and adjusting a couple parameters to get the NIC driver to have zero errors and just didn't know where to go to look for the upstream drops. I will definitely check into dropwatch as well. I can't believe I forgot to attach my Debian version. Old age must be catching up with me.

admin@truenas[~]$ lsb_release -a
No LSB modules are available.
Distributor ID: Debian
Description: Debian GNU/Linux 12 (bookworm)
Release: 12
Codename: bookworm
admin@truenas[~]$ uname -r
6.6.29-production+truenas
admin@truenas[~]$

[Aki]

Thanks for the info, very helpful. I spent considerable time testing and adjusting a couple parameters to get the NIC driver to have zero errors and just didn't know where to go to look for the upstream drops. I will definitely check into dropwatch as well.

I'm glad it was helpful. Please, let me know when you are sorted.

I can't believe I forgot to attach my Debian version. Old age must be catching up with me. :)

Code: Select all

admin@truenas[~]$ lsb_release -a
No LSB modules are available.
Distributor ID: Debian
Description:    Debian GNU/Linux 12 (bookworm)
Release:        12
Codename:       bookworm
admin@truenas[~]$ uname -r
6.6.29-production+truenas
admin@truenas[~]$

Don't worry :D It is not the kernel currently distributed with Debian Stable. Probably it is a custom version by truenas.

This is not an usual Debian installation, after all.

[Aki]

Hello @gffdeb,

Have you sorted it out ?

[gffdeb]

I would not say sorted but I have pretty much given up on being able to fix it. Truenas frowns upon allowing dropwatch to be installed and after looking at the kernel parameters that I can tune I have not found a way to match those up with the counters. Luckily it is a very low rate of drops so I don't think its affecting performance. It's just my engineer brain wanting to know what causes it. Truenas will not even try to answer it as they just say thats an upstream Debian issue. Being that a NAS relies heavily on networking I would have expected a bit more knowlege or concern from them.

Anyway I apreciate everyones input, this is truly an awesome forum. Let's consider the issue closed for now.

[Aki]

Anyway I apreciate everyones input, this is truly an awesome forum. Let's consider the issue closed for now.

OK.