Scheduled Maintenance: We are aware of an issue with Google, AOL, and Yahoo services as email providers which are blocking new registrations. We are trying to fix the issue and we have several internal and external support tickets in process to resolve the issue. Please see: viewtopic.php?t=158230

 

 

 

NRPE service doesn't received package from network interface

Linux Kernel, Network, and Services configuration.
Post Reply
Message
Author
Seiken
Posts: 11
Joined: 2022-04-21 14:18

NRPE service doesn't received package from network interface

#1 Post by Seiken »

Hello everyone,

I have a particular problem and I confess that I am short of ideas to find the origin of the problem.
On a Debian10 server (4.19.183-1), I have a nagios-nrpe-server that listen on the port 5666. It's used to run locally some commands from an external Centreon poller.
On the nrpe.cfg, 'allowed_hosts' is correctly configured.

The service running without any problem but I can't connect from the poller (or any server in the same lan) to the TCP port 5666.
I run the last version of NRPE:

Code: Select all

/usr/sbin/nrpe -V
NRPE - Nagios Remote Plugin Executor
Version: 4.1.0
The service is up:

Code: Select all

 systemctl status nagios-nrpe-server
● nagios-nrpe-server.service - Nagios Remote Plugin Executor
   Loaded: loaded (/lib/systemd/system/nagios-nrpe-server.service; enabled; vendor preset: enabled)
  Drop-In: /etc/systemd/system/nagios-nrpe-server.service.d
           └─user-override.conf
   Active: active (running) since Fri 2022-08-05 15:33:05 CEST; 2h 6min ago
     Docs: http://www.nagios.org/documentation
 Main PID: 873 (nrpe)
    Tasks: 1 (limit: 4915)
   Memory: 1.4M
   CGroup: /system.slice/nagios-nrpe-server.service
           └─873 /usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f

Aug 05 15:33:05 myserver systemd[1]: Started Nagios Remote Plugin Executor.
The process run correctly:

Code: Select all

# ps -aux | grep nrpe
monitor+   873  0.0  0.0   6412  4968 ?        Ss   15:33   0:00 /usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f
I have the process that listen on all interface on the port TCP 5666:

Code: Select all

# netstat -ltpna | awk 'NR==2 || /:5666/'
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 0.0.0.0:5666            0.0.0.0:*               LISTEN      873/nrpe            
tcp6       0      0 :::5666                 :::*                    LISTEN      873/nrpe
But from an external server, the port is still see as 'filtered':

Code: Select all

$ nmap 10.25.34.89 -p 5666
Starting Nmap 7.80 ( https://nmap.org ) at 2022-08-05 15:33 UTC
Nmap scan report for (10.25.34.89)
Host is up (0.00088s latency).

PORT     STATE    SERVICE
5666/tcp filtered nrpe
My iptables's rule is flush from all kind of rules that can block anything (doing that for debug purpose of course):

Code: Select all

# iptables -S
-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT
-A INPUT -i eth0 -p tcp -m tcp --dport 5666 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 5666 -j LOG --log-prefix "** NRPE **"
-A FORWARD -p tcp -m tcp --dport 5666 -j LOG --log-prefix "** NRPE **"
-A OUTPUT -p tcp -m tcp --dport 5666 -j LOG --log-prefix "** NRPE **"
On the server himself, I can run 'telnet 127.0.0.1 5666' without any problem:

Code: Select all

# telnet 127.0.0.1 5666
Trying 127.0.0.1...
Connected to 127.0.0.1.
Escape character is '^]'.
Connection closed by foreign host.

From a server in the same LAN or my poller, I can't run 'telnet 10.25.34.89 5666' (you have guessed that 10.25.34.89 is my debian10 server).
With a TCMPDUMP, I correctly see packages from external server:

Code: Select all

# tcpdump -i eth0 port 5666 -vvvvv
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
17:36:10.192023 IP (tos 0x10, ttl 63, id 30978, offset 0, flags [DF], proto TCP (6), length 60)
    10.25.29.33.37728 > 10.25.34.89.nrpe: Flags [S], cksum 0x2058 (correct), seq 937319388, win 64240, options [mss 1460,sackOK,TS val 3613508716 ecr 0,nop,wscale 7], length 0
17:36:11.196915 IP (tos 0x10, ttl 63, id 30979, offset 0, flags [DF], proto TCP (6), length 60)
    10.25.29.33.37728 > 10.25.34.89.nrpe: Flags [S], cksum 0x1c6b (correct), seq 937319388, win 64240, options [mss 1460,sackOK,TS val 3613509721 ecr 0,nop,wscale 7], length 0
Same from the kernel with the log from iptables:

Code: Select all

Aug  5 17:37:41 myserver kernel: IN=eth0 OUT= MAC=00:50:56:b1:cd:74:00:c4:c4:c4:c4:04:08:00 SRC=10.25.29.33 DST=10.25.34.89 LEN=60 TOS=0x10 PREC=0x00 TTL=63 ID=13334 DF PROTO=TCP SPT=50168 DPT=5666 WINDOW=64240 RES=0x00 SYN URGP=0 
Aug  5 17:37:42 myserver kernel: IN=eth0 OUT= MAC=00:50:56:b1:cd:74:00:c4:c4:c4:c4:04:08:00 SRC=10.25.29.33 DST=10.25.34.89 LEN=60 TOS=0x10 PREC=0x00 TTL=63 ID=13335 DF PROTO=TCP SPT=50168 DPT=5666 WINDOW=64240 RES=0x00 SYN URGP=0

I run a strace on the PID himself and I see nothing from the external telnet test:

Code: Select all

strace -f -e trace=network -s 10000 -p 873
I run these commands too but without a clear goal on my part :

Code: Select all

readlink /proc/873/fd/3
	socket:[17763]
	
cat /proc/net/tcp | grep 17763
2: 00000000:1622 00000000:0000 0A 00000000:00000000 00:00000000 00000000   998        0 17763 1 0000000000000000 100 0 0 10 0 

head -1 /proc/873/net/tcp; grep 17763 /proc/873/net/tcp
  sl  local_address rem_address   st tx_queue rx_queue tr tm->when retrnsmt   uid  timeout inode                                                     
   2: 00000000:1622 00000000:0000 0A 00000000:00000000 00:00000000 00000000   998        0 17763 1 0000000000000000 100 0 0 10 0
There is no Apparmor unning on this server.

I am out of ideas.
Something seems to block the sending between the network stack and the process, because I see nothin with the strace but I don't know why.
Is it possible to analyze what is happening on the network stack side?
Can I open the socket file from its inode to see its content?

Thanks in advance for your help.

lindi
Debian Developer
Debian Developer
Posts: 412
Joined: 2022-07-12 14:10
Has thanked: 1 time
Been thanked: 77 times

Re: NRPE service doesn't received package from network interface

#2 Post by lindi »

Does the problem occur if you simply listen to the port using e.g. netcat? That would let us know if this is related to NRPE.

You could try https://lindi.iki.fi/lindi/iptables-debugger to see if there are some other iptables tables that might make a difference here. You should run it with

Code: Select all

sudo ./iptables-debugger -p tcp --dport 5666

User avatar
ralph.ronnquist
Posts: 342
Joined: 2015-12-19 01:07
Location: Melbourne, Australia
Been thanked: 6 times

Re: NRPE service doesn't received package from network interface

#3 Post by ralph.ronnquist »

Maybe add the routing table... does 10.25.34.89 know a route to 10.25.29.33 ?

Seiken
Posts: 11
Joined: 2022-04-21 14:18

Re: NRPE service doesn't received package from network interface

#4 Post by Seiken »

ralph.ronnquist wrote: 2022-08-06 07:52 Maybe add the routing table... does 10.25.34.89 know a route to 10.25.29.33 ?
Yes, I connect to the server from a machine in 10.25.29.33 and can ping machines from 10.25.29.0/24:

Code: Select all

myserver# ping 10.25.29.33
PING 10.25.29.33 (10.25.29.33) 56(84) bytes of data.
64 bytes from 10.25.29.33: icmp_seq=1 ttl=63 time=0.608 ms
64 bytes from 10.25.29.33: icmp_seq=2 ttl=63 time=0.682 ms
Does the problem occur if you simply listen to the port using e.g. netcat? That would let us know if this is related to NRPE.
Yep I tried that too :

Code: Select all

myserver# systemctl stop nagios-nrpe-server
myserver# netcat -l 5666
And from my other machine in 10.25.29.33:

Code: Select all

$ telnet 10.25.34.89 5666
Trying 10.25.34.89...
The return from your script (pretty usefull !) :

Code: Select all

# ./iptables-debugger.sh -p tcp --dport 5666
 m PREROUTING 1 IN=eth0 OUT= SRC=10.25.29.33 DST=10.25.34.89 PROTO=TCP SPT=60650 DPT=5666 SYN 
 n PREROUTING 1 IN=eth0 OUT= SRC=10.25.29.33 DST=10.25.34.89 PROTO=TCP SPT=60650 DPT=5666 SYN 
 n PREROUTING 2 IN=eth0 OUT= SRC=10.25.29.33 DST=10.25.34.89 PROTO=TCP SPT=60650 DPT=5666 SYN 
 m INPUT 1 IN=eth0 OUT= SRC=10.25.29.33 DST=10.25.34.89 PROTO=TCP SPT=60650 DPT=5666 SYN 
 f INPUT 1 IN=eth0 OUT= SRC=10.25.29.33 DST=10.25.34.89 PROTO=TCP SPT=60650 DPT=5666 SYN
 ...(I truncated the other lines, they were the same for each new network packet that arrived)

User avatar
ralph.ronnquist
Posts: 342
Joined: 2015-12-19 01:07
Location: Melbourne, Australia
Been thanked: 6 times

Re: NRPE service doesn't received package from network interface

#5 Post by ralph.ronnquist »

I was looking for the output of "ip route" on your server (10.25.34.89), which is not on the 10.25.29.0/24 network.

Seiken
Posts: 11
Joined: 2022-04-21 14:18

Re: NRPE service doesn't received package from network interface

#6 Post by Seiken »

sorry, I didn't understand that.

So, the current ip route is:

Code: Select all

# ip route
default via 10.25.34.65 dev eth0 onlink 
10.25.34.64/26 dev eth0 proto kernel scope link src 10.25.34.89 
172.17.0.0/16 dev br-5320787b8dfb proto kernel scope link src 172.17.0.1 linkdown

User avatar
ralph.ronnquist
Posts: 342
Joined: 2015-12-19 01:07
Location: Melbourne, Australia
Been thanked: 6 times

Re: NRPE service doesn't received package from network interface

#7 Post by ralph.ronnquist »

RIght, 10.25.34.89 will send any packet for 10.25.29.33 by sending it to its router, 10.25.34.65, which is supposed to send it onward onto the 10.25.29.0/24 network, provided that a) it (10.25.34.65) has such route and b) it forwards packets for its 10.25.34.64/26 network. Does it?

Seiken
Posts: 11
Joined: 2022-04-21 14:18

Re: NRPE service doesn't received package from network interface

#8 Post by Seiken »

Yes, the router 10.25.34.65 knows how to reach the network 10.25.29.0/24.
I usually connect to my server with SSH from 10.25.29.33.

But to be sure, this time I connect on my server directly from VCenter Web Console and do a Ping to 10.25.29.33:

Code: Select all

# ping 10.25.29.33
PING 10.25.29.33 (10.25.29.33) 56(84) bytes of data.
64 bytes from 10.25.29.33: icmp_seq=1 ttl=63 time=0.387 ms
64 bytes from 10.25.29.33: icmp_seq=2 ttl=63 time=0.387 ms
64 bytes from 10.25.29.33: icmp_seq=3 ttl=63 time=0.407 ms
^C
--- 10.25.29.33 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 5ms
rtt min/avg/max/mdev = 0.387/0.393/0.407/0.024 ms
So it's confirm that myServer 10.25.34.89 know how to reach 10.25.29.33

User avatar
ralph.ronnquist
Posts: 342
Joined: 2015-12-19 01:07
Location: Melbourne, Australia
Been thanked: 6 times

Re: NRPE service doesn't received package from network interface

#9 Post by ralph.ronnquist »

Right. I suppose next is the question about firewall on that router. Perhaps that's where port 5666 is filtered?

Seiken
Posts: 11
Joined: 2022-04-21 14:18

Re: NRPE service doesn't received package from network interface

#10 Post by Seiken »

I already thought of this and there is no filtrering by the firewall.
It's confirmed from my first post by correctly seeing the incoming packages with a TCPDUMP on the server himself.

For me, on the network side, the packets arrive well on the server but never to the process.

User avatar
ralph.ronnquist
Posts: 342
Joined: 2015-12-19 01:07
Location: Melbourne, Australia
Been thanked: 6 times

Re: NRPE service doesn't received package from network interface

#11 Post by ralph.ronnquist »

Yes, I'm just making useless noise I'm afraid :( ...
I suppose you've already gone through Nagios support "general troubleshouting tips".

hamishm
Posts: 28
Joined: 2022-07-06 10:28
Has thanked: 2 times
Been thanked: 1 time

Re: NRPE service doesn't received package from network interface

#12 Post by hamishm »

Can you ping FROM 10.25.29.33 to the server? Can you connect with other services like ssh?

Since it doesn't work when you listen with netcat instead of the nrpe server it doesn't seem to be Nagios related, but network/firewall/routing.

Seiken
Posts: 11
Joined: 2022-04-21 14:18

Re: NRPE service doesn't received package from network interface

#13 Post by Seiken »

Yes I can ping my server from10.25.29.33 and I connect to the server with SSH from 10.25.29.33 too.
As I said, the package arrived correctly on the server but not to the process (see strace logs).

So I don't think it's a network/firewall/routing problem, at least not outside the server.

User avatar
ralph.ronnquist
Posts: 342
Joined: 2015-12-19 01:07
Location: Melbourne, Australia
Been thanked: 6 times

Re: NRPE service doesn't received package from network interface

#14 Post by ralph.ronnquist »

Just a final thought: is there an issue with the loopback, lo, interface (on the server)?

Seiken
Posts: 11
Joined: 2022-04-21 14:18

Re: NRPE service doesn't received package from network interface

#15 Post by Seiken »

how can I confirm/infirm that it comes from the loopback interface ?
I don't have a specific test idea to verify this.

hamishm
Posts: 28
Joined: 2022-07-06 10:28
Has thanked: 2 times
Been thanked: 1 time

Re: NRPE service doesn't received package from network interface

#16 Post by hamishm »

Seiken wrote: 2022-08-08 07:57 Yes I can ping my server from10.25.29.33 and I connect to the server with SSH from 10.25.29.33 too.
As I said, the package arrived correctly on the server but not to the process (see strace logs).

So I don't think it's a network/firewall/routing problem, at least not outside the server.
How about an strace of the netcat listen? And strace everything, not just trace=network.

Seiken
Posts: 11
Joined: 2022-04-21 14:18

Re: NRPE service doesn't received package from network interface

#17 Post by Seiken »

I just tried it and I have no good results.

From myserver 10.25.34.89:

Code: Select all

# netcat -l 5666

# ps -aux | grep netcat
root     17254  0.0  0.0   2288  1412 pts/0    S+   08:39   0:00 netcat -l 5666
root     17722  0.0  0.0   6080   816 pts/1    S+   08:39   0:00 grep netcat

# strace -f  -s 10000 -p 17254
strace: Process 17254 attached
accept(3,
From the server 10.25.29.33:

Code: Select all

$ telnet 10.25.34.89 5666
Trying 10.25.34.89...
telnet: Unable to connect to remote host: Connection timed out

And, to be sure, a launched a tcpdump and clearly see packages comming from ETH0, on 10.25.34.89. But strace see nothing:

Code: Select all

# tcpdump -i eth0 port 5666 -vvv
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
08:45:38.848314 IP (tos 0x10, ttl 63, id 51192, offset 0, flags [DF], proto TCP (6), length 60)
    10.25.29.33.59088 > 10.25.34.89.nrpe: Flags [S], cksum 0x413c (correct), seq 1455027003, win 64240, options [mss 1460,sackOK,TS val 3813641711 ecr 0,nop,wscale 7], length 0
08:45:39.872509 IP (tos 0x10, ttl 63, id 51193, offset 0, flags [DF], proto TCP (6), length 60)
    10.25.29.33.59088 > 10.25.34.89.nrpe: Flags [S], cksum 0x3d3c (correct), seq 1455027003, win 64240, options [mss 1460,sackOK,TS val 3813642735 ecr 0,nop,wscale 7], length 0
08:45:41.888003 IP (tos 0x10, ttl 63, id 51194, offset 0, flags [DF], proto TCP (6), length 60)
    10.25.29.33.59088 > 10.25.34.89.nrpe: Flags [S], cksum 0x355c (correct), seq 1455027003, win 64240, options [mss 1460,sackOK,TS val 3813644751 ecr 0,nop,wscale 7], length 0

Seiken
Posts: 11
Joined: 2022-04-21 14:18

Re: NRPE service doesn't received package from network interface

#18 Post by Seiken »

Well, I finally found the origin of the problem. My server had an nftable service enabled (this was not in place to my knowledge).
The file "/etc/nftables.conf" contains a rule indicating which ports are allowed from the outside and, obviously, my nrpe port was not inside.

Thank you all for your help and the time spent on my problem.

Post Reply