[SOLVED] Server with 3 NICs one NIC down, no IP for the LAN

[juhaz]

Server with 3 NICs one NIC down.

It has been great fun setting up the server but now it's time to ask advice to get further.

So I can't figure out why one of the network interface cards won't work. It won't give IP to client PC at LAN.

At my home server/router/firewall. Have a Debian Buster installed on older PC.
Network topology is thru ISP->LTE router/modem->server->LANs. Have 2 LANs thru 2 NICs. One works one not.
Problem could be some setting or hardware.
Configuration should be quite straightforward and the third LAN-config is quite a copy of working one. LAN1 is for Home network, Samba Internet etc. LAN2 is open for Guest and CellPhone LAN. That's the plan, but now the current config is just 2 LANs and attempt to get the basics work.

Following info is quite a mouth full, but it was not so bad once again to take a closer look at the configuration.

Basic hardware is tested and should work despite that certain doubts about that third NIC.

---
NICs are:

Modem/router with DHCP 192.168.1.1
Primary: server with DHCP, nftables enp4s0 192.168.1.2
LAN1: enp5s0 192.168.10.0 gw 192.168.10.1
LAN2: enp5s2 192.168.20.0 gw 192.168.20.1

Server got the DHCP IP from Modem and it is locked to stay same/static.
Server DHCP feeds the LANs and filters the network with nftables.
LAN gateway IPs are static.
---
Latest Debian Buster

root@servu:~# uname -a
Linux servu 4.19.0-6-amd64 #1 SMP Debian 4.19.67-2+deb10u2 (2019-11-11) x86_64 GNU/Linux
---
Server pings ok and another PC too at different (LAN1) 192.168.10.0 network.
root@servu:/# ping 192.168.20.1

PING 192.168.20.1 (192.168.20.1) 56(84) bytes of data.
64 bytes from 192.168.20.1: icmp_seq=1 ttl=64 time=0.075 ms
---
root@servu:~# ip -br link show
lo UNKNOWN 00:00:00:00:00:00 <LOOPBACK,UP,LOWER_UP>
enp5s0 UP 00:22:b0:62:31:4f <BROADCAST,MULTICAST,UP,LOWER_UP>
enp5s2 UP 00:21:91:8d:42:2c <BROADCAST,MULTICAST,UP,LOWER_UP>
enp4s0 UP 00:1c:c0:4e:4b:ce <BROADCAST,MULTICAST,UP,LOWER_UP>
---
arp don't find the clinet PC at 192.168.20.0 network.

root@servu:/# arp
Address HWtype HWaddress Flags Mask Iface
192.168.10.13 ether 04:f1:28:75:2c:09 C enp5s0
nokia6 ether 20:39:56:48:7d:a6 C enp5s0
zyxel ether b8:ec:a3:ec:ef:7f C enp4s0
juhax ether 2c:4d:54:e9:4b:9c C enp5s0
---
root@servu:/# arping -A -I enp5s2 192.168.20.1
ARPING 192.168.20.1
Timeout
---
root@servu:/etc# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: enp5s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 00:22:b0:62:31:4f brd ff:ff:ff:ff:ff:ff
inet 192.168.10.1/24 brd 192.168.10.255 scope global enp5s0
valid_lft forever preferred_lft forever
inet6 fe80::222:b0ff:fe62:314f/64 scope link
valid_lft forever preferred_lft forever
3: enp5s2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 00:21:91:8d:42:2c brd ff:ff:ff:ff:ff:ff
inet 192.168.20.1/24 brd 192.168.20.255 scope global enp5s2
valid_lft forever preferred_lft forever
inet6 fe80::221:91ff:fe8d:422c/64 scope link
valid_lft forever preferred_lft forever
4: enp4s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 00:1c:c0:4e:4b:ce brd ff:ff:ff:ff:ff:ff
inet 192.168.1.2/24 brd 192.168.1.255 scope global dynamic enp4s0
valid_lft 74961sec preferred_lft 74961sec
inet6 fe80::21c:c0ff:fe4e:4bce/64 scope link
valid_lft forever preferred_lft forever
---
Routing:

root@servu:/etc# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 192.168.1.1 0.0.0.0 UG 0 0 0 enp4s0
127.0.0.1 0.0.0.0 255.255.255.255 UH 0 0 0 lo
192.168.1.0 0.0.0.0 255.255.255.0 U 0 0 0 enp4s0
192.168.10.0 0.0.0.0 255.255.255.0 U 0 0 0 enp5s0
192.168.20.0 0.0.0.0 255.255.255.0 U 0 0 0 enp5s2

Tried several routing scenarions which failed.
---
Interface settings:

root@servu:/etc/network# cat interfaces

source /etc/network/interfaces.d/*

# The loopback network interface
auto lo
iface lo inet loopback

# WAN
auto enp4s0
allow-hotplug enp4s0
iface enp4s0 inet dhcp

#LAN1 Kotiverkko suojattu ulkopuolisilta
auto enp5s0
allow-hotplug enp5s0
iface enp5s0 inet static
address 192.168.10.1
netmask 255.255.255.0
network 192.168.10.0
broadcast 192.168.10.255

#LAN2 Avoin ulkopuolisille, vain internet
auto enp5s2
allow-hotplug enp5s2
iface enp5s2 inet static
address 192.168.20.1
netmask 255.255.255.0
network 192.168.20.0
broadcast 192.168.20.255
---
DHCP Server:

root@servu:/etc/dhcp# cat dhcpd.conf
# dhcpd.conf

# servu Debian buster 18.11.2019, 3.12.2019, 13.1.2020, 18.1.2020

log-facility local7;
#log-facility local10;


# Kotiverkko 192.168.10.0 vain omaan käyttöön
# ei vierailijoille, Zyxel -kytkimen kautta
# NIC1
subnet 192.168.10.0 netmask 255.255.255.0 {
authoritative;
range 192.168.10.150 192.168.10.200;
option domain-name-servers 192.168.1.1 , 195.197.54.100 , 195.74.0.47 , 62.121.35.14;
option broadcast-address 192.168.10.255;
option routers 192.168.10.1;
one-lease-per-client true;
get-lease-hostnames true;
default-lease-time 432000;
max-lease-time 864000;
}

# Vierailija verkko 192.168.20.0 Dlink -kytkimen kautta
# NIC2
subnet 192.168.20.0 netmask 255.255.255.0 {
authoritative;
range 192.168.20.2 192.168.20.100;
option domain-name-servers 192.168.1.1 , 195.197.54.100 , 195.74.0.47 , 62.121.35.14;
option broadcast-address 192.168.20.255;
option routers 192.168.20.1;
default-lease-time 360;
max-lease-time 720;
}

# Gateway1, NIC1
host gw1 {
hardware ethernet 00:22:b0:62:31:4f;
}

# Gateway2, NIC2
host gw2 {
hardware ethernet 00:21:91:8d:42:2c;
}

# Zyxel LTE3301 modeemi 192.168.1.1, DefaultGateway
host zyxel {
hardware ethernet b8:ec:a3:ec:ef:7f;
option host-name "zyxel";
}
---
tcpdump and dhclient test:

root@servu:/etc/dhcp# tcpdump -lni enp5s2 arp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on enp5s2, link-type EN10MB (Ethernet), capture size 262144 bytes

and another PC at 192.168.20.0 network is trying to get IP:

root@juha-msi:~#dhclient enp2s0 -v
DHCPDISCOVER on enp2s0 to 255.255.255.255 port 67 .... etc...

No luck...
---
Firewall / NFTables:

root@servu:~# nft list ruleset
table inet filter {
chain input {
type filter hook input priority 0; policy drop;
iif "lo" accept
iif "enp5s0" accept
iif "enp5s2" accept
ct state established,related accept
tcp dport { netbios-ns, netbios-dgm, netbios-ssn, microsoft-ds } accept
udp dport { netbios-ns, netbios-dgm, netbios-ssn, microsoft-ds } accept
tcp dport { ssh, http, https } accept
ip protocol icmp accept
ip6 nexthdr ipv6-icmp accept
ip protocol igmp accept
counter packets 4488 bytes 500585 reject with icmpx type admin-prohibited
}

chain forward {
type filter hook forward priority 0; policy accept;
iif "enp5s2" oif "enp4s0" accept
}

chain output {
type filter hook output priority 0; policy accept;
}
}
table ip nat {
chain prerouting {
type nat hook prerouting priority 0; policy accept;
}

chain postrouting {
type nat hook postrouting priority 0; policy accept;
oif "enp4s0" masquerade
}
}
---
var/log/syslog and dmesg did not show any glues I think.
---
No errors:

>systemctl status networking.service
>systemctl status isc-dhcp-server.service
>systemctl status resolvconf.service

root@servu:~# systemctl --failed
0 loaded units listed.
---
>netdiscover

Currently scanning: 192.168.47.0/16 | Screen View: Unique Hosts

4 Captured ARP Req/Rep packets, from 4 hosts. Total size: 240
_____________________________________________________________________________
IP At MAC Address Count Len MAC Vendor / Hostname
-----------------------------------------------------------------------------
192.168.10.2 2c:4d:54:e9:4b:9c 1 60 ASUSTek COMPUTER INC.
192.168.10.10 20:39:56:48:7d:a6 1 60 HMD Global Oy
192.168.10.101 00:16:01:0d:2c:fa 1 60 BUFFALO.INC
192.168.10.13 04:f1:28:75:2c:09 1 60 HMD Global Oy
---
root@servu:~# ethtool enp5s2
Settings for enp5s2:
Supported ports: [ TP MII ]
Supported link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
Supported pause frame use: No
Supports auto-negotiation: Yes
Supported FEC modes: Not reported
Advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
Advertised pause frame use: Symmetric
Advertised auto-negotiation: Yes
Advertised FEC modes: Not reported
Link partner advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
Link partner advertised pause frame use: Symmetric
Link partner advertised auto-negotiation: Yes
Link partner advertised FEC modes: Not reported
Speed: 100Mb/s
Duplex: Full
Port: MII
PHYAD: 1
Transceiver: internal
Auto-negotiation: on
Supports Wake-on: pumbg
Wake-on: d
Current message level: 0x00000000 (0)

Link detected: yes
---
2020-1-20: edited some typos.
---
All help appreciated!
/j.

[fabien]

Have you tried to swap cards to rule out hardware problem?

[juhaz]

Most warm thanks you all that have been interested the question.
Looks like new MOBO is needed.
Both NICs worked ok. Did swap the cards, no luck. These NICs were DLink PCI NICs. Current Intel MOBO accepted only one add-on card at the time.
Think to get 1GB PCIexp NICs to new MOBO. Will see.
Cheers!