Scheduled Maintenance: We are aware of an issue with Google, AOL, and Yahoo services as email providers which are blocking new registrations. We are trying to fix the issue and we have several internal and external support tickets in process to resolve the issue. Please see: viewtopic.php?t=158230

 

 

 

[SOLVED] DNS Suddenly STOPPED! Stretch 9.5 updated regularly

Linux Kernel, Network, and Services configuration.
Post Reply
Message
Author
User avatar
w4kh
Posts: 98
Joined: 2006-09-09 19:10
Location: Alabama, USA

[SOLVED] DNS Suddenly STOPPED! Stretch 9.5 updated regularly

#1 Post by w4kh »

I was just "surfing" this morning and suddenly the music stream stopped, and every tab on the browser showed:
No internet
Try:
Checking the network cables, modem, and router
Reconnecting to Wi-Fi
DNS_PROBE_FINISHED_NO_INTERNET

Initially, I thought my internet connection had gone down - much construction on the main road past the subdivision, and utilities being moved to new poles - but Windows was still going while Stretch gagged and halted.

I had not finished my usual round before running the daily updater - given the DNS issue I did NOT run this, which is usually run daily:
#! /bin/bash
# Update the repositories and then process upgrades
/usr/bin/apt-get update
sleep 5
UPLOCK=/var/lib/apt/lists/lock
if [ -f $UPLOCK ]; then
rm $UPLOCK
fi
echo "Now, to upgrade"
/usr/bin/apt-get -f upgrade
# Run rkhunter
echo "Running Rootkit Hunter"
/usr/bin/rkhunter --propupd
#And, now to update the tripwire database
echo "updating the tripwire database"
/usr/local/bin/uptripw

But instead ran just 'apt-get update' which returned:
Err:1 http://security.debian.org stretch/updates InRelease
Temporary failure resolving 'security.debian.org'

I knew something bad had happened... but what?

Hardware all seems solid - nothing new added in a long time
32GB RAM, 8-core AMD CPU...
System kept up-to-date daily.
No joy on Google or searching the Forums (unless I used a poor choice of search terms)
I am at a loss (and resigned to having to address this forum with <gag> Windows10 on my wife's desktop!)
Last edited by w4kh on 2018-08-25 20:09, edited 1 time in total.
4.19.0-9-amd64 #1 SMP Debian 4.19.118-2+deb10u1
CPU: AMD FX(tm)-8350 Eight-Core Processor
RAM: 32GB (8x8GB) DDR3 DRAM
Video: GeForce 8400 GS to VIZIO E320VA Monitor

Segfault
Posts: 993
Joined: 2005-09-24 12:24
Has thanked: 5 times
Been thanked: 17 times

Re: DNS Suddenly STOPPED! Stretch 9.5 updated regularly

#2 Post by Segfault »

Basics first. Can you access web sites by IP address? http://217.196.43.138/

p.H
Global Moderator
Global Moderator
Posts: 3049
Joined: 2017-09-17 07:12
Has thanked: 5 times
Been thanked: 132 times

Re: DNS Suddenly STOPPED! Stretch 9.5 updated regularly

#3 Post by p.H »

w4kh wrote:DNS_PROBE_FINISHED_NO_INTERNET
What does this come from ?
Segfault wrote:Basics first. Can you access web sites by IP address?
"Basics" ? A web browser is an overcomplicated piece of software unsuitable for network troubleshooting.
To me, basics are IP configuration lookup with ip addr and ip route, IP connectivity check with ping and traceroute, DNS lookup with host, dig or nslookup.

Segfault
Posts: 993
Joined: 2005-09-24 12:24
Has thanked: 5 times
Been thanked: 17 times

Re: DNS Suddenly STOPPED! Stretch 9.5 updated regularly

#4 Post by Segfault »

Segfault wrote:Basics first. Can you access web sites by IP address?
"Basics" ? A web browser is an overcomplicated piece of software unsuitable for network troubleshooting.
To me, basics are IP configuration lookup with ip addr and ip route, IP connectivity check with ping and traceroute, DNS lookup with host, dig or nslookup.
I never said to use a web browser. I asked if web sites are accessible by IP address. Leaving it for OP to decide how. Try to think more clearly.

User avatar
w4kh
Posts: 98
Joined: 2006-09-09 19:10
Location: Alabama, USA

Re: DNS Suddenly STOPPED! Stretch 9.5 updated regularly

#5 Post by w4kh »

Segfault wrote:
Basics first. Can you access web sites by IP address? http://217.196.43.138/
Yes... I can maneuver by IPv4 address - with some limits...

a ping by IPv4 address results in:
ping: sendmsg: Operation not permitted
ping: sendmsg: Operation not permitted
ping: sendmsg: Operation not permitted
ping: sendmsg: Operation not permitted
ping: sendmsg: Operation not permitted
and,
a ping by URL gets:
ping: http://www.siteyouwant.com: Temporary failure in name resolution

Oddest thing is, I can get to some Debian URLs with no problem:
i.e., https://bugs.debian.org/release-critical/
However, I suspect that is from cache, since that URL gets hit daily
p.H wrote:
"DNS_PROBE_FINISHED_NO_INTERNET"
What does this come from ?
Google Chrome Browser... everything just suddenly stopped and that came up as the last "sentence" of the error message.
4.19.0-9-amd64 #1 SMP Debian 4.19.118-2+deb10u1
CPU: AMD FX(tm)-8350 Eight-Core Processor
RAM: 32GB (8x8GB) DDR3 DRAM
Video: GeForce 8400 GS to VIZIO E320VA Monitor

p.H
Global Moderator
Global Moderator
Posts: 3049
Joined: 2017-09-17 07:12
Has thanked: 5 times
Been thanked: 132 times

Re: DNS Suddenly STOPPED! Stretch 9.5 updated regularly

#6 Post by p.H »

Segfault wrote:I never said to use a web browser. I asked if web sites are accessible by IP address. Leaving it for OP to decide how. Try to think more clearly.
How then do you suggest to check if websites are available ? Try to write more clearly.
Non-browser HTTP clients which understand URLs such as wget or curl are nearly as complicated as web browsers.
w4kh wrote: that came up as the last "sentence" of the error message.
Why didn't you provide provide the full error message ?
w4kh wrote:ping: sendmsg: Operation not permitted
This error usually means that outgoing packets are blocked by firewall rules. Check iptables rules with iptables-save.
If it is intended that outgoing ICMP echo request (aka "ping") are blocked for security reasons, then you have to use another tool and protocol to check connectivity. Standard traceroute sends UDP packets, tcptraceroute sends TCP packets as you can guess.
w4kh wrote:a ping by URL gets:
ping: http://www.siteyouwant.com: Temporary failure in name resolution
This command cannot work : ping does not understand URLs and expects a host/domain name or an IP address.

If the dnsutils package is installed, you can test the DNS resolution with something like

Code: Select all

dig some.domain.name
host some.domain.name
Also, check the contents of /etc/resolv.conf.

User avatar
w4kh
Posts: 98
Joined: 2006-09-09 19:10
Location: Alabama, USA

Re: DNS Suddenly STOPPED! Stretch 9.5 updated regularly

#7 Post by w4kh »

p.H wrote:
w4kh wrote: that came up as the last "sentence" of the error message.
Why didn't you provide provide the full error message ?
Actually, I did... the entire error message is:

Code: Select all

No internet
Try:
Checking the network cables, modem, and router
Reconnecting to Wi-Fi
DNS_PROBE_FINISHED_NO_INTERNET
And when I tried:

Code: Select all

dig www.charter.com
host www.charter.com
This happened:

Code: Select all

; <<>> DiG 9.10.3-P4-Debian <<>> www.charter.com
;; global options: +cmd
;; connection timed out; no servers could be reached
root@bigmutt:~# host www.charter.com
;; connection timed out; no servers could be reached
w4kh wrote:ping: sendmsg: Operation not permitted
p.H wrote: This error usually means that outgoing packets are blocked by firewall rules. Check iptables rules with iptables-save.
If it is intended that outgoing ICMP echo request (aka "ping") are blocked for security reasons, then you have to use another tool and protocol to check connectivity. Standard traceroute sends UDP packets, tcptraceroute sends TCP packets as you can guess.
NOTHING changed (that I am aware of)... Auto-update is not active; I had not yet run my daily apt-get update/apt-get upgrade... the system just suddenly went sideways and DNS/BIND9 ceased working. Since I was working using the Debian machine, I assumed (yeah, I know...) that I had lost internet due to the road construction behind our house - it had happened before. Then I found the Windows machine could surf and Debian was knocked out, and I tried to figure out what was going on.
Also, check the contents of /etc/resolv.conf.

Code: Select all

search mydomain.com
nameserver 8.8.8.8
# NOTE: the libc resolver may not support more than 3 nameservers.
# The nameservers listed below may not be recognized.
nameserver 2602:306:3bef:51b0::1
4.19.0-9-amd64 #1 SMP Debian 4.19.118-2+deb10u1
CPU: AMD FX(tm)-8350 Eight-Core Processor
RAM: 32GB (8x8GB) DDR3 DRAM
Video: GeForce 8400 GS to VIZIO E320VA Monitor

User avatar
debiman
Posts: 3063
Joined: 2013-03-12 07:18

Re: DNS Suddenly STOPPED! Stretch 9.5 updated regularly

#8 Post by debiman »

Code: Select all

systemctl status systemd-resolved
sudo ping -c3 8.8.8.8
sudo ping -c3 google.com
oh and what software do you use to connect to the internet? networkmanager?

i don't use tripwire; could it be responsible somehow?

User avatar
w4kh
Posts: 98
Joined: 2006-09-09 19:10
Location: Alabama, USA

Re: DNS Suddenly STOPPED! Stretch 9.5 updated regularly

#9 Post by w4kh »

debiman wrote:

Code: Select all

systemctl status systemd-resolved

Code: Select all

systemctl status systemd-resolved
● systemd-resolved.service - Network Name Resolution
   Loaded: loaded (/lib/systemd/system/systemd-resolved.service; disabled; vendor preset: enabled)
  Drop-In: /lib/systemd/system/systemd-resolved.service.d
           └─resolvconf.conf
   Active: inactive (dead)
     Docs: man:systemd-resolved.service(8)
           http://www.freedesktop.org/wiki/Software/systemd/resolved
           http://www.freedesktop.org/wiki/Software/systemd/writing-network-configuration-manag
           http://www.freedesktop.org/wiki/Software/systemd/writing-resolver-clients
It appears as though /lib/systemd/system/systemd-resolved.service is disabled, and the man pages give no hint that my fatigued eyes could find as to how to enable it. Looking a tad further, Both the man page for systemd-resolved and for systemd-resolved.service refer to a file /run/systemd/resolve/resolv.conf, HOWEVER, there is NO "resolve" directory under /run/systemd, and (obviously?) no resolv.conf in the non-existent /run/systemd/resolve directory
debiman wrote:oh and what software do you use to connect to the internet? networkmanager?
NetworkManager
debiman wrote:i don't use tripwire; could it be responsible somehow?
I doubt it... TripWire's primary function is to monitor changes to files, especially "system files", as indications of intrusion
debiman wrote:

Code: Select all

sudo ping -c3 8.8.8.8

Code: Select all

ping -c3 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
ping: sendmsg: Operation not permitted
ping: sendmsg: Operation not permitted
ping: sendmsg: Operation not permitted

--- 8.8.8.8 ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 2027ms
debiman wrote:

Code: Select all

sudo ping -c3 google.com

Code: Select all

ping -c3 google.com
ping: google.com: Temporary failure in name resolution
Me thinks the missing "/run/systemd/resolve/resolv.conf" has a great deal to do with the failure, but should its structure and content be exactly like /etc/resolv.conf??
Keeping track of actions taken, in order than reversal is "possible"

Code: Select all

mkdir /run/systemd/resolve
chown root:root /run/systemd/resolve
chmod 755 /run/systemd/resolve
cp -p /etc/resolv.conf /run/systemd/resolve/
ls -l /run/systemd/resolve/resolv.conf
-rw-r--r-- 1 root root 250 Aug 18 10:33 /run/systemd/resolve/resolv.conf
Still NO joy! :( and, I am at a loss to explain what has happened. Real problem, for me, is how to explain everything working, and suddenly "POOF!" with no change made... I was just visiting the daily sites...
4.19.0-9-amd64 #1 SMP Debian 4.19.118-2+deb10u1
CPU: AMD FX(tm)-8350 Eight-Core Processor
RAM: 32GB (8x8GB) DDR3 DRAM
Video: GeForce 8400 GS to VIZIO E320VA Monitor

User avatar
Head_on_a_Stick
Posts: 14114
Joined: 2014-06-01 17:46
Location: London, England
Has thanked: 81 times
Been thanked: 132 times

Re: DNS Suddenly STOPPED! Stretch 9.5 updated regularly

#10 Post by Head_on_a_Stick »

w4kh wrote:Me thinks the missing "/run/systemd/resolve/resolv.conf" has a great deal to do with the failure
No, /run/systemd/resolve/resolv.conf is only created if systemd-resolved.service is running.

You are not using (and do not need to use) systemd-resolved and furthermore the recommended mode of operation is

Code: Select all

# ln -sf /lib/systemd/resolv.conf /etc/resolv.conf
Which should give

Code: Select all

root@hegel:~# grep -v '^#\|^$' /etc/resolv.conf 
nameserver 127.0.0.53
root@hegel:~#
Anyway, I have no idea why debiman has brought systemd-resolved up, it doesn't seem to have anything to do with your problem.

Can we please see the full output of

Code: Select all

ip a
Thanks!
deadbang

User avatar
w4kh
Posts: 98
Joined: 2006-09-09 19:10
Location: Alabama, USA

Re: DNS Suddenly STOPPED! Stretch 9.5 updated regularly

#11 Post by w4kh »

Head_on_a_Stick wrote:
w4kh wrote:Me thinks the missing "/run/systemd/resolve/resolv.conf" has a great deal to do with the failure
No, /run/systemd/resolve/resolv.conf is only created if systemd-resolved.service is running.

You are not using (and do not need to use) systemd-resolved and furthermore the recommended mode of operation is

Code: Select all

# ln -sf /lib/systemd/resolv.conf /etc/resolv.conf
Which should give

Code: Select all

root@hegel:~# grep -v '^#\|^$' /etc/resolv.conf 
nameserver 127.0.0.53
root@hegel:~#
And It did...
Head_on_a_Stick wrote:Anyway, I have no idea why debiman has brought systemd-resolved up, it doesn't seem to have anything to do with your problem.

Can we please see the full output of

Code: Select all

ip a
Thanks!
Here it is:

Code: Select all

ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether fc:aa:14:58:58:c7 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.42/24 brd 192.168.1.255 scope global eth0
       valid_lft forever preferred_lft forever
In case it is not obvious, eth0 address is static... "things" (and a couple laptops) are the only dhcp clients on the LAN
4.19.0-9-amd64 #1 SMP Debian 4.19.118-2+deb10u1
CPU: AMD FX(tm)-8350 Eight-Core Processor
RAM: 32GB (8x8GB) DDR3 DRAM
Video: GeForce 8400 GS to VIZIO E320VA Monitor

User avatar
Head_on_a_Stick
Posts: 14114
Joined: 2014-06-01 17:46
Location: London, England
Has thanked: 81 times
Been thanked: 132 times

Re: DNS Suddenly STOPPED! Stretch 9.5 updated regularly

#12 Post by Head_on_a_Stick »

w4kh wrote:And It did...
That wasn't a suggestion, I was just pointing out how to configure systemd-resolved, I *do not* think you should be trying systemd-resolved, it just adds another thing that might be going wrong.

Remove the symlink and replace /etc/resolv.conf with a single line:

Code: Select all

nameserver 8.8.8.8
Does `ping google.com` work now?

EDIT: we should probably also see

Code: Select all

ip r
deadbang

User avatar
w4kh
Posts: 98
Joined: 2006-09-09 19:10
Location: Alabama, USA

Re: DNS Suddenly STOPPED! Stretch 9.5 updated regularly

#13 Post by w4kh »

Head_on_a_Stick wrote:
w4kh wrote:And It did...
That wasn't a suggestion, I was just pointing out how to configure systemd-resolved, I *do not* think you should be trying systemd-resolved, it just adds another thing that might be going wrong.
Agreed...
Head_on_a_Stick wrote:Remove the symlink and replace /etc/resolv.conf with a single line:

Code: Select all

nameserver 8.8.8.8
Does `ping google.com` work now?

Code: Select all

ping google.com
ping: google.com: Temporary failure in name resolution
Head_on_a_Stick wrote:EDIT: we should probably also see

Code: Select all

ip r

Code: Select all

ip r
default via 192.168.1.254 dev eth0 onlink
192.168.1.0/24 dev eth0 proto kernel scope link src 192.168.1.42
4.19.0-9-amd64 #1 SMP Debian 4.19.118-2+deb10u1
CPU: AMD FX(tm)-8350 Eight-Core Processor
RAM: 32GB (8x8GB) DDR3 DRAM
Video: GeForce 8400 GS to VIZIO E320VA Monitor

User avatar
Head_on_a_Stick
Posts: 14114
Joined: 2014-06-01 17:46
Location: London, England
Has thanked: 81 times
Been thanked: 132 times

Re: DNS Suddenly STOPPED! Stretch 9.5 updated regularly

#14 Post by Head_on_a_Stick »

w4kh wrote:

Code: Select all

default via 192.168.1.254 dev eth0 onlink
Is that correct? Is your gateway at that address?

Does the ethernet card work if you load up a Debian "live" image?
deadbang

p.H
Global Moderator
Global Moderator
Posts: 3049
Joined: 2017-09-17 07:12
Has thanked: 5 times
Been thanked: 132 times

Re: DNS Suddenly STOPPED! Stretch 9.5 updated regularly

#15 Post by p.H »

Also, after checking the output of iptables-save, could you provide the full output of

Code: Select all

traceroute -n 8.8.8.8

User avatar
debiman
Posts: 3063
Joined: 2013-03-12 07:18

Re: DNS Suddenly STOPPED! Stretch 9.5 updated regularly

#16 Post by debiman »

w4kh wrote:It appears as though /lib/systemd/system/systemd-resolved.service is disabled
it is well possible that name resolution is achieved by different means.
but since you can't ping IPs directly either, this is not crucial atm.
i didn't think this through, just went by thread title...

i could throw in a few wild guesses, but i will wait what HoaS' line of enquiry brings up.

kevinthefixer
Posts: 190
Joined: 2018-05-05 22:30

Re: DNS Suddenly STOPPED! Stretch 9.5 updated regularly

#17 Post by kevinthefixer »

Not to interrupt the gurus, but the first thing I'd try is rebooting the hardware, all of it, with special attention given to the router if present. Amazing how often something that simple fixes major problems.

As an aside, I once had a user on an Android forum explain that he didn't think he had to go to "all the trouble" because he didn't think there was anything wrong with his router. After he'd asked for advice! Trouble rebooting a router? I recommend that be done at least monthly just as a routine maintenance item, it takes about 5 minutes, mostly waiting 2-3 minutes to be sure the RAM is voided!

User avatar
w4kh
Posts: 98
Joined: 2006-09-09 19:10
Location: Alabama, USA

Re: DNS Suddenly STOPPED! Stretch 9.5 updated regularly

#18 Post by w4kh »

Head_on_a_Stick wrote:
w4kh wrote:

Code: Select all

default via 192.168.1.254 dev eth0 onlink
Is that correct? Is your gateway at that address?
Yes. That is the correct Gateway address
Head_on_a_Stick wrote:Does the ethernet card work if you load up a Debian "live" image?
I am able to ping around the LAN, and connect through the eth0 ethernet card in both directions: that is, I can connect via FileZilla to another machine on the network, and I am connected to the NAS, to iSCSI drives, and, as well, other systems on the LAN can connect and move files to and from the Linux machine.
DNS isn't working
4.19.0-9-amd64 #1 SMP Debian 4.19.118-2+deb10u1
CPU: AMD FX(tm)-8350 Eight-Core Processor
RAM: 32GB (8x8GB) DDR3 DRAM
Video: GeForce 8400 GS to VIZIO E320VA Monitor

User avatar
Head_on_a_Stick
Posts: 14114
Joined: 2014-06-01 17:46
Location: London, England
Has thanked: 81 times
Been thanked: 132 times

Re: DNS Suddenly STOPPED! Stretch 9.5 updated regularly

#19 Post by Head_on_a_Stick »

Have you tried setting the gateway as the nameserver (via /etc/resolv.conf)?

What exactly is the nature of your set up: are you using an ISP-supplied router as the gateway device?

EDIT: please also post the output requested by @p.H
deadbang

User avatar
w4kh
Posts: 98
Joined: 2006-09-09 19:10
Location: Alabama, USA

[SOLVED]: DNS Suddenly STOPPED! Stretch 9.5 updated regularl

#20 Post by w4kh »

Head_on_a_Stick wrote:Have you tried setting the gateway as the nameserver (via /etc/resolv.conf)?
Network Manager seems to do that as a matter of course (with other nameserver lines after)
Head_on_a_Stick wrote:What exactly is the nature of your set up: are you using an ISP-supplied router as the gateway device?
No, I have an ARRIS Cable Modem and Linksys WRT3200ACM router instead...
The router was selected for its feature set, AND that it could be updated with ExpressVPN.

Therein was the problem. Apparently, early in the morning the system went sideways, ExpressVPN dumped its version into iptables.
Once I discovered that, I copied the iptables-save data into a separate file for potential reloading if that were NOT the issue. Then, I retrieved a backup of the previous iptable, reviewed the settings, and loaded it into the active iptable. That solved the problem.

Thank you very much, especially to Head on a Stick, p.H ... fixing iptables, which I had *NOT* changed myself, was the key. I really appreciate the help offered, and especially the new things I learned.

One last thing:
Head_on_a_Stick wrote:EDIT: please also post the output requested by @p.H
And here it is, with the good iptables...

Code: Select all

traceroute -n 8.8.8.8
traceroute to 8.8.8.8 (8.8.8.8), 30 hops max, 60 byte packets
 1  192.168.1.254  0.288 ms  0.262 ms  0.232 ms
 2  10.167.0.1  55.985 ms  55.998 ms  55.985 ms
 3  147.75.192.244  58.972 ms  58.994 ms  58.955 ms
 4  * * 147.75.98.20  73.530 ms
 5  198.16.4.86  59.829 ms 198.16.4.90  59.841 ms 198.16.4.88  57.163 ms
 6  198.32.160.130  58.749 ms  58.663 ms 206.130.10.41  59.045 ms
 7  108.170.248.65  58.621 ms  62.028 ms 108.170.248.1  63.342 ms
 8  108.170.235.179  61.988 ms 209.85.243.189  60.807 ms 108.170.238.199  59.718 ms
 9  8.8.8.8  59.592 ms  58.999 ms  59.027 ms
4.19.0-9-amd64 #1 SMP Debian 4.19.118-2+deb10u1
CPU: AMD FX(tm)-8350 Eight-Core Processor
RAM: 32GB (8x8GB) DDR3 DRAM
Video: GeForce 8400 GS to VIZIO E320VA Monitor

Post Reply