[SOLVED] DNS Suddenly STOPPED! Stretch 9.5 updated regularly

Kernels & Hardware, configuring network, installing services

[SOLVED] DNS Suddenly STOPPED! Stretch 9.5 updated regularly

Postby w4kh » 2018-08-16 21:01

I was just "surfing" this morning and suddenly the music stream stopped, and every tab on the browser showed:
No internet
Try:
Checking the network cables, modem, and router
Reconnecting to Wi-Fi
DNS_PROBE_FINISHED_NO_INTERNET

Initially, I thought my internet connection had gone down - much construction on the main road past the subdivision, and utilities being moved to new poles - but Windows was still going while Stretch gagged and halted.

I had not finished my usual round before running the daily updater - given the DNS issue I did NOT run this, which is usually run daily:
#! /bin/bash
# Update the repositories and then process upgrades
/usr/bin/apt-get update
sleep 5
UPLOCK=/var/lib/apt/lists/lock
if [ -f $UPLOCK ]; then
rm $UPLOCK
fi
echo "Now, to upgrade"
/usr/bin/apt-get -f upgrade
# Run rkhunter
echo "Running Rootkit Hunter"
/usr/bin/rkhunter --propupd
#And, now to update the tripwire database
echo "updating the tripwire database"
/usr/local/bin/uptripw

But instead ran just 'apt-get update' which returned:
Err:1 http://security.debian.org stretch/updates InRelease
Temporary failure resolving 'security.debian.org'

I knew something bad had happened... but what?

Hardware all seems solid - nothing new added in a long time
32GB RAM, 8-core AMD CPU...
System kept up-to-date daily.
No joy on Google or searching the Forums (unless I used a poor choice of search terms)
I am at a loss (and resigned to having to address this forum with <gag> Windows10 on my wife's desktop!)
Last edited by w4kh on 2018-08-25 20:09, edited 1 time in total.
User avatar
w4kh
 
Posts: 62
Joined: 2006-09-09 19:10
Location: Tennessee, USA

Re: DNS Suddenly STOPPED! Stretch 9.5 updated regularly

Postby Segfault » 2018-08-17 01:51

Basics first. Can you access web sites by IP address? http://217.196.43.138/
Segfault
 
Posts: 700
Joined: 2005-09-24 12:24

Re: DNS Suddenly STOPPED! Stretch 9.5 updated regularly

Postby p.H » 2018-08-17 08:27

w4kh wrote:DNS_PROBE_FINISHED_NO_INTERNET

What does this come from ?

Segfault wrote:Basics first. Can you access web sites by IP address?

"Basics" ? A web browser is an overcomplicated piece of software unsuitable for network troubleshooting.
To me, basics are IP configuration lookup with ip addr and ip route, IP connectivity check with ping and traceroute, DNS lookup with host, dig or nslookup.
p.H
 
Posts: 521
Joined: 2017-09-17 07:12

Re: DNS Suddenly STOPPED! Stretch 9.5 updated regularly

Postby Segfault » 2018-08-17 10:48

Segfault wrote:Basics first. Can you access web sites by IP address?

"Basics" ? A web browser is an overcomplicated piece of software unsuitable for network troubleshooting.
To me, basics are IP configuration lookup with ip addr and ip route, IP connectivity check with ping and traceroute, DNS lookup with host, dig or nslookup.

I never said to use a web browser. I asked if web sites are accessible by IP address. Leaving it for OP to decide how. Try to think more clearly.
Segfault
 
Posts: 700
Joined: 2005-09-24 12:24

Re: DNS Suddenly STOPPED! Stretch 9.5 updated regularly

Postby w4kh » 2018-08-17 12:33

Segfault wrote:
Basics first. Can you access web sites by IP address? http://217.196.43.138/

Yes... I can maneuver by IPv4 address - with some limits...

a ping by IPv4 address results in:
ping: sendmsg: Operation not permitted
ping: sendmsg: Operation not permitted
ping: sendmsg: Operation not permitted
ping: sendmsg: Operation not permitted
ping: sendmsg: Operation not permitted
and,
a ping by URL gets:
ping: http://www.siteyouwant.com: Temporary failure in name resolution

Oddest thing is, I can get to some Debian URLs with no problem:
i.e., https://bugs.debian.org/release-critical/
However, I suspect that is from cache, since that URL gets hit daily

p.H wrote:
"DNS_PROBE_FINISHED_NO_INTERNET"
What does this come from ?

Google Chrome Browser... everything just suddenly stopped and that came up as the last "sentence" of the error message.
User avatar
w4kh
 
Posts: 62
Joined: 2006-09-09 19:10
Location: Tennessee, USA

Re: DNS Suddenly STOPPED! Stretch 9.5 updated regularly

Postby p.H » 2018-08-17 17:47

Segfault wrote:I never said to use a web browser. I asked if web sites are accessible by IP address. Leaving it for OP to decide how. Try to think more clearly.

How then do you suggest to check if websites are available ? Try to write more clearly.
Non-browser HTTP clients which understand URLs such as wget or curl are nearly as complicated as web browsers.

w4kh wrote: that came up as the last "sentence" of the error message.

Why didn't you provide provide the full error message ?

w4kh wrote:ping: sendmsg: Operation not permitted

This error usually means that outgoing packets are blocked by firewall rules. Check iptables rules with iptables-save.
If it is intended that outgoing ICMP echo request (aka "ping") are blocked for security reasons, then you have to use another tool and protocol to check connectivity. Standard traceroute sends UDP packets, tcptraceroute sends TCP packets as you can guess.

w4kh wrote:a ping by URL gets:
ping: http://www.siteyouwant.com: Temporary failure in name resolution

This command cannot work : ping does not understand URLs and expects a host/domain name or an IP address.

If the dnsutils package is installed, you can test the DNS resolution with something like
Code: Select all
dig some.domain.name
host some.domain.name

Also, check the contents of /etc/resolv.conf.
p.H
 
Posts: 521
Joined: 2017-09-17 07:12

Re: DNS Suddenly STOPPED! Stretch 9.5 updated regularly

Postby w4kh » 2018-08-17 21:27

p.H wrote:
w4kh wrote: that came up as the last "sentence" of the error message.

Why didn't you provide provide the full error message ?


Actually, I did... the entire error message is:
Code: Select all
No internet
Try:
Checking the network cables, modem, and router
Reconnecting to Wi-Fi
DNS_PROBE_FINISHED_NO_INTERNET


And when I tried:
Code: Select all
dig www.charter.com
host www.charter.com

This happened:

Code: Select all
; <<>> DiG 9.10.3-P4-Debian <<>> www.charter.com
;; global options: +cmd
;; connection timed out; no servers could be reached
root@bigmutt:~# host www.charter.com
;; connection timed out; no servers could be reached


w4kh wrote:ping: sendmsg: Operation not permitted

p.H wrote:This error usually means that outgoing packets are blocked by firewall rules. Check iptables rules with iptables-save.
If it is intended that outgoing ICMP echo request (aka "ping") are blocked for security reasons, then you have to use another tool and protocol to check connectivity. Standard traceroute sends UDP packets, tcptraceroute sends TCP packets as you can guess.

NOTHING changed (that I am aware of)... Auto-update is not active; I had not yet run my daily apt-get update/apt-get upgrade... the system just suddenly went sideways and DNS/BIND9 ceased working. Since I was working using the Debian machine, I assumed (yeah, I know...) that I had lost internet due to the road construction behind our house - it had happened before. Then I found the Windows machine could surf and Debian was knocked out, and I tried to figure out what was going on.

Also, check the contents of /etc/resolv.conf.


Code: Select all
search mydomain.com
nameserver 8.8.8.8
# NOTE: the libc resolver may not support more than 3 nameservers.
# The nameservers listed below may not be recognized.
nameserver 2602:306:3bef:51b0::1
User avatar
w4kh
 
Posts: 62
Joined: 2006-09-09 19:10
Location: Tennessee, USA

Re: DNS Suddenly STOPPED! Stretch 9.5 updated regularly

Postby debiman » 2018-08-18 07:26

Code: Select all
systemctl status systemd-resolved
sudo ping -c3 8.8.8.8
sudo ping -c3 google.com

oh and what software do you use to connect to the internet? networkmanager?

i don't use tripwire; could it be responsible somehow?
User avatar
debiman
 
Posts: 2814
Joined: 2013-03-12 07:18

Re: DNS Suddenly STOPPED! Stretch 9.5 updated regularly

Postby w4kh » 2018-08-18 15:32

debiman wrote:
Code: Select all
systemctl status systemd-resolved


Code: Select all
systemctl status systemd-resolved
● systemd-resolved.service - Network Name Resolution
   Loaded: loaded (/lib/systemd/system/systemd-resolved.service; disabled; vendor preset: enabled)
  Drop-In: /lib/systemd/system/systemd-resolved.service.d
           └─resolvconf.conf
   Active: inactive (dead)
     Docs: man:systemd-resolved.service(8)
           http://www.freedesktop.org/wiki/Software/systemd/resolved
           http://www.freedesktop.org/wiki/Software/systemd/writing-network-configuration-manag
           http://www.freedesktop.org/wiki/Software/systemd/writing-resolver-clients


It appears as though /lib/systemd/system/systemd-resolved.service is disabled, and the man pages give no hint that my fatigued eyes could find as to how to enable it. Looking a tad further, Both the man page for systemd-resolved and for systemd-resolved.service refer to a file /run/systemd/resolve/resolv.conf, HOWEVER, there is NO "resolve" directory under /run/systemd, and (obviously?) no resolv.conf in the non-existent /run/systemd/resolve directory

debiman wrote:oh and what software do you use to connect to the internet? networkmanager?


NetworkManager

debiman wrote:i don't use tripwire; could it be responsible somehow?


I doubt it... TripWire's primary function is to monitor changes to files, especially "system files", as indications of intrusion

debiman wrote:
Code: Select all
sudo ping -c3 8.8.8.8


Code: Select all
ping -c3 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
ping: sendmsg: Operation not permitted
ping: sendmsg: Operation not permitted
ping: sendmsg: Operation not permitted

--- 8.8.8.8 ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 2027ms


debiman wrote:
Code: Select all
sudo ping -c3 google.com


Code: Select all
ping -c3 google.com
ping: google.com: Temporary failure in name resolution


Me thinks the missing "/run/systemd/resolve/resolv.conf" has a great deal to do with the failure, but should its structure and content be exactly like /etc/resolv.conf??
Keeping track of actions taken, in order than reversal is "possible"
Code: Select all
mkdir /run/systemd/resolve
chown root:root /run/systemd/resolve
chmod 755 /run/systemd/resolve
cp -p /etc/resolv.conf /run/systemd/resolve/
ls -l /run/systemd/resolve/resolv.conf
-rw-r--r-- 1 root root 250 Aug 18 10:33 /run/systemd/resolve/resolv.conf


Still NO joy! :( and, I am at a loss to explain what has happened. Real problem, for me, is how to explain everything working, and suddenly "POOF!" with no change made... I was just visiting the daily sites...
User avatar
w4kh
 
Posts: 62
Joined: 2006-09-09 19:10
Location: Tennessee, USA

Re: DNS Suddenly STOPPED! Stretch 9.5 updated regularly

Postby Head_on_a_Stick » 2018-08-18 16:08

w4kh wrote:Me thinks the missing "/run/systemd/resolve/resolv.conf" has a great deal to do with the failure

No, /run/systemd/resolve/resolv.conf is only created if systemd-resolved.service is running.

You are not using (and do not need to use) systemd-resolved and furthermore the recommended mode of operation is
Code: Select all
# ln -sf /lib/systemd/resolv.conf /etc/resolv.conf

Which should give
Code: Select all
root@hegel:~# grep -v '^#\|^$' /etc/resolv.conf
nameserver 127.0.0.53
root@hegel:~#

Anyway, I have no idea why debiman has brought systemd-resolved up, it doesn't seem to have anything to do with your problem.

Can we please see the full output of
Code: Select all
ip a

Thanks!
ESTRAGON: We always find something, eh, Didi, to give us the impression we exist?
VLADIMIR (impatiently): Yes, yes, we're magicians.
User avatar
Head_on_a_Stick
 
Posts: 7893
Joined: 2014-06-01 17:46
Location: /dev/chair

Re: DNS Suddenly STOPPED! Stretch 9.5 updated regularly

Postby w4kh » 2018-08-18 17:01

Head_on_a_Stick wrote:
w4kh wrote:Me thinks the missing "/run/systemd/resolve/resolv.conf" has a great deal to do with the failure

No, /run/systemd/resolve/resolv.conf is only created if systemd-resolved.service is running.

You are not using (and do not need to use) systemd-resolved and furthermore the recommended mode of operation is
Code: Select all
# ln -sf /lib/systemd/resolv.conf /etc/resolv.conf

Which should give
Code: Select all
root@hegel:~# grep -v '^#\|^$' /etc/resolv.conf
nameserver 127.0.0.53
root@hegel:~#

And It did...
Head_on_a_Stick wrote:Anyway, I have no idea why debiman has brought systemd-resolved up, it doesn't seem to have anything to do with your problem.

Can we please see the full output of
Code: Select all
ip a

Thanks!


Here it is:
Code: Select all
ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether fc:aa:14:58:58:c7 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.42/24 brd 192.168.1.255 scope global eth0
       valid_lft forever preferred_lft forever


In case it is not obvious, eth0 address is static... "things" (and a couple laptops) are the only dhcp clients on the LAN
User avatar
w4kh
 
Posts: 62
Joined: 2006-09-09 19:10
Location: Tennessee, USA

Re: DNS Suddenly STOPPED! Stretch 9.5 updated regularly

Postby Head_on_a_Stick » 2018-08-18 17:05

w4kh wrote:And It did...

That wasn't a suggestion, I was just pointing out how to configure systemd-resolved, I *do not* think you should be trying systemd-resolved, it just adds another thing that might be going wrong.

Remove the symlink and replace /etc/resolv.conf with a single line:
Code: Select all
nameserver 8.8.8.8

Does `ping google.com` work now?

EDIT: we should probably also see
Code: Select all
ip r
ESTRAGON: We always find something, eh, Didi, to give us the impression we exist?
VLADIMIR (impatiently): Yes, yes, we're magicians.
User avatar
Head_on_a_Stick
 
Posts: 7893
Joined: 2014-06-01 17:46
Location: /dev/chair

Re: DNS Suddenly STOPPED! Stretch 9.5 updated regularly

Postby w4kh » 2018-08-18 17:16

Head_on_a_Stick wrote:
w4kh wrote:And It did...

That wasn't a suggestion, I was just pointing out how to configure systemd-resolved, I *do not* think you should be trying systemd-resolved, it just adds another thing that might be going wrong.


Agreed...

Head_on_a_Stick wrote:Remove the symlink and replace /etc/resolv.conf with a single line:
Code: Select all
nameserver 8.8.8.8

Does `ping google.com` work now?

Code: Select all
ping google.com
ping: google.com: Temporary failure in name resolution


Head_on_a_Stick wrote:EDIT: we should probably also see
Code: Select all
ip r

Code: Select all
ip r
default via 192.168.1.254 dev eth0 onlink
192.168.1.0/24 dev eth0 proto kernel scope link src 192.168.1.42
User avatar
w4kh
 
Posts: 62
Joined: 2006-09-09 19:10
Location: Tennessee, USA

Re: DNS Suddenly STOPPED! Stretch 9.5 updated regularly

Postby Head_on_a_Stick » 2018-08-18 17:21

w4kh wrote:
Code: Select all
default via 192.168.1.254 dev eth0 onlink

Is that correct? Is your gateway at that address?

Does the ethernet card work if you load up a Debian "live" image?
ESTRAGON: We always find something, eh, Didi, to give us the impression we exist?
VLADIMIR (impatiently): Yes, yes, we're magicians.
User avatar
Head_on_a_Stick
 
Posts: 7893
Joined: 2014-06-01 17:46
Location: /dev/chair

Re: DNS Suddenly STOPPED! Stretch 9.5 updated regularly

Postby p.H » 2018-08-18 17:57

Also, after checking the output of iptables-save, could you provide the full output of
Code: Select all
traceroute -n 8.8.8.8
p.H
 
Posts: 521
Joined: 2017-09-17 07:12

Next

Return to System configuration

Who is online

Users browsing this forum: No registered users and 8 guests

fashionable