SSH Connection Refused

Kernels & Hardware, configuring network, installing services

SSH Connection Refused

Postby schmidtbag » 2019-01-01 19:25

I have a fresh new install of Debian Buster. There is no GUI installed (I intend this to run headless), and aside from some python libraries, closed-source nvidia drivers (for GPGPU purposes), openssh, and avahi, nothing else is installed on this system.

I'm trying to log in as root via SSH, and it tells me the connection is refused on port 22.
BUT....
If I log in as root locally, I can SSH into this computer just fine. I don't have to do anything, I just have to log in and suddenly it works.

In /etc/ssh/sshd_config, the only line I changed was "PermitRootLogin yes". I also tried allowing all hosts in /etc/hosts.allow
I haven't modified any other config files.
I tried running "ssh-keygen -A" which didn't help either.
Checking "systemctl status ssh" doesn't give any errors.

I'm out of ideas and this is driving me nuts. Any suggestions?
schmidtbag
 
Posts: 246
Joined: 2010-04-14 20:51
Location: MA

Re: SSH Connection Refused

Postby reinob » 2019-01-01 20:09

are you using any kind of (non-block-device) encryption?
if so, the ssh daemon may not be able to access your ~/.ssh/authorized_keys until you actually log in..
reinob
 
Posts: 699
Joined: 2014-06-30 11:42

Re: SSH Connection Refused

Postby reinob » 2019-01-01 20:16

found this: https://bugs.launchpad.net/ubuntu/+sour ... bug/319909

A possible workaround is to configure sshd to read authorized_keys from some unencrypted location, like
Code: Select all
AuthorizedKeysFile   /etc/authorized_keys


I'm not sure if that implies that keys placed there will be used independently of the user logging in, so they might be considered OK to log in as *any* user in the system. If you can, please test it and report back.

Cheers.
reinob
 
Posts: 699
Joined: 2014-06-30 11:42

Re: SSH Connection Refused

Postby schmidtbag » 2019-01-01 20:16

reinob wrote:are you using any kind of (non-block-device) encryption?
if so, the ssh daemon may not be able to access your ~/.ssh/authorized_keys until you actually log in..

I'm using whatever the Debian default is.
I also don't have a ~/.ssh/authorized_keys file, though, lacking that doesn't seem to prevent me from SSH'ing in while I'm also logged in locally.
schmidtbag
 
Posts: 246
Joined: 2010-04-14 20:51
Location: MA

Re: SSH Connection Refused

Postby schmidtbag » 2019-01-01 20:21

reinob wrote:found this: https://bugs.launchpad.net/ubuntu/+sour ... bug/319909

A possible workaround is to configure sshd to read authorized_keys from some unencrypted location, like
Code: Select all
AuthorizedKeysFile   /etc/authorized_keys


I'm not sure if that implies that keys placed there will be used independently of the user logging in, so they might be considered OK to log in as *any* user in the system. If you can, please test it and report back.

Cheers.


This doesn't seem to do anything. /etc/authorized_keys doesn't exist either, if that makes a difference.
schmidtbag
 
Posts: 246
Joined: 2010-04-14 20:51
Location: MA

Re: SSH Connection Refused

Postby reinob » 2019-01-01 20:23

Can you post the output of (as root) "systemctl status sshd"

It should always start when booting (and not when logging in). I assume you have not configured some kind of firewall or iptables script running only when root logs in (like opening port 22 in .bashrc or some crazy stuff..)
reinob
 
Posts: 699
Joined: 2014-06-30 11:42

Re: SSH Connection Refused

Postby reinob » 2019-01-01 20:25

schmidtbag wrote:
reinob wrote:found this: https://bugs.launchpad.net/ubuntu/+sour ... bug/319909

A possible workaround is to configure sshd to read authorized_keys from some unencrypted location, like
Code: Select all
AuthorizedKeysFile   /etc/authorized_keys


I'm not sure if that implies that keys placed there will be used independently of the user logging in, so they might be considered OK to log in as *any* user in the system. If you can, please test it and report back.

Cheers.


This doesn't seem to do anything. /etc/authorized_keys doesn't exist either, if that makes a difference.


You would need to create that file yourself, and place your authorized keys there. But since you are apparently not using keys, then this is all irrelevant. Password authentication should work (if enabled), and should allow the system (PAM) to decrypt your $HOME folder (if encrypted).
reinob
 
Posts: 699
Joined: 2014-06-30 11:42

Re: SSH Connection Refused

Postby schmidtbag » 2019-01-01 20:32

reinob wrote:Can you post the output of (as root) "systemctl status sshd"

It should always start when booting (and not when logging in). I assume you have not configured some kind of firewall or iptables script running only when root logs in (like opening port 22 in .bashrc or some crazy stuff..)

What I mentioned in my first post is literally all I changed from a fresh Buster install. Nothing else. There's no firewall, there's no iptables script, there's no .bashrc modifications, etc etc. This is pretty much as stock as it gets.

Anyway, I don't see how it's much help, but here's the sshd status:
Code: Select all
● ssh.service - OpenBSD Secure Shell server
   Loaded: loaded (/lib/systemd/system/ssh.service; enabled; vendor preset: enabled)
   Active: active (running) since Tue 2019-01-01 16:09:23 EST; 8min ago
     Docs: man:sshd(8)
           man:sshd_config(5)
  Process: 319 ExecStartPre=/usr/sbin/sshd -t (code=exited, status=0/SUCCESS)
 Main PID: 385 (sshd)
    Tasks: 1 (limit: 2358)
   Memory: 5.8M
   CGroup: /system.slice/ssh.service
           └─385 /usr/sbin/sshd -D

Keep in mind, the problem goes away when I log in, so I don't think the status is ever going to tell me exactly what's wrong.


reinob wrote:You would need to create that file yourself, and place your authorized keys there. But since you are apparently not using keys, then this is all irrelevant. Password authentication should work (if enabled), and should allow the system (PAM) to decrypt your $HOME folder (if encrypted).

I didn't encrypt the $HOME folder (not sure if it is by default).

EDIT:
It's worth pointing out that I can SSH to any other system on my network no problem. This 1 computer is the only one with issues. Of those devices, one is Debian Jessie, another is Buster, and a third runs Arch Linux.
schmidtbag
 
Posts: 246
Joined: 2010-04-14 20:51
Location: MA

Re: SSH Connection Refused

Postby xepan » 2019-01-02 04:11

can you ssh as user?

does:
ssh -v remote
offer any further insight?
If in doubt add more v's ...

did you restart ssh after editing ssh_config?

I don't really understand what you mean with "works after logging in". Logging in where, and what works then? It is much easier to get the head in it when seeing the exact commands and the exact output they give.
xepan
 
Posts: 89
Joined: 2018-11-28 06:38

Re: SSH Connection Refused

Postby reinob » 2019-01-02 13:14

schmidtbag wrote:It's worth pointing out that I can SSH to any other system on my network no problem. This 1 computer is the only one with issues. Of those devices, one is Debian Jessie, another is Buster, and a third runs Arch Linux.


According to https://bugs.debian.org/cgi-bin/bugrepo ... bug=912087 the openssh/openssl/kernel version in buster (which, I remind you, is not stable yet..) takes long to start if not enough entropy is available. Your headless server, with probably not much traffic, is a good candidate for low entropy..

Logging in at the console may provide enough entropy to allow the kernel to initialize the RNG (which openssl needs, which in turn openssh needs). It would be interesting to test if you can log in after a while without having physically logged on at the console.

Reading the above thread it seems that it's all a mess, either one or more of at least { kernel, openssl, systemd } seems to be the culprit. A workaround seems to be "apt install haveged". As we speak, I'm installing that everywhere to avoid (potential) problems when I update to buster.

(Note that I also have a debian sid laptop which does not show this problem, but (1) it's a highly interactive system and (2) I don't usually ssh into my laptop shortly after booting..)

Please report back if installing haveged solved the issue.
Good luck.
reinob
 
Posts: 699
Joined: 2014-06-30 11:42

Re: SSH Connection Refused

Postby schmidtbag » 2019-01-02 13:30

reinob wrote:
schmidtbag wrote:It's worth pointing out that I can SSH to any other system on my network no problem. This 1 computer is the only one with issues. Of those devices, one is Debian Jessie, another is Buster, and a third runs Arch Linux.


According to https://bugs.debian.org/cgi-bin/bugrepo ... bug=912087 the openssh/openssl/kernel version in buster (which, I remind you, is not stable yet..) takes long to start if not enough entropy is available. Your headless server, with probably not much traffic, is a good candidate for low entropy..

Logging in at the console may provide enough entropy to allow the kernel to initialize the RNG (which openssl needs, which in turn openssh needs). It would be interesting to test if you can log in after a while without having physically logged on at the console.

Reading the above thread it seems that it's all a mess, either one or more of at least { kernel, openssl, systemd } seems to be the culprit. A workaround seems to be "apt install haveged". As we speak, I'm installing that everywhere to avoid (potential) problems when I update to buster.

(Note that I also have a debian sid laptop which does not show this problem, but (1) it's a highly interactive system and (2) I don't usually ssh into my laptop shortly after booting..)

Please report back if installing haveged solved the issue.
Good luck.

Very interesting. I'm not sure I'd have been able to find that on my own. The odd thing is I do have another Buster system that I can SSH into just fine right after it boots up, but, that has also been in use for much longer, so maybe it grandfathered in something that works fine.
If I have no luck with haveged, I'll likely just switch to Ubuntu 18.10. I've wasted way too much time on this problem as-is, and I can't revert back to the stable version of Debian because the packages I need are too outdated.

Thanks for the help.
schmidtbag
 
Posts: 246
Joined: 2010-04-14 20:51
Location: MA

Re: SSH Connection Refused

Postby reinob » 2019-01-02 13:52

schmidtbag wrote:Very interesting. I'm not sure I'd have been able to find that on my own. The odd thing is I do have another Buster system that I can SSH into just fine right after it boots up, but, that has also been in use for much longer, so maybe it grandfathered in something that works fine.


It would be nice if you could post the output of "dmesg | grep crng" on both systems. My laptop shows
Code: Select all
[    0.176571] random: get_random_bytes called from start_kernel+0x93/0x537 with crng_init=0
[    6.023801] random: crng init done


which I take to mean that it takes 6 seconds until enough entropy is collected.

A debian (stretch) virtual server I use shows it took 132 seconds until "crng init done", but the server was forcibly
rebooted and took about 40 seconds to fsck..

A raspberry pi (model 1) running raspbian (based off stretch) shows 28 seconds.
reinob
 
Posts: 699
Joined: 2014-06-30 11:42


Return to System configuration

Who is online

Users browsing this forum: fender0107401 and 10 guests

fashionable