SSH Connection Refused

[schmidtbag]

I have a fresh new install of Debian Buster. There is no GUI installed (I intend this to run headless), and aside from some python libraries, closed-source nvidia drivers (for GPGPU purposes), openssh, and avahi, nothing else is installed on this system.

I'm trying to log in as root via SSH, and it tells me the connection is refused on port 22.
BUT....
If I log in as root locally, I can SSH into this computer just fine. I don't have to do anything, I just have to log in and suddenly it works.

In /etc/ssh/sshd_config, the only line I changed was "PermitRootLogin yes". I also tried allowing all hosts in /etc/hosts.allow
I haven't modified any other config files.
I tried running "ssh-keygen -A" which didn't help either.
Checking "systemctl status ssh" doesn't give any errors.

I'm out of ideas and this is driving me nuts. Any suggestions?

[reinob]

are you using any kind of (non-block-device) encryption?
if so, the ssh daemon may not be able to access your ~/.ssh/authorized_keys until you actually log in..

[reinob]

found this: https://bugs.launchpad.net/ubuntu/+sour ... bug/319909

A possible workaround is to configure sshd to read authorized_keys from some unencrypted location, like

Code: Select all

AuthorizedKeysFile	/etc/authorized_keys

I'm not sure if that implies that keys placed there will be used independently of the user logging in, so they might be considered OK to log in as *any* user in the system. If you can, please test it and report back.

Cheers.

[schmidtbag]

are you using any kind of (non-block-device) encryption?
if so, the ssh daemon may not be able to access your ~/.ssh/authorized_keys until you actually log in..

I'm using whatever the Debian default is.
I also don't have a ~/.ssh/authorized_keys file, though, lacking that doesn't seem to prevent me from SSH'ing in while I'm also logged in locally.

[schmidtbag]

found this: https://bugs.launchpad.net/ubuntu/+sour ... bug/319909

A possible workaround is to configure sshd to read authorized_keys from some unencrypted location, like

Code: Select all

AuthorizedKeysFile	/etc/authorized_keys

I'm not sure if that implies that keys placed there will be used independently of the user logging in, so they might be considered OK to log in as *any* user in the system. If you can, please test it and report back.

Cheers.

This doesn't seem to do anything. /etc/authorized_keys doesn't exist either, if that makes a difference.

[reinob]

Can you post the output of (as root) "systemctl status sshd"

It should always start when booting (and not when logging in). I assume you have not configured some kind of firewall or iptables script running only when root logs in (like opening port 22 in .bashrc or some crazy stuff..)

[reinob]

found this: https://bugs.launchpad.net/ubuntu/+sour ... bug/319909

A possible workaround is to configure sshd to read authorized_keys from some unencrypted location, like

Code: Select all

AuthorizedKeysFile	/etc/authorized_keys

I'm not sure if that implies that keys placed there will be used independently of the user logging in, so they might be considered OK to log in as *any* user in the system. If you can, please test it and report back.

Cheers.

This doesn't seem to do anything. /etc/authorized_keys doesn't exist either, if that makes a difference.

You would need to create that file yourself, and place your authorized keys there. But since you are apparently not using keys, then this is all irrelevant. Password authentication should work (if enabled), and should allow the system (PAM) to decrypt your $HOME folder (if encrypted).

[schmidtbag]

Can you post the output of (as root) "systemctl status sshd"

It should always start when booting (and not when logging in). I assume you have not configured some kind of firewall or iptables script running only when root logs in (like opening port 22 in .bashrc or some crazy stuff..)

What I mentioned in my first post is literally all I changed from a fresh Buster install. Nothing else. There's no firewall, there's no iptables script, there's no .bashrc modifications, etc etc. This is pretty much as stock as it gets.

Anyway, I don't see how it's much help, but here's the sshd status:

Code: Select all

● ssh.service - OpenBSD Secure Shell server
   Loaded: loaded (/lib/systemd/system/ssh.service; enabled; vendor preset: enabled)
   Active: active (running) since Tue 2019-01-01 16:09:23 EST; 8min ago
     Docs: man:sshd(8)
           man:sshd_config(5)
  Process: 319 ExecStartPre=/usr/sbin/sshd -t (code=exited, status=0/SUCCESS)
 Main PID: 385 (sshd)
    Tasks: 1 (limit: 2358)
   Memory: 5.8M
   CGroup: /system.slice/ssh.service
           └─385 /usr/sbin/sshd -D

Keep in mind, the problem goes away when I log in, so I don't think the status is ever going to tell me exactly what's wrong.

You would need to create that file yourself, and place your authorized keys there. But since you are apparently not using keys, then this is all irrelevant. Password authentication should work (if enabled), and should allow the system (PAM) to decrypt your $HOME folder (if encrypted).

I didn't encrypt the $HOME folder (not sure if it is by default).

EDIT:
It's worth pointing out that I can SSH to any other system on my network no problem. This 1 computer is the only one with issues. Of those devices, one is Debian Jessie, another is Buster, and a third runs Arch Linux.

[xepan]

can you ssh as user?

does:
ssh -v remote
offer any further insight?
If in doubt add more v's ...

did you restart ssh after editing ssh_config?

I don't really understand what you mean with "works after logging in". Logging in where, and what works then? It is much easier to get the head in it when seeing the exact commands and the exact output they give.

[reinob]

It's worth pointing out that I can SSH to any other system on my network no problem. This 1 computer is the only one with issues. Of those devices, one is Debian Jessie, another is Buster, and a third runs Arch Linux.

According to https://bugs.debian.org/cgi-bin/bugrepo ... bug=912087 the openssh/openssl/kernel version in buster (which, I remind you, is not stable yet..) takes long to start if not enough entropy is available. Your headless server, with probably not much traffic, is a good candidate for low entropy..

Logging in at the console may provide enough entropy to allow the kernel to initialize the RNG (which openssl needs, which in turn openssh needs). It would be interesting to test if you can log in after a while without having physically logged on at the console.

Reading the above thread it seems that it's all a mess, either one or more of at least { kernel, openssl, systemd } seems to be the culprit. A workaround seems to be "apt install haveged". As we speak, I'm installing that everywhere to avoid (potential) problems when I update to buster.

(Note that I also have a debian sid laptop which does not show this problem, but (1) it's a highly interactive system and (2) I don't usually ssh into my laptop shortly after booting..)

Please report back if installing haveged solved the issue.
Good luck.

[schmidtbag]

It's worth pointing out that I can SSH to any other system on my network no problem. This 1 computer is the only one with issues. Of those devices, one is Debian Jessie, another is Buster, and a third runs Arch Linux.

According to https://bugs.debian.org/cgi-bin/bugrepo ... bug=912087 the openssh/openssl/kernel version in buster (which, I remind you, is not stable yet..) takes long to start if not enough entropy is available. Your headless server, with probably not much traffic, is a good candidate for low entropy..

Logging in at the console may provide enough entropy to allow the kernel to initialize the RNG (which openssl needs, which in turn openssh needs). It would be interesting to test if you can log in after a while without having physically logged on at the console.

Reading the above thread it seems that it's all a mess, either one or more of at least { kernel, openssl, systemd } seems to be the culprit. A workaround seems to be "apt install haveged". As we speak, I'm installing that everywhere to avoid (potential) problems when I update to buster.

(Note that I also have a debian sid laptop which does not show this problem, but (1) it's a highly interactive system and (2) I don't usually ssh into my laptop shortly after booting..)

Please report back if installing haveged solved the issue.
Good luck.

Very interesting. I'm not sure I'd have been able to find that on my own. The odd thing is I do have another Buster system that I can SSH into just fine right after it boots up, but, that has also been in use for much longer, so maybe it grandfathered in something that works fine.
If I have no luck with haveged, I'll likely just switch to Ubuntu 18.10. I've wasted way too much time on this problem as-is, and I can't revert back to the stable version of Debian because the packages I need are too outdated.

Thanks for the help.

[reinob]

Very interesting. I'm not sure I'd have been able to find that on my own. The odd thing is I do have another Buster system that I can SSH into just fine right after it boots up, but, that has also been in use for much longer, so maybe it grandfathered in something that works fine.

It would be nice if you could post the output of "dmesg | grep crng" on both systems. My laptop shows

Code: Select all

[    0.176571] random: get_random_bytes called from start_kernel+0x93/0x537 with crng_init=0
[    6.023801] random: crng init done

which I take to mean that it takes 6 seconds until enough entropy is collected.

A debian (stretch) virtual server I use shows it took 132 seconds until "crng init done", but the server was forcibly
rebooted and took about 40 seconds to fsck..

A raspberry pi (model 1) running raspbian (based off stretch) shows 28 seconds.