[Solved] Unable to ssh after upgrade from bullseye to bookworm

[joesysadmin]

I am trying to update an AWS EC2 from Debian v11.9 to bookworm.
Every time I've tried I find I am unable to ssh in after the upgrade.

For some brackground: this instance started out as Debian 6.0 (squeeze) and has been updated regularly over its lifetime.
While I can't be 100% sure, I am fairly confident that on every upgrade package maintainer / distribution configs would be installed and then changes applied as necessary.
This is the first test instance to sort out the update process after which it will be applied to several dozen instances.

I can successfully ssh into fresh install of bookworm with no issues.

Without debug logging on, all I get in the log on the server is:

Code: Select all

sshd[4774]: error: sys_get_rdomain: cannot determine VRF for fd=4 : Protocol not available

Logs from both the client and server (and fresh server) below.

I have access to the server via Session Manager so I can access it for the moment.

Client:

Code: Select all

user@ip-192-168-30-10 ~ % ssh -vvvv admin@bookworm.example.com
OpenSSH_9.0p1, LibreSSL 3.3.6
debug1: Reading configuration data /Users/user/.ssh/config
debug3: /Users/user/.ssh/config line 1: Including file /Users/user/.ssh/config.aws-ssm/int depth 0
debug1: Reading configuration data /Users/user/.ssh/config.aws-ssm/int
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 21: include /etc/ssh/ssh_config.d/* matched no files
debug1: /etc/ssh/ssh_config line 54: Applying options for *
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts' -> '/Users/user/.ssh/known_hosts'
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts2' -> '/Users/user/.ssh/known_hosts2'
debug1: Authenticator provider $SSH_SK_PROVIDER did not resolve; disabling
debug1: Connecting to bookworm.example.com port 22.
debug1: Connection established.
debug1: identity file /Users/user/.ssh/id_rsa type 0
debug1: identity file /Users/user/.ssh/id_rsa-cert type -1
debug1: identity file /Users/user/.ssh/id_ecdsa type -1
debug1: identity file /Users/user/.ssh/id_ecdsa-cert type -1
debug1: identity file /Users/user/.ssh/id_ecdsa_sk type -1
debug1: identity file /Users/user/.ssh/id_ecdsa_sk-cert type -1
debug1: identity file /Users/user/.ssh/id_ed25519 type 3
debug1: identity file /Users/user/.ssh/id_ed25519-cert type -1
debug1: identity file /Users/user/.ssh/id_ed25519_sk type -1
debug1: identity file /Users/user/.ssh/id_ed25519_sk-cert type -1
debug1: identity file /Users/user/.ssh/id_xmss type -1
debug1: identity file /Users/user/.ssh/id_xmss-cert type -1
debug1: identity file /Users/user/.ssh/id_dsa type 1
debug1: identity file /Users/user/.ssh/id_dsa-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_9.0
debug1: Remote protocol version 2.0, remote software version OpenSSH_9.2p1 Debian-2+deb12u2
debug1: compat_banner: match: OpenSSH_9.2p1 Debian-2+deb12u2 pat OpenSSH* compat 0x04000000
debug3: fd 6 is O_NONBLOCK
debug1: Authenticating to bookworm.example.com:22 as 'admin'
debug3: record_hostkey: found key type ED25519 in file /Users/user/.ssh/known_hosts:697
debug3: load_hostkeys_file: loaded 1 keys from bookworm.example.com
debug1: load_hostkeys: fopen /Users/user/.ssh/known_hosts2: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
debug3: order_hostkeyalgs: have matching best-preference key type ssh-ed25519-cert-v01@openssh.com, using HostkeyAlgorithms verbatim
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent

Server (unable to ssh):

Code: Select all

sshd[4056]: debug1: Forked child 4774.
sshd[4774]: debug1: Set /proc/self/oom_score_adj to 0
sshd[4774]: debug1: rexec start in 6 out 6 newsock 6 pipe 8 sock 9
sshd[4774]: debug1: inetd sockets after dupping: 4, 4
sshd[4774]: error: sys_get_rdomain: cannot determine VRF for fd=4 : Protocol not available
sshd[4774]: Connection from ___.___.___.___ port 61663 on 172.31.1.169 port 22
sshd[4774]: debug1: Local version string SSH-2.0-OpenSSH_9.2p1 Debian-2+deb12u2
sshd[4774]: debug1: Remote protocol version 2.0, remote software version OpenSSH_9.0
sshd[4774]: debug1: compat_banner: match: OpenSSH_9.0 pat OpenSSH* compat 0x04000000
sshd[4774]: debug1: permanently_set_uid: 101/65534 [preauth]
sshd[4774]: debug1: ssh_sandbox_child: prctl(PR_SET_NO_NEW_PRIVS): Invalid argument [preauth]
sshd[4774]: debug1: ssh_sandbox_child: prctl(PR_SET_SECCOMP): Invalid argument [preauth]
sshd[4774]: debug1: list_hostkey_types: rsa-sha2-512,rsa-sha2-256,ecdsa-sha2-nistp256,ssh-ed25519 [preauth]
sshd[4774]: debug1: do_cleanup
sshd[4774]: debug1: Killing privsep child 4775
sshd[4774]: debug1: audit_event: unhandled event 12

Server (able to ssh):

Code: Select all

sshd[9809]: debug1: Forked child 9863.
sshd[9809]: debug3: send_rexec_state: entering fd = 8 config len 3247
sshd[9809]: debug3: ssh_msg_send: type 0
sshd[9809]: debug3: send_rexec_state: done
sshd[9863]: debug3: oom_adjust_restore
sshd[9863]: debug1: Set /proc/self/oom_score_adj to 0
sshd[9863]: debug1: rexec start in 5 out 5 newsock 5 pipe 7 sock 8
sshd[9863]: debug1: inetd sockets after dupping: 4, 4
sshd[9863]: debug3: process_channel_timeouts: setting 0 timeouts
sshd[9863]: debug3: channel_clear_timeouts: clearing
sshd[9863]: Connection from ___.___.___.___ port 63268 on 172.31.1.244 port 22 rdomain ""
sshd[9863]: debug1: Local version string SSH-2.0-OpenSSH_9.2p1 Debian-2+deb12u2
sshd[9863]: debug1: Remote protocol version 2.0, remote software version OpenSSH_9.0
sshd[9863]: debug1: compat_banner: match: OpenSSH_9.0 pat OpenSSH* compat 0x04000000
sshd[9863]: debug2: fd 4 setting O_NONBLOCK
sshd[9863]: debug3: ssh_sandbox_init: preparing seccomp filter sandbox
sshd[9863]: debug2: Network child is on pid 9864
sshd[9863]: debug3: preauth child monitor started
sshd[9863]: debug3: privsep user:group 103:65534 [preauth]
sshd[9863]: debug1: permanently_set_uid: 103/65534 [preauth]
sshd[9863]: debug3: ssh_sandbox_child: setting PR_SET_NO_NEW_PRIVS [preauth]
sshd[9863]: debug3: ssh_sandbox_child: attaching seccomp filter program [preauth]
sshd[9863]: debug3: append_hostkey_type: ssh-rsa key not permitted by HostkeyAlgorithms [preauth]
sshd[9863]: debug1: list_hostkey_types: rsa-sha2-512,rsa-sha2-256,ecdsa-sha2-nistp256,ssh-ed25519 [preauth]
sshd[9863]: debug3: send packet: type 20 [preauth]
sshd[9863]: debug1: SSH2_MSG_KEXINIT sent [preauth]
sshd[9863]: debug3: receive packet: type 20 [preauth]
sshd[9863]: debug1: SSH2_MSG_KEXINIT received [preauth]
sshd[9863]: debug2: local server KEXINIT proposal [preauth]
sshd[9863]: debug2: KEX algorithms: sntrup761x25519-sha512@openssh.com,curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,kex-strict-s-v00@openssh.com [preauth]
sshd[9863]: debug2: host key algorithms: rsa-sha2-512,rsa-sha2-256,ecdsa-sha2-nistp256,ssh-ed25519 [preauth]
sshd[9863]: debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com [preauth]
sshd[9863]: debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com [preauth]

...

[CwF]

sshd[4774]: error: sys_get_rdomain: cannot determine VRF for fd=4 : Protocol not available

It's vague to me right now, but there was a change in allowed protocols awhile back. I remember something in listchanges alerted me to it, now I can't recall?

[Aki]

Hello,

These message logs from not working sshd (with newer kernel) could be interesting:

Code: Select all

sshd[4774]: debug1: ssh_sandbox_child: prctl(PR_SET_NO_NEW_PRIVS): Invalid argument [preauth]
sshd[4774]: debug1: ssh_sandbox_child: prctl(PR_SET_SECCOMP): Invalid argument [preauth]

Just a guess: your guest cloud kernel (or guest user space configuration) or the container host configuration restrict the prctl kernel function to access PR_SET_NO_NEW_PRIVS and PR_SET_SECCOMP parameters for the sshd sandbox.

What is your current kernel version in the guest ? Is there an apparmor profile for sshd ?

[joesysadmin]

What is your current kernel version in the guest ?

This was it.

While updated kernels were being installed when updating the instances they never get selected so I was still running one that was very out of date.

Thanks for the assist!

[Aki]

I'm glad you sorted it out. :)

Please, mark the discussion as "solved" manually adding the text tag "[Solved]" at the beginning of the subject of the first message (after other tags, if any). It is also an opportunity to the make the subject more understandable to other forum users and readers, if necessary; i.e. :

[Solved] Unable to ssh after upgrade from bullseye to bookworm

Happy Debian !