Scheduled Maintenance: We are aware of an issue with Google, AOL, and Yahoo services as email providers which are blocking new registrations. We are trying to fix the issue and we have several internal and external support tickets in process to resolve the issue. Please see: viewtopic.php?t=158230

 

 

 

ssh.service will not start at boot if using ListenAddress

Linux Kernel, Network, and Services configuration.
Post Reply
Message
Author
imthenachoman
Posts: 8
Joined: 2019-02-10 04:53

ssh.service will not start at boot if using ListenAddress

#1 Post by imthenachoman »

I have ListenAddress in my /etc/ssh/sshd_config.

ssh.service will not start at boot but it will start if I run service ssh start after boot.

Here is the output from journalctl -xe | egrep "ssh|network":

Code: Select all

Feb 10 00:17:46 nook systemd[1]: Starting Raise network interfaces...
-- Subject: Unit networking.service has begun start-up
-- Unit networking.service has begun starting up.
Feb 10 00:17:46 nook systemd[1]: Started Raise network interfaces.
-- Subject: Unit networking.service has finished start-up
-- Unit networking.service has finished starting up.
-- Subject: Unit network.target has finished start-up
-- Unit network.target has finished starting up.
-- Subject: Unit network-online.target has finished start-up
-- Unit network-online.target has finished starting up.
-- Subject: Unit ssh.service has begun start-up
-- Unit ssh.service has begun starting up.
Feb 10 00:17:46 nook sshd[563]: error: Bind to port 22 on 192.168.1.100 failed: Cannot assign requested address.
Feb 10 00:17:46 nook sshd[563]: fatal: Cannot bind any address.
Feb 10 00:17:46 nook systemd[1]: ssh.service: Main process exited, code=exited, status=255/n/a
-- Subject: Unit ssh.service has failed
-- Unit ssh.service has failed.
Feb 10 00:17:46 nook systemd[1]: ssh.service: Unit entered failed state.
Feb 10 00:17:46 nook systemd[1]: ssh.service: Failed with result 'exit-code'.
/etc/systemd/system/ssh.service:

Code: Select all

[Unit]
Description=OpenBSD Secure Shell server
After=network.target auditd.service network-online.target
Wants=network-online.target
ConditionPathExists=!/etc/ssh/sshd_not_to_be_run

[Service]
EnvironmentFile=-/etc/default/ssh
ExecStartPre=/usr/sbin/sshd -t
ExecStart=/usr/sbin/sshd -D $SSHD_OPTS
ExecReload=/usr/sbin/sshd -t
ExecReload=/bin/kill -HUP $MAINPID
KillMode=process
Restart=on-failure
RestartPreventExitStatus=255
Type=notify

[Install]
WantedBy=multi-user.target
Alias=sshd.service
/etc/systemd/system/sshd.service:

Code: Select all

[Unit]
Description=OpenBSD Secure Shell server
After=network.target auditd.service network-online.target
Wants=network-online.target
ConditionPathExists=!/etc/ssh/sshd_not_to_be_run

[Service]
EnvironmentFile=-/etc/default/ssh
ExecStartPre=/usr/sbin/sshd -t
ExecStart=/usr/sbin/sshd -D $SSHD_OPTS
ExecReload=/usr/sbin/sshd -t
ExecReload=/bin/kill -HUP $MAINPID
KillMode=process
Restart=on-failure
RestartPreventExitStatus=255
Type=notify

[Install]
WantedBy=multi-user.target
Alias=sshd.service
I've looked online but can't seem to figure out what is wrong. It works if I start it manually after boot but it won't start at boot. :/

It works if I take out ListenAddress from /etc/ssh/sshd_config but I need/want it there.

User avatar
bw123
Posts: 4015
Joined: 2011-05-09 06:02
Has thanked: 1 time
Been thanked: 28 times

Re: ssh.service will not start at boot if using ListenAddres

#2 Post by bw123 »

Maybe the Wants=network-online.target is problematic, are you really using networkmanager to bring up the network?
Have you tried making the daemon wait to start by instead using wants multi-user.target or something like that?

maybe if you post your whole sshd_config someone can spot something, and debian ver and openssh-server ver is probably a good idea.

Why did you have to make all the services and put them in /etc/systemd/system instead of using the pkg service files?
resigned by AI ChatGPT

User avatar
Head_on_a_Stick
Posts: 14114
Joined: 2014-06-01 17:46
Location: London, England
Has thanked: 81 times
Been thanked: 132 times

Re: ssh.service will not start at boot if using ListenAddres

#3 Post by Head_on_a_Stick »

bw123 wrote:Why did you have to make all the services and put them in /etc/systemd/system instead of using the pkg service files?
I am also wondering this.

The stock file just seems to have

Code: Select all

After=network.target auditd.service
So I presume the OP has added network-online.target, along with the subsequent Wants= line, in an attempt to solve this.

If so then the OP really should have told us :)
imthenachoman wrote:ssh.service will not start at boot but it will start if I run service ssh start after boot.
It's a dirty hack but you could add a delay to /lib/systemd/system/ssh.service by running

Code: Select all

# systemctl edit ssh.service
and then adding

Code: Select all

[Service]
ExecStartPre=
ExecStartPre=/bin/sleep 30
ExecStartPre=/usr/sbin/sshd -t
You may have to play around with the `sleep` interval to get the optimal value.

Remove your custom unit files in /etc/systemd/system before trying this though (and run `systemctl daemon-reload` to update the configuration), they obviously don't work.

Disclaimer: I don't use sshd.
deadbang

imthenachoman
Posts: 8
Joined: 2019-02-10 04:53

Re: ssh.service will not start at boot if using ListenAddres

#4 Post by imthenachoman »

Yes, I made changes to the service files to add the network-online.wants. Sorry for not disclosing that.

I just did a fresh Debian 9 expert install in a VM. The only thing I selected was ssh server and core stuff. Then added ListenAddress to /etc/ssh/sshd_config and am having the same problem.

This is what is in /etc/ssh/sshd_config:

Code: Select all

ChallengeResponseAuthentication no
UsePAM yes
X11Forwarding yes
PrintMotd no
AcceptEnv LANG LC_*
Subsystem       sftp    /usr/lib/openssh/sftp-server
ListenAddress 10.10.20.152
It looks like network-manager is not used:

Code: Select all

root@debian:~# dpkg --get-selections | grep -i network
root@debian:~#
Here is what is in /etc/systemd/system:

Code: Select all

drwxr-xr-x 2 root root 4096 Feb 10 08:32 getty.target.wants
drwxr-xr-x 2 root root 4096 Feb 10 08:35 multi-user.target.wants
drwxr-xr-x 2 root root 4096 Feb 10 08:32 network-online.target.wants
lrwxrwxrwx 1 root root   31 Feb 10 08:35 sshd.service -> /lib/systemd/system/ssh.service
drwxr-xr-x 2 root root 4096 Feb 10 08:33 sysinit.target.wants
lrwxrwxrwx 1 root root   35 Feb 10 08:32 syslog.service -> /lib/systemd/system/rsyslog.service
drwxr-xr-x 2 root root 4096 Feb 10 08:32 timers.target.wants
And here is the content of /lib/systemd/system/ssh.service:

Code: Select all

[Unit]
Description=OpenBSD Secure Shell server
After=network.target auditd.service
ConditionPathExists=!/etc/ssh/sshd_not_to_be_run

[Service]
EnvironmentFile=-/etc/default/ssh
ExecStartPre=/usr/sbin/sshd -t
ExecStart=/usr/sbin/sshd -D $SSHD_OPTS
ExecReload=/usr/sbin/sshd -t
ExecReload=/bin/kill -HUP $MAINPID
KillMode=process
Restart=on-failure
RestartPreventExitStatus=255
Type=notify

[Install]
WantedBy=multi-user.target
Alias=sshd.service
Here is output of journalctl -xe | egrep "ssh|network":

Code: Select all

root@debian:~# journalctl -xe | egrep "ssh|network"
Feb 10 08:39:17 debian systemd[1]: Started Raise network interfaces.
-- Subject: Unit networking.service has finished start-up
-- Unit networking.service has finished starting up.
-- Subject: Unit network.target has finished start-up
-- Unit network.target has finished starting up.
-- Subject: Unit ssh.service has begun start-up
-- Unit ssh.service has begun starting up.
Feb 10 08:39:18 debian sshd[320]: error: Bind to port 22 on 10.10.20.152 failed: Cannot assign requested address.
Feb 10 08:39:18 debian sshd[320]: fatal: Cannot bind any address.
Feb 10 08:39:18 debian systemd[1]: ssh.service: Main process exited, code=exited, status=255/n/a
-- Subject: Unit ssh.service has failed
-- Unit ssh.service has failed.
Feb 10 08:39:18 debian systemd[1]: ssh.service: Unit entered failed state.
Feb 10 08:39:18 debian systemd[1]: ssh.service: Failed with result 'exit-code'.
Is there anyway to get this to work without resorting to sleep hacks or delaying SSH to start till the end. There must be a right way to get this to work...

User avatar
Head_on_a_Stick
Posts: 14114
Joined: 2014-06-01 17:46
Location: London, England
Has thanked: 81 times
Been thanked: 132 times

Re: ssh.service will not start at boot if using ListenAddres

#5 Post by Head_on_a_Stick »

imthenachoman wrote:Here is what is in /etc/systemd/system
That doesn't actually show the full unit file configuration, try this instead:

Code: Select all

ls -lR /etc/systemd/system
Or

Code: Select all

systemctl list-unit-files --state=enabled
But anyway I can see that you're using networking.service (and hence ifupdown via /etc/network/interfaces{,.d/*}) from the journal output.

Have you tried using systemd-networkd instead? That is very well integrated with the rest of systemd and so may work better. See systemd-networkd(8) & systemd.network(5) for the configuration details, it is very simple to set up.

Be sure to disable networking.service first though to avoid conflicts.
deadbang

imthenachoman
Posts: 8
Joined: 2019-02-10 04:53

Re: ssh.service will not start at boot if using ListenAddres

#6 Post by imthenachoman »

Okay, I will see if I can figure that out. This is a fresh install of Debian so I am just using what it configured.

Code: Select all

# systemctl list-unit-files --state=enabled
UNIT FILE                 STATE
autovt@.service           enabled
console-setup.service     enabled
cron.service              enabled
getty@.service            enabled
keyboard-setup.service    enabled
networking.service        enabled
rsyslog.service           enabled
ssh.service               enabled
sshd.service              enabled
syslog.service            enabled
systemd-timesyncd.service enabled
remote-fs.target          enabled
apt-daily-upgrade.timer   enabled
apt-daily.timer           enabled

14 unit files listed.

reinob
Posts: 1189
Joined: 2014-06-30 11:42
Has thanked: 97 times
Been thanked: 47 times

Re: ssh.service will not start at boot if using ListenAddres

#7 Post by reinob »

In your first post sshd fails to bind to 192.168.1.100
But later you show your sshd_config, which has:

Code: Select all

ListenAddress 10.10.20.152
Can you clarify this?
and can you tell us which IP address you actually use?
(you know, you can't just bind to anything..)

User avatar
bw123
Posts: 4015
Joined: 2011-05-09 06:02
Has thanked: 1 time
Been thanked: 28 times

Re: ssh.service will not start at boot if using ListenAddres

#8 Post by bw123 »

imthenachoman wrote: ...
This is a fresh install of Debian so I am just using what it configured.
...
Then added ListenAddress to /etc/ssh/sshd_config and am having the same problem.
...
Is there anyway to get this to work without resorting to sleep hacks or delaying SSH to start till the end. There must be a right way to get this to work...
I'd guess since you are already willing to change your "fresh" install by adding ListenAddress to sshd_config, it shouldn't be too hackish to also add in some delay to make sure the network is up? There's probably several ways that would work, is that what you're after, or just one "right" way?

I read some interesting things this morning in bugs.debian.org/openssh-server have you checked it out there?

p.s. I like systemctl edit ssh.service like HoAs suggested. This is the better way IMO to alter the service files, instead of copying them around. It does the whole job in one go. By have an override.conf it's easy to see exactly what lines you are adding or changing from the default config.
resigned by AI ChatGPT

imthenachoman
Posts: 8
Joined: 2019-02-10 04:53

Re: ssh.service will not start at boot if using ListenAddres

#9 Post by imthenachoman »

reinob wrote:In your first post sshd fails to bind to 192.168.1.100
But later you show your sshd_config, which has:

Code: Select all

ListenAddress 10.10.20.152
Can you clarify this?
and can you tell us which IP address you actually use?
(you know, you can't just bind to anything..)
My network is actually on 10.10.20.*. In the first post I changed it to 192.168.1.* in an attempt to hide potentially sensitive information. In the second post I forgot. Heh.

And yes, I know you can't just bind to anything. I am binding to the IP address of the server.

imthenachoman
Posts: 8
Joined: 2019-02-10 04:53

Re: ssh.service will not start at boot if using ListenAddres

#10 Post by imthenachoman »

bw123 wrote:
imthenachoman wrote: ...
This is a fresh install of Debian so I am just using what it configured.
...
Then added ListenAddress to /etc/ssh/sshd_config and am having the same problem.
...
Is there anyway to get this to work without resorting to sleep hacks or delaying SSH to start till the end. There must be a right way to get this to work...
I'd guess since you are already willing to change your "fresh" install by adding ListenAddress to sshd_config, it shouldn't be too hackish to also add in some delay to make sure the network is up? There's probably several ways that would work, is that what you're after, or just one "right" way?

I read some interesting things this morning in bugs.debian.org/openssh-server have you checked it out there?

p.s. I like systemctl edit ssh.service like HoAs suggested. This is the better way IMO to alter the service files, instead of copying them around. It does the whole job in one go. By have an override.conf it's easy to see exactly what lines you are adding or changing from the default config.
ListenAddress is a very standard configuration option of SSHD. Setting it should not break system start-up. Changing system start up code to accommodate an applications configuration is very hacky and not the right answer.

If I tell one service (i.e. ssh.service) to start after another service has finished (i.e. network-wants.online) then it is not unrealistic to expect that to work. If it doesn't then that is a bug -- right?

And yes, I read many of the bugs, many highlight the same issue I am having but none seem to have a real fix -- one that makes ssh.service start after network-wants.online actually finishes.

User avatar
bw123
Posts: 4015
Joined: 2011-05-09 06:02
Has thanked: 1 time
Been thanked: 28 times

Re: ssh.service will not start at boot if using ListenAddres

#11 Post by bw123 »

I haven;t tried it myself, but it seems that ssh.service does "start after network-wants.online actually finishes." The problem seems to be that not all methods of configuring the network are immediate, and they don't notify anything anywhere of anything. It's just an address after all.
The problem goes way back, at least to 2010 https://bugs.debian.org/cgi-bin/bugrepo ... bug=590578

Breaks system startup, isn't that an exaggeration?

How do you recommend it get fixed? How would you determine that the network address in a config file exists/active on the system before trying to use it? Do you want to ping something? I am not sure I want that myself by default. Would you halt startup on error, or wait? How long would your timeout be? Would you try to accomodate every single method available for configuring a network device? DHCP can take awhile, what if the router is down, then what?

You could chime in with some ideas, instead of pointing out the issue. From my searches, it seems pretty well known issue for "a very standard configuration option of SSHD."

p.s. Nice use of bold font the documentation I read (found by running 'systemctl status network-online.target') is at https://www.freedesktop.org/wiki/Softwa ... orkTarget/ and it says use one of the two *wait-online.service either networkd-wait-online or networkmanager-wait-online to solve it.
resigned by AI ChatGPT

imthenachoman
Posts: 8
Joined: 2019-02-10 04:53

Re: ssh.service will not start at boot if using ListenAddres

#12 Post by imthenachoman »

bw123 wrote:Breaks system startup, isn't that an exaggeration?
Well, the system starts but SSH does not which makes a server rather useless. Maybe I am being overly dramatic but to me "system startup" means the system is in running and usable. For me, not having SSH start means system startup is broken. :/
bw123 wrote:How do you recommend it get fixed? How would you determine that the network address in a config file exists/active on the system before trying to use it? Do you want to ping something? I am not sure I want that myself by default. Would you halt startup on error, or wait? How long would your timeout be? Would you try to accomodate every single method available for configuring a network device? DHCP can take awhile, what if the router is down, then what?
Humm. I had not thought of that. You raise a good point. I need to digest this notion. Maybe I don't need to use ListenAddress in sshd_config.
bw123 wrote:p.s. Nice use of bold font
Thanks. I think -- I can't tell if you're being sarcastic. I am used to markdown where I can use in-line code blocks to make posts easier to read/understand. This forum doesn't have in-line code blocks so I figured bold would work.
bw123 wrote:the documentation I read (found by running 'systemctl status network-online.target') is at https://www.freedesktop.org/wiki/Softwa ... orkTarget/ and it says use one of the two *wait-online.service either networkd-wait-online or networkmanager-wait-online to solve it.
Tried this already -- no dice. But, like you said, maybe I shouldn't be doing this.

reinob
Posts: 1189
Joined: 2014-06-30 11:42
Has thanked: 97 times
Been thanked: 47 times

Re: ssh.service will not start at boot if using ListenAddres

#13 Post by reinob »

imthenachoman wrote: Humm. I had not thought of that. You raise a good point. I need to digest this notion. Maybe I don't need to use ListenAddress in sshd_config.
You could always remove ListenAddress (or use "0.0.0.0" or "::", etc.) and then use the firewall (if needed) to block the unwanted connections.

I think systemd's network-online.target does not do what people expect. Probably nobody really knows what it does :)
and in my (little) experience it's very hit-and-miss. Even the official documentation says "Usually it indicates a configured, routable IP address of some kind", which in many cases is not what you want (i.e. which specific interface, etc.) plus it will depend a lot on whether you use systemd-networkd, ifupdown, network manager, etc.

So much for a "deterministic" boot, especially combined with headless servers...

imthenachoman
Posts: 8
Joined: 2019-02-10 04:53

Re: ssh.service will not start at boot if using ListenAddres

#14 Post by imthenachoman »

reinob wrote:You could always remove ListenAddress (or use "0.0.0.0" or "::", etc.) and then use the firewall (if needed) to block the unwanted connections.
Yup. That is what I am doing now. I do wish that service did behave as expected though. It's not unrealistic to want something to start after network is up. I would think maybe it would wait till the rest of the computer is up, at login prompt, and then wait for network and then run the dependent services. Although, this approach could go crazy.

Post Reply