ssh.service will not start at boot if using ListenAddress

Kernels & Hardware, configuring network, installing services

ssh.service will not start at boot if using ListenAddress

Postby imthenachoman » 2019-02-10 05:15

I have ListenAddress in my /etc/ssh/sshd_config.

ssh.service will not start at boot but it will start if I run service ssh start after boot.

Here is the output from journalctl -xe | egrep "ssh|network":

Code: Select all
Feb 10 00:17:46 nook systemd[1]: Starting Raise network interfaces...
-- Subject: Unit networking.service has begun start-up
-- Unit networking.service has begun starting up.
Feb 10 00:17:46 nook systemd[1]: Started Raise network interfaces.
-- Subject: Unit networking.service has finished start-up
-- Unit networking.service has finished starting up.
-- Subject: Unit network.target has finished start-up
-- Unit network.target has finished starting up.
-- Subject: Unit network-online.target has finished start-up
-- Unit network-online.target has finished starting up.
-- Subject: Unit ssh.service has begun start-up
-- Unit ssh.service has begun starting up.
Feb 10 00:17:46 nook sshd[563]: error: Bind to port 22 on 192.168.1.100 failed: Cannot assign requested address.
Feb 10 00:17:46 nook sshd[563]: fatal: Cannot bind any address.
Feb 10 00:17:46 nook systemd[1]: ssh.service: Main process exited, code=exited, status=255/n/a
-- Subject: Unit ssh.service has failed
-- Unit ssh.service has failed.
Feb 10 00:17:46 nook systemd[1]: ssh.service: Unit entered failed state.
Feb 10 00:17:46 nook systemd[1]: ssh.service: Failed with result 'exit-code'.


/etc/systemd/system/ssh.service:

Code: Select all
[Unit]
Description=OpenBSD Secure Shell server
After=network.target auditd.service network-online.target
Wants=network-online.target
ConditionPathExists=!/etc/ssh/sshd_not_to_be_run

[Service]
EnvironmentFile=-/etc/default/ssh
ExecStartPre=/usr/sbin/sshd -t
ExecStart=/usr/sbin/sshd -D $SSHD_OPTS
ExecReload=/usr/sbin/sshd -t
ExecReload=/bin/kill -HUP $MAINPID
KillMode=process
Restart=on-failure
RestartPreventExitStatus=255
Type=notify

[Install]
WantedBy=multi-user.target
Alias=sshd.service


/etc/systemd/system/sshd.service:

Code: Select all
[Unit]
Description=OpenBSD Secure Shell server
After=network.target auditd.service network-online.target
Wants=network-online.target
ConditionPathExists=!/etc/ssh/sshd_not_to_be_run

[Service]
EnvironmentFile=-/etc/default/ssh
ExecStartPre=/usr/sbin/sshd -t
ExecStart=/usr/sbin/sshd -D $SSHD_OPTS
ExecReload=/usr/sbin/sshd -t
ExecReload=/bin/kill -HUP $MAINPID
KillMode=process
Restart=on-failure
RestartPreventExitStatus=255
Type=notify

[Install]
WantedBy=multi-user.target
Alias=sshd.service


I've looked online but can't seem to figure out what is wrong. It works if I start it manually after boot but it won't start at boot. :/

It works if I take out ListenAddress from /etc/ssh/sshd_config but I need/want it there.
imthenachoman
 
Posts: 7
Joined: 2019-02-10 04:53

Re: ssh.service will not start at boot if using ListenAddres

Postby bw123 » 2019-02-10 07:38

Maybe the Wants=network-online.target is problematic, are you really using networkmanager to bring up the network?
Have you tried making the daemon wait to start by instead using wants multi-user.target or something like that?

maybe if you post your whole sshd_config someone can spot something, and debian ver and openssh-server ver is probably a good idea.

Why did you have to make all the services and put them in /etc/systemd/system instead of using the pkg service files?
User avatar
bw123
 
Posts: 3787
Joined: 2011-05-09 06:02

Re: ssh.service will not start at boot if using ListenAddres

Postby Head_on_a_Stick » 2019-02-10 12:11

bw123 wrote:Why did you have to make all the services and put them in /etc/systemd/system instead of using the pkg service files?

I am also wondering this.

The stock file just seems to have
Code: Select all
After=network.target auditd.service

So I presume the OP has added network-online.target, along with the subsequent Wants= line, in an attempt to solve this.

If so then the OP really should have told us :)

imthenachoman wrote:ssh.service will not start at boot but it will start if I run service ssh start after boot.

It's a dirty hack but you could add a delay to /lib/systemd/system/ssh.service by running
Code: Select all
# systemctl edit ssh.service

and then adding
Code: Select all
[Service]
ExecStartPre=
ExecStartPre=/bin/sleep 30
ExecStartPre=/usr/sbin/sshd -t

You may have to play around with the `sleep` interval to get the optimal value.

Remove your custom unit files in /etc/systemd/system before trying this though (and run `systemctl daemon-reload` to update the configuration), they obviously don't work.

Disclaimer: I don't use sshd.
User avatar
Head_on_a_Stick
 
Posts: 8900
Joined: 2014-06-01 17:46
Location: /dev/chair

Re: ssh.service will not start at boot if using ListenAddres

Postby imthenachoman » 2019-02-10 12:58

Yes, I made changes to the service files to add the network-online.wants. Sorry for not disclosing that.

I just did a fresh Debian 9 expert install in a VM. The only thing I selected was ssh server and core stuff. Then added ListenAddress to /etc/ssh/sshd_config and am having the same problem.

This is what is in /etc/ssh/sshd_config:

Code: Select all
ChallengeResponseAuthentication no
UsePAM yes
X11Forwarding yes
PrintMotd no
AcceptEnv LANG LC_*
Subsystem       sftp    /usr/lib/openssh/sftp-server
ListenAddress 10.10.20.152


It looks like network-manager is not used:

Code: Select all
root@debian:~# dpkg --get-selections | grep -i network
root@debian:~#


Here is what is in /etc/systemd/system:

Code: Select all
drwxr-xr-x 2 root root 4096 Feb 10 08:32 getty.target.wants
drwxr-xr-x 2 root root 4096 Feb 10 08:35 multi-user.target.wants
drwxr-xr-x 2 root root 4096 Feb 10 08:32 network-online.target.wants
lrwxrwxrwx 1 root root   31 Feb 10 08:35 sshd.service -> /lib/systemd/system/ssh.service
drwxr-xr-x 2 root root 4096 Feb 10 08:33 sysinit.target.wants
lrwxrwxrwx 1 root root   35 Feb 10 08:32 syslog.service -> /lib/systemd/system/rsyslog.service
drwxr-xr-x 2 root root 4096 Feb 10 08:32 timers.target.wants


And here is the content of /lib/systemd/system/ssh.service:

Code: Select all
[Unit]
Description=OpenBSD Secure Shell server
After=network.target auditd.service
ConditionPathExists=!/etc/ssh/sshd_not_to_be_run

[Service]
EnvironmentFile=-/etc/default/ssh
ExecStartPre=/usr/sbin/sshd -t
ExecStart=/usr/sbin/sshd -D $SSHD_OPTS
ExecReload=/usr/sbin/sshd -t
ExecReload=/bin/kill -HUP $MAINPID
KillMode=process
Restart=on-failure
RestartPreventExitStatus=255
Type=notify

[Install]
WantedBy=multi-user.target
Alias=sshd.service


Here is output of journalctl -xe | egrep "ssh|network":

Code: Select all
root@debian:~# journalctl -xe | egrep "ssh|network"
Feb 10 08:39:17 debian systemd[1]: Started Raise network interfaces.
-- Subject: Unit networking.service has finished start-up
-- Unit networking.service has finished starting up.
-- Subject: Unit network.target has finished start-up
-- Unit network.target has finished starting up.
-- Subject: Unit ssh.service has begun start-up
-- Unit ssh.service has begun starting up.
Feb 10 08:39:18 debian sshd[320]: error: Bind to port 22 on 10.10.20.152 failed: Cannot assign requested address.
Feb 10 08:39:18 debian sshd[320]: fatal: Cannot bind any address.
Feb 10 08:39:18 debian systemd[1]: ssh.service: Main process exited, code=exited, status=255/n/a
-- Subject: Unit ssh.service has failed
-- Unit ssh.service has failed.
Feb 10 08:39:18 debian systemd[1]: ssh.service: Unit entered failed state.
Feb 10 08:39:18 debian systemd[1]: ssh.service: Failed with result 'exit-code'.


Is there anyway to get this to work without resorting to sleep hacks or delaying SSH to start till the end. There must be a right way to get this to work...
imthenachoman
 
Posts: 7
Joined: 2019-02-10 04:53

Re: ssh.service will not start at boot if using ListenAddres

Postby Head_on_a_Stick » 2019-02-10 13:27

imthenachoman wrote:Here is what is in /etc/systemd/system

That doesn't actually show the full unit file configuration, try this instead:
Code: Select all
ls -lR /etc/systemd/system

Or
Code: Select all
systemctl list-unit-files --state=enabled

But anyway I can see that you're using networking.service (and hence ifupdown via /etc/network/interfaces{,.d/*}) from the journal output.

Have you tried using systemd-networkd instead? That is very well integrated with the rest of systemd and so may work better. See systemd-networkd(8) & systemd.network(5) for the configuration details, it is very simple to set up.

Be sure to disable networking.service first though to avoid conflicts.
User avatar
Head_on_a_Stick
 
Posts: 8900
Joined: 2014-06-01 17:46
Location: /dev/chair

Re: ssh.service will not start at boot if using ListenAddres

Postby imthenachoman » 2019-02-10 13:42

Okay, I will see if I can figure that out. This is a fresh install of Debian so I am just using what it configured.

Code: Select all
# systemctl list-unit-files --state=enabled
UNIT FILE                 STATE
autovt@.service           enabled
console-setup.service     enabled
cron.service              enabled
getty@.service            enabled
keyboard-setup.service    enabled
networking.service        enabled
rsyslog.service           enabled
ssh.service               enabled
sshd.service              enabled
syslog.service            enabled
systemd-timesyncd.service enabled
remote-fs.target          enabled
apt-daily-upgrade.timer   enabled
apt-daily.timer           enabled

14 unit files listed.
imthenachoman
 
Posts: 7
Joined: 2019-02-10 04:53

Re: ssh.service will not start at boot if using ListenAddres

Postby reinob » 2019-02-10 14:52

In your first post sshd fails to bind to 192.168.1.100
But later you show your sshd_config, which has:
Code: Select all
ListenAddress 10.10.20.152


Can you clarify this?
and can you tell us which IP address you actually use?
(you know, you can't just bind to anything..)
reinob
 
Posts: 597
Joined: 2014-06-30 11:42

Re: ssh.service will not start at boot if using ListenAddres

Postby bw123 » 2019-02-10 15:14

imthenachoman wrote:...
This is a fresh install of Debian so I am just using what it configured.
...
Then added ListenAddress to /etc/ssh/sshd_config and am having the same problem.
...
Is there anyway to get this to work without resorting to sleep hacks or delaying SSH to start till the end. There must be a right way to get this to work...


I'd guess since you are already willing to change your "fresh" install by adding ListenAddress to sshd_config, it shouldn't be too hackish to also add in some delay to make sure the network is up? There's probably several ways that would work, is that what you're after, or just one "right" way?

I read some interesting things this morning in bugs.debian.org/openssh-server have you checked it out there?

p.s. I like systemctl edit ssh.service like HoAs suggested. This is the better way IMO to alter the service files, instead of copying them around. It does the whole job in one go. By have an override.conf it's easy to see exactly what lines you are adding or changing from the default config.
User avatar
bw123
 
Posts: 3787
Joined: 2011-05-09 06:02

Re: ssh.service will not start at boot if using ListenAddres

Postby imthenachoman » 2019-02-10 23:13

reinob wrote:In your first post sshd fails to bind to 192.168.1.100
But later you show your sshd_config, which has:
Code: Select all
ListenAddress 10.10.20.152


Can you clarify this?
and can you tell us which IP address you actually use?
(you know, you can't just bind to anything..)


My network is actually on 10.10.20.*. In the first post I changed it to 192.168.1.* in an attempt to hide potentially sensitive information. In the second post I forgot. Heh.

And yes, I know you can't just bind to anything. I am binding to the IP address of the server.
imthenachoman
 
Posts: 7
Joined: 2019-02-10 04:53

Re: ssh.service will not start at boot if using ListenAddres

Postby imthenachoman » 2019-02-10 23:18

bw123 wrote:
imthenachoman wrote:...
This is a fresh install of Debian so I am just using what it configured.
...
Then added ListenAddress to /etc/ssh/sshd_config and am having the same problem.
...
Is there anyway to get this to work without resorting to sleep hacks or delaying SSH to start till the end. There must be a right way to get this to work...


I'd guess since you are already willing to change your "fresh" install by adding ListenAddress to sshd_config, it shouldn't be too hackish to also add in some delay to make sure the network is up? There's probably several ways that would work, is that what you're after, or just one "right" way?

I read some interesting things this morning in bugs.debian.org/openssh-server have you checked it out there?

p.s. I like systemctl edit ssh.service like HoAs suggested. This is the better way IMO to alter the service files, instead of copying them around. It does the whole job in one go. By have an override.conf it's easy to see exactly what lines you are adding or changing from the default config.


ListenAddress is a very standard configuration option of SSHD. Setting it should not break system start-up. Changing system start up code to accommodate an applications configuration is very hacky and not the right answer.

If I tell one service (i.e. ssh.service) to start after another service has finished (i.e. network-wants.online) then it is not unrealistic to expect that to work. If it doesn't then that is a bug -- right?

And yes, I read many of the bugs, many highlight the same issue I am having but none seem to have a real fix -- one that makes ssh.service start after network-wants.online actually finishes.
imthenachoman
 
Posts: 7
Joined: 2019-02-10 04:53

Re: ssh.service will not start at boot if using ListenAddres

Postby bw123 » 2019-02-11 01:07

I haven;t tried it myself, but it seems that ssh.service does "start after network-wants.online actually finishes." The problem seems to be that not all methods of configuring the network are immediate, and they don't notify anything anywhere of anything. It's just an address after all.
The problem goes way back, at least to 2010 https://bugs.debian.org/cgi-bin/bugrepo ... bug=590578

Breaks system startup, isn't that an exaggeration?

How do you recommend it get fixed? How would you determine that the network address in a config file exists/active on the system before trying to use it? Do you want to ping something? I am not sure I want that myself by default. Would you halt startup on error, or wait? How long would your timeout be? Would you try to accomodate every single method available for configuring a network device? DHCP can take awhile, what if the router is down, then what?

You could chime in with some ideas, instead of pointing out the issue. From my searches, it seems pretty well known issue for "a very standard configuration option of SSHD."

p.s. Nice use of bold font the documentation I read (found by running 'systemctl status network-online.target') is at https://www.freedesktop.org/wiki/Softwa ... orkTarget/ and it says use one of the two *wait-online.service either networkd-wait-online or networkmanager-wait-online to solve it.
User avatar
bw123
 
Posts: 3787
Joined: 2011-05-09 06:02

Re: ssh.service will not start at boot if using ListenAddres

Postby imthenachoman » 2019-02-11 02:46

bw123 wrote:Breaks system startup, isn't that an exaggeration?


Well, the system starts but SSH does not which makes a server rather useless. Maybe I am being overly dramatic but to me "system startup" means the system is in running and usable. For me, not having SSH start means system startup is broken. :/

bw123 wrote:How do you recommend it get fixed? How would you determine that the network address in a config file exists/active on the system before trying to use it? Do you want to ping something? I am not sure I want that myself by default. Would you halt startup on error, or wait? How long would your timeout be? Would you try to accomodate every single method available for configuring a network device? DHCP can take awhile, what if the router is down, then what?


Humm. I had not thought of that. You raise a good point. I need to digest this notion. Maybe I don't need to use ListenAddress in sshd_config.

bw123 wrote:p.s. Nice use of bold font


Thanks. I think -- I can't tell if you're being sarcastic. I am used to markdown where I can use in-line code blocks to make posts easier to read/understand. This forum doesn't have in-line code blocks so I figured bold would work.

bw123 wrote:the documentation I read (found by running 'systemctl status network-online.target') is at https://www.freedesktop.org/wiki/Softwa ... orkTarget/ and it says use one of the two *wait-online.service either networkd-wait-online or networkmanager-wait-online to solve it.


Tried this already -- no dice. But, like you said, maybe I shouldn't be doing this.
imthenachoman
 
Posts: 7
Joined: 2019-02-10 04:53

Re: ssh.service will not start at boot if using ListenAddres

Postby reinob » 2019-02-12 09:39

imthenachoman wrote:Humm. I had not thought of that. You raise a good point. I need to digest this notion. Maybe I don't need to use ListenAddress in sshd_config.


You could always remove ListenAddress (or use "0.0.0.0" or "::", etc.) and then use the firewall (if needed) to block the unwanted connections.

I think systemd's network-online.target does not do what people expect. Probably nobody really knows what it does :)
and in my (little) experience it's very hit-and-miss. Even the official documentation says "Usually it indicates a configured, routable IP address of some kind", which in many cases is not what you want (i.e. which specific interface, etc.) plus it will depend a lot on whether you use systemd-networkd, ifupdown, network manager, etc.

So much for a "deterministic" boot, especially combined with headless servers...
reinob
 
Posts: 597
Joined: 2014-06-30 11:42

Re: ssh.service will not start at boot if using ListenAddres

Postby imthenachoman » 2019-02-12 12:58

reinob wrote:You could always remove ListenAddress (or use "0.0.0.0" or "::", etc.) and then use the firewall (if needed) to block the unwanted connections.


Yup. That is what I am doing now. I do wish that service did behave as expected though. It's not unrealistic to want something to start after network is up. I would think maybe it would wait till the rest of the computer is up, at login prompt, and then wait for network and then run the dependent services. Although, this approach could go crazy.
imthenachoman
 
Posts: 7
Joined: 2019-02-10 04:53


Return to System configuration

Who is online

Users browsing this forum: No registered users and 10 guests

fashionable