Tracking down cause of systemd mount events

Kernels & Hardware, configuring network, installing services

Tracking down cause of systemd mount events

Postby troffasky » 2020-11-07 09:22

These events are logged in the journal, every 6 minutes:

Code: Select all
Nov 07 09:11:21 westogre systemd[1]: home-.zfs-snapshot-autosnap_2020\x2d08\x2d23_22:08:01_monthly.mount: Succeeded.
Nov 07 09:17:21 westogre systemd[1]: home-.zfs-snapshot-autosnap_2020\x2d08\x2d23_22:08:01_monthly.mount: Succeeded.
Nov 07 09:17:21 westogre systemd[92140]: home-.zfs-snapshot-autosnap_2020\x2d08\x2d23_22:08:01_monthly.mount: Succeeded.
Nov 07 09:23:21 westogre systemd[92140]: home-.zfs-snapshot-autosnap_2020\x2d08\x2d23_22:08:01_monthly.mount: Succeeded.
Nov 07 09:23:21 westogre systemd[1]: home-.zfs-snapshot-autosnap_2020\x2d08\x2d23_22:08:01_monthly.mount: Succeeded.
Nov 07 09:29:22 westogre systemd[1]: home-.zfs-snapshot-autosnap_2020\x2d08\x2d23_22:08:01_monthly.mount: Succeeded.
Nov 07 09:29:22 westogre systemd[92140]: home-.zfs-snapshot-autosnap_2020\x2d08\x2d23_22:08:01_monthly.mount: Succeeded.
Nov 07 09:35:22 westogre systemd[1]: home-.zfs-snapshot-autosnap_2020\x2d08\x2d23_22:08:01_monthly.mount: Succeeded.
Nov 07 09:35:22 westogre systemd[92140]: home-.zfs-snapshot-autosnap_2020\x2d08\x2d23_22:08:01_monthly.mount: Succeeded.


That snapshot is the oldest snapshot on my /home filesystem and it was created by sanoid. It's not currently mounted.
I don't think it's sanoid causing these .mount events, as sanoid isn't responsible for mounting snapshots, only creating and destroying them. I also have three other filesystems in this pool which sanoid is managing snapshots on, and which do not have any events like the above.
I cannot find a .mount file anywhere that corresponds with this.
I cannot find a timer that corresponds with this:
Code: Select all
 # systemctl list-timers --all
NEXT                        LEFT         LAST                        PASSED       UNIT                         ACTIVATES
Sat 2020-11-07 10:00:00 GMT 8min left    Sat 2020-11-07 09:45:02 GMT 6min ago     sanoid.timer                 sanoid.service
Sat 2020-11-07 10:09:00 GMT 17min left   Sat 2020-11-07 09:39:02 GMT 12min ago    phpsessionclean.timer        phpsessionclean.service
Sat 2020-11-07 10:33:15 GMT 41min left   Sat 2020-11-07 09:30:57 GMT 20min ago    anacron.timer                anacron.service
Sat 2020-11-07 11:00:32 GMT 1h 8min left Fri 2020-11-06 11:00:32 GMT 22h ago      systemd-tmpfiles-clean.timer systemd-tmpfiles-clean.service
Sat 2020-11-07 21:26:22 GMT 11h left     Sat 2020-11-07 07:02:29 GMT 2h 49min ago certbot.timer                certbot.service
Sat 2020-11-07 21:30:15 GMT 11h left     Sat 2020-11-07 09:28:02 GMT 23min ago    apt-daily.timer              apt-daily.service
Sun 2020-11-08 00:00:00 GMT 14h left     Sat 2020-11-07 00:00:02 GMT 9h ago       exim4-base.timer             exim4-base.service
Sun 2020-11-08 00:00:00 GMT 14h left     Sat 2020-11-07 00:00:02 GMT 9h ago       logrotate.timer              logrotate.service
Sun 2020-11-08 00:00:00 GMT 14h left     Sat 2020-11-07 00:00:02 GMT 9h ago       man-db.timer                 man-db.service
Sun 2020-11-08 03:10:18 GMT 17h left     Sun 2020-11-01 03:10:42 GMT 6 days ago   e2scrub_all.timer            e2scrub_all.service
Sun 2020-11-08 06:33:56 GMT 20h left     Sat 2020-11-07 06:49:02 GMT 3h 2min ago  apt-daily-upgrade.timer      apt-daily-upgrade.service

sanoid has a timer but it's every 15 minutes, not every 5.
There is no /etc/fstab entry for this snapshot.
troffasky
 
Posts: 25
Joined: 2017-11-05 11:00

Re: Tracking down cause of systemd mount events

Postby troffasky » 2020-11-07 09:34

This is so weird, it actually *is* mounted, but I swear it wasn't showing up in mount or df before!
I actually found this with "systemctl | grep zfs" [I had been doing "systemctl status | grep zfs" and coming up with nothing....gahhhh]:
Code: Select all
# systemctl  | grep zfs
  home-.zfs-snapshot-autosnap_2020\x2d08\x2d23_22:08:01_monthly.mount                            loaded active mounted   /home/.zfs/snapshot/autosnap_2020-08-23_22:08:01_monthly             
  zfs-import-cache.service                                                                       loaded active exited    Import ZFS pools by cache file                                       
  zfs-load-module.service                                                                        loaded active exited    Install ZFS kernel module                                             
  zfs-mount.service                                                                              loaded active exited    Mount ZFS filesystems                                                 
  zfs-share.service                                                                              loaded active exited    ZFS file system shares                                               
  zfs-volume-wait.service                                                                        loaded active exited    Wait for ZFS Volume (zvol) links in /dev                             
  zfs-zed.service                                                                                loaded active running   ZFS Event Daemon (zed)                                               
  zfs-import.target                                                                              loaded active active    ZFS pool import target                                               
  zfs-volumes.target                                                                             loaded active active    ZFS volumes are ready                                                 
  zfs.target                                                                                     loaded active active    ZFS startup target
# systemctl  status  'home-.zfs-snapshot-autosnap_2020\x2d08\x2d23_22:08:01_monthly.mount'
* home-.zfs-snapshot-autosnap_2020\x2d08\x2d23_22:08:01_monthly.mount - /home/.zfs/snapshot/autosnap_2020-08-23_22:08:01_monthly
     Loaded: loaded (/proc/self/mountinfo)
     Active: active (mounted) since Sat 2020-11-07 10:06:09 GMT; 4min 42s ago
      Where: /home/.zfs/snapshot/autosnap_2020-08-23_22:08:01_monthly
       What: HomePool/homes@autosnap_2020-08-23_22:08:01_monthly


So that's a bit more info but I'm not really any closer to knowing *why* it's doing this. I guess I might have manually mounted that snapshot at some point in the past after setting up the automation of snapshots, just to confirm that it was working. But I don't want it mounted automatically, least of all every 6 minutes!
troffasky
 
Posts: 25
Joined: 2017-11-05 11:00

Re: Tracking down cause of systemd mount events

Postby troffasky » 2020-11-07 10:14

'ls -al /home/.zfs/snapshots/' caused every snapshot to be mounted automatically. 6 minutes later, they were all unmounted again, apart from the original snapshot I referred to in this post [I have tracked this with 'while sleep 30; do date && mount | grep homes; done']
This is weird though - most of the time that snapshot is mounted, but occasionally, it's not. "Occasionally" is of course, every 6 minutes.
Code: Select all
Sat  7 Nov 10:46:39 GMT 2020
HomePool/homes on /home type zfs (rw,noatime,xattr,noacl)
Sat  7 Nov 10:47:09 GMT 2020
HomePool/homes on /home type zfs (rw,noatime,xattr,noacl)
HomePool/homes@autosnap_2020-08-23_22:08:01_monthly on /home/.zfs/snapshot/autosnap_2020-08-23_22:08:01_monthly type zfs (ro,relatime,xattr,noacl)
Sat  7 Nov 10:47:39 GMT 2020
HomePool/homes on /home type zfs (rw,noatime,xattr,noacl)
HomePool/homes@autosnap_2020-08-23_22:08:01_monthly on /home/.zfs/snapshot/autosnap_2020-08-23_22:08:01_monthly type zfs (ro,relatime,xattr,noacl)
Sat  7 Nov 10:48:09 GMT 2020
HomePool/homes on /home type zfs (rw,noatime,xattr,noacl)
HomePool/homes@autosnap_2020-08-23_22:08:01_monthly on /home/.zfs/snapshot/autosnap_2020-08-23_22:08:01_monthly type zfs (ro,relatime,xattr,noacl)
Sat  7 Nov 10:48:39 GMT 2020
HomePool/homes on /home type zfs (rw,noatime,xattr,noacl)
HomePool/homes@autosnap_2020-08-23_22:08:01_monthly on /home/.zfs/snapshot/autosnap_2020-08-23_22:08:01_monthly type zfs (ro,relatime,xattr,noacl)
Sat  7 Nov 10:49:09 GMT 2020
HomePool/homes on /home type zfs (rw,noatime,xattr,noacl)
HomePool/homes@autosnap_2020-08-23_22:08:01_monthly on /home/.zfs/snapshot/autosnap_2020-08-23_22:08:01_monthly type zfs (ro,relatime,xattr,noacl)
Sat  7 Nov 10:49:39 GMT 2020
HomePool/homes on /home type zfs (rw,noatime,xattr,noacl)
HomePool/homes@autosnap_2020-08-23_22:08:01_monthly on /home/.zfs/snapshot/autosnap_2020-08-23_22:08:01_monthly type zfs (ro,relatime,xattr,noacl)
Sat  7 Nov 10:50:09 GMT 2020
HomePool/homes on /home type zfs (rw,noatime,xattr,noacl)
HomePool/homes@autosnap_2020-08-23_22:08:01_monthly on /home/.zfs/snapshot/autosnap_2020-08-23_22:08:01_monthly type zfs (ro,relatime,xattr,noacl)
Sat  7 Nov 10:50:39 GMT 2020
HomePool/homes on /home type zfs (rw,noatime,xattr,noacl)
HomePool/homes@autosnap_2020-08-23_22:08:01_monthly on /home/.zfs/snapshot/autosnap_2020-08-23_22:08:01_monthly type zfs (ro,relatime,xattr,noacl)
Sat  7 Nov 10:51:09 GMT 2020
HomePool/homes on /home type zfs (rw,noatime,xattr,noacl)
HomePool/homes@autosnap_2020-08-23_22:08:01_monthly on /home/.zfs/snapshot/autosnap_2020-08-23_22:08:01_monthly type zfs (ro,relatime,xattr,noacl)
Sat  7 Nov 10:51:39 GMT 2020
HomePool/homes on /home type zfs (rw,noatime,xattr,noacl)
HomePool/homes@autosnap_2020-08-23_22:08:01_monthly on /home/.zfs/snapshot/autosnap_2020-08-23_22:08:01_monthly type zfs (ro,relatime,xattr,noacl)
Sat  7 Nov 10:52:09 GMT 2020
HomePool/homes on /home type zfs (rw,noatime,xattr,noacl)
HomePool/homes@autosnap_2020-08-23_22:08:01_monthly on /home/.zfs/snapshot/autosnap_2020-08-23_22:08:01_monthly type zfs (ro,relatime,xattr,noacl)
Sat  7 Nov 10:52:39 GMT 2020
HomePool/homes on /home type zfs (rw,noatime,xattr,noacl)
Sat  7 Nov 10:53:09 GMT 2020
HomePool/homes on /home type zfs (rw,noatime,xattr,noacl)
HomePool/homes@autosnap_2020-08-23_22:08:01_monthly on /home/.zfs/snapshot/autosnap_2020-08-23_22:08:01_monthly type zfs (ro,relatime,xattr,noacl)
Sat  7 Nov 10:53:39 GMT 2020
HomePool/homes on /home type zfs (rw,noatime,xattr,noacl)
HomePool/homes@autosnap_2020-08-23_22:08:01_monthly on /home/.zfs/snapshot/autosnap_2020-08-23_22:08:01_monthly type zfs (ro,relatime,xattr,noacl)
Sat  7 Nov 10:54:09 GMT 2020
HomePool/homes on /home type zfs (rw,noatime,xattr,noacl)
HomePool/homes@autosnap_2020-08-23_22:08:01_monthly on /home/.zfs/snapshot/autosnap_2020-08-23_22:08:01_monthly type zfs (ro,relatime,xattr,noacl)
Sat  7 Nov 10:54:39 GMT 2020
HomePool/homes on /home type zfs (rw,noatime,xattr,noacl)
HomePool/homes@autosnap_2020-08-23_22:08:01_monthly on /home/.zfs/snapshot/autosnap_2020-08-23_22:08:01_monthly type zfs (ro,relatime,xattr,noacl)
Sat  7 Nov 10:55:09 GMT 2020
HomePool/homes on /home type zfs (rw,noatime,xattr,noacl)
HomePool/homes@autosnap_2020-08-23_22:08:01_monthly on /home/.zfs/snapshot/autosnap_2020-08-23_22:08:01_monthly type zfs (ro,relatime,xattr,noacl)
Sat  7 Nov 10:55:39 GMT 2020
HomePool/homes on /home type zfs (rw,noatime,xattr,noacl)
HomePool/homes@autosnap_2020-08-23_22:08:01_monthly on /home/.zfs/snapshot/autosnap_2020-08-23_22:08:01_monthly type zfs (ro,relatime,xattr,noacl)
Sat  7 Nov 10:56:09 GMT 2020
HomePool/homes on /home type zfs (rw,noatime,xattr,noacl)
HomePool/homes@autosnap_2020-08-23_22:08:01_monthly on /home/.zfs/snapshot/autosnap_2020-08-23_22:08:01_monthly type zfs (ro,relatime,xattr,noacl)
Sat  7 Nov 10:56:39 GMT 2020
HomePool/homes on /home type zfs (rw,noatime,xattr,noacl)
HomePool/homes@autosnap_2020-08-23_22:08:01_monthly on /home/.zfs/snapshot/autosnap_2020-08-23_22:08:01_monthly type zfs (ro,relatime,xattr,noacl)
Sat  7 Nov 10:57:09 GMT 2020
HomePool/homes on /home type zfs (rw,noatime,xattr,noacl)
HomePool/homes@autosnap_2020-08-23_22:08:01_monthly on /home/.zfs/snapshot/autosnap_2020-08-23_22:08:01_monthly type zfs (ro,relatime,xattr,noacl)
Sat  7 Nov 10:57:39 GMT 2020
HomePool/homes on /home type zfs (rw,noatime,xattr,noacl)
HomePool/homes@autosnap_2020-08-23_22:08:01_monthly on /home/.zfs/snapshot/autosnap_2020-08-23_22:08:01_monthly type zfs (ro,relatime,xattr,noacl)
Sat  7 Nov 10:58:09 GMT 2020
HomePool/homes on /home type zfs (rw,noatime,xattr,noacl)
HomePool/homes@autosnap_2020-08-23_22:08:01_monthly on /home/.zfs/snapshot/autosnap_2020-08-23_22:08:01_monthly type zfs (ro,relatime,xattr,noacl)
Sat  7 Nov 10:58:39 GMT 2020
HomePool/homes on /home type zfs (rw,noatime,xattr,noacl)
Sat  7 Nov 10:59:09 GMT 2020
HomePool/homes on /home type zfs (rw,noatime,xattr,noacl)
HomePool/homes@autosnap_2020-08-23_22:08:01_monthly on /home/.zfs/snapshot/autosnap_2020-08-23_22:08:01_monthly type zfs (ro,relatime,xattr,noacl)
troffasky
 
Posts: 25
Joined: 2017-11-05 11:00

Re: Tracking down cause of systemd mount events

Postby Head_on_a_Stick » 2020-11-07 16:17

Thread moved to Offtopic section, Ubuntu is not supported here.
Black Lives Matter

Debian buster-backports ISO image: for new hardware support
User avatar
Head_on_a_Stick
 
Posts: 12795
Joined: 2014-06-01 17:46
Location: /dev/chair

Re: Tracking down cause of systemd mount events

Postby troffasky » 2020-11-07 22:44

Uhhh, what? Ubuntu? This is *definitely* Debian.

Code: Select all
 # cat /etc/issue
Debian GNU/Linux bullseye/sid \n \l
troffasky
 
Posts: 25
Joined: 2017-11-05 11:00

Re: Tracking down cause of systemd mount events

Postby sickpig » 2020-11-07 23:46

Code: Select all
# systemctl  status  'home-.zfs-snapshot-autosnap_2020\x2d08\x2d23_22:08:01_monthly.mount'
* home-.zfs-snapshot-autosnap_2020\x2d08\x2d23_22:08:01_monthly.mount - /home/.zfs/snapshot/autosnap_2020-08-23_22:08:01_monthly
     Loaded: loaded (/proc/self/mountinfo)


If you check the Loaded.. line you will see that systemd is reading info from /proc, systemd is not mounting the snapshot. Had systemd mounted it you would seen the path to the mount unit like /usr/lib/systemd/system/..mount.

You must have a script or cron or something mounting it. systemd is just reading the mount.
User avatar
sickpig
 
Posts: 589
Joined: 2019-01-23 10:34

Re: Tracking down cause of systemd mount events

Postby Head_on_a_Stick » 2020-11-08 07:54

troffasky wrote:Uhhh, what? Ubuntu? This is *definitely* Debian.

Oh, sorry, I must have missed the bit in your OP that specified the version. Thread moved back to the System Configuration section.

It would probably help if you explained exactly how you installed Debian — the developers actually respect the licences of the software (unlike Ubuntu) so it is not possible to install onto ZFS using the official ISO images.
Black Lives Matter

Debian buster-backports ISO image: for new hardware support
User avatar
Head_on_a_Stick
 
Posts: 12795
Joined: 2014-06-01 17:46
Location: /dev/chair

Re: Tracking down cause of systemd mount events

Postby troffasky » 2020-11-08 09:49

sickpig wrote:
Code: Select all
# systemctl  status  'home-.zfs-snapshot-autosnap_2020\x2d08\x2d23_22:08:01_monthly.mount'
* home-.zfs-snapshot-autosnap_2020\x2d08\x2d23_22:08:01_monthly.mount - /home/.zfs/snapshot/autosnap_2020-08-23_22:08:01_monthly
     Loaded: loaded (/proc/self/mountinfo)


If you check the Loaded.. line you will see that systemd is reading info from /proc, systemd is not mounting the snapshot. Had systemd mounted it you would seen the path to the mount unit like /usr/lib/systemd/system/..mount.

You must have a script or cron or something mounting it. systemd is just reading the mount.


I don't have any cron jobs that run every 6 minutes. There are no CRON events in journalctl that coincide with it being mounted. I am a bit stumped as to what would be mounting this. I would have remembered writing a script that mounts a filesystem.
I copied some files from this snapshot as my unprivileged user on the 9th Oct, then I manually umounted this snapshot as root on the 10th Oct, presumably after noticing it was mounted. The journal doesn't go back that far but I assume that's when these events started.
Using lsof to see if anything is open in here seems to cause it to be mounted automatically. If that's the case then almost anything at all accessing that path could be causing this. Short of an access.log for filesystems, I don't know how I'll ever track this down!
troffasky
 
Posts: 25
Joined: 2017-11-05 11:00

Re: Tracking down cause of systemd mount events

Postby sgosnell » 2020-11-08 15:56

Have you rebooted the machine?
Take my advice, I'm not using it.
sgosnell
 
Posts: 906
Joined: 2011-03-14 01:49

Re: Tracking down cause of systemd mount events

Postby troffasky » 2020-11-08 18:28

Funny you should ask that...I did copy some files out of a snapshot on a different volume which isn't having these events now, but I have rebooted since then. So maybe a reboot will "fix" this.
troffasky
 
Posts: 25
Joined: 2017-11-05 11:00

Re: Tracking down cause of systemd mount events

Postby Head_on_a_Stick » 2020-11-10 18:02

troffasky wrote:I cannot find a timer that corresponds with this

Did you try
Code: Select all
systemctl --user list-timers --all
Black Lives Matter

Debian buster-backports ISO image: for new hardware support
User avatar
Head_on_a_Stick
 
Posts: 12795
Joined: 2014-06-01 17:46
Location: /dev/chair

Re: Tracking down cause of systemd mount events

Postby troffasky » 2020-11-12 20:52

Yes, it's in my first post.
troffasky
 
Posts: 25
Joined: 2017-11-05 11:00

Re: Tracking down cause of systemd mount events

Postby Head_on_a_Stick » 2020-11-14 17:43

No, it isn't. My command lists timers for your user but your OP only shows system timers.
Black Lives Matter

Debian buster-backports ISO image: for new hardware support
User avatar
Head_on_a_Stick
 
Posts: 12795
Joined: 2014-06-01 17:46
Location: /dev/chair

Re: Tracking down cause of systemd mount events

Postby troffasky » 2020-11-18 20:58

My user:
Code: Select all
$ systemctl --user list-timers --all
NEXT LEFT LAST PASSED UNIT ACTIVATES

0 timers listed.



Not sure if this is normal or not but here it is as root anyway:
Code: Select all
# systemctl --user list-timers --all
Failed to connect to bus: Operation not permitted
troffasky
 
Posts: 25
Joined: 2017-11-05 11:00


Return to System configuration

Who is online

Users browsing this forum: No registered users and 18 guests

fashionable