Scheduled Maintenance: We are aware of an issue with Google, AOL, and Yahoo services as email providers which are blocking new registrations. We are trying to fix the issue and we have several internal and external support tickets in process to resolve the issue. Please see: viewtopic.php?t=158230

 

 

 

Tracking down cause of systemd mount events

Linux Kernel, Network, and Services configuration.
Post Reply
Message
Author
troffasky
Posts: 37
Joined: 2017-11-05 11:00
Has thanked: 1 time
Been thanked: 1 time

Tracking down cause of systemd mount events

#1 Post by troffasky »

These events are logged in the journal, every 6 minutes:

Code: Select all

Nov 07 09:11:21 westogre systemd[1]: home-.zfs-snapshot-autosnap_2020\x2d08\x2d23_22:08:01_monthly.mount: Succeeded.
Nov 07 09:17:21 westogre systemd[1]: home-.zfs-snapshot-autosnap_2020\x2d08\x2d23_22:08:01_monthly.mount: Succeeded.
Nov 07 09:17:21 westogre systemd[92140]: home-.zfs-snapshot-autosnap_2020\x2d08\x2d23_22:08:01_monthly.mount: Succeeded.
Nov 07 09:23:21 westogre systemd[92140]: home-.zfs-snapshot-autosnap_2020\x2d08\x2d23_22:08:01_monthly.mount: Succeeded.
Nov 07 09:23:21 westogre systemd[1]: home-.zfs-snapshot-autosnap_2020\x2d08\x2d23_22:08:01_monthly.mount: Succeeded.
Nov 07 09:29:22 westogre systemd[1]: home-.zfs-snapshot-autosnap_2020\x2d08\x2d23_22:08:01_monthly.mount: Succeeded.
Nov 07 09:29:22 westogre systemd[92140]: home-.zfs-snapshot-autosnap_2020\x2d08\x2d23_22:08:01_monthly.mount: Succeeded.
Nov 07 09:35:22 westogre systemd[1]: home-.zfs-snapshot-autosnap_2020\x2d08\x2d23_22:08:01_monthly.mount: Succeeded.
Nov 07 09:35:22 westogre systemd[92140]: home-.zfs-snapshot-autosnap_2020\x2d08\x2d23_22:08:01_monthly.mount: Succeeded.
That snapshot is the oldest snapshot on my /home filesystem and it was created by sanoid. It's not currently mounted.
I don't think it's sanoid causing these .mount events, as sanoid isn't responsible for mounting snapshots, only creating and destroying them. I also have three other filesystems in this pool which sanoid is managing snapshots on, and which do not have any events like the above.
I cannot find a .mount file anywhere that corresponds with this.
I cannot find a timer that corresponds with this:

Code: Select all

 # systemctl list-timers --all
NEXT                        LEFT         LAST                        PASSED       UNIT                         ACTIVATES
Sat 2020-11-07 10:00:00 GMT 8min left    Sat 2020-11-07 09:45:02 GMT 6min ago     sanoid.timer                 sanoid.service
Sat 2020-11-07 10:09:00 GMT 17min left   Sat 2020-11-07 09:39:02 GMT 12min ago    phpsessionclean.timer        phpsessionclean.service
Sat 2020-11-07 10:33:15 GMT 41min left   Sat 2020-11-07 09:30:57 GMT 20min ago    anacron.timer                anacron.service
Sat 2020-11-07 11:00:32 GMT 1h 8min left Fri 2020-11-06 11:00:32 GMT 22h ago      systemd-tmpfiles-clean.timer systemd-tmpfiles-clean.service
Sat 2020-11-07 21:26:22 GMT 11h left     Sat 2020-11-07 07:02:29 GMT 2h 49min ago certbot.timer                certbot.service
Sat 2020-11-07 21:30:15 GMT 11h left     Sat 2020-11-07 09:28:02 GMT 23min ago    apt-daily.timer              apt-daily.service
Sun 2020-11-08 00:00:00 GMT 14h left     Sat 2020-11-07 00:00:02 GMT 9h ago       exim4-base.timer             exim4-base.service
Sun 2020-11-08 00:00:00 GMT 14h left     Sat 2020-11-07 00:00:02 GMT 9h ago       logrotate.timer              logrotate.service
Sun 2020-11-08 00:00:00 GMT 14h left     Sat 2020-11-07 00:00:02 GMT 9h ago       man-db.timer                 man-db.service
Sun 2020-11-08 03:10:18 GMT 17h left     Sun 2020-11-01 03:10:42 GMT 6 days ago   e2scrub_all.timer            e2scrub_all.service
Sun 2020-11-08 06:33:56 GMT 20h left     Sat 2020-11-07 06:49:02 GMT 3h 2min ago  apt-daily-upgrade.timer      apt-daily-upgrade.service
sanoid has a timer but it's every 15 minutes, not every 5.
There is no /etc/fstab entry for this snapshot.
Top
troffasky
Posts: 37
Joined: 2017-11-05 11:00
Has thanked: 1 time
Been thanked: 1 time

Re: Tracking down cause of systemd mount events

#2 Post by troffasky »

This is so weird, it actually *is* mounted, but I swear it wasn't showing up in mount or df before!
I actually found this with "systemctl | grep zfs" [I had been doing "systemctl status | grep zfs" and coming up with nothing....gahhhh]:

Code: Select all

# systemctl  | grep zfs
  home-.zfs-snapshot-autosnap_2020\x2d08\x2d23_22:08:01_monthly.mount                            loaded active mounted   /home/.zfs/snapshot/autosnap_2020-08-23_22:08:01_monthly              
  zfs-import-cache.service                                                                       loaded active exited    Import ZFS pools by cache file                                        
  zfs-load-module.service                                                                        loaded active exited    Install ZFS kernel module                                             
  zfs-mount.service                                                                              loaded active exited    Mount ZFS filesystems                                                 
  zfs-share.service                                                                              loaded active exited    ZFS file system shares                                                
  zfs-volume-wait.service                                                                        loaded active exited    Wait for ZFS Volume (zvol) links in /dev                              
  zfs-zed.service                                                                                loaded active running   ZFS Event Daemon (zed)                                                
  zfs-import.target                                                                              loaded active active    ZFS pool import target                                                
  zfs-volumes.target                                                                             loaded active active    ZFS volumes are ready                                                 
  zfs.target                                                                                     loaded active active    ZFS startup target
# systemctl  status  'home-.zfs-snapshot-autosnap_2020\x2d08\x2d23_22:08:01_monthly.mount'
* home-.zfs-snapshot-autosnap_2020\x2d08\x2d23_22:08:01_monthly.mount - /home/.zfs/snapshot/autosnap_2020-08-23_22:08:01_monthly
     Loaded: loaded (/proc/self/mountinfo)
     Active: active (mounted) since Sat 2020-11-07 10:06:09 GMT; 4min 42s ago
      Where: /home/.zfs/snapshot/autosnap_2020-08-23_22:08:01_monthly
       What: HomePool/homes@autosnap_2020-08-23_22:08:01_monthly
So that's a bit more info but I'm not really any closer to knowing *why* it's doing this. I guess I might have manually mounted that snapshot at some point in the past after setting up the automation of snapshots, just to confirm that it was working. But I don't want it mounted automatically, least of all every 6 minutes!
Top
troffasky
Posts: 37
Joined: 2017-11-05 11:00
Has thanked: 1 time
Been thanked: 1 time

Re: Tracking down cause of systemd mount events

#3 Post by troffasky »

'ls -al /home/.zfs/snapshots/' caused every snapshot to be mounted automatically. 6 minutes later, they were all unmounted again, apart from the original snapshot I referred to in this post [I have tracked this with 'while sleep 30; do date && mount | grep homes; done']
This is weird though - most of the time that snapshot is mounted, but occasionally, it's not. "Occasionally" is of course, every 6 minutes.

Code: Select all

Sat  7 Nov 10:46:39 GMT 2020
HomePool/homes on /home type zfs (rw,noatime,xattr,noacl)
Sat  7 Nov 10:47:09 GMT 2020
HomePool/homes on /home type zfs (rw,noatime,xattr,noacl)
HomePool/homes@autosnap_2020-08-23_22:08:01_monthly on /home/.zfs/snapshot/autosnap_2020-08-23_22:08:01_monthly type zfs (ro,relatime,xattr,noacl)
Sat  7 Nov 10:47:39 GMT 2020
HomePool/homes on /home type zfs (rw,noatime,xattr,noacl)
HomePool/homes@autosnap_2020-08-23_22:08:01_monthly on /home/.zfs/snapshot/autosnap_2020-08-23_22:08:01_monthly type zfs (ro,relatime,xattr,noacl)
Sat  7 Nov 10:48:09 GMT 2020
HomePool/homes on /home type zfs (rw,noatime,xattr,noacl)
HomePool/homes@autosnap_2020-08-23_22:08:01_monthly on /home/.zfs/snapshot/autosnap_2020-08-23_22:08:01_monthly type zfs (ro,relatime,xattr,noacl)
Sat  7 Nov 10:48:39 GMT 2020
HomePool/homes on /home type zfs (rw,noatime,xattr,noacl)
HomePool/homes@autosnap_2020-08-23_22:08:01_monthly on /home/.zfs/snapshot/autosnap_2020-08-23_22:08:01_monthly type zfs (ro,relatime,xattr,noacl)
Sat  7 Nov 10:49:09 GMT 2020
HomePool/homes on /home type zfs (rw,noatime,xattr,noacl)
HomePool/homes@autosnap_2020-08-23_22:08:01_monthly on /home/.zfs/snapshot/autosnap_2020-08-23_22:08:01_monthly type zfs (ro,relatime,xattr,noacl)
Sat  7 Nov 10:49:39 GMT 2020
HomePool/homes on /home type zfs (rw,noatime,xattr,noacl)
HomePool/homes@autosnap_2020-08-23_22:08:01_monthly on /home/.zfs/snapshot/autosnap_2020-08-23_22:08:01_monthly type zfs (ro,relatime,xattr,noacl)
Sat  7 Nov 10:50:09 GMT 2020
HomePool/homes on /home type zfs (rw,noatime,xattr,noacl)
HomePool/homes@autosnap_2020-08-23_22:08:01_monthly on /home/.zfs/snapshot/autosnap_2020-08-23_22:08:01_monthly type zfs (ro,relatime,xattr,noacl)
Sat  7 Nov 10:50:39 GMT 2020
HomePool/homes on /home type zfs (rw,noatime,xattr,noacl)
HomePool/homes@autosnap_2020-08-23_22:08:01_monthly on /home/.zfs/snapshot/autosnap_2020-08-23_22:08:01_monthly type zfs (ro,relatime,xattr,noacl)
Sat  7 Nov 10:51:09 GMT 2020
HomePool/homes on /home type zfs (rw,noatime,xattr,noacl)
HomePool/homes@autosnap_2020-08-23_22:08:01_monthly on /home/.zfs/snapshot/autosnap_2020-08-23_22:08:01_monthly type zfs (ro,relatime,xattr,noacl)
Sat  7 Nov 10:51:39 GMT 2020
HomePool/homes on /home type zfs (rw,noatime,xattr,noacl)
HomePool/homes@autosnap_2020-08-23_22:08:01_monthly on /home/.zfs/snapshot/autosnap_2020-08-23_22:08:01_monthly type zfs (ro,relatime,xattr,noacl)
Sat  7 Nov 10:52:09 GMT 2020
HomePool/homes on /home type zfs (rw,noatime,xattr,noacl)
HomePool/homes@autosnap_2020-08-23_22:08:01_monthly on /home/.zfs/snapshot/autosnap_2020-08-23_22:08:01_monthly type zfs (ro,relatime,xattr,noacl)
Sat  7 Nov 10:52:39 GMT 2020
HomePool/homes on /home type zfs (rw,noatime,xattr,noacl)
Sat  7 Nov 10:53:09 GMT 2020
HomePool/homes on /home type zfs (rw,noatime,xattr,noacl)
HomePool/homes@autosnap_2020-08-23_22:08:01_monthly on /home/.zfs/snapshot/autosnap_2020-08-23_22:08:01_monthly type zfs (ro,relatime,xattr,noacl)
Sat  7 Nov 10:53:39 GMT 2020
HomePool/homes on /home type zfs (rw,noatime,xattr,noacl)
HomePool/homes@autosnap_2020-08-23_22:08:01_monthly on /home/.zfs/snapshot/autosnap_2020-08-23_22:08:01_monthly type zfs (ro,relatime,xattr,noacl)
Sat  7 Nov 10:54:09 GMT 2020
HomePool/homes on /home type zfs (rw,noatime,xattr,noacl)
HomePool/homes@autosnap_2020-08-23_22:08:01_monthly on /home/.zfs/snapshot/autosnap_2020-08-23_22:08:01_monthly type zfs (ro,relatime,xattr,noacl)
Sat  7 Nov 10:54:39 GMT 2020
HomePool/homes on /home type zfs (rw,noatime,xattr,noacl)
HomePool/homes@autosnap_2020-08-23_22:08:01_monthly on /home/.zfs/snapshot/autosnap_2020-08-23_22:08:01_monthly type zfs (ro,relatime,xattr,noacl)
Sat  7 Nov 10:55:09 GMT 2020
HomePool/homes on /home type zfs (rw,noatime,xattr,noacl)
HomePool/homes@autosnap_2020-08-23_22:08:01_monthly on /home/.zfs/snapshot/autosnap_2020-08-23_22:08:01_monthly type zfs (ro,relatime,xattr,noacl)
Sat  7 Nov 10:55:39 GMT 2020
HomePool/homes on /home type zfs (rw,noatime,xattr,noacl)
HomePool/homes@autosnap_2020-08-23_22:08:01_monthly on /home/.zfs/snapshot/autosnap_2020-08-23_22:08:01_monthly type zfs (ro,relatime,xattr,noacl)
Sat  7 Nov 10:56:09 GMT 2020
HomePool/homes on /home type zfs (rw,noatime,xattr,noacl)
HomePool/homes@autosnap_2020-08-23_22:08:01_monthly on /home/.zfs/snapshot/autosnap_2020-08-23_22:08:01_monthly type zfs (ro,relatime,xattr,noacl)
Sat  7 Nov 10:56:39 GMT 2020
HomePool/homes on /home type zfs (rw,noatime,xattr,noacl)
HomePool/homes@autosnap_2020-08-23_22:08:01_monthly on /home/.zfs/snapshot/autosnap_2020-08-23_22:08:01_monthly type zfs (ro,relatime,xattr,noacl)
Sat  7 Nov 10:57:09 GMT 2020
HomePool/homes on /home type zfs (rw,noatime,xattr,noacl)
HomePool/homes@autosnap_2020-08-23_22:08:01_monthly on /home/.zfs/snapshot/autosnap_2020-08-23_22:08:01_monthly type zfs (ro,relatime,xattr,noacl)
Sat  7 Nov 10:57:39 GMT 2020
HomePool/homes on /home type zfs (rw,noatime,xattr,noacl)
HomePool/homes@autosnap_2020-08-23_22:08:01_monthly on /home/.zfs/snapshot/autosnap_2020-08-23_22:08:01_monthly type zfs (ro,relatime,xattr,noacl)
Sat  7 Nov 10:58:09 GMT 2020
HomePool/homes on /home type zfs (rw,noatime,xattr,noacl)
HomePool/homes@autosnap_2020-08-23_22:08:01_monthly on /home/.zfs/snapshot/autosnap_2020-08-23_22:08:01_monthly type zfs (ro,relatime,xattr,noacl)
Sat  7 Nov 10:58:39 GMT 2020
HomePool/homes on /home type zfs (rw,noatime,xattr,noacl)
Sat  7 Nov 10:59:09 GMT 2020
HomePool/homes on /home type zfs (rw,noatime,xattr,noacl)
HomePool/homes@autosnap_2020-08-23_22:08:01_monthly on /home/.zfs/snapshot/autosnap_2020-08-23_22:08:01_monthly type zfs (ro,relatime,xattr,noacl)
Top
User avatar
Head_on_a_Stick
Posts: 14114
Joined: 2014-06-01 17:46
Location: London, England
Has thanked: 81 times
Been thanked: 133 times

Re: Tracking down cause of systemd mount events

#4 Post by Head_on_a_Stick »

Thread moved to Offtopic section, Ubuntu is not supported here.
deadbang
Top
troffasky
Posts: 37
Joined: 2017-11-05 11:00
Has thanked: 1 time
Been thanked: 1 time

Re: Tracking down cause of systemd mount events

#5 Post by troffasky »

Uhhh, what? Ubuntu? This is *definitely* Debian.

Code: Select all

 # cat /etc/issue
Debian GNU/Linux bullseye/sid \n \l
Top
User avatar
sickpig
Posts: 589
Joined: 2019-01-23 10:34

Re: Tracking down cause of systemd mount events

#6 Post by sickpig »

Code: Select all

# systemctl  status  'home-.zfs-snapshot-autosnap_2020\x2d08\x2d23_22:08:01_monthly.mount'
* home-.zfs-snapshot-autosnap_2020\x2d08\x2d23_22:08:01_monthly.mount - /home/.zfs/snapshot/autosnap_2020-08-23_22:08:01_monthly
     Loaded: loaded (/proc/self/mountinfo)
If you check the Loaded.. line you will see that systemd is reading info from /proc, systemd is not mounting the snapshot. Had systemd mounted it you would seen the path to the mount unit like /usr/lib/systemd/system/..mount.

You must have a script or cron or something mounting it. systemd is just reading the mount.
Top
User avatar
Head_on_a_Stick
Posts: 14114
Joined: 2014-06-01 17:46
Location: London, England
Has thanked: 81 times
Been thanked: 133 times

Re: Tracking down cause of systemd mount events

#7 Post by Head_on_a_Stick »

troffasky wrote:Uhhh, what? Ubuntu? This is *definitely* Debian.
Oh, sorry, I must have missed the bit in your OP that specified the version. Thread moved back to the System Configuration section.

It would probably help if you explained exactly how you installed Debian — the developers actually respect the licences of the software (unlike Ubuntu) so it is not possible to install onto ZFS using the official ISO images.
deadbang
Top
troffasky
Posts: 37
Joined: 2017-11-05 11:00
Has thanked: 1 time
Been thanked: 1 time

Re: Tracking down cause of systemd mount events

#8 Post by troffasky »

sickpig wrote:

Code: Select all

# systemctl  status  'home-.zfs-snapshot-autosnap_2020\x2d08\x2d23_22:08:01_monthly.mount'
* home-.zfs-snapshot-autosnap_2020\x2d08\x2d23_22:08:01_monthly.mount - /home/.zfs/snapshot/autosnap_2020-08-23_22:08:01_monthly
     Loaded: loaded (/proc/self/mountinfo)
If you check the Loaded.. line you will see that systemd is reading info from /proc, systemd is not mounting the snapshot. Had systemd mounted it you would seen the path to the mount unit like /usr/lib/systemd/system/..mount.

You must have a script or cron or something mounting it. systemd is just reading the mount.
I don't have any cron jobs that run every 6 minutes. There are no CRON events in journalctl that coincide with it being mounted. I am a bit stumped as to what would be mounting this. I would have remembered writing a script that mounts a filesystem.
I copied some files from this snapshot as my unprivileged user on the 9th Oct, then I manually umounted this snapshot as root on the 10th Oct, presumably after noticing it was mounted. The journal doesn't go back that far but I assume that's when these events started.
Using lsof to see if anything is open in here seems to cause it to be mounted automatically. If that's the case then almost anything at all accessing that path could be causing this. Short of an access.log for filesystems, I don't know how I'll ever track this down!
Top
sgosnell
Posts: 975
Joined: 2011-03-14 01:49

Re: Tracking down cause of systemd mount events

#9 Post by sgosnell »

Have you rebooted the machine?
Take my advice, I'm not using it.
Top
troffasky
Posts: 37
Joined: 2017-11-05 11:00
Has thanked: 1 time
Been thanked: 1 time

Re: Tracking down cause of systemd mount events

#10 Post by troffasky »

Funny you should ask that...I did copy some files out of a snapshot on a different volume which isn't having these events now, but I have rebooted since then. So maybe a reboot will "fix" this.
Top
User avatar
Head_on_a_Stick
Posts: 14114
Joined: 2014-06-01 17:46
Location: London, England
Has thanked: 81 times
Been thanked: 133 times

Re: Tracking down cause of systemd mount events

#11 Post by Head_on_a_Stick »

troffasky wrote:I cannot find a timer that corresponds with this
Did you try

Code: Select all

systemctl --user list-timers --all
deadbang
Top
troffasky
Posts: 37
Joined: 2017-11-05 11:00
Has thanked: 1 time
Been thanked: 1 time

Re: Tracking down cause of systemd mount events

#12 Post by troffasky »

Yes, it's in my first post.
Top
User avatar
Head_on_a_Stick
Posts: 14114
Joined: 2014-06-01 17:46
Location: London, England
Has thanked: 81 times
Been thanked: 133 times

Re: Tracking down cause of systemd mount events

#13 Post by Head_on_a_Stick »

No, it isn't. My command lists timers for your user but your OP only shows system timers.
deadbang
Top
troffasky
Posts: 37
Joined: 2017-11-05 11:00
Has thanked: 1 time
Been thanked: 1 time

Re: Tracking down cause of systemd mount events

#14 Post by troffasky »

My user:

Code: Select all

$ systemctl --user list-timers --all
NEXT LEFT LAST PASSED UNIT ACTIVATES

0 timers listed.
Not sure if this is normal or not but here it is as root anyway:

Code: Select all

# systemctl --user list-timers --all
Failed to connect to bus: Operation not permitted
Top
Post Reply

Return to “System and Network configuration”