fscache service won't start

Kernels & Hardware, configuring network, installing services

fscache service won't start

Postby blazini36 » 2020-07-11 19:49

Not sure that I'll get any help on this one but .....why not

I have a NAS server I setup using OpenMediaVault 5 which is running Debian 10 kernel 5.6. I mount NFS shares on it and I'm trying to get the write performance up a bit. I just upgraded it to 10gbe but the data disks are all spinning rust except the OS drive and a Caching SSD I just added. Basically the 10gbe does me no good on my main share since it is running on a mergerfs file system. Basically that uses software to present all drives assigned to the share as a single drive/share, Because of the overhead I can't get any higher than 50mb/s with a "sync" mount and an "async" mount is not really stable, because it bursts to 1000mb/s, but sinks back to 50mb/s to catch up. Basically it doesn't help get the client out of the transfer any faster. OMV has no real caching mechanism so I'm trying a few things with Debian. I have tons of data on ext4 hdd's so I'm not really looking at these newer filesystems like zfs or btrfs or whatever.

FScache looks promising but I ran into an issue I can't really find an answer to. I've mostly followed this https://blog.frehi.be/2019/01/03/fs-cache-for-nfs-clients/ but I get an error trying to start the service.
Code: Select all
~# systemctl start cachefilesd
Job for cachefilesd.service failed because the control process exited with error code.
See "systemctl status cachefilesd.service" and "journalctl -xe" for details.

Code: Select all
# systemctl status cachefilesd.service
● cachefilesd.service - LSB: CacheFiles daemon
   Loaded: loaded (/etc/init.d/cachefilesd; generated)
   Active: failed (Result: exit-code) since Sat 2020-07-11 16:36:26 EDT; 18s ago
     Docs: man:systemd-sysv-generator(8)
  Process: 3034 ExecStart=/etc/init.d/cachefilesd start (code=exited, status=1/FAILURE)

Jul 11 16:36:26 openmediavault.local systemd[1]: Starting LSB: CacheFiles daemon...
Jul 11 16:36:26 openmediavault.local cachefilesd[3034]: Starting FilesCache daemon : cachefilesd
Jul 11 16:36:26 openmediavault.local cachefilesd[3034]: Warning: Fake start-stop-daemon called, doing nothing.
Jul 11 16:36:26 openmediavault.local cachefilesd[3034]:  failed!
Jul 11 16:36:26 openmediavault.local systemd[1]: cachefilesd.service: Control process exited, code=exited, status=1/FAILURE
Jul 11 16:36:26 openmediavault.local systemd[1]: cachefilesd.service: Failed with result 'exit-code'.
Jul 11 16:36:26 openmediavault.local systemd[1]: Failed to start LSB: CacheFiles daemon.

I don't see anything worth looking at in journalctl. The tutorial mentions a Debian bug that is supposed to be fixed by editing the /etc/cachefilesd.conf and commenting out this line:
Code: Select all
#secctx system_u:system_r:cachefiles_kernel_t:s0

But it's already commented out in the Debian package.

The server side fstab mount for the cache ssd is
Code: Select all
# fscache
UUID=a86f82b3-b4e8-49df-9162-532f7bb108d4   /var/cache/fscache                      ext4    rw,acl,user_xattr 0 2

I read elsewhere that user_xattr is important for this but I'm not sure it matters for getting the service to start. Any ideas?
Posts: 7
Joined: 2020-04-04 17:24

Re: fscache service won't start

Postby ruwolf » 2020-07-13 01:48

Did you have any problems during installation or upgrade?
When I try to google Warning: Fake start-stop-daemon called, doing nothing, it reccomends me:
Code: Select all
apt update --fix-missing

/sbin/start-stop-daemon usually belongs to dpkg package (paradoxically package for installation), so it could be broken by installation/update in my opinion.

There was posted solution here, but previous solution is recommended, because more issues can be broken...
User avatar
Posts: 429
Joined: 2008-02-18 05:04
Location: Slovakia, Banovce nad Bebravou, Matice slovenskej 1260/4-7

Re: fscache service won't start

Postby zenlord » 2020-07-15 14:35

Are you sure that it is not running?

I remember seeing a similar (but not identical) error related to cachefilesd. It turned out that manually starting the service would work, while start-at-boot would consistently fail.

I filed this bugreport - maybe the solution helps you too: https://bugs.debian.org/cgi-bin/bugrepo ... bug=946125

Posts: 77
Joined: 2009-06-17 15:23

Re: fscache service won't start

Postby blazini36 » 2020-07-19 01:42

So I wound up having to reinstall the OS on a different drive because I was having issues with running it from a USB flash drive. What I did was install the OS to the NVME and then shrink the OS partition to allow most of the NVME for a cache partition.

I can verify that it did not actually run before, but now for whatever reason I don't have the same error and it actually does start the service. Thing is that it doesn't actually do anything. Either I misunderstood the purpose of cachefsd or I set it up wrong.

What I did was modify the /etc/cachefilesd.conf on the server to use "dir <directory-inside-cache-partition>" rather than "dir /var/cache/fscache". Then I mount the NFS shares from the client with the "fsc" option but I don't see any difference, and I don't see the cache drive consuming any extra space as if it were actually being used. The expectation was that I could write from the client to the server, and data would hit the cache drive, then move out to the filesystem target (on the slow drive).
Posts: 7
Joined: 2020-04-04 17:24

Re: fscache service won't start

Postby zenlord » 2020-07-20 06:24

I was able to verify that the cache was being used (but not at my desktop right no):
Code: Select all
# cat /proc/fs/fscache/stats

If it's not caching, see if apparmor is not blocking access to that custom location you've set up.
Posts: 77
Joined: 2009-06-17 15:23

Re: fscache service won't start

Postby blazini36 » 2020-07-20 22:53

This suggests that nothing is happening
Code: Select all
# cat /proc/fs/fscache/stats
FS-Cache statistics
Cookies: idx=0 dat=0 spc=0
Objects: alc=0 nal=0 avl=0 ded=0
ChkAux : non=0 ok=0 upd=0 obs=0
Pages  : mrk=0 unc=0
Acquire: n=0 nul=0 noc=0 ok=0 nbf=0 oom=0
Lookups: n=0 neg=0 pos=0 crt=0 tmo=0
Invals : n=0 run=0
Updates: n=0 nul=0 run=0
Relinqs: n=0 nul=0 wcr=0 rtr=0
AttrChg: n=0 ok=0 nbf=0 oom=0 run=0
Allocs : n=0 ok=0 wt=0 nbf=0 int=0
Allocs : ops=0 owt=0 abt=0
Retrvls: n=0 ok=0 wt=0 nod=0 nbf=0 int=0 oom=0
Retrvls: ops=0 owt=0 abt=0
Stores : n=0 ok=0 agn=0 nbf=0 oom=0
Stores : ops=0 run=0 pgs=0 rxd=0 olm=0
VmScan : nos=0 gon=0 bsy=0 can=0 wt=0
Ops    : pend=0 run=0 enq=0 can=0 rej=0
Ops    : ini=0 dfr=0 rel=0 gc=0
CacheOp: alo=0 luo=0 luc=0 gro=0
CacheOp: inv=0 upo=0 dro=0 pto=0 atc=0 syn=0
CacheOp: rap=0 ras=0 alp=0 als=0 wrp=0 ucp=0 dsp=0
CacheEv: nsp=0 stl=0 rtr=0 cul=0

like at all lol

I assume that AppArmor isn't running because
Code: Select all
# sudo aa-status
sudo: aa-status: command not found

There are however apparmor directories like /etc/apparmor.d

How would I verify that fs-cache is actually able to use the cache directory?
Posts: 7
Joined: 2020-04-04 17:24

Re: fscache service won't start

Postby zenlord » 2020-07-22 07:48

To use aa-status, you have to install apparmor-utils (OOTH).

To verify whether apparmor is not blocking anything, you can quickly scan your journal for messages containing the word "DENIED". If you need to loosen the apparmor restrictions, create an override file and include the override file instead of editing the file that is supplied by the Debian package. There are manuals available how to do this correctly and how to then reload the apparmor configuration.
Posts: 77
Joined: 2009-06-17 15:23

Re: fscache service won't start

Postby blazini36 » 2020-07-26 01:16

I installed apparmor-utils and aa-status shows nothing is being blocked. I can't find anything in journalctl either.

I did some more looking into it and it appears I do have the usage of FS-cache misunderstood. FS-cache is for client side caching, I have it installed on the Server. I don't have a use for client side caching, my local disks and network speed are much faster than what the server can ingest to HDDs, espescially the ones mounted behind the mergerFS share.

Anyone have a suggestion for server side caching? As said above, I'd like to figure out some means of writing to the server quickly and have the server's cache handle the delay of writing to permanent storage. I played with tempfs a bit before cachefs but couldn't get anywhere.
Posts: 7
Joined: 2020-04-04 17:24

Return to System configuration

Who is online

Users browsing this forum: No registered users and 22 guests