ext4 root filesystem full for unknown reason

[yunchih]

The ext4 root filesystem is full, but ncdu didn't show any large file.

Code: Select all

$ df  /
Filesystem          1K-blocks     Used Available Use% Mounted on
/dev/mapper/lv-root  40684008 40455320         0 100% /
$ df -h /
Filesystem           Size  Used Avail Use% Mounted on
/dev/mapper/lv-root   39G   39G     0 100% /
$ ncdu -x /
  549.6 MiB [##########] /usr                                                                                                                                                                                      
  489.0 MiB [########  ] /lib
   65.1 MiB [#         ] /boot
   16.9 MiB [          ] /root
   10.5 MiB [          ] /sbin
    9.3 MiB [          ] /bin
    5.1 MiB [          ] /etc
  136.0 KiB [          ] /tmp
e  16.0 KiB [          ] /lost+found
    8.0 KiB [          ] /media
    4.0 KiB [          ] /lib64
e   4.0 KiB [          ] /srv
e   4.0 KiB [          ] /opt
e   4.0 KiB [          ] /mnt
    4.0 KiB [          ] /e
@   0.0   B [          ]  initrd.img
@   0.0   B [          ]  vmlinuz
@   0.0   B [          ]  initrd.img.old
@   0.0   B [          ]  vmlinuz.old
>   0.0   B [          ] /var
>   0.0   B [          ] /sys
>   0.0   B [          ] /run
>   0.0   B [          ] /proc
>   0.0   B [          ] /home
>   0.0   B [          ] /dev

It's possible that some large files are unlinked but still opened by some processes, thus taking space.
But lsof didn't show any large file:

Code: Select all

$ sudo lsof -s | egrep 'REG|DIR' | sort -k8 -n | uniq | numfmt --field=8 --to=iec --invalid=ignore | tail -n 10
rpc.mount 18893                  root    6u      REG                0,4        0       3.8G /proc/18880/net/rpc/nfsd.fh/channel
rpc.mount 18894                  root    6u      REG                0,4        0       3.8G /proc/18880/net/rpc/nfsd.fh/channel
rpc.mount 18895                  root    6u      REG                0,4        0       3.8G /proc/18880/net/rpc/nfsd.fh/channel
rpc.mount 18896                  root    6u      REG                0,4        0       3.8G /proc/18880/net/rpc/nfsd.fh/channel
rpc.mount 18897                  root    6u      REG                0,4        0       3.8G /proc/18880/net/rpc/nfsd.fh/channel
rpc.mount 18898                  root    6u      REG                0,4        0       3.8G /proc/18880/net/rpc/nfsd.fh/channel
rpc.mount 18899                  root    6u      REG                0,4        0       3.8G /proc/18880/net/rpc/nfsd.fh/channel
rpc.mount 18900                  root    6u      REG                0,4        0       3.8G /proc/18880/net/rpc/nfsd.fh/channel
rpc.idmap 18882                  root    8u      REG                0,4        0       3.8G /proc/18882/net/rpc/nfs4.idtoname/channel
rpc.idmap 18882                  root    7u      REG                0,4        0       3.8G /proc/18882/net/rpc/nfs4.nametoid/channel

This is an NFS server: it mounts numerous iscsi disks and exports them to around 20 clients.
It runs Debian 9.4 (stable) with backported 4.16 kernel.

Is there any other possible tool that I can use for more investigation? ex: debugfs?
If so, what kind of information aid debugging problem like this one?

Thanks in advanced, folks!

[p.H]

lsof didn't show any large file

I am not convinced that this output is relevant. The file size is field 7, not 8. Also, /proc contents does not take space on the / filesystem.
Shouldn't you search deleted files instead (grep "deleted") ?

Also, could there be files "hidden" under a mount ?

[yunchih]

Hi,
Thanks for helping.
Once I stopped autofs and unmount everything, I found a backup archive dumped to a directory
that was supposed to live on a NFS mount .... This is indeed "hidden under a mount"!