Systemdisk/OS disk almost full

[SonOfThor]

I have a problem to locate what is filling up my space on my OS disk (rootfs if this is the more proper terminology). I am no Linux expert, frankly a novice in Linux.
I believe that the problem started with a a container in Docker not properly set up - hence it started to store files to the OS disk not to the proper mount point - the disk filled up almost 100%.
I deleted the file(s) and thought everything was ok. This did not free up space. I decided to uninstall all containers and included also Docker. This did not solve the problem with the space.
To ensure I have enough space I have re-sized the partition. Now I have more space, but the "ghost" files are still stealing my space on the disk.
My rootfs is on an SSD disk.
I am using OpenMediaVault (OMV), based on Debian. I have addressed the issue in the OMV forum - but so far not being able to solve the storage issue. As a consequence I am bringing the issue here to ask if anyone would be so kind as to point me what to try in order to figure this out. I have spent a few weeks searching google and various forums, and have almost given up.

This is what I have done so far:

Code: Select all

root@OMV:/# df -h /
Filesystem      Size  Used Avail Use% Mounted on
/dev/sdg1        91G   49G   38G  57% /

From df it seems I am using 49 GB of disk space. This is also reported to OMV GUI.

Code: Select all

root@OMV:/# du -xh --max-depth=1
3.7G    ./var
4.0K    ./lib64
5.4G    ./home
8.0K    ./media
936M    ./usr
4.0K    ./sharedfolders
7.3M    ./etc
4.0K    ./mnt
40K     ./root
13M     ./bin
4.0K    ./opt
4.0K    ./export
15M     ./sbin
12K     ./srv
950M    ./lib
16K     ./lost+found
131M    ./boot
12G     .

I am OK with the ~ 12 GB size on the drive, but df tells me that it is 49GB used on the drive ...?

Code: Select all

 root@OMV:/# ncdu -x

5.4 GiB [##########] /home
    3.6 GiB [######    ] /var
  949.5 MiB [#         ] /lib
  935.1 MiB [#         ] /usr
  130.7 MiB [          ] /boot
   14.0 MiB [          ] /sbin
   12.6 MiB [          ] /bin
    7.3 MiB [          ] /etc
   40.0 KiB [          ] /root
e  16.0 KiB [          ] /lost+found
   12.0 KiB [          ] /srv
    8.0 KiB [          ] /media
    4.0 KiB [          ] /lib64
    4.0 KiB [          ] /sharedfolders
e   4.0 KiB [          ] /opt
e   4.0 KiB [          ] /mnt
e   4.0 KiB [          ] /export
@   0.0   B [          ]  initrd.img.old
@   0.0   B [          ]  initrd.img
@   0.0   B [          ]  vmlinuz.old
@   0.0   B [          ]  vmlinuz
>   0.0   B [          ] /tmp
>   0.0   B [          ] /sys
>   0.0   B [          ] /run
>   0.0   B [          ] /proc
>   0.0   B [          ] /dev

Checked also with ncdu which gives me the "same" picture as that of du.

Code: Select all

root@OMV:/# lsof +L1
COMMAND     PID                  USER   FD   TYPE DEVICE SIZE/OFF NLINK  NODE NAME
php-fpm7.  1412                  root    3u   REG   0,40        0     0 35254 /tmp/.ZendSem.e6bLh3 (deleted)
php-fpm7.  6530              www-data    3u   REG   0,40        0     0 35254 /tmp/.ZendSem.e6bLh3 (deleted)
php-fpm7.  6561              www-data    3u   REG   0,40        0     0 35254 /tmp/.ZendSem.e6bLh3 (deleted)
php-fpm7. 24239 openmediavault-webgui    3u   REG   0,40        0     0 35254 /tmp/.ZendSem.e6bLh3 (deleted)

Thought maybe that there are some deleted files that could take up space. But it seems that all these deleted files have a size of 0? I have tried to kill these processes e.g.

Code: Select all

kill -9 1412

But this did not help, and they come back up after a short time, with a new PID.

Any experts here that can point in the right direction? What can possibly take up space on my disk?

[Hallvor]

Try installing xdiskusage. It is way easier than digging through the CLI for a novice.

[p.H]

I have a problem to locate what is filling up my space on my OS disk (rootfs if this is the more proper terminology)

No, rootfs is something different. See https://www.kernel.org/doc/Documentatio ... tramfs.txt.

I am OK with the ~ 12 GB size on the drive, but df tells me that it is 49GB used on the drive ...?

It could be
- deleted files which are still opened by a process. Did you reboot after deleting the files ?
- files "hidden" under a mounted filesystem

[pylkko]

Please describe in detail how you removed the docker images.

[SonOfThor]

Try installing xdiskusage. It is way easier than digging through the CLI for a novice.

I tried installing it with apt-get, but that did not work. I will have another look at it again when I find the time.

[SonOfThor]

Please describe in detail how you removed the docker images.

I just used the functionality in the GUI of OpenMediaVault. Deleted the containers using "delete" in GUI, then checked the openmediavault-docker-gui 4.1and selected uninstall in the plugin section of the GUI. The server has been restarted a number of times after that - and before the re-installation of the Docker plugin. Do you think that the Docker service is still addressing some space on the disk? If so, would that not show up using LSOF? How would you suggest to identify this further?

[SonOfThor]

It could be
- deleted files which are still opened by a process. Did you reboot after deleting the files ?
- files "hidden" under a mounted filesystem

Yes, it seems likely. I did reboot after deleting the files, and have done a number of reboots after the problem started.
Do you have any suggestions as to show "hidden" files under a mounted filesystem?

[Head_on_a_Stick]

Which filesystem is in use?

[SonOfThor]

Which filesystem is in use?

I am using ext4 as filesystem.

[p.H]

Do you have any suggestions as to show "hidden" files under a mounted filesystem?

"Bind mount" the / filesystem on a temporary mount point.

Code: Select all

mount --bind / /mnt
du -h --max-depth=1 /mnt

[SonOfThor]

Do you have any suggestions as to show "hidden" files under a mounted filesystem?

"Bind mount" the / filesystem on a temporary mount point.

Code: Select all

mount --bind / /mnt
du -h --max-depth=1 /mnt

Wow, that was useful. Doing that I found a folder with files and folders that does not appear when I do a normal

Code: Select all

du -h --max-depth=1 /

What do I do now to remove the "hidden" files and folders on my /? If I remove them from the mount bind it is not removed from the / right? Since I am a novice, I want to be careful here ... so please let me know what to do next.

[SonOfThor]

I was adventurous and tried a bit. It turns out that if you delete files/folders on the mount bind it also deletes the files on / after it has been umounted
So I have now managed to get this fixed!

Thanks a lot p.H! You have saved me for a lot of grey hair!