Systemdisk/OS disk almost full

Kernels & Hardware, configuring network, installing services

Systemdisk/OS disk almost full

Postby SonOfThor » 2018-11-11 17:41

I have a problem to locate what is filling up my space on my OS disk (rootfs if this is the more proper terminology). I am no Linux expert, frankly a novice in Linux.
I believe that the problem started with a a container in Docker not properly set up - hence it started to store files to the OS disk not to the proper mount point - the disk filled up almost 100%.
I deleted the file(s) and thought everything was ok. This did not free up space. I decided to uninstall all containers and included also Docker. This did not solve the problem with the space.
To ensure I have enough space I have re-sized the partition. Now I have more space, but the "ghost" files are still stealing my space on the disk.
My rootfs is on an SSD disk.
I am using OpenMediaVault (OMV), based on Debian. I have addressed the issue in the OMV forum - but so far not being able to solve the storage issue. As a consequence I am bringing the issue here to ask if anyone would be so kind as to point me what to try in order to figure this out. I have spent a few weeks searching google and various forums, and have almost given up.

This is what I have done so far:

Code: Select all
root@OMV:/# df -h /
Filesystem      Size  Used Avail Use% Mounted on
/dev/sdg1        91G   49G   38G  57% /


From df it seems I am using 49 GB of disk space. This is also reported to OMV GUI.

Code: Select all
root@OMV:/# du -xh --max-depth=1
3.7G    ./var
4.0K    ./lib64
5.4G    ./home
8.0K    ./media
936M    ./usr
4.0K    ./sharedfolders
7.3M    ./etc
4.0K    ./mnt
40K     ./root
13M     ./bin
4.0K    ./opt
4.0K    ./export
15M     ./sbin
12K     ./srv
950M    ./lib
16K     ./lost+found
131M    ./boot
12G     .


I am OK with the ~ 12 GB size on the drive, but df tells me that it is 49GB used on the drive ...?

Code: Select all
 root@OMV:/# ncdu -x

5.4 GiB [##########] /home
    3.6 GiB [######    ] /var
  949.5 MiB [#         ] /lib
  935.1 MiB [#         ] /usr
  130.7 MiB [          ] /boot
   14.0 MiB [          ] /sbin
   12.6 MiB [          ] /bin
    7.3 MiB [          ] /etc
   40.0 KiB [          ] /root
e  16.0 KiB [          ] /lost+found
   12.0 KiB [          ] /srv
    8.0 KiB [          ] /media
    4.0 KiB [          ] /lib64
    4.0 KiB [          ] /sharedfolders
e   4.0 KiB [          ] /opt
e   4.0 KiB [          ] /mnt
e   4.0 KiB [          ] /export
@   0.0   B [          ]  initrd.img.old
@   0.0   B [          ]  initrd.img
@   0.0   B [          ]  vmlinuz.old
@   0.0   B [          ]  vmlinuz
>   0.0   B [          ] /tmp
>   0.0   B [          ] /sys
>   0.0   B [          ] /run
>   0.0   B [          ] /proc
>   0.0   B [          ] /dev

Checked also with ncdu which gives me the "same" picture as that of du.


Code: Select all
root@OMV:/# lsof +L1
COMMAND     PID                  USER   FD   TYPE DEVICE SIZE/OFF NLINK  NODE NAME
php-fpm7.  1412                  root    3u   REG   0,40        0     0 35254 /tmp/.ZendSem.e6bLh3 (deleted)
php-fpm7.  6530              www-data    3u   REG   0,40        0     0 35254 /tmp/.ZendSem.e6bLh3 (deleted)
php-fpm7.  6561              www-data    3u   REG   0,40        0     0 35254 /tmp/.ZendSem.e6bLh3 (deleted)
php-fpm7. 24239 openmediavault-webgui    3u   REG   0,40        0     0 35254 /tmp/.ZendSem.e6bLh3 (deleted)


Thought maybe that there are some deleted files that could take up space. But it seems that all these deleted files have a size of 0? I have tried to kill these processes e.g.

Code: Select all
kill -9 1412


But this did not help, and they come back up after a short time, with a new PID.

Any experts here that can point in the right direction? What can possibly take up space on my disk?
SonOfThor
 
Posts: 7
Joined: 2018-10-18 13:46

Re: Systemdisk/OS disk almost full

Postby Hallvor » 2018-11-11 20:23

Try installing xdiskusage. It is way easier than digging through the CLI for a novice.
HP Elitebook 2570p, Intel Core i5 3320-M CPU @ 2.60 GHz, 6 GB RAM, Intel HD 4000 graphics, 240 GB SSD, Debian Stretch (KDE)
User avatar
Hallvor
 
Posts: 847
Joined: 2009-04-16 18:35
Location: Norway

Re: Systemdisk/OS disk almost full

Postby p.H » 2018-11-11 20:37

SonOfThor wrote:I have a problem to locate what is filling up my space on my OS disk (rootfs if this is the more proper terminology)

No, rootfs is something different. See https://www.kernel.org/doc/Documentation/filesystems/ramfs-rootfs-initramfs.txt.

SonOfThor wrote:I am OK with the ~ 12 GB size on the drive, but df tells me that it is 49GB used on the drive ...?

It could be
- deleted files which are still opened by a process. Did you reboot after deleting the files ?
- files "hidden" under a mounted filesystem
p.H
 
Posts: 645
Joined: 2017-09-17 07:12

Re: Systemdisk/OS disk almost full

Postby pylkko » 2018-11-11 21:23

Please describe in detail how you removed the docker images.
User avatar
pylkko
 
Posts: 1357
Joined: 2014-11-06 19:02

Re: Systemdisk/OS disk almost full

Postby SonOfThor » 2018-11-13 06:00

Hallvor wrote:Try installing xdiskusage. It is way easier than digging through the CLI for a novice.


I tried installing it with apt-get, but that did not work. I will have another look at it again when I find the time.
SonOfThor
 
Posts: 7
Joined: 2018-10-18 13:46

Re: Systemdisk/OS disk almost full

Postby SonOfThor » 2018-11-13 06:06

pylkko wrote:Please describe in detail how you removed the docker images.


I just used the functionality in the GUI of OpenMediaVault. Deleted the containers using "delete" in GUI, then checked the openmediavault-docker-gui 4.1and selected uninstall in the plugin section of the GUI. The server has been restarted a number of times after that - and before the re-installation of the Docker plugin. Do you think that the Docker service is still addressing some space on the disk? If so, would that not show up using LSOF? How would you suggest to identify this further?
SonOfThor
 
Posts: 7
Joined: 2018-10-18 13:46

Re: Systemdisk/OS disk almost full

Postby SonOfThor » 2018-11-13 06:08

p.H wrote:It could be
- deleted files which are still opened by a process. Did you reboot after deleting the files ?
- files "hidden" under a mounted filesystem


Yes, it seems likely. I did reboot after deleting the files, and have done a number of reboots after the problem started.
Do you have any suggestions as to show "hidden" files under a mounted filesystem?
SonOfThor
 
Posts: 7
Joined: 2018-10-18 13:46

Re: Systemdisk/OS disk almost full

Postby Head_on_a_Stick » 2018-11-13 06:19

Which filesystem is in use?
dbruce wrote:Ubuntu forums try to be like a coffee shop in Seattle. Debian forums strive for the charm and ambience of a skinhead bar in Bacau. We intend to keep it that way.
User avatar
Head_on_a_Stick
 
Posts: 8322
Joined: 2014-06-01 17:46
Location: /dev/chair

Re: Systemdisk/OS disk almost full

Postby SonOfThor » 2018-11-13 07:15

Head_on_a_Stick wrote:Which filesystem is in use?


I am using ext4 as filesystem.
SonOfThor
 
Posts: 7
Joined: 2018-10-18 13:46

Re: Systemdisk/OS disk almost full

Postby p.H » 2018-11-13 09:00

SonOfThor wrote:Do you have any suggestions as to show "hidden" files under a mounted filesystem?

"Bind mount" the / filesystem on a temporary mount point.
Code: Select all
mount --bind / /mnt
du -h --max-depth=1 /mnt
p.H
 
Posts: 645
Joined: 2017-09-17 07:12

Re: Systemdisk/OS disk almost full

Postby SonOfThor » 2018-11-13 18:54

p.H wrote:
SonOfThor wrote:Do you have any suggestions as to show "hidden" files under a mounted filesystem?

"Bind mount" the / filesystem on a temporary mount point.
Code: Select all
mount --bind / /mnt
du -h --max-depth=1 /mnt


Wow, that was useful. Doing that I found a folder with files and folders that does not appear when I do a normal
Code: Select all
du -h --max-depth=1 /


What do I do now to remove the "hidden" files and folders on my /? If I remove them from the mount bind it is not removed from the / right? Since I am a novice, I want to be careful here ... so please let me know what to do next. :)
SonOfThor
 
Posts: 7
Joined: 2018-10-18 13:46

Re: Systemdisk/OS disk almost full

Postby SonOfThor » 2018-11-13 19:23

I was adventurous and tried a bit. It turns out that if you delete files/folders on the mount bind it also deletes the files on / after it has been umounted
So I have now managed to get this fixed!

Thanks a lot p.H! You have saved me for a lot of grey hair!
SonOfThor
 
Posts: 7
Joined: 2018-10-18 13:46


Return to System configuration

Who is online

Users browsing this forum: No registered users and 6 guests

fashionable