Scheduled Maintenance: We are aware of an issue with Google, AOL, and Yahoo services as email providers which are blocking new registrations. We are trying to fix the issue and we have several internal and external support tickets in process to resolve the issue. Please see: viewtopic.php?t=158230

 

 

 

[Software] What is using up my /dev/sda2 space?

Need help with peripherals or devices?
Post Reply
Message
Author
instantdreams
Posts: 3
Joined: 2023-01-27 00:00

[Software] What is using up my /dev/sda2 space?

#1 Post by instantdreams »

I have 4 servers running Debian or Debian equivalents:
  • 2 x Edge servers : Raspberry Pi 4b RAM 2GB SSD 120GB
  • 1 x Services server : Intel NUC7CJYHN RAM 16GB SSD 500GB
  • 1 x Security server : Intel NUC8i3BEH RAM 32GB SSD 250GB
These four servers run in headless mode with minimal installs, and the Docker Engine and Docker Compose plugin installed. They are used to manage a set of containers for my home network.
  • OS Version/build: Debian 11.5 bullseye (or Raspberry Pi OS equivalent)
  • App version: Docker version 20.10.22, build 3a2c30b, Docker Compose version v2.14.1
  • Exclusive use of docker-compose.yml files for managing containers
  • Configuration-as-code stored in private github repositories
  • Server specific information held in .env.example files, .env files created before container start
My issue is that one server has significantly less drive space available than the other three, and I can't work out why.

--

Edge Servers
The edge servers boot from a 120GB SSD, and run 9 Docker containers. They report having 91GB of the 110GB allocated free:
$ df --human-readable --local --total --print-type --exclude-type=tmpfs --exclude-type=devtmpfs | (sed --unbuffered 1q;sort --key 1)
Filesystem     Type  Size  Used Avail Use% Mounted on
/dev/root      ext4  110G   15G   91G  14% /
/dev/sda1      vfat  253M   50M  203M  20% /boot
total          -     111G   15G   91G  14% -

Services Server
My services server runs from a 500GB SSD and runs 31 Docker containers. It reports having 362GB available from the 456GB allocated:
$ df --human-readable --local --total --print-type --exclude-type=tmpfs --exclude-type=devtmpfs | (sed --unbuffered 1q;sort --key 1)
Filesystem     Type  Size  Used Avail Use% Mounted on
/dev/sda1      vfat  511M  5.8M  506M   2% /boot/efi
/dev/sda2      ext4  456G   72G  362G  17% /
total          -     457G   72G  362G  17% -

Security Server
The security server, which runs various NVR containers (and 6 containers in total), has a 250GB SSD with ZERO space available:
$ df --human-readable --local --total --print-type --exclude-type=tmpfs --exclude-type=devtmpfs --exclude-type=zfs | (sed --unbuffered
 1q;sort --key 1)
Filesystem     Type  Size  Used Avail Use% Mounted on
/dev/sda1      vfat  511M  5.8M  506M   2% /boot/efi
/dev/sda2      ext4  227G  218G     0 100% /
total          -     228G  218G  506M 100% -

I've looked at the directories in the root file system and created a list that appear to be on the /dev/sda2 device. Using this list, I've pulled the disk usage for each server.

Edge Servers
This looks like roughly 19GB, which matches the disk free reports:
$ chk_dirs=("boot" "dev" "etc" "home" "lost+found" "mnt" "opt" "root" "run" "srv" "sys" "tmp" "usr" "var");for chk_dir in ${chk_dirs[@]};do sudo du --summarize --human-readable /$chk_dir;done
50M     /boot
0       /dev
3.8M    /etc
300K    /home
16K     /lost+found
4.0K    /mnt
12M     /opt
4.8M    /root
3.2M    /run
3.2G    /srv
0       /sys
40K     /tmp
1.8G    /usr
9.8G    /var

Services Server
This seems to be around 61GB, which also matches the disk free reports:
$ chk_dirs=("boot" "dev" "etc" "home" "lost+found" "mnt" "opt" "root" "run" "srv" "sys" "tmp" "usr" "var");for chk_dir in ${chk_dirs[@]};do sudo du --summarize --human-readable /$chk_dir;done
92M     /boot
0       /dev
3.8M    /etc
192K    /home
16K     /lost+found
4.0K    /mnt
16K     /opt
48K     /root
4.9M    /run
22G     /srv
0       /sys
196K    /tmp
2.1G    /usr
34G     /var


Security Server
Roughly around 12GB of space used, but no space left at all:
$ chk_dirs=("boot" "dev" "etc" "home" "lost+found" "mnt" "opt" "root" "run" "srv" "sys" "tmp" "usr" "var");for chk_dir in ${chk_dirs[@]};do sudo du --summarize --human-readable /$chk_dir;done
54M     /boot
0       /dev
4.3M    /etc
192K    /home
16K     /lost+found
4.0K    /mnt
16K     /opt
36K     /root
3.1M    /run
621M    /srv
0       /sys
48K     /tmp
2.0G    /usr
6.5G    /var

What commands or process could I try to find where the space is being allocated?

I've already taken down every running container and performed the following:
$ docker system prune --all --force
$ docker image prune --all --force
$ docker container prune --force
$ docker network prune --force
$ docker volume prune --force
$ sudo find /var/lib/docker/containers/ -type f -name "*.log" -delete

I am not sure where else to look, so any guidance is very appreciated. Thank you!

p.H
Global Moderator
Global Moderator
Posts: 3049
Joined: 2017-09-17 07:12
Has thanked: 5 times
Been thanked: 132 times

Re: [Software] What is using up my /dev/sda2 space?

#2 Post by p.H »

"Lost space" may be hidden under a mount

Code: Select all

mount --bind / /mnt
du -hxd1 /mnt | sort -h
or deleted still opened files (reboot or check with lsof)
or filesystem inconsistencies (check with e2fsck).
Last edited by p.H on 2023-01-27 18:40, edited 1 time in total.

instantdreams
Posts: 3
Joined: 2023-01-27 00:00

Re: [Software] What is using up my /dev/sda2 space?

#3 Post by instantdreams »

Here's the result:

Code: Select all

$ sudo mount --bind / /mnt
 sudo du -hxd1 --total /mnt | sort -n
2.0G    /mnt/usr
3.6G    /mnt/var
4.0K    /mnt/jbod
4.0K    /mnt/mnt
4.0K    /mnt/proc
4.0K    /mnt/run
4.0K    /mnt/sys
4.3M    /mnt/etc
8.0K    /mnt/media
12K     /mnt/dev
16K     /mnt/lost+found
16K     /mnt/opt
36K     /mnt/root
48K     /mnt/tmp
49M     /mnt/boot
192K    /mnt/home
212G    /mnt/data
218G    /mnt
218G    total
621M    /mnt/srv
This is not what I expected, because the /data directory is mounted from a ZFS pool called jbod.

Code: Select all

$ sudo du -hxd1 /jbod | sort -n
512     /jbod

$ sudo du -hxd1 /data | sort -n
4.0K    /data

$ sudo du -hxd1 --total /data/backups | sort -n
898G    /data/backups/[a bunch of stuff]
898G    /data/backups
898G    total

$ sudo du -hxd1 --total /data/security | sort -n
11M     /data/security/kerberos
20G     /data/security/ispyagentdvr
72G     /data/security/frigate
531K    /data/security/wyze-bridge
913G    /data/security/teslacam
1004G   /data/security
1004G   total

$ zpool list
NAME   SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
jbod  7.25T  1.86T  5.39T        -         -     1%    25%  1.00x    ONLINE  -

$ zfs list
NAME            USED  AVAIL     REFER  MOUNTPOINT
jbod           1.86T  5.27T       96K  /jbod
jbod/backups    899G  5.27T      899G  /data/backups
jbod/security  1004G  5.27T     1004G  /data/security
So why is data showing as using 212G in /mnt/data?

instantdreams
Posts: 3
Joined: 2023-01-27 00:00

Re: [Software] What is using up my /dev/sda2 space?

#4 Post by instantdreams »

I think we can chalk this up to me being a novice.

Some of the data written to /data/security and /data/backups was written to /dev/sda2 rather than the ZFS mount. I have fixed this:

Code: Select all

$ sudo zfs set mountpoint=/jbod/security jbod/security
$ sudo zfs set mountpoint=/jbod/backups jbod/backups
$ sudo rm -r /data/security
$ sudo rm -r /data/backups
I then adjusted my environment variables and share mounts, and rebooted. Things are looking good now.

Thank you @p.H for pointing me in the right direction!

p.H
Global Moderator
Global Moderator
Posts: 3049
Joined: 2017-09-17 07:12
Has thanked: 5 times
Been thanked: 132 times

Re: [Software] What is using up my /dev/sda2 space?

#5 Post by p.H »

instantdreams wrote: 2023-01-27 15:55 212G /mnt/data
This is not what I expected, because the /data directory is mounted from a ZFS pool
This is totally what I expected when files are hidden by a mount.
instantdreams wrote: 2023-01-27 15:55 So why is data showing as using 212G in /mnt/data?
Because those 212 GiB were written into /data while the ZFS filesystems were not mounted on /data for some reason.

PS: I made a typo in my previous post, use -h for proper sorting instead of -n.

Post Reply