- 2 x Edge servers : Raspberry Pi 4b RAM 2GB SSD 120GB
- 1 x Services server : Intel NUC7CJYHN RAM 16GB SSD 500GB
- 1 x Security server : Intel NUC8i3BEH RAM 32GB SSD 250GB
- OS Version/build: Debian 11.5 bullseye (or Raspberry Pi OS equivalent)
- App version: Docker version 20.10.22, build 3a2c30b, Docker Compose version v2.14.1
- Exclusive use of docker-compose.yml files for managing containers
- Configuration-as-code stored in private github repositories
- Server specific information held in .env.example files, .env files created before container start
--
Edge Servers
The edge servers boot from a 120GB SSD, and run 9 Docker containers. They report having 91GB of the 110GB allocated free:
$ df --human-readable --local --total --print-type --exclude-type=tmpfs --exclude-type=devtmpfs | (sed --unbuffered 1q;sort --key 1) Filesystem Type Size Used Avail Use% Mounted on /dev/root ext4 110G 15G 91G 14% / /dev/sda1 vfat 253M 50M 203M 20% /boot total - 111G 15G 91G 14% -
Services Server
My services server runs from a 500GB SSD and runs 31 Docker containers. It reports having 362GB available from the 456GB allocated:
$ df --human-readable --local --total --print-type --exclude-type=tmpfs --exclude-type=devtmpfs | (sed --unbuffered 1q;sort --key 1) Filesystem Type Size Used Avail Use% Mounted on /dev/sda1 vfat 511M 5.8M 506M 2% /boot/efi /dev/sda2 ext4 456G 72G 362G 17% / total - 457G 72G 362G 17% -
Security Server
The security server, which runs various NVR containers (and 6 containers in total), has a 250GB SSD with ZERO space available:
$ df --human-readable --local --total --print-type --exclude-type=tmpfs --exclude-type=devtmpfs --exclude-type=zfs | (sed --unbuffered 1q;sort --key 1) Filesystem Type Size Used Avail Use% Mounted on /dev/sda1 vfat 511M 5.8M 506M 2% /boot/efi /dev/sda2 ext4 227G 218G 0 100% / total - 228G 218G 506M 100% -
I've looked at the directories in the root file system and created a list that appear to be on the /dev/sda2 device. Using this list, I've pulled the disk usage for each server.
Edge Servers
This looks like roughly 19GB, which matches the disk free reports:
$ chk_dirs=("boot" "dev" "etc" "home" "lost+found" "mnt" "opt" "root" "run" "srv" "sys" "tmp" "usr" "var");for chk_dir in ${chk_dirs[@]};do sudo du --summarize --human-readable /$chk_dir;done 50M /boot 0 /dev 3.8M /etc 300K /home 16K /lost+found 4.0K /mnt 12M /opt 4.8M /root 3.2M /run 3.2G /srv 0 /sys 40K /tmp 1.8G /usr 9.8G /var
Services Server
This seems to be around 61GB, which also matches the disk free reports:
$ chk_dirs=("boot" "dev" "etc" "home" "lost+found" "mnt" "opt" "root" "run" "srv" "sys" "tmp" "usr" "var");for chk_dir in ${chk_dirs[@]};do sudo du --summarize --human-readable /$chk_dir;done 92M /boot 0 /dev 3.8M /etc 192K /home 16K /lost+found 4.0K /mnt 16K /opt 48K /root 4.9M /run 22G /srv 0 /sys 196K /tmp 2.1G /usr 34G /var
Security Server
Roughly around 12GB of space used, but no space left at all:
$ chk_dirs=("boot" "dev" "etc" "home" "lost+found" "mnt" "opt" "root" "run" "srv" "sys" "tmp" "usr" "var");for chk_dir in ${chk_dirs[@]};do sudo du --summarize --human-readable /$chk_dir;done 54M /boot 0 /dev 4.3M /etc 192K /home 16K /lost+found 4.0K /mnt 16K /opt 36K /root 3.1M /run 621M /srv 0 /sys 48K /tmp 2.0G /usr 6.5G /var
What commands or process could I try to find where the space is being allocated?
I've already taken down every running container and performed the following:
$ docker system prune --all --force $ docker image prune --all --force $ docker container prune --force $ docker network prune --force $ docker volume prune --force $ sudo find /var/lib/docker/containers/ -type f -name "*.log" -delete
I am not sure where else to look, so any guidance is very appreciated. Thank you!