[SOLVED] BTRFS root filesystem full

If none of the more specific forums is the right place to ask

[SOLVED] BTRFS root filesystem full

Postby vanmaal » 2017-10-28 12:52

Hi!,

Not sure if general questions is the correct place, so I apologise in advance if not the case.

I have a headless server running Jessie and OpenMediaVault 3. Root filesystem is btrfs, and it has grown until it fills a whole 240 GB SSD. I'm absolutely uncapable to understand where the free space has gone. I have no snapshots, and although Docker has lots of subvolumes, all but a few space is eaten by the root subvolume. But the weird thing is btrfs fi du only shows 25,40 GB used.

I tried to find anything related to my problem but so far unsuccessfully, either on google, the Debian forums or wiki. Most solutions seem to rely on balance, defrag or other btrfs maintenance options, and although my diagnostics seem to show otherwise, I tried them. Unsuccesfully, of course.

I suppose I could try reformatting to ext4 and reinstalling all software, but I'd like to avoid it. In the worst case, I have a backup from a previous working state, but I'd like to avoid it, too, if possible.

Here follow some diagnostics:

btrfs fi sh:
Code: Select all
# btrfs fi sh
Label: none  uuid: 952a3691-b40c-4996-b408-317644ffe94f
        Total devices 1 FS bytes used 214.13GiB
        devid    1 size 215.57GiB used 215.56GiB path /dev/sda2

Label: 'nas2'  uuid: 3af75ca8-e28b-46e8-9afa-4095e3a67763
        Total devices 1 FS bytes used 15.37GiB
        devid    1 size 3.64TiB used 18.06GiB path /dev/mapper/vgnas-nas2lv

Label: 'backup'  uuid: 1da0c5f8-a9fc-4ee8-ab80-79ec3a83e359
        Total devices 2 FS bytes used 1.57TiB
        devid    1 size 3.64TiB used 1.57TiB path /dev/sdb1
        devid    2 size 3.64TiB used 1.57TiB path /dev/sdc1

Label: 'nas1'  uuid: fefaad79-68e5-447a-90de-e0334ff827e0
        Total devices 1 FS bytes used 4.59TiB
        devid    1 size 5.46TiB used 4.65TiB path /dev/mapper/vgnas-nas1lv


btrfs fi df / shows more or less the same thing:
Code: Select all
# btrfs fi df /
Data, single: total=213.53GiB, used=213.53GiB
System, single: total=32.00MiB, used=48.00KiB
Metadata, single: total=2.00GiB, used=609.06MiB
GlobalReserve, single: total=238.91MiB, used=0.00B


Or btrfs qgroup show -reF /:
Code: Select all
# btrfs qgroup show -reF /
qgroupid         rfer         excl     max_rfer     max_excl
--------         ----         ----     --------     --------
0/5         212.75GiB    212.75GiB         none         none


btrfs qgroup show /:
Code: Select all
# btrfs qgroup show /
qgroupid         rfer         excl
--------         ----         ----
0/5         212.75GiB    212.75GiB
0/260         4.33MiB    328.00KiB
0/261        24.48MiB      1.77MiB
0/262        24.48MiB      1.77MiB
0/263        25.22MiB      1.78MiB
0/264        25.22MiB      1.78MiB
0/273        25.22MiB      1.78MiB
0/274        25.23MiB      1.86MiB
0/276       120.39MiB      4.36MiB
0/277       120.39MiB      4.34MiB
0/278        99.22MiB      4.34MiB
0/279        99.22MiB      4.34MiB
0/280        99.22MiB      4.36MiB
0/281        99.22MiB      4.36MiB
0/282       127.50MiB      6.94MiB
0/283       127.50MiB      6.92MiB
0/284       391.58MiB      8.80MiB
0/286       391.58MiB      8.80MiB
0/292        16.00KiB     16.00KiB
0/293        16.00KiB     16.00KiB
0/294       391.59MiB      8.83MiB
0/295       391.64MiB      9.42MiB
0/304         4.32MiB    312.00KiB
0/305        24.47MiB      1.78MiB
0/306        24.47MiB      1.78MiB
0/307       137.77MiB      4.16MiB
0/308       201.66MiB      6.55MiB
0/309       201.66MiB      6.55MiB
0/312       201.66MiB      6.55MiB
0/313       523.83MiB    332.89MiB
0/317        41.12MiB      1.91MiB
0/318        41.14MiB      1.91MiB
0/319        41.14MiB      1.91MiB
0/320        48.47MiB      9.33MiB
0/342       152.89MiB      5.45MiB
0/343       152.89MiB      5.45MiB
0/347       152.89MiB      5.45MiB
0/348       158.17MiB     12.01MiB
0/3015          0.00B        0.00B
0/3043       16.00KiB        0.00B
0/3068        2.94MiB        0.00B


But, the strange thing is:

btrfs fi du / -s:
Code: Select all
btrfs fi du / -s
     Total   Exclusive  Set shared  Filename
  25.40GiB    21.45GiB   643.47MiB  /


If we develop it:
Code: Select all
# btrfs fi du / -s
     Total   Exclusive  Set shared  Filename
  25.40GiB    21.45GiB   643.47MiB  /
root@oriolnas:~# btrfs fi du /* -s
     Total   Exclusive  Set shared  Filename
  13.37MiB    13.37MiB       0.00B  /bin
  57.54MiB    57.54MiB       0.00B  /boot
ERROR: cannot check space of '/dev': Inappropriate ioctl for device
  19.19MiB    19.19MiB       0.00B  /etc
     0.00B       0.00B       0.00B  /export
     0.00B       0.00B       0.00B  /home
  18.15MiB    18.15MiB       0.00B  /initrd.img
  16.04MiB    16.04MiB       0.00B  /initrd.img.old
 386.39MiB   386.39MiB       0.00B  /lib
   3.50MiB     3.50MiB       0.00B  /lib32
     0.00B       0.00B       0.00B  /lib64
     0.00B       0.00B       0.00B  /media
     0.00B       0.00B       0.00B  /mnt
     0.00B       0.00B       0.00B  /opt
ERROR: cannot check space of '/proc': Inappropriate ioctl for device
   8.29GiB     8.29GiB       0.00B  /root
ERROR: cannot check space of '/run': Inappropriate ioctl for device
  12.61MiB    12.61MiB       0.00B  /sbin
  15.31GiB    15.31GiB       0.00B  /srv
ERROR: cannot check space of '/sys': Inappropriate ioctl for device
ERROR: cannot check space of '/tmp': Inappropriate ioctl for device
   1.49GiB     1.49GiB       0.00B  /usr
  15.13GiB    11.18GiB   643.47MiB  /var
   3.86MiB     3.86MiB       0.00B  /vmlinuz
   2.98MiB     2.98MiB       0.00B  /vmlinuz.old
     0.00B       0.00B       0.00B  /webmin-setup.out


It is strange that /srv shows a 15,31 GB usage, as the only subdir that is hosted on / fs is /srv/ftp, currently empty. I also tried a classic du, hoping the "inappropiate ioctl for device" for the /dev fs or similar could hide some drive usage. Here is the output:
Code: Select all
# du -sh /*
14M     /bin
59M     /boot
0       /dev
27M     /etc
0       /export
20K     /home
4,0K    /initrd.img
4,0K    /initrd.img.old
388M    /lib
3,6M    /lib32
4,0K    /lib64
0       /media
0       /mnt
0       /opt
du: no s’ha pogut accedir a «/proc/29690/task/29690/fd/3»: El fitxer o directori no existeix
du: no s’ha pogut accedir a «/proc/29690/task/29690/fdinfo/3»: El fitxer o directori no existeix
du: no s’ha pogut accedir a «/proc/29690/fd/3»: El fitxer o directori no existeix
du: no s’ha pogut accedir a «/proc/29690/fdinfo/3»: El fitxer o directori no existeix
0       /proc
8,4G    /root
148M    /run
13M     /sbin
du: no s’ha pogut accedir a «/srv/dev-disk-by-label-nas1/.snapshots/384»: El fitxer o directori no existeix
du: no s’ha pogut accedir a «/srv/dev-disk-by-label-nas1/.snapshots/385»: El fitxer o directori no existeix
du: no s’ha pogut accedir a «/srv/dev-disk-by-label-nas1/.snapshots/386»: El fitxer o directori no existeix
du: no s’ha pogut accedir a «/srv/dev-disk-by-label-nas1/.snapshots/387»: El fitxer o directori no existeix
du: no s’ha pogut accedir a «/srv/dev-disk-by-label-nas1/.snapshots/388»: El fitxer o directori no existeix
du: no s’ha pogut accedir a «/srv/dev-disk-by-label-nas1/.snapshots/389»: El fitxer o directori no existeix
du: no s’ha pogut accedir a «/srv/dev-disk-by-label-nas1/.snapshots/390»: El fitxer o directori no existeix
du: no s’ha pogut accedir a «/srv/dev-disk-by-label-nas1/.snapshots/391»: El fitxer o directori no existeix
du: no s’ha pogut accedir a «/srv/dev-disk-by-label-nas1/.snapshots/392»: El fitxer o directori no existeix
du: no s’ha pogut accedir a «/srv/dev-disk-by-label-nas1/.snapshots/393»: El fitxer o directori no existeix
du: no s’ha pogut accedir a «/srv/dev-disk-by-label-nas1/.snapshots/394»: El fitxer o directori no existeix
du: no s’ha pogut accedir a «/srv/dev-disk-by-label-nas1/.snapshots/395»: El fitxer o directori no existeix
du: no s’ha pogut accedir a «/srv/dev-disk-by-label-nas1/.snapshots/396»: El fitxer o directori no existeix
du: no s’ha pogut accedir a «/srv/dev-disk-by-label-nas1/.snapshots/397»: El fitxer o directori no existeix
du: no s’ha pogut accedir a «/srv/dev-disk-by-label-nas1/.snapshots/398»: El fitxer o directori no existeix
du: no s’ha pogut accedir a «/srv/dev-disk-by-label-nas1/.snapshots/399»: El fitxer o directori no existeix
du: no s’ha pogut accedir a «/srv/dev-disk-by-label-nas1/.snapshots/400»: El fitxer o directori no existeix
du: no s’ha pogut accedir a «/srv/dev-disk-by-label-nas1/.snapshots/401»: El fitxer o directori no existeix
du: no s’ha pogut accedir a «/srv/dev-disk-by-label-nas1/.snapshots/402»: El fitxer o directori no existeix
du: no s’ha pogut accedir a «/srv/dev-disk-by-label-nas1/.snapshots/403»: El fitxer o directori no existeix
du: no s’ha pogut accedir a «/srv/dev-disk-by-label-nas1/.snapshots/405»: El fitxer o directori no existeix
du: no s’ha pogut accedir a «/srv/dev-disk-by-label-nas1/.snapshots/406»: El fitxer o directori no existeix
du: no s’ha pogut accedir a «/srv/dev-disk-by-label-nas1/.snapshots/407»: El fitxer o directori no existeix
139T    /srv
0       /sys
8,0K    /tmp
1,6G    /usr
16G     /var
4,0K    /vmlinuz
4,0K    /vmlinuz.old
4,0K    /webmin-setup.out


The 139T in /srv account for the various snapshots in the filesystems hosted on my mechanical drives. But, so far, no hints on where my space is being eaten.

Sorry for the long post, and my apologies if this has already been asked or I'm making another obviously stupid error. Thanks in advance for any suggestion.

Cheers!
Last edited by vanmaal on 2017-10-29 23:49, edited 1 time in total.
vanmaal
 
Posts: 5
Joined: 2017-10-28 11:15

Re: BTRFS root filesystem full

Postby Head_on_a_Stick » 2017-10-28 13:18

Have you seen https://btrfs.wiki.kernel.org/index.php/FAQ#Help.21_I_ran_out_of_disk_space.21?

See also http://marc.merlins.org/perso/btrfs/pos ... blems.html

Personally, I find that removing old subvolumes usually clears things but your outputs are a little strange, for example:
Code: Select all
# btrfs qgroup show /
qgroupid         rfer         excl
--------         ----         ----
0/5         212.75GiB    212.75GiB

Here is mine:
Code: Select all
root@Xanadu:~ # btrfs qgroup show /
qgroupid         rfer         excl
--------         ----         ----
0/5          16.00KiB     16.00KiB

Not sure what this actually means though, sorry.
"Only the mediocre are always at their best." — Jean Giraudoux
User avatar
Head_on_a_Stick
 
Posts: 6666
Joined: 2014-06-01 17:46
Location: /dev/chair

Re: BTRFS root filesystem full

Postby vanmaal » 2017-10-28 15:45

Thanks, Head_on_a_Stick.

Yes, I tried the suggestions on both links. They are focused on cleaning up snapshots, rebalancing when all space is allocated but not used, or when metadata is full. Unfortunately I don't have any snapshots (at least according to snapper and to btrfs qgroup show), my space seems to be allocated and used, even if I can't find what are the directories (if any) that are eating up it, and have plenty of metadata space available.

Thank you anyway for helping, I appreciate it!
vanmaal
 
Posts: 5
Joined: 2017-10-28 11:15

Re: BTRFS root filesystem full

Postby pylkko » 2017-10-28 19:37

Just so that I can understand better, can you show
Code: Select all
btrfs subvolumes list /


What does the root contain? only Debian?
User avatar
pylkko
 
Posts: 1174
Joined: 2014-11-06 19:02

Re: BTRFS root filesystem full

Postby vanmaal » 2017-10-28 19:52

Hi, pylkko, thanks

Code: Select all
# btrfs subvolume list /
ID 260 gen 27500527 top level 5 path var/lib/docker/btrfs/subvolumes/c27553482d6e96afee9705b635ea729c3d3da50151027c246a1240ca14c5ed0a
ID 261 gen 27500527 top level 5 path var/lib/docker/btrfs/subvolumes/3486f382e07837f1e94b84673ecb5d697e07a3fae4738ca551ba69c267821e53
ID 262 gen 27500527 top level 5 path var/lib/docker/btrfs/subvolumes/51374748f86954260e285eaf7a7875ac26b6cc2fbec940b6d2664558c17f2b42
ID 263 gen 27500527 top level 5 path var/lib/docker/btrfs/subvolumes/ebc1f18b39d54770b0e3fa7dc30910d068c4886e0e0ed083467c6c538afa4931
ID 264 gen 27500527 top level 5 path var/lib/docker/btrfs/subvolumes/cdeae4d14523e3ff6291ee7a76633af85ee683267eeb281c8663cb76dce7d45d
ID 273 gen 27500527 top level 5 path var/lib/docker/btrfs/subvolumes/5119a4832d040c4964b54315ab173d44fab42c7eaf6eed1c4a04441ebd983631-init
ID 274 gen 27808917 top level 5 path var/lib/docker/btrfs/subvolumes/5119a4832d040c4964b54315ab173d44fab42c7eaf6eed1c4a04441ebd983631
ID 276 gen 27500527 top level 5 path var/lib/docker/btrfs/subvolumes/2d75462ac7fb7b0e89fce8216e23cf9be310411f38e7e2c2143427f517776896
ID 277 gen 27500527 top level 5 path var/lib/docker/btrfs/subvolumes/32fe5ce84f42793e78c8053b3842e5153460c87d0d550512a3d46c8d71455714
ID 278 gen 27500527 top level 5 path var/lib/docker/btrfs/subvolumes/1515af29fe6e920655c26e1c0f80bb07b4121dbeec7150fe54cecaae8b931514
ID 279 gen 27500527 top level 5 path var/lib/docker/btrfs/subvolumes/6bf97af6792a71395895733953c982811e40ea4119beef337d9b32b4499b0810
ID 280 gen 27500527 top level 5 path var/lib/docker/btrfs/subvolumes/ccf836673735f312f8aa9e7b2df33a1430d0c27d94c7298b45ea4725e169d694
ID 281 gen 27500527 top level 5 path var/lib/docker/btrfs/subvolumes/866ddcc4f78e0db50c05ef846f632fc4f5a5422aa97e2389a42259a1d5a0fd53
ID 282 gen 27500527 top level 5 path var/lib/docker/btrfs/subvolumes/3355ca9198ae3d5549ffccc7d95594dc274d4fd5a1604b0b505e3af52d084543
ID 283 gen 27500527 top level 5 path var/lib/docker/btrfs/subvolumes/5f512200af40fdf602e81daa3b5398bed47f4491c04c3050b7d66a12be97ce8f
ID 284 gen 27500527 top level 5 path var/lib/docker/btrfs/subvolumes/20f2e4f3e10caf9b73c3d4d470ccc9fd1a22cd3b22940f37a4ecf58e0406ab4b
ID 286 gen 27500527 top level 5 path var/lib/docker/btrfs/subvolumes/b9a3e54ce818044893042a4d5a549ae1cfa6d0dc443da1a1494186d0c7240042
ID 292 gen 27500527 top level 5 path var/lib/docker/btrfs/subvolumes/bc6d5ca374d043dc57cbeb1c67214435da634ae13cc711fbfe6b3d0ed8e98a7f-init
ID 293 gen 27500527 top level 5 path var/lib/docker/btrfs/subvolumes/bc6d5ca374d043dc57cbeb1c67214435da634ae13cc711fbfe6b3d0ed8e98a7f
ID 294 gen 27500527 top level 5 path var/lib/docker/btrfs/subvolumes/4c79d9694c28094b6cab59eabd4d4149cb30fde5ed64311618d34a79e868fac7-init
ID 295 gen 27737725 top level 5 path var/lib/docker/btrfs/subvolumes/4c79d9694c28094b6cab59eabd4d4149cb30fde5ed64311618d34a79e868fac7
ID 304 gen 27500527 top level 5 path var/lib/docker/btrfs/subvolumes/0f8da760146b08ca002c45af708cc277153f8a441aadda0689b0fdb3ad343100
ID 305 gen 27500527 top level 5 path var/lib/docker/btrfs/subvolumes/827e5828ebad1724c565edc92e548781da977783ae2e138fa94fd214c3f480fa
ID 306 gen 27500527 top level 5 path var/lib/docker/btrfs/subvolumes/ab51774d7567f0ed44a81add375fe415965bbf4ee9c735b2fb1018f018d67c39
ID 307 gen 27500527 top level 5 path var/lib/docker/btrfs/subvolumes/c2ce0e0577c80089ebc5533cf3e5fe42a23b9cd7f7d2913fd782cdb02621e355
ID 308 gen 27500527 top level 5 path var/lib/docker/btrfs/subvolumes/9fe7e04a433d9f9a5b1fa6313046f0041f54ee8a712826500d1216214d7f4203
ID 309 gen 27500527 top level 5 path var/lib/docker/btrfs/subvolumes/2f71615dc662e51ca9e01d78679f65f12b0d69ca618e97bef7780c633e03de64
ID 312 gen 27500527 top level 5 path var/lib/docker/btrfs/subvolumes/1f8d857a0317e4dc6eab508d4c59c2c1e0d4dde2ac0f16799e20aa86b3539e2c-init
ID 313 gen 27810824 top level 5 path var/lib/docker/btrfs/subvolumes/1f8d857a0317e4dc6eab508d4c59c2c1e0d4dde2ac0f16799e20aa86b3539e2c
ID 317 gen 27500527 top level 5 path var/lib/docker/btrfs/subvolumes/af7888811183ec4299cfbfea347ce87fdb2a0c998642cd8f074082040fb4afd1
ID 318 gen 27500527 top level 5 path var/lib/docker/btrfs/subvolumes/316c9ed010530c7a293f0f168df4fac2a8828429a1260723c69c5536ba4805da
ID 319 gen 27500527 top level 5 path var/lib/docker/btrfs/subvolumes/76df8bed03d08a5a1deb9a9613ef635160273eaf2825fb8838e48627741d023b-init
ID 320 gen 27616404 top level 5 path var/lib/docker/btrfs/subvolumes/76df8bed03d08a5a1deb9a9613ef635160273eaf2825fb8838e48627741d023b
ID 342 gen 27500527 top level 5 path var/lib/docker/btrfs/subvolumes/65c2463204e291a296474da0ae1271de16fb5c9bb4de8ceccc142f74825d27cd
ID 343 gen 27500527 top level 5 path var/lib/docker/btrfs/subvolumes/79ef8fce91ef316bbbf73d33440fb6413f683c9e5a1e690022f2477f5c5eb4f0
ID 347 gen 27500527 top level 5 path var/lib/docker/btrfs/subvolumes/ee78ef521c1ad5c35bee2362901944d00d8e023c0568339de6c953443412a2bc-init
ID 348 gen 27825552 top level 5 path var/lib/docker/btrfs/subvolumes/ee78ef521c1ad5c35bee2362901944d00d8e023c0568339de6c953443412a2bc


What does the root contain? only Debian?


Yes, as far as I can tell. I installed openmediavault and webmin from their own repositories, though. And I have no separate partitions for /home, /var or /usr.

Thanks for your help!
vanmaal
 
Posts: 5
Joined: 2017-10-28 11:15

Re: BTRFS root filesystem full

Postby pylkko » 2017-10-28 21:35

What kind of symptoms are you having? How does not having enough space manifest itself in your case? Do some of these containers write much stuff? Do diffs from any container stand out? Have you cleaned out unused ones?
User avatar
pylkko
 
Posts: 1174
Joined: 2014-11-06 19:02

Re: BTRFS root filesystem full

Postby vanmaal » 2017-10-28 22:48

Symptoms...? Well, I cannot write to the filesystem, of course... This translates in that when I try to access the different web interfaces, I get a software error message, or the relevant process crashes. About the containers, yes, there are some that write stuff, but they only use the root filesystem for temporary files and data is moved to a mechanical disk afterwards. Anyway, this is the output of a du command:

Code: Select all
/var/lib/docker/btrfs/subvolumes# du ./* -sh
5,6M    ./0f8da760146b08ca002c45af708cc277153f8a441aadda0689b0fdb3ad343100
106M    ./1515af29fe6e920655c26e1c0f80bb07b4121dbeec7150fe54cecaae8b931514
537M    ./1f8d857a0317e4dc6eab508d4c59c2c1e0d4dde2ac0f16799e20aa86b3539e2c
214M    ./1f8d857a0317e4dc6eab508d4c59c2c1e0d4dde2ac0f16799e20aa86b3539e2c-init
409M    ./20f2e4f3e10caf9b73c3d4d470ccc9fd1a22cd3b22940f37a4ecf58e0406ab4b
127M    ./2d75462ac7fb7b0e89fce8216e23cf9be310411f38e7e2c2143427f517776896
214M    ./2f71615dc662e51ca9e01d78679f65f12b0d69ca618e97bef7780c633e03de64
49M     ./316c9ed010530c7a293f0f168df4fac2a8828429a1260723c69c5536ba4805da
127M    ./32fe5ce84f42793e78c8053b3842e5153460c87d0d550512a3d46c8d71455714
142M    ./3355ca9198ae3d5549ffccc7d95594dc274d4fd5a1604b0b505e3af52d084543
32M     ./3486f382e07837f1e94b84673ecb5d697e07a3fae4738ca551ba69c267821e53
409M    ./4c79d9694c28094b6cab59eabd4d4149cb30fde5ed64311618d34a79e868fac7
409M    ./4c79d9694c28094b6cab59eabd4d4149cb30fde5ed64311618d34a79e868fac7-init
33M     ./5119a4832d040c4964b54315ab173d44fab42c7eaf6eed1c4a04441ebd983631
33M     ./5119a4832d040c4964b54315ab173d44fab42c7eaf6eed1c4a04441ebd983631-init
32M     ./51374748f86954260e285eaf7a7875ac26b6cc2fbec940b6d2664558c17f2b42
142M    ./5f512200af40fdf602e81daa3b5398bed47f4491c04c3050b7d66a12be97ce8f
162M    ./65c2463204e291a296474da0ae1271de16fb5c9bb4de8ceccc142f74825d27cd
106M    ./6bf97af6792a71395895733953c982811e40ea4119beef337d9b32b4499b0810
56M     ./76df8bed03d08a5a1deb9a9613ef635160273eaf2825fb8838e48627741d023b
49M     ./76df8bed03d08a5a1deb9a9613ef635160273eaf2825fb8838e48627741d023b-init
162M    ./79ef8fce91ef316bbbf73d33440fb6413f683c9e5a1e690022f2477f5c5eb4f0
32M     ./827e5828ebad1724c565edc92e548781da977783ae2e138fa94fd214c3f480fa
106M    ./866ddcc4f78e0db50c05ef846f632fc4f5a5422aa97e2389a42259a1d5a0fd53
214M    ./9fe7e04a433d9f9a5b1fa6313046f0041f54ee8a712826500d1216214d7f4203
32M     ./ab51774d7567f0ed44a81add375fe415965bbf4ee9c735b2fb1018f018d67c39
49M     ./af7888811183ec4299cfbfea347ce87fdb2a0c998642cd8f074082040fb4afd1
409M    ./b9a3e54ce818044893042a4d5a549ae1cfa6d0dc443da1a1494186d0c7240042
4,0K    ./bc6d5ca374d043dc57cbeb1c67214435da634ae13cc711fbfe6b3d0ed8e98a7f
4,0K    ./bc6d5ca374d043dc57cbeb1c67214435da634ae13cc711fbfe6b3d0ed8e98a7f-init
5,6M    ./c27553482d6e96afee9705b635ea729c3d3da50151027c246a1240ca14c5ed0a
147M    ./c2ce0e0577c80089ebc5533cf3e5fe42a23b9cd7f7d2913fd782cdb02621e355
106M    ./ccf836673735f312f8aa9e7b2df33a1430d0c27d94c7298b45ea4725e169d694
33M     ./cdeae4d14523e3ff6291ee7a76633af85ee683267eeb281c8663cb76dce7d45d
33M     ./ebc1f18b39d54770b0e3fa7dc30910d068c4886e0e0ed083467c6c538afa4931
168M    ./ee78ef521c1ad5c35bee2362901944d00d8e023c0568339de6c953443412a2bc
162M    ./ee78ef521c1ad5c35bee2362901944d00d8e023c0568339de6c953443412a2bc-init



Or:

Code: Select all
/var/lib/docker/btrfs/subvolumes# btrfs fi du ./* -s
     Total   Exclusive  Set shared  Filename
   4.04MiB    24.00KiB     4.01MiB  ./0f8da760146b08ca002c45af708cc277153f8a441aadda0689b0fdb3ad343100
  92.83MiB       0.00B    92.83MiB  ./1515af29fe6e920655c26e1c0f80bb07b4121dbeec7150fe54cecaae8b931514
 503.78MiB   320.36MiB   183.42MiB  ./1f8d857a0317e4dc6eab508d4c59c2c1e0d4dde2ac0f16799e20aa86b3539e2c
 183.42MiB       0.00B   183.42MiB  ./1f8d857a0317e4dc6eab508d4c59c2c1e0d4dde2ac0f16799e20aa86b3539e2c-init
 375.64MiB       0.00B   375.64MiB  ./20f2e4f3e10caf9b73c3d4d470ccc9fd1a22cd3b22940f37a4ecf58e0406ab4b
 114.00MiB       0.00B   114.00MiB  ./2d75462ac7fb7b0e89fce8216e23cf9be310411f38e7e2c2143427f517776896
 183.42MiB       0.00B   183.42MiB  ./2f71615dc662e51ca9e01d78679f65f12b0d69ca618e97bef7780c633e03de64
  34.95MiB       0.00B    34.95MiB  ./316c9ed010530c7a293f0f168df4fac2a8828429a1260723c69c5536ba4805da
 114.00MiB       0.00B   114.00MiB  ./32fe5ce84f42793e78c8053b3842e5153460c87d0d550512a3d46c8d71455714
 116.26MiB       0.00B   116.26MiB  ./3355ca9198ae3d5549ffccc7d95594dc274d4fd5a1604b0b505e3af52d084543
  18.44MiB       0.00B    18.44MiB  ./3486f382e07837f1e94b84673ecb5d697e07a3fae4738ca551ba69c267821e53
 375.64MiB       0.00B   375.64MiB  ./4c79d9694c28094b6cab59eabd4d4149cb30fde5ed64311618d34a79e868fac7
 375.64MiB       0.00B   375.64MiB  ./4c79d9694c28094b6cab59eabd4d4149cb30fde5ed64311618d34a79e868fac7-init
  19.16MiB       0.00B    19.16MiB  ./5119a4832d040c4964b54315ab173d44fab42c7eaf6eed1c4a04441ebd983631
  19.16MiB       0.00B    19.16MiB  ./5119a4832d040c4964b54315ab173d44fab42c7eaf6eed1c4a04441ebd983631-init
  18.44MiB       0.00B    18.44MiB  ./51374748f86954260e285eaf7a7875ac26b6cc2fbec940b6d2664558c17f2b42
 116.26MiB       0.00B   116.26MiB  ./5f512200af40fdf602e81daa3b5398bed47f4491c04c3050b7d66a12be97ce8f
 138.39MiB       0.00B   138.39MiB  ./65c2463204e291a296474da0ae1271de16fb5c9bb4de8ceccc142f74825d27cd
  92.83MiB       0.00B    92.83MiB  ./6bf97af6792a71395895733953c982811e40ea4119beef337d9b32b4499b0810
  42.25MiB     7.30MiB    34.95MiB  ./76df8bed03d08a5a1deb9a9613ef635160273eaf2825fb8838e48627741d023b
  34.95MiB       0.00B    34.95MiB  ./76df8bed03d08a5a1deb9a9613ef635160273eaf2825fb8838e48627741d023b-init
 138.39MiB       0.00B   138.39MiB  ./79ef8fce91ef316bbbf73d33440fb6413f683c9e5a1e690022f2477f5c5eb4f0
  18.44MiB       0.00B    18.44MiB  ./827e5828ebad1724c565edc92e548781da977783ae2e138fa94fd214c3f480fa
  92.83MiB       0.00B    92.83MiB  ./866ddcc4f78e0db50c05ef846f632fc4f5a5422aa97e2389a42259a1d5a0fd53
 183.42MiB       0.00B   183.42MiB  ./9fe7e04a433d9f9a5b1fa6313046f0041f54ee8a712826500d1216214d7f4203
  18.44MiB       0.00B    18.44MiB  ./ab51774d7567f0ed44a81add375fe415965bbf4ee9c735b2fb1018f018d67c39
  34.95MiB       0.00B    34.95MiB  ./af7888811183ec4299cfbfea347ce87fdb2a0c998642cd8f074082040fb4afd1
 375.64MiB       0.00B   375.64MiB  ./b9a3e54ce818044893042a4d5a549ae1cfa6d0dc443da1a1494186d0c7240042
     0.00B       0.00B       0.00B  ./bc6d5ca374d043dc57cbeb1c67214435da634ae13cc711fbfe6b3d0ed8e98a7f
     0.00B       0.00B       0.00B  ./bc6d5ca374d043dc57cbeb1c67214435da634ae13cc711fbfe6b3d0ed8e98a7f-init
   4.04MiB    24.00KiB     4.01MiB  ./c27553482d6e96afee9705b635ea729c3d3da50151027c246a1240ca14c5ed0a
 124.96MiB       0.00B   124.96MiB  ./c2ce0e0577c80089ebc5533cf3e5fe42a23b9cd7f7d2913fd782cdb02621e355
  92.83MiB       0.00B    92.83MiB  ./ccf836673735f312f8aa9e7b2df33a1430d0c27d94c7298b45ea4725e169d694
  19.16MiB       0.00B    19.16MiB  ./cdeae4d14523e3ff6291ee7a76633af85ee683267eeb281c8663cb76dce7d45d
  19.16MiB       0.00B    19.16MiB  ./ebc1f18b39d54770b0e3fa7dc30910d068c4886e0e0ed083467c6c538afa4931
 142.87MiB     4.48MiB   138.39MiB  ./ee78ef521c1ad5c35bee2362901944d00d8e023c0568339de6c953443412a2bc
 138.39MiB       0.00B   138.39MiB  ./ee78ef521c1ad5c35bee2362901944d00d8e023c0568339de6c953443412a2bc-init


As you can see, very few exclusive data, so hardly an issue, at least how I understand it...

Following your suggestion:

Code: Select all
/var/lib/docker/btrfs/subvolumes# docker volume prune
WARNING! This will remove all volumes not used by at least one container.
Are you sure you want to continue? [y/N] y
Deleted Volumes:
8ff85c6a51668c42ec3e4b3a39a5bc641dcfcd5ea947da4ca40493cc4560a73b
efd4b404eafeac47a4294fde920f61104fe192d8e7b1fb7944e980fc7b3dd525

Total reclaimed space: 0B


0B does not sound good...

Fortunately, I'm learning a lot, even if I cannot solve the issue...

Thanks for helping, again!
vanmaal
 
Posts: 5
Joined: 2017-10-28 11:15

Re: BTRFS root filesystem full

Postby pylkko » 2017-10-29 14:02

have you ever deleted any subvolumes or qgroups? have you tried rescan?
User avatar
pylkko
 
Posts: 1174
Joined: 2014-11-06 19:02

Re: BTRFS root filesystem full

Postby vanmaal » 2017-10-29 23:48

Sorry for late response. Trying to solve the issue I made the mistake of extending the filesystem to an lvm lv... All went good until I rebooted the system, when it didn't mount root and fell back to initramfs, so I lost my remote connection. Now I know that I probably had to reinstall grub with lvm activated... Although I'm not entirely sure that this would have solved the issue.

Anyway, I tried a different approach and mounted the root filesystem from an ubuntu live cd, and found that those docker containers that were supposed to write to mechanical disks were not doing so, but writing directly to those disks mount points in the parent fs. That's probably why btrfs fi du did not detect the disk usage. I have to think now how I solve this. I suppose I can write to someplace in the root fs and set up a cron job that moves any new files to the correct places, but this is a much easier task. Anyway, thank you for all your suggestions!

Cheers!

[P.S.] If someone encounters the same situation, the problem was that docker initialized the containers before the target filesystem was mounted, so it created a directory structure in root filesystem that was invisible once all filesystems were in place. Workaround has been adding RequiresMountsFor=/path/to/mountpoint to [Unit] section of /lib/systemd/system/docker.service. Now it seems to work as intended.
vanmaal
 
Posts: 5
Joined: 2017-10-28 11:15


Return to General Questions

Who is online

Users browsing this forum: No registered users and 7 guests

fashionable