Scheduled Maintenance: We are aware of an issue with Google, AOL, and Yahoo services as email providers which are blocking new registrations. We are trying to fix the issue and we have several internal and external support tickets in process to resolve the issue. Please see: viewtopic.php?t=158230

 

 

 

[SOLVED] BTRFS root filesystem full

If none of the specific sub-forums seem right for your thread, ask here.
Post Reply
Message
Author
vanmaal
Posts: 5
Joined: 2017-10-28 11:15

[SOLVED] BTRFS root filesystem full

#1 Post by vanmaal »

Hi!,

Not sure if general questions is the correct place, so I apologise in advance if not the case.

I have a headless server running Jessie and OpenMediaVault 3. Root filesystem is btrfs, and it has grown until it fills a whole 240 GB SSD. I'm absolutely uncapable to understand where the free space has gone. I have no snapshots, and although Docker has lots of subvolumes, all but a few space is eaten by the root subvolume. But the weird thing is btrfs fi du only shows 25,40 GB used.

I tried to find anything related to my problem but so far unsuccessfully, either on google, the Debian forums or wiki. Most solutions seem to rely on balance, defrag or other btrfs maintenance options, and although my diagnostics seem to show otherwise, I tried them. Unsuccesfully, of course.

I suppose I could try reformatting to ext4 and reinstalling all software, but I'd like to avoid it. In the worst case, I have a backup from a previous working state, but I'd like to avoid it, too, if possible.

Here follow some diagnostics:

btrfs fi sh:

Code: Select all

# btrfs fi sh
Label: none  uuid: 952a3691-b40c-4996-b408-317644ffe94f
        Total devices 1 FS bytes used 214.13GiB
        devid    1 size 215.57GiB used 215.56GiB path /dev/sda2

Label: 'nas2'  uuid: 3af75ca8-e28b-46e8-9afa-4095e3a67763
        Total devices 1 FS bytes used 15.37GiB
        devid    1 size 3.64TiB used 18.06GiB path /dev/mapper/vgnas-nas2lv

Label: 'backup'  uuid: 1da0c5f8-a9fc-4ee8-ab80-79ec3a83e359
        Total devices 2 FS bytes used 1.57TiB
        devid    1 size 3.64TiB used 1.57TiB path /dev/sdb1
        devid    2 size 3.64TiB used 1.57TiB path /dev/sdc1

Label: 'nas1'  uuid: fefaad79-68e5-447a-90de-e0334ff827e0
        Total devices 1 FS bytes used 4.59TiB
        devid    1 size 5.46TiB used 4.65TiB path /dev/mapper/vgnas-nas1lv
btrfs fi df / shows more or less the same thing:

Code: Select all

# btrfs fi df /
Data, single: total=213.53GiB, used=213.53GiB
System, single: total=32.00MiB, used=48.00KiB
Metadata, single: total=2.00GiB, used=609.06MiB
GlobalReserve, single: total=238.91MiB, used=0.00B
Or btrfs qgroup show -reF /:

Code: Select all

# btrfs qgroup show -reF /
qgroupid         rfer         excl     max_rfer     max_excl
--------         ----         ----     --------     --------
0/5         212.75GiB    212.75GiB         none         none
btrfs qgroup show /:

Code: Select all

# btrfs qgroup show /
qgroupid         rfer         excl
--------         ----         ----
0/5         212.75GiB    212.75GiB
0/260         4.33MiB    328.00KiB
0/261        24.48MiB      1.77MiB
0/262        24.48MiB      1.77MiB
0/263        25.22MiB      1.78MiB
0/264        25.22MiB      1.78MiB
0/273        25.22MiB      1.78MiB
0/274        25.23MiB      1.86MiB
0/276       120.39MiB      4.36MiB
0/277       120.39MiB      4.34MiB
0/278        99.22MiB      4.34MiB
0/279        99.22MiB      4.34MiB
0/280        99.22MiB      4.36MiB
0/281        99.22MiB      4.36MiB
0/282       127.50MiB      6.94MiB
0/283       127.50MiB      6.92MiB
0/284       391.58MiB      8.80MiB
0/286       391.58MiB      8.80MiB
0/292        16.00KiB     16.00KiB
0/293        16.00KiB     16.00KiB
0/294       391.59MiB      8.83MiB
0/295       391.64MiB      9.42MiB
0/304         4.32MiB    312.00KiB
0/305        24.47MiB      1.78MiB
0/306        24.47MiB      1.78MiB
0/307       137.77MiB      4.16MiB
0/308       201.66MiB      6.55MiB
0/309       201.66MiB      6.55MiB
0/312       201.66MiB      6.55MiB
0/313       523.83MiB    332.89MiB
0/317        41.12MiB      1.91MiB
0/318        41.14MiB      1.91MiB
0/319        41.14MiB      1.91MiB
0/320        48.47MiB      9.33MiB
0/342       152.89MiB      5.45MiB
0/343       152.89MiB      5.45MiB
0/347       152.89MiB      5.45MiB
0/348       158.17MiB     12.01MiB
0/3015          0.00B        0.00B
0/3043       16.00KiB        0.00B
0/3068        2.94MiB        0.00B
But, the strange thing is:

btrfs fi du / -s:

Code: Select all

btrfs fi du / -s
     Total   Exclusive  Set shared  Filename
  25.40GiB    21.45GiB   643.47MiB  /
If we develop it:

Code: Select all

# btrfs fi du / -s
     Total   Exclusive  Set shared  Filename
  25.40GiB    21.45GiB   643.47MiB  /
root@oriolnas:~# btrfs fi du /* -s
     Total   Exclusive  Set shared  Filename
  13.37MiB    13.37MiB       0.00B  /bin
  57.54MiB    57.54MiB       0.00B  /boot
ERROR: cannot check space of '/dev': Inappropriate ioctl for device
  19.19MiB    19.19MiB       0.00B  /etc
     0.00B       0.00B       0.00B  /export
     0.00B       0.00B       0.00B  /home
  18.15MiB    18.15MiB       0.00B  /initrd.img
  16.04MiB    16.04MiB       0.00B  /initrd.img.old
 386.39MiB   386.39MiB       0.00B  /lib
   3.50MiB     3.50MiB       0.00B  /lib32
     0.00B       0.00B       0.00B  /lib64
     0.00B       0.00B       0.00B  /media
     0.00B       0.00B       0.00B  /mnt
     0.00B       0.00B       0.00B  /opt
ERROR: cannot check space of '/proc': Inappropriate ioctl for device
   8.29GiB     8.29GiB       0.00B  /root
ERROR: cannot check space of '/run': Inappropriate ioctl for device
  12.61MiB    12.61MiB       0.00B  /sbin
  15.31GiB    15.31GiB       0.00B  /srv
ERROR: cannot check space of '/sys': Inappropriate ioctl for device
ERROR: cannot check space of '/tmp': Inappropriate ioctl for device
   1.49GiB     1.49GiB       0.00B  /usr
  15.13GiB    11.18GiB   643.47MiB  /var
   3.86MiB     3.86MiB       0.00B  /vmlinuz
   2.98MiB     2.98MiB       0.00B  /vmlinuz.old
     0.00B       0.00B       0.00B  /webmin-setup.out
It is strange that /srv shows a 15,31 GB usage, as the only subdir that is hosted on / fs is /srv/ftp, currently empty. I also tried a classic du, hoping the "inappropiate ioctl for device" for the /dev fs or similar could hide some drive usage. Here is the output:

Code: Select all

# du -sh /*
14M     /bin
59M     /boot
0       /dev
27M     /etc
0       /export
20K     /home
4,0K    /initrd.img
4,0K    /initrd.img.old
388M    /lib
3,6M    /lib32
4,0K    /lib64
0       /media
0       /mnt
0       /opt
du: no s’ha pogut accedir a «/proc/29690/task/29690/fd/3»: El fitxer o directori no existeix
du: no s’ha pogut accedir a «/proc/29690/task/29690/fdinfo/3»: El fitxer o directori no existeix
du: no s’ha pogut accedir a «/proc/29690/fd/3»: El fitxer o directori no existeix
du: no s’ha pogut accedir a «/proc/29690/fdinfo/3»: El fitxer o directori no existeix
0       /proc
8,4G    /root
148M    /run
13M     /sbin
du: no s’ha pogut accedir a «/srv/dev-disk-by-label-nas1/.snapshots/384»: El fitxer o directori no existeix
du: no s’ha pogut accedir a «/srv/dev-disk-by-label-nas1/.snapshots/385»: El fitxer o directori no existeix
du: no s’ha pogut accedir a «/srv/dev-disk-by-label-nas1/.snapshots/386»: El fitxer o directori no existeix
du: no s’ha pogut accedir a «/srv/dev-disk-by-label-nas1/.snapshots/387»: El fitxer o directori no existeix
du: no s’ha pogut accedir a «/srv/dev-disk-by-label-nas1/.snapshots/388»: El fitxer o directori no existeix
du: no s’ha pogut accedir a «/srv/dev-disk-by-label-nas1/.snapshots/389»: El fitxer o directori no existeix
du: no s’ha pogut accedir a «/srv/dev-disk-by-label-nas1/.snapshots/390»: El fitxer o directori no existeix
du: no s’ha pogut accedir a «/srv/dev-disk-by-label-nas1/.snapshots/391»: El fitxer o directori no existeix
du: no s’ha pogut accedir a «/srv/dev-disk-by-label-nas1/.snapshots/392»: El fitxer o directori no existeix
du: no s’ha pogut accedir a «/srv/dev-disk-by-label-nas1/.snapshots/393»: El fitxer o directori no existeix
du: no s’ha pogut accedir a «/srv/dev-disk-by-label-nas1/.snapshots/394»: El fitxer o directori no existeix
du: no s’ha pogut accedir a «/srv/dev-disk-by-label-nas1/.snapshots/395»: El fitxer o directori no existeix
du: no s’ha pogut accedir a «/srv/dev-disk-by-label-nas1/.snapshots/396»: El fitxer o directori no existeix
du: no s’ha pogut accedir a «/srv/dev-disk-by-label-nas1/.snapshots/397»: El fitxer o directori no existeix
du: no s’ha pogut accedir a «/srv/dev-disk-by-label-nas1/.snapshots/398»: El fitxer o directori no existeix
du: no s’ha pogut accedir a «/srv/dev-disk-by-label-nas1/.snapshots/399»: El fitxer o directori no existeix
du: no s’ha pogut accedir a «/srv/dev-disk-by-label-nas1/.snapshots/400»: El fitxer o directori no existeix
du: no s’ha pogut accedir a «/srv/dev-disk-by-label-nas1/.snapshots/401»: El fitxer o directori no existeix
du: no s’ha pogut accedir a «/srv/dev-disk-by-label-nas1/.snapshots/402»: El fitxer o directori no existeix
du: no s’ha pogut accedir a «/srv/dev-disk-by-label-nas1/.snapshots/403»: El fitxer o directori no existeix
du: no s’ha pogut accedir a «/srv/dev-disk-by-label-nas1/.snapshots/405»: El fitxer o directori no existeix
du: no s’ha pogut accedir a «/srv/dev-disk-by-label-nas1/.snapshots/406»: El fitxer o directori no existeix
du: no s’ha pogut accedir a «/srv/dev-disk-by-label-nas1/.snapshots/407»: El fitxer o directori no existeix
139T    /srv
0       /sys
8,0K    /tmp
1,6G    /usr
16G     /var
4,0K    /vmlinuz
4,0K    /vmlinuz.old
4,0K    /webmin-setup.out
The 139T in /srv account for the various snapshots in the filesystems hosted on my mechanical drives. But, so far, no hints on where my space is being eaten.

Sorry for the long post, and my apologies if this has already been asked or I'm making another obviously stupid error. Thanks in advance for any suggestion.

Cheers!
Last edited by vanmaal on 2017-10-29 23:49, edited 1 time in total.

User avatar
Head_on_a_Stick
Posts: 14114
Joined: 2014-06-01 17:46
Location: London, England
Has thanked: 81 times
Been thanked: 132 times

Re: BTRFS root filesystem full

#2 Post by Head_on_a_Stick »

Have you seen https://btrfs.wiki.kernel.org/index.php ... k_space.21?

See also http://marc.merlins.org/perso/btrfs/pos ... blems.html

Personally, I find that removing old subvolumes usually clears things but your outputs are a little strange, for example:

Code: Select all

# btrfs qgroup show /
qgroupid         rfer         excl
--------         ----         ----
0/5         212.75GiB    212.75GiB
Here is mine:

Code: Select all

root@Xanadu:~ # btrfs qgroup show /
qgroupid         rfer         excl 
--------         ----         ---- 
0/5          16.00KiB     16.00KiB 
Not sure what this actually means though, sorry.
deadbang

vanmaal
Posts: 5
Joined: 2017-10-28 11:15

Re: BTRFS root filesystem full

#3 Post by vanmaal »

Thanks, Head_on_a_Stick.

Yes, I tried the suggestions on both links. They are focused on cleaning up snapshots, rebalancing when all space is allocated but not used, or when metadata is full. Unfortunately I don't have any snapshots (at least according to snapper and to btrfs qgroup show), my space seems to be allocated and used, even if I can't find what are the directories (if any) that are eating up it, and have plenty of metadata space available.

Thank you anyway for helping, I appreciate it!

User avatar
pylkko
Posts: 1802
Joined: 2014-11-06 19:02

Re: BTRFS root filesystem full

#4 Post by pylkko »

Just so that I can understand better, can you show

Code: Select all

btrfs subvolumes list /
What does the root contain? only Debian?

vanmaal
Posts: 5
Joined: 2017-10-28 11:15

Re: BTRFS root filesystem full

#5 Post by vanmaal »

Hi, pylkko, thanks

Code: Select all

# btrfs subvolume list /
ID 260 gen 27500527 top level 5 path var/lib/docker/btrfs/subvolumes/c27553482d6e96afee9705b635ea729c3d3da50151027c246a1240ca14c5ed0a
ID 261 gen 27500527 top level 5 path var/lib/docker/btrfs/subvolumes/3486f382e07837f1e94b84673ecb5d697e07a3fae4738ca551ba69c267821e53
ID 262 gen 27500527 top level 5 path var/lib/docker/btrfs/subvolumes/51374748f86954260e285eaf7a7875ac26b6cc2fbec940b6d2664558c17f2b42
ID 263 gen 27500527 top level 5 path var/lib/docker/btrfs/subvolumes/ebc1f18b39d54770b0e3fa7dc30910d068c4886e0e0ed083467c6c538afa4931
ID 264 gen 27500527 top level 5 path var/lib/docker/btrfs/subvolumes/cdeae4d14523e3ff6291ee7a76633af85ee683267eeb281c8663cb76dce7d45d
ID 273 gen 27500527 top level 5 path var/lib/docker/btrfs/subvolumes/5119a4832d040c4964b54315ab173d44fab42c7eaf6eed1c4a04441ebd983631-init
ID 274 gen 27808917 top level 5 path var/lib/docker/btrfs/subvolumes/5119a4832d040c4964b54315ab173d44fab42c7eaf6eed1c4a04441ebd983631
ID 276 gen 27500527 top level 5 path var/lib/docker/btrfs/subvolumes/2d75462ac7fb7b0e89fce8216e23cf9be310411f38e7e2c2143427f517776896
ID 277 gen 27500527 top level 5 path var/lib/docker/btrfs/subvolumes/32fe5ce84f42793e78c8053b3842e5153460c87d0d550512a3d46c8d71455714
ID 278 gen 27500527 top level 5 path var/lib/docker/btrfs/subvolumes/1515af29fe6e920655c26e1c0f80bb07b4121dbeec7150fe54cecaae8b931514
ID 279 gen 27500527 top level 5 path var/lib/docker/btrfs/subvolumes/6bf97af6792a71395895733953c982811e40ea4119beef337d9b32b4499b0810
ID 280 gen 27500527 top level 5 path var/lib/docker/btrfs/subvolumes/ccf836673735f312f8aa9e7b2df33a1430d0c27d94c7298b45ea4725e169d694
ID 281 gen 27500527 top level 5 path var/lib/docker/btrfs/subvolumes/866ddcc4f78e0db50c05ef846f632fc4f5a5422aa97e2389a42259a1d5a0fd53
ID 282 gen 27500527 top level 5 path var/lib/docker/btrfs/subvolumes/3355ca9198ae3d5549ffccc7d95594dc274d4fd5a1604b0b505e3af52d084543
ID 283 gen 27500527 top level 5 path var/lib/docker/btrfs/subvolumes/5f512200af40fdf602e81daa3b5398bed47f4491c04c3050b7d66a12be97ce8f
ID 284 gen 27500527 top level 5 path var/lib/docker/btrfs/subvolumes/20f2e4f3e10caf9b73c3d4d470ccc9fd1a22cd3b22940f37a4ecf58e0406ab4b
ID 286 gen 27500527 top level 5 path var/lib/docker/btrfs/subvolumes/b9a3e54ce818044893042a4d5a549ae1cfa6d0dc443da1a1494186d0c7240042
ID 292 gen 27500527 top level 5 path var/lib/docker/btrfs/subvolumes/bc6d5ca374d043dc57cbeb1c67214435da634ae13cc711fbfe6b3d0ed8e98a7f-init
ID 293 gen 27500527 top level 5 path var/lib/docker/btrfs/subvolumes/bc6d5ca374d043dc57cbeb1c67214435da634ae13cc711fbfe6b3d0ed8e98a7f
ID 294 gen 27500527 top level 5 path var/lib/docker/btrfs/subvolumes/4c79d9694c28094b6cab59eabd4d4149cb30fde5ed64311618d34a79e868fac7-init
ID 295 gen 27737725 top level 5 path var/lib/docker/btrfs/subvolumes/4c79d9694c28094b6cab59eabd4d4149cb30fde5ed64311618d34a79e868fac7
ID 304 gen 27500527 top level 5 path var/lib/docker/btrfs/subvolumes/0f8da760146b08ca002c45af708cc277153f8a441aadda0689b0fdb3ad343100
ID 305 gen 27500527 top level 5 path var/lib/docker/btrfs/subvolumes/827e5828ebad1724c565edc92e548781da977783ae2e138fa94fd214c3f480fa
ID 306 gen 27500527 top level 5 path var/lib/docker/btrfs/subvolumes/ab51774d7567f0ed44a81add375fe415965bbf4ee9c735b2fb1018f018d67c39
ID 307 gen 27500527 top level 5 path var/lib/docker/btrfs/subvolumes/c2ce0e0577c80089ebc5533cf3e5fe42a23b9cd7f7d2913fd782cdb02621e355
ID 308 gen 27500527 top level 5 path var/lib/docker/btrfs/subvolumes/9fe7e04a433d9f9a5b1fa6313046f0041f54ee8a712826500d1216214d7f4203
ID 309 gen 27500527 top level 5 path var/lib/docker/btrfs/subvolumes/2f71615dc662e51ca9e01d78679f65f12b0d69ca618e97bef7780c633e03de64
ID 312 gen 27500527 top level 5 path var/lib/docker/btrfs/subvolumes/1f8d857a0317e4dc6eab508d4c59c2c1e0d4dde2ac0f16799e20aa86b3539e2c-init
ID 313 gen 27810824 top level 5 path var/lib/docker/btrfs/subvolumes/1f8d857a0317e4dc6eab508d4c59c2c1e0d4dde2ac0f16799e20aa86b3539e2c
ID 317 gen 27500527 top level 5 path var/lib/docker/btrfs/subvolumes/af7888811183ec4299cfbfea347ce87fdb2a0c998642cd8f074082040fb4afd1
ID 318 gen 27500527 top level 5 path var/lib/docker/btrfs/subvolumes/316c9ed010530c7a293f0f168df4fac2a8828429a1260723c69c5536ba4805da
ID 319 gen 27500527 top level 5 path var/lib/docker/btrfs/subvolumes/76df8bed03d08a5a1deb9a9613ef635160273eaf2825fb8838e48627741d023b-init
ID 320 gen 27616404 top level 5 path var/lib/docker/btrfs/subvolumes/76df8bed03d08a5a1deb9a9613ef635160273eaf2825fb8838e48627741d023b
ID 342 gen 27500527 top level 5 path var/lib/docker/btrfs/subvolumes/65c2463204e291a296474da0ae1271de16fb5c9bb4de8ceccc142f74825d27cd
ID 343 gen 27500527 top level 5 path var/lib/docker/btrfs/subvolumes/79ef8fce91ef316bbbf73d33440fb6413f683c9e5a1e690022f2477f5c5eb4f0
ID 347 gen 27500527 top level 5 path var/lib/docker/btrfs/subvolumes/ee78ef521c1ad5c35bee2362901944d00d8e023c0568339de6c953443412a2bc-init
ID 348 gen 27825552 top level 5 path var/lib/docker/btrfs/subvolumes/ee78ef521c1ad5c35bee2362901944d00d8e023c0568339de6c953443412a2bc
What does the root contain? only Debian?
Yes, as far as I can tell. I installed openmediavault and webmin from their own repositories, though. And I have no separate partitions for /home, /var or /usr.

Thanks for your help!

User avatar
pylkko
Posts: 1802
Joined: 2014-11-06 19:02

Re: BTRFS root filesystem full

#6 Post by pylkko »

What kind of symptoms are you having? How does not having enough space manifest itself in your case? Do some of these containers write much stuff? Do diffs from any container stand out? Have you cleaned out unused ones?

vanmaal
Posts: 5
Joined: 2017-10-28 11:15

Re: BTRFS root filesystem full

#7 Post by vanmaal »

Symptoms...? Well, I cannot write to the filesystem, of course... This translates in that when I try to access the different web interfaces, I get a software error message, or the relevant process crashes. About the containers, yes, there are some that write stuff, but they only use the root filesystem for temporary files and data is moved to a mechanical disk afterwards. Anyway, this is the output of a du command:

Code: Select all

/var/lib/docker/btrfs/subvolumes# du ./* -sh
5,6M    ./0f8da760146b08ca002c45af708cc277153f8a441aadda0689b0fdb3ad343100
106M    ./1515af29fe6e920655c26e1c0f80bb07b4121dbeec7150fe54cecaae8b931514
537M    ./1f8d857a0317e4dc6eab508d4c59c2c1e0d4dde2ac0f16799e20aa86b3539e2c
214M    ./1f8d857a0317e4dc6eab508d4c59c2c1e0d4dde2ac0f16799e20aa86b3539e2c-init
409M    ./20f2e4f3e10caf9b73c3d4d470ccc9fd1a22cd3b22940f37a4ecf58e0406ab4b
127M    ./2d75462ac7fb7b0e89fce8216e23cf9be310411f38e7e2c2143427f517776896
214M    ./2f71615dc662e51ca9e01d78679f65f12b0d69ca618e97bef7780c633e03de64
49M     ./316c9ed010530c7a293f0f168df4fac2a8828429a1260723c69c5536ba4805da
127M    ./32fe5ce84f42793e78c8053b3842e5153460c87d0d550512a3d46c8d71455714
142M    ./3355ca9198ae3d5549ffccc7d95594dc274d4fd5a1604b0b505e3af52d084543
32M     ./3486f382e07837f1e94b84673ecb5d697e07a3fae4738ca551ba69c267821e53
409M    ./4c79d9694c28094b6cab59eabd4d4149cb30fde5ed64311618d34a79e868fac7
409M    ./4c79d9694c28094b6cab59eabd4d4149cb30fde5ed64311618d34a79e868fac7-init
33M     ./5119a4832d040c4964b54315ab173d44fab42c7eaf6eed1c4a04441ebd983631
33M     ./5119a4832d040c4964b54315ab173d44fab42c7eaf6eed1c4a04441ebd983631-init
32M     ./51374748f86954260e285eaf7a7875ac26b6cc2fbec940b6d2664558c17f2b42
142M    ./5f512200af40fdf602e81daa3b5398bed47f4491c04c3050b7d66a12be97ce8f
162M    ./65c2463204e291a296474da0ae1271de16fb5c9bb4de8ceccc142f74825d27cd
106M    ./6bf97af6792a71395895733953c982811e40ea4119beef337d9b32b4499b0810
56M     ./76df8bed03d08a5a1deb9a9613ef635160273eaf2825fb8838e48627741d023b
49M     ./76df8bed03d08a5a1deb9a9613ef635160273eaf2825fb8838e48627741d023b-init
162M    ./79ef8fce91ef316bbbf73d33440fb6413f683c9e5a1e690022f2477f5c5eb4f0
32M     ./827e5828ebad1724c565edc92e548781da977783ae2e138fa94fd214c3f480fa
106M    ./866ddcc4f78e0db50c05ef846f632fc4f5a5422aa97e2389a42259a1d5a0fd53
214M    ./9fe7e04a433d9f9a5b1fa6313046f0041f54ee8a712826500d1216214d7f4203
32M     ./ab51774d7567f0ed44a81add375fe415965bbf4ee9c735b2fb1018f018d67c39
49M     ./af7888811183ec4299cfbfea347ce87fdb2a0c998642cd8f074082040fb4afd1
409M    ./b9a3e54ce818044893042a4d5a549ae1cfa6d0dc443da1a1494186d0c7240042
4,0K    ./bc6d5ca374d043dc57cbeb1c67214435da634ae13cc711fbfe6b3d0ed8e98a7f
4,0K    ./bc6d5ca374d043dc57cbeb1c67214435da634ae13cc711fbfe6b3d0ed8e98a7f-init
5,6M    ./c27553482d6e96afee9705b635ea729c3d3da50151027c246a1240ca14c5ed0a
147M    ./c2ce0e0577c80089ebc5533cf3e5fe42a23b9cd7f7d2913fd782cdb02621e355
106M    ./ccf836673735f312f8aa9e7b2df33a1430d0c27d94c7298b45ea4725e169d694
33M     ./cdeae4d14523e3ff6291ee7a76633af85ee683267eeb281c8663cb76dce7d45d
33M     ./ebc1f18b39d54770b0e3fa7dc30910d068c4886e0e0ed083467c6c538afa4931
168M    ./ee78ef521c1ad5c35bee2362901944d00d8e023c0568339de6c953443412a2bc
162M    ./ee78ef521c1ad5c35bee2362901944d00d8e023c0568339de6c953443412a2bc-init

Or:

Code: Select all

/var/lib/docker/btrfs/subvolumes# btrfs fi du ./* -s
     Total   Exclusive  Set shared  Filename
   4.04MiB    24.00KiB     4.01MiB  ./0f8da760146b08ca002c45af708cc277153f8a441aadda0689b0fdb3ad343100
  92.83MiB       0.00B    92.83MiB  ./1515af29fe6e920655c26e1c0f80bb07b4121dbeec7150fe54cecaae8b931514
 503.78MiB   320.36MiB   183.42MiB  ./1f8d857a0317e4dc6eab508d4c59c2c1e0d4dde2ac0f16799e20aa86b3539e2c
 183.42MiB       0.00B   183.42MiB  ./1f8d857a0317e4dc6eab508d4c59c2c1e0d4dde2ac0f16799e20aa86b3539e2c-init
 375.64MiB       0.00B   375.64MiB  ./20f2e4f3e10caf9b73c3d4d470ccc9fd1a22cd3b22940f37a4ecf58e0406ab4b
 114.00MiB       0.00B   114.00MiB  ./2d75462ac7fb7b0e89fce8216e23cf9be310411f38e7e2c2143427f517776896
 183.42MiB       0.00B   183.42MiB  ./2f71615dc662e51ca9e01d78679f65f12b0d69ca618e97bef7780c633e03de64
  34.95MiB       0.00B    34.95MiB  ./316c9ed010530c7a293f0f168df4fac2a8828429a1260723c69c5536ba4805da
 114.00MiB       0.00B   114.00MiB  ./32fe5ce84f42793e78c8053b3842e5153460c87d0d550512a3d46c8d71455714
 116.26MiB       0.00B   116.26MiB  ./3355ca9198ae3d5549ffccc7d95594dc274d4fd5a1604b0b505e3af52d084543
  18.44MiB       0.00B    18.44MiB  ./3486f382e07837f1e94b84673ecb5d697e07a3fae4738ca551ba69c267821e53
 375.64MiB       0.00B   375.64MiB  ./4c79d9694c28094b6cab59eabd4d4149cb30fde5ed64311618d34a79e868fac7
 375.64MiB       0.00B   375.64MiB  ./4c79d9694c28094b6cab59eabd4d4149cb30fde5ed64311618d34a79e868fac7-init
  19.16MiB       0.00B    19.16MiB  ./5119a4832d040c4964b54315ab173d44fab42c7eaf6eed1c4a04441ebd983631
  19.16MiB       0.00B    19.16MiB  ./5119a4832d040c4964b54315ab173d44fab42c7eaf6eed1c4a04441ebd983631-init
  18.44MiB       0.00B    18.44MiB  ./51374748f86954260e285eaf7a7875ac26b6cc2fbec940b6d2664558c17f2b42
 116.26MiB       0.00B   116.26MiB  ./5f512200af40fdf602e81daa3b5398bed47f4491c04c3050b7d66a12be97ce8f
 138.39MiB       0.00B   138.39MiB  ./65c2463204e291a296474da0ae1271de16fb5c9bb4de8ceccc142f74825d27cd
  92.83MiB       0.00B    92.83MiB  ./6bf97af6792a71395895733953c982811e40ea4119beef337d9b32b4499b0810
  42.25MiB     7.30MiB    34.95MiB  ./76df8bed03d08a5a1deb9a9613ef635160273eaf2825fb8838e48627741d023b
  34.95MiB       0.00B    34.95MiB  ./76df8bed03d08a5a1deb9a9613ef635160273eaf2825fb8838e48627741d023b-init
 138.39MiB       0.00B   138.39MiB  ./79ef8fce91ef316bbbf73d33440fb6413f683c9e5a1e690022f2477f5c5eb4f0
  18.44MiB       0.00B    18.44MiB  ./827e5828ebad1724c565edc92e548781da977783ae2e138fa94fd214c3f480fa
  92.83MiB       0.00B    92.83MiB  ./866ddcc4f78e0db50c05ef846f632fc4f5a5422aa97e2389a42259a1d5a0fd53
 183.42MiB       0.00B   183.42MiB  ./9fe7e04a433d9f9a5b1fa6313046f0041f54ee8a712826500d1216214d7f4203
  18.44MiB       0.00B    18.44MiB  ./ab51774d7567f0ed44a81add375fe415965bbf4ee9c735b2fb1018f018d67c39
  34.95MiB       0.00B    34.95MiB  ./af7888811183ec4299cfbfea347ce87fdb2a0c998642cd8f074082040fb4afd1
 375.64MiB       0.00B   375.64MiB  ./b9a3e54ce818044893042a4d5a549ae1cfa6d0dc443da1a1494186d0c7240042
     0.00B       0.00B       0.00B  ./bc6d5ca374d043dc57cbeb1c67214435da634ae13cc711fbfe6b3d0ed8e98a7f
     0.00B       0.00B       0.00B  ./bc6d5ca374d043dc57cbeb1c67214435da634ae13cc711fbfe6b3d0ed8e98a7f-init
   4.04MiB    24.00KiB     4.01MiB  ./c27553482d6e96afee9705b635ea729c3d3da50151027c246a1240ca14c5ed0a
 124.96MiB       0.00B   124.96MiB  ./c2ce0e0577c80089ebc5533cf3e5fe42a23b9cd7f7d2913fd782cdb02621e355
  92.83MiB       0.00B    92.83MiB  ./ccf836673735f312f8aa9e7b2df33a1430d0c27d94c7298b45ea4725e169d694
  19.16MiB       0.00B    19.16MiB  ./cdeae4d14523e3ff6291ee7a76633af85ee683267eeb281c8663cb76dce7d45d
  19.16MiB       0.00B    19.16MiB  ./ebc1f18b39d54770b0e3fa7dc30910d068c4886e0e0ed083467c6c538afa4931
 142.87MiB     4.48MiB   138.39MiB  ./ee78ef521c1ad5c35bee2362901944d00d8e023c0568339de6c953443412a2bc
 138.39MiB       0.00B   138.39MiB  ./ee78ef521c1ad5c35bee2362901944d00d8e023c0568339de6c953443412a2bc-init
As you can see, very few exclusive data, so hardly an issue, at least how I understand it...

Following your suggestion:

Code: Select all

/var/lib/docker/btrfs/subvolumes# docker volume prune
WARNING! This will remove all volumes not used by at least one container.
Are you sure you want to continue? [y/N] y
Deleted Volumes:
8ff85c6a51668c42ec3e4b3a39a5bc641dcfcd5ea947da4ca40493cc4560a73b
efd4b404eafeac47a4294fde920f61104fe192d8e7b1fb7944e980fc7b3dd525

Total reclaimed space: 0B
0B does not sound good...

Fortunately, I'm learning a lot, even if I cannot solve the issue...

Thanks for helping, again!

User avatar
pylkko
Posts: 1802
Joined: 2014-11-06 19:02

Re: BTRFS root filesystem full

#8 Post by pylkko »

have you ever deleted any subvolumes or qgroups? have you tried rescan?

vanmaal
Posts: 5
Joined: 2017-10-28 11:15

Re: BTRFS root filesystem full

#9 Post by vanmaal »

Sorry for late response. Trying to solve the issue I made the mistake of extending the filesystem to an lvm lv... All went good until I rebooted the system, when it didn't mount root and fell back to initramfs, so I lost my remote connection. Now I know that I probably had to reinstall grub with lvm activated... Although I'm not entirely sure that this would have solved the issue.

Anyway, I tried a different approach and mounted the root filesystem from an ubuntu live cd, and found that those docker containers that were supposed to write to mechanical disks were not doing so, but writing directly to those disks mount points in the parent fs. That's probably why btrfs fi du did not detect the disk usage. I have to think now how I solve this. I suppose I can write to someplace in the root fs and set up a cron job that moves any new files to the correct places, but this is a much easier task. Anyway, thank you for all your suggestions!

Cheers!

[P.S.] If someone encounters the same situation, the problem was that docker initialized the containers before the target filesystem was mounted, so it created a directory structure in root filesystem that was invisible once all filesystems were in place. Workaround has been adding RequiresMountsFor=/path/to/mountpoint to [Unit] section of /lib/systemd/system/docker.service. Now it seems to work as intended.

Post Reply