No space left on device, du says otherwise

If none of the more specific forums is the right place to ask

No space left on device, du says otherwise

Postby ychaouche » 2017-09-04 16:18

Dear debian users,

df -h says that my root partition is full, while du -sh fails to show that

Code: Select all
root@PIGE:/media# df -h
Filesystem                       Size  Used Avail Use% Mounted on
udev                             1.9G     0  1.9G   0% /dev
tmpfs                            383M   42M  342M  11% /run
/dev/sda5                         37G   35G     0 100% /
tmpfs                            1.9G   84K  1.9G   1% /dev/shm
tmpfs                            5.0M  4.0K  5.0M   1% /run/lock
tmpfs                            1.9G     0  1.9G   0% /sys/fs/cgroup
/dev/sda7                        176G  125G   43G  75% /var
/dev/mapper/isw_cjeajaiadh_DATA  3.6T  2.8T  579G  84% /media/pige/Data
tmpfs                            383M  8.0K  383M   1% /run/user/1000
root@PIGE:/media# du -h --max-depth=1 --exclude='/proc/*' / | sort -h
0       /proc
0       /sys
4.0K    /cdrom
4.0K    /lib64
4.0K    /opt
4.0K    /snap
4.0K    /srv
16K     /lost+found
48K     /mnt
52K     /tmp
84K     /dev
664K    /root
9.6M    /etc
13M     /bin
14M     /sbin
42M     /run
107M    /boot
587M    /lib
588M    /home
2.8G    /usr
125G    /var
2.8T    /media
3.0T    /
root@PIGE:/media#


- /media/pige/Data and /var are separate partitions,
- /media itself is empty (contains only pige/Data which is on another parition),

What remains on / is :

Code: Select all
107M    /boot
587M    /lib
588M    /home
2.8G    /usr


We're far away from the 35Go advertised by df -h.

Any ideas on how to correctly troubleshoot this ?
ychaouche
 
Posts: 55
Joined: 2014-12-11 09:45

Re: No space left on device, du says otherwise

Postby Bulkley » 2017-09-04 16:27

Try
Code: Select all
apt-get clean


The usual problem is that Apt has filled up /root.
Bulkley
 
Posts: 5360
Joined: 2006-02-11 18:35

Re: No space left on device, du says otherwise

Postby ychaouche » 2017-09-04 19:13

Still 100%

Code: Select all
pige@PIGE:~$ sudo bash
[sudo] password for pige:
root@PIGE:~# apt-get clean
root@PIGE:~# df -h
Filesystem                       Size  Used Avail Use% Mounted on
udev                             1.9G     0  1.9G   0% /dev
tmpfs                            383M   42M  342M  11% /run
/dev/sda5                         37G   35G     0 100% /
tmpfs                            1.9G   84K  1.9G   1% /dev/shm
tmpfs                            5.0M  4.0K  5.0M   1% /run/lock
tmpfs                            1.9G     0  1.9G   0% /sys/fs/cgroup
/dev/sda7                        176G  127G   41G  76% /var
/dev/mapper/isw_cjeajaiadh_DATA  3.6T  2.8T  579G  84% /media/pige/Data
tmpfs                            383M  8.0K  383M   1% /run/user/1000
root@PIGE:~#
ychaouche
 
Posts: 55
Joined: 2014-12-11 09:45

Re: No space left on device, du says otherwise

Postby Head_on_a_Stick » 2017-09-04 19:36

Are you using btrfs?

This command is needed for that filesystem:
Code: Select all
btrfs filesystem df -h /

See also https://btrfs.wiki.kernel.org/index.php ... k_space.21 (recursively).

You could try using ncdu to see what's taking up all the space.

If you can't install it in your system then install it from a "live" ISO then mount /dev/sda5 and run the program from there.
"Only the mediocre are always at their best." — Jean Giraudoux
User avatar
Head_on_a_Stick
 
Posts: 6782
Joined: 2014-06-01 17:46
Location: /dev/chair

Re: No space left on device, du says otherwise

Postby ychaouche » 2017-09-05 12:32

I'm not using btrf on any partition. Here's a list of the different filesystems mounted on that machine :

Code: Select all
root@PIGE:~# blkid | grep -oP 'TYPE=".*?"' | cut -f2 -d= | sed 's/"//g' | sort | uniq
dos
ext2
ext4
isw_raid_member
LVM2_member
ntfs
swap
root@PIGE:~#


In any case I restarted the machine and disk usage went down to 11% ! (was 100%),

Image

and got back to 100% after I tried a mysql table repair !

Code: Select all
root@PIGE:~# df -h
Filesystem                       Size  Used Avail Use% Mounted on
udev                             1.9G     0  1.9G   0% /dev
tmpfs                            383M   12M  372M   3% /run
/dev/sda5                         37G   35G     0 100% /
tmpfs                            1.9G  140K  1.9G   1% /dev/shm
tmpfs                            5.0M  4.0K  5.0M   1% /run/lock
tmpfs                            1.9G     0  1.9G   0% /sys/fs/cgroup
/dev/sda7                        176G  106G   61G  64% /var
/dev/mapper/isw_cjeajaiadh_DATA  3.6T  2.8T  578G  84% /media/pige/Data
tmpfs                            383M  8.0K  383M   1% /run/user/1000


I'm thinking maybe the system tried an auto repair, that's what inflated the / partition, then for some reason the operation failed but didn't clean the mess.
ychaouche
 
Posts: 55
Joined: 2014-12-11 09:45

Re: No space left on device, du says otherwise

Postby Head_on_a_Stick » 2017-09-05 17:02

Head_on_a_Stick wrote:You could try using ncdu to see what's taking up all the space.
"Only the mediocre are always at their best." — Jean Giraudoux
User avatar
Head_on_a_Stick
 
Posts: 6782
Joined: 2014-06-01 17:46
Location: /dev/chair

Re: No space left on device, du says otherwise

Postby ychaouche » 2017-11-29 08:36

Sorry for reporing this so late. So yes the problem was that the mysql repair operation took an incredible amount of disk space to perform.
The table was 21G in size, and repairing it requires twice that amount (https://stackoverflow.com/questions/144 ... pace-issue,) the root partition had only 35 available.

I configured the $tmpdir variable to be /var/ (which is in it sole partition) instead of /tmp (root partition) and the problem was fixed.
ychaouche
 
Posts: 55
Joined: 2014-12-11 09:45


Return to General Questions

Who is online

Users browsing this forum: No registered users and 6 guests

fashionable