Scheduled Maintenance: We are aware of an issue with Google, AOL, and Yahoo services as email providers which are blocking new registrations. We are trying to fix the issue and we have several internal and external support tickets in process to resolve the issue. Please see: viewtopic.php?t=158230

 

 

 

[SOLVED] No space left on device, du says otherwise

If none of the specific sub-forums seem right for your thread, ask here.
Post Reply
Message
Author
ychaouche
Posts: 87
Joined: 2014-12-11 09:45

[SOLVED] No space left on device, du says otherwise

#1 Post by ychaouche »

Dear debian users,

df -h says that my root partition is full, while du -sh fails to show that

Code: Select all

root@PIGE:/media# df -h
Filesystem                       Size  Used Avail Use% Mounted on
udev                             1.9G     0  1.9G   0% /dev
tmpfs                            383M   42M  342M  11% /run
/dev/sda5                         37G   35G     0 100% /
tmpfs                            1.9G   84K  1.9G   1% /dev/shm
tmpfs                            5.0M  4.0K  5.0M   1% /run/lock
tmpfs                            1.9G     0  1.9G   0% /sys/fs/cgroup
/dev/sda7                        176G  125G   43G  75% /var
/dev/mapper/isw_cjeajaiadh_DATA  3.6T  2.8T  579G  84% /media/pige/Data
tmpfs                            383M  8.0K  383M   1% /run/user/1000
root@PIGE:/media# du -h --max-depth=1 --exclude='/proc/*' / | sort -h
0       /proc
0       /sys
4.0K    /cdrom
4.0K    /lib64
4.0K    /opt
4.0K    /snap
4.0K    /srv
16K     /lost+found
48K     /mnt
52K     /tmp
84K     /dev
664K    /root
9.6M    /etc
13M     /bin
14M     /sbin
42M     /run
107M    /boot
587M    /lib
588M    /home
2.8G    /usr
125G    /var
2.8T    /media
3.0T    /
root@PIGE:/media# 
- /media/pige/Data and /var are separate partitions,
- /media itself is empty (contains only pige/Data which is on another parition),

What remains on / is :

Code: Select all

107M    /boot
587M    /lib
588M    /home
2.8G    /usr
We're far away from the 35Go advertised by df -h.

Any ideas on how to correctly troubleshoot this ?
Last edited by ychaouche on 2020-11-05 12:17, edited 1 time in total.

Bulkley
Posts: 6383
Joined: 2006-02-11 18:35
Has thanked: 2 times
Been thanked: 39 times

Re: No space left on device, du says otherwise

#2 Post by Bulkley »

Try

Code: Select all

apt-get clean
The usual problem is that Apt has filled up /root.

ychaouche
Posts: 87
Joined: 2014-12-11 09:45

Re: No space left on device, du says otherwise

#3 Post by ychaouche »

Still 100%

Code: Select all

pige@PIGE:~$ sudo bash
[sudo] password for pige:
root@PIGE:~# apt-get clean
root@PIGE:~# df -h
Filesystem                       Size  Used Avail Use% Mounted on
udev                             1.9G     0  1.9G   0% /dev
tmpfs                            383M   42M  342M  11% /run
/dev/sda5                         37G   35G     0 100% /
tmpfs                            1.9G   84K  1.9G   1% /dev/shm
tmpfs                            5.0M  4.0K  5.0M   1% /run/lock
tmpfs                            1.9G     0  1.9G   0% /sys/fs/cgroup
/dev/sda7                        176G  127G   41G  76% /var
/dev/mapper/isw_cjeajaiadh_DATA  3.6T  2.8T  579G  84% /media/pige/Data
tmpfs                            383M  8.0K  383M   1% /run/user/1000
root@PIGE:~#

User avatar
Head_on_a_Stick
Posts: 14114
Joined: 2014-06-01 17:46
Location: London, England
Has thanked: 81 times
Been thanked: 132 times

Re: No space left on device, du says otherwise

#4 Post by Head_on_a_Stick »

Are you using btrfs?

This command is needed for that filesystem:

Code: Select all

btrfs filesystem df -h /
See also https://btrfs.wiki.kernel.org/index.php ... k_space.21 (recursively).

You could try using ncdu to see what's taking up all the space.

If you can't install it in your system then install it from a "live" ISO then mount /dev/sda5 and run the program from there.
deadbang

ychaouche
Posts: 87
Joined: 2014-12-11 09:45

Re: No space left on device, du says otherwise

#5 Post by ychaouche »

I'm not using btrf on any partition. Here's a list of the different filesystems mounted on that machine :

Code: Select all

root@PIGE:~# blkid | grep -oP 'TYPE=".*?"' | cut -f2 -d= | sed 's/"//g' | sort | uniq
dos
ext2
ext4
isw_raid_member
LVM2_member
ntfs
swap
root@PIGE:~# 
In any case I restarted the machine and disk usage went down to 11% ! (was 100%),

Image

and got back to 100% after I tried a mysql table repair !

Code: Select all

root@PIGE:~# df -h
Filesystem                       Size  Used Avail Use% Mounted on
udev                             1.9G     0  1.9G   0% /dev
tmpfs                            383M   12M  372M   3% /run
/dev/sda5                         37G   35G     0 100% /
tmpfs                            1.9G  140K  1.9G   1% /dev/shm
tmpfs                            5.0M  4.0K  5.0M   1% /run/lock
tmpfs                            1.9G     0  1.9G   0% /sys/fs/cgroup
/dev/sda7                        176G  106G   61G  64% /var
/dev/mapper/isw_cjeajaiadh_DATA  3.6T  2.8T  578G  84% /media/pige/Data
tmpfs                            383M  8.0K  383M   1% /run/user/1000
I'm thinking maybe the system tried an auto repair, that's what inflated the / partition, then for some reason the operation failed but didn't clean the mess.

User avatar
Head_on_a_Stick
Posts: 14114
Joined: 2014-06-01 17:46
Location: London, England
Has thanked: 81 times
Been thanked: 132 times

Re: No space left on device, du says otherwise

#6 Post by Head_on_a_Stick »

Head_on_a_Stick wrote:You could try using ncdu to see what's taking up all the space.
deadbang

ychaouche
Posts: 87
Joined: 2014-12-11 09:45

Re: No space left on device, du says otherwise

#7 Post by ychaouche »

Sorry for reporing this so late. So yes the problem was that the mysql repair operation took an incredible amount of disk space to perform.
The table was 21G in size, and repairing it requires twice that amount (https://stackoverflow.com/questions/144 ... pace-issue,) the root partition had only 35 available.

I configured the $tmpdir variable to be /var/ (which is in it sole partition) instead of /tmp (root partition) and the problem was fixed.

Post Reply