Scheduled Maintenance: We are aware of an issue with Google, AOL, and Yahoo services as email providers which are blocking new registrations. We are trying to fix the issue and we have several internal and external support tickets in process to resolve the issue. Please see: viewtopic.php?t=158230

 

 

 

File system: ROOT -- full -- X closes with error SOLVED

New to Debian (Or Linux in general)? Ask your questions here!
Post Reply
Message
Author
User avatar
mike acker
Posts: 131
Joined: 2017-06-28 21:23

File system: ROOT -- full -- X closes with error SOLVED

#1 Post by mike acker »

Need guidance on fixing this:

while working with Oracle/Virtual box I received a message : root file system full.

I was unable to close the virtual box and had to kill it with the power switch. now when I try to re-boot it it gives an error: X window somethin -- and then just tries to restart

Can I: use gparted and increase the size of the /root file system ?
the failed disc is in SDA; this system is from SDC. so: I should be able to launch gparted and access the partition table on SDA ( I hope )
it's at 32GB now, I'd like to increase it to 64 GB

any help/guidance will be appreciated
should i do this using the live installer ??
Last edited by mike acker on 2018-09-21 19:36, edited 1 time in total.
Viva la Resistencia

User avatar
debiman
Posts: 3063
Joined: 2013-03-12 07:18

Re: File system: ROOT -- full -- X closes with error

#2 Post by debiman »

virtual discs can be resized somehow; depends a little on the type you chose iirc.
what did your web searches on the topic reveal?

i hope it just works again after that.
if not, and you still require help:
mike acker wrote:error: X window somethin --
we would need to know a lot more about this "somethin".

User avatar
mike acker
Posts: 131
Joined: 2017-06-28 21:23

Re: File system: ROOT -- full -- X closes with error

#3 Post by mike acker »

debiman wrote:virtual discs can be resized somehow; depends a little on the type you chose iirc.
what did your web searches on the topic reveal?

i hope it just works again after that.
if not, and you still require help:
mike acker wrote:error: X window somethin --
we would need to know a lot more about this "somethin".
alas-- i wasn't able to capture the exact message. however: i can go back and re-create it.

from my web searches it appears that an apt clean command might do the trick. the system boots ok -- but just cant complete the x-window (cinamon ) log on

there should be a key sequence to drop the x-window and get a command prompt; from that i might be able to issue the apt cleanup. i suspect there have been too many writes to /var in the / partition

i'll go back after the message now
Viva la Resistencia

User avatar
debiman
Posts: 3063
Joined: 2013-03-12 07:18

Re: File system: ROOT -- full -- X closes with error

#4 Post by debiman »

are you actually talking about a virtual machine?
suddenly i'm not so sure anymore.

User avatar
mike acker
Posts: 131
Joined: 2017-06-28 21:23

Re: File system: ROOT -- full -- X closes with error

#5 Post by mike acker »

debiman wrote:are you actually talking about a virtual machine?
suddenly i'm not so sure anymore.
thanks so much for helping here !!

here is the error:

Code: Select all

Xsession: warning: unable to write to /tmp; X session may exit with an error
it loops back into the logon screen.

my web search suggests that writes to /var might have filled the root partition and suggests running apt clean -- or possibly using gparted to re-size the root partition.

can i kill the x-windows and get a command line prompt to run the apt clean command. seems like i know there's a way to do that

the root partition was set at 32GB

I was running a photo program called ACDSee in Oracle's Virtual Box when the original error was presented. there's plent of free space on the SDA drive: the backups only took a little over 100 GB
Viva la Resistencia

User avatar
mike acker
Posts: 131
Joined: 2017-06-28 21:23

Re: File system: ROOT -- full -- X closes with error

#6 Post by mike acker »

it's showing 22 GB in the /var/log directory with 3 files the apparent offenders:

Code: Select all

-rw-r----- 1 root adm  7281709056 Sep 21 15:25 kern.log
-rw-r----- 1 root adm      233339 Sep 21 11:51 kern.log.1
-rw-r----- 1 root adm       58208 Sep 12 12:44 kern.log.2.gz
-rw-r----- 1 root adm       99010 Sep  3 14:49 kern.log.3.gz
-rw-r----- 1 root adm       78852 Aug 26 10:07 kern.log.4.gz
-rw-rw-r-- 1 root utmp     292292 Feb 22  2018 lastlog
drwx--x--x 2 root root       4096 Sep 21 15:26 lightdm
-rw-r----- 1 root adm  7281664000 Sep 21 15:26 messages
-rw-r----- 1 root adm      231480 Sep 21 11:51 messages.1
-rw-r----- 1 root adm       60847 Sep 12 12:47 messages.2.gz
-rw-r----- 1 root adm      106680 Sep  3 14:52 messages.3.gz
-rw-r----- 1 root adm       80279 Aug 26 10:11 messages.4.gz
drwx------ 2 mdm  root       4096 Aug  7  2017 speech-dispatcher
-rw-r----- 1 root adm  7281778688 Sep 21 15:26 syslog
-rw-r----- 1 root adm      194502 Sep 21 11:51 syslog.1
-rw-r----- 1 root adm       32341 Sep 13 13:44 syslog.2.gz
-rw-r----- 1 root adm       31481 Sep 12 12:47 syslog.3.gz
-rw-r----- 1 root adm       32729 Sep  5 12:32 syslog.4.gz
-rw-r----- 1 root adm       31730 Sep  4 07:33 syslog.5.gz
-rw-r----- 1 root adm       68975 Sep  3 14:52 syslog.6.gz
-rw-r----- 1 root adm       33022 Aug 28 12:40 syslog.7.gz
can i just delete these 3 logs ?
Viva la Resistencia

User avatar
mike acker
Posts: 131
Joined: 2017-06-28 21:23

Re: File system: ROOT -- full -- X closes with error

#7 Post by mike acker »

post mortem

i deleted the 3 monster log files and the we are running OK again

thanks for the help!!
Viva la Resistencia

User avatar
sunrat
Administrator
Administrator
Posts: 6412
Joined: 2006-08-29 09:12
Location: Melbourne, Australia
Has thanked: 116 times
Been thanked: 461 times

Re: File system: ROOT -- full -- X closes with error SOLVED

#8 Post by sunrat »

I had that happen once with those same log files. Make sure you monitor them now to see why they are growing so large, it's not a normal occurrence.

Viva la Illuminati. :mrgreen:
“ computer users can be divided into 2 categories:
Those who have lost data
...and those who have not lost data YET ”
Remember to BACKUP!

User avatar
mike acker
Posts: 131
Joined: 2017-06-28 21:23

Re: File system: ROOT -- full -- X closes with error SOLVED

#9 Post by mike acker »

sunrat wrote:I had that happen once with those same log files. Make sure you monitor them now to see why they are growing so large, it's not a normal occurrence.

Viva la Illuminati. :mrgreen:
interesting. I suspect the problem was related to my activity: I was beating the crap out of a Photo Library Management package called ACDSee -- but running in Oracle's Virtual box. I had some 900 files that i wanted re-indexed and so i was hammering that program pretty bad.

i see from my googling that "back in the day" we used to put /var in its own partition because of this problem

obviously in my case the logs are the problem -- not the root partition: i had 21 GB of logs, the / partition was at 32 GB.

maybe we can get lennart to change the logging mechanism to output into a circular queue and put this problem to bed.
Viva la Resistencia

p.H
Global Moderator
Global Moderator
Posts: 3049
Joined: 2017-09-17 07:12
Has thanked: 5 times
Been thanked: 132 times

Re: File system: ROOT -- full -- X closes with error SOLVED

#10 Post by p.H »

mike acker wrote:i see from my googling that "back in the day" we used to put /var in its own partition because of this problem
obviously in my case the logs are the problem
Putting /var in a separate filesystem is not the solution to the overgrowing logs issue, because you'll get a full /var and some parts of the system may behave badly under this condition too.

User avatar
GarryRicketson
Posts: 5644
Joined: 2015-01-20 22:16
Location: Durango, Mexico

Re: File system: ROOT -- full -- X closes with error SOLVED

#11 Post by GarryRicketson »

maybe we can get lennart to change the logging mechanism to output into a circular queue and put this problem to bed.
Better then relying on Lennart or other people to do things for you, maybe learn how to configure your system your self .
How to configure the logging mechanism on linux, so that old logs are deleted

One hit that is educational: https://www.loggly.com/ultimate-guide/m ... inux-logs/

Another, says:
. For the
directory to automatically delete old archived logs, define a log file deletion ...
You might want to look into "logrotate" and how to configure it:
It is commonly
also used to delete old log files so as not to fill your server's hard drive. ... In stock--snip--
You show here : http://forums.debian.net/posting.php?mo ... 1#pr681622
All those .gz files are the older logs, the large files are ones that have not yet been compressed, and those have the more recent logs, it is normal that they appear to be large, depending on how you have things configured, at some point they get compressed, IE:

Code: Select all

Sep 21 15:26 syslog
194502 Sep 21 11:51 syslog.1
-rw-r----- 1 root adm       32341 Sep 13 13:44 syslog.2.gz
-rw-r----- 1 root adm       31481 Sep 12 12:47 syslog.3.gz
-rw-r----- 1 root adm       32729 Sep  5 12:32 syslog.4.gz
-rw-r----- 1 root adm       31730 Sep  4 07:33 syslog.5.gz
-rw-r----- 1 root adm       68975 Sep  3 14:52 syslog.6.gz
-rw-r----- 1 root adm       33022 Aug 28 12:40 syslog.7.gz
 
You could, and should have just deleted the .gz files, or moved them to a storage device if you want to save them. The system log could contain information about your system, you should have at least reviewed it, it might of had some information about what is going on in your system.
Also you did not need to delete it, you could have just emptied it.
IE:

Code: Select all

echo 0 > /var/log/system  
Where it says system, would be any log file, for example again:

Code: Select all

echo 0 > /var/log/messages  

User avatar
sunrat
Administrator
Administrator
Posts: 6412
Joined: 2006-08-29 09:12
Location: Melbourne, Australia
Has thanked: 116 times
Been thanked: 461 times

Re: File system: ROOT -- full -- X closes with error SOLVED

#12 Post by sunrat »

GarryRicketson wrote:You might want to look into "logrotate" and how to configure it:
It is commonly
also used to delete old log files so as not to fill your server's hard drive. ... In stock--snip--
logrotate wouldn't help here. It doesn't delete the latest file, which was the problem here.
Also you did not need to delete it, you could have just emptied it.
Same result. System just creates a new one.
“ computer users can be divided into 2 categories:
Those who have lost data
...and those who have not lost data YET ”
Remember to BACKUP!

p.H
Global Moderator
Global Moderator
Posts: 3049
Joined: 2017-09-17 07:12
Has thanked: 5 times
Been thanked: 132 times

Re: File system: ROOT -- full -- X closes with error SOLVED

#13 Post by p.H »

GarryRicketson wrote:You might want to look into "logrotate" and how to configure it
I doubt that configuring logrotate with a maximum file size would have helped much in this situation.
logrotate runs daily, but the current log files grew up in less that 4 hours.
GarryRicketson wrote:You could, and should have just deleted the .gz files
The old log files are several orders of magnitude smaller than the current ones. Deleting them would have left very little free space, which would have been filled up within minutes if the cause of the huge logs persisted.
GarryRicketson wrote:The system log could contain information about your system, you should have at least reviewed it, it might of had some information about what is going on in your system.
I agree, but we can see that the three huge log files have approximately the same size so it is likely that they all contain a copy of the same huge messages. So the OP could have deleted one or two of them to free up space quickly and reviewed the remaining one.
GarryRicketson wrote:Also you did not need to delete it, you could have just emptied it.
Good point. If the log files are still opened by rsyslog, deleting them would not really delete them (just make them invisible) until rsyslog closes them.

User avatar
debiman
Posts: 3063
Joined: 2013-03-12 07:18

Re: File system: ROOT -- full -- X closes with error SOLVED

#14 Post by debiman »

mike acker wrote:maybe we can get lennart to change the logging mechanism to output into a circular queue and put this problem to bed.
systemd actually does this.
these are non-systemd log files.
but that is beside the point; you have a big fat red warning lamp coming on: you need to look at what creates the errors and fix that.

User avatar
mike acker
Posts: 131
Joined: 2017-06-28 21:23

Re: File system: ROOT -- full -- X closes with error SOLVED

#15 Post by mike acker »

debiman wrote:
mike acker wrote:maybe we can get lennart to change the logging mechanism to output into a circular queue and put this problem to bed.
systemd actually does this.
these are non-systemd log files.
but that is beside the point; you have a big fat red warning lamp coming on: you need to look at what creates the errors and fix that.
something related to pushing that ACDSee program hard inside the Virtual box is almost certainly the problem. several times the ACDSee just hanged and I had to kill the VM in order to get moving again.

most likely the user app had gone into a hard loop part of which involved causing the VM to write to the error logs. this would be the sort of thing that would result in 3 log files each going up to 7GB size

next time i start the Research box I'll look into those log files. I curious to find out if this is a 1-off issue or if running that stack is hard on my log files.
Viva la Resistencia

Post Reply