mariadb filled the /var partition - how to fix?

If none of the more specific forums is the right place to ask
Post Reply
Message
Author
mikedodd
Posts: 67
Joined: 2009-03-17 18:44

mariadb filled the /var partition - how to fix?

#1 Post by mikedodd »

I'm running Buster on a home-network server with mariadb and a number of other applications.

Something went wrong with a Windows application that was connected to one of the databases hosted by mariadb on the server. Immediately mariadb spewed out a bunch of error messages to /var/log, completely filling the 229GB /var partition. Now it won't start, probably because there's no space for more logs.

This happened once before, and my solution was to add a second SSD partitioned for use solely by /var.

Now that drive is filled. I tried deleting all large .1, .2, etc. log files in /var/ but df -h still shows 0 space available on /var.

I created a /home/varcopy directory on the main SSD, and copied the entire /var directory it. Sure enough, df -h shows the additional usage in /home, and when I delete large files there, df -h shows more free space in /home. but deleting files in /var does not free-up any space.

How can I recover the space the deleted log files should have freed up? I can't start mariadb with /var filled.

Thanks for any and all advice. I'm stumped.

--- Mike

p.H
Posts: 1969
Joined: 2017-09-17 07:12
Been thanked: 9 times

Re: mariadb filled the /var partition - how to fix?

#2 Post by p.H »

mikedodd wrote: 2021-07-28 18:28 Now it won't start, probably because there's no space for more logs.
Or because mariadb database files are also in /var.
Advice : on a server, use a separate filesystem for /var/log (and everything which may grow out of control).
mikedodd wrote: 2021-07-28 18:28 I tried deleting all large .1, .2, etc. log files in /var/ but df -h still shows 0 space available on /var.
If the deleted files were still held opened by some process, the disk space will not be freed until the files are closed.
When you want to free space used by an open file, you can overwrite it.

Code: Select all

echo > /var/log/logfile
mikedodd wrote: 2021-07-28 18:28 How can I recover the space the deleted log files should have freed up?
Restart rsyslogd or reboot.
Last edited by p.H on 2021-07-31 08:13, edited 1 time in total.

mikedodd
Posts: 67
Joined: 2009-03-17 18:44

SOLVED! Re: mariadb filled the /var partition - how to fix?

#3 Post by mikedodd »

Thanks for the advice. It turns out that the mariadb logs were not the space-eaters -- it was cups.

I poked around the 'Net, and learned about the du command, and du -sh showed that cups logs were using almost the entire 291 GB. I shut down cups and deleted its logs, and the /var partition was back to normal. I started mariadb with systemctl, and it launched and ran fine.

I used systemctl to disable cups. It's not even configured for any of my network printers in the first place, and I never print anything from this server anyway.

Does anyone know of a reason my Debian 10 server needs cups running?

--- Mike

p.H
Posts: 1969
Joined: 2017-09-17 07:12
Been thanked: 9 times

Re: mariadb filled the /var partition - how to fix?

#4 Post by p.H »

mikedodd wrote: 2021-07-30 13:49 cups logs were using almost the entire 291 GB
Hmm. You wrote earlier :
mikedodd wrote: 2021-07-28 18:28 I tried deleting all large .1, .2, etc. log files in /var/
What log files did you delete exactly and how "large" were they in total ? Anything less than 5% of the filesystem size cannot be acounted as "large".
mikedodd wrote: 2021-07-30 13:49 Does anyone know of a reason my Debian 10 server needs cups running?
Only you can tell if your server needs CUPS. Either you chose to install it, or it was pulled by some dependency.

mikedodd
Posts: 67
Joined: 2009-03-17 18:44

Re: mariadb filled the /var partition - how to fix?

#5 Post by mikedodd »

"What log files did you delete exactly and how "large" were they in total ? Anything less than 5% of the filesystem size cannot be acounted as 'large'."

There were approximately 10 files in /var/log/cups, and the largest one was 290 GB in size -- nearly the size of the entire partition. IMO, that qualifies as (very) "large." :-)

Why was cups installed? I don't know. It's entirely possible that I selected it during installation, thinking I might, sometime, want to print something from the server. No matter; it's disabled, and df -h shows /var is using only 1% of its partition.

--- Mike

p.H
Posts: 1969
Joined: 2017-09-17 07:12
Been thanked: 9 times

Re: mariadb filled the /var partition - how to fix?

#6 Post by p.H »

Please pay attention. You wrote first
mikedodd wrote: I tried deleting all large .1, .2, etc. log files in /var/
before later finding out that
mikedodd wrote: cups logs were using almost the entire 291 GB
My question was about the supposedly "large .1, .2, etc. log files in /var/" you deleted the first time.

Post Reply