Scheduled Maintenance: We are aware of an issue with Google, AOL, and Yahoo services as email providers which are blocking new registrations. We are trying to fix the issue and we have several internal and external support tickets in process to resolve the issue. Please see: viewtopic.php?t=158230

 

 

 

MariaDB/MySQL Took a Dump Last Night

If none of the specific sub-forums seem right for your thread, ask here.
Post Reply
Message
Author
Gerowen
Posts: 182
Joined: 2011-04-11 05:12
Location: Kentucky
XMMP/Jabber: gerowen@conversations.im
Has thanked: 5 times
Been thanked: 2 times
Contact:

MariaDB/MySQL Took a Dump Last Night

#1 Post by Gerowen »

So I woke up late today, around 1300, to find that my Nextcloud instance was down. I'm hosting it on Debian Bullseye via the regular old tarball manually set up with Apache, MariaDB/MySQL, PHP, etc. It's been running great for literally years across multiple in-place upgrades to both Nextcloud and Debian.

After doing some tinkering it came to my attention that I MySQL was complaining it couldn't connect to the database. Easy enough I figured, I'll just log into MySQL and see what's wrong. Upon trying to launch the MySQL shell though it would ask for the password and then error out saying it couldn't connect to the server.

Code: Select all

"ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/run/mysqld/mysqld.sock' (111)"
So I thought maybe the .sock file got messed with during an update or something and wasn't being removed properly, so I verified the location of the correct file by looking at the configs, all of which pointed to the same file, and I then deleted that mysqld.sock file and tried restarting MySQL, but still no dice. I tried rebooting the whole server just for kicks, no luck.

I tried reinstalling MariaDB/MySQL but that apparently doesn't get rid of the existing configuration files, so what I ended up doing was apt purge --autoremove on mariadb-server, deleting /var/run/mysql, then reinstalling it and re-importing my most recent database backup (yesterday). It's just a personal instance with myself, my wife and kids on it, and I've got it scheduled to do daily backups of the database, so it wasn't a huge issue. What I'm curious about is why it crapped out in the first place.

While poking around in syslog I found the following line:

Code: Select all

mariadbd[1115]: 2022-01-01 11:55:02 0 [ERROR] [FATAL] InnoDB: You should dump + drop + reimport the table to fix the corruption.
That timestamp is hours after any kind of automatic update/reboot would have taken place.

So something crazy happened that corrupted the actual database, but why would that have broken my ability to log into the MySQL shell to try and correct it? It's saying I should dump and reimport the database, but I couldn't do that without having access to the MySQL shell.

I've checked the logs for apt and I don't see any kind of updates that would have been applied by unattended-upgrades; my last automatic update was December 18th.

Did anybody else have anything happen today with their database? I guess it's definitely possible that Nextcloud encountered some kind of bug and corrupted its database. I've done short SMART tests on all the drives in the system and found no issues, and the server is running on an UPS so there shouldn't have been any kind of power fluctuation or outage to cause any issues. My UPS is reporting no events since the 17th either.

I guess I'm posting all this just to try and fish for thoughts from any of you who may have encountered this kind of thing in the past, or who may have some idea as to what happened. I've restored a backup and everything is fine, but if there's something I can do to prevent the issue in the future, I'd like to do so.

Post Reply