Update 1/11: Bot is coming back online. It has around 2 million submissions it needs to process which will take most of the night.

If your Subreddit is registered, it will start checking submissions from 2024-01-09 23:51:42 forward.

I apologize for the extended downtime. This was a bit of a doozy. I'm getting a much better backup plan in place so if something like this happens again it should be pretty fast to restore.

------------------------------------------------------------------------------------------------

I had some type of crash on my NAS that took down all my VMs.

I have everything back online but MySQL is not happy and something got corrupted. I'm currently pulling a backup of the MySQL VM before I start trying recover it.

I don't have an ETA at this point but it will probably be down for most of today.

Update: Database is completely corrupt. I'll be restoring from a backup when I get out of work tonight.

Update 1/9: Still working on it. There were some issues with my backup process that I'm working around.

Update 1/10: Had limited time to work on it today. I did get the backup working and I'm currently importing the table data. It has a few hours left on it.

Looks like it will be back up and running tomorrow afternoon.

For any nerds that want details, the database server is Percona MySQL, roughly 1tb in size and has around 2 billion rows. I take backups with Xtrabackup. However, what I did not realize is backing up a single database, verses the whole server with Xtrabackup makes the restore process a pain. Instead of being able to execute a single restore, each table has to be imported manually. Along with that the Xtrabackup Prepare command different than with a full server backup. I messed the prepare step up so my local backup copy is junk. I'm currently waiting for a clean backup copy to download for Google Drive before I attempt another restore.

Suffice to say my disaster planning wasn't great and I've never actually tested a full restore. Once we're back up and running I'm putting together and much more robust process.