Well, folks, its been a heck of a day. Believe it or not, the website, in general has a 99.994% uptime statistic, but we sort of tarnished that today.
What happened?
The short answer is that the 500GB hard drive that runs the website, etc. overflowed during a backup. That overflow caused the worst corruption problems we've had with the DB since I've been here. Usually, we've just had to run a mysqlchk and we're good to go. One of the things I've liked about mysql is that it really is pretty hardy.
This time that didn't happen. This time, for the first time since I've been here, I had to bring the whole database down and run a separate program to solve the issue. While that took a couple of hours, it did solve it and we have no database loss because of this issue.
Now, that won't happen again. Why? An entirely new 500GB hard drive has been purchased and put on Cirrus for the purposes of backups and SQL logs. Before we had about 10% of available space on Cirrus, which may not seem like a lot, but in reality its several GB of space. However, the file in question was more and that's what caused the problem. Now, after the change I made, we have 35% of availalbe space on the hard drive, and I suspect we'll find more as we do another check. Historically we've tried to be somewhat frugal with the hard drive space on Cirrus in order to best use the space, but that decision needed to be somewhat reevaluated, so I made the call to get the new hard drive.
Based on checks by both myself and Will, everything again looks like its nicely up and running.
I very much regret the downtime today and look forward to getting back to our normal 99.94% general web uptime. Thanks go out to Sara, Matt, Stella, Will, and several of our observers and members for their help and support today. Thank you all.
And now let's get back to "Clear Skys!"
Doc Kinne
AAVSO Astronomical Technologist
"You never miss the water 'til the well runs dry!"
Good work. I figured it was something like a server problem, so I didn't complain.
Lew
Hi,
Thanks, Doc, for giving up your Saturday.
You're a star!
John
Mike
Thanks guys
Mike
Great work Doc! But, I am a little confused, I thought it was all running on the Amazon cloud servers, why would you need to buy a new disk?
Also, I though cloud servers gave you "overdraft privileges" rather than hard limits on space? Any explanation would be appreciated. Thanks,
Mike
Considering the size and complexity of all this, thanks for getting the site back up so fast!
Might I suggest a Facebook and or Tweet post when unanticipated maintenance occurs? That day I went to each of the AAVSO FB and Twitter accounts to see if there was some news about this. Alas, there was not.
Michael