Data Loss Announcement
.thumb.png.930e14f6185274fc318c83d22a72a188.png)
Entry posted by Admin ·
15,531 views
First, so you don't have to read through all my rambling, the website database is set back to November 2010, and there is no way around it.
I was working with the forum software on Sunday, 4-17, and I was just very pleased with how things were working. I saw there was an update for the bug tracker application, and the change list just mentioned some bug fixes, so I felt okay installing it. The new version removed custom issue fields, which we use extensively for listing hardware, drivers, operating systems, etc.
It's not very easy to revert to previous software versions. I had performed a full website backup earlier that day, so I figured the best way to revert the tracker app would just be to have the IT admin revert the server to the backup I had made. I'd downloaded it to my local hard drive and it was intact, and also still available on the server. I also had an earlier full website backup I made on April 2. I figured this would be a minor inconvenience that would take the forum offline for a few hours.
Here's the interface in CPanel:
I also talked to the IT admin and they assured me that the full website backup does includes MySQL databases. I sent the following message to them Sunday afternoon:
There is a backup file on my server called “__________.tar.gz”. I would like my domain www.leadwerks.com restored to the state it was at the time this backup was created.
Please note we have several MySQL databases and some complex forum software installed. I need the backup to restore everything it to its previous state at the time the backup was created. I chose “Download or generate a full website backup” in CPanel > Backups to generate this file.
My caution in this email might indicate I had a suspicion something was wrong. Did I somehow know something would go wrong? Should I have? This thought is nagging me right now.
I've been on the phone with tech support for a couple days, but you don't need the whole story. The outcome is the server was erased and repopulated with the contents of the backup archive. All files were intact, but the databases were not saved when the backup was performed. The previous backup I had was from April 2, which would not be a terrible loss. It appears the same occurred then. In fact, since I have been dutifully saving full website backups since November, thinking I was making extra effort to be on the safe side, the databases were never being saved in the archive. So the last copy of the forum database I have is from November 2010. I've been quite diligent with backing the site up, but that doesn't mean anything if the backups aren't working. The host automatically backs up sites under 10 gb every 24-36 hours, but that size excludes us.
Although it was easy to bring the forum software up to date, all forum data since November 2010 is gone forever. The loss of this is staggering. To me, and to others here.
You can get your screenshots here in a single package. They're randomly named, but if you really need something you can find it, and it's divided up by month:
http://leadwerks.com/werkspace/index.php?/topic/3098-screenshot-archive/page__pid__28584#entry28584
I still have all the attached files from posts, blogs, downloads, etc., but they are randomly named and I don't know how possible it is to go through them.
The thing that really hurts is the lost documentation that was created after November. That's going to take time to recover from. We'll recover, but the loss is still sinking in for me. Jorn worked really hard on that at a reasonable pay rate and I feel really bad about it.
I have records of all registered users. If you registered for your Leadwerks account after November of 2010, your account will be recreated and your password will be set to your registration key you received.
The wiki and old forum are fine.
The new forum skin is fine, but not installed at the moment.
This must never happen again.
First, a better backup protocol is needed. The most important thing is that the backups actually be valid backups. Obviously, I've learned that MySQL databases must be backed up individually and that a full website backup from CPanel is not reliable. Data is saved to a 2 TB external hard drive. Additionally, I am opening a safe deposit box where code and site data are regular deposited. I already thought I was doing everything right, so at this point I feel like I can't be too careful.
Second, I am locking the forum software at the current versions, unless a critical vulnerability is discovered. In 2009, all we had was an installation of PHPBB. The idea for Werkspace was something Annika came up with and gradually sold me on. We listed all the features we wanted and found forum software that would allow it. Customizing the look and feel of the software to fit in with the theme of Leadwerks was a long process, but we finally got a suitable skin made. I am satisfied with the features and functionality of the system, and now we can just leave it be, indefinitely. The data loss does not affect the skin we had developed. The skin will continue to get minor improvements, but those kind of changes are easily performed and rolled back, if need be.
Third, semi-annual tests are needed to make sure data can be restored successfully to a test server. Backups don't help if you wait until you need them to find out they don't work. A test server will be used to install the latest backup on every six months, or before any restoration takes place. Maybe this is overkill, but nothing was supposed to go wrong before, and we found out otherwise.
It seems redundant to say I'm sorry about this occurrence. I know we'll get back to normal, but it hurts pretty bad. The key point is my data backup technique was flawed, so all the backups I've been performing were worthless.
-Josh
41 Comments
Recommended Comments