This last week has been crazy – a study of what can go wrong with even a simple plan. It would be comedy if user’s websites and businesses weren’t at stake.
On a group of shared hosting servers I manage, we rely on a local backup to an internal drive in each box and a periodic remote backup. The internal backups run nightly and backup all home directories, database files and website files. The remote backups FTP a full backup from the server appliance to a remote FTP server in the same datacenter. These appliance backups vary depending on the appliance (Ensim or Plesk) but each archives each account and some or all of the appliance itself and its settings.
Early last week, after some performance issues, we tasked the datacenter to add another stick of memory to the box. Afterward, the box wouldn’t boot – kernel panic trying to open the main volume. This is normally caused by a forced shutdown; commodity (ie. cheap) drives can spew data across a volume, corrupting good data. If it happens to corrupt the system’s fstab (File System Table), the entire drive can become unusable. Another major cause is bad memory causing a system error on boot which in turn causes a kernel panic and forced shutdown.
To make a long story short, the datacenter installed a new drive, re-imaged with Redhat and Ensim and set the corrupted drive as a slave. Then they noticed that there was one too many drives in the server. We were paying for one additional drive, but the server actually contained two. One, apparently being left over from another HD crash that they never removed. They removed the extra drive and notified us. When I finally could login to the server (I didn’t know imaging a drive could take four hours), they had removed our good backup drive with the previous night’s backup run. By the time the wonderfully proficient tech support located the drive, it was in the midst of a disk test. All the most recent backup data was gone.
The only option now was to try to recover as much as possible off of the corrupted drive. In the next few days, I’ll post some of the steps I used to remount the corrupted drive and the attempts at data recovery.
The whole point… BACKUP YOUR DATA! You never know when something is going to happen. Also worthy of note, is to verify your backups periodically. A backup is worthless if you can’t open it. Just take the time to download a day’s backup files every month and unzip them to make sure they are valid and that the data is good.
Links
These applications are old and simple, but they work (and they are easy to configure).
Popularity: 1%
Most Popular Posts
