|Notes from a disk rescue||[Jul. 3rd, 2008|04:09 pm]|
For 9 or 10 months, I've had a media server at home running MythTV on Ubuntu. It does the usual DVR operations and network streaming fairly well (there's only one tuner, so we can only receive one channel at once, but it's quite possible to, for example, watch two recordings at once in different places). On Monday, we noticed that machine was complaining of disk errors in syslog.|
Rescuing 500GB is not fun, but every last byte was eventually recovered. The main tool used was GNU ddrescue (not to be confused with dd_rescue). It can do partial copies (leaving blanks where errors occur), keep a state file to cope with interruptions, go back to the areas with errors and do binary chops to retrieve as much information as possible. That's essentially what I did (first pass -n, then ran with -r 3 a few times, using a log file (and sacrificing a steady stream of small rodents) throughout). A very odd thing happened when the first pass hit the first few errors, at about 380GB: the speed of copying slowed down from about 50MB/s to about 7MB/s. At first I thought some increased error correction/paranoia had kicked in somewhere in the hardware, but I realised the disk or head might be in a physically disadvantageous state, so as a shot in the dark, I stopped the copy, did hdparm -Y (sleep command) and resumed. This worked; it was 50MB/s again. Determining why this voodoo worked is left as an exercise for the reader. :-)
Getting into an environment where I could run ddrescue was slightly complicated, because the only big enough disk I had available to rescue onto was the replacement disk, both of them are SATA, and no machine in the building has more than two SATA ports, meaning I needed to borrow the IDE disk from dougalwuff's machine to boot (I didn't like the idea of doing this from a CD, needed somewhere to keep the state file and had no USB keys in the building). Since the new disk is twice the size, I could have done a temporary install onto it first, but I wouldn't then have been able to easily grow the filesystem afterwards to the full size of the disk (asking said temporary install to obliterate itself). As a final piece of silliness, I will of course need to redo grub onto the new disk by running grub-install on the media server itself (running it on the spare machine didn't seem to produce something bootable).