Broken hard drive woes and viable backup options for huge data amounts

Hardware sucks. It just totally and utterly sucks.

I purchased a 300 GB hard drive (plus a 3.5" USB disk drive enclosure) roughly 8 months ago which I encrypted using dm-crypt and then used it as a backup medium. And now the disk has died. No, this is not a software problem (that would be easy to deal with), the hardware is simply dead (it's making funny "klck" noises all the time).

Mounting it via USB (using the USB enclosure) doesn't work at all anymore. I connected the disk via IDE in a PC and was able to mount it just fine (with cryptsetup luksOpen and all). A simple ls works, too (at least in the top-level directory). So I tried to copy the data over to another medium, but when accessing certain directories or files the system simply hangs and I get tons of funny kernel errors (and the disk makes even more and louder funny noises). Great.

Stupid as I am, I also put data on the drive which I did not have backups of on other media (mostly because the data is huge, e.g. conference video recordings, large amounts of Creative Commons MP3s etc). OK, so I managed to get at least some small parts of my data, but now the disk is completely dead, I fear.

I'll try to convice the vendor to give me a new drive, but I won't let them get their fingers on my data, i.e., I will not send the disk to them (yes, even though it is encrypted; crypto can be broken). Any ideas what else I could do? A professional "data recovery" service is not an option either, they usually cost 1000-2000 Euros (in the best case).

What do you do for storing huge amounts of data nowadays? The only viable option I can think of is a RAID-5 (mdadm) with 3 or more disks in a silent PC which I can leave turned on 24/7. Unfortunately that costs non-trivial amounts of money. CDs are too small and they don't last too long either; the same is true for DVDs.

Sigh...

Comments

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

Hello! I read this guide to

Hello! I read this guide to online backup and must admit it’s great. Complete, thorough and not too complicated.

The funny thing is that

The funny thing is that right now I am hearing the same "klck', well it's not that funny, I expect my hard drive to crush next weeks. I know the "symptoms", last time I heard that I end up desperately searching for hard drive recovery solutions.

Wiping Contents

If the goal is to wipe the data off the hard drive so that it can be sent back, I assume for some kind of financial benefit, then why does the solution have to software? Let's not forget what a hard drive is - magnetic media. Why not unscrew the hard drive, get some of those strong rare-earth magnets from a craft store, and just make every bit a 1 the brute-force way?

wiping hard drive

Using magnets is not a ideal way to get rid of data. If you intend on selling a second hand drive just be careful. Anything can usually be retrieved with the correct software

Urban legends

You really need a big ass magnet to do any harm to the data on current hard drives. The data on current hard drives is so dense by now, that it is compleatly enough to overwrite the whole disk with some garbage data and _NO_ software can retrieve them afterwords. There may only be some sectors on the disk that you do not reach by software, to overwrite, because they got hidden due to a error correction or some internal logic of the disk. But if you could not reach them to write data to, then another person will not be able to read them either, not without real indeep knowledge of at least the ATA protocol. ^^

had similar problem

Small transfers (200MB or so) would turn out corrupted and big ones would just fail. HD was making noises too. I thought It was over with the HD (and the damn thing was pretty new) but turned out it was the USB enclosure. I gave the USB thing to a friend (trying to get rid of it) and the same thing happened.

Gruss
P

USB-enclosure

You mean the USB-enclosure was broken, or the USB-enclosure broke the harddrive?

I connected the disk in a normal PC via IDE and it didn't work there either, so it's not the USB-encolure itself which is broken...

Switching acoustic

Switching acoustic management to quit (128) seems to increase the lifetime of harddrives.
I think you can use dd-rescue even for encrypted discs by using /dev/mapper/crypt instead of /dev/hdax but I do not know what dm-crypt does if an read-error occurs.

a few points

a) don't worry about them reading the disk. They've got a legal and moral reason not to do it; because you've encrypted the filesystem they also have a large technical barrier to surmount. If you are paranoid enough not to let an encrypted disk out of your hands, then you need to be prepared to write off the cost of a hard disk now and then :)

b) have you tried asking it to perform SMART self-tests? "smartctl" is a very useful tool. Ask it to perform a short, conveyance and long self-test and see if they report success or failure.

c) as you put more disks in a RAID-5, the likelyhood of a failure goes up (law of averages). As RAID-5 can only survive one disk failure (same as RAID-1) it's not any more safe than just using mirrored disks.

Stuff

a) Yes, I'm paranoid enough ;-) The data on this particular drive is not that important (mostly publically available MP3s, videos etc.), but in general, if I have a choice I'd rather choose the lower risk, i.e. not giving away the disk at all (even if it's encrypted).

b) I don't think smartctl will help anymore at this stage. I'm 100% sure I have a hardware failure; smartctl will most probably just hang (as does any other operation which accesses the disk). I'll give it a try, though...

c) Yep. RAID-1 is probably better in my case, except if I plan to add more drives later in which case I'd rather use RAID-5 in order to get more disk space in total...

Thanks for the "few points" :)

Thoughts on storage

RAID is not a backup. ;-)

I've been using the RAID5-in-a-PC method as a way of backing up data for a bit over a year now, with the 'main' copies either living on my laptop or a USB hard drive. During that time, I've had two drives out of the four fail (one WD and one Seagate). Consumer hard drives just don't seem to like being spun up permanently, and they don't like high temperatures. (One thing I've been meaning to look into for a while is making sure the drives are spun down for most of the time. Linux sucks for this.) The machine I have is quiet, but far from silent - if you went any quieter, you'd be keeping the drives at still higher temperatures and increasing the chance of failure.

The upside of the Linux+RAID approach is that smartd can monitor your drives and tell you about failures well before you lose any data. You can maintain regular backups of several machines automatically using rsync and cron. The downside is the cost - not just the cost of building the machine, but also of keeping it powered up all the time.

Cheaper than a dedicated box for this would be multiple USB drive boxes, whether RAIDed or just with copies of the data on each. You win on price, drive longevity (since they're powered down mostly), power usage and your backups can be carried around in a backpack. You lose on convenience, and have no access to SMART monitoring.

RAID5-in-a-PC

Hm, maybe a RAID-5 (or RAID-1), but not having it turned on 24/7 would work? I.e. only power it on once a week or so for a backup-run?

I won't buy any "consumer" drives anymore. Drives which are explicitly marked as "RAID-suitable" should be a lot more robust, and they don't cost a lot more than "consumer" disks...

And yeah, RAID is not backup ;-) But I want backups and I have to store them somewhere; a RAID is better for that than a single drive.

Uwe.

Don't worry about samsung

Don't worry about samsung breaking the encryption on the drive. If they had the time and desire to do that they could just as easily break into your house to install hardware trojans, etc.

Try the freezer trick. Wrap the drive in plastic. Put it in the freezer for 6+ hours (overnight, a full day, whatever). Remove it from the freezer. Leave wrapped in plastic. Try to read/recover data every half hour. My drive needed about 2 hours to warm up before it would work.

Freeze

Yep, will try. Why "leave it wrapped in plastic"? Does it make a difference if I remove the plastic?

Keeping frozen drives in plastic.

I think that it is simply to avoid water condensation from the air.

gddrescue

using ddrescue, you can have your filesystem back on another drive. then you fsck it, and get the files you need. (this means having a bigger storage device than your 300G).

ddrescue --verbose /dev/hdc1 hdh1 hdc1.log

works wonders.

dd_rhelp is a wrapper for

dd_rhelp is a wrapper for ddrescue that makes it even better. Especially since you can keep going through power cycles and or cooling cycles. You even get a visual representation of what is being saved.

ddrescue

I would agree in normal cases, but it's a bit more complicated in my case as I have a completely encrypted disk. Not sure whether I will be able to even decrypt/mount it when a few sectors are damaged (probably depends on which sectors are damaged).

Anyway, in this case it's more like a few sectors which are not damaged (almost all seem to be damaged!); almost any disk access hangs, so I think I'm out of luck...

Try to put it in a fridge

Try to put it in a fridge for a day or two (vacuum packed [or "safe enough that it cannot get wet"]) and then retry to copy the data.

>What do you do for storing huge amounts of data nowadays?
I'm curretly setting up a new fileserver. I'll use a jetway embedded board and a raid 0+1. not the cheapest solution but IMHO solid.

a alternative would be a Linksys NSLU2.

for a second backup I use DVD Rams. (25-50 years warranty and even with some holes in it you should be able to recover some data [Verbatim])

NSLU2

Yeah, I've been thinking about a NSLU2... It adds to the total cost of course (as I don't own one yet), but I guess it's the most quiet solution!?

I'll have to read some documentation on the NSLU2, e.g. whether I can set up a RAID + LVM + dm-crypt using the NSLU2, but I guess the answer is yes...

Depending on the nature of the failure

Spinrite may or may not be able to recover the data. It is not free in any sense though Steve Gibson, the guy that writes it, does offer a money back guarentee if buy it to recover data and it doesn't work. I've no affliation with Steve or grc.com, just a happy customer.

I've used it to recover a few drives suffering from the click of death.

As for backup solutions for large amounts of data, I have my /home in a software raid 5, + 4 hour rsync's to another disc in the same system + a nightly rsync to another a server I have in a data center which is on a software raid 1.

Spinrite

Hm, apart from the fact that Spinrite seems to be a Windows application (and I don't use Windows), and that it's a closed-source application (and I don't like closed-source software), I'm not too optimistic that it could handle dm-crypted disks well!?

Yeah, RAID seems to be the only viable solution here. I need to find a really silent/cheap passively-cooled mainboard (or embedded device, maybe a NSLU2) and some very silent disks...

Spinrite is a DOS

Spinrite is a DOS application, not Windows, and can be configured on a bootable floppy or CD with its own DOS clone. As such it will work for Windows, Linux, even OS X, and whether the data is encrypted or not doesn't matter: bits are bits.

re: hd failure

Hi there, just thought i would mention http://geeksaresexy.blogspot.com/2006/01/freeze-your-hard-drive-to-recover-data.html , sometimes you can be "lucky" and the hard disk after cooled for a while, will spin up long enough to mount and backup the data. Helped me in the past.

I've found this to be

I've found this to be effective on older drives that usually suffered an untimely death due to "stiction" where the bearings would sieze after being run for long periods of time. After being shut down, the platters would fail to spin back up. Freezing the drive would shrink the metals in spindle at different rates, freeing it up so it could spin again.

freeze hard drive

Wow! This is just crazy ;-) But it might work, so I'll probably try it. Thanks for the pointer!

Uwe.

seems like the right advice

I've restored 20GB data from IBM TroubleStar cooling notebook with big German air-fan from the bottom-side.

Was it a maxtor drive?

Was it a maxtor drive?

Samsung

No, Samsung. Does that matter?

My personal advise for such

My personal advise for such data amounts is simply... another harddisk. Really.
Whenever you deem necessary, make a cp -a or tar cf to that disk. Or if you want, create a RAID 1 (mdadm) on it.
The just released mdadm 2.6 has an (experimental) --incremental parameter to be used with udev to automatically add a newly plugged in disk to the array it belongs to. So just plug the disk in once a month or so. And subscribe to the linux-raid mailinglist to submit feedback :)

Yeah, I'm thinking of doing

Yeah, I'm thinking of doing something like that.

cp or tar (or rsync) are not an option, as I want to have the same state all the time, and not only when I explicitly sync the data (which means work, and I could forget to do it often enough).

RAID1 is nice as it only needs 2 drives, RAID5 would have the benefit of more space as you add more disks, but unfortunately more disks cost more money ;-)

Hmm, need money...

One more thing

A raid1 or raid5 won't cover all your bases. You'll still have uncovered failure modes:
1) entire machine goes up in smoke/flames/drowns
2) Filesystem corruption
3) User error (delete something you actually wanted to keep)

RAID

Sure, that's clear. I just don't want to lose data from failing disks anymore. That's a start ;-)

If you want the same state everywhere

perhaps you should look into OpenAFS?

OpenAFS

Yeah, I'll try out OpenAFS sooner or later anyway, but in this case I meant "same state everywhere" as in RAID-1/RAID-5 (i.e. not having to do manual cp/rsync all the time).

Uwe.