Yes, it's that time of the year again. A disk in my desktop-replacement laptop with 2 disks and a RAID-1 has died. Time for recovery.
This laptop has been running 24/7 for the last 3 years or such, so it's not too surprising that a disk dies. Surprisingly though, for the first time in a long series of dead disks, smartctl -a does indeed show errors for this disk. Here's a short snippet of those:
$ smartctl -a /dev/sda [...] Error 1341 occurred at disk power-on lifetime: 17614 hours (733 days + 22 hours) When the command that caused the error occurred, the device was active or idle. After command completion occurred, registers were: ER ST SC SN CL CH DH -- -- -- -- -- -- -- 40 41 02 1f c0 9c 40 Error: UNC at LBA = 0x009cc01f = 10272799 Commands leading to the command that caused the error were: CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name -- -- -- -- -- -- -- -- ---------------- -------------------- 60 f8 08 20 c0 9c 40 00 41d+01:51:50.974 READ FPDMA QUEUED 60 08 00 18 c0 9c 40 00 41d+01:51:50.972 READ FPDMA QUEUED ef 10 02 00 00 00 a0 00 41d+01:51:50.972 SET FEATURES [Reserved for Serial ATA] ec 00 00 00 00 00 a0 00 41d+01:51:50.971 IDENTIFY DEVICE ef 03 45 00 00 00 a0 00 41d+01:51:50.971 SET FEATURES [Set transfer mode] SMART Self-test log structure revision number 1 Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error # 1 Short offline Completed: read failure 90% 20511 156170102 [...]
The status of the degraded RAID array looks like this:
$ cat /proc/mdstat Personalities : [raid1] md1 : active raid1 sdb7 409845696 blocks [2/1] [_U] md0 : active raid1 sda6 sdb6 291776 blocks [2/2] [UU]
The [_U] means that one of two disks has failed, it should normally be [UU]. There are two RAID-1s actually, a small md0 (sda6 + sdb6) for /boot and the main md1 (sda7 + sdb7) which holds the OS and my data. Apparently (at first at least), only sda7 was faulty and got kicked out of the array:
$ dmesg | grep kick md: kicking non-fresh sda7 from array!
Anyway, so I ordered a replacement disk, removed the dead disk (I checked the serial number and brand before, so I don't accidentally remove the wrong one), inserted the new disk and rebooted.
Note: In order for this to work you have to have (previously) installed the bootloader (usually GRUB) onto both disks, otherwise you won't be able to boot from either of them (which you'll want to do if one of them dies, of course). In my case, sda was now dead, so I put sdb into its place (physically, by using the other SATA connector/port) and the new replacement disk would become the new sdb.
After the reboot, the new disk needs to be partitioned like the other RAID disk. This can be done easily by copying the partition layout of the "good" disk (now sda after the reboot) onto the empty disk (sdb):
$ sfdisk -d /dev/sda | sfdisk /dev/sdb
Specifically, the RAID disks/partitions need to have the type/ID "fd" ("Linux raid autodetect"), check if that is the case. Then, you can add the new disk to the RAIDs:
$ mdadm /dev/md0 --add /dev/sdb6 $ mdadm /dev/md1 --add /dev/sdb7
After a few hours the RAID will be re-synced properly and all is good again. You can check the progress via:
$ watch -n 1 cat /proc/mdstat
You should probably not reboot during the resync (though I'm not 100% sure if that would be an issue in practice; please leave a comment if you know).
Also, don't forget to install GRUB on the new disk so you can still boot when the next disk dies:
$ grub-mkdevicemap $ grub-install /dev/sdb
And it might be a good idea to use S.M.A.R.T. to check the new disk, just in case. I did a quick run for the new disk via:
$ smartctl -t short /dev/sdb # Wait a few minutes after this. $ smartctl -a /dev/sdb [...] SMART Self-test log structure revision number 1 Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error # 1 Short offline Completed without error 00% 22 - [...]
Looks good. So far.
This is what I set up for backups recently using a cheap USB-enclosure which can house 2 SATA disks and shows them as 2 USB mass-storage devices to my system (using only one USB cable). Without any further introduction, here goes the HOWTO:
First, create one big partition on each of the two disks (/dev/sdc and /dev/sdd in my case) of the exact same size. The cfdisk details are omitted here.
$ cfdisk /dev/sdc $ cfdisk /dev/sdd
Then, create a new RAID array using the mdadm utility:
$ mdadm --create /dev/md0 --level=1 --raid-devices=2 /dev/sdc1 /dev/sdd1
The array is named md0, consists of the two devices (--raid-devices=2) /dev/sdc1 and /dev/sdd1, and it's a RAID-1 array, i.e. data is simply mirrored on both disks so if one of them fails you don't lose data (--level=1). After this has been done the array will be synchronized so that both disks contain the same data (this process will take a long time). You can watch the current status via:
$ cat /proc/mdstat Personalities : [raid1] md0 : active raid1 sdd1 sdc1 1465135869 blocks super 1.1 [2/2] [UU] [>....................] resync = 0.0% (70016/1465135869) finish=2440.6min speed=10002K/sec unused devices:
Some more info is also available from mdadm:
$ mdadm --detail --scan ARRAY /dev/md0 metadata=1.01 name=foobar:0 UUID=1234578:1234578:1234578:1234578 $ mdadm --detail /dev/md0 /dev/md0: Version : 1.01 Creation Time : Sat Feb 6 23:58:51 2010 Raid Level : raid1 Array Size : 1465135869 (1397.26 GiB 1500.30 GB) Used Dev Size : 1465135869 (1397.26 GiB 1500.30 GB) Raid Devices : 2 Total Devices : 2 Persistence : Superblock is persistent Update Time : Sun Feb 7 00:03:21 2010 State : active, resyncing Active Devices : 2 Working Devices : 2 Failed Devices : 0 Spare Devices : 0 Rebuild Status : 0% complete Name : foobar:0 (local to host foobar) UUID : 1234578:1234578:1234578:1234578 Events : 1 Number Major Minor RaidDevice State 0 8 33 0 active sync /dev/sdc1 1 8 49 1 active sync /dev/sdd1
Next, you'll want to create a big partition on the RAID device (cfdisk details omitted)...
$ cfdisk /dev/md0
...and then encrypt all the (future) data on the device using dm-crypt+LUKS and cryptsetup:
$ cryptsetup --verbose --verify-passphrase luksFormat /dev/md0p1 Enter your desired pasphrase here (twice) $ cryptsetup luksOpen /dev/md0p1 myraid
After opening the encrypted container with cryptsetup luksOpen you can create a filesystem on it (ext3 in my case):
$ mkfs.ext3 -j -m 0 /dev/mapper/myraid
That's about it. In future you can access the RAID data by using the steps below.
Starting the RAID and mouting the drive:
$ mdadm --assemble /dev/md0 /dev/sdc1 /dev/sdd1 $ cryptsetup luksOpen /dev/md0p1 myraid $ mount -t ext3 /dev/mapper/myraid /mnt
Shutting down the RAID:
$ umount /mnt $ cryptsetup luksClose myraid $ mdadm --stop /dev/md0
That's all. Performance is shitty due to all the data being shoved out over one USB cable (and USB itself being too slow for these amounts of data), but I don't care too much about that as this setup is meant for backups, not performance-critical stuff.
Update 04/2011: Thanks to Bohdan Zograf there's a Belorussian translation of this article now!
As I have bought a new 300 GB external USB disk drive on Friday, I have tried something new this time: disk encryption using dm-crypt / LUKS. It has been suggested to me multiple times that dm-crypt is superior to loop-aes, however I didn't get a real reason. Yes, it doesn't require any kernel patches and is easier to setup. But has any serious cryptographer looked at it sharply, yet? Did it withhold his eye contact?
Anyways, here's how I encrypted my 300 GB drive. I largely followed the guide at the EncryptedDeviceUsingLUKS wiki page...
badblocks -c 10240 -s -w -t random -v /dev/sdb
/dev/sdbwith whatever is correct on your system. If you're really paranoid, and are willing to wait one or two days, do this:
dd if=/dev/urandom of=/dev/sdb
apt-get install cryptsetup
cryptsetup --verbose --verify-passphrase luksFormat /dev/sdb1
cryptsetup luksOpen /dev/sdb1 samsung300gb
mkfs.ext3 -j -m 1 -O dir_index,filetype,sparse_super /dev/mapper/samsung300gb
mount /dev/mapper/samsung300gb /mnt/samsung300gb
/mnt/samsung300gbwill be encrypted transparently.
cryptsetup luksClose /dev/mapper/samsung300gb
After unmounting, nobody will be able to see your data without knowing the correct passphrase. Drive is stolen? No problem. Drive is broken, and you want to send it in for repair without the guys there poking in your data? No problem. You leave the USB drive at home and some jerk breaks into your house, steals your drive, rapes your wife, and kills your kids? No problem. Well, sort of, but you get the idea ;-)
There's more things you can do, thanks to LUKS: have multiple passphrases which unlock your data, change/add/remove passphrases as you see fit, etc.
Update 2006-04-17: You have to use cryptsetup from unstable if you want LUKS support. cryptsetup in testing does not support this (thanks Ariel).