backup

How to setup an encrypted USB-disk software-RAID-1 on Debian GNU/Linux using mdadm and cryptsetup

This is what I set up for backups recently using a cheap USB-enclosure which can house 2 SATA disks and shows them as 2 USB mass-storage devices to my system (using only one USB cable). Without any further introduction, here goes the HOWTO:

First, create one big partition on each of the two disks (/dev/sdc and /dev/sdd in my case) of the exact same size. The cfdisk details are omitted here.

  $ cfdisk /dev/sdc
  $ cfdisk /dev/sdd

Then, create a new RAID array using the mdadm utility:

  $ mdadm --create /dev/md0 --level=1 --raid-devices=2 /dev/sdc1 /dev/sdd1

The array is named md0, consists of the two devices (--raid-devices=2) /dev/sdc1 and /dev/sdd1, and it's a RAID-1 array, i.e. data is simply mirrored on both disks so if one of them fails you don't lose data (--level=1). After this has been done the array will be synchronized so that both disks contain the same data (this process will take a long time). You can watch the current status via:

  $ cat /proc/mdstat
  Personalities : [raid1]
  md0 : active raid1 sdd1[1] sdc1[0]
        1465135869 blocks super 1.1 [2/2] [UU]
        [>....................]  resync =  0.0% (70016/1465135869) finish=2440.6min speed=10002K/sec
  unused devices: 

Some more info is also available from mdadm:

  $ mdadm --detail --scan
  ARRAY /dev/md0 metadata=1.01 name=foobar:0 UUID=1234578:1234578:1234578:1234578

  $ mdadm --detail /dev/md0
  /dev/md0:
          Version : 1.01
    Creation Time : Sat Feb  6 23:58:51 2010
       Raid Level : raid1
       Array Size : 1465135869 (1397.26 GiB 1500.30 GB)
    Used Dev Size : 1465135869 (1397.26 GiB 1500.30 GB)
     Raid Devices : 2
    Total Devices : 2
      Persistence : Superblock is persistent
      Update Time : Sun Feb  7 00:03:21 2010
            State : active, resyncing
   Active Devices : 2
  Working Devices : 2
   Failed Devices : 0
    Spare Devices : 0
   Rebuild Status : 0% complete
             Name : foobar:0  (local to host foobar)
             UUID : 1234578:1234578:1234578:1234578
           Events : 1
      Number   Major   Minor   RaidDevice State
         0       8       33        0      active sync   /dev/sdc1
         1       8       49        1      active sync   /dev/sdd1

Next, you'll want to create a big partition on the RAID device (cfdisk details omitted)...

  $ cfdisk /dev/md0

...and then encrypt all the (future) data on the device using dm-crypt+LUKS and cryptsetup:

  $ cryptsetup --verbose --verify-passphrase luksFormat /dev/md0p1
  Enter your desired pasphrase here (twice)
  $ cryptsetup luksOpen /dev/md0p1 myraid

After opening the encrypted container with cryptsetup luksOpen you can create a filesystem on it (ext3 in my case):

  $ mkfs.ext3 -j -m 0 /dev/mapper/myraid

That's about it. In future you can access the RAID data by using the steps below.

Starting the RAID and mouting the drive:

  $ mdadm --assemble /dev/md0 /dev/sdc1 /dev/sdd1
  $ cryptsetup luksOpen /dev/md0p1 myraid
  $ mount -t ext3 /dev/mapper/myraid /mnt

Shutting down the RAID:

  $ umount /mnt
  $ cryptsetup luksClose myraid
  $ mdadm --stop /dev/md0

That's all. Performance is shitty due to all the data being shoved out over one USB cable (and USB itself being too slow for these amounts of data), but I don't care too much about that as this setup is meant for backups, not performance-critical stuff.

Update 04/2011: Thanks to Bohdan Zograf there's a Belorussian translation of this article now!

Broken hard drive woes and viable backup options for huge data amounts

Hardware sucks. It just totally and utterly sucks.

I purchased a 300 GB hard drive (plus a 3.5" USB disk drive enclosure) roughly 8 months ago which I encrypted using dm-crypt and then used it as a backup medium. And now the disk has died. No, this is not a software problem (that would be easy to deal with), the hardware is simply dead (it's making funny "klck" noises all the time).

Mounting it via USB (using the USB enclosure) doesn't work at all anymore. I connected the disk via IDE in a PC and was able to mount it just fine (with cryptsetup luksOpen and all). A simple ls works, too (at least in the top-level directory). So I tried to copy the data over to another medium, but when accessing certain directories or files the system simply hangs and I get tons of funny kernel errors (and the disk makes even more and louder funny noises). Great.

Stupid as I am, I also put data on the drive which I did not have backups of on other media (mostly because the data is huge, e.g. conference video recordings, large amounts of Creative Commons MP3s etc). OK, so I managed to get at least some small parts of my data, but now the disk is completely dead, I fear.

I'll try to convice the vendor to give me a new drive, but I won't let them get their fingers on my data, i.e., I will not send the disk to them (yes, even though it is encrypted; crypto can be broken). Any ideas what else I could do? A professional "data recovery" service is not an option either, they usually cost 1000-2000 Euros (in the best case).

What do you do for storing huge amounts of data nowadays? The only viable option I can think of is a RAID-5 (mdadm) with 3 or more disks in a silent PC which I can leave turned on 24/7. Unfortunately that costs non-trivial amounts of money. CDs are too small and they don't last too long either; the same is true for DVDs.

Sigh...

My VServer Still Down

More than two weeks ago I blogged about my server being down. After multiple emails, phone calls, and even a fax trying to reach the support team, the server is still dead. But at least I know (a little bit) more now.

The Good
I managed to get someone from support on the phone and he fixed the system at least so far that I could ssh to it again. I was able to pull a complete backup of the system, including a database dump.
That means that Unmaintained Free Software and all other sites hosted on the server will eventually return, no data will be lost.

The Bad
After I created the backup, I wanted to reinstall the whole system and then install the backup to restore all services. As it turned out, the (automatic) reboot- and reinstall-script they use is obviously broken, I cannot reach the server anymore after I initated the reinstall. This is probably something more serious, as other people seem to be affected, too.

The Ugly
I have not the slightest idea what the hell happened on the server. There was something really, really strange going on. An example:

# ls -l /usr/bin/traceroute
-rw-rw---- 1 mysql mysql 310872 Jun 21 03:21 traceroute

Why the hell is traceroute not executable and belongs to user/group mysql? There are several other anomalies there: /usr/share/doc/apt is not a directory as it is supposed to be, but a Perl script. /usr/bin/id is a directory. Multiple system tools (awk, sed, ...) are not executable and partly directories with strange stuff in them. What gives?

One possible explanation is that the server was hacked and some rootkit wrecked havoc on the server. After a quick glance at the logs, I couldn't find any hints for a successful breakin, though. Another possibility is that the hard drive simply died and/or the filesystem was (heavily) corrupted. I don't know...

Has anybody ever seen something like this? Please enlighten me what could have happened...

Syndicate content