How to setup an encrypted USB-disk software-RAID-1 on Debian GNU/Linux using mdadm and cryptsetup

This is what I set up for backups recently using a cheap USB-enclosure which can house 2 SATA disks and shows them as 2 USB mass-storage devices to my system (using only one USB cable). Without any further introduction, here goes the HOWTO:

First, create one big partition on each of the two disks (/dev/sdc and /dev/sdd in my case) of the exact same size. The cfdisk details are omitted here.

  $ cfdisk /dev/sdc
  $ cfdisk /dev/sdd

Then, create a new RAID array using the mdadm utility:

  $ mdadm --create /dev/md0 --level=1 --raid-devices=2 /dev/sdc1 /dev/sdd1

The array is named md0, consists of the two devices (--raid-devices=2) /dev/sdc1 and /dev/sdd1, and it's a RAID-1 array, i.e. data is simply mirrored on both disks so if one of them fails you don't lose data (--level=1). After this has been done the array will be synchronized so that both disks contain the same data (this process will take a long time). You can watch the current status via:

  $ cat /proc/mdstat
  Personalities : [raid1]
  md0 : active raid1 sdd1[1] sdc1[0]
        1465135869 blocks super 1.1 [2/2] [UU]
        [>....................]  resync =  0.0% (70016/1465135869) finish=2440.6min speed=10002K/sec
  unused devices: 

Some more info is also available from mdadm:

  $ mdadm --detail --scan
  ARRAY /dev/md0 metadata=1.01 name=foobar:0 UUID=1234578:1234578:1234578:1234578

  $ mdadm --detail /dev/md0
          Version : 1.01
    Creation Time : Sat Feb  6 23:58:51 2010
       Raid Level : raid1
       Array Size : 1465135869 (1397.26 GiB 1500.30 GB)
    Used Dev Size : 1465135869 (1397.26 GiB 1500.30 GB)
     Raid Devices : 2
    Total Devices : 2
      Persistence : Superblock is persistent
      Update Time : Sun Feb  7 00:03:21 2010
            State : active, resyncing
   Active Devices : 2
  Working Devices : 2
   Failed Devices : 0
    Spare Devices : 0
   Rebuild Status : 0% complete
             Name : foobar:0  (local to host foobar)
             UUID : 1234578:1234578:1234578:1234578
           Events : 1
      Number   Major   Minor   RaidDevice State
         0       8       33        0      active sync   /dev/sdc1
         1       8       49        1      active sync   /dev/sdd1

Next, you'll want to create a big partition on the RAID device (cfdisk details omitted)...

  $ cfdisk /dev/md0

...and then encrypt all the (future) data on the device using dm-crypt+LUKS and cryptsetup:

  $ cryptsetup --verbose --verify-passphrase luksFormat /dev/md0p1
  Enter your desired pasphrase here (twice)
  $ cryptsetup luksOpen /dev/md0p1 myraid

After opening the encrypted container with cryptsetup luksOpen you can create a filesystem on it (ext3 in my case):

  $ mkfs.ext3 -j -m 0 /dev/mapper/myraid

That's about it. In future you can access the RAID data by using the steps below.

Starting the RAID and mouting the drive:

  $ mdadm --assemble /dev/md0 /dev/sdc1 /dev/sdd1
  $ cryptsetup luksOpen /dev/md0p1 myraid
  $ mount -t ext3 /dev/mapper/myraid /mnt

Shutting down the RAID:

  $ umount /mnt
  $ cryptsetup luksClose myraid
  $ mdadm --stop /dev/md0

That's all. Performance is shitty due to all the data being shoved out over one USB cable (and USB itself being too slow for these amounts of data), but I don't care too much about that as this setup is meant for backups, not performance-critical stuff.

Update 04/2011: Thanks to Bohdan Zograf there's a Belorussian translation of this article now!

Note to self: Missing lvm2 and cryptsetup packages lead to non-working initrd very, very soon

I recently almost died from a heart attack because after a really horrible crash (don't ask), Debian unstable on my laptop wouldn't boot anymore. The system hung at "Waiting for root filesystem...", and I was in panic mode as I feared I lost all my data (and as usual my backups were waaay too old).

At first I was suspecting that something actually got erased or mangled due to the crash, either at the dm-crypt layer, or the LVM layer, or the ext3 filesystem on top of those. After various hours of messing with live CDs, cryptsetup, lvm commands (such as pvscan, pvs, vgchange, vgs, vgck) and finally fsck I still had not managed to successfully boot my laptop.

I finally was able to boot by changing the initrd from initrd.img-2.6.30-2-686 to initrd.img-2.6.30-2-686.bak in the GRUB2 menu (at boot-time), at which point it was clear that something was wrong with my current initrd. A bit of debugging and some initrd comparisons revealed the cause:

Both, the cryptsetup and lvm2 packages were no longer installed on my laptop, which made all update-initramfs invokations (e.g. upon kernel package updates) create initrds which did not contain the proper dm-crypt and lvm functionality support. Hence, no booting for me. I only noticed because of the crash, as I usually do not reboot the laptop very often (two or three times per year maybe).

Now, as to why those packages were removed I have absolutely no idea. I did not remove them knowingly, so I suspect some dist-upgrade did it and I didn't notice (but I do carefully check which packages dist-upgrade tries to remove, usually)...

Speed up Linux crypto operations on the One A110 laptop with VIA Padlock

One Mini A110 subnotebook

OK, so I've been hacking on and testing my shiny new One A110 mini-laptop during the last few days and I must say I'm very happy with it. I'll write up some more details later (check the wiki if you're impatient), but today I want to highlight a very nice feature of this laptop (compared to, for instance, the Eee PC): The VIA C7-M ULV CPU in the laptop has VIA Padlock support.

VIA Padlock is a hardware feature in recent VIA CPUs which provides hardware-accelerated AES and SHA-1/SHA-256 support, among other things. This can be used in Linux (with the proper drivers and patches) to improve performance of dm-crypt, OpenSSL (and all programs using it), scp, sha1sum, OpenVPN, etc. etc.

I have written a quite extensive VIA Padlock HOWTO and benchmarks in the A110 wiki (but all of this will work on other systems which have VIA Padlock, too). To summarize, here are the most important benchmarks:

dm-crypt (256bit AES, cbc-essiv:sha256)

VIA Padlock dm-crypt benchmark

Without VIA Padlock support:

$ hdparm -tT /dev/mapper/hdc2_crypt
 Timing cached reads:   448 MB in  2.00 seconds = 223.47 MB/sec
 Timing buffered disk reads:   22 MB in  3.07 seconds =   7.17 MB/sec

With VIA Padlock support:

$ hdparm -tT /dev/mapper/hdc2_crypt
 Timing cached reads:   502 MB in  2.00 seconds = 250.41 MB/sec
 Timing buffered disk reads:   90 MB in  3.07 seconds =  29.36 MB/sec

The native speed of the SSD in the laptop is 31.01 MB/sec, so there is almost no performance penalty when using VIA Padlock.


VIA Padlock OpenSSL benchmark

OpenSSL speed benchmark, first line without Padlock, second line with Padlock enabled:

$ openssl speed -evp aes-256-cbc [-engine padlock]
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
aes-256-cbc       9187.18k    10572.28k    11054.32k    11179.36k    11218.02k
aes-256-cbc      47955.92k   150619.73k   325730.73k   458320.11k   520520.79k


VIA Padlock scp benchmark

Without VIA Padlock support:

$ scp -c aes256-cbc bigfile.dat localhost:/dev/null
bigfile.dat                100%  159MB   5.9MB/s   00:27

With VIA Padlock support:

$ scp -c aes256-cbc bigfile.dat localhost:/dev/null
bigfile.dat                100%  159MB  14.5MB/s   00:11


A real speed benchmark is pending (not measurable easily on 100MBit LAN, will try on a slower link), but as OpenVPN uses OpenSSL it should have roughly the same speedup iff you tell OpenVPN to use AES (it uses Blowfish per default).

Also, there's a measurable difference in CPU load while tranferring large files over OpenVPN: 8% CPU load with VIA Padlock (vs. 20% CPU load without VIA Padlock).

sha1sum / phe_sum

VIA Padlock sha1sum / phe_sum benchmark

phe_sum is a small C program which can be used as drop-in replacement for sha1sum (which doesn't support VIA Padlock yet). Quick benchmark:

sha1sum, without VIA Padlock:

$ time sha1sum bigfile.dat
real    0m6.511s
user    0m5.864s
sys     0m0.412s

phe_sum (with VIA Padlock support):

$ time ./phe_sum bigfile.dat
real    0m1.149s
user    0m0.704s
sys     0m0.424s

All in all VIA Padlock gives you a pretty impressive speedup for many crypto-using applications on Linux, which is especially useful on the A110 mini-laptop (think OpenVPN or scp for mobile usage, and dm-crypt for an encrypted SSD, of course).

Lest We Remember: Cold Boot Attacks on Encryption Keys

Just in case you haven't already read about this... Some researchers from Princeton have published a paper about methods which can be used to attack full-disk-encryption (FDE) schemes.

They have demonstrated that at least BitLocker (Windows Vista), FileVault (MacOS X) and dm-crypt (Linux) are vulnerable to this type of (partly hardware-based) attack scenarios. Quite likely lots of similar other solutions are vulnerable as well.

The main problem is that (contrary to popular belief) RAM does indeed retain its data for a non-trivial amount of time after power is cut (seconds, even minutes or hours if it's cooled down enough), so you can mount some new attacks such as:

  • Get physical access to laptop/computer, cut power to it (the hard way), reboot with a special live CD or USB thumb drive and some special software which dumps the RAM contents to an external disk (or sends it via network). As RAM contents are still there a few seconds after the power is cut, this works astonishingly well.
  • Get physical access to laptop/computer, open it, remove RAM DIMMs while the computer is running, insert them into your own prepared computer and read the RAM contents using some special software.

Yes, all attacks assume that the attacker has physical access to your PC/RAM, in which case you already have several other problems. Still, the new thing about this is that even full-disk-encryption doesn't help much in some cases. You probably shouldn't depend too much on it (but you shouldn't stop using disk encryption either, of course!).

Full paper: coldboot.pdf. There are also some demo videos and pictures.

More coverage at Boing Boing, Bruce Schneier's weblog, Freedom to Tinker, Slashdot, Heise (German), and many more...

Make sure to read the comments of the various articles for more scenarios and possible ideas for how to prevent such attacks. Some ideas include enabling the BIOS RAM checks (which might explicitly erase RAM contents on reboot; that doesn't help in all cases, though) or using coreboot (previously LinuxBIOS) to erase RAM contents at boot-up and/or shutdown.

It's a highly non-trivial issue, though, there's no easy and complete fix so far. The only sure way is to not have your laptop or PC stolen and to not give attackers physical access to your computers.

RAID5 + dm-crypt + LVM + ext3 Debian install and benchmarks

OK, so I've setup a RAID5 at home because I'm getting tired of failed disk drives and data losses.

Some notes:

  • The system consists of 3 x 300 GB IDE drives in software RAID5 (standard Linux kernel and mdadm), thus ca. 600 GB usable storage space.
  • I've used the stock Debian installer to set up all of this, no custom hacks or anything needed.
  • Each drive is on an extra IDE bus/controller (1x onboard/internal, 2x on an PCI IDE controller card), as broken IDE disks (lacking hot-swap capabilities) often take down the whole IDE bus with them; it's not a good idea to put two disks on one IDE bus.
  • The software stack is: RAID5 at the botton, dm-crypt on top of that to encrypt the whole RAID, LVM on top of that to partition the system into /, /usr, /var, /tmp, /home, and swap.
  • /boot is on an extra 1 GB partition (replicated on each drive) as GRUB doesn't work on RAIDed disks and I want to use GRUB, not LILO. GRUB is also installed on the MBR of each drive, so if one of them fails, the other two can still come up.
  • I installed and configured smartmontools to check the status of the drives, and hddtemp to check their temperature.
  • Stability tests so far: While the system is running, pull out one of the IDE drives (yes, they're not hot-swappable and that may not be such a good idea, usually). The system survived without data loss. Time for rebuilding the array: ca. 1 hour. Second test: while the system is running, pull the plug. The system survived that, too.

Some stats from bonnie++ if anybody cares:

Version  1.03       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
bonsai           2G 26727  72 39426  19 16690   7 28725  65 34164   7 215.3   0
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++

(Now, if I only knew what all those figures mean ;-)

No, neither the software RAID5, nor the dm-crypt layer nor LVM cause any measurable performance degradation whatsoever (from my subjective feeling). I don't care enough to measure anything. The CPU is idling all the time.

Power consumption is rather high (partly due to the mainboard and CPU, but also because of the disks + fans) and the system is pretty loud, which both sucks on the long run. I plan an ultra-silent, ultra-low-power RAID5 with 2.5" disks attached via USB to a (silent, low-power) NSLU2 for later.

Syndicate content