Uwe Hermann | A slightly paranoid Debian developer

I've bought a new hard drive for my laptop recently, because I finally got fed up with my constantly-full disk. Having to browse around in $HOME looking for stuff which can be safely deleted just because I want to run fetchmail (and that would fill up my disk) just sucks. So, after getting a cheapo 160 GB 2.5" disk (the old one was 80 GB), I had to move all my data to the new disk.

As I didn't want to re-install from scratch I started with dd'ing the whole disk over to the new one (using a live CD and an external USB hard-drive enclosure). This took pretty long, but went fine otherwise.

The new disk then contained all my partitions (hda1-hda3) and also GRUB in the MBR etc., as expected, but was still only 80 GB in size, of course. So the first step is to enlarge the hda3 partition, which is a dm-crypt volume that contains various LVM logical volumes (for /home, /usr, /var, swap, etc.), each of them using the ext3 filesystem (except for the swap volume, of course).

0. Perform backups, boot from a live CD

Important: If you plan to perform any of these steps, make sure you have recent backups! I take no responsibility for any data loss you might experience. You have been warned!

First off, you should boot from a live CD which has all the tools you'll need, including cryptsetup, LVM tools, resize2fs, etc. You can use the nice grml live CD for instance.

1. Resize partition

This sounds scary (and it is!), but the way I enlarged the encrypted hda3 partition was by first deleting it via fdisk. First, issue the "p" command in fdisk, write down the exact start cylinder of hda3. Then delete hda3. Now create a new hda3 partition which starts at exactly the same cylinder as the old hda3 but is larger, i.e. in my case it has ca. 80 GB additional space.

Your data will still be there if you don't screw up, and the partition is bigger now. Using something like gparted will likely not work as expected, as the partition is encrypted!

2. Resize dm-crypt volume

Nothing to be done, it seems dm-crypt automatically adapts and notices that the partition is bigger. Just "open" the encrypted volume using cryptsetup now:

  $ cryptsetup luksOpen /dev/hda3 foo

3. Resize LVM physical volume

Next step is to tell LVM about the new space. We first resize the LVM physical volume on the foo "partition" to use up all newly-available space.

  $ pvresize /dev/mapper/foo

4. Resize LVM logical volume

Now we can pump the new space into any of the logical volumes (or into multiple ones). I only increased one logical volume, my /home:

  $ lvresize -L +74 GB /dev/vg-whole/lv-home

5. Resize ext3 filesystem

The final step is to resize the ext3 filesystem on the lv-home logical volume (after running the obligatory fsck -n). I first used ext2resize, but that failed horribly:

  $ fsck -n /dev/vg-whole/lv-home
  $ ext2resize /dev/vg-whole/lv-home
  error: Invalid argument: seeking to 3258921205760

This seems to be a known bug, ext2resize apparently cannot handle large disks or something, and as I found out a few minutes later it's pretty much deprecated anyway. The better solution is to use resize2fs:

  $ fsck -n /dev/vg-whole/lv-home
  $ resize2fs /dev/vg-whole/lv-home

That's it. We can now reboot the system from disk and enjoy ca. 80 GB of additional hard drive space. Yay!

Just in case you haven't already read about this... Some researchers from Princeton have published a paper about methods which can be used to attack full-disk-encryption (FDE) schemes.

They have demonstrated that at least BitLocker (Windows Vista), FileVault (MacOS X) and dm-crypt (Linux) are vulnerable to this type of (partly hardware-based) attack scenarios. Quite likely lots of similar other solutions are vulnerable as well.

The main problem is that (contrary to popular belief) RAM does indeed retain its data for a non-trivial amount of time after power is cut (seconds, even minutes or hours if it's cooled down enough), so you can mount some new attacks such as:

• Get physical access to laptop/computer, cut power to it (the hard way), reboot with a special live CD or USB thumb drive and some special software which dumps the RAM contents to an external disk (or sends it via network). As RAM contents are still there a few seconds after the power is cut, this works astonishingly well.
• Get physical access to laptop/computer, open it, remove RAM DIMMs while the computer is running, insert them into your own prepared computer and read the RAM contents using some special software.

Yes, all attacks assume that the attacker has physical access to your PC/RAM, in which case you already have several other problems. Still, the new thing about this is that even full-disk-encryption doesn't help much in some cases. You probably shouldn't depend too much on it (but you shouldn't stop using disk encryption either, of course!).

Full paper: coldboot.pdf. There are also some demo videos and pictures.

More coverage at Boing Boing, Bruce Schneier's weblog, Freedom to Tinker, Slashdot, Heise (German), and many more...

Make sure to read the comments of the various articles for more scenarios and possible ideas for how to prevent such attacks. Some ideas include enabling the BIOS RAM checks (which might explicitly erase RAM contents on reboot; that doesn't help in all cases, though) or using coreboot (previously LinuxBIOS) to erase RAM contents at boot-up and/or shutdown.

It's a highly non-trivial issue, though, there's no easy and complete fix so far. The only sure way is to not have your laptop or PC stolen and to not give attackers physical access to your computers.

OK, so I've setup a RAID5 at home because I'm getting tired of failed disk drives and data losses.

Some notes:

• The system consists of 3 x 300 GB IDE drives in software RAID5 (standard Linux kernel and mdadm), thus ca. 600 GB usable storage space.
• I've used the stock Debian installer to set up all of this, no custom hacks or anything needed.
• Each drive is on an extra IDE bus/controller (1x onboard/internal, 2x on an PCI IDE controller card), as broken IDE disks (lacking hot-swap capabilities) often take down the whole IDE bus with them; it's not a good idea to put two disks on one IDE bus.
• The software stack is: RAID5 at the botton, dm-crypt on top of that to encrypt the whole RAID, LVM on top of that to partition the system into /, /usr, /var, /tmp, /home, and swap.
• /boot is on an extra 1 GB partition (replicated on each drive) as GRUB doesn't work on RAIDed disks and I want to use GRUB, not LILO. GRUB is also installed on the MBR of each drive, so if one of them fails, the other two can still come up.
• I installed and configured smartmontools to check the status of the drives, and hddtemp to check their temperature.
• Stability tests so far: While the system is running, pull out one of the IDE drives (yes, they're not hot-swappable and that may not be such a good idea, usually). The system survived without data loss. Time for rebuilding the array: ca. 1 hour. Second test: while the system is running, pull the plug. The system survived that, too.

Some stats from bonnie++ if anybody cares:

Version  1.03       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
bonsai           2G 26727  72 39426  19 16690   7 28725  65 34164   7 215.3   0
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++
bonsai,2G,26727,72,39426,19,16690,7,28725,65,34164,7,215.3,0,16,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++

(Now, if I only knew what all those figures mean ;-)

No, neither the software RAID5, nor the dm-crypt layer nor LVM cause any measurable performance degradation whatsoever (from my subjective feeling). I don't care enough to measure anything. The CPU is idling all the time.

Power consumption is rather high (partly due to the mainboard and CPU, but also because of the disks + fans) and the system is pretty loud, which both sucks on the long run. I plan an ultra-silent, ultra-low-power RAID5 with 2.5" disks attached via USB to a (silent, low-power) NSLU2 for later.

Hardware sucks. It just totally and utterly sucks.

I purchased a 300 GB hard drive (plus a 3.5" USB disk drive enclosure) roughly 8 months ago which I encrypted using dm-crypt and then used it as a backup medium. And now the disk has died. No, this is not a software problem (that would be easy to deal with), the hardware is simply dead (it's making funny "klck" noises all the time).

Mounting it via USB (using the USB enclosure) doesn't work at all anymore. I connected the disk via IDE in a PC and was able to mount it just fine (with cryptsetup luksOpen and all). A simple ls works, too (at least in the top-level directory). So I tried to copy the data over to another medium, but when accessing certain directories or files the system simply hangs and I get tons of funny kernel errors (and the disk makes even more and louder funny noises). Great.

Stupid as I am, I also put data on the drive which I did not have backups of on other media (mostly because the data is huge, e.g. conference video recordings, large amounts of Creative Commons MP3s etc). OK, so I managed to get at least some small parts of my data, but now the disk is completely dead, I fear.

I'll try to convice the vendor to give me a new drive, but I won't let them get their fingers on my data, i.e., I will not send the disk to them (yes, even though it is encrypted; crypto can be broken). Any ideas what else I could do? A professional "data recovery" service is not an option either, they usually cost 1000-2000 Euros (in the best case).

What do you do for storing huge amounts of data nowadays? The only viable option I can think of is a RAID-5 (mdadm) with 3 or more disks in a silent PC which I can leave turned on 24/7. Unfortunately that costs non-trivial amounts of money. CDs are too small and they don't last too long either; the same is true for DVDs.

Sigh...

I was planning to set up my laptop from scratch for a while now... so I did.

Preparation

• First, go home. No, really! Do all of this at home in a non-hostile, firewalled network. You don't want to be in a crowded place such as a conference where people can shoulder-surf your passwords, nor do you want your network traffic sniffed or MITM'd in a hostile network.
• Backup all your data! You'll be wiping your whole drive soon, so make sure you have recent, tested backups.
• Get the most recent Debian-installer ISO image (currently etch-beta3), as well as the MD5SUMS and MD5SUMS.sign files:
wget http://cdimage.debian.org/cdimage/etch_di_beta3/i386/iso-cd/debian-testing-i386-binary-1.iso
wget http://cdimage.debian.org/cdimage/etch_di_beta3/i386/iso-cd/MD5SUMS
wget http://cdimage.debian.org/cdimage/etch_di_beta3/i386/iso-cd/MD5SUMS.sign
• Run gpg --verify MD5SUMS.sign, which will fail but tell you the signing key ID (88C7C1F7 in this case). Get the key and re-run the verification: gpg --recv-key --keyserver subkeys.pgp.net 88C7C1F7 && gpg --verify MD5SUMS.sign. The output should now say "Good signature from [...]".
• Now check the MD5 checksums via md5sum -c MD5SUMS. The output should contain debian-testing-i386-binary-1.iso: OK.
• As you now have (somewhat) verified the integrity of the ISO image, burn it on a CD-R: wodim debian-testing-i386-binary-1.iso.
• Put trusted versions of some files on a USB thumb drive (or CD-ROM); at least a firewall script, but maybe also your bashrc, bash_logout, inputrc, vimrc, muttrc.
• Disconnect your laptop from any kinds of networks. Pull all ethernet cables. Disable WLAN (via hardware killswitch). Disable Bluetooth. Disable/remove Firewire, USB, serial, whatever.
• Put on your tin-foil hat (optional).

BIOS

• Set a good BIOS boot password (which you need to boot any OS). Set a (different) good BIOS boot setup password (which you need to enter the BIOS).
• Disable all boot possibilities in the BIOS, except for CD-ROM. This means it should not be possible to boot via USB, hard drive, network, PXE, Firewire, floppy, whatever. The BIOS setup password helps to prevent tampering with this setting.
• Finally, never rely on BIOS passwords alone for security! They can often be circumvented very easily.

Installation / Setting up full-disk encryption using dm-crypt

• Insert the installer CD and boot in expert-mode (don't hit ENTER when you boot, but rather type "expert").
• As for networking: select "Do not configure the network at this time". We'll fix and enable networking later.
• Partitioning:
  • Select manual partitioning. Remove all partitions (if any). Create a 100 MB /boot (ext3) as primary partition, and make the rest of the hard drive one huge partition which has "Use as:" set to "physical volume for encryption".
  • The standard options for cipher, key size, IV mode etc. should be fine (AES, 256 bit, CBC-ESSIV-SHA256, dm-crypt).
  • After the erasing is done (this is important!), use the whole encrypted space as "physical volume for LVM". Then select "Configure the Logical Volume Manager". Create one big volume group and a bunch of logical volumes for the various partitions we'll use (lv-root, lv-usr, lv-var, lv-tmp, lv-swap, lv-home).
  • It is extremely important that your swap space is encrypted (in this case it is, as all partitions except for /boot reside on a dm-crypt device)! Never set up unencrypted swap!
• Enable shadow passwords. Allow login as root (I feel confident that I won't do stupid things as root).
• Choose a good root password, and a (different) good user password. Don't enter a full name for the user.
• Choose the latest kernel (old kernels might have security issues). Do not participate in popcon.
• Do not install any tasks (no "desktop", no "base system"). We want the smallest installation possible, and add only the packages we really need. Fewer packages means fewer security issues (statistically).
• That's it. Eject the CD-ROM, reboot, change the BIOS to only allow booting from hard drive.

Post-installation tasks

• Enter the USB thumb drive, copy all config-files to /root and /home/uwe. Log out and log in again to make ~/.bashrc and ~/.inputrc take effect.
• Enable the firewall: mkdir /etc/rc.boot && cp fw_laptop /etc/rc.boot && chmod 700 /etc/rc.boot/fw_laptop && sh /etc/rc.boot/fw_laptop
• Shut down all networked daemons (if any): /etc/init.d/foo stop.
• Tighten home-directory permissions: chmod 700 /root /home/uwe.
• Edit /etc/passwd: give all users except for root, sync, uucp and your user account /usr/sbin/nologin as login shell. None of these accounts really needs a valid login shell (nologin will log any login attempts for those accounts).
• Edit /etc/group: remove your user account from the dialout, cdrom, and floppy group. The groups audio, video, and plugdev can stay.
• Edit /etc/fstab: add some mount options such as ro, nosuid, noexec, or nodev as you see fit. Example:
  

  /dev/mapper/vg--whole-lv--root /     ext3 defaults,errors=remount-ro      0 0
  /dev/sda2                      /boot ext3 defaults,nodev,nosuid,noexec,ro 0 0
  /dev/mapper/vg--whole-lv--home /home ext3 defaults,nodev,nosuid           0 0
  /dev/mapper/vg--whole-lv--tmp  /tmp  ext3 defaults,nodev,nosuid           0 0
  /dev/mapper/vg--whole-lv--usr  /usr  ext3 defaults,nodev,ro               0 0
  /dev/mapper/vg--whole-lv--var  /var  ext3 defaults,nodev                  0 0
  /dev/mapper/vg--whole-lv--swap none  swap sw                              0 0
  /dev/scd0 /media/cdrom iso9660 noauto,nodev,nosuid,noexec,uid=uwe,gid=uwe 0 0
  

• If you have read-only (ro) file systems, configure Apt so that it can remount them read-write when installing/removing packages. Add this to /etc/apt/apt.conf:
  

  DPkg
  {
    Pre-Invoke { "mount -o remount,rw /usr; mount -o remount,rw /boot"; }
    Post-Invoke { "mount -o remount,ro /usr; mount -o remount,ro /boot"; }
  }
  

• Fix the GRUB configuration. Replace the "password foo" line (which contains the GRUB password in plain-text) from your /boot/grub/menu.lst with a "password --md5 $1$1234567890..." line, where the MD5 hash ($1$1234567890...) can be generated with grub-md5-crypt. Additionally, add such a password line after each "title" line in the GRUB config-file, so that nobody can boot any OS installed on the laptop without a password!

Networking, Upgrading and Apt-secure

• Now that we have a small, hardened system, it should be reasonably safe to enable networking. Add this to /etc/network/interfaces:
  

  auto eth0
  iface eth0 inet dhcp
    pre-up /etc/rc.boot/fw_laptop
  

  Run /etc/init.d/networking restart. The firewall script will run every time the network is started.

• Now add this (tweak as you see fit) to /etc/apt/sources.list:
deb http://ftp.de.debian.org/debian unstable main
deb-src http://ftp.de.debian.org/debian unstable main
• Time for upgrading: apt-get update && apt-get dist-upgrade. All packages are GnuPG-signed and will be verified by Apt. The installer already ships the required key (for 2006), so everything should just work. Still, you should read about SecureApt.
• Install the rest of your system now, and restore your data from backups.
• Use sysv-rc-conf to disable all daemons you don't want to start per default: sysv-rc-conf foo off.
• Install and set up Samhain (or any other file integrity checker): apt-get install samhain. You want to be notified if your system files are being tampered with (e.g. replaced by a rootkit).
• Install and configure Tor for anonymous browsing. More details here.
• Install and configure more security-related programs, e.g. logcheck, snort, rkhunter, chkrootkit, tiger, sxid, etc.

SELinux

Now install and set up SELinux. This section is based on notes from Erich Schubert (thanks!), and will soon appear in the SELinuxSetup wiki page, too.

• Install the base packages and an SELinux policy: apt-get install selinux-basics selinux-policy-refpolicy-targeted.
• Edit /boot/grub/menu.lst and add selinux=1 to your kernel command line to enable SELinux upon booting.
• In /etc/pam.d/login uncomment the "session required pam_selinux.so multiple" line. Do the same in /etc/pam.d/ssh if you have ssh installed.
• In /etc/default/rcS set FSCKFIX=yes.
• In /etc/init.d/bootmisc.sh search for "Update motd" and comment the two lines below that line. Then rm /var/run/motd.
• If you have exim installed, you must either install postfix or write an exim policy, as none currently exists. But even postfix needs some fixing (no pun intended ;-). Disable chroot-support (change all "chroot" fields to "n" in /etc/postfix/master.cf and execute echo 'SYNC_CHROOT="n" >> /etc/default/postfix').
• Use check-selinux-installation to check for common SELinux problems on Debian (such as the above mentioned).
• touch /.autorelabel. Reboot. touch /.autorelabel (again). Reboot (again).
• Done. You should now have a working SELinux system. If no critical audit errors appear and you feel comfortable with SELinux, enable enforcing mode via setenforce 1 or by adding enforcing=1 to the kernel command line in /boot/grub/menu.lst.

Behaviour

• Never leave your laptop unattended!
• Always lock your terminal (using vlock) when you move more than 30 cm away from the laptop!
• Don't run insecure and/or closed-source software (which you can never trust!). No NVIDIA/ATI drivers, no VMware, no Google Earth, no Flash Plugin (except for Gnash maybe), no Adobe Acrobat. You get the idea.
• Keep the number of installed packages small and try to configure each of them as secure as possible.
• Never enable networking or WLAN or Bluetooth if you don't absolutely have to.
• Trust no one. Don't let other people use you laptop, don't give out shell accounts.

Further ideas

• The /boot partition is still unencrypted, so an attacker can tamper with it. Boot from a CD-R, forbid booting from hard drive (BIOS). Sign/mark the CD-R physically, so you'll know when someone replaced your CD-R with his own, back-doored one.
• Another idea is to use an additionaly USB thumb drive or CD-ROM or smartcard for two-factor authentication.
• Install another Debian into a QEMU image. Use it as a sandbox for stuff you don't trust: qemu -snapshot -net none foo.img.
• At all costs, disable Firewire! If possible via hardware or BIOS, or at least don't load the drivers and/or fix them (page 19).
• Replace the proprietary, closed-source BIOS with LinuxBIOS, if possible.

That's it. You can take off that stupid tin-foil hat now.

Update 2006-09-29: Fixed typos. Mentioned sxid. Added two-factor authentication.