RAID5 + dm-crypt + LVM + ext3 Debian install and benchmarks

OK, so I've setup a RAID5 at home because I'm getting tired of failed disk drives and data losses.

Some notes:

  • The system consists of 3 x 300 GB IDE drives in software RAID5 (standard Linux kernel and mdadm), thus ca. 600 GB usable storage space.
  • I've used the stock Debian installer to set up all of this, no custom hacks or anything needed.
  • Each drive is on an extra IDE bus/controller (1x onboard/internal, 2x on an PCI IDE controller card), as broken IDE disks (lacking hot-swap capabilities) often take down the whole IDE bus with them; it's not a good idea to put two disks on one IDE bus.
  • The software stack is: RAID5 at the botton, dm-crypt on top of that to encrypt the whole RAID, LVM on top of that to partition the system into /, /usr, /var, /tmp, /home, and swap.
  • /boot is on an extra 1 GB partition (replicated on each drive) as GRUB doesn't work on RAIDed disks and I want to use GRUB, not LILO. GRUB is also installed on the MBR of each drive, so if one of them fails, the other two can still come up.
  • I installed and configured smartmontools to check the status of the drives, and hddtemp to check their temperature.
  • Stability tests so far: While the system is running, pull out one of the IDE drives (yes, they're not hot-swappable and that may not be such a good idea, usually). The system survived without data loss. Time for rebuilding the array: ca. 1 hour. Second test: while the system is running, pull the plug. The system survived that, too.

Some stats from bonnie++ if anybody cares:

Version  1.03       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
bonsai           2G 26727  72 39426  19 16690   7 28725  65 34164   7 215.3   0
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++
bonsai,2G,26727,72,39426,19,16690,7,28725,65,34164,7,215.3,0,16,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++

(Now, if I only knew what all those figures mean ;-)

No, neither the software RAID5, nor the dm-crypt layer nor LVM cause any measurable performance degradation whatsoever (from my subjective feeling). I don't care enough to measure anything. The CPU is idling all the time.

Power consumption is rather high (partly due to the mainboard and CPU, but also because of the disks + fans) and the system is pretty loud, which both sucks on the long run. I plan an ultra-silent, ultra-low-power RAID5 with 2.5" disks attached via USB to a (silent, low-power) NSLU2 for later.

Comments

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

howto please

Can you post a howto of this install please ? I'm beginner and i'd like to have full encryption on my debian install and the benefits of raid5 fr my 3 disks.

Thanks

raid1 + dmcrypt ... then what? howto lvm on top?

Please tell me how you did this! Im stuck.

I created software raid1 with mdadm on two eSATA 500GB MyBooks:

1. ran badblocks random on both (took 48 hours)
2. created partitions on each disk with gparted (cfdisk complained, probably about no label and I didn't dare using fdisk). did not put any filesystem on the partitions.
3. used fdisk to change partition types to fd.
4. mdadm –-create --verbose /dev/md0 –-metadata=1.0 –-raid-devices=2 –-level=raid1 –-name=backupArray /dev/sd[bc]1
5. mdadm --examine --scan | tee /etc/mdadm.conf
6. did not need to assemble it here...it was done automagically.
7. used your dmcrypt howto and ran cryptsetup on the /dev/md0 device
8. opened dmcrypt and created mkfs.ext3 and mounted it. I know I don't need to do this but I just wanted to test it so far and it seems ok.

How do i now put lvm on top of this to use the whole dmcrypt device?? Doesn't pvcreate overwrite the partition table?

Lower power file server

Hi Uwe,

Just stumbled about this article while looking for the (very good!) Disk encryption how-to on your page.
However, I just started to run a T-Vision 100 box as low power (40 Watts) file server with an external usb disk drive. The box has no fan as well as the usb drive and thus is really super silent. I got one for free but you can get them on ebay for about 30 euros at the moment. I installed Debian 4.0 and it runs very well so far. Ok, the data transfer rate could be faster, but a silent and low power server is more important for me.

Thanks again for the dm-crypt tutorial,
Ben

what the Bonnie++ results mean

39M/s block writes and 34M/s blog reads is not particularly exciting for a single modern disk, for a RAID-5 it's less than I would hope to see.

The "++++" bits mean that the results were too fast to be accurately recorded (any test that completes in less than 500ms is not considered to be sufficiently accurate to display). Use an option such as "-n32" to do a larger test which will display results (or use "-n0" to skip that test).

Bonnie++

Hi Russel, thanks for the explanations!

Unfortunately I don't have the numbers from a single disk (with no RAID or LVM) to compare to. But I'll do one more bonnie++ run with "-n32" and post the results here...

Cheers, Uwe.

bonnie++ -n32

OK, so here are the numbers for "bonnie++ -n32":

Version 1.03 ------Sequential Output------ --Sequential Input- --Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
bonsai 2G 29467 72 43277 17 16808 7 31594 66 35520 8 366.6 1
------Sequential Create------ --------Random Create--------
-Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
32 31493 98 +++++ +++ 53570 100 35364 75 +++++ +++ 54380 99
bonsai,2G,29467,72,43277,17,16808,7,31594,66,35520,8,366.6,1,32,31493,98,+++++,+++,53570,100,35364,75,+++++,+++,54380,99

Cheers, Uwe.

PCI IDE Adapter

I've built arrays putting an active disk and a spare in a PCI IDE adapter, so if the adapter fails the array is degraded but not destroyed. This saved me lots of frustration at least once, when a cheap adapter failed completely taking both disks down with it but allowing the rest of the array to survive just long enough to backup data, shut down the system and replace the PCI card.
You might want to consider putting two disks in two on-board IDE adapters or using two independent PCI IDE Adapters to avoid a single point of failure. If you're short on IDE channels consider moving your CD-ROM drive to the PCI IDE card or getting a SATA disk and setting it up as a RAID spare.

PCI IDE Adapter

Two disks in onboard-IDE is not possible unfortunately, this board only has one. I could use a SATA disk, but I already have three IDE drives, no need to spend more money for a SATA drive. There's no CDROM in this PC either, btw. (not needed).

I'll just add another PCI IDE adapter, so that each drive is completely separate from the others.

Thanks, Uwe.

Its much easier to make your

Its much easier to make your /boot partition raid1. Grub will happily boot a mirrored raid partition. This way, data automatically gets replicated, and if a drive needs to drop off for a while, the partition easily gets rebuilt.

GRUB on RAID1

Hm, true, I was thinking about this too, but when I tried it, the installer won't let me use GRUB (only lilo). Maybe I did something else which killed the GRUB option though, not sure, but IIRC RAID1 should indeed work with GRUB...

Uwe.

If you don't use ext3 for

If you don't use ext3 for the boot partition, the installer refuses to use GRUB. For instance, use XFS for /boot, and it will auto install lilo. Atleased thats how the Debian & Ubuntu installers react.