15 July 2007, 18:27 by mark hoekstra

OpenBSD encrypted NAS HOWTO

update I implemented the changes as pointed out in this comment by Otto, thanks!

In this document I will try to explain what it takes to get an encrypted NAS on OpenBSD. This involves software RAID and encrypted filesystems, as mentioned in this article.

First of all, this how-to is a combination of the how-to of Manolis Tzanidakis on NewsForge on Software RAID on OpenBSD using RAIDframe and my own OpenBSD encrypted fileserver HOWTO (which was partly based on stuff done by backwatcher.org (still down as it seems)). This now reflects in a much better way on what I’m running at home for quite some months now than my original old how-to did.

In this setup we’ll have one separate disk for the OS (in this case sd0) and two disks (in our case, wd0 and wd1) for the RAID 1 array. In total we’re going to mirror the two big disks and on top of that array we’re going to build an encrypted filesystem :-)

About further hardware requirements. Well, you will need some motherboard to which you can connect three hard drives (that won’t be too hard, I guess ^_^). Then for the encryption, I’ve used an MP1800+ before and that gave me around 8MB/sec. Nowadays I use an Opteron 146 (2.0GHz) and that gives me around 20MB/sec (25MB/sec read, 16MB/sec write). So if you take anything which is not too old (AMD64s are great for this purpose imho), you’ll be fine. Personally I also use OpenBSD/AMD64 but there’s nothing wrong in using OpenBSD/i386 for this. Whatever suits you.

For this setup, a minimum installation of bsd kernel and base, comp, and etc tarballs will be fine. Feel free to install anything else you might want. Configure the network, passwords, and services as usual and reboot into the new installation.

Source code for the kernel is included on the install CD-ROM as src.tar.gz and on the OpenBSD mirrors as sys.tar.gz. Uncompress the source code from the CD-ROM by issuing the following commands as root:

mount /dev/cd0a /mnt
tar -zxvpf /mnt/src.tar.gz -C /usr/src ./sys
umount /mnt

Replace cd0 to match your CD-ROM drive device. (It would be better not to connect a CD-ROM drive as slave on one of the array drive’s IDE channels, but since CDs are rarely used on a server it should not cause any troubles.)

Now is a good time to apply any patches issued since the release of your OpenBSD version, in order to avoid another time-consuming compilation later. Patches are announced on the errata page, and each patch includes instructions on how to apply the software.

Now it’s time to compile and install a kernel with RAIDframe support and RAID auto-configuration. Move to the conf directory for the kernel using cd /sys/arch/‘uname -m’/conf and create a new kernel configuration, GENERIC.RAID, by issuing the commands:

cat >> GENERIC.RAID << EOF
include “arch/‘uname -m’/conf/GENERIC” # include GENERIC configuration

option RAID_AUTOCONFIG # automatically configure RAIDframe arrays on boot

pseudo-device raid 4 # RAIDframe disk driver
EOF

Now, run the following commands to configure, compile, and install the kernel:

config GENERIC.RAID
cd ../compile/GENERIC.RAID
make clean depend && make
cp /bsd /bsd.noraid
install -o root -g wheel -m 644 bsd /

After that’s done, reboot the system into the RAIDframe-enabled kernel. In case something goes wrong and the new kernel doesn’t boot, the old one is saved as ‘bsd.noraid’ and can be booted by issuing the following command at the boot prompt:

boot> boot sd0a:/boot.noraid

After successfully booting into the new kernel, initialize the second disk, wd0, using fdisk. If OpenBSD is the only installed operating system, you can use the whole disk as an OpenBSD partition with the command:

fdisk -i wd0

Now you could go out and create multiple partitions. In my experience I couldn’t get the ‘a’ partition to become of type FS_RAID for some reason (that probably has to do that the first partition could be needed for booting and booting from a software RAID array is a chicken and egg situation… ;-)). But leaving the first partition of type FS_BSDFFS (e.g. 4.2BSD) will also work. If this somehow bothers you, you can make a small first partition and make a second partition of type FS_RAID (RAID)

disklabel -E wd0

newfs wd0a

You can repeat this for the second disk, in short:

fdisk -i wd1
disklabel -E wd1
(make one big partition)
newfs wd1a

Now you can create raid0.conf, our configuration file for the raid-array.

On my system, it looks like this:

# cat raid0.conf
START array
1 2 0
START disks
/dev/wd0a
/dev/wd1a
START layout
128 1 1 1
START queue
fifo 100

After you created this, we need to start the array:

raidctl -C /etc/raid0.conf raid0 (the first time we need to use -C, once it’s created and no configuration changes have been made, -c suffices, read ‘man raidctl’ for more info)

raidctl -I raid0

Now this serial-number of the array is user-specified. If you plan on experimenting with multiple arrays, it’s highly recommended to make use of these, for instance the date and trailing number: 0707150

so the command becomes: raidctl -I 0707150 raid0

If you somehow get errors about raidctl not wanting to add drives or other, reboot the system here

Let’s check up on our array (if you either rebooted or not):

raidctl -s raid0

Now the parity-status will be dirty, we’ll get to that later. If wd1a failed, we can get it to reconstruct by using:

raidctl -R /dev/wd1a raid0

and after that, raidctl -s raid0 will show ‘reconstructing’

After this we need to get the parity clean and we can do that by:

raidctl -vP raid0

(with the -v (verbose) option, we get a ETA and then you know how long you need to wait, on my system, with two 320GB drives and a 2.0GHz Opteron, it takes about 1 hour and 45 minutes)

Once this is done, we should have a healthy RAID 1 array, on my system it gives me this:

# raidctl -s raid0
raid0 Components:
/dev/wd0a: optimal
/dev/wd1a: optimal
No spares.
Parity status: clean
Reconstruction is 100% complete.
Parity Re-write is 100% complete.
Copyback is 100% complete.
#

Then we need to initialize the array and we can do that by:

fdisk -i raid0
disklabel -E raid0 (and again, make one big partition, of type 4.2BSD, of this)
newfs raid0a

So, now we have our base, in all RAID-i-fied glory, time to build an encrypted filesystem on top of that!

I’ve got a /datacrypt dir especially for holding the encrypted filesystem, the decrypted one is going to be mounted elsewhere.

cd /
mkdir datacrypt
chmod 655 datacrypt
cd /datacrypt
mkdir raid0
chmod 655 raid0

Then we need to mount the just made raid-array and also add this to our fstab, in mine it’s this line:

/dev/raid0a /datacrypt/raid0 ffs rw,nodev,nosuid 1 2

By hand we can mount it simply like this:

mount /dev/raid0a /datacrypt/raid0

Once we’ve done that, we need to make one big file on this, as a base for our encrypted filesystem.

touch /datacrypt/raid0/cryptfile

and in the original how-to I filled that file with zeroes. Now some people argued that it’s better (more secure) to fill it with the outcome of /dev/prandom and indeed, that’s even better, so here we go:

dd if=/dev/prandom of=/datacrypt/raid0/cryptfile bs=512 count=615278084

From the disklabel and fdisk we did earlier, we should know how many blocks (being 512 or 1024-byte blocks) there are on the disk. (my experience is, that it still works, even if you set the count too high)

Now we need to make up where we want the accessible/readable part to be mounted. In my case, that’s /data

We’re going to use vnconfig to associate the cryptfile on the disk with this mountpoint and of course including the encryption…

/usr/sbin/vnconfig -ck -v svnd0 /datacrypt/raid0/cryptfile

Now you will be asked for the encryption-key… enter something good (and something you’ll never ever forget!)

If everything worked out, it’ll also show you how big your new ‘disk’ is:

# /usr/sbin/vnconfig -ck -v svnd0 /datacrypt/raid0/cryptfile
Encryption key:
svnd0: 314945404928 bytes on /datacrypt/raid0/cryptfile
#

Now we have to initialize the disk and make a new filesystem on the encrypted partition:

fdisk -i svnd0
disklabel -E svnd0 (and again, make one ‘a’ big partition, of type 4.2BSD, of this)

newfs /dev/rsvnd0a

Now we can go off and mount it…

mount /dev/svnd0a /data

and tada! /data is just like your everyday mounted volume… only it’s fully encrypted :-)

Now you can go off and, for example, configure Samba to share this volume in a windows-network (if those machines only knew…)

You can unmount the volume by:

umount /data
vnconfig -u -v /dev/svnd0a

I’ve added a script for convenience, cryptfs (and with multiple disks, you can just copy this script as cryptfs1 cryptfs2 and such), put it somewhere in your path and you can mount it by

1. cryptfs -m Encryption key: secretstring

and unmount it by:
2. cryptfs -u

To monitor the status of the array and the smart-status of the individual drives (thank you, Scott!) automatically you should create a shell script similar to this one:

#!/bin/sh

ARRAY=raid0
DRIVE1=wd0
DRIVE2=wd1
MAILTO=mark@geektechnique.org
if ! raidctl -s $ARRAY | grep -q ‘Parity status: clean’; then
raidctl -s $ARRAY 2>&1 | mail -s “`hostname -s`‘s Array failed: $ARRAY” $MAILTO
fi
if ! atactl $DRIVE1 smartstatus | grep -q ‘No SMART threshold exceeded’; then
atactl $DRIVE1 smartstatus 2>&1 | mail -s “`hostname -s`‘s SMART-errors on drive: $DRIVE1” $MAILTO
fi
if ! atactl $DRIVE2 smartstatus | grep -q ‘No SMART threshold exceeded’; then
atactl $DRIVE2 smartstatus 2>&1 | mail -s “`hostname -s`‘s SMART-errors on drive: $DRIVE2” $MAILTO
fi

Run the script with cron every 15 minutes and it will notify you via email if a drive starts to go south and/or completely fails. Because CMSs like mine never get the accents right in a shell script, I also put it here: raidcheck

(and if you’re going to use an ‘outside’ mail address(like me), please check on the commandline first with ‘mail’ that this machine can mail to the outside world ;-))

what if the youknow hits the fan

Now I got to writing this how-to, after I was forced into refamiliarizing myself again with all this after a power outage in my home. My machine (I’ve been running an encrypted fileserver now for more than two years and with RAID for more than 8 months) easily gets uptimes of 100+ days. In my particular case, when booting, the machine would hang(!) at recalculating the parity… not nice. One of the downsides of running software RAID on OpenBSD is that all the work that needs to be done after an unclean shutdown, gets done before the machine boots up completely, that’s also before any SSH-daemon is running ;-). In a home-setup, with physical access to the machine that’s not a big problem, but remember this when you would even think of running something like this in a production-like environment(which I would totally not recommend). Anyway, at boot, you can ctrl-c your way through the parity calculation and have the machine boot anyway (and be glad you have a separate OS-disk!). Now the last few days, since I didn’t knew why it would hang, I’ve been playing around with a lot of the possibilities of raidctl. I even made two degraded raid-arrays (instead of one ‘normal’ array) of my two disks, both with a bogus ‘failing’ drive, both with an encrypted filesystem on top and copy all the contents (300GB) from one to the other… (which still went with something like 15MB/sec)

Anyway, in the end it seemed I had a rather full root-filesystem (*ahem* I should’ve spotted that earlier on…) and even though I did filesystem checks on the OS-disk (by going back to single usermode) and on the raid-array, I somehow forgot to do filesystem-checks on the ‘virtual’ encrypted filesystem and exactly there were a lot of errors to repair (and that caused the machine to hang and/or go back to the debugger even when copying to it). So, as a lesson, if the shit really hit the fan and your machine acts up like mine did (for the first time in 2+ years), make sure you also fsck the svnd0a (in my case).

this HOWTO is part of this article

DIY trackback:
slashdot
openbsd.org – press ^_^
net-security
G.C.U. (french)
astalavista
BSD news
del.icio.us (at a certain point it was very hot!)
hackaday

permalink

  1. otto@ @ 16 July 2007, 08:40 :

    Never make filesystem on a raw c partitition. fdisk (if your platforms needs it) and disklabel the svnd partition like any other disk.
    If things go wrong in a particular way, fsck_ffs will not be able to handle a raw partition.

    otto@



  2. newbie @ 16 July 2007, 09:20 :

    raw partition? what does that mean?



  3. adamo @ 16 July 2007, 09:49 :

    Raw partition is the “c” partition. In the case of this document this is /dev/svnd0c



  4. markie @ 16 July 2007, 10:34 :

    >fdisk (if your platforms needs it) and disklabel the svnd partition like any other disk.

    That would mean yet another filesystem on top of of growing list in this particular setup…

    I never had any problems with the encryption like it is pointed out here, it’s running for over two years like this. This week I had a problem and in the end (like I point out at the bottom of the how-to) I was able to fsck svnd0c… There were no problems doing so… So yes, you’ve got a point, but I’m reluctant in building yet another filesystem on top of this all… Anyone else have something to say about this?



  5. It's not another filesystem.. @ 16 July 2007, 10:55 :

    It’s just a disklabel, setting up a slice; one more step, and then you build things on /dev/svnd0a (the a slice, instead of the ‘c’ whole disk slice)



  6. markie @ 16 July 2007, 11:03 :

    Oh, yes, (geez I need to wake up), you’re right…

    I guess you can point as easily to svnd0a instead of svnd0c here:

    > /usr/sbin/vnconfig -ck -v /dev/svnd0c /datacrypt/raid0/cryptfile

    But like I said, fsck doesn’t seem to have a problem fsck-ing svnd0c



  7. bigm @ 16 July 2007, 11:08 :

    you mean raid1 right…..?



  8. markie @ 16 July 2007, 11:12 :

    euhm, it’s the first raid-array and therefore called raid0, but it’s a raid 1 array yes ;-)



  9. Kevin Maguire @ 16 July 2007, 12:55 :

    Only in this kind of forum would someone write “its the first raid-array and therefore called raid0 “ without intending any irony :-)



  10. otto@ @ 16 July 2007, 14:11 :

    You are not having problem since you’ve probably not encountered one particular kind of corruption. But let me tell you will regret not following this advice the moment your primary superblock is corrupted.
    The code to scan for an alternate superblock won’t work on raw (that is: c) partitions.

    Otto

  11. markie @ 16 July 2007, 14:28 :

    >But let me tell you will regret not following this advice the moment your primary superblock is corrupted.

    Okay, well, it does make sense what you say. I will check up on this and if necessary change the how-to okay? (I’m not at home atm)



  12. otto@ @ 16 July 2007, 14:32 :

    This would be the steps:

    vnconfig … svnd0 …
    fdisk -i svnd0
    disklabel -E svnd0
    [ create an ‘a’ partition spanning the whole disk, with type 4.2BSD ]
    and then create the filesystem on rsvnd0a

    Of couse the fdisk and disklabel step only need to be done a single time.



  13. LuSiDe @ 16 July 2007, 15:19 :

    To be frank, I find your performance crap compared to a Sempron 3000 (which is quite low power) w/o disk encryption saturating at 55 MB/sec with GELI whereas GELI has a 55% overhead. It seems like a horror to setup compared to solutions I’ve used, and I put huge questionmarks around several of your claims.

    However, what strikes me most is why on Earth did you pick OpenBSD instead of a different OS like say Linux, FreeBSD, or even NetBSD? OpenBSD is not the right tool for this job, given you are so devoted to disk encryption. If you want a decent implementation of disk encryption, look here for a comparison: http://mareichelt.de/pub/notmine/linuxbsd-comparison.html it is slightly out of date as LUKS supports LRW since Linux 2.6.20. What one can conclude from that paper is that the self-proclaimed most secure OS on the world’s disk encryption implementation is vulnerable to a dictionary attack since it doesn’t implement PKCS#5 PBKDF2. It has no key management whatsoever and it lacks other basic features which LUKS, GELI, and CGD do have. Even TrueCrypt is better. If you want a NAS with disk encryption there are several user-friendly solutions for that such as FreeNAS and CryptoBox.



  14. ferzkopp @ 16 July 2007, 15:21 :

    Now all we need to add is an LCD screen and number keypad on the front to: – show the mount/encryption/raid status – interactively unmount on keypress (i.e. when you leave the house) – re-mount after a secret code entry (i.e. when you come back)

    Because what’s the point of encryption if the NAS is always on and the passphrase is stored in the bootup shell script.



  15. scot bontrager @ 16 July 2007, 16:44 :

    You need to add some SMART to your mail script:

    /sbin/atactl /dev/wd0c smartstatus >/dev/null
    /sbin/atactl /dev/wd1c smartstatus >/dev/null

    This will (may) let you know if a drive is going south BEFORE it dies. Checking the RAID is good, but so is checking the physical device.



  16. John @ 16 July 2007, 17:15 :

    > LuSiDe @ 16 July 2007, 14:19 :
    > To be frank, I find your performance crap compared to a Sempron 3000 (which is quite low power) w/o disk encryption…

    Yeah, w/o encryption, you Linux warrior simpleton…



  17. LuSiDe @ 16 July 2007, 17:44 :

    Actually, that machine ran FreeBSD, and the overhead of GELI is about 55%, whereas that machine is less powerful than this Opteron. I provided links why OpenBSD’s disk encryption is crap. Feel free to refute them w/your usual Theocratic illusions.



  18. sparky @ 16 July 2007, 18:35 :

    I have always stayed away from OBSD for my file servers (tho i use it for everything else). Anyone have experience with mega-filesystems on OpenBSD? say ~5-6TB. do fdisk and disklabel still work? how about GPT?



  19. beano @ 16 July 2007, 18:54 :

    markie, I wouldn’t bother asking for second opinions on what otto told you… the end of “otto@” is “openbsd.org”, so he’s probably giving you good advice :)



  20. ddp @ 16 July 2007, 19:03 :

    > LuSiDe @ 16 July 2007, 14:19 :
    > disk encryption implementation is vulnerable to a
    > dictionary attack since it doesn’t implement
    > PKCS#5 PBKDF2.

    Check mount_vnd(8) for the -K option. Time to find a more up to date link. ;)



  21. LuSiDe @ 16 July 2007, 20:47 :

    That is good news!

    However, “The mount_vnd command first appeared in OpenBSD 4.2.” Yeah, go run -CURRENT on a production server. See you again on Novembre 1, 2007. ;)



  22. drm @ 16 July 2007, 21:29 :

    Thanks for the interesting article.

    It looks like there is a minor cut-and-paste error in the instructions for creating the partition on the second disk. (Commands are referencing wd0, instead of wd1.) While this would only trip up a complete novice, it does mar the cookbook-like simplicity of your howto.



  23. markie @ 16 July 2007, 21:47 :

    @otto:

    I changed the how-to accordingly… (I just implemented the changes on the box itself)

    (you guys probably don’t want to know how I did this with data on the drives… to keep the data, I once again splitted the raid array in two degraded raid-arrays, both with a bogus-disk (which is failing of course), I implemented Otto’s changes on one of them and build a new filesystem on svnd0a and now I’m copying all the data from the one array to the other… 0_o) (after which I’m gonna rebuild the second drive from the first and then we’re back! without any external data copying)

    anyway, thanks! I guess this whole encryption setup now finally is theoretically sound (well, more than it was anyway) ^_^



  24. otto@ @ 16 July 2007, 21:57 :

    >However, “The mount_vnd command first appeared in OpenBSD 4.2.” Yeah, go run -CURRENT on a production server. See you again on Novembre 1, 2007. ;)

    You show a lack of research. The -K option is available since 4.0 in vnconfig...
    mount_vnd is a new command in current, which acts like a mount helper, so svnd can be more easily used in fstab.



  25. markie @ 16 July 2007, 22:22 :

    @drm:

    >It looks like there is a minor cut-and-paste error in the instructions for creating the partition on the second disk.

    yes, ahem, you’re right, fixed it, thanks!



  26. necr0man6er @ 16 July 2007, 22:47 :

    you got slash-dotted again :)



  27. LuSiDe @ 16 July 2007, 22:57 :

    Good Otto, congratulations. I did try this with an earlier OpenBSD version and I switched to LUKS and later GELI. Apparently OpenBSD is catching up. Now key management, LRW, changing password without re-encrypting, emergency destruction, and perhaps someone who makes an easy NAS solution based on OpenBSD, but like e.g. CryptoBox/FreeNAS… see you Novembre 1, 2007 ;)



  28. badmacktuck @ 17 July 2007, 02:24 :

    i’m totally new to bsd variants. any chance of this working on m0n0wall?



  29. Natalia G @ 17 July 2007, 06:42 :

    > i’m totally new to bsd variants. any chance of this working on m0n0wall?

    Son, at that point, use freenas.



  30. markie @ 17 July 2007, 17:28 :

    Alright, after a night of copying and quite some hours reconstructing, I guess we’re there… I changed the cryptfs -script accordingly and everything should be good to go, including Otto’s changes. I just rebooted, the raid-array is healthy:

    # raidctl -s raid0
    raid0 Components:
    /dev/wd0a: optimal
    /dev/wd1a: optimal
    No spares.
    Parity status: clean
    Reconstruction is 100% complete.
    Parity Re-write is 100% complete.
    Copyback is 100% complete.
    #

    and after mounting, it looks like this:

    # cryptfs -m
    Encryption key:
    # df
    Filesystem 512-blocks Used Avail Capacity Mounted on
    /dev/sd0a 4124380 3621560 296604 92% /
    /dev/sd0d 11211304 286320 10364420 3% /var
    /dev/raid0a 615278084 615278052 -30763872 105% /datacrypt/raid0
    /dev/svnd0a 605208900 551019196 23929260 96% /data
    #



  31. markie @ 17 July 2007, 18:28 :

    Thanks Scott, I also added a SMART-status check to the script.

    (we’re getting somewhere! ^_^)



  32. lessaid @ 5 August 2007, 16:28 :

    This is the French traduction made by Matthieu Angeli for OpenBSD-France (French OpenBSD Group)
    http://www.openbsd-france.org/documentations/OpenBSD-NAS-encrypte.html



  33. boris @ 15 September 2007, 00:35 :

    Would it be possible, and how, to mount /var to encrypted partition.



  34. FlingeR @ 14 March 2009, 01:52 :

    Aquí pueden encontrar varios manuales para OpenBSD: (in spanish)
    http://www.fencus.net/foros/-indice-manuales-para-openbsd-t-552.html



  35. Takeshi @ 19 August 2009, 01:36 :

    Firstly: nice how-to!

    Three comments, though:

    1) Instead of -k, you should use -K in conjunction with -S, and store the salt file somewhere else. That protects against some attacks and makes it harder to decrypt even with the passphrase given.

    2) You should also use a non-obvious number of rounds with -K, so that you will additionally have to know that in order to decrypt, even given the passphrase AND salt file.

    3) Actually, you don’t need to back svnd with a file, a device will do. At least it doesn’t complain about me using /dev/wd1c, so I assume that /dev/raid0c would do just as well. One less FS to become corrupted, and no need to fdisk / disklabel it (so it’s hard to prove there’s anything besides random junk on the drive).

    Instead of the file, you can fill the raw device with randomness, or you can fill the encrypted raw device (before creating an FS on it) with anything you like, as that will, through getting encrypted, also produce random-looking data all over the drive. Of course, you can use random data, and / or fill this with an svnd device set up using different (random…) encryption key / salt / rounds than what is going to be used for the actual storage…



  36. Takeshi @ 19 August 2009, 02:05 :

    Almost forgot… there’s now also an “encrypting RAID discipline” (RAID C), see bioctl(8). AFAIK it uses AES128 with XTS, but does not seem to have support for a salt file or anything besides a plain passphrase. Of course, that can be emulated using a mount-script, but it’d still be nice to have it out of the box, plus selecting key size and cipher would be better still, since AES, especially with only 128 key bits, seems on the verge of being weak to me, though of course I don’t know if the implementation can make up for that. Of course, if your passphrase is “god”... ;)



  37. FlingeR @ 2 August 2010, 22:18 :

    Nos mudamos de fencus a www.opentz.com.ar.

    Nuevo link:
    http://www.opentz.com.ar/viewforum.php?f=4



As mentioned in the Message from Mark's family this site has been made static. This means that it will be no longer possible to comment on his ideas and projects, but that we all can continue to cherish his creativity.

             you can find all of my projects overhere