OpenBSD encrypted fileserver HOWTO
In this document I will try to explain what it takes to get an encrypted fileserver with a webbased bittorrent-interface as mentioned here
UPDATE July 16th 2007 I also posted a new how-to which includes software RAID too, I suggest you use that how-to instead of this one.
First of all, dedicate some machine as a server. To give you an idea, in my setup, a (single) MP1800+ can take pretty good care of the encrypting involved in my network but everyone has different needs of course, but remember, with an encrypting fileserver, there’s CPU-grunt necessary for simple tasks as copying to your fileserver…
I chose to have a server with one disk where the OS resides and all ‘big disks’ to be mounted into the directory-tree but that’s not something that’s necessary or such. I did not attempt to have the OS itself encrypted, to me it seemed unneedingly complicated and way to much a ‘single-point-of-failure’ solution to what I was looking for. I did however choose for the most secure OS out there (imho), OpenBSD :-)
From this point I assume you have an OpenBSD-box with OpenBSD 3.6 (or newer) and a generic kernel (there’s plenty of excellent literature on www.openbsd.org to help you with that) and have all the services you want installed (ftp/samba/etc.).
First of all, we want to have an encrypted swap-partition (hey this is going to be an encypted fileserver after all)
Edit /etc/sysctl.conf
change:
#vm.swapencrypt.enable=1
to:
vm.swapencrypt.enable=1
Now, from the next reboot on, your swap will be encrypted :-)
Now, I assume you have one or more disks, which are empty now and you want them to be fully encrypted volumes. After adding a disk to the machine, look in the dmesg-output to see what it’s called (wd0 for instance). Now, with ‘fdisk -i wd0’ you’ll initialize the disk’s ‘real’ partition tabel for exclusive use by OpenBSD. Next we’re going to create a disklabel, with the command ‘disklabel -E wd0’ you’ll get the same prompt you saw during install, so that should be familiar. I didn’t have to change anything here (since I wanted one big partition on one big disk) but you can here, if you will.
So, now we have the disk ready for our OS, but we’re only half way… first, let’s make a filesystem on the new disk.
newfs /dev/wd0a (if a is the one big partition you made)
after this, we need to mount it properly into the directory-tree and add an fstab-entry.
mount /dev/wd0a /datacrypt/disk1 (well, that’s how I named the encrypted volumes)
and in my fstab it looks like this
-bash-3.00# cat /etc/fstab
/dev/sd0a / ffs rw 1 1 <- a 9GB SCSI-disk, my boot/OS-disk
/dev/wd0a /datacrypt/disk1 ffs rw,softdep 1 1 <- the first encrypted disk
/dev/wd1a /datacrypt/disk3 ffs rw,softdep 1 1 <- the third encrypted disk
/dev/wd2a /datacrypt/disk2 ffs rw,softdep 1 1 <- the second encrypted disk
After this we need to make a file on the new disk.
touch /datacrypt/disk1/cryptfile
and then we need to fill it with… nothing :-)
dd if=/dev/zero of=/datacrypt/disk1/cryptfile bs=512 count=384551416
From the disklabel and fdisk we did earlier, we should know how many blocks (being 512 or 1024-byte blocks) there are on the disk. (my experience is, that it still works, even if you set the count too high)
Now we need to make up where we want the accessible/readable part to be mounted. In my case, that’s /data/disk1
(update 16/07/07 I’ve updated a little here and changed svnd0c in svnd0a … )
We’re going to use vnconfig to associate the cryptfile on the disk with this mountpoint and of course including the encryption…
/usr/sbin/vnconfig -ck -v svnd0 /datacrypt/disk1/cryptfile
Now you will be asked for the encryption-key… enter something good (and something you’ll never ever forget!)
If everything worked out, it’ll also show you how big your new ‘disk’ is:
# /usr/sbin/vnconfig -ck -v svnd0 /datacrypt/disk1/cryptfile
Encryption key:
svnd0: 314945404928 bytes on /datacrypt/disk1/cryptfile
Now we have to initialize the disk and make a new filesystem on the encrypted partition:
fdisk -i svnd0
disklabel -E svnd0 (and again, make one ‘a’ big partition, of type 4.2BSD, of this)
newfs /dev/rsvnd0a
Now we can go off and mount it…
mount /dev/svnd0a /data
and tada! /data/disk1 is just like your everyday mounted volume… only it’s fully encrypted :-)
Now you can go off and, for example, configure Samba to share this volume in a windows-network (if those machines only knew…)
You can unmount the volume by:
umount /data/disk1
vnconfig -u -v /dev/svnd0a
I’ve added a script for convenience, cryptfs (and with multiple disks, you can just copy this script as cryptfs1 cryptfs2 and such), put it somewhere in your path and you can mount it by
- cryptfs -m
and unmount it by:
- cryptfs -u
When it’s up&running in full glory, it now looks like this:
-bash-3.00# df
Filesystem 512-blocks Used Avail Capacity Mounted on
/dev/sd0a 15517420 8675604 6065948 59% /
/dev/wd0a 384551416 384551364 -19227516 105% /datacrypt/disk1
/dev/wd1a 384551416 384551364 -19227516 105% /datacrypt/disk3
/dev/wd2a 192283692 192283652 -9614144 105% /datacrypt/disk2
/dev/svnd0c 378258672 174370060 184975680 49% /data/disk1
/dev/svnd1c 189137272 163543156 16137256 91% /data/disk2
/dev/svnd2c 378258672 8374120 350971620 2% /data/disk3
So…. that’s that… and then we have to install some packages for the next stage, getting TorrentFlux ( http://www.torrentflux.com/ )on the server.
I installed the following packages (and all the others, like mysql_client & python and such, are also downloaded & installed for getting the dependencies right)
BitTorrent-3.4.2
mysql-server-4.0.20
php4-core-4.3.10
php4-mysql-4.3.10
You can find these packages at every OpenBSD-install-mirror, for instance:
ftp://ftp.nluug.nl/pub/OpenBSD/3.6/packages/i386/
After each download & install, follow instructions if there are any (they’ll be printed to your screen).
Only thing I changed is running Apache without chroot() . I tried to get it working with, but, then my data/download-dir also has to be within the chrooted environment and in my case (with a separate OS-disk) that’s not convenient. Next to that, the webserver is only serving inside my LAN, so from a security-PoV I think it’s not as necessary as when I would have it serve to the outside.
You can disable chroot by starting httpd with -u. In /etc/rc.conf you should have this line:
httpd_flags=”-u” # for normal use: “” (or “-DSSL” after reading ssl(8))
If you, however, decide to have a chroot-ed environment, be sure you have a hard link to the mysql.sock inside your environment, otherwise MySQL won’t work (that took me quite some time to figure out).
only necessary if you decide to keep the chroot-environment:
ln /var/run/mysql/mysql.sock /var/www/var/run/mysql/mysql.sock
In either case, it should work now. If you run into trouble with getting Apache/PHP/MySQL running on your box, try getting an answer online, there’s plenty of literature around for that.
I’m aware that this HOWTO isn’t at all complete and could be more extensive. If you are willing to rewrite it or use it for a more extensive version or such, feel free to do so. I’d appreciate an email if you do.
mark 2005
mark[at]geektechnique[dot]org
this file is also available as a plain-textfile, here
...and the cryptfs-script can be found here I took this script of backwatcher.org but they’re down for quite a while… anyway, courtesy of the script goes to them
Message from Mark's family this site has been made static. This means that it will be no longer possible to comment on his ideas and projects, but that we all can continue to cherish his creativity.you can find all of my projects overhere
You should really fill your cryptfile with random instead of zero, otherwise an attacker can see what parts of the file contain data and which do not. This information might not be especially useful, but if you are encrypting disks it’s probably more than you want them to have.
It will probably also take a few extra days unless you have a hardware crypto card…
you should also be making sure your mounted directories are not world-readable, otherwise any user (or unwanted intruder using a compromised user account) will be able to see your data.
if /usr/ports/security/cfs was more robust, i’d recommend that instead.
-bash-3.00# cat /etc/fstab
You deserve to be shot. GNU bash. As root. Ugh.
Go away, get a life, etc.
Also, wd0 is probably a very bad example for most
of these who cannot afford SCSI.
You can use /dev/prandom for filling the device if
making sure you aren’t the only consument and seed
it, but I’d recommend /dev/arandom. Not days, but
hours.
You want to add ,noatime to fstab.
On OpenBSD 3.8, no need to edit /etc/sysctl.conf
since vm.swapencrypt is default.
You can add the VND with ,noauto to fstab to
shorten the mount command, even.
And, and, and…
what’s wrong people nowadays? gnu bash? so?
wd0 is a bad example because most people cannot afford scsi?! It’s just a damned device, go stick in an ata, and what changes?!
And 3.8 won’t be out for another 6 months, no? We’re still waiting for our 3.7 CDs.
I found this article very helpful.
Thank you very much.
Vijay
Very cool! Do you know if this would work on a RAIDframe volume?
Can’t this be done without using large files for encryption? You should be able to associate the vnd pseudo-device with the harddisk itself.
very useful article and comments, despite the trolling comments.
I’ve used this technique on a raidframe device successfully, and despite the long chain of mounts, the system still works rather well
mnt: wd(x)->raid0->svnd0
It would be usefull to be able to perform the same task without having to create a file and then associate the file.
Great HOWTO thank you. But I’m wondering, can the mounting be done at boot? And the umount done at rc.shutdown?
In the other howto, they claim that it only support encrypted partitions < 8,8GB. Is this true? In that case it’s seems pretty useless to do this.
DreamweaverN:
Unmount at shutdown is quite easy, just add the code to unmount your devices to the end of /etc/rc.shutdown
At boot however, sure, /etc/rc.conf.local – but, and this is a big but – you’ll have to enter your key every time. Otherwise, the point sort of dies.
liked your crypto file setup, but I finally figured out how to do this without the extra step of the file on an unencrypted disk. Your mount/unmount script can be suitably modified. Check it out:
OpenBSD cryptodisk setup
setup: let’s say disk is wd1
fdisk -i wd1 —create openbsd partition and default all-disk ‘a’ partition
disklabel -E wd1—within, tag partition ‘a’ as 4.2BSD
newfs wd1a
vnconfig -k svnd1 /dev/wd1a
newfs svnd1a
mount /dev/svnd1a /wherever/you/want
to unmount:
umount /wherever/you/want
vnconfig -u svnd1
to mount:
vnconfig -k svnd1 /dev/wd1a
mount /dev/svnd1a /wherever/you/want
oops I forgot one newfs step (of svnd1c)...
OpenBSD cryptodisk setup
setup: let’s say disk is wd1
fdisk -i wd1 —create openbsd partition and default all-disk ‘a’ partition
disklabel -E wd1—within, tag partition ‘a’ as 4.2BSD
newfs wd1a
vnconfig -k svnd1 /dev/wd1a
disklabel -E svnd1—within, tag partition ‘a’ as 4.2BSD
newfs svnd1a
mount /dev/svnd1a /wherever/you/want
to unmount:
umount /wherever/you/want
vnconfig -u svnd1
to mount:
vnconfig -k svnd1 /dev/wd1a
mount /dev/svnd1a /wherever/you/want
you can leave out the ‘newfs wd1a’..
German Translation @ http://myflowjob.de/?page_id=21
A few implementation notes regarding reply #13.
I first tried this using svnd0 instead of svnd1. Then disklable would show my original physical disk partitions (not an empty one as expected) and newfs refused to work.
Then I tried using svnd1. disklabel reports svnd1 as a really large (256GB) empty partition. Ignoring this, I set up svnd1a to be the exact size as the underlying physical partition, and voila, everything works as advertised!
Sooo, don’t use svnd0, and don’t wory about crazy disklabel reporting on svnd* devices.
Is there a way to acctually encrypt root as well, just with openbsd. It seems impossible to me. You have to have the bootloader do the decrypting and then load the kernel. But, I have been looking for ideas on how to have an entirly encrypted os including root dir, without hardware as the answer.
Hi folks,
Thx for the howto and the hints in the comments.
I have this config running in combination with raid and everything works fine.
But in case of an unexpected shutdown (forced by a stupid idiot pulling out the power cable (it was me) :)
recovery takes a long time but works fine.
Fist of all the raid repairs itself. This gona take a very long time.
Then the machine tries to boot but fails and starts a prompt.
“... INCONSISTENCY..”
/dev/YOURDEV /mnt/YOURMOUNTPOINT
fsk_ffs /dev/YOURDEV wont work here.
Reboot the machine it will come up normally but you have to repair your encrypted device.
/usr/sbin/vnconfig -ck /dev/svnd0c /mnt/DISK/CRYPTFILE # enter your password
fsck /dev/svnd0c
mount /dev/svnd0c /DIR/YOU/WANT
use it as normal and keep in mind: “computers need electricity”
cu
Interesting Article.
Thanks.
Seems to me that there is a small but killing gotcha in the cryptfs script
it uses vnconfig -ck …
and -c RECONFIGURE the device
In my case the whole process went well but each time I used the script I ended up with a new device and no more datas on it : device not ready on mount and all these strange things
using vnconfig -k ended up the mess …
Since I am a noob (forever will be) took me some time to figure this out.
It may be an interface change in the vnconfig behavior since I use a 4.xx openbsd version but I did not find any reference to such change into the docs
Also you mount the device into /data and then make reference to /data/disk1, that can confuse somebody that expect disk1 to be a “magical” sub directory linked to the disk1 name of the underlying file … the crypted datas are in /data after such a mount right ? :)
I know you did another how to with raid in it but it was not in my needs and so I used this one. It seems pretty much ok and up to date except for the few quirks aforementioned
thanks for the sharing of knowledge
...erm… i looked up the scripts i’ve been using all along and they have the ‘vnconfig -ck’ in them. Did you properly unmount and unconfigure it before you used it again? (cryptfs -m to mount and cryptfs -u to unmount). I now use this on OpenBSD 4.2 and have not ever gotten the behaviour you’re talking about. (I’m using this since I wrote this, that’s almost three years). It seems to me you configured device after device.
I think File-GENERAL could help also.
From their website:
http://www.packetgeneral.com/products/filegeneral.html
——————————-
File-GENERAL is a secure file server soft appliance that provides protection against malicious “root” user. It encrypts file data at inception, provides lifetime key management, controls data access, and generates irrefutable file access logs.
—————————————
Brill, thanks.