Until I get to the point that additional iron is warranted (and then I might decide Amazon EC2 is a better fit for us), this machine has to do everything:
- File Server
- Virtual Machine Host
- Kerberos Server
- Home Music Server
- Personal Web Server
This machine has to have rock-solid file storage with backup. At least until btrfs comes out, that means ZFS for the important stuff. There’s lots of good information out there about ZFS; I’m not going to flog it here; look it up
Obviously I need RAID for redundancy in case a disk fails. I have it on good authority that the Mean Time Between Failures (MTBF) for big, cheap SATA disks like mine (the “I” in RAID stands for “inexpensive”) is short enough that if one fails, there’s a very good chance another will fail while you’re sync’ing the failed drive’s replacement. So that means two bits of RAID parity – RAIDZ2.
As it turns out, I’ll end up running Linux. While you can hack linux into booting/rooting off ZFS, there are loads of problems (read about that here). So that means, additionally, mdRAID6 for my base system. I’ll just have to live with the RAID5/6 write hole and rely on my UPS and, failing that, backups to ZFS and offsite, for recovery of the base system.
Oh, and since I need all the snapshotty goodness of ZFS for making backups, I’ll need to put my whole root partition on LVM.
But we’re not done yet. GRUB-legacy can only boot off RAID1 disks. I tried valiantly to install GRUB2 on RAID6 (they claim it works), but failed miserably with the Ubuntu packages, and the GRUB2 source wouldn’t even build on amd64 [Update/News Flash: I may be wrong] [Update #2: still no dice, though]. My /boot partition is thus on a 3-disk mdRAID1 mirror.
- KVM – requires CPU virtualization support. Too bad; the KVM machines I’ve run on my laptop are very snappy.
- Xen – requires CPU virtualization support if you need to virtualize Windows, and I do. Too bad; I understand it is a really efficient virtualization scheme. Oh, and Ubuntu has dropped support for running as dom0, banking on KVM instead.
- VMWare ESXi – Free and high-performance, but my low-budget SATA controllers are at best off in the weeds of experimental, community-supported, workaround-required hackery, and most likely even if I got it working I wouldn’t be able to run a RAID configuration. [Update 2009-03-09: some helpful souls on the #vmware IRC channel IIRC seem to think it should work as long as most of my VMs are willing to wait for the VM serving ZFS via iSCSI to boot... sheesh]
- VirtualBox – I really want to like this one, but there are obstacles.
- the licensing: it looks like the full-featured version of VBox isn’t really free for commercial use, so I’d be giving up several of the usability features that make VirtualBox cool, unless I paid some amount of money between $30 and $500 yearly; they don’t seem to say what you get with the “Premium Server Subscription” or which plan is consistent with my use cases.
- FreeBSD only “works partially” or so says their guest OS status page (although a recent release seems to have some related fixes), and specifically that “FreeBSD 6.2 is known to cause problems.” Since boostpro.com runs on FreeBSD 6.2, I’ve really wanted the ability to virtualize it here for testing purposes. However, I’ve never really used that ability for much, so maybe it shouldn’t be such a big consideration. I suppose my virtual FreeBSD installation could be on different hardware.
- VMware server – Free, does everything I really need it to. I don’t know how well it performs.
- VirtualBox on Xen – yes, I considered this as a way of virtualizing Windows on a Xen-based system. Doesn’t work.
- VMWare on Xen – One guy claims it worked but most of what I’ve seen says it won’t. Actually, whadda ya know; here are some instructions for installing it (you install over a non-Xen kernel). But seriously, does it work once you’re back to running on the Xen kernel? I guess this is something worth trying.
Looks like VMWare Server wins, with a possibility of trying it along with Xen.
As stated earlier, I want to run ZFS, so there are two obvious alternatives: FreeBSD and OpenSolaris, which both have native implementations. Unfortunately, FreeBSD has essentially no virtualization technology (jails don’t let you run other OSes) and Solaris only supports VirtualBox.
Fortunately, there is a userspace implementation of ZFS for Linux. If VMWare on Xen actually works, I might consider trying a Solaris or FreeBSD Xen domU to run the fileserver. Since my ZFS is running on partitions, I should be able to read it just fine from the Xen guest.
I’d have liked to start with Gentoo or something else that builds from scratch, to squeeze every last possible bit of performance out of my hardware. However, I didn’t have time to wrestle with a new distro, and since the Gentoo Wiki was recently wiped from the face of the web (why didn’t they just crawl Google’s cache to retrieve the pages—they showed up in searches for months?) I didn’t think it was a good bet I’d readily find all the information I wanted. For now, I’m sticking with what I know best – Ubuntu.