"Steve's Place"

Proxmox is a virtualization server that I have become quite fond of because LXC containers are first-class citizens. Easy to back up, light on CPU load, and you can change CPU/RAM on the fly. Backups basically snapshot and copy, though you can pause as well. It runs on everything I've put it on so far, and it clusters.

technical debt

“Technical debt” is a phrase we geeks use to describe the situation where we built something and it evolved on the fly and we later learned that we should have done it differently but are in a situation where we can't afford (time, materials, data, whatever) to go back and do it right. So we often end up hacking at stuff and doing it even wronger in order to avoid that heavy lift of going back and doing it right.

What I'm about to describe is 'technical debt'. It's one of those “don't try this at home, kids” things, and I'm describing it for folks who may find themselves in the same exact situation of technical debt.

SPOF

Yes, I realize that this process renders one node of my Proxmox cluster a “SPOF” (single point of failure). And that this compromises any concept of “HA” one might gain from a “cluster”. But this is a home lab and I have a standalone NAS for backups. There's little mission critical going on here other than stuff I can back up and restore. So my risk exposure is just a bit of downtime, no more. Most of what runs on my cluster is dogfooding as I learn Kubernetes and BGP and the like, while I run my Fediverse services (like this blog, my PixelFed, my Mastodon and Pleroma) and other useful stuff like hastebin, flashpaper, and picoshare. If you think you need five nines of uptime, you probably shouldn't be hosting your own stuff in your basement like I am :D

Virtualization

I figured out a long time ago that I could get old enterprise servers for cheap and stick them in a room in my basement (my “datacenter”) and have significant server power for peanuts. The current incarnation of this concept is a pair of HP DL385p boxes with 32 Opteron cores each and 128G RAM each. These boxes cost me roughly $250 each, delivered, and the kill-o-watt attached to their PSUs says they cost me about $6/month. (I have fairly cheap power). I DARE you to find a VPS for $6 a month – or even an order-of-magnitude more – that has 64 cores and 256GB RAM. ( that bit is just there for all the folks that always say “Why not use a VPS?” )

The one drawback of enterprise servers is theyr'e picky. Picky about RAM, picky about all kinds of things, but particularly picky about hard drives. These machines have... eight? 2.5 drive bays per, but if you put anything but HP drives in 'em they complain. What's WORSE is that if you put more than FOUR unrecognized drives in 'em, the chassis fans all kick on “JET ENGINE MODE”. This is unacceptable even in my little lab. As it is, the brief thirty seconds of JEM we get on reboot is enough to wake you up if power bounces in the middle of the night.

Before these boxen I ran VMWare's free edition, but it didn't like the Opteron processors on these machines and I wasn't going to fight with it, so I switched to proxmox and I'm glad. I can see how VMWare's enterprise tools might be valuable to ... an enterprise ... but they're useless for my little lab.

So I put four random 2.5” drives I had laying around into the server and installed Proxmox and created storage willy-nilly. If you're just starting out, be aware that it supports shared storage like Ceph and if you can buy identical drives for three nodes, by all means, close this page immediately and go do that very thing. When I can afford to buy eight 2 or 4TB SATA SSDs, I'll probably back this whole cluster up and rebuild it that way, but until then...

Storage

I have this drive bay I bought years ago that fits in three full-sized CD drive slots and holds five drives vertically. It's been laying around the house, but when I decided I needed more storage in my life I ordered four ( lol ) HGZT 8TB drives and stuck these drive bays in my ex-windows box (A Ryzen R7). I threw the four drives in four of the five bays, hammered it into my old cooler-master case (it's an awesome case but I don't need a windows box anymore), and installed Proxmox and added it to the cluster, then made the four drives into a 24TB RAIDZ pool.

If you're paying attention, you'll probably guess that the storage is only accessible from that proxmox node, not the others. And you'd be right. But I'm just crazy enough to violate RFC1925 and add YET ANOTHER layer of indirection, and also to move the problem to a different part of the network. What did I do? I downloaded TrueNAS Core, of course. I then created a virtual machine and installed TrueNAS on it, and gave it a big chunk of that storage to work with. I THEN created an NFS share on that TrueNAS server.

Storageception

At this point you might want to close your laptop in disgust and walk away as you start to comprehend the technological abomination I'm describing. I wouldn't blame you. I came pretty close myself, while writing this. But we're just to the juicy part, so here we go.

I then went to the Proxmox UI in the Datacenter and added NFS storage to all nodes, pointing to the TrueNAS vm that I had just configured. It happily added the storage and it showed up on all three nodes and et voila! I have RAIDZ shared storage.

To test it, I created a container on one node using the NFS storage. I then right clicked on the container and told it to move it to another node. It promptly did so. MISSION ACCOMPLISHED.

Ultimately, this does not present more risk than simply running a single-node external NAS server, but adds the benefit of using the other six Ryzen cores for compute rather than simply storage. My other NAS is an old 2012 Mac Pro with 12 Xeon 3.46GZ cores and I find myself running services on that to avoid wasting the compute power serving files, so this is kind of a shortcut.

You do have to make sure that your “NAS” vm comes up first, but you can make that happen in Proxmox fairly easily.