I've recently completed my CKA certification after a multi-year journey of learning and experimenting. I started with minikube and went through all the various iterations, up to and including OpenShift (OKD) 4.10. I maintain a cluster in my lab to keep my hand in and extend and expand my knowledge. Frankly, for my small applications, Proxmox's LXC containers are a superior solution, but I need K8S chops in my day job, and it's fun in a perverse sort of way. It's entirely overkill – as is my entire home lab – but you don't need to have the homelab illness I have to operate a kubernetes cluster.

minikube

If you have the ram and the CPU power, minikube is a great intro. It spins up a tiny single-node k8s cluster by default, though you can scale it to multiple nodes if your computer has the gas to pull it off. Great way to get your feet wet, or to do local development of apps and packages. You could technically actually use it for a k8s cluster but it's probably not the best solution for non-development workloads.

'Vanilla K8S'

My next step was a cluster of VMs in Proxmox running Debian and “k8s from scratch”. Just stepping through the manual install process, even with kubeadm, learning about CNIs and config details was very useful and critical. This is a great learning process and will help you understand the details around kublet and etcd that you'll need if you wanna get certified. Pretty easy with a single master, but HA vanilla K8S is harder; you'll need proxy chops to configure your API proxy if you want more than one master. You'll also need a lot of machines, or you'll need to enable workloads on your masters. I found out that the triple etcd database itself put significant load on my VMs, and DR was a PITA. A power failure killed my etcd instances and I eventually deleted that cluster and started over.

Openshift/OKD

Next was Openshift via OKD. OKD's relationship to Openshift is somewhat similar to the relationship between Fedora and RHEL; it's the opensource core that RedHat adds proprietary tools and support to and calls Openshift. Lemme just say Openshift is a HOG. You'll need five machines for HA (vms work, but they have to be pretty robust). You'll need a lot of network setup, including DNS and probably vlans. Openshift offers a lot of canned features that are not in vanilla K8S, from a web console to an operator store, security and routing advancements, etc. There's an Openshift solution similar to minikube called CodeReady Containers that lets you run a minimal OKD 4 single-node cluster on your local box for development and learning. I've not spent much time with it, tbh.

The biggest problem I had with OKD was that every time I had a power outage I had to rebuild the cluster. It wasn't hard because I'd created an ansible playbook that did everthing. I'd kick off “build-cluster” and it would delete the old virtual machines, create new ones. configure them, install OKD, and then I'd restore the backup. But it was a PITA. Also, I wanted my CKA and Openshift was teaching me bad habits (not the least of which was thinking in oc instead of kubectl).

Microk8s (from canonical)

So I went looking for a “light” K8S version. Microk8s looked interesting. I liked the very lightweight database (dqlite) and low idle load I got from deploying it. The built in options are awesome; you can just enable stuff like istio, k8s dashboard, metallb, observability, traefik – all sorts of stuff. Easy-peasy. This was great for several months. Then I had a power outage that crashed the whole system. I restored from node backups and unfortunagely could never quite get the cluster happy again. It had something to do with the k8s backing store (dqlite in this instance) but I was unable to discover a solution, though I found quite a few posts from people who were in the same position; unable to recover their microk8s cluster despite having good recent ( like, six hours old) backups.

K3S

So recently I decided to give k3s a go. I made this decision because several products I'm familiar with run k3s under the hood and they have been pretty reliable, and I like the fact that it can use an external database – in my case mariadb – that I can back up separately and eventually even make into an HA configuration. The documentation isn't as clear and clean as microk8s, and I had a few false starts along the way, but once you sort it out it's pretty straighforward. It'll run in very small VMs (though for my home lab I have a four-node cluster of VMs with 8 VCPU and 16+ GB RAM). I've only been running k3s for a couple of weeks, but so far it's been pretty tight; I had a power drop during a storm and lost two nodes, but they came back online with no trouble. It recovers rapidly from a lost node, it has very simple HA config (though my DB isn't yet HA, I have four API nodes that work just fine).

Ultimately, every step in that path was a learning experience important to my understanding of the overall k8s system. You may or may not want to recapitulate that sequence, depending on how deep you need your k8s knowledge to be, and how deep your existing infrastructure knowledge is to start with. But if you wanna skip to the end, I don't blame you. I'll be adding some articles on specific solutions I've come up with in my lab, but here are a few of the resources I referenced while building out my own k3s cluster:

• Setting up your own k3s cluster (fullstaq)
• K3S quickstart guide (project page)
• Getting Started with K3S (baeldung)