Hmm this was something i actually wanted to look at.... i want to offer SaaS services on K8's
Discussion
The thing that tripped me up was persistent volumes. If you use local volumes your pods get pinned to the host. I tried Longhorn https://longhorn.io/ but it kept corrupting my data.
I also had frequent problems with nodes losing connection and then not being able to get them to rejoin the cluster.
Not to mention you're running a ton of code in Go for these operators and I was digging into their source code a lot more frequently than I would have liked.
Most problems are issues with running a multi-node system. So if I had to do it again I'd try doing it on one giant server. But there are simpler solutions.
FWIW, the client-server interaction is pretty nice, eg: https://gitlab.com/tribes-host/tribes/-/blob/develop/lib/tribes_deploy/k8s_resource.ex It's the internals of K8s itself I really struggled with.
Yea storage has always been the worst part to manage, i mostly use NFS, using nfs-subdir-provisioner and then just plain manual provisioning (subPath)
My previous setup was 1 Master + 3 Worker nodes in the same data enter, i never had any problems, but i guess it really depends exactly what setup you use.
For me i used kubeadm + flannel, a very basic setup, and also MetalLB in the new setup (and home lab)