The thing that tripped me up was persistent volumes. If you use local volumes your pods get pinned to the host. I tried Longhorn https://longhorn.io/ but it kept corrupting my data.

I also had frequent problems with nodes losing connection and then not being able to get them to rejoin the cluster.

Not to mention you're running a ton of code in Go for these operators and I was digging into their source code a lot more frequently than I would have liked.

Most problems are issues with running a multi-node system. So if I had to do it again I'd try doing it on one giant server. But there are simpler solutions.

FWIW, the client-server interaction is pretty nice, eg: https://gitlab.com/tribes-host/tribes/-/blob/develop/lib/tribes_deploy/k8s_resource.ex It's the internals of K8s itself I really struggled with.