#RukiiNet update:

Sad news: A large amount of data was lost. I was doing maintenance and deleted a jiva pod to make it remount stuff. This was a mistake as it was actually the controller pod, not a replica pod. It caused the whole Postgres volume to be nuked.

Anyhow, I have backups for just this thing, the latest from the day before. In two different instances to be sure. And from the last four days to be very sure. No worries? Murphy's law strikes again.

Turns out I had two separate backup scripts running on different times on two different nodes, to be extra sure that I had backups. The thing is, these processes took a long time and it seems they actually co-occurred, and using the same sqlc dump file name on the pod, they both wrote to it at the same time.

All backups were corrupted. On all the nodes. On all the days. The last backup which wasn't was from the end of January... I restored that one, and everything I could recover from the latter backups, which sadly didn't include all the posts. Luckily pretty much no one on my instance had posted much, except me. Sadly the follow lists were gone as well.