If you have the space, any old computer. If you like it compact, you get a lot of bang for your buck with the small 1-liter PCs. I like the Lenovo m920x in particular: 2 x m.2 plus a 2.5 SSD.

For the OS: Proxmox. That will let you play around with VMs and help you learn how to run stuff without having to reinstall the entire computer when something goes wrong.

When you feel like you've outgrown that, a next step could be a dedicated NAS.

Are you sure you want to host your own mailserver? It's a hazzle, and also something the VPS host might not allow.

If you don't do the actual mailserver, #Nextcloud has calendar, contacts and webmail apps. (Plus a lot more)

Regarding the VPS I would suggest one in a datacenter with good connections to where you usual use it from. I've used Digital Ocean and Hetzner. Start with a small one and play around until you are comfortable with and move up to a larger more permanent one.

Did you notice any improvents in usage when going from Docker to NixOS?

I've moved between the two a couple of times before really starting to use Nextcloud, and I didn't really feel an improvement in normal use.