The prime intellect folks are very much worth paying attention to for large scale decentralized AI training:

https://www.primeintellect.ai/blog/intellect-2-release