Replying to Avatar iru@localhot $_

We need distributed training so we can create foundational models in a permissionless way with thousands of consumer GPUs. Maybe with Bitcoin bids for processing requests so it will kinda feel like mining in the old days.
Avatar
iru@localhot $_ 1y ago

Like… I post the dataset, along with a bid in a coordinator. The coordinator breaks the tasks into small chunks and distributes the rewards to everyone running the training.

Reply to this note

Please Login to reply.

Discussion

Avatar
someone 1y ago

the training needs big amount of GPU on the same machine but i am sure some will figure out to scale out. mixture of experts is the popular thing nowadays. maybe each of those experts can be trained on a single machine and later combined.