Andrej Karparhy said something interesting in his interview with Dwarkesh. We know that training these models on large amounts of data produces cognitive capabilities, that's why larger models are better at doing things. But we don't exactly know what parts of the model is actually responsible for it. If we can figure it out, then we can make smaller models much more capable even though they'd be much less knowledgeable

If that's the direction it takes, then eventually I expect to have some standard smallish models that can be fed huge datasets according to your needs, and companies will use these to create knowledge specific models that can run on your regular devices. Or models that can learn stuff on demand on your own system nostr:note1n4xw4pha62lt706w0u76xj3gs8qnrallulkwac324a4zm3mp97usmys6ux