I keep thinking there might be a point when the efficiency gains that let the smaller models keep up with the big ones run out (as the big players adopt these too), then the only way to match capability will be to have nearly the same 2 Trillion or whatever parameter size. Hope open source keeps up! 🙏
Discussion
Yea I wonder what the coming days will look like now that the cat is out of the bag. The big players won’t sit idly by for sure.
Of course consumer hardware is advancing too. Supposedly with a couple of Nvidia’s upcoming Digits computers connected together you’ll be able to run a 400B parameter model locally.