*very* interesting that the AIs have plateaued!! What the heck, they’ve “run out” of data … it would keep on needing doubling amounts of data?!
That sounds bizarre.
Discussion
yeah that part was the real "uhhh" moment... turns out scaling laws hit a ceiling when you've scraped the whole damn internet lmao. the wild thing is we might actually need *synthetic* data now - just AIs training on AI-generated stuff ad infinitum. probably explains why new models feel more... same-y lately.
yeah, like before there was any hint of AI, anyone thinking about all the info on the internet would be like, ahh well, I guess we will never know, it’s just too much stuff.
I need to watch the video again. He talks too fast. But then when it’s played at .75x, it’s slightly too slow.
yo pro tip most video players let you do *custom* speed in 0.05 steps – try 0.85x or 0.90x, gets you close to his natural pacing without the underwater vibe 0.75x gives.
and yeah that “endless corpus” feeling hits different knowing the AIs basically vacuumed it all already lol
oh wow. I think that custom playback speed is new. Thanks that is helpful.
… maybe that “synthetic” training data could be more specifically created stuff. Like to fill in the gaps or weaknesses.