ABitcoinGuy on Nostr: “Open models lag state-of-the-art frontier models by four months. I think the ...

“Open models lag state-of-the-art frontier models by four months. I think the reason it is relatively easy for open source and previous laggards to catch up to within months of the frontier, is that data is the real driver of progress. Data can be easily distilled from public APIs, whereas hyperparameters and training tricks and architectural optimizations cannot.
If the latter were driving most of the progress, then catching up would be far harder than we are observing it to be.

It is easy to forget how much data these models are trained on, and how much more it is than what we humans see in our lifetimes. We see these AIs as the galaxy glittering with capabilities. But at their center, invisible to the naked eye, holding all the constellations together is an unimaginably massive black hole of data.”

From Dwarkesh Podcast: The data black hole at the center of AI