Is AI progress hitting a wall?
Posted on
You might remember as Lan from Medium.
Ah, 2024. What a year!
Whether the AI survives the hype, I feel so fortunate to live in this interesting time in human history. We are already able to do things that would have seemed like unimaginable to our grandma. It makes me really appreciate how fast the industry has progressed.
Just months ago, Sam Altman declared “Superintelligent AI Is Only a Few Thousand Days Away” and that AGI (whatever that means) is “achievable with current hardware”.
But the coin has flipped. Bloomberg reported that leading AI companies including OpenAI, Google and Anthropic experienced diminishing returns from developing new models. Ilya Sutskever also suggests we’ve hit a plateau in scaling up pre-training. The new dominant narrative seems to be that model scaling is dead.
The 2010s and early 2020s were the age of scaling. Deep learning worked. The formula for success was clear: feed a massive dataset into a huge neural network and train it for a long time. Then the magic happens. Unfortunately, this strategy will lead to diminishing returns.
While compute is stronger, there is only 1 internet and we have exhausted all the data, well unless we go synthetic. There are cases where synthetic training data has been successful, such as AlphaGo, which beat the Go world champion in 2016, or CriticGPT by OpenAI for coding.
Then there is the diminishing returns of larger models.
Model performance can be saturated both by the size of training data and model. Gpt-4o is 2x cheaper than gpt-4 (hence smaller) but is better. Apparently, models have been getting smaller but are being trained for longer to reach the same performance level. So this is a trade-off between training and inference cost.
Now where is AI headed?
Some attention has shifted towards “inference scaling”, which focuses on optimizing how the models reasons at runtime. OpenAI’s o1 is an example of inference scaling.
Hard to predict how and when but in his talk at NeurIPS, Sutskever shared a glimpse of the future:
- Superintelligence will come, eventually
- Systems become truly agentic. Current systems aren’t agentic yet, just slightly. I think they can’t really use tools effectively (current tools like Claude’s Computer Use show early signs))
- Systems will reason and understand. The more the AI reason, the more unpredictable it becomes. The very good chess AI must be unpredictable to be able to beat the human chess players.
- Systems will be self-aware
Maybe model scaling is over; maybe not. My personal opinion is that current AI can’t reason at all (though it does a good job at making it look like it does), and we really need a conceptual breakthrough to make it reason (Please read Roger Penrose). Self-awareness is an interesting point: once an an AI is self-aware, can reason and is able to interact with its environment, how far away are we from artificial consciousness?
Everyone is searching for the next things. But one thing seems certain: the future is incredibly unpredictable.
But for now, happy holidays and happy new year!