For decades, the Moore’s law was thought to be the standard of how rapid technological advancement could be. All the way back in 1965, Gordon Moore had predicted that the number of transistors on an IC would double every two years, and this prediction has held true for decades, which has led to the enormous capabilities of our computers today. But there are indications that AI infrastructure could be getting better even faster.
NVIDIA CEO Jensen Huang has said that computing power in AI is growing four-fold every year, which would lead to a million times improvement in AI capabilities in 10 years. “Scaling laws have shown predictable improvements in AI model performance as they scale with model size, data, and computing power. The industry’s current trajectory scales computing power four fold annually, projecting a million fold increase over a decade. For comparison, Moore’s law achieved a hundred fold increase per decade,” Huang said.
“These scaling laws apply not only to LLM training, but with the advent of OpenAI Strawberry, also to inference. Over the next decade, we will accelerate our roadmap to keep pace with training and inference scaling demands, and to discover the next plateaus of intelligence,” he added.
This is a pretty extraordinary pronouncement. Huang is saying that AI computing power is growing eight times as fast as those for conventional computers, and its progress could be dramatically more rapid than what we were used to with conventional computers. The progress of conventional computing was itself extremely impressive — an ordinary smartphone that most people have today is orders of magnitude more powerful than the most powerful computers that had sent man to the moon in the 60s. Huang seems to be saying that AI’s pace of progress is eight times faster, which could make AI systems incredibly powerful just a few years down the line. It’s perhaps this progress which is making tech leaders predict that AI is a few years away, and companies are spending hundreds of billions of dollars into building their AI infrastructure. While there could be many bumps along the way, it certainly does appear that AI is moving at breathtaking pace, and the world would do well to fasten their seatbelts.