It had been widely believed that there were no longer any significant gains to be made in pre-training for frontier models, but Google Gemini 3’s creators are crediting the model’s impressive performance to improvements made in pre-training as well.
Oriol Vinyals, who is the Gemini co-lead and VP of Research and Deep Learning Lead at Google DeepMind, has said that contrary to popular belief, pre-training hasn’t hit a wall, and improvements in pre-training helped them create the Gemini 3 model. “The secret behind Gemini 3? Simple: Improving pre-training & post-training,” he posted on X.
“Pre-training: Contra the popular belief that scaling is over—which we discussed in our NeurIPS ’25 talk with Ilya Sutskever and Quoc Le—the team delivered a drastic jump. The delta between 2.5 and 3.0 is as big as we’ve ever seen. No walls in sight!” he added.
“Post-training: Still a total greenfield. There’s lots of room for algorithmic progress and improvement, and 3.0 hasn’t been an exception, thanks to our stellar team,” he continued.
It had been believed that pre-training — which involves feeding the model massive amounts of data, which includes most of the internet and many published books — had already been pushed to its limits, and would lead to no further performance improvements. Most frontier models had already scraped the open internet, and there simply wasn’t any new data to feed these models. This had been alluded to by former OpenAI Chief Scientist Ilya Sutskever, who’d said in December last year that data was the fossil fuel of AI, and it would get exhausted. Elon Musk had said in January this year that we’d already run out of human data to train AI models.
But Google now says that it has made some breakthroughs in pre-training which have helped Gemini 3’s performance, and pushed it to the top of nearly all benchmarks. It’s possible that Google had access to some data that not on the open web that it’s now using — Google’s been around for 25 years, and likely has managed to collect plenty of data that other companies don’t have access to. It’s also possible that Google has found some new techniques to process this data that are leading to better results. But for Google to come and say that are still improvements left in pre-training would be bullish for LLMs, and the overall AI space in general — it shows that there are still two available axes, pre-training and post-training, that companies can use to improve their models.