The AI community hangs on to everything that Andrej Karpathy says, and that’s in part because few people have been so prescient about AI for so long.
A tweet of his from August 5, 2017 has been doing the rounds again, and reading it nine years later feels almost unsettling. The man who would go on to co-found OpenAI, run AI at Tesla, and now work on pretraining at Anthropic, posted a single line: “Gradient descent can write code better than you. I’m sorry.” No thread, no elaboration, just what seemed like a joke aimed at an audience that mostly took it as one.
A few months later, in November 2017, he expanded the idea into a full essay titled Software 2.0, arguing that neural networks weren’t simply another tool sitting alongside traditional programming, but the beginning of an entirely different way of building software. The tweet was the punchline. The essay was the argument.
What gradient descent actually is
Gradient descent is the optimization method that trains most modern machine learning systems, and the concept is far less intimidating than it sounds. Picture a neural network as an enormous mathematical function with millions or billions of tunable parameters, and picture the network’s mistakes on a training dataset as a landscape with hills and valleys. The lowest points in that landscape represent the fewest errors. Gradient descent is simply the process of repeatedly checking which direction leads downhill and taking a small step that way, over and over, until the network settles somewhere near the bottom.
The programmer doesn’t write out the rules the network follows. Instead, the programmer defines a goal, sets up the architecture, feeds it data, and lets this iterative search find the weights that produce the desired behavior. Karpathy’s framing was that this search process was, in effect, doing the job a human engineer used to do by hand — writing the function. Datasets became the new source code, the neural network architecture became the rough skeleton, and gradient descent became the compiler that filled in the details no human had explicitly written.
Where the field actually stood in 2017
It’s worth remembering how modest AI’s coding ability was at the time. Deep learning had proven itself in narrow domains — AlexNet had shown in 2012 that convolutional networks could crush image classification benchmarks, and by 2017 the field had moved on to sequence models, recurrent networks, and the first stirrings of what would become the Transformer architecture, published by Google that same year. Karpathy himself was deep in computer vision work at Tesla, building perception systems for Autopilot. There was no ChatGPT, no Copilot, no agentic coding tool of any kind. Writing code with the help of a neural network wasn’t a category that existed for most engineers. Karpathy’s tweet and his subsequent essay were describing a shift in how software would eventually get made, not something anyone could point to in production yet.
That’s what makes the prediction remarkable in hindsight. He wasn’t reacting to an existing wave of AI coding tools, because none existed. He was reasoning from first principles about what optimization over large datasets would eventually be capable of, years before large language models made that reasoning obvious to everyone else.
Where things stand now
The gap between 2017 and 2026 is difficult to overstate. On SWE-bench Verified, the benchmark built from real GitHub issues that require an AI system to actually understand a codebase and ship a working patch, frontier models now clear the high 80s in percentage terms, a level that would have sounded like fiction in Karpathy’s Tesla days. Anthropic’s Claude Fable 5, released this year, reportedly completed a 50-million-line Ruby migration for a customer in a single day, a job Stripe had separately estimated would take a human engineering team over two months. Multiple Claude Opus agents working in parallel have been used to write a C compiler from scratch, and OpenAI beat all human participants at the AtCoder coding competition this year. Karpathy’s own open-source project, autoresearch, lets AI agents run machine learning experiments overnight without a person in the loop, closing a version of the exact feedback cycle his Software 2.0 essay gestured toward.
Karpathy has kept shaping the vocabulary the industry uses to talk about all this. He’s the one who coined the term vibe coding in February 2025, describing a style of building software where you describe what you want and stop reading the diffs. What started as a weekend habit of his has become how a meaningful share of software gets written today, from indie developers to CEOs of large companies who say they haven’t personally written a line of code in over a year. He’s also been vocal that the biggest gains now come from removing humans as the approval bottleneck in AI coding workflows altogether, a natural extension of the same argument he was making back in 2017: that the machine, left to its own optimization process, ends up writing the better program.
It’s not as though coding is solved. Harder, less contaminated benchmarks like SWE-bench Pro still show scores well below what Verified suggests, and plenty of real-world engineering work continues to need serious human oversight. But the trajectory Karpathy sketched with a joke and a follow-up essay has held up in a way that few technology predictions from 2017 can claim. He wasn’t guessing that AI would get better at writing code. He was pointing at the mechanism — gradient descent, searching a space of programs no human wrote by hand — and betting that the mechanism would eventually outrun the programmer holding the keyboard.