Ever since models like ChatGPT burst into prominence a couple of years ago, some people have been putting much in store by what the “AI” says. “Ask an AI” is thought to be akin to asking a question to the grand supercomputer in the Hitchhiker’s Guide to the Galaxy, and people expect that it’ll synthesize all the known knowledge in the universe to generate a wise, all-knowing answer. But a prominent voice in AI has explained why that’s currently not quite the case.
Andrej Karpathy, who has worked at OpenAI and was the Director of AI at Tesla, has explained why “ask an AI” isn’t what many people think it is. “People have too inflated sense of what it means to “ask an AI” about something,” he posted on X. “The AI are language models trained basically by imitation on data from human labelers. Instead of the mysticism of “asking an AI”, think of it more as “asking the average data labeler” on the internet,” he added.
“Few caveats apply because in many domains (e.g. code, math, creative writing) the companies hire skilled data labelers (so think of it as asking them instead), and this is not 100% true when reinforcement learning is involved. But roughly speaking (and today), you’re not asking some magical AI. You’re asking a human data labeler, whose average essence was lossily distilled into statistical token tumblers that are LLMs,” Karpathy said.
Karpathy said he wanted to clarify this because some people were suggesting that we should “ask AI” about critical things such as how to run governments. “You’re not asking an AI, you’re asking some mashup spirit of its average data labeler,” he explained.
“Example when you ask something like “top 10 sights in Amsterdam”, some hired data labeler probably saw a similar question at some point, researched it for 20 minutes using Google and Trip Advisor or something, and came up with some list of 10, which literally then becomes the correct answer, (and trains) the AI to give that answer for that question. If the exact place in question is not in the finetuning training set, the neural net imputes a list of statistically similar vibes based on its knowledge gained from the pretraining stage (language modeling of internet documents),” Karpathy explained.
Modern LLMs are trained on vast amounts of data, but the final step towards creating the model relies on humans giving the model feedback on which of their outputs are better. This is called Reinforcement Learning with Human Feedback (RLHF), and relies on contractors who usually see two model outputs, and simply rate which one they prefer. This causes the model to be influenced greatly by the opinions of these human evaluators. As such, as Andrej Karpathy explains, asking an AI isn’t as grandiose as it seems — its outputs are heavily influenced by human contractors who’ve been asked to manually train it.