DeepSeek-R1's release made headlines this week, reaching far beyond the AI community into mainstream media. While much attention has focused on the geopolitical angle - a Chinese model matching frontier Western models - and the market reaction, including the largest single-day market cap loss in U.S. corporate history, there is a different aspect of the release I find the most interesting.
R1-Zero
The paper introduces two models: R1-Zero and its more polished version, R1, which you can talk to in DeepSeek chat. Let’s first see how R1-Zero was trained.
Essentially, they took a Deepseek-V3, a capable “$5.5 million“ base model, and trained it with reinforcement learning on problems with verifiable answers: math and programming.
The main training signal provided was whether the final response was correct. This is the simplest way to do this kind of training: there were no rewards for correct intermediate steps or simultaneously generating multiple parts of the answer and continuing with the best ones in the Monte-Carlo Tree Search style.
The model could also use <think> tags to enclose the text which was not included in the evaluation of the final response, except to check it did not mess up the formatting (hence "main" in the previous paragraph). This is the part referred to as “reasoning”.
This was enough to make the model really good at math and programming.
Moreover, it learnt to reason for longer and longer, even though there was no reward for this specifically. But when the model used more tokens in the scratchpad before giving the final answer ("thinks for longer"), the answer was more likely to be correct, reinforcing this behaviour.
This paradigm is not new: the first reasoning frontier model was OpenAI's o1, with the second generation o3 expected to be released soon. There were several theories on how the o1 worked, but R1-Zero shows that the approach that worked may be the simplest one.
(R1)
The R1-Zero is very good at giving you the answer in math/programming, but it has some rough edges.
For example, R1-Zero sometimes mixed languages in the reasoning process -- the only thing it cared for was the final answer. So, they added a penalty for using words from different languages in the reasoning process.
They fine-tuned the model with more data: the usual helpful/harmless mix, but also tasks with no easily verifiable answer where the original V3 decides if the answer is correct.
One more key difference between the R1 and o1 models is that with o1, you can only see a summary of the reasoning, while R1 shows you the exact text generated in the scratchpad.
The Bitter Lesson
There is a phenomenon formulated by Richard Sutton known as the Bitter Lesson. AI researchers have a strong tendency to encode human knowledge and patterns into AI systems. Yet, time and time again, methods that are able to scale well with more computation end up outperforming them while often being much simpler. Take AlphaZero as one example: it knows nothing about Chess or Go except the rules of the games, yet it's superhuman in both.
"We want AI agents that can discover like we can, not which contain what we have discovered. Building in our discoveries only makes it harder to see how the discovering process can be done." (R. Sutton)
The backbone of large language models is arguably the simplest imaginable objective: predict the next token accurately. Now, with R1-Zero, we again see the plain training goal: get the final answer right. Use as many reasoning tokens as you want in any language you like. And it works.