generated by grok 2
introduction
deepseek has shown r1 to the world and made everyone talk about it. it changed the assumptions about the cost of the knowledge set by openaI, making it cheaper and more accessible (about 96% less costly than o1 api costs). while the world was taken by this model, one of their other experiments went under the radar — deepseek’s r1 zero. while the experiment was not exactly successful, it marks a decided turn on how we approach model training and model scaling.
the problem with the current setup
traditionally, large language models (llms) rely heavily on supervised fine-tuning (sft) (a training process, which enables the model to follow chat-instruction format and make the model mimic certain responses). sft is done with human-labeled data or with synthetic data (training data generated from another llm — distilled data from larger models to train smaller models)
inorder to get the sft to work, the models need a huge amount of data samples — millions of training data pairs (prompt & response). importantly, all the data should be of high quality (whatever it would be — human-labelled data or synthetic data). generating these datasets is expensive and should be carefully curated.
another important thing with sft is that the model will never surpass the model or the human who created the training data. because it just tries to imitate the training data, it can only get closer to the level of knowledge present in the data but not above that (just never).
another scaling technique would be solving problems step by step, in recent studies, the llms are more robust with the chain-of-thought (cot) fine-tuning for solving complex problems. while generating a solution the model undergoes several steps like breaking down the problem, reasoning on how to solve and reflextions on its steps. this seems to be working — but not enough (again this is not scaled enough because it's trained with sft).
the solution?
this problem statement is not new, this is been solved 10 years before in the era of alphago zero. they beat the world’s best champion on go — lee se-dol, by letting the model self-play (in the training phase) rather than imitate the player's data (player moves and decisions).
alphago zero vs lee se-dol
in alphago zero, deepmind took a revolutionary step by discarding the need for human expert data entirely. by relying solely on self-play, the model was able to explore millions of possible moves, gradually uncovering strategies that even human experts had not considered. this self-directed learning process not only pushed the boundaries of what was achievable in the game of go but also served as a powerful demonstration of how reinforcement learning can transform raw memorization into true strategic reasoning
applying the same magic
the solution deepmind found not only sticks with board games, atari games (arcade learning environment**)** or 3d games, but also to llms. now the llms play a very important role in most of the nlp (and vision) tasks. people have a vision of the future for these models that these models could help humanity in many other ways and in different fields i.e., education, computer science, healthcare etc. but “sft only” approaches can’t help with those generalizations.
you might have guessed what we can use instead of sft (i shouldn’t say instead of, but i have clarified on my stance in my final thoughts below) — yes, it is reinforcement learning (rl). rl is a type of machine learning where an agent (model) learns to make decisions by interacting with an environment. the primary objective is for the agent to develop a policy — a strategy for choosing actions (ie., response for a user prompt) that maximizes the total accumulated reward (assume that reward is a scalar value given to the model response based on the helpfulness) over time.
specifically for language models, the rl helps the model to find the path to the solution (rather than mug up like in sft). under the process of rl, the model undergoes exploration & change in the way of solving the problem.
just to recall, we have seen the two components — sft & cot earlier in the section. both are methods — sft is a method to train & cot is an inference method to scale the problem-solving ability in llms. introducing rl can help with what sft is missing out on and take the cot to the next level.
rl is the magic that applies to the llms to fix these problems — with sft & utilizing cot effectively. rl generally helps the models to find the optimal path by itself by exploration & rewarding it if its going in a right path.
visualizing the training pipeline for deepSeek-r1(-zero) by @sirrahchan.
assume that the model is being trained on grad-level math data which contains problems and final answers. the model undergoes various steps to solve each problem — this is what we call reasoning. it naturally learns how humans solve problems through reasoning and having some actual time to think before answering (unlike the previous models i.e., gpt-4o, claude). this way the model generalizes and learns really how to approach a problem.
a small clarification here, you may know even some models i.e, gpt-4o, llama 3, claude are trained with sft and also done with some alignment using rl, but still, those models are assumed to be sft models because 70% of the post-training is with sft and only remaining 30% with rl.
because the reasoning is scaling up, the models tend to think more time — thus resulting in scaling compute in inference time too (prediction time). this is called test-time computing. letting the model think (after reasoning training with rl) for a while is better than training a model for a while with sft in terms of the cognitive ability of llms — also rl training takes less time than sft.
clarification & final thoughts
don’t get me wrong — sft is still needed. it sets up the baseline for the model’s “cold start” during training. but reducing reliance on sft-only training can shift more computational burden from training to inference, and also make the model more generalized.
these days, powerful models like o1, r1, o3, etc., are just playing around with natural language (text) and vision (images), yet they’re already achieving human-level performance in various fields like coding and even phd-level science. imagine if these models had more degrees of freedom in their environment, like embodied agents with an actual “thinking” mind. that’s when we’ll see the next generation of AI transform the world.
rl brings us one step closer to that future, a future where AI transcends imitation and becomes a true partner in problem-solving and discovery.
