{"id":48,"date":"2025-05-04T14:16:03","date_gmt":"2025-05-04T06:16:03","guid":{"rendered":"http:\/\/120.55.12.129\/?p=48"},"modified":"2025-05-05T16:22:13","modified_gmt":"2025-05-05T08:22:13","slug":"%e4%ba%ba%e5%b7%a5%e6%99%ba%e8%83%bd%e7%9a%84%e4%b8%ad%e5%9c%ba%e9%98%b6%e6%ae%b5%ef%bc%9a%e4%bb%8e%e8%a7%a3%e5%86%b3%e9%97%ae%e9%a2%98%e5%88%b0%e5%ae%9a%e4%b9%89%e9%97%ae%e9%a2%98%ef%bc%88we","status":"publish","type":"post","link":"https:\/\/www.agidt.com\/?p=48","title":{"rendered":"\u4eba\u5de5\u667a\u80fd\u7684\u4e2d\u573a\u9636\u6bb5\uff1a\u4ece\u89e3\u51b3\u95ee\u9898\u5230\u5b9a\u4e49\u95ee\u9898\uff08We\u2019re at AI\u2019s halftime.\uff09"},"content":{"rendered":"\n
\"\"<\/figure>\n\n\n\n

tldr: We\u2019re at AI\u2019s halftime.<\/p>\n\n\n\n

For decades, AI has largely been about developing new training methods and models. And it worked: from beating world champions at chess and Go, surpassing most humans on the SAT and bar exams, to earning IMO and IOI gold medals. Behind these milestones in the history book \u2014 DeepBlue, AlphaGo, GPT-4, and the o-series \u2014 are fundamental innovations in AI methods: search, deep RL, scaling, and reasoning. Things just get better over time.<\/p>\n\n\n\n

So what\u2019s suddenly different now?<\/p>\n\n\n\n

In three words: RL finally works. More precisely: RL finally generalizes. After several major detours and a culmination of milestones, we\u2019ve landed on a working recipe to solve a wide range of RL tasks using language and reasoning. Even a year ago, if you told most AI researchers that a single recipe could tackle software engineering, creative writing, IMO-level math, mouse-and-keyboard manipulation, and long-form question answering \u2014 they\u2019d laugh at your hallucinations. Each of these tasks is incredibly difficult and many researchers spend their entire PhDs focused on just one narrow slice.<\/p>\n\n\n\n

Yet it happened.<\/p>\n\n\n\n

So what comes next? The second half of AI \u2014 starting now \u2014 will shift focus from solving problems to defining problems. In this new era, evaluation becomes more important than training. Instead of just asking, \u201cCan we train a model to solve X?\u201d, we\u2019re asking, \u201cWhat should we be training AI to do, and how do we measure real progress?\u201d To thrive in this second half, we\u2019ll need a timely shift in mindset and skill set, ones perhaps closer to a product manager.<\/p>\n\n\n\n

The first half<\/h2>\n\n\n\n

To make sense of the first half, look at its winners. What do you consider to be the most impactful AI papers so far?<\/p>\n\n\n\n

I tried the quiz in Stanford 224N, and the answers were not surprising: Transformer, AlexNet, GPT-3, etc. What\u2019s common about these papers? They propose some fundamental breakthroughs to train better models. But also, they managed to publish their papers by showing some (significant) improvements on some benchmarks.<\/p>\n\n\n\n

There is a latent commonality though: these \u201cwinners\u201d are all training methods or models, not benchmarks or tasks. Even arguably the most impactful benchmark of all, ImageNet, has less than one third of the citation of AlexNet. The contrast of method vs benchmark is even more drastic anywhere else \u2014- for example, the main benchmark of Transformer is WMT\u201914, whose workshop report has ~1,300 citations, while Transformer had >160,000.<\/p>\n\n\n\n

\"\"\/<\/figure>\n\n\n\n

That illustrates the game of the first half: focus on building new models and methods, and evaluation and benchmark are secondary (although necessary to make the paper system work).<\/p>\n\n\n\n

Why? A big reason is that, in the first half of AI, methods were harder and more exciting than tasks. Creating a new algorithm or model architecture from scratch \u2013 think of breakthroughs like the backpropagation algorithm, convolutional networks (AlexNet), or the Transformer used in GPT-3 \u2013 required remarkable insight and engineering. In contrast, defining tasks for AI often felt more straightforward: we simply took tasks humans already do (like translation, image recognition, or chess) and turned them into benchmarks. Not much insight or even engineering.<\/p>\n\n\n\n

Methods also tended to be more general and widely applicable than individual tasks, making them especially valuable. For example, the Transformer architecture ended up powering progress in CV, NLP, RL, and many other domains \u2013 far beyond the single dataset (WMT\u201914 translation) where it first proved itself. A great new method can hillclimb many different benchmarks because it\u2019s simple and general, thus the impact tends to go beyond an individual task.<\/p>\n\n\n\n

This game has worked for decades and sparked world-changing ideas and breakthroughs, which manifested themselves by ever-increasing benchmark performances in various domains. Why would the game change at all? Because the cumulation of these ideas and breakthroughs have made a qualitative difference in creating a working recipe in solving tasks.<\/p>\n\n\n\n

The recipe<\/h2>\n\n\n\n

What\u2019s the recipe? Its ingredients, not surprisingly, include massive language pre-training, scale (in data and compute), and the idea of reasoning and acting. These might sound like buzzwords that you hear daily in SF, but why call them a recipe??<\/p>\n\n\n\n

We can understand this by looking through the lens of reinforcement learning (RL), which is often thought of as the \u201cend game\u201d of AI \u2014 after all, RL is theoretically guaranteed to win games, and empirically it\u2019s hard to imagine any superhuman systems (e.g. AlphaGo) without RL.<\/p>\n\n\n\n

In RL, there are three key components: algorithm, environment, and priors<\/strong>. For a long time, RL researchers focused mostly on the algorithm (e.g. REINFORCE, DQN, TD-learning, actor-critic, PPO, TRPO\u2026) \u2013 the intellectual core of how an agent learns \u2013 while treating the environment and priors as fixed or minimal. For example, Sutton and Barto\u2019s classical textbook is all about algorithms and almost nothing about environments or priors.<\/p>\n\n\n\n

\"\"\/<\/figure>\n\n\n\n

However, in the era of deep RL, it became clear that environments matter a lot empirically: an algorithm\u2019s performance is often highly specific to the environment it was developed and tested in. If you ignore the environment, you risk building an \u201coptimal\u201d algorithm that only excels in toy settings. So why don\u2019t we first figure out the environment we actually want to solve, then find the algorithm best suited for it?<\/p>\n\n\n\n

That\u2019s exactly OpenAI\u2019s initial plan. It built gym<\/a>, a standard RL environment for various games, then the World of Bits and Universe projects<\/a>, trying to turn the Internet or computer into a game. A good plan, isn\u2019t it? Once we turn all digital worlds into an environment, solve it with smart RL algorithms, we have digital AGI.<\/p>\n\n\n\n

A good plan, but not entirely working. OpenAI made tremendous progress down the path, using RL to solve Dota<\/a>, robotic hands<\/a>, etc. But it never came close to solving computer use or web navigation, and the RL agents working in one domain do not transfer to another. Something is missing.<\/p>\n\n\n\n

Only after GPT-2 or GPT-3, it turned out that the missing piece is priors. You need powerful language pre-training to distill general commonsense and language knowledge into models, which then can be fine-tuned to become web (WebGPT) or chat (ChatGPT) agents (and change the world). It turned out the most important part of RL might not even be the RL algorithm or environment, but the priors, which can be obtained in a way totally unrelated from RL.<\/strong><\/p>\n\n\n\n

Language pre-training created good priors for chatting, but not equally good for controlling computers or playing video games. Why? These domains are further from the distribution of Internet text, and naively doing SFT \/ RL on these domains generalizes poorly. I noticed the problem in 2019, when GPT-2 just came out and I did SFT \/ RL on top of it to solve text-based games – CALM<\/a> was the first agent in the world built via pre-trained language models. But it took millions of RL steps for the agent to hillclimb a single game, and it doesn\u2019t transfer to new games. Though that\u2019s exactly the characteristic of RL and nothing strange to RL researchers, I found it weird because we humans can easily play a new game and be significantly better zero-shot. Then I hit one of the first eureka moment in my life – we generalize because we can choose to do more than \u201cgo to cabinet 2\u201d or \u201copen chest 3 with key 1\u201d or \u201ckill dungeon with sword\u201d, we can also choose to think about things like \u201cThe dungeon is dangerous and I need a weapon to fight with it. There is no visible weapon so maybe I need to find one in locked boxes or chests. Chest 3 is in Cabinet 2, let me first go there and unlock it\u201d.<\/p>\n\n\n\n

\"\"\/<\/figure>\n\n\n\n

Thinking, or reasoning, is a strange<\/strong> kind of action – it does not directly affect the external world, yet the space of reasoning is open-ended and combintocially infinite \u2014 you can think about a word, a sentence, a whole passage, or 10000 random English words, but the world around you doesn\u2019t immediate change. In the classical RL theory, it is a terrible deal and makes decision-making impossible. Imagine you need to choose one out of two boxes, and there\u2019s only one box with $1M and the other one empty. You\u2019re expected to earn $500k. Now imagine I add infinite empty boxes. You\u2019re expected to earn nothing. But by adding reasoning into the action space of any RL environment, we make use of the language pre-training priors to generalize, and we afford to have flexible test-time compute for different decisions. It is a really magical<\/strong> thing and I apologize for not fully making sense of it here, I might need to write another blog post just for it. You\u2019re welcome to read ReAct<\/a> for the original story of reasoning for agents and read my vibes at the time. For now, my intuitive explanation is: even though you add infinite empty boxes, you have seen them throughout your life in all kinds of games, and choosing these boxes prepare you to better choose the box with money for any given game. My abstract explanation would be: language generalizes through reasoning in agents<\/strong>.<\/p>\n\n\n\n

Once we have the right RL priors (language pre-training) and RL environment (adding language reasoning as actions), it turns out RL algorithm might be the most trivial part. Thus we have o-series, R1, deep research, computer-using agent, and so much more to come. What a sarcastic turn of events! For so long RL researchers cared about algorithms way more than environments, and no one paid any attention to priors \u2014 all RL experiments essentially start from scratch. But it took us decades of detours to realize maybe our prioritization should have be completely reversed.<\/p>\n\n\n\n

But just like Steve Jobs said: You can\u2019t connect the dots looking forward; you can only connect them looking backward.<\/p>\n\n\n\n

The second half<\/h2>\n\n\n\n

This recipe is completely changing the game. To recap the game of the first half:<\/p>\n\n\n\n