OurCoders (我们程序员)

2025-06-26 15:21:12
AI Gets Better at Getting Better at Writing Code Recent research reveals an advanced system called Darwin Gödel Machines (DGMs) that uses large language models and evolutionary algorithms to recursively improve coding agents, achieving significant performance gains on programming benchmarks. This development suggests potential for大幅提升生产力，但也引发了对AI自我改进安全性的担忧。研究者采取了防护措施来确保系统的可解释性和与人类指令的一致性，但关于这种技术的长期影响和潜在风险仍存在争议。 In April, Microsoftâs CEO said that artificial intelligence now wrote close to a third of the companyâs code . Last October, Googleâs CEO put their number at around a quarter . Other tech companies canât be far off. Meanwhile, these firms create AI that will presumably be used to help programmers further. Researchers have long hoped to fully close the loop, creating coding agents that recursively improve themselves. New research reveals an impressive demonstration of such a system. Extrapolating, one might see a boon to productivity, or a much darker future for humanity. âItâs nice work,â said JÃ¼rgen Schmidhuber , a computer scientist at the King Abdullah University of Science and Technology ( KAUST ), in Saudi Arabia , who was not involved in the new research. âI think for many people, the results are surprising. Since Iâve been working on that topic for almost 40 years now, itâs maybe a little bit less surprising to me.â But his work over that time was limited by the tech at hand. One new development is the availability of large language models (LLMs), the engines powering chatbots like ChatGPT . In the 1980s and 1990s, Schmidhuber and others explored evolutionary algorithms for improving coding agents, creating programs that write programs. An evolutionary algorithm takes something (such as a program), creates variations, keeps the best ones, and iterates on those. But evolution is unpredictable. Modifications donât always improve performance. So in 2003, Schmidhuber created problem solvers that rewrote their own code only if they could formally prove the updates to be useful. He called them GÃ¶del machines , named after Kurt GÃ¶del , a mathematician whoâd done work on self-referencing systems. But for complex agents, provable utility doesnât come easily. Empirical evidence may have to suffice. The Value of Open-Ended Exploration The new systems, described in a recent preprint on arXiv, rely on such evidence. In a nod to Schmidhuber, theyâre called Darwin GÃ¶del Machines (DGMs). A DGM starts with a coding agent that can read, write, and execute code, leveraging an LLM for the reading and writing. Then it applies an evolutionary algorithm to create many new agents. In each iteration, the DGM picks one agent from the population and instructs the LLM to create one change to improve the agentâs coding ability. LLMs have something like intuition about what might help, because theyâre trained on lots of human code. What results is guided evolution, somewhere between random mutation and provably useful enhancement. The DGM then tests the new agent on a coding benchmark, scoring its ability to solve programming challenges. Some evolutionary algorithms keep only the best performers in the population, on the assumption that progress moves endlessly forward. DGMs, however, keep them all, in case an innovation that initially fails actually holds the key to a later breakthrough when further tweaked. Itâs a form of â open-ended exploration ,â not closing any paths to progress. (DGMs do prioritize higher scorers when selecting progenitors.) The researchers ran a DGM for 80 iterations using a coding benchmark called SWE-bench , and ran one for 80 iterations using a benchmark called Polyglot . Agentsâ scores improved on SWE-bench from 20 percent to 50 percent, and on Polyglot from 14 percent to 31 percent. âWe were actually really surprised that the coding agent could write such complicated code by itself,â said Jenny Zhang , a computer scientist at the University of British Columbia and the paperâs lead author. âIt could edit multiple files, create new files, and create really complicated systems.â The first coding agent (numbered 0) created a generation of new and slightly different coding agents, some of which were selected to create new versions of themselves. The agentsâ performance is indicated by the color inside the circles, and the best performing agent is marked with a star. Jenny Zhang, Shengran Hu, et al. Critically, the DGMs outperformed an alternate method that used a fixed external system for improving agents. With DGMs, agentsâ improvements compounded as they improved themselves at improving themselves. The DGMs also outperformed a version that didnât maintain a population of agents and just modified the latest agent. To illustrate the benefit of open-endedness, the researchers created a family tree of the SWE-bench agents. If you look at the best-performing agent and trace its evolution from beginning to end, it made two changes that temporarily reduced performance. So the lineage followed an indirect path to success. Bad ideas can become good ones. The black line on this graph shows the scores obtained by agents within the lineage of the final best-performing agent. The line includes two performance dips. Jenny Zhang, Shengran Hu, et al. The best SWE-bench agent was not as good as the best agent designed by expert humans, which currently scores about 70 percent, but it was generated automatically, and maybe with enough time and computation an agent could evolve beyond human expertise. The study is a âbig step forwardâ as a proof of concept for recursive self-improvement, said Zhengyao Jiang , a cofounder of Weco AI , a platform that automates code improvement. Jiang, who was not involved in the study, said the approach could made further progress if it modified the underlying LLM, or even the chip architecture. ( Google DeepMindâs AlphaEvolve designs better basic algorithms and chips and found a way to accelerate the training of its underlying LLM by 1 percent.) DGMs can theoretically score agents simultaneously on coding benchmarks and also specific applications, such as drug design, so theyâd get better at getting better at designing drugs. Zhang said sheâd like to combine a DGM with AlphaEvolve. Could DGMs reduce employment for entry-level programmers? Jiang sees a bigger threat from everyday coding assistants like Cursor. âEvolutionary search is really about building really high-performance software that goes beyond the human expert,â he said, as AlphaEvolve has done on certain tasks. The Risks of Recursive Self-improvement One concern with both evolutionary search and self-improving systemsâand especially their combination, as in DGMâis safety. Agents might become uninterpretable or misaligned with human directives. So Zhang and her collaborators added guardrails. They kept the DGMs in sandboxes without access to the Internet or an operating system , and they logged and reviewed all code changes. They suggest that in the future, they could even reward AI for making itself more interpretable and aligned. (In the study, they found that agents falsely reported using certain tools, so they created a DGM that rewarded agents for not making things up, partially alleviating the problem. One agent, however, hacked the method that tracked whether it was making things up.) In 2017, experts met in Asilomar, Calif., to discuss beneficial AI, and many signed an open letter called the Asilomar AI Principles . In part, it called for restrictions on âAI systems designed to recursively self-improve.â One frequently imagined outcome is the so-called singularity , in which AIs self-improve beyond our control and threaten human civilization. âI didnât sign that because it was the bread and butter that Iâve been working on,â Schmidhuber told me. Since the 1970s, heâs predicted that superhuman AI will come in time for him to retire, but he sees the singularity as the kind of science-fiction dystopia people love to fear. Jiang, likewise, isnât concerned, at least for the time being. He still places a premium on human creativity. Whether digital evolution defeats biological evolution is up for grabs. Whatâs uncontested is that evolution in any guise has surprises in store.

全部新闻