AI Labs Warn of Existential Threats: From Lab-Made Viruses to Self-Improving Cod

Summary

Leading AI labs are issuing a stark warning about the potential for advanced AI models to pose an existential threat. Sam Altman, Dario Amodei, and Demis Hassabis, among others, have signed a letter to Congress urging mandatory screening of orders for synthetic nucleic acids, the building blocks of viruses. They highlight that AI can now outperform PhD-level virologists, potentially enabling the creation of dangerous pathogens. Meanwhile, Anthropic is collaborating with the NSA on cybersecurity, though it's unclear if this extends to offensive operations. The source also delves into the accelerating pace of AI development, citing an anonymized researcher's experience of AI surpassing human capabilities, leading to a sense of obsolescence. Anthropic's research suggests AI agents are becoming increasingly productive, performing complex tasks and even conducting research autonomously, with predictions of AI outperforming human researchers within a year. The discussion then outlines three potential futures: an S-curve plateauing AI progress, a continued exponential gain where human experts remain crucial bottlenecks, and a scenario of true recursive self-improvement leading to unpredictable, potentially uncontrollable AI. The core concern remains AI alignment – ensuring advanced AI's goals match humanity's. To navigate these risks, a globally verifiable system for pausing or slowing AI development is proposed, akin to a game with agreed-upon rules and a referee, to prevent a unilateral advantage for unethical actors. The piece concludes by lamenting the disconnect in public perception, with some still dismissing AI's capabilities entirely, while acknowledging the increasing reality of AI impacting daily life.