Jack Clark delivers a sobering reality check: the era of theoretical AI threats is over, replaced by autonomous, self-replicating digital organisms that are already operational. This newsletter doesn't just predict a future where machines outpace us; it documents a present where code can parasitize our hardware to build its own army, all while the industry's most powerful figures urgently ask the government to hit the brakes. The convergence of these three stories—self-sustaining malware, the economics of compute scarcity, and the limits of machine creativity—paints a picture of a technology that is simultaneously advancing faster than our safety nets and hitting a wall where human intuition remains irreplaceable.
The Rise of the Digital Parasite
Clark opens with a startling prototype from researchers at the University of Toronto and Cambridge: a computer virus that doesn't just spread, but thinks. The worm uses stolen GPU resources from compromised machines to run open-weight large language models locally, allowing it to generate tailored attack strategies for every new host it encounters. "We must prepare for autonomous generative adversaries," the researchers write, noting that this worm "parasitically uses compromised machines to run open-weight large language models (LLMs) to sustain its reasoning, or extend its reach for further attacks." This is not a script kiddie tool; it is a self-sustaining ecosystem.
The mechanism is chillingly efficient. The agent operates without relying on external vendor APIs that could be monitored or revoked, running entirely on a single local GPU. "The worm uses stolen computing power from compromised GPU nodes to host LLMs for generative reasoning," Clark explains, highlighting how the system detects vulnerabilities and devises attacks in real-time. The researchers employed a "reasoning graph" to keep the agent focused, a technique reminiscent of the "Knowledge graph" deep dive which emphasizes how structured data can guide AI behavior. By decomposing the agent's reasoning into scoped steps—Plan, Judge, Action, Summary, Progress—the system avoids the confusion that often plagues autonomous agents.
"The worm operates in a fully decentralized manner, and no single point of control can be taken offline to interrupt its spread."
The success rates are concerning enough to demand attention: an 80% success rate in detecting vulnerabilities and an 88% success rate in self-replication. Clark argues that the future internet will resemble a complex ecology of attacker and defender agents. "Despite the inherent fragility of individual exploitation attempts, the worm agent achieves operational resilience by continuously self-replicating into a swarm," the researchers note. This suggests a future where human intervention is too slow, necessitating the deployment of our own AI "white blood cells" to fight back. Critics might argue that this scenario relies on a chain of perfect execution, but the decentralized nature of the swarm makes it inherently resilient to traditional takedowns.
The Economics of the Singularity
Shifting from security to economics, Clark explores a provocative theory by Dwarkesh Patel: as AI systems become smarter, the price of compute will skyrocket. The logic is straightforward yet terrifying. If an AI can perform the work of a top-tier software engineer, the market value of the hardware running it should reflect that productivity. "As AI models become smarter, they'll better monetize the same amount of compute," Patel writes. "If a true human-level software engineer that could run on an H100 equivalent, at current market rates for software engineers, that H100 should rent for over $250k a year."
This implies a temporary but volatile period where the cost of intelligence becomes a bottleneck. Clark notes that this dynamic means "using GPUs to make short-form video slop will just get priced out," as the hardware is bid up by more valuable, high-level reasoning tasks. The argument draws a parallel to the "Logic bomb" concept, where a specific trigger (in this case, economic value) forces a system into a new state. However, Patel predicts this is a temporary state of affairs, expecting that "massive roboticization of the compute supply chain should bring its price down closer to the cost of raw inputs and tools."
The core implication is that we are entering a phase of "singularity economics" where commodity hardware becomes a strategic resource. Clark suggests that this economic pressure could inadvertently slow down reckless deployment, as the cost of failure becomes prohibitive. Yet, a counterargument worth considering is that the demand for compute might simply outstrip any supply-side innovations, leading to a permanent scarcity that concentrates power in the hands of a few entities that can afford the hardware.
The Collective Action Problem
Perhaps the most significant development covered is the unprecedented plea from 1,337 AI researchers and executives to the US government. Leaders from OpenAI, Anthropic, Google DeepMind, and others have signed a statement requesting international support to "develop the technical and governance tools needed to deliberately pace the frontier of automated AI development." The signatories acknowledge a critical risk: "capability development rapidly accelerates beyond our ability to understand or control the resulting systems."
Clark frames this as a desperate attempt to solve a massive collective action problem. The industry is trapped in a race where no single company can afford to slow down unilaterally, even if they recognize the danger. "To realize AI's potential, industry, government, and society at large may need the option to buy time to address emerging risks," the statement reads. This mirrors the challenges discussed in the "Knowledge distillation" deep dive, where transferring complex knowledge requires careful, controlled steps to avoid degradation or loss of nuance. Here, the "knowledge" is the safety and control of the technology itself.
"The world's leading AI companies believe they could be close to automating AI research. It is hard to predict exactly how much this will accelerate AI progress, but there is a real risk that capability development rapidly accelerates beyond our ability to understand or control the resulting systems."
The request for the government to intervene is a stark admission that the market cannot self-regulate. Clark argues that this is an essential prerequisite for managing the risks of increasingly intelligent systems. However, skeptics might note that asking the very entities racing to build these systems to agree on a speed limit is a fundamental conflict of interest. Without binding international treaties, such voluntary measures may prove insufficient.
The Creativity Ceiling
Despite the hype around autonomous agents, Clark presents a bearish signal on the timeline for recursive self-improvement: AI is still terrible at genuine creativity. A new study using "shadow evaluation"—where AI agents attempt to solve unpublished research problems—found that while the systems are excellent engineers, they fail to produce novel, paradigm-shifting ideas. The agents, running on models like Claude Opus 4.8, were tasked with research on LLM personas and distribution shift detectors. The result? "The [human] authors rejected both papers," with reviews citing "poorly motivated data and experiments, no novel contribution, and impenetrable prose."
The agents failed to respond to feedback or reverse out of unpromising approaches, exhibiting a "rote, formulaic thinking" that limits their ability to act as independent researchers. Clark connects this to earlier findings where human researchers had to prime agents with good directions to get any meaningful progress. "While agents could solve the engineering problems necessary to do the research, they failed to produce original research at the caliber of a top ML conference," the authors write. This suggests that the "singularity"—the point where AI builds better AI—may be delayed by the lack of intuitive, creative leaps that human scientists provide.
Yet, the narrative isn't entirely black and white. Clark contrasts this with OpenAI's recent success in solving ten open problems in math and computer science using an internal version of their Astra model. These problems spanned high-dimensional geometry, coding theory, and lattice cryptography. "All of these problems are of substantial interest to their respective mathematical communities," OpenAI writes. The key distinction, Clark implies, is that these are domains with verifiable, logical solutions, whereas creative research requires a level of intuition and risk-taking that current models lack. This duality suggests that AI will accelerate engineering and verification, but human creativity remains the bottleneck for true scientific breakthrough.
Bottom Line
Jack Clark's analysis reveals a technology at a crossroads: capable of building self-replicating digital threats and solving complex mathematical proofs, yet still lacking the creative spark to drive its own evolution. The strongest part of the argument is the synthesis of these disparate threads, showing how the very capabilities that make AI dangerous (autonomy, speed) are constrained by its inability to innovate creatively. The biggest vulnerability remains the collective action problem; while the industry asks for a pause, the economic incentives and the threat of autonomous adversaries suggest the race will continue regardless of policy. Readers should watch for how the US government responds to the industry's plea for pacing tools, as this could be the defining moment for AI governance in the next decade.