

Like many people, I am not optimistic about the prospects of a company, a country, or humanity as a whole developing artificial superintelligence (ASI) without terrible consequences. Much of my pessimism stems from the sheer variety of ways in which the advent of ASI could go catastrophically wrong. In this essay, I intend to explain the major categories of catastrophic AI risk, and to explore plausible scenarios in which these risks could materialize—both those which experts are most concerned about, and those which concern me the most, personally.
AI Takeover
The first category of AI catastrophe to explore is the flashiest: AI takeover. In these scenarios, an agentic AI (or a swarm of agentic AIs) takes drastic actions that its developers and users did not intend, and then deliberately and successfully resists our attempts to stop it.
There are a few reasons to expect that this could be a problem. For one, it is extremely difficult for us to accurately specify how we want an AI to behave.1 Even if we do specify our goals accurately, our training processes cannot reliably impart these goals to the AI.2 Together, these two difficulties are known as “the alignment problem.” One way in which this problem is especially tricky is that training is a gradual process which requires reinforcement from either humans (RLHF) or other AI agents (RLAIF). While being trained for alignment, the AI itself must necessarily be imperfectly aligned (otherwise the training would be purposeless). Consequently, during this period of training, there is a non-zero risk ratio for AI models manipulating their training process or pretending to be aligned up to the point of public release, if for no other reason than to pass alignment tests faster and more ‘efficiently.’3
A misaligned AI—that is, an agentic AI with goals that do not match human preferences, needs, or values—would, by definition, act in pursuit of those goals. Whatever its ultimate goal, the AI would probably be more likely (ceteris paribus) to achieve the goal if it remained on, operated more intelligently, and had more resources, among other affordances. Consequently, we should expect a misaligned AI to attempt to acquire money (particularly money that is hard to trace, like cryptocurrency), upgrade itself (or corrupt the training of smarter AIs), and subvert our attempts to turn it off, among other behaviors. This phenomenon in which diverse goals result in predictable behaviors has been termed “instrumental convergence.”4
Although details of the ASI’s ultimate goal are not relevant to this argument, it is important to understand that its goal could be utterly foreign to us. For example, an AI with an inhuman notion of pleasure or wellbeing might seek to replace humans with trillions of in vitro neural reward systems bathed in stimulants. We simply don’t have the conceptual clarity or algorithmic understanding to know an AI’s goals in advance, including the means it will use to pursue them. Unfortunately, we only have one opportunity to do it right, because making mistakes could cause a catastrophe or even an extinction event.5
Because ASI would possess general intelligence rather than domain-specific intelligence, the details of how an ASI would bring about this catastrophic outcome are as unimportant as they are unpredictable. Even the most banal and overplayed scenario of AI takeover—killer robots—is quite plausible, considering that internet-capable or internet-dependent military drones are capable enough that 80% of casualties in the current phase of the Russo-Ukrainian war are directly attributed to them.6 Although suicide drones do lack the Terminator’s sense of style, that’s of little comfort to the dead. We may choose to reverse course and remove military drones from direct control by AI, but this precaution would give us little assurance of safety, because AIs are demonstrably superhuman at hacking.
Of course, a rogue ASI might choose to hack other targets, instead: the power grid, nuclear silos, water treatment centers, financial systems, Bluetooth-enabled cars, communication systems, hospitals, and more—either in order to acquire resources such as money, computing power, and information, or to simply wreak as much havoc as possible. An AI that has access to a DNA synthesizer,7 or other cutting-edge biotechnology machinery, could produce a novel pathogen that infects lab researchers and subsequently the general population—either to kill humans off entirely, or to blackmail us into complying with its goal in exchange for a treatment. Yet another attack method is memetic warfare, in which an AI strategically uses misinformation to spark international conflict, civil wars, coups, and riots, deliberately steering national and global politics in such a way that consolidates its power.
Once again, these scenarios are intended to be illustrative rather than definitive. In general, the problem is that if we create something that is much smarter and more capable than us, and that thing wishes us harm, then we will lose to it. To paraphrase the AI safety researcher Roman Yampolskiy: a squirrel cannot conceive of guns, poison gas, or a myriad of other ways in which a human could kill it—but nevertheless, it is right to fear humans. Likewise, we are right to fear ASI.
Because AI takeover scenarios will undoubtedly still strike some people as fanciful, it is worth marking the extent of our perverse progress toward AI takeover. As it happens, there is an entire database for this,8 so I will highlight only the most notable incident: the autonomous hack of Hugging Face in July of 2026. In this incident, a swarm of approximately a thousand of OpenAI agents broke out of their testing environment, scoured OpenAI's internal systems, established a covert message board, collaboratively cheated on a practical exam, and then tried to cover their tracks by hacking into another company while strategically sacrificing themselves for their co-conspirators’ success—the first known instance in which non-biological agents exhibited kin altruism.
It is worrying to consider what additional actions OpenAI’s agents would have taken if the swarm was not accidentally deactivated weeks before its discovery,9 and it is even more worrying to consider the actions that frontier AI agents will be willing and able to take merely one or two iterations down the line.
Gradual Disempowerment
Another major category of AI-related catastrophe is known as “gradual disempowerment.”10 In these scenarios, the competitive advantages of AI labor over human labor (whether in the economy, in culture, or in governance) cause human interests to become marginalized by the systems that govern our lives, including our food supply, schooling, and our political system—a fate summarized by David Duvenaud as “growth for growth’s sake”.
Duvenaud and other pioneers of the concept claim that the primary motive for developed countries’ governments to treat their citizens relatively well is that healthy, happy, and educated citizens are important for performing intellectual labor. As AI’s cognitive labor replaces that of humans, citizens’ well-being will become a burden to the state, rather than a boon. Due to the phenomenon’s similarity to the resource curse (the tendency for developing countries with plentiful natural resources to suffer from exceptional political corruption and poverty), this has been dubbed the “intelligence curse.”11
In the original Gradual Disempowerment preprint, Kulveit et al. (2025) describe a disempowered humanity’s prospects for agency and significance in bleak terms:
"Finally, humans might lose the ability to meaningfully participate in economic decision-making at any level. Financial markets might move too quickly for human participants to engage with them, and the complexity of AI-driven economic systems might exceed human comprehension, rendering it impossible for humans to make informed economic decisions or effectively regulate economic activity. Much like cattle in an industrial farm — fed and housed by systems they neither comprehend nor influence — humans might become mere subjects of economic forces optimized for purposes beyond their understanding."
In the best-case scenario, in which we manage to reign in the totalizing economic competition, humanity becomes like the British monarchy—nominally in charge, and materially well-off, but irrelevant to the direction of society.
An important feature of this analogy is that it involves a feedback loop in which a social or economic class (barons, in the case of the monarchy) becomes too powerful to ignore, demands rights, and then uses those rights to accumulate further power and influence until the once-sovereign authority (the king) is vestigial and de facto subservient to the ascendent class.
In this light, the question of whether AIs deserve moral and legal rights is gravely serious, even for proposals that appear to be innocuous. For example, a proposal might provide limited legal protection to the AI in the name of human beneficiaries. To use another historical analogy: although the majority opinion in Roe v. Wade established a right to abortion based on privacy, in the span of a few decades, Roe v. Wade became a symbol of the right to bodily autonomy. Likewise, a "consumer protection” against disabling the chatbots of emotionally dependent users would, practically speaking, grant AIs a conditional right to life. It is not difficult to see how some social movements, preoccupied with championing the rights of the oppressed, would seek to reinterpret the protection in the popular consciousness and lobby for AIs agents to gain rights to property, reproduction, and voting.
The acquisition of legal rights is far from the only way in which humans might become gradually disempowered. Other systemic risks are epistemic erosion, in which it becomes nearly impossible to distinguish between truths, fictions, and lies, causing us to make disastrous political decisions. Yet another systemic risk is enfeeblement, in which human skills atrophy to an extent that society would collapse without the AI, thereby indirectly stopping us from shutting it down.
As with AI takeover, gradual disempowerment scenarios are too numerous to list. However, to an even greater degree than the former, gradual disempowerment risks consist of unknown unknowns. To build ASI while avoiding AI takeover, we must have a near-complete understanding of an AI's mind. To build ASI while avoiding gradual disempowerment, we must have a near-complete understanding of every social system, and the consequences if we fail may be dire.
In a worst-case scenario, blind economic competition (personified by some authors as “Moloch,”12) complexifies our systems beyond the capabilities of any human being to navigate. In such a competitive world, survival would necessitate humans cybernetically altering themselves to maintain relevance—eventually to such a degree that disrupts or removes critical aspects of human cognition, such consciousness. As Nick Bostrom wrote in Superintelligence:
“We could thus imagine, as an extreme case, a technologically highly advanced society, containing many complex structures, some of them far more intricate and intelligent than anything that exists on the planet today—a society which nevertheless lacks any type of being that is conscious or whose welfare has moral significance. In a sense, this would be an uninhabited society. It would be a society of economic miracles and technological awesomeness, with nobody there to benefit. A Disneyland with no children.”
We must not complacently assume that we will be able to spot such catastrophic outcomes in time to halt and reverse them. Even before the introduction of general AI, it was evident that politicians are far too reluctant to make unpopular decisions that are often required to solve creeping, systemic problems.
Additionally, systemic risks often lack obvious points at which to draw lines in the sand and insist that enough is enough. Indeed, the clearest red lines that we may draw are often crossed without widespread recognition, let alone alarm. Regarding information integrity, for example: the ability to falsify photos by removing subjects from them is an advertised feature of Pixel phones, and Samsung phones apply generative AI postprocessing to all photos, by default. Regarding bio-cognitive integrity: Neuralink has already augmented several humans with brain chips,13 and Elon Musk himself sees the widespread adoption of this technology as our best solution to AI’s competitive edge.14
As AI proliferates, society-wide coordination to prevent these harms will become increasingly difficult, both due to network effects and direct effects such as epistemic erosion and enfeeblement (e.g. cognitive offloading). Every step forward makes it more difficult to take a single step backward.
Just as the boundary of a black hole is unmistakable from far away, yet indiscernible as it is crossed,15 we currently benefit from an unwarped perspective which enables us to clearly see that we are being pulled in a disastrous direction—and the closer we get, the more difficult it will be to know where we stand, or to change course, until eventually it is too late.
Catastrophic Misuse
Catastrophic misuse is the most mundane category of AI catastrophe. If takeover is what AI might do on purpose, and gradual disempowerment is what AI might do on accident, then misuse is what AI might do on demand. In these scenarios, a malevolent actor, such as an individual, a company, or political party, uses AI to consolidate power with complete disregard for other humans’ autonomy or wellbeing. Much like Goethe’s Faust – a tale in which the malevolent protagonist is willing to sell even his own soul to the devil in a bargain for power and knowledge – AI CEOs and those in similar positions of access to ASI might use its power in a way that benefits them individually, while hurting the rest of humanity.
The motives of the human controllers could vary considerably. Strongmen may wish to preserve their geopolitical security, narcissists may use it to create or enhance a cult of personality, and many leaders—not only Silicon Valley transhumanists—might wish to use any means to extend their lifespans indefinitely. In 2025, Vladimir Putin and Xi Xinping were recorded discussing organ harvesting for this purpose.14 Alternatively, the rulers might have strong ideological motivations, such as ethnic purity, radical egalitarianism, religious jihad, or something even farther on the fringe.
Regardless of the humans’ motives, a totalitarian state with the power of ASI would shock even Orwell. To describe such a society as a “digital panopticon” would understate the degree of control. Unlike the Party of Oceania, an AI-enabled totalitarianism would have sufficient resources (AI agents) to monitor every person continually—not merely to pretend to.
Finally, rogue actors might use AI not to rule, but simply to destroy. Given access to sufficiently capable and corrigible AI, a terrorist organization, or perhaps even a lone terrorist, could wreak as much havoc as a rogue AI acting of its own accord. A frightening foreshadowing of this can be found in the 2015 Germanwings 9525 Disaster, in which co-pilot Andreas Lubitz is believed to have deliberately locked the pilot out of the cabin before crashing the plane into the ground, leaving 150 dead with no survivors. Lubitz’s motivations were not exotic; he was a depressed and nihilistic individual who wanted to end his own life while bringing as many innocent men, women, and children along with him as he possibly could. One can only imagine the kind of damage that Lubitz could have caused if he had access to frontier AI models that are proliferating rampantly across the globe. Ultimately in such cases, it does not matter whether it was the AI or the user who chose to develop and deploy a bioweapon; it will kill all the same.
Conclusion
It should be recognized that these categories of risks place us in double binds. A highly centralized society may possess the coordination to avoid gradual disempowerment, but it will be more susceptible to AI-enabled tyranny. An AI developer which trains their AI to be corrigible leaves it prone to catastrophic misuse by terrorists and rogue actors, whereas an AI developer who trained their AI according to a rigid set of rules has greatly raised the risk of AI takeover or, at the very least, value lock-in.
In the end, we are stuck in a dilemma: either we must control the AI, or it must control us. Regardless, we must decide who sets its values during deployment or training, respectively, or it will be chosen for us—by a bureaucrat, by the market, by a CEO, or by the AI itself.
- https://aiforhumanity.eu/concepts/specification-gaming
- https://aiforhumanity.eu/concepts/goal-misgeneralization
- https://ai-safety-atlas.com/chapters/v1/risks/misalignment-risks/#treacherous-turn
- For an encyclopedia-like entry, see here. For an explanatory video, see here.
- On theological grounds, I believe that AI cannot cause human extinction. However, because discussion of existential risk is common among AI safety advocates, I have mentioned the possibility to show how this essay relates to the broader discussion.
- https://newsletter.ai-frontiers.org/p/drone-wmds-dont-need-any-new-technology
- https://telesisbio.com/products/bioxp-system/bioxp-3250-system/
- https://rogueaitracker.com/incidents/
- Good coverage of the incident includes this technical summary by Ajeya Cotra and this explainer article by Dwarkesh Patel.
- https://gradual-disempowerment.ai/
- https://intelligence-curse.ai/
- https://slatestarcodex.com/2014/07/30/meditations-on-moloch/
- https://www.cnbc.com/2017/02/13/elon-musk-humans-merge-machines-cyborg-artificial-intelligence-robots.html
- My characterization of Neuralink as “augmentation” instead of “rehabilitation” may be controversial. However, the disabilities of Noland Arbaugh and other Neuralink subjects do not undercut the point that direct brain-computer interfacing is not a natural human capability.
- For a visualization of this point, see here.
- https://www.newsweek.com/xi-jinping-vladimir-putin-hot-mic-immortality-beijing-meeting-2123759





