HOW MIGHT

ARTIFICIAL
INTELLIGENCE
AFFECT THE RISK OF
NUCLEAR
WAR?
EDWARD GEIST | ANDREW J. LOHN

Perspective
EXPERT INSIGHTS ON A TIMELY POLICY ISSUE C O R P O R AT I O N
 Limited Print and Electronic Distribution Rights
This document and trademark(s) contained herein are protected by law. This
representation of RAND intellectual property is provided for noncommercial use
only. Unauthorized posting of this publication online is prohibited. Permission is
given to duplicate this document for personal use only, as long as it is unaltered
and complete. Permission is required from RAND to reproduce, or reuse in another
form, any of our research documents for commercial use. For information on
reprint and linking permissions, please visit
[Link]/pubs/[Link].
RAND’s publications do not necessarily reflect the opinions of its research clients
and sponsors. R® is a registered trademark.
For more information on this publication, visit [Link]/t/PE296.
© Copyright 2018 RAND Corporation
T
oday’s nuclear balance relies on several conditions but the tracking and targeting system needs only to be perceived
that may not hold. Progress in computing and data as capable to be destabilizing. A capability that is nearly effec-
availability are making it possible for machines to tive might be even more dangerous than one that already works.
accomplish many tasks that once required human effort The trajectory of AI development, together with that of comple-
or were considered altogether impossible. This artificial intelli- mentary information technology and other advancements, will have
gence (AI) might portend new capabilities that could spur arms a large effect on nuclear-security issues in the next quarter century.
races or increase the likelihood of states escalating to nuclear AI technology could continue to evolve rapidly, as it has in recent
use—either intentionally or accidentally—during a crisis. 1
years, or it could plateau once current techniques mature. Some theo-
The R AND Corporation convened a series of workshops that rists postulate that machines might develop the ability to improve
brought together experts in AI and nuclear security to explore their own intelligence at some point, resulting in “superintelligences”
ways that AI might be a stabilizing—or destabilizing—force by with abilities that humans could neither comprehend nor control,
the year 2040. but there is little consensus about how AI will advance, includ-
The effect of AI on nuclear strategy depends as much or ing the plausibility of superintelligences. Some envision an initial
more on adversaries’ perceptions of its capabilities as on what it breakthrough followed by setbacks; others suspect that progress will
can actually do. For instance, it is extremely technically chal- remain incremental.
lenging for a state to develop the ability to locate and target all The two extreme cases have only limited relevance for the
enemy nuclear-weapon launchers, but such an ability also yields future of nuclear warfare. Stalling development (also referred
an immense strategic advantage. States therefore covet this to as AI winter) would result in only minor changes from the
capability and might pursue it irrespective of technical difficul- current nuclear-security environment. With superintelligence,
ties and the potential to alarm rivals and increase the likelihood AI would render the world unrecognizable and either save or
of conflict. The case could be made on technical grounds that destroy humanity in the process. The other two cases, in which
advanced AI would still struggle to overcome obstacles originat- AI progresses substantially and enables many new capabilities
ing from data limitations and information-theoretic arguments, while still remaining fallible and inferior to humans in at least

1
some respects, seem to have more support from the expert com-
munity, although experts disagree about the national security Without being directly connected to
implications of such capabilities. Some fall in the category of the nuclear launchers, an AI could still
“Complacents”: These tend to believe that producing an AI provide advice to humans on matters
capable of performing the types of tasks that would destabilize
of escalation.
the nuclear balance is sufficiently difficult that it is unlikely
to be achieved. “Alarmists” hold the opposite view, that an AI
multipolar world. Effective deterrence will require us to con-
could be capable of certain tasks but should not be included
tend with the rapidly changing set of capabilities being driven
in any aspect of nuclear war. A third group, “Subversionists,”
by progress in AI. Key considerations include the impact of the
focus on an adversary’s ability to alter, mislead, divert, or
actual capabilities, the perceived potential of those capabilities
otherwise trick the AI, which could prove either stabilizing or
(whether they exist or not), and the premature use or fallibility
destabilizing.
of those capabilities, especially as a result of adversarial actions.
One example discussed in the workshops was an AI that
With care and some forward-thinking, these risks can poten-
acts as a decision support system. Without being directly con-
tially be identified and mitigated.
nected to the nuclear launchers, an AI could still provide advice
to humans on matters of escalation. It seems reasonable that
such a capability, at least for some aspects of the decisionmak-
Hints of Major Changes Ahead for the Nuclear
ing process, could be achieved by 2040 given the progress AI
Balance
is making in increasingly complex and poorly specified tasks.
November 2015, Russia revealed that it was developing the ultimate
Alarmists might be concerned that such a capability could be
“killer robot”: a nuclear powered undersea drone designed to carry
incorporated before it is sufficiently robust or without fully
an enormous thermonuclear warhead. Russian television revealed
understanding its limitations. If an AI adviser were proven effec-
the existence of this nightmarish weapon in an “accidental” leak that
tive, however, it could increase stability by reducing the likeli-
most Western observers concluded was intentional. Television cam-
hood of human error and by providing radical transparency,
eras lingered momentarily on an ostensibly classified briefing slide
which could reduce the risk of miscalculation. But many experts
for President Vladimir Putin describing the “Oceanic Multipurpose
were concerned by the potential for an adversary to subvert even
System Status-6.” Shaped like an enormous torpedo and powered
a very capable AI by hacking, poisoning its training data, or
by a compact nuclear reactor (see the figure on page 3), Status-6
manipulating its inputs.
would overcome enemy defenses through a combination of speed and
Maintaining strategic stability in the coming decades will
range that would enable it to outrun almost anything in the ocean
require revisiting the foundations of deterrence theory in a

2
(Sutyagin, 2016). The drone would be launched from submarines in sian leaders’ concerns about the credibility of their retaliatory
the Russian arctic, traverse the ocean at perhaps 100 km/hr while forces in the face of U.S. counterforce targeting capability and
autonomously circumventing antisubmarine defenses, and deliver its missile defenses. Unable to match these capabilities in kind,
deadly payload to the U.S. coastline, presumably after a villainous contemporary Russia hopes to exploit AI to ensure the cred-
American first-strike attack destroying the Kremlin. The difficulty ibility of its deterrent. It might succeed by 2040 because the
of communicating underwater would require a degree of autonomous Kremlin continues to explore novel ways of employing AI for
capability on the part of the drone that has become possible only military purposes. This effort is in keeping with its decades-
recently as a result of progress in AI. 2
old strategy of developing “asymmetric responses” to superior
Status-6 is not just a concrete application of AI; it is a reflection U.S. capabilities. Russia’s undersea “doomsday drone” is merely
of AI’s potential looming impact on nuclear deterrence—the use of the most extreme example of this phenomenon so far.4
retaliatory threats to dissuade an adversary from attacking a state or Will nuclear deterrence be recognizable in 2040? Status-6 is
its allies. The nuclear drone is the latest manifestation of Rus-
3
a stark warning that if technological progress undermines nuclear

Components of Status-6

NUCLEAR SUBMARINE CARRIERS OF SELF-PROPELLED UNDERWATER VEHICLES

Project 09852 SSN Project 09851 SSN

SELF-PROPELLED UNDERWATER VEHICLE

sensors warhead guidance reactor drive system

system compartment
Operational depth = <1,000 m; speed = <185 km/hr; range = <10,000 km; diameter = 1.6 m

The “Oceanic Multipurpose System Status-6,” shaped like an enormous torpedo and powered by a compact nuclear reactor,
would overcome enemy defenses through a combination of speed and range that would enable it to outrun almost anything in
the ocean.

3
powers’ sense of security, those powers could attempt to salvage
their nuclear deterrents by embracing unprecedented new weapon These unfamiliar strategic arrangements
systems and force postures. These unfamiliar strategic arrangements could prove less stable than those that kept
could prove less stable than those that kept an uneasy peace between an uneasy peace between the United States
the United States and the Soviet Union, and instability increases
and the Soviet Union.
the probability of nuclear war. The extent to which risk increases
depends in considerable part on the rate and extent of progress in
building on the premise that the future geostrategic order is more
AI, which could enable new ways of both delivering nuclear weap-
predictable than the development of AI technology. Discussion was
ons and defending against nuclear attack. In May and June of 2017,
seeded with several specific scenarios in which conflicts between the
the RAND Corporation convened three workshops with nuclear-
nuclear powers became more acute. These included:
security professionals and AI researchers to discuss the impact of AI
1. a “resurgent Russia” scenario, in which the New START treaty
on nuclear security. Participants appeared to agree that advanced
collapses and Russia achieves a significant advantage in strate-
AI could severely compromise nuclear strategic stability and thereby
gic nuclear arms over the United States by the early 2030s
increase the risk of nuclear war. However, there was not agreement
2. a “rising China” scenario, in which China gradually expands
about how and why AI would have this effect, even within respective
its strategic nuclear arsenal and achieves parity with the United
constituencies.
States and Russia
3. a “successful limited use” scenario, in which Pakistan success-
Methodology and Description of Workshops
fully uses tactical nuclear weapons to persuade India to with-
To investigate the potential influence of advanced AI on nuclear
draw an invading force, breaking the “nuclear taboo”
security in the next quarter century, RAND conducted a series
4. a “regional nuclear war” scenario, in which a North Korean
of workshops in May and June of 2017. These workshops brought
regime undergoing collapse lashes out against South Korea,
together a variety of expert groups, including both nuclear-security
Japan, and China, resulting in the devastation of the region.
professionals and AI researchers, as well as participants from govern-
ment and industry, resulting in a variety of diverse perspectives.
Workshop participants were asked to flesh out these scenarios with
technical details of respective powers’ nuclear forces—including
Workshop 1
number and capabilities of delivery systems and C4ISR (command,
The first workshop was held at RAND’s Santa Monica office on
control, communications, computers, intelligence, surveillance,
May 1, 2017, and many of the 16 participants were RAND research-
and reconnaissance)—with the aim of identifying AI applications
ers working in nuclear or AI-related fields. The aim of the workshop
that might be of interest to nuclear states. Future combat systems
was to envision strategic environments with which AI might interact,

4
were presumed to be similar to those in development today because is intrinsic to most of the learning techniques of most machines, so
military acquisition time lines are slow, but AI progress can be made states will be able to employ these approaches as a means to prevent
much faster than defense system acquisitions. Participants seemed an adversary from tracking its launchers.
to agree that applying advanced AI to these systems would likely be The second issue at this workshop addressed was the use of AI
a destabilizing influence in a future standoff. However, participants in decision support systems to advise decisionmakers on strategic
also postulated that for every destabilizing technology, there is a nuclear issues in crisis or conflict. The groups disagreed considerably
counter, stabilizing technology. This theme was developed further in about the use of AI for these tasks, with some saying they should be
the second workshop. kept under strictly human control, while others declared that to be
unrealistic. This topic is discussed in detail later on.
Workshop 2 Finally, the workshop characterized possible lessons from nuclear
A group of 19 participants contributed to the second workshop, arms control for future AI applications. Participants seemed to
which was held in San Francisco on May 25, 2017. Seven of these agree that it is not possible to replicate the kind of legal structures
participants described themselves as being in the AI field, five and norms that have been used to forestall nuclear proliferation to
described themselves as being in national security, three said they head off military AI applications because nuclear technology and AI
were in both, and four in neither. The AI-focused contributors are too different. In the specific case of AI for nuclear war–related
included prominent figures from commercial, academic, and non- tasks, participants pointed out that controlling AI might be difficult,
profit AI research organizations, as well as AI policy communities, but other components essential for those applications (i.e., sensor
while the national security participants included nuclear-weapon platforms) could be subjected to monitoring and control. Participants
experts from the national laboratories. Participants divided into sub- discussed whether it might be possible to control AI by control-
groups to discuss three issues. ling data, human talent, or processing resources. Several of the AI
The first issue was whether AI could enable states to track and researcher participants argued that the current shortage of human
target adversary retaliatory forces and thereby undermine the premise talent is temporary and training data would become less important
of assured retaliation that forms the basis of much of nuclear strategic as simulations improve, but that hardware might then become the
theory (discussed in depth in a later section). One group, dominated limiting factor. In their view, the limited number of factories making
by nuclear-security experts, concluded that AI could accomplish such components as graphics processing units might make it possible
this but met with disagreement from a second group that included to construct some sort of control regime, but many other participants
a prominent expert on generative adversarial networks (a technique were skeptical of this.
in which a generator neural network interacts with a classifier neural
network to learn to create increasingly realistic fake examples). In
this group’s view, vulnerability to adversarial manipulation attacks

5
One subgroup suggested provocatively that a future AI system could essentially be the arms
control regime, monitoring compliance and adjudicating violations without human input.

Workshop 3 programs in light of the possibility that AI might significantly

The third and final workshop took place at RAND’s office in Arling- reshape the strategic landscape. Participants noted that the current
ton, Virginia, on June 9, 2017. This workshop had 15 participants, program has many vulnerabilities but that apparent alternatives are
including eight who described themselves as focusing on nuclear not obviously better even if they probably merit at least some analy-
issues and five as focused on AI, although most of the latter work in sis. Domestic and international institutional pressures discourage the
the policy space rather than as AI research practitioners. The remain- United States from departing significantly from the current “triad” of
ing two participants contributed valuable expertise with respect to intercontinental ballistic missiles (ICBMs), submarines, and manned
acquisition policy. The group included both RAND researchers bombers.
and representatives from the U.S. Army, the Office of the Secretary The third discussion addressed how AI might contribute to
of Defense, and the State Department Bureau for Arms Control, nuclear arms control. AI might be used for such tasks as treaty verifi-
Verification, and Compliance. This workshop built on the findings cation, by enabling increased transparency and trust. One subgroup
of the previous two, asking the attending policy practitioners how suggested provocatively that a future AI system could essentially be
they would address the challenges identified in the previous two the arms control regime, monitoring compliance and adjudicating
workshops. violations without human input. Finally, the participants considered
The first discussion focused on the tracking and targeting prob- whether it is possible or desirable to apply arms control to AI itself.
lem and asked participants to consider how they would try to thwart Most participants were skeptical of the feasibility and desirability of
an adversary seeking to render strategic forces vulnerable using AI. this goal; many regarded it as either a practical impossibility or some-
Participants suggested trying to neutralize this capability by attack- thing that would require extreme and unacceptable interventions,
ing the associated sensors and communications network rather than such as interning AI researchers.
the AI itself. In the subsequent discussion, participants considered
the challenges posed by the generative adversarial techniques empha-
sized by the AI researchers in the second workshop, although no Theoretical and Historical Background for
consensus emerged (possibly due to the unfamiliarity of most of the Assessing AI’s Potential Influence
attendees with technical details of these methods). During the Cold War, both the United States and the Soviet
The second discussion addressed whether the United States Union begrudgingly accepted the condition of mutual assured
needs to reconsider the trajectory of its nuclear force-modernization destruction (MAD)—the premise that any all-out attack would be

6
met with an apocalyptic retaliatory strike ensuring that both societies adversary from attacking oneself or one’s allies. Deterrence can be
would be destroyed. MAD was a condition, rather than a strategy— categorized into central deterrence (deterrence of an attack on one’s
one that both superpowers hoped to escape if possible (Buchan et al., homeland) and extended deterrence (deterrence of an attack on one’s
2003). Even if mutual vulnerability made a general nuclear exchange strategic partners) (Cimbala, 2002). Nuclear weapons can also be
less likely, the omnipresent possibility that war might still occur by used for compellence—coercing the enemy into doing something
accident or miscalculation weighed heavily on the minds of super- that it does not want to do (Long, 2008, p. 9). In addition to coer-
power leaders. Ronald Reagan, for instance, called on scientists to cive deterrence and compellence threats, nuclear weapons can be
create a missile defense that would render nuclear weapons “impotent employed for warfighting, the way they were at the end of the Second
and obsolete,” while the Soviet Union developed an elaborate civil World War. The practical complexities of nuclear strategy stem from
defense program (Garthoff, 1987; Geist, 2012). Nor was MAD a the challenges of assurance—making extended deterrence credible.
sufficient basis for U.S. or Soviet nuclear strategy. While MAD cred- During the Cold War, the United States accumulated its massive
ibly deterred a Soviet preemptive strike on the United States, it also stockpile of strategic and tactical nuclear weapons to convince its
undermined the plausibility of U.S. promises to defend its European allies that that it would be willing to retaliate to conventional Soviet
allies in the North Atlantic Treaty Organization, even at the risk of attacks in Europe with nuclear responses. As United Kingdom
nuclear war. If Washington relied solely on MAD, the Soviet Union Defence Minister Denis Healey observed, it took “only five per cent
could exploit its conventional superiority to invade western Europe credibility of American retaliation to deter the Russians, but ninety-
and the United States would face a stark choice between capitulation
Categories of Nuclear Strategy Goals
or an all-out nuclear war. As a consequence, American strategists and
government officials developed the more comprehensive doctrine of
Aspect Definition
assured retaliation—the prospect that any enemy provocation would
be met by an appropriate and effective response (Long, 2008). By Coercion

threatening a retaliation scaled to likely enemy provocations, “assured

Deterrence Dissuade adversaries from doing something they want to do
retaliation” sought to credibly deter minor and all-out attacks. In
the later decades of the Cold War, a variant of this approach called Compellence Force adversaries to do something they do not wish to do

the countervailing strategy sought to deter all manner of attacks,

including preemptive counterforce attacks, by assuring that any such Assurance Convince allies that security guarantees are credible

attacks would fail to accomplish their objectives because of U.S.

retaliation (Slocombe, 1981).
Convince adversaries that they will not be attacked so long
Reassurance
Nuclear strategy is about more than just deterrence (see the table as they refrain from provocative behavior

opposite). Deterrence is the use of retaliatory threats to dissuade an

7
five per cent credibility to reassure the Europeans” (Healey, 1989). In an extreme case, AI could undermine the condition of MAD
The scale of the U.S. nuclear arsenal, however, alarmed Soviet lead- and make nuclear war winnable, but it takes much less to undermine
ers, who believed that the Americans might be attempting to develop strategic stability. AI advancements merely need to cast doubt on
a first-strike capability against them. This distrust underscored the the credibility of retaliation at some level of conflict. Major nuclear
need for reassurance—convincing adversaries that they will not powers, such as the United States, Russia, and China, have a shared
be attacked as long as they refrain from the behavior that is being interest in maintaining the credibility of central deterrence, but they
deterred (Schelling, 1966). seek regional advantages in pursuit of what they regard as their core
Strategic stability exists when adversaries lack a significant strategic interests. Areas where credibility is already strained, such as
incentive to engage in provocative behavior. There are several kinds
5
certain extended deterrence guarantees, are particularly vulnerable
of strategic stability that are distinguished by their varying temporal to destabilization. The increasingly multipolar strategic environment
scales. First-strike stability exists when no state can carry out an attack is also encouraging forms of competition that threaten stability. For
out of the blue against its opponent without significant fear of a dev- instance, the United States is interested in developing the capability
astating retaliation. Such a possibility is best deterred by the threat to track and target a minor nuclear power’s mobile missile launchers,
of overwhelming and automatic retaliation from secure second-strike but Russia and China fear that the same technology could mature
forces (Cimbala, 2002, p. 66). Crisis stability, by contrast, aims to into a threat to their more sophisticated retaliatory forces. In a crisis
prevent or manage escalation during crises, as occurred in Berlin and situation, the employment or availability of AI-enabled intelligence,
Cuba in the early 1960s (Cimbala, 2002, p. 98). In these circum- surveillance, and reconnaissance (ISR) or weapon systems could
stances, national leaders are under immense pressure not to show stoke tensions and increase the chances of inadvertent escalation.
weakness by backing down, but the chance of inadvertent escala- Finally, the pursuit of advanced military capabilities is liable to cause
tion increases significantly as states attempt to maneuver the nuclear arms race instability even if those technologies are nonviable, as in
forces for signaling purposes. In this context, the kind of large the historical case of missile defense.
automatic retaliation that is ideal for maximizing first-strike stability The challenge AI poses to strategic stability is not unique to this
is a recipe for disaster. particular technology, but it is more acute because of rapid technical
Finally, arms race stability is achieved when there are no exploit- progress in AI and its many potential intersections with nuclear strat-
able inequalities in adversaries’ military capabilities (Cimbala, 2002, egy. Most of the specific applications AI are likely to be used for, such
p. 110). States avoid these inequalities to manage the risks and costs
of long-term competition and to avoid compromising first-strike
Strategic stability exists when adversaries
stability and crisis stability in the future. Nuclear strategy is difficult
because these objectives are in tension with each other. lack a significant incentive to engage in
provocative behavior.

8
as analysis of ISR data, controlling autonomous sensor platforms, and ware, and the rise of cloud computing and big data, AI has advanced
automated target recognition (ATR) have been eagerly sought for rapidly in the past few years, most prominently in the field of “deep
decades but were beyond the capability of available technology. Even neural networks” (DNNs), or neural networks with many layers
without further breakthroughs, incremental progress using existing (Goodfellow, Bengio, and Courville, 2016). The increase in the per-
AI techniques may make these long-sought goals practical realities in formance of DNNs has been so spectacular that they have become
the foreseeable future. almost synonymous with AI, but in actuality the older paradigms are
Both Russia and China appear to believe that the United States also continuing to progress and are in widespread commercial and
is attempting to leverage AI to threaten the survivability of their military use. Some impressive recent AI systems, such as Alphabet
strategic nuclear forces, stoking mutual distrust that could prove DeepMind’s AlphaGo program, which beat the world Go champion,
catastrophic in a crisis. As Paul Bracken observes, ongoing improve- employ DNNs in combination with such older techniques as search-
ments in technology such as AI threaten to “undermine minimum ing the tree of possible moves. One thing that has remained constant
deterrence strategies” and “blur the line between conventional and over AI’s 60-year history is its proponents’ high hopes. With enough
nuclear war” (Bracken, 2017). intelligence, might it be possible to conquer such seemingly impos-
sible problems as poverty and illness—or even win a nuclear war?
AI in the Cold War The intersection between AI and nuclear warfare became a
AI pioneer Marvin Minsky defined AI as “the science of making science fiction cliché more than 50 years ago, but their real-world
machines do things that would require intelligence if done by men” connections are even older. The earliest AI researchers were deeply
(Minsky, 1968, p. v). Since AI research began in the 1950s, the involved in national security work and secured government support
boundaries of the field have shifted as computers have reshaped how by suggesting that their theoretical studies would soon translate into
humans comprehend “intelligence.” AI has also evolved as theoretical practical military applications. Claude Shannon asserted in his foun-
paradigms have shifted in and out of vogue. From the 1950s until dational 1950 article “Programming a Computer for Playing Chess”
the 1980s, a “symbolic” paradigm that aimed to replicate high-level that making computers play that venerable game would impart
human reasoning predominated, only to be supplanted by a “con- theoretical insights that would make “machines for making strategic
nectionist” paradigm that sought to emulate the biological basis decisions in simplified military operations” possible “in the near-
of human cognition using artificial neural networks. In the 20th term future” (Shannon, 1950, p. 256). In the mid-1950s, research-
century, neither paradigm worked particularly well outside labora- ers created the earliest working AI programs with support from the
tory demonstrations. This triggered occasional periods (sometimes U.S. Air Force (Simon and Newell, 1958; Newell, Shaw, and Simon,
characterized as AI winters) during which funding for AI research 1959). Potential applications of such machines soon began appearing
was scarce. Thanks to decades of progress in computer science, in the writings of strategic theorists. In the late 1950s, Herman Kahn
advances in computing and communications hardware and soft- postulated the notion of “doomsday machines” that would employ

9
computers programmed to recognize unacceptable enemy provoca- the Survivable Adaptive Planning Experiment (SAPE), sought to use
tions and retaliate (Kahn, 1960, pp. 145–154). While Kahn intended the AI technology of the time to enable the United States to target
these as thought experiments illustrating how not to conduct nuclear the Soviet Union’s mobile ICBM launchers. The SAPE would not
strategy, science fiction authors latched onto the idea of intelligent control nuclear weapons directly; rather, it would employ expert
computers controlling nuclear weapons, inspiring numerous novels systems to translate reconnaissance data into nuclear targeting plans
and such films as Colossus (1970), WarGames (1983), and Terminator that would then be carried out by manned B-2 bombers. The SAPE
(1984). was just one part of an envisioned suite of systems and capabilities
While fictional thrillers spin tales of nuclear armed computers that, if actualized, would have severely challenged the survivability
run amok, real-world attempts to apply AI to nuclear strategic of the Soviet Union’s nuclear arsenal (Roland and Shiman, 2002,
problems tended to be much more mundane. Neither U.S. nor Soviet p. 305; Long and Green, 2012).
officials were inclined to entrust launch decisions to computers,
both because they jealously reserved this prerogative for themselves AI and the Emerging Geopolitical Order
and because automating retaliation was not a logical response to Although 20th-century AI struggled to actualize these applications,
difficult strategic problems, such as compellence or crisis stability. more-recent advances in computing could release their potential.
The sole notable exception came from the Soviet Union at the end Such contemporary techniques as deep learning are dramatically
of the Cold War. Perceiving that the United States aspired to a advancing machine vision and other signal processing applications,
first-strike capability and anxious that they might be the objects which can enhance autonomy and sensor fusion. Autonomy and
of a decapitation strike, Soviet leaders sought measures to ensure sensor fusion may be of paramount strategic relevance because they
that capitalist aggressors would never go unpunished. The Soviet
6
could greatly improve ISR, ATR, and terminal guidance capabilities.
Union reportedly considered developing a system that would have All of these might severely erode the means by which nuclear powers
automatically launched surviving ICBMs at the United States assure the survivability of their nuclear forces. Because increased
following a first strike if it could not contact the Soviet political weapon accuracy has long since undermined the survivability of
leadership. It seems that the fully automated version, nicknamed silo-based ICBMs, the United States, Russia, and China put nuclear
the “Dead Hand,” was rejected in favor of a version, dubbed weapons on submarines and mobile ICBMs that were deemed more
“Perimetr,” that would automatically delegate launch authority likely to survive a first strike. Technologies that make it more likely
to field commanders but would always require a human in the that survivable forces (such as submarine and mobile missiles) could
loop (Hoffman, 2009). According to Russian media accounts, the be targeted and destroyed make it more plausible that one country
Perimetr system still exists and uses some kind of AI. The United
7
might threaten a first strike. This undermines strategic stability,
States, meanwhile, explored the possibility that AI could be used to because even if the state possessing these capabilities has no intention
bolster its counterforce capability. One late 1980s research project, of actually using them, the adversary cannot be sure of that. Thus,

10
the capabilities can still be used to pressure potential adversaries and engaged in a vociferous debate in the military press about the extent
perhaps extract concessions during a crisis. Such a capability does of their country’s strategic vulnerabilities.8 Their tendency to assume
not have to be exploited during a crisis to be politically useful. As that current and future U.S. capabilities pose a dire threat to Russia’s
Alfred T. Mahan observed, “force is never more operative than when security stokes these anxieties.
it is known to exist but is not brandished” (Mahan, 1912, p. 105). A major challenge of nuclear strategy is that adversaries may
As long as adversaries fear that the capability may exist, they can be interpret one nation’s secure retaliatory forces as a first-strike threat
cowed into submission without explicit confrontation—the more or a doomsday machine and react accordingly. For instance, the
powerful state can in effect preemptively “win” the crisis. As a conse- Russians probably conceived of Status-6 as a last-ditch second-strike
quence, counterforce targeting capability is an enticing prospect for option exploiting AI to autonomously circumvent U.S. defenses, but
many despite its potential to compromise strategic stability. Western observers interpreted it as a Strangelovean “cobalt bomb.”
AI technologies could help enable new breakthroughs in AI progress is also contributing to Russia’s doubling down on older
tracking and targeting and in antisubmarine warfare or make types of systems with undesirable strategic properties. For example,
it easier for high-precision conventional munitions to destroy with its RS-28 “Sarmat” missile, Russia is reinvesting in large, silo-
hardened ICBM silos (Holmes, 2016). Such capabilities would be based ICBMs with multiple independently targetable reentry vehicle
especially destabilizing because decisionmakers could threaten to (MIRV) warheads, a category of weapon it once planned to abandon
employ conventional weapons much more plausibly than any kind under the now-defunct Strategic Arms Reduction Talks II treaty.
of nuclear attack. A conventional threat would place the adversary Western strategic theory generally considers large MIRVed ICBMs to
under enormous pressure during a crisis, which could force it to be destabilizing because they are ideal for preemptive strikes and are
capitulate—but could also spiral into nuclear war. Such escalation vulnerable to preemption.
could happen because the adversary felt the need to use its weapons At the dawn of the millennium, Moscow believed that it
before being disarmed, in retaliation for an unsuccessful disarming could ensure the survivability of its forces by emphasizing mobile
strike, or simply because the crisis triggered accidental use. ICBMs and scrapping large silo-based missiles inherited from the
Potential U.S. adversaries, such as Russia, take seriously the Soviet Union. However, Russian leaders’ anxieties about potential
possibility that the United States might leverage its advantage in such U.S. threats to the survivability of the mobile ICBMs seem to
technologies as AI to radically improve its counterforce capabili- have changed this calculus and led them to try to ensure retali-
ties. For the past several years, Russian military analysts have been ation by launching during a U.S. attack instead of riding it out.

A major challenge of nuclear strategy is that adversaries may interpret one nation’s secure
retaliatory forces as a first-strike threat or a doomsday machine and react accordingly.

11
 consider the various means by which AI could raise or lower the
The increasingly multipolar nuclear risk of intentional or accidental thermonuclear war. Even with our
environment also aggravates the potential current imperfect understanding, it is possible to start considering
strategic impact of AI. the impact of some emerging capabilities and their interactions.

This is tantamount to the adoption of a launch-under-attack pos- Commonly Held Expert Opinions on Possible AI
ture that could place great pressure on Russian leaders to launch Futures
first in a crisis, increasing the chances of accidental escalation. As discussed earlier, several perspectives dominated discussions at the
The Russians recognize that the Sarmat silo would be unlikely to workshops.
survive a preemptive attack on its own, so its survivability hinges
on an associated active defense system, code-named “Mozyr’,” Anticipating Progress in AI
that would attempt to force enemy warheads to detonate at a There are four main schools of thought regarding progress in AI. It
slight distance from a silo, allowing it to survive the nuclear is difficult to provide rigorous justification in favor of any one school
explosions. over any other. Nonetheless, their proponents often argue emphati-
The increasingly multipolar nuclear environment also aggra- cally on their behalf. For the most part, the experts who attended in
vates the potential strategic impact of AI. While six states had the our workshops were familiar with the various schools of thought and
bomb during the Cold War, five of them considered the Soviet were able to make arguments from all perspectives.
Union their primary enemy, making the strategic order essentially For each of the possible future states of AI, the common views
bipolar. This bipolarity encouraged both crisis and arms race from each of the categories of expert opinion with respect to nuclear
stability. Today, there are nine nuclear-weapon states and multiple security are summarized in the table on page 13. These views and the
strategic rivalries that indirectly affect one another. The United arguments supporting or refuting them are discussed further in the
States worries about Russia and China; Russia plans for confron- following sections.
tations with both the United States and China; China regards the
Superintelligence
United States, Russia, and India as potential adversaries; India is
Superintelligence is anticipated by some to be an inevitable state
embroiled in strategic competition with China and Pakistan; and
where machines come to hopelessly outmatch humans intellectually.
North Korea is a headache for almost everyone.
Such theorists as Oxford philosopher Nick Bostrom (2014) argue
Much work remains in developing theories of strategic stabil-
that once a superintelligence exists, two outcomes are possible: The
ity applicable to these complex multipolar conflicts. As analysts
superintelligence is benevolent and solves all humanity’s problems,
develop approaches to this challenging problem, they need to
or the superintelligence destroys humanity, either maliciously or

12
incidentally. Bostrom believes that recursively self-improving AIs increased intellects that would still be subhuman in at least some
might evolve to superhuman intelligence extremely quickly while respects. This could happen, for example, if a recursively reprogram-
committing few or no mistakes. This “intelligence explosion” might mable software system were to rapidly increase intelligence until
take hours or minutes. reaching the peak of what its hardware is capable of and being unable
In this case, the role of nuclear security is made trivial: if to advance further.
benevolent, superintelligence would save humanity from nuclear In this case, depending on the exact capabilities that emerged
war; if malevolent, nuclear strikes would be just one of many possible relative to humans, the AI would most likely be used to exploit
methods for extinction. its comparative advantages and humans would be used to maxi-
Superintelligence does not seem to be viewed as imminent or mize theirs. The AI remains fallible, as do humans, and the range
inevitable by the majority of experts in AI, but many supporters and impact of possible outcomes rely heavily on the nature of that
believe it merits attention because of the extreme nature of its costs fallibility.
and benefits, even if the likelihood of its occurrence is low.
Continuous Incremental Progress
Limited Breakout A third possibility is that a state similar to the one outlined above is
Short of creation of a true superintelligence, large and discontinu- reached not through discontinuous jumps in the progress of AI but
ous jumps in intelligence might also be possible, leading to greatly as a result of continuous incremental advancement. This progress

Alternative AI Futures from Expert Opinions

Possible Future States of AI

Categories of Limited Breakout or

Expert Opinion AI Winter Continued Incremental Progress Superintelligence

Data insufficient and problems too complex Unlikely to exist but would probably be safer than
Complacents Likely
for even advanced AI humans

Algorithms that barely work would alarm Inevitable eventually and likely to destroy humanity either
Alarmists Unlikely
adversaries and could fail if used intentionally or inadvertently

AI could be made to fail catastrophically,

Superintelligence resistant to both subversion and human
Subversionists Neutral or ability to cause AI to fail could provide
control
stabilizing assurances

13
would rely on increases in computing speeds, hardware architectures, such high levels of investment, projections of progress can become
algorithmic development, data availability, and decreasing costs. This self-fulfilling, much as Moore’s Law drove semiconductor develop-
is arguably the most plausible interpretation of recent trends in AI, ment for decades (Mack, 2011). Moreover, an “AI plateau” might
where the increases in AI capability are mundane when viewed at any occur after considerable progress from present-day capabilities, creat-
one moment in time but compelling when viewed over a few years— ing many of the same challenges for nuclear security as the previous
when projected over more than two decades to the year 2040, prog- two scenarios.
ress could be astonishing in the same way that the internet of today
would be barely recognizable to nontechnologists in the mid-1990s. Anticipated Impacts on Nuclear Security
While perhaps distinct from a philosophical or preventive policy The focus of this Perspective is mainly on the limited breakout and
perspective, continuous incremental progress over more than two continuous incremental progress cases because the other two are of
decades is largely indistinguishable from the limited breakout school limited relevance from a nuclear perspective. The limited breakout
in terms of outcomes and impact on the world of nuclear security. and continuous incremental progress projections are characterized
Just as for limited breakout, there would likely be aspects of superior- by AI that far exceeds human capacity in increasingly complex and
ity for both humans and machines while both remain fallible, and data-limited tasks. Experts disagree about what such capabilities
that fallibility would drive the risks. imply for nuclear security.

AI Plateau Complacents
A final perspective expressed by a few workshop participants pos- One common view is that AI will not profoundly change the status
ited that AI progress might plateau once current techniques reach quo aside from improving efficiency and transparency. Subscribers
technological maturity. Such an outcome would be distinct from to this “complacent” view tend to be focused more on technological
historical AI winters, during which AI research continued to advance issues than on strategy or policy. It should be noted that, although
even though funding and popular interest dropped markedly. For including some of the world’s most capable AI engineers, participants
instance, computer hardware might stagnate and deprive AI of the were probably more interested in AI safety than is typical in their
computational resources required to reach its theoretical potential. fields, so this view may have been underrepresented. Complacents
Currently active AI researchers are frequently skeptical of this would be more likely to believe that the complexity of nuclear war is
line of thought, although they acknowledge it is conceivable. Current too challenging for AI to contribute significantly and therefore AI’s
levels of direct investment in AI development around the world are impact on the existing balance would be negligible. Complacents
unprecedented and may not be sustainable, but the expressed level of would assert, for example, that challenges in data collection and in
commitment from private firms and such governments as China sug- distinguishing between real systems or actions and decoys would be
gest that funding will remain robust for the foreseeable future. With impossible for an AI to overcome, even by 2040. They would also

14
 Subversionists
Moreover, an “AI plateau” might occur A third perspective that can lead to positions that fall between the
after considerable progress from present- Complacent and Alarmist camps is rooted in concerns over AI’s
day capabilities, creating many of the susceptibility to adversarial actions. This stems from theoretical
considerations and demonstrations that such adversarial attacks are
same challenges for nuclear security as the
likely to be highly effective. This view does not always lead to the
previous two scenarios. same conclusions as the cases in which AI shows limited progress or
is perceived to be effective but is not, although there is overlap with
be more likely to view the problem of identifying and interpreting
both.
inputs for decisions regarding nuclear escalation as being sufficiently
In numerous convincing demonstrations, small amounts of
broad to be AI-complete. That is to say, any computer that could out-
adversarial effort toward subverting machine learning algorithms
perform humans at this task would necessarily have made the jump
have shown outsized effect. Some researchers argue that this is a
to being able to outperform humans generally.
pervasive trait of machine learning and that they expect that it will
Alarmists persist for years to come. Where an effective AI for tracking and
At the opposite extreme are Alarmists, who tend to believe that AI targeting might be destabilizing and lead to proliferation or worse,
will render existing systems vulnerable or will upset the present an adversary may regain trust in the survivability of its second-strike
strategic balance enough to be of grave concern. This camp includes forces if it is confident in its ability to forestall detection using these
those who would never entrust any aspect of nuclear decisionmak- adversarial methods, thereby reestablishing strategic stability. On the
ing to an algorithm. Some of the participants had personal experi- other hand, an actor may believe that it can subvert an AI’s ability to
ence with historical attempts to create algorithms to achieve related identify a preemptive first strike, making such a strike a viable option
objectives. In some cases, the ineptitude of those algorithms and their and therefore destabilizing.
inability to consider the emotional and ethical aspects of a decision
had made the participants uncomfortable with the intersection of Illustrative Case: Tracking Mobile Missile
AI and nuclear issues. Alarmists also argue that an AI needs only to Launchers
be perceived as highly effective to be destabilizing—for example, in AI may be strategically destabilizing not because it works too
the tracking and targeting of adversary launchers. Threatened with well but because it works just well enough to feed uncertainty. To
potential loss of its second-strike capability, an adversary would be illustrate this point, in this section we describe the results of ear-
pressured into a preemptive first strike or into expanding its arsenal, lier RAND research on the problem of targeting mobile missile
both undesirable outcomes. launchers.

15
Impact on Secure Second Strike
Most nuclear powers favor mobile missile launchers because they are
Fired from very close distances . . . , even
difficult to track and target and therefore are considered survivable. conventional munitions could become viable
These missiles move regularly via road or rail, and unless the enemy options, thereby significantly increasing the
can keep apprised of their locations at all times, the only way to
credibility of preemptive counterforce strikes.
threaten them (other than a first strike destroying the weapons before
they are deployed to the field) is by attempting to target their sizable the weapon velocities and kill radii.9 While some of these limitations
patrol areas with nuclear weapons. Even such bombardment strate- could be overcome by advances in AI over relatively short time lines,
gies are really practical only if the possible locations of the missiles others are less likely by the year 2040. For example, even with perfect
can be narrowed down at least somewhat. Cold War–era schemes to knowledge of the target location, mobile targets can move between
target Soviet mobile ICBM launchers combined bombardment strat- the time a weapon is launched and the time it arrives. Weapons
egies with intelligence about patterns in the way the Soviet Union for targeting mobile systems might be able to fly faster and adjust
moved its missiles. course better, but weapons would still need extremely sophisticated
AI could make critical contributions to ISR and analysis terminal guidance capabilities to substantially reduce the amount of
systems, upending these assumptions and making mobile missile ordnance required. As a result, even with advances in image process-
launchers vulnerable to preemption. This possibility seriously alarms ing and target recognition, many large weapons would be needed,
Russian and Chinese defense planners because those states rely or smaller ones would need to be launched from close range. The
heavily on mobile ICBMs for deterrence. Even if AI only modestly figure on page 17 shows the number of warheads of various types
improves the ability to integrate data about the disposition of enemy that would be required to destroy a mobile target with a weapon
missiles, it might substantially undermine a state’s sense of security radius of effect between 0 and 5 kilometers. Despite their huge “kill
and undermine crisis stability. At present, the requisite capabilities radius” measuring kilometers in diameter, multiple thermonuclear
for ATR, sensor integration, and signal processing remain forbid- warheads delivered by ballistic missiles would be required to have a
dingly difficult. But it appears plausible that these challenges could high assurance of destroying a missile launcher. For instance, three
fall in the uncomfortable median between working well enough to 475-kT W88 warheads delivered by Trident II missiles with a ten-
render these weapons entirely obsolete and utterly lacking credibility. minute flight time would be required to cover one target, while five
100-kT W76 warheads would be necessary to cover it. The analysis
A Difficult Technical Challenge finds, however, that accurate cruise missiles (CMs) launched from
RAND has developed a model for tracking and targeting adversary a position close to the targets (30-kT CM and 200-kT CM in the
forces that incorporates limitations in sensing, image processing, and figure) could cover the mobile missile launchers with only one or two

16
 Minimum Number of Weapons Required to Cover Target

5.0

1
2
weapon 3
4
required
4.0 W88 5
warhead
MMIII
with update 6–10 MMIII
3.0
200-kT CM
Weapon radius
of effect (km)

2.0
W76 11–20
30-kT CM warhead

1.0
ATACM
>20

JDAM
0.0
0 15 30

Weapon flight time after last update (min.)

This figure shows the number of warheads of various types that would be required to destroy a mobile target with a weapon
radius of effect between 0 and 5 kilometers. Despite their huge “kill radius” measuring kilometers in diameter, multiple thermo-
nuclear warheads delivered by ballistic missiles would be required to have a high assurance of destroying a missile launcher.
ATACM = Army Tactical Missile System; JDAM = Joint Direct Attack Munition; kT = kiloton; MMIII = Minuteman III.

warheads. Fired from very close distances (i.e., flight times of a few of strategically destabilizing threats to the survivability of mobile
minutes), even conventional munitions could become viable options, ICBM launchers but also offer some hope that arms control could
thereby significantly increasing the credibility of preemptive counter- help forestall threats. To pose a credible threat to the mobile ICBM
force strikes. launchers, the attacking forces need to be based very close to them,
These finding suggest that AI in conjunction with mobile, pos- even if the AI system that is tracking and targeting the launchers is
sibly autonomous, sensor platforms could enable the development relatively sophisticated. Even under such conditions, “windows of

17
vulnerability” during which the attack might be carried out would
last a period of only minutes, pressuring the would-be attacker to AI is making rapid progress, exhibiting
seize the opportunity if it arose. States worried about a disarming superhuman performance at increasingly
strike would be extremely alarmed by the appearance of such forces complex tasks.
around their periphery, perhaps leading them to believe that they
were in a “use it or lose it” situation. Therefore, such moves could and clearly defined. But DeepMind’s developers have been working
create a vicious cycle of distrust and actually incite a conflict that toward AI that can play the computer game Starcraft (Woyke and
neither side intended. Such undesirable outcomes might be avoided, Kim, 2017), which mirrors a military engagement complete with
however, by verifiable agreements not to base or deploy weapons that logistics, infrastructure, and a range of moves and strategies that are
might be used for such a disarming strike within a certain distance of difficult to specify. Starcraft, too, is far simpler than nuclear war, but
the mobile missile launchers. by the year 2040, it does not seem unreasonable to expect that an AI
system might be able to play aspects or stages of military wargames
Illustrative Case—AI as a Trusted Adviser or exercises at superhuman levels. Once that capability has been
In addition to potentially reducing confidence in second- demonstrated, it is likely that humans making command decisions
strike forces, AI could inadvertently compromise a state’s ability to will treat the AI system’s suggestions as on par with or better than
navigate road-to-war, escalation, and launch decisions. Autonomous those of human advisers. This potentially unjustified trust presents
control is unlikely to be implemented directly in any of the domes- new risks that must be considered.
tic launchers or command-and-control platforms, but doing so is Some workshops participants were convinced that humans
not necessary for AI to exert influence. This already happens to the would be unwilling to let the computer influence decisions about
degree that computer programs, simulations, or data analysis proce- nuclear war, while others could easily envision growing comfortable
dures are used to inform human decisions. AI is expected to become with the idea. Anecdotally, the difference in perspective was gen-
more widely used in aids to decisionmaking (commonly termed deci- erational, suggesting that those who will have inherited the reins by
sion support systems). 2040 will be more comfortable with abdicating some degree of con-
trol, especially as AI continues to prove itself in increasingly complex
Probable Roles for AI in Decisionmaking and day-to-day tasks over the coming decades. It is already common
AI is making rapid progress, exhibiting superhuman performance at for Americans to rely on AI to make routing decisions when driving,
increasingly complex tasks. Alphabet DeepMind’s AlphaGo defeat- facilitate scheduling tasks, and respond to simple e-mails. Perilously,
ing the world champion at Go astonished even AI and strategy these successes may build confidence that is unwarranted considering
experts (Etherington, 2017). To be sure, the decisionmaking in Go is the chasm between routine decisions and nuclear war.
far simpler to address than in nuclear war; the moves are sequential

18
Limits to Effectiveness Because of Training Data Attacks
Adversarial Actions Another way to subvert AI is to tamper with the training data. That
There are two main challenges in developing AI for tasks relating can be achieved in several ways: insiders replacing data, hacking to
to nuclear war. First, nuclear weapons have not been used since the switch out data, including erroneous samples in openly available
U.S. bombings of Japan in 1945 triggered unconditional surrender, data, or an adversary carefully selecting its behaviors in ways that set
and there has never been a nuclear exchange. Therefore, there is a false precedent.
complete lack of real training data. However, simulations, wargames, A range of studies have started to outline strategies for, and
and exercises might help alleviate that problem, and it should not be effects of, poisoning training data for various machine learning
forgotten that the same lack of real data limits human learning and algorithms (Anderson et al., 2017; Biggio, Nelson, and Laskov,
decisionmaking as well. 2012; Kearns and Li, 1992), but much work is left to be done and
The second distinguishing trait is that all parties involved in many more discoveries should be anticipated. Much of this work
such common pursuits as navigation or scheduling tend to have the is being led by the antivirus community, which is among the few
same incentive to complete the task successfully, whereas nuclear war other application spaces that are adversarial by nature—in recent
is inherently adversarial. There is a range of different approaches to years, this community has turned to machine learning as opposed
subverting AI systems, and it appears that subversion is likely to be to more-traditional, signature-based methods. Some have sought
an effective option for a long time to come. We will briefly discuss ways to ensure that machine learning remains effective despite data
hacking, training data attacks, and input manipulation as illustra- manipulation attacks, but those efforts remain nascent (Kegelmeyer
tions of the types of concerns that exist. et al., 2015). Data tampering is expected to be a threat for a long time
to come.
Hacking
Hacking is not specific to AI, but as long as AI involves computers, it Input Manipulation
must be considered vulnerable to hacking. The intelligence itself can A third opportunity to subvert AI comes after it is fully trained.
be hacked, as will be described later, but data might also be altered Manipulating the inputs in subtle ways can lead even a high-
at the inputs, outputs, or en route from the output to the display, for performing AI system to come to any of its possible conclusions
example. Of course, any AI that played a role in the nuclear enter- that the attacker prefers. This has been demonstrated for image
prise would be carefully protected, but it would also be a high-value recognition, where changes so small that they are undetectable
target. to humans have been made to an image and caused the AI to
classify the altered image as a category of the attacker’s choosing
(Karpathy, 2015). This may be more difficult in nuclear matters,
where a human may not have precise knowledge of all the inputs or

19
possible classifications. In the image-recognition case, the adversary
range of inputs was simple, restricted to pixels. For other tasks, Progress in AI appears inexorable, with firms
adversaries may need to mobilize forces in a specific pattern or release and governments rushing to employ it for
statements with a specific message in a specific sequence, but it is still an ever-widening range of applications,
possible—at least in principle—to “trick” fully trained AI systems.
including both offensive and defensive uses.
Importantly, input manipulation attacks do not require the adversary
to have access to the trained system, so even a well-protected AI
should have to go through rigorous testing that would include
can still be vulnerable (Papernot, McDaniel, and Goodfellow,
adversarial approaches. The simulation of adversaries in testing is
2016). More research will be needed to understand the extent of
fully effective only if the tester can envision the full range of attacks
vulnerabilities and to understand which parts of the inputs, outputs,
an adversary might create. This impossibly tall order is nonetheless
and data would need to be kept secure.
faced for all military systems that are deployed.

Impact of Limited Effectiveness on Nuclear Security

Some Possible Stability-Enhancing Effects of AI
In the previous case of tracking and targeting, AI threats relate to
Given the many decades that have passed without nuclear
undermining strategic stability because of the adversary’s exagger-
attack, it is easy to take strategic stability for granted. While the
ated faith in its effectiveness, but the opposite could also be the
previous sections have outlined ways in which AI progress could
case. In the case of decision support systems, it is more perni-
undermine strategic stability, this need not be the case. Progress
cious for the force employing the AI to believe that it is effective
in AI appears inexorable, with firms and governments rushing to
when it is not. It is also possible that an adversary could become
employ it for an ever-widening range of applications, including
convinced that it is able to subvert an AI and avoid retaliation,
both offensive and defensive uses. The effects of AI on these
leading it to pursue paths that would otherwise be escalatory in
strategic applications will become apparent only with time. AI has
nature, up to and including preemptive first strike. For example,
the potential to exacerbate the tensions among different aspects
the adversary might be convinced that it has discovered a pattern
of nuclear strategy, but it might, under favorable circumstances,
of launches and trajectories that would lead the AI to view the
alleviate these tensions and enhance strategic stability instead.
data and conclude that such a pattern is safe even as missiles are
Despite their mutual distrust, nuclear states may be motivated by
en route to targets.
self-interest to coordinate toward this end.
AI presents an array of new vulnerabilities that are difficult
to detect in real time. Yet it will almost certainly—eventually or
gradually—be given more prominence in road-to-war, escalation,
and even launch decisions. Any system with those responsibilities

20
The Periods Beyond High Fallibility to risk provocative actions from which uncertainty might otherwise
Workshop participants agreed that the riskiest periods will occur dissuade it.
immediately after AI enables a new capability, such as tracking and Improved accuracy in intelligence collection and analysis could also
targeting or decision support about escalation. During this break-in reinforce strategic stability by making deterrence, assurance, and
period, errors and misunderstandings are relatively likely. With time reassurance more credible. If potential adversaries had less oppor-
and increased technological progress, those risks would be expected tunity to prepare an attack in secret, threats to use force against
to diminish. If the main enabling capabilities are developed during oneself or one’s allies would be less plausible. If strategic partners
peacetime, then it may be reasonable to expect progress to continue had access to more-comprehensive intelligence and analysis, they
beyond the point at which they could be initially fielded, allowing could be assured more easily. With smaller forces needed for assur-
time for them to increase in reliability or for their limitations to ance, a nuclear power such as the United States could reduce the
become well understood. Eventually, the AI system would develop size of its nuclear arsenal, which would enhance reassurance of the
capabilities that, while fallible, would be less error-prone than their enemy. This process could develop into a virtuous cycle, ultimately
human alternatives and therefore be stabilizing in the long term. greatly reducing the risk of war. Unfortunately, this outcome would
require fortuitous conditions to materialize, irrespective of the state
Potential Cooperation for Strategic Stability of AI technology. First, all actors would require equivalent access to
One of the factors that perpetuates the risk of nuclear war is intelligence and analysis capabilities. The weaker state in an emerging
the contradiction between the first-strike stability requirement intelligence asymmetry would probably consider itself unacceptably
of assured retaliation, which encourages governments to adopt vulnerable and deepen its suspicions of the adversary. Furthermore,
“launch under attack” postures, and the possibility of an accident the intentions of rival states would need to be genuinely benign.
or malfunction. For instance, a 1983 malfunction of the Soviet Finally, officials’ confidence in the intelligence collection and analysis
Union’s early warning systems led to the “detection” of a nonexistent system (including non-AI components) needs to be well justified.
U.S. attack (Hoffman, 2009, pp. 6–11). Particularly during a crisis To actualize the potential of AI to bolster strategic stability, states
situation, such an incident might lead officials to order a retaliatory need to begin coordinating as the technology matures to avoid these
strike in response to a phantom assault. AI could help alleviate pitfalls. These discussions should include diplomatic and military
this contradiction by enabling the creation of more-reliable early officials, as well as technology experts.
warning systems. Greater first-strike stability should, in turn, help
reduce the danger of accidental escalation in crises. Even so, this Radical Transparency
kind of confidence could be a mixed blessing. An aggressor state In one highly optimistic possibility, an AI algorithm being used
believing in its ability to predict escalation might feel emboldened to provide support for decisions about escalation could be shared
with the adversary. Such radical transparency would come with

21
many risks. The adversary might then be able to pursue undesirable Some experts fear that an increased reliance on AI could lead to
actions up to the very edge of the escalation threshold. It might also new types of catastrophic mistakes. There may be pressure to use it
try to subvert the AI. At the same time, any AI that would be used before it is technologically mature; it may be susceptible to adversar-
as an aid in such decisionmaking should be required to undergo ial subversion; or adversaries may believe that the AI is more capable
extensive testing, including of an adversarial nature. It is good than it is, leading them to make catastrophic mistakes.
practice in any case to attempt to design AI in such a way that it On the other hand, if the nuclear powers manage to establish a
would remain secure even if an enemy were to get the algorithm; it is form of strategic stability compatible with the emerging capabilities
dangerous to presume that an adversary would be unable to obtain that AI might provide, the machines could reduce distrust and allevi-
it (Kerckhoff, 1883). If the AI computer system must meet that high ate international tensions, thereby decreasing the risk of nuclear war.
standard of robustness prior to fielding, disseminating it widely At present, we cannot predict which—if any—of these scenarios
might alleviate fears and make miscalculations nearly impossible. will come to pass, but we need to begin considering the potential
impact of AI on nuclear security before these challenges become
Conclusions acute. Maintaining strategic stability in the coming decades may
Overall workshop participants agreed that AI has significant poten- prove extremely difficult, and all nuclear powers will have to partici-
tial to upset the foundations of nuclear stability and undermine pate in the cultivation of institutions to help limit nuclear risk. This
deterrence by the year 2040, especially in the increasingly multipolar goal will demand a fortuitous combination of technological, military,
strategic environment. Dismissing the Hollywood nightmare of and diplomatic measures that will require rival states to cooperate.
malevolent AIs trying to destroy humanity with nuclear weapons, We hope that this Perspective will begin that discussion and open a
experts were instead concerned with more-mundane issues arising path toward pragmatism and realism on these controversial and often
from improving capabilities. AI applications discussed included polarizing topics.
the ability to track and target adversary launchers for counterforce
targeting and the incorporation of AI into decision support systems
informing choices about the use of nuclear weapons.

22
Notes (2008). Vladimir Putin continues to echo this language—for instance, in his 2012
declaration that “Russia’s military-technical response to American global antimissile
defense and its component in Europe will be effective and asymmetric” (Putin,
1
In this Perspective, we employ the term artificial intelligence in an informal sense 2012).
that includes many computer science achievements in research programs broadly 5
The 2010 Nuclear Posture Review states that the goal of U.S. nuclear strategy
associated with AI, even though these accomplishments ultimately had little to
is to “strengthen deterrence of regional adversaries,” such as North Korea, while
do with emulating human intelligence per se. Such programs resulted in pattern-
“reinforcing strategic stability” with Russia and China. The report does not provide
recognition algorithms, new programming languages, natural-language processing,
a concise definition of “strategic stability,” however (U.S. Department of Defense,
and a host of other functions that were referred to as AI in previous decades but have
2010).
long since entered the mainstream of computing.
6
The United States and Soviet Union each sought to develop a preemptive strike
2
Russian press accounts attest that the Status-6 employs iskusstvennyi intellekt (AI)
capability that could be used to disarm the other if an attack appeared to be
to achieve its autonomous capabilities. For instance, see Tuchkov (2016) and “Ros-
imminent, but this should be distinguished from a first-strike capability designed
siiskii proekt ‘Status-6’ meniaet sootnosheniia iadernykh sil v mire” (2016). The
to mount a “bolt from the blue” attack. The practical difficulty of distinguishing
latter article asserts that Status-6, “being equipped with artificial intelligence,” could
strategic forces intended for a preemptive attack from those intended for a first strike
circumvent antisubmarine warfare measures by following “otherwise unreachable
led officials in both superpowers to fear that the other side might be preparing to
routes” to attack the enemy “where he least expects it.”
start a nuclear war.
3
In keeping with common parlance, we are colloquially including both future and 7
The assertion that Perimetr employs some kind of iskusstvennyi intellekt (AI) has
recent advances within the definition of AI, but not those with a long history of
appeared repeatedly in Russian state media. For example, see Timoshenko (2015)
application, even if the tasks required humans at some point in the past.
and Valagin (2014).
4
During the final years of the Cold War, the Soviet Union elected to counter 8
For instance, see Akhmerov, Akhmerov, and Valeev (2016).
prospective U.S. missile defenses by developing missile technologies designed
to defeat them. For a Russian account of the Soviet “asymmetric response” to 9
Unpublished RAND research by Brien Alkire and Jim Powers.
Ronald Reagan’s Strategic Defense Initiative, see Oznobishev, Potapov, and Skokov

23
References Healey, D., The Time of My Life, London: Michael Joseph, 1989, p. 243.

Hoffman, D. E., The Dead Hand, New York: Doubleday, 2009.

Akhmerov, D. E., E. N. Akhmerov, and M. G. Valeev, “‘Uiazvimost’ kontseptsii Holmes, J. R., “Sea Changes: The Future of Nuclear Deterrence,” Bulletin of the
neiadernogo razoruzheniia strategicheskikh iaderniyi sil Rossii” [“The Dubiousness Atomic Scientists, Vol. 72, No. 4, 2016, pp. 228–233.
of the Concept of a Non-Nuclear Disarming Strike Against Russia’s Strategic
Kahn, H., On Thermonuclear War, Princeton, N.J.: Princeton University Press,
Nuclear Forces”], Vestnik akademii voennykh nauk, Vol. 54, No. 1, 2016, pp. 37–41.
1960.
Anderson, H. S., A. Kharkar, B. Filar, and P. Roth, Evading Machine Learning
Karpathy, A., “Breaking Linear Classifiers on ImageNet,” Andrej Karpathy blog,
Malware Detection, [Link], July 2017. As of August 15, 2017:
March 30, 2015. As of August 15, 2017:
[Link]
[Link]
[Link]
Kearns, M., and M. Li, “Learning in the Presence of Malicious Errors,” SIAM
Biggio, B., B. Nelson, and P. Laskov, “Poisoning Attacks Against Support Vector
Journal on Computing, Vol. 22, No. 4, March 1992, pp. 807–837. As of
Machines,” Proceedings of the 29th International Conference on Machine Learning,
August 15, 2017:
July 2012, pp. 1467–1474. As of August 15, 2017:
[Link]
[Link]
Kegelmeyer, P., T. M. Shead, J. Crussell, K. Rodhouse, D. Robinson, C. Johnson,
Bostrom, Nick, Superintelligence: Paths, Dangers, Strategies, Oxford: Oxford
D. Zage, W. Davis, J. Wendt, J. Doak, T. Cayton, R. Colbaugh, K. Glass, B. Jones,
University Press, 2014.
and J. Shelburg, Counter Adversarial Data Analytics, Albuquerque, N.M.: Sandia
Bracken, P., “The Intersection of Cyber and Nuclear War,” The Strategy Bridge, blog National Laboratories, SAND2015-3711, May 2015. As of August 15, 2017:
post, January 17, 2017. As of August 15, 2017: [Link]
[Link]
Kerckhoff, A., “La Cryptographie Militaire,” Journal des Sciences Militaires,
and-nuclear-war
January 1883.
Buchan, G., D. Matonick, C. Shipbaugh, and R. Mesic, Future Roles of U.S. Nuclear
Long, A., Deterrence from Cold War to Long War: Lessons from Six Decades of RAND
Forces: Implications for U.S. Strategy, Santa Monica, Calif.: RAND Corporation,
Research, Santa Monica, Calif.: RAND Corporation, MG-636-OSD/AF, 2008. As
MR-1231-AF, 2003. As of March 8, 2018:
of March 8, 2018:
[Link]
[Link]
Cimbala, S. J., The Dead Volcano: The Background and Effects of Nuclear War
Long, A., and B. R. Green, “Stalking the Secure Second Strike: Intelligence,
Complacency, Westport, Conn.: Praeger, 2002.
Counterforce, and Nuclear Strategy,” Journal of Strategic Studies, Vol. 38,
Etherington, D., “Google’s AlphaGo AI Beats the World’s Best Human Go Player,” Nos. 1–2, August 2012, pp. 38–76.
TechCrunch, May 23, 2017. As of August 15, 2017:
Mack, C. A., “Fifty Years of Moore’s Law,” IEEE Transaction on Semiconductor
[Link]
Manufacturing, Vol. 24, No. 2, May 2011, pp. 202–207. As of August 15, 2017:
best-human-go-player/
[Link]
Garthoff, R. L., “Refocusing the SDI Debate,” Bulletin of the Atomic Scientists,
Mahan, A. T., Armaments and Arbitration: Or, The Place of Force in the International
Vol. 43, No. 7, September 1987.
Relations of States, New York: Harper & Brothers, 1912.
Geist, E., “Was There a Real ‘Mineshaft Gap’? Bomb Shelters in the USSR,
Minsky, M., Semantic Information Processing, Cambridge, Mass.: MIT Press, 1968.
1945–62,” Journal of Cold War Studies, Vol. 14, No. 2, Spring 2012, pp. 3–28.
Newell, A., J. C. Shaw, and H. A. Simon, Report on a General Problem-Solving
Goodfellow, I., Y. Bengio, and A. Courville, Deep Learning, Cambridge, Mass.:
Program, Santa Monica, Calif., RAND Corporation, Report P-1584, revised
MIT Press, 2016.
February 9, 1959.

24
Oznobishev, S. K., V. Ia. Potapov, and V. V. Skokov, Kak gotovilisia “asymmetrichnyi Slocombe, W., “The Countervailing Strategy,” International Security, Vol. 5, No. 4,
otvet” na “Strategicheskuiu oboronnyiu initsiativu” [Link]. Velikhov, Kokoshin i Spring 1981, pp. 18–27.
drugie [How the “Asymmetric Response” to R. Reagan’s “Strategic Defense Initiative”
Sutyagin, I., “Russia’s Underwater “Doomsday Drone”: Science Fiction, but Real
Was Prepared], Moscow: Legand, 2008.
Danger,” Bulletin of the Atomic Scientists, Vol. 72, No. 4, June 2016,
Papernot, N., P. McDaniel, and I. Goodfellow, “Transferability in Machine pp. 243–246.
Learning: from Phenomena to Black-Box Attacks Using Adversarial Samples,”
Timoshenko, M., ‘Mertvaia ruka’ na strazhe perimetra Rossii” [“The ‘Dead Hand’
arXiv, May 24, 2016. As of August 15, 2017:
Guarding Russia’s Periphery”], Telekanal “Zvezda,” February 18, 2015. As of
[Link]
August 15, 2017:
Putin, V., “Byt’ sil’nymi: garantii natsional’noi bezopasnosti dlia Rossii” [“Being [Link]
Strong Is the Guarantee of National Security for Russia”], Rossiiskaia gazeta,
Tuchkov, V., Status-6: Oruzhie Vosmezdia, Vognavshee Pentagon v Stupor [Status-6:
February 20, 2012. As of December 5, 2017:
The Retaliatory Weapon That Drove the Pentagon into a Stupor], Svobodnaia Pressa,
[Link]
December 11, 2016. As of December 4, 2016:
Roland, A., and P. Shiman, Strategic Computing: DARPA and the Quest for Machine [Link]
Intelligence, 1983–1993, Cambridge, Mass.: MIT Press, 2002.
U.S. Department of Defense, Nuclear Posture Review Report, Washington, D.C.,
“Rossiiskii Proekt ‘Status-6’ Meniaet Sootnosheniia Iadernykh Sil v Mire” [“The April 2010.
Russian Status-6 Project Is Changing the World’s Nuclear Balance of Forces”],
Valagin, A., “Garantirovannoe vozmezdie: Kak rabotaet rossiiskaia sistema
Russkaia Politika, November 14, 2016. As of December 4, 2016:
‘Perimetr’” [“Assured Retaliation: How the Russian ‘Perimetr’ System Works”],
[Link]
Rossiiskaia gazeta, January 22, 2014. As of August 15, 2017:
rossiyskiy-proekt-status-6-menyaet-​sootnoshenie-​yadernyih-sil-v-mire/
[Link]
Schelling, T., Arms and Influence, New Haven, Conn.: Yale University Press, 1966.
Woyke, E., and Y. Kim, “Starcraft Pros Are Ready to Battle AI,” MIT Technology
Shannon, C. E. “Programming a Computer for Playing Chess,” Philosophical Review, May 19, 2017. As of August 15, 2017:
Magazine, Vol. 41, No. 7, 1950, pp. 256–275. [Link]

Simon, H. A., and A. Newell, “Heuristic Problem Solving: The Next Advance in
Operations Research,” Operations Research, Vol. 6, No. 1, January–February 1958,
pp. 1–10.

25
About This Perspective RAND Ventures
RAND is a research organization that develops solutions to public
We would like to thank RAND’s Center for Global Risk and Secu-
policy challenges to help make communities throughout the world
rity and its director, Andrew Parasiliti, for initiating this effort, and
safer and more secure, healthier and more prosperous. RAND is non-
RAND Ventures for sponsoring it. We also thank Angela O’Mahoney
profit, nonpartisan, and committed to the public interest.
and Bill Welser for their guidance throughout. We also indebted to
RAND Ventures is a vehicle for investing in policy solutions.
Sonni Efron, Doug Irving, and Greg Baumann for their help in find-
Philanthropic contributions support our ability to take the long view,
ing the stories as they emerged and Hosay Yaqub for making the events
tackle tough and often-controversial topics, and share our findings
themselves a success. Finally, we would like to thank the workshop
in innovative and compelling ways. RAND’s research findings and
participants we leave unnamed in accordance with the Chatham
recommendations are based on data and evidence, and therefore do
House Rule.
not necessarily reflect the policy preferences or interests of its clients,
Security 2040 donors, or supporters.
This Perspective is part of a broader effort, an initiative of RAND Funding for this venture was provided by gifts from RAND sup-
Ventures, to envision critical security challenges in the world of 2040, porters and income from operations.
considering the effects of political, technological, social, and demo-
About the Authors
graphic trends that will shape those security challenges in the coming
decades. The research was conducted within the RAND Center for Edward Geist is an associate policy researcher at RAND. Previously
Global Risk and Security. a MacArthur Nuclear Security fellow at Stanford University’s Center
for International Security and Cooperation (CISAC) and a Stanton
RAND Center for Global Risk and Security
Nuclear Security Fellow in RAND’s Washington office, Edward
The Center for Global Risk and Security (CGRS) works across the
received his doctorate in Russian history from the University of
RAND Corporation to develop multidisciplinary research and policy
North Carolina in May 2013.
analysis dealing with systemic risks to global security. The center
draws on RAND’s expertise to complement and expand RAND Andrew J. Lohn is an engineer at the RAND Corporation. He
research in many fields, including security, economics, health, and applies a wide range of mathematical and machine learning techniques
technology. A board of distinguished business leaders, philanthro- to provide new insights into highly technical policy issues, such as
pists, and former policymakers advises and supports the center activi- cyberwarfare, artificial intelligence, or drone delivery. Lohn holds
ties, which are increasingly focused on global security trends and a doctorate in electrical engineering from the University of California,
the impact of disruptive technologies on risk and security. For more Santa Cruz.
information about the RAND Center for Global Risk and Security,
visit [Link]/international/cgrs

26