Journal of the American Statistical Association

ISSN: (Print) (Online) Journal homepage: [Link]/journals/uasa20

Adversarial Machine Learning: Bayesian Perspectives

David Rios Insua, Roi Naveiro, Víctor Gallego & Jason Poulos

To cite this article: David Rios Insua, Roi Naveiro, Víctor Gallego & Jason Poulos (2023)
Adversarial Machine Learning: Bayesian Perspectives, Journal of the American Statistical
Association, 118:543, 2195-2206, DOI: 10.1080/01621459.2023.2183129

To link to this article: [Link]

© 2023 The Author(s). Published with

license by Taylor & Francis Group, LLC.

Published online: 31 Mar 2023.

Submit your article to this journal

Article views: 5774

View related articles

View Crossmark data

Citing articles: 8 View citing articles

Full Terms & Conditions of access and use can be found at

[Link]
JOURNAL OF THE AMERICAN STATISTICAL ASSOCIATION
2023, VOL. 118, NO. 543, 2195–2206: Review
[Link]

Adversarial Machine Learning: Bayesian Perspectives

David Rios Insua∗ a , Roi Naveiro∗ a,b
, Víctor Gallego∗ a,c , and Jason Poulosd
a
ICMAT-CSIC, Madrid, Spain; b CUNEF University, Madrid, Spain; c Komorebi AI Technologies, Madrid, Spain; d Harvard Medical School, Boston, MA

ABSTRACT ARTICLE HISTORY

Adversarial Machine Learning (AML) is emerging as a major field aimed at protecting Machine Learning Received May 2020
(ML) systems against security threats: in certain scenarios there may be adversaries that actively manipulate Accepted February 2023
input data to fool learning systems. This creates a new class of security vulnerabilities that ML systems
may face, and a new desirable property called adversarial robustness essential to trust operations based KEYWORDS
on ML outputs. Most work in AML is built upon a game-theoretic modeling of the conflict between a Adversarial risk analysis;
Bayesian methods;
learning system and an adversary, ready to manipulate input data. This assumes that each agent knows Cybersecurity; Machine
their opponent’s interests and uncertainty judgments, facilitating inferences based on Nash equilibria. learning; Security
However, such common knowledge assumption is not realistic in the security scenarios typical of AML.
After reviewing such game-theoretic approaches, we discuss the benefits that Bayesian perspectives provide
when defending ML-based systems. We demonstrate how the Bayesian approach allows us to explicitly
model our uncertainty about the opponent’s beliefs and interests, relaxing unrealistic assumptions, and
providing more robust inferences. We illustrate this approach in supervised learning settings, and identify
relevant future research problems. Supplementary materials for this article are available online.

1. Introduction hypothesis, given the presence of adaptive adversaries ready to

intervene to modify the data and obtain a benefit. Stemming
Over the last decade, an increasing number of processes have
from the pioneering work in adversarial classification (Dalvi
been automated through Machine Learning (ML) algorithms,
et al. 2004), the prevailing paradigm in AML has modeled the
making it more crucial that these algorithms become robust
confrontation between learning-based systems and adversaries
and reliable if we are to trust operations based on their out-
through game theory (Menache and Ozdaglar 2011). This entails
put. State-of-the-art ML methods perform extraordinarily well
common knowledge (CK) assumptions (Hargreaves-Heap and
on standard data, but have been shown to be vulnerable to
Varoufakis 2004), which are questionable in the security domain
adversarial examples (Goodfellow, Shlens, and Szegedy 2015),
as adversaries try to hide and conceal information. Thus, there
data instances targeted at fooling them. The presence of adver-
is a need for developing a better founded paradigm: as Fan,
saries has been highlighted in areas such as spam detection
Ma, and Zhong (2021) point out, a framework that guarantees
(Zeager et al. 2017), computer vision (Goodfellow, Shlens, and
robustness of ML against adversarial manipulations in a princi-
Szegedy 2015) and automated driving systems (ADS, Caballero,
pled manner is required.
Ríos Insua, and Banks 2021). In those contexts, algorithms
After providing an overview of key concepts and methods
should acknowledge the presence of possible adversaries to
in AML emphasizing the underlying game theoretical assump-
protect from their eventual data manipulations. Comiter (2019)
tions, we suggest an alternative formal Bayesian decision the-
provides a review from a policy perspective showing how many
oretical framework based on Adversarial Risk Analysis (ARA,
AI systems, including content filters, and military and law
Rios Insua, Rios, and Banks 2009) and illustrate it in supervised
enforcement systems, are vulnerable to attacks. As a motivating
learning settings. We end by suggesting a research agenda.
example, consider fraud detection: as ML algorithms are incor-
porated to such a task, fraudsters learn how to evade them. For 2. Motivating Examples
instance, they could find out that making a huge transaction
increases the probability of being detected, and instead would Two examples serve us to motivate key issues in AML. They
issue smaller transactions. showcase how the performance of ML systems may considerably
As a fundamental assumption, ML systems rely on using degrade under subtle data manipulations, suggesting the need to
iid data for both training and operations (Zhang et al. 2021). take into account the presence of adversaries.
However, the security aspects of ML, part of the emerging Case 1. Attacking spam detection algorithms. Consider spam
field of Adversarial Machine Learning (AML), challenge such detection, an example of content filters which are at the

CONTACT Roi Naveiro [Link]@[Link] CUNEF University, Madrid, Spain.

∗ These authors contributed equally to this work.
Supplementary materials for this article are available online. Please go to [Link]/r/JASA.
© 2023 The Author(s). Published with license by Taylor & Francis Group, LLC.
This is an Open Access article distributed under the terms of the Creative Commons Attribution-NonCommercial-NoDerivatives License ([Link] which
permits non-commercial re-use, distribution, and reproduction in any medium, provided the original work is properly cited, and is not altered, transformed, or built upon in any way. The terms on
which this article has been published allow the posting of the Accepted Manuscript in a repository by the author(s) or with their consent.
2196 D. R. INSUA ET AL.

Table 1. Accuracy (with precision) of four algorithms on clean data (untainted); attacked data and unprotected; protected through ARA during operations; protected
through ARA during training.

Algorithm Untainted Unprotected ARA op. ARA tr.

Naive Bayes 0.882 ± 0.004 0.754 ± 0.027 0.939 ± 0.006 —
Logistic Regression 0.932 ± 0.004 0.673 ± 0.005 0.898 ± 0.008 0.946 ± 0.003
Neural Network 0.904 ± 0.029 0.607 ± 0.009 0.882 ± 0.025 0.960 ± 0.002
Random Forest 0.912 ± 0.005 0.731 ± 0.008 0.807 ± 0.007 —

This is debatable in most security settings and is a driving force

in this article. 

3. Adversarial Machine Learning: A Review

We review the main results and concepts in AML. We focus
on key ideas in the brief history of the field, motivating a
reflection that will lead to our alternative Bayesian framework
Figure 1. Original input and attacked version. in Section 4. Further perspectives may be seen in Vorobeychik
and Kantarcioglu (2018), Joseph et al. (2019), Biggio and Roli
backbone of many security systems. We study the performance (2018), and Dasgupta and Collins (2019). We describe first the
degradation of different algorithms under the Good-Words- usual workflow in AML within which most previous research
Insertion attacks described in Naveiro et al. (2019a): the adver- can be embedded. In general, we refer to the learning system
sary attacks spam E-mails by inserting at most two good words1 as defender (D, she), and to the adversary manipulating data as
into them. Table 1 presents the accuracy of four standard algo- attacker (A, he).
rithms — naive Bayes, logistic regression, neural network (NN)
and random forest — when facing clean and attacked data,2 using
3.1. An Adversarial Machine Learning workflow
the Spambase Data Set from the UCI ML repository (Dua and
Graff 2017). Accuracy means and standard deviations are esti- Guaranteeing protection of a learning system against attacks
mated via repeated hold-out validation over 10 repetitions (Kim involves undertaking various activities related to security eval-
2009). Observe, columns 2 and 3, the important loss in accuracy uation, threat modeling, and attack simulation. Let us frame
of the four algorithms: major performance degradation may these activities within a workflow for AML with three steps, as
affect them when ignoring the possible presence of adversaries. in Biggio and Roli (2018): (a) gathering intelligence to study
Columns 4 and 5 are discussed in Section 5. the likely attacks that a system may face; (b) forecasting likely
attacks; and (c), protecting learning systems from such attacks.
Case 2. Attacking vision algorithms. Computer vision algorithms
are at the core of many AI applications such as perception 1. Gathering intelligence. This activity is critical to ensure ML
systems in ADS (Caballero, Ríos Insua, and Banks 2021). The security in adversarial environments. Obviously, any algorithm
simplest and most notorious attacks targeting such algorithms could be fooled if adversarial data modifications are not some-
consist of modifications of images so that the alteration becomes how restricted: in an extreme case, if an instance in a binary
irrelevant to the human eye, yet drives a model trained on classification problem is modified so that it is indistinguishable
millions of images to misclassify the attacked ones. This attack from an instance of the other class, clearly, the algorithm would
entails potentially relevant security consequences. As an exam- misclassify such instance. However, the adversary is probably
ple, with a relatively simple convolutional NN (CNN, Goodfel- not interested in making such data modifications as the new
low et al. 2016), we achieve 99% test set accuracy predicting the instance might lose its malicious purposes. Thus, an in-depth
handwritten digits in the MNIST dataset (LeCun, Cortes, and study of likely attackers is key in AML. In general, we should
Burges 1998). However, accuracy reduces to 59% if we attack gather information about three attacker features.
those data with the fast gradient sign method (FGSM, Szegedy First, we assess their goals, which may range from appropri-
et al. 2014). Figure 1 provides examples of an original MNIST ating funds to causing harm to people or organizations (Couce-
image and an attacked one. Our CNN correctly classifies the Vieira, Insua, and Kosgodagan 2020). Prior to deploying a ML
original image (left) as a 4; however, it misclassifies the attacked system, it is crucial to guarantee robustness against attackers
one (right) as a 9. FGSM and related attacks described below are with the most common goals. For instance, in fraud detection,
easily built through low cost computational methods. In certain the attacker usually obfuscates fraudulent transactions to make
settings, they require the attacker to have precise knowledge the system classify them as legitimate in search of an economic
about the architecture of the corresponding predictive model. benefit: a fraud detection system should be robust against such
attacks, trying to minimize economic losses. In general, attacker
1
Words that are common in legitimate e-mail but rare in spam. goals are classified along two dimensions. For the first one,
2
The logistic regression is applied with L1 regularization, which is equivalent violation type, a usual distinction is between integrity (aimed
to performing maximum a posteriori estimation in a logistic regression
model with a Laplace prior (Park and Casella 2008). The NN has two hidden at moving the prediction about particular instances toward
layers. the attacker’s target, for example, to have malicious samples
 JOURNAL OF THE AMERICAN STATISTICAL ASSOCIATION 2197

misclassified as legitimate); availability (aimed at increasing function assesses attack effectiveness, taking into account the
the predictive error to make the system unusable) and privacy assumptions about the attacker features. The constraints frame
(exploratory attacks to gather information about the ML system) assumptions such as the adversary wanting to avoid detection
violations. The second dimension refers to attack specificity, or having available a maximum attacking budget. However,
where the usual distinction is between targeted attacks, address- full knowledge assumptions are generally unrealistic in the
ing a few specific defenders, and indiscriminate attacks, affecting AML realm as adversaries try to conceal information: adversary
many defenders in a random manner (Rios Insua et al. 2021). modeling must take into account the lack of information and
Second, we assess the knowledge that the adversary could have corresponding uncertainty that we have about the adversary.
about the ML system. At one end of the spectrum, we find white Beyond the aleatoric and epistemic uncertainties typical in risk
box or perfect knowledge attacks: the adversary knows every analysis, in AML, analysts need to consider as well concept uncer-
aspect of the system. This is almost never the case in security tainty (Banks et al. 2020). Thus, given some data, we associate
scenarios, except perhaps for insider attacks (Joshi, Aliaga, and an attacking model with a probability distribution over attacks
Insua 2021). Yet, they could be useful in sequential settings which encodes our uncertainty about how the adversary will act
where the ML system moves first, training an algorithm to fit when seeing a particular instance.5
its parameters; and the adversary, who moves afterwards, has
3. Protecting ML algorithms. Once relevant adversarial models
time to observe the behavior of the system and learn about it.3
have been produced, the last step consists of protecting learning
At the other end, black box or zero knowledge attacks assume that
systems against the modeled attacks. Broadly speaking, two
the adversary has capabilities to query the system but does not
types of defenses have been proposed. Reactive defenses aim
have any information about the data, feature space, or specific
to mitigate, even eliminate, the effects of an eventual attack.
algorithms used. This is the most reasonable assumption when
They include timely detection of attacks (e.g., Naveiro et al.
attacking and defending decisions are made simultaneously. In
2019b); frequent retraining of learning algorithms; or verifi-
between, attacks are called gray box or limited knowledge, the
cation of algorithmic decisions by experts. The second type,
most common type of attacks in security settings, especially
proactive defenses, aim to prevent attack execution. They can
when attacking and defending decisions are made sequentially
entail security-by-design approaches such as explicitly account-
but there is private information that agents are not willing to
ing for adversarial manipulations (Naveiro et al. 2019a) or pro-
share.
ducing provably secure algorithms against perturbations (Gowal
Finally, the third feature refers to the adversary’s capabilities
et al. 2018); or security-by-obscurity techniques such as ran-
to influence on data and other features. With poisoning attacks,
domization of algorithm responses, or gradient obfuscation to
he may obfuscate training data to later induce errors during
make attacks less likely to succeed (Athalye, Carlini, and Wag-
operations. Alternatively, evasion attacks have no influence on
ner 2018). A more interesting classification of defenses, later
training data, but perform modifications during operations, for
emphasized, refers to whether the protection is carried out
instance when trying to evade a detection system. These data
at training or at operation time. Defenses of the former class,
crafting activities are typical in AML and designated to come
train learning systems robustly, anticipating future adversarial
from a so-called data-fiddler. There could be as well attackers
attacks. Defenses of the latter class, when receiving a potentially
capable of changing the underlying structure of the problem
attacked instance, undertake inference about possible originat-
affecting process parameters, called structural attackers. More-
ing instances to make the corresponding decision.
over, some adversaries could be making decisions in parallel to
those of the defender with the agents’ losses depending on both
decisions, which we term parallel attackers.4 3.2. Core Concepts in Adversarial Machine Learning
2. Forecasting likely attacks. In our path to enhance protection, Most work in AML has dealt with supervised learning fac-
once having gathered intelligence about the potential attacks ing adversarial threats, with a focus on either proposing new
to a learning system, we should produce models for how the attacks to learning systems to showcase their vulnerabilities
adversary may behave when facing new data. A central argument (thus, related to Steps 1–2 of the above workflow) or proposing
for us is that such models must take into account our uncer- defenses to protect algorithms from common attacks (Step 3).
tainty about adversarial aspects. As mentioned, most previous
research along these lines has been usually based on game theory Attacks to learning systems. The most common goal of pro-
assuming full knowledge about the adversary. Thus, given some posed attacks is to modify instance covariates to induce the
data, the adversary would behave deterministically: the stan- learning system into making wrong decisions upon observing
dard approach forecasts attacks solving constrained optimiza- or analyzing such contaminated covariates. One of the most
tion problems with different assumptions about the adversary’s influential concepts triggering the current interest in AML are
knowledge, goals and capabilities. The corresponding objective adversarial examples. They are introduced within NN models
as perturbed data instances aimed at fooling NNs, obtained
3
through solving certain optimization problems (Szegedy et al.
However, although the adversary may have some knowledge, assuming that
this knowledge is perfect is not realistic and has been criticized (Dalvi et al. 2014). These models are highly sensitive to such examples; recall
2004). case 2 in Section 2.
4
Some attackers could combine the three capabilities in certain scenarios.
For example, in cybersecurity an attacker might add spam modifying its
proportion (structural); alter some spam messages (data-fiddler); and, in
5
addition, undertake his own business decisions (parallel). See Ríos Insua Previous attacking models can be easily recovered in this framework assum-
et al. (2018). ing degenerate distributions as later illustrated.
2198 D. R. INSUA ET AL.

Adversarial examples have traditionally targeted computer Subsequent AC approaches, reviewed in Biggio, Fumera, and
vision systems, using techniques such as FGSM or projected Roli (2014), have focused on analyzing attacks over algorithms
gradient descent (PGD, Madry et al. 2018). These techniques and upgrading their robustness against them, always making
find a constrained perturbation of an image that maximizes strong assumptions about the adversary. For instance, Lowd
the loss function used to train the computer vision system. and Meek (2005) consider that the attacker can send mem-
This optimization is usually approximated using gradient ascent bership queries to the classifier to issue optimal attacks. Other
routines. In addition, attacks like these have been extended to approaches have focused on improving Dalvi et al.’s model but,
target other systems such as natural language processing (Zhang as far as we know, none have disposed of the unrealistic CK
et al. 2020b), due to their increasing relevance. Attacking strate- assumptions.
gies targeting tabular data are usually application specific. Most Protection during training.. An important family of AML
approaches model the confrontation between the attacker and defenses try to robustify learning systems by modifying the
the learning system as a game (Brückner, Kanzow, and Scheffer way training is performed. A relevant source of AML defenses
2012). Assuming that each agent knows their opponent’s inter- of this type are Adversarial Prediction Problems (APPs), whose
ests and uncertainty judgments, the adversary will perform the focus is on building adversarially robust predictive models. It
attacks dictated by the Nash equilibrium strategy of that game is assumed that during operations an adversary that exercises
or emerging as best responses. some control over the data generation process will be present:
Much less AML work is available in relation with unsuper- the data generation distributions at operations and training will
vised learning. Kos, Fischer, and Song (2018) describe adversar- be different, jeopardizing standard prediction techniques. To
ial attacks to generative models, where slight perturbations to the address this, APPs model interactions between the predictor and
model input may yield a reconstructed output that is very differ- a fictitious adversary during training as a two-agent game with a
ent from the original input. Biggio et al. (2013) study clustering system aimed at learning a parametric predictive model and an
under adversarial disturbances. The authors describe how to cre- adversary trying to transform the distribution governing data.
ate attacks that significantly alter cluster assignments, as well as The predictor will therefore minimize the expected cost under
obfuscation attacks that slightly perturb inputs to be clustered in the operations data distribution. In turn, the fictitious adversary
a predefined assignment, showing that single-link hierarchical will modify data optimizing his expected cost under this distri-
clustering is sensitive to such attacks. Lastly, adversarial attacks bution. As such distributions are not known, agents optimize
targeting time series forecasting systems have started to attract their regularized empirical costs, based on training data: the
interest. Alfeld, Zhu, and Barford (2016) describes an attacker predictor chooses her model minimizing her expected cost with
manipulating the inputs to drive the latent space of a linear respect to an attacked version of the training data chosen so as to
autoregressive (AR) model toward a region of interest. Similarly, minimize the adversary’s cost. The final optimization problems
Papernot et al. (2016) propose adversarial perturbations over are case dependent.
recurrent NNs, and Naveiro (2021) studies adversarial attacks In Stackelberg prediction games, Brückner and Scheffer (2011)
against Bayesian dynamic models. assume full information of the attacker about the predictive
model used by the defender who, in addition, has full informa-
Defenses against attacks. As mentioned in Section 3.1, we dis- tion about the adversary’s costs and action space. D acts first
tinguish two types of AML defenses: those that promote pro- choosing her parameters; then, A, who observes this decision,
tection strategies during training, and those that protect during chooses the optimal data transformation. Finding NE in these
operations. All these defenses assume that a clean training set is games leads to a bi-level optimization problem, minimizing the
available and the attacks happen once the ML system is deployed, defender’s cost function subject to the adversary, after observing
the most common case in realistic scenarios. the defender’s choice, minimizing his cost function. As nested
Protection during operations. The pioneering defense of this optimization problems are intrinsically hard, the authors restrict
type, proposed in Dalvi et al. (2004), was devoted to protect to simple classes where analytical solutions are available.6
classification systems. Given the importance of classification In Nash prediction games both agents act simultaneously.
in many cutting-edge ML applications, this article opens up Brückner, Kanzow, and Scheffer (2012) provide conditions for
a research area known as adversarial classification (AC). The existence and uniqueness of NE in certain subclasses of these
authors view AC as a game between a classifier (defender), games. Notice that APPs propose training using instances that
and an adversary (attacker). During operations, upon observ- are attacked by “fictitious attackers,” and hope that this will
ing a new vector of covariates, D aims at finding an optimal serve as a proxy for dealing with real attackers. However, it
classification strategy against A’s optimal attacks. Computing is assumed that the fictitious attackers’ costs and probabilities
Nash equilibria in such general games quickly becomes very are CK, which is not realistic in security scenarios. Deviations
complex. Thus, the authors propose a forward myopic version: from the assumed attackers’ models potentially lead to severe
D first assumes that data is untainted, computing her optimal performance degradation as we later illustrate.
classification decision; then, A deploys his optimal attack against Another important family of defenses that affect the train-
it. Subsequently, D implements her best response against such ing stage are those that aim at robustifying models against
attack, and so on. They assume CK as all parameters of both adversarial examples. Adversarial training (AT) (Madry et al.
players are known to each other. Although standard in game
theory, this assumption is unrealistic in security settings typical
in AML, an issue acknowledged in Dalvi et al. (2004) and largely 6
More recently, Naveiro and Insua (2019) provide efficient gradient methods
unsolved. to approximate solutions in more general problems.
 JOURNAL OF THE AMERICAN STATISTICAL ASSOCIATION 2199

2018) is the most important one. It aims at choosing a para- friend-or-foe-Q learning, but these come with unrealistic CK
metric model (usually a NN) that minimizes the empirical risk assumptions or can only be applied in restrictive scenarios.
evaluated under worst-case data perturbations. Thus, it can be
viewed as a zero-sum version of an APP: in AT, the fictitious Further comments. Practically all ML methods have been
attacker is assumed to select the data manipulation that maxi- touched upon from an adversarial perspective. Of major impor-
mizes the defender’s costs within some constrained region. AT tance in this field is the cleverhans (Papernot et al. 2018)
approximates the inner optimization through the PGD algo- library, aimed at accelerating research in developing new attack
rithm, ensuring that the perturbed input falls within a tolerable threats and more robust defenses specifically for deep neural
boundary, usually specified through some restriction on a norm models.
distance. Attack complexity depends on the chosen norm. How- AML is a difficult area which evolves rapidly and leads to an
ever, recent pointers urge modelers to depart from using norm arms race in which the community alternates cycles of proposing
based approaches (Carlini et al. 2019) and develop more realistic attacks and implementing defenses that deal with them. Thus, it
attack models as in Brown et al.’s (2017) adversarial patches. is important to develop sound techniques. Note that, stemming
Liu et al. (2018) adapted the idea of AT to Bayesian from Dalvi et al. (2004), most of AML research has been framed,
NNs, using the notion that incorporating randomness in the sometimes implicitly, within a standard game theory approach
NN weights enhances their robustness. They propose training characterized by NE and refinements. However, these entail CK
Bayesian NNs using mean-field variational inference. However, assumptions which are hard to maintain in the security contexts
as in AT, instead of maximizing the evidence lower bound typical of AML. We next propose a Bayesian decision theoretical
(ELBO) under the original training instances, they propose methodology to solve AML problems, using an ARA perspective
maximizing the ELBO under a worst-case attacker that chooses (Rios Insua, Rios, and Banks 2009) to model the confrontation
the best data manipulation inside a ball in a normed space. As between attackers and defenders mitigating questionable CK
with AT, this heuristic implicitly assumes full knowledge in the assumptions.
construction of the fictitious attacker, not taking into account
the existing uncertainty. This may produce performance degra-
dation when dealing with actual, unknown attackers. Finally,
4. A Bayesian Workflow for AML
another relevant but more heuristic family of defenses is called
adversarial logit pairing (Kannan, Kurakin, and Goodfellow We now revisit the workflow in Section 3.1 proposing a Bayesian
2018) in which logits of pairs of attacked and clean instances are decision theoretical alternative to AML. It is based on ARA,
encouraged to be the same, thus, yielding the same prediction which operationalizes the Bayesian approach to games (Kadane
for both. and Larkey 1982) and facilitates a procedure to forecast adver-
All defenses presented assume full knowledge in the attacking sarial attacks. ARA provides prescriptive support to a decision
models, leading to deterministic attacks. Taking into account maker (DM), the ML system in our case, facing one or more
existing uncertainties about adversaries would be crucial to attackers whose actions affect her decision making process. The
produce sensible defenses. This is the goal of the Bayesian frame- DM is assumed to be a rational, expected utility maximizing
work for AML presented in Section 3.1. agent. Her utility and beliefs (epistemic uncertainty) depends
on her decision, the adversaries’ rationality and decisions (con-
Adversarial Reinforcenment Learning. While there is consider- cept uncertainty), and possibly some other random variables
able AML research in supervised and unsupervised learning, (aleatoric uncertainty). Since CK is not assumed, random vari-
much less work is available in relation to reinforcement learning ables model adversaries’ decisions that must be integrated out
(RL). In it, adversarial aspects refer to the presence of agents to compute expected utilities. ARA provides a coherent pro-
whose decisions affect the reward perceived by our supported cedure to obtain probabilistic forecasts of adversaries’ actions.
agent. The prevailing solution approach in standard RL is Q- The main idea is to model the adversaries’ decision making
learning (Sutton and Barto 1998); its adaptation to large prob- process, putting priors on unknown quantities to reflect the
lems, deep Q-learning, has faced an incredible growth recently lack of knowledge. This way, the optimal adversarial decision
(Silver et al. 2017). It relies on models, such as convolutional becomes probabilistic. Simulations from such random optimal
NNs, to process input information. Consequently, the adver- decisions are used to compute the DM’s expected utility.
sarial examples described above apply when fooling RL systems Our focus will be on protecting a supervised learning system
(Lin et al. 2017). (D) which receives instances described by covariates x ∈ Rd ,
Single-agent RL methods fail in presence of other agents that with each instance having an associated output y. Uncertainty
interfere with their learning process, as they do not take into about the instances’ output given its covariates is modeled
account the nonstationarity due to the other agents’ actions: Q- through a distribution p(y|x). This distribution can arise from
learning may lead to sub-optimal results (Buşoniu, Babuška, and a generative model, where distributions p(x) and p(x|y) are
De Schutter 2010). Thus, a deployed RL system must be able modeled explicitly and p(y|x) is obtained via Bayes formula; or
to reason about and forecast the adversaries’ behavior. Several from a discriminative model, in which p(y|x) is modeled directly
methods to enhance Q-learning in multiagent systems have been (Bishop 2006). It may be derived through maximum likelihood
proposed, mostly focusing on adapting ideas from game theory or in a Bayesian way using training data which, by assumption, is
into RL, mainly focusing on modeling the multiagent system free of attacks. Whichever estimation method is adopted, upon
through Markov games. Three well-known solutions (Tuyls and observing a new instance with covariates x, the Defender must
Weiss 2012) are minimax-Q learning; Nash-Q learning; and decide the corresponding output. As D is rational, she decides
2200 D. R. INSUA ET AL.

based on maximum predictive utility through optimal adversarial data modification that becomes random.
 Its associated probability distribution conforms to the attack-
arg max u(yD , y)p(y|x)dy, ing model used to protect the learning algorithm. In general,
yD
evaluating analytically such model will be unfeasible. However,
where u(yD , y) is the utility that she perceives when an instance in most cases it is conceptually and computationally simple
whose actual output is y is assigned output yD .7 In adversarial to sample from it. This just entails sampling from our priors
settings, agent A applies an attack a to the features x leading to and, for each sample, solving the adversary’s decision making
the transformation x = a(x), the observation actually received problem, which provides a sample from the random optimal
by D. We focus on exploratory attacks, which affect just over data manipulation. In our adversarial supervised learning con-
operational data, leaving training data untainted. Let us revisit text, the adversary will produce data manipulations solving (1).
the three stages of the workflow. Under standard CK assumptions,
 our attacking model  would be
p(x |x) = δ x − arg maxz uA (yD , y)pA (yD |z)dyD . However,
1. Gathering intelligence. This stage entails modeling the as argued, CK rarely holds in security domains there being
attacker’s problem. Assessing his goals requires determining the multiple sources of uncertainty. First of all, unlike the adversary,
actions that he may undertake and the utility that he perceives we do not know the actual output y for a given instance x.
when performing a specific action, given a defender’s strategy: Thus, our attackingmodel must account for this uncertainty
the output is the set of attacker’s decisions and a functional form through p(x |x) = p(x |x, y)p(y|x)dy. Sampling from p(y|x)
for his utility, generally dependent on his and the defender’s is standard. Sampling from p(x |x, y) is more complex, as we
decisions. Assessing the attacker knowledge entails looking for usually have uncertainty about the adversary’s utility uA (yD , y)
information that he may have when performing the attack, and and his probability estimates pA (yD |z). We propose modeling
his degree of knowledge about it, as we do not assume CK. such uncertainty with, respectively, random utilities UA and
This requires not only a modeling activity, but also a security y
random probabilities PAD defined, without loss of generality, over
assessment of the ML system determining which of its elements an appropriate common probability space (, A, P) with atomic
(training data, feature space, architecture, loss function, param- elements ω ∈ . This induces a distribution over the Attacker’s
eters, etc.) are accessible to the attacker. Finally, identifying his optimal attack defined through
capabilities requires determining which part of the defender 
problem the attacker has influence on. Xω (x, y) = arg max UAω (yD , y)PAω (yD |z)dyD ,
Consider an adversary aiming at fooling a supervised learn- z
ing system by modifying the value of the covariates. The adver- leading to p(x |x, y) = P(Xω (x, y) = x ). In such a way, our
sary receives objects with covariates x and output y and manip- 
model for p(x |x, y) properly accounts for the existing uncer-
ulates x, transforming them into x . His goal is to induce the tainty about the adversary. By construction, if we sample utilities
defender to make nonoptimal decisions for the output cor- and probabilities from their corresponding priors and solve
responding to the observed covariates. Following a norma- (1), this solution would be distributed according to p(x |x, y).
tive decision theoretical perspective, we model the adversary Overall, to produce samples from p(x |x), we first sample from
as a rational agent choosing data manipulations to maximize y from the posterior predictive distribution p(y|x), and then x
expected utility. Let uA (yD , y) be the adversary’s utility when the from p(x |x, y).
defender assigns output yD to an instance whose actual output is The specifications of random utilities and random proba-
y. This utility can also depend on the specific data manipulation, bilities are application-specific. Guidelines for adversarial clas-
as distinct manipulations can entail different costs; however, to sification are given in Gallego et al. (2020). Notice that, to
simplify notation we do not include explicitly this dependence. model our uncertainty about the adversary, we study his deci-
The adversary thus chooses data manipulations through sion making problem from our point of view. Obviously, when

analyzing the Attacker’s problem, we must take into account his
x (x, y) = arg max uA (yD , y)pA (yD |z = a(x))dyD , (1) uncertainty about our elements; for example, his uncertainty
z
about the defender’s decision upon observing the manipulated
where pA (yD |z = a(x)) models the adversary’s belief about the instance. This could lead to a infinite hierarchy of decision
defender’s decision upon observing the manipulated instance making problems as presented in Rios and Insua (2012), albeit
z = a(x). in a simpler context. One would typically model several steps in
2. Forecasting likely attacks. Based on Step 1, we produce models the hierarchy and stop at a level in which no more information is
for how the adversary would modify data, encoding not only available. At that stage, noninformative priors over the involved
the information gathered, but also our uncertainty about the probabilities and utilities can be used.
adversary’s elements. The output of this stage is an attacking To sum up, we have now a general, formal, decision-theoretic
model, a probability distribution over adversarial manipulations alternative to produce attacking models, that is, samples from
that encodes all relevant uncertainties. A formal Bayesian way p(x |x), keeping CK assumptions at a minimum. Note, however,
to do this, as suggested by ARA, is to place priors on every that previous attacks proposed in the literature can be adopted
unknown element of the adversary’s decision making problem. within the proposed workflow. For instance, consider the FGSM
The uncertainty implied by these priors is propagated to the attack (case 2, Section 2): it assumes that the defender uses
a parameterized model with parameters θ , trained minimiz-
7
If the output is discrete rather than continuous (as in classification problems) ing a loss function L(θ , x, y); the attacker has full knowledge
the integral is replaced by a sum. about such loss, or at least its gradient and has resources to
 JOURNAL OF THE AMERICAN STATISTICAL ASSOCIATION 2201

perturb the covariates by adding a small vector . Under this Protection during training. As mentioned, other defenses mod-
CK setting,
 the proposed
 (deterministic) attack is x = x +  · ify how training is performed in order to take into account
sign ∇x L(θ , x, y) , leading to an attacking model p(x |x, y, θ ), the possible presence of an adversary during operations. Their
degenerated at such x . goal is to train using artificial data that somehow mimic actual,
Our proposed attacking model would be classified as gray potentially attacked, operational data through several heuris-
box since, even if perfect knowledge is not assumed, certain tics. Most of them model how the attacker would modify the
assumptions about the adversary are being made such as him instances in the training set. Having trained the classifier in
being an expected utility maximizer. One of the advantages of this manner, p(y|x ) could be directly evaluated at the operation
the proposed approach is its ability to create more complex stage as this probability has been inferred taking into account
attacking models; for example, through mixtures of attackers the presence of an attacker. As discussed, these methods assume
with different solution concepts (Rios Insua, Banks, and Rios models for how the attacker would modify training instances
2016). Finally, for purely black-box settings in which the attacker that do not take into account the existing uncertainty. For
is only assumed to have query access to the target learning instance, AT, as a proxy to robustify classifiers against attacks,
system, an interesting approach to produce attacking models has considers an attacker that produces the worst data modifica-
been recently introduced (Lee et al. 2022). Here, the authors tion for the classifier, assuming explicitly that the classifier has
propose to generate attacks using Bayesian optimization, model- knowledge about the attacker’s objectives, and implicitly that he
ing the (unknown) attacker’s objective function with a Gaussian has knowledge about the classifier’s utility. However, in realistic
process that is sequentially updated after the queries’ results are settings, we would not have precise information about how the
received. attacker modifies a given instance, as we do not know, in gen-
eral, his intentions and probability assessments. Thus, assuming
3. Protecting ML algorithms. Once with a reasonable probabilistic a deterministic attack for robustification purposes may result
attacking model, we protect our learning system against such inappropriate. We believe that it is crucial to account for such
attacks, either at operations or training. uncertainty explicitly.
Protection during operations. The Defender observes a poten- Ye and Zhu (2018) take a step in this direction. They provide
tially attacked vector of covariates x . Based on it, she has a Bayesian counterpart of AT, designated Bayesian Adversarial
to assign an output yD . An adversary-unaware D will make Learning, assuming that the Defender has observed clean train-
this decision
 by maximizing the posterior predictive utility, ing data D = {xi , yi }Ni=1 which are samples from an unknown
arg maxyD u(yD , y)p(y|x )dy. In terms of the spam detection distribution. Based on this, an adversary unaware defender using
example (Section 2), x would represent the words used in an a model parameterized by θ will simply compute the posterior
E-mail, potentially manipulated by a spammer, and yD corre- p(θ |D) and employ this to calculate the predictive distribution
sponds to labeling such E-mail as spam or legitimate. As illus- used during operations. However, the presence of an adversary
trated, solving this classification using an adversary-unaware D at operations changes the data-generation mechanism, Thus,
could lead to serious performance degradation. using the original D for inference could lead to large perfor-
Upon observing x , D is uncertain about the actual originat- mance degradation. Instead, the authors suggest computing a
ing vector of covariates x (the words of the originating E-mail). robustadversarial posterior distribution (RAPD) over the param-
She may model this uncertainty through a distribution p(x|x ) eters p(θ |D)p(D|D)dD, where D refers to the manipulated
and decide based on the posterior predictive utility, where x has training data. Gibbs sampling provides samples from it iterating
been marginalized out through
 
D(t) |θ (t−1) , D ∼ p(D|θ (t−1) , D), (3)
arg max u(yD , y) p(y|x)p(x|x )dx dy, (2)
yD
θ (t) |D(t) ∼ p(θ |D(t) ). (4)
where conditional independence of y and x given x is assumed.
Thus, when adversaries are present, rather than deciding based After a burn-in period, samples {θ (T) , D(T) } follow the joint
on the posterior predictive distribution of a new instance, we posterior p(θ , D|D) and, consequently, sample θ (T) follows the
do it based on what we designate the robust adversarial poste- RAPD. The distribution p(D|θ , D) quantifies our uncertainty
rior predictive distribution (RAPPD) p(y|x)p(x|x )dx. This is about the data generation process; that is, about how the adver-
generally not available in closed form and has to be evaluated sary will modify data D.
numerically using Monte Carlo methods. The key step for this As with attacks, if we assume a high degree of CK, earlier
is the ability to sample from p(x|x ), that is, the distribution of defense mechanisms can be framed within our workflow. In
possible originating covariates given the observed ones x . Here AT, the defender uses a parametric model with parameters θ .
is where the attacker models from Step 2 come into play. Having An adversary unaware D makes inference about θ minimiz-
constructed a model for p(x |x) and being able to sample from it, ing a loss function N i=1 L(θ , xi , yi ), with N training points.
all we need is to generate samples from the inverse distribution When taking into account the adversary, AT proposes minimiz-
p(x|x ). Techniques for this based on Approximate Bayesian ing N i=1 maxγ ≤ L(θ , xi + γ , yi ); that is, minimize the loss
Computation (ABC) are discussed in Gallego et al. (2020). In evaluated under worst-case perturbations in some constrained
terms of the spam detection case, the attack model p(x |x) would region.
reflect D’s uncertainty about the manipulated words x that the Most common losses can be written as negative log posterior
adversary selects, given the E-mail with words x. distributions. Thus, for the rest of the discussion, assume that
2202 D. R. INSUA ET AL.

the loss function can be written as parameters’ values. Accounting for this would entail that the
N N previous sampling scheme is done over an uncertain defender
L(θ , xi , yi ) = − log p(xi , yi |θ ) − log p(θ ). (5) model p(y|x, θ ) and start a hierarchy of level-k thinking (Rios
i=1 i=1 and Insua 2012), which can be computationally intractable.
Instead, we propose mixing both steps and sample from the
If we assume that, for some fixed value of θ , p(D|D) has the form posterior distribution of the defended model rather than just
N arriving at θ MAP . To do so efficiently, if the model is optimized
p(D|D) = p(xi , yi |xi , yi ) using gradient descent routines (as usual with deep NN models),
i=1 we can again leverage SG-MCMC techniques to sample from the
N    robust posterior p(θ |{(xi∗ , yi )}N
i=1 ), by repeating the following
= δ xi − xi + arg max L(θ , xi + γ , yi ) , procedure for some number of training iterations:
γ ≤
i=1
1. Sample perturbed samples x1 , . . . , xK ∼ p(D|D, θ ) using the
and we recover AT as a maximum a posteriori (MAP) estimate sampler from (6), or from the natural distribution, for a mini-
of θ under the robust adversarial posterior distribution. Indeed, batch of size K.
notice that the robust posterior can be written 2. Update θt+1 = θt − ∇ Ki=1 L(θt , xi , yi ) + N (0, 2I)

In the end, we collect S samples {θi }Si=1 from the robust posterior
p(θ |D)p(D|D)dD = p(θ |{(xi∗ , yi )}N
i=1 ), distribution. Then, given an instance x, we compute the pre-
dicted output yD approximating the posterior predictive utility
where xi∗ = xi +arg maxγ ≤ L(θ , xi +γ , yi ). The MAP estimate
using MC.
of θ is
 
θ MAP = arg max log p(θ |{(xi∗ , yi )}N
i=1 )
θ 5. Case Studies
 N

= arg min − log p(xi∗ , yi |θ ) − log p(θ ) , We illustrate the proposed defense mechanisms through the
θ
i=1
motivating examples from Section 2.8

which, according to (5) is nothing else but the loss evaluated Case 1. Spam detection. Consider the set-up from Section 2.
under the worst-case transformation xi∗ . Thus, AT is a special Protection during operation. For the first batch of experiments,
case of our workflow, in which we assume CK in the sense we use the same algorithms in Section 2. Recall the severe per-
that the attacking model p(D|D) is deterministic; that is, a formance degradation resulting from Good-Words-Insertion
degenerate distribution. attacks (Table 1, cols. 2 and 3). Once the models are trained,
There are several ways of sampling from the conditionals we perform attacks over the instances in the test set, solving
(3) and (4). Rios Insua, Naveiro, and Gallego (2020) propose problem (1) for each test spam E-mail, assuming certain val-
a scalable way of doing it leveraging efficient SG-MCMC sam- ues for the attacker’s utilities and probability judgements. D
pling algorithms, and, in particular, stochastic gradient Langevin is uncertain about the attacker’s elements and models these
dynamics (SGLD, Welling and Teh 2011). First, to account for uncertainties with random utilities and probabilities: we use
the uncertainty that the defender has over the attacking model, beta distributions centered at the attacker’s actual utility and
the authors propose defining p(D|D, θ ) ∝ exp {L(θ , xi , yi )}. probability values with variances chosen to guarantee that the
Under SGLD, sampling iterations adopt the form distribution is concave in its support (they must be bounded
 
xi,t+1 = xi,t −  ∇x (log p(y|xi,t , θ ) + log p(θ )) + ξt , (6) from above by min [μ2 (1 − μ)]/(1 + μ), [μ(1 − μ)2 ]/(2 − μ) ,
where μ is the corresponding mean). The variance size informs
with ξt ∼ N (0, 2), and t = 1, . . . , T with xi,1 = xi . Note that about the degree of knowledge the defender is assumed to have
this is a sampler from the distribution p(D|D, θ ) and we approx- about the attacker; reflecting a moderate lack of knowledge, we
imate this distribution with the set of attacked samples {xi∗ }Ni=1 , set the variance to be 10% of this upper bound. Of course, we
setting xi∗ = xi,T . Further uncertainties can also be accounted are assuming certain degree of knowledge about the adversary,
for; let us denote with λ the vector of hyperparameters of the as the expected values of the random utilities and probabilities
 t or the number T of iterations.
optimizer, such as the step sizes coincide with the actual values used by the attacker. We later
Then, we have p(D|D, θ ) = p(D|D, θ , λ)p(λ)dλ. To generate a study how deviations from the assumed attacker behavior affect
perturbed data sample from the previous distribution, we need performance.
to sample from p(λ). For instance, we could sample the step Having a model for the attacker, for each instance x of the
sizes t from a beta distribution over [1e − 5, 1e − 3], which test set, the defender computes the robust adversarial posterior

are typical values in computer vision tasks using deep NNs; predictive distribution p(y|x)p(x|x )dx, and assigns x to the
the number of iterations T could be sampled from a Poisson class maximizing the posterior predictive utility (2). To compute
distribution. Moreover, we could consider mixtures of different the RAPPD, samples from p(x |x) are obtained leveraging the
attackers, for instance by sampling a Bernoulli random variable ability to sample from the attacker model and using ABC as in
and then choosing the gradient corresponding to either FGSM Gallego et al. (2020).
or another attack, such as Carlini and Wagner’s (2017).
The attacker might have also uncertainty over the model the 8
Code to reproduce these experiments is available at [Link]
defender adopts, let it be the concrete model architecture or its roinaveiro/aml_bayes.
 JOURNAL OF THE AMERICAN STATISTICAL ASSOCIATION 2203

Table 2. Average accuracy plus minus one standard deviation of four algorithms on uncertainty estimates from the posterior. Next, we attack the
attacked data without defense (col 2); with CK defense (col 3); and with ARA defense
(col 4).
clean test set using the procedure in Section 2 and evaluate
the performance of our robustification proposal. Since we are
Classifier Acc. Taint. Acc. CK Taint. Acc. ARA Taint. dealing with discrete attacks, we cannot use the uncertainty over
Naive Bayes 0.793 ± 0.005 0.867 ± 0.004 0.883 ± 0.005 attacks as in (6). Instead, we model it using the distribution
Logistic Reg. 0.687 ± 0.008 0.803 ± 0.007 0.864 ± 0.005 p(x |x) and take samples from it as discussed in step 3 from
Neural Network 0.774 ± 0.007 0.767 ± 0.007 0.792 ± 0.006
Random Forest 0.682 ± 0.005 0.819 ± 0.007 0.821 ± 0.007 our Section 4 workflow. We evaluate the Bayesian predictive
distribution using S = 5 posterior samples obtained after
T = 2000 SGLD iterations, and present the results in Table 1,
Column 4 in Table 1 compares the average accuracy of the col. 5. Observe again that the proposed robustification process
robustified during operation classifiers on tainted data. As can be protects differentiable classifiers, recovering from the degraded
seen, our approach allows us to reduce the performance degra- performance under attacked data. Note that the robustified algo-
dation of the four original classifiers, showcasing the benefits of rithms achieve even higher accuracies than those attained by the
explicitly modeling the attacker’s behavior in adversarial envi- original classifier over clean data, due to the regularizing effect
ronments. Interestingly, in the naïve Bayes case, our approach mentioned above.
even outperforms the algorithm behavior under untainted data
(column 2). This effect has been observed also in Naveiro et al. Case 2. Vision. When the input data is high-dimensional (such
(2019a) and Goodfellow, Shlens, and Szegedy (2015) for other as with images), our ARA robustification at operation easily
algorithms and application areas. This is likely due to the fact becomes computational intractable. We thus robustify model
that the presence of an adversary has a regularizing effect, being at training. For illustration purposes, we perform additional
able to improve the original accuracy of the base algorithm, and experiments using two benchmarks: Fashion-MNIST, a clothing
making it more robust. classification problem (Xiao, Rasul, and Vollgraf 2017), and
The previous experiment used beta distributions centered Kuzushiji-MNIST, a traditional Japanese handwritten character
around the values actually employed by the attacker to quantify recognition problem (Clanuwat et al. 2018). For both datasets
the uncertainty about the attacker’s utility and probability. It we trained standard deep NNs over their respective training sets
is natural to explore how deviations from the assumed values (consisting of 60.000 images each) using SGD (i.e., no defense),
affect performance. Our second batch of experiments tests the adversarial training (AT defense), and our robustification pro-
approach against an attacker whose utilities and probabilities cedure from Section 4 (ARA defense). We then attacked the
are different from those assumed by the defender. In particu- respective test sets using five iterations of PGD, with varying
lar, for each attack, A deviates uniformly around the assumed attack intensities (the step-size  in (6)), and evaluated the
probability and utility. The size of the deviation is constrained accuracies of both models under these attacked test sets. Figure 2
to be less than 50% the assumed value: if we center our beta displays these results. Note that our scalable approach from Sec-
distribution for, for example, the attacker’s probability at value μ, tion 4 offers fairly superior robustification defenses compared
the attacker will deviate from the assumed behavior in the range to the AT defense mechanism, showing that incorporating the
(0.5 · μ, 1.5 · μ). Thus, in this experiment, our beta distributions uncertainties provided by the ARA methodology has additional
will be centered around wrong values. We set the variance of the benefits when adversarially training a ML model in diverse
beta priors to be relatively high, at 50% of the upper bound, and datasets.
compare our approach with the CK one, in which the elements of
the attacker are assumed to be known, and thus are point masses
(on wrong values). 6. Conclusion
Table 2 shows average accuracy plus minus one standard
deviation (estimated through repeated hold out validation) of We have provided a review of key approaches, models, and
the four algorithms on attacked data without defense (col. 2), concepts in AML. This area is of major importance in security
the standard CK defense (col. 3) and, finally, our ARA defense and cybersecurity to protect systems that increasingly rely on
(col. 4). Note first the overall performance drop with respect to ML algorithms (Comiter 2019; Ciancaglini et al. 2020). The pio-
the results in Table 1 col. 4: when the attacker deviates from neering work by Dalvi et al. (2004) framed most of this research
his assumed behavior, the performance of both ARA-based and within the game theory realm, with entailed CK conditions
CK defenses is lower. However, we can also observe that the which hardly hold in AML security contexts.
ARA-based defense outperforms the CK defense for all clas- We have proposed a Bayesian alternative to AML. Its main
sifiers: when the attacker deviates from the assumed behavior, difference with respect to previous approaches is that unreal-
accounting for the uncertainty over his elements is beneficial. istic CK conditions are not entailed. As a consequence of the
This experiment showcases the increase in robustness due to increased realism, the resulting models are more robust, espe-
modeling uncertainty in scenarios in which CK is not realistic. cially with respect to deviations in the assumed attacker behav-
ior, given the better reflection of the involved uncertainties as
Protection during training. We next assess ARA based robustifi- empirically illustrated. Our Bayesian framework enjoys greater
cation during training. This requires the underlying model to be flexibility than previous game-theoretic approaches, some of
differentiable in the parameters, thus, leaving just two candidates which can be framed as limit or degenerate cases of our proposal.
among the original models: logistic regression and NN. Both However, the increased robustness and flexibility comes at a
models can be trained using SGD plus noise methods to obtain higher computational and modeling costs. Investigating how to
2204 D. R. INSUA ET AL.

Figure 2. Robustness of a deep network against the PGD attack under three defense mechanisms (NONE, AT, ARA). (a) depicts the security evaluation curves for the attacked
Fashion-M. dataset. (b) depicts the respective curves for the attacked Kuzushiji-M. dataset.

alleviate these costs is an open research question. Related to this, counterfactual inference in observational studies (Johansson,
we sketch several promising avenues for future work. Shalit, and Sontag 2016); and (iv) leveraging ideas from causal
A promising research line consists of developing efficient inference to improve adversarial robustness (Schölkopf et al.
algorithms for approximate Bayesian inference with robust- 2021). Several recent works, for instance, aim to improve the
ness guarantees. For example, regarding opponent modeling in robustness of deep NNs for image classification by leveraging a
sequential decision making, an agent has uncertainty over her causally informed model of unseen perturbations or adversarial
opponent type initially; as more information is gathered, she examples (Zhang, Zhang, and Li 2020a; Zhang et al. 2021).
might reduce her uncertainty via Bayesian updating. Similarly,
work in robust Bayesian analysis (Rios Insua and Ruggeri 2000),
in particular referring to likelihood robustness, is relevant. Not Supplementary Materials
taking into account an attacked data generation process is an Python code to reproduce the results presented in this article is avail-
example of model misspecification; robustness of Bayesian infer- able at [Link] Reproducibility workflow
ence to such issue has been revisited recently in Miller and is included in the repository’s readme file.
Dunson (2019).
There are also several enhancements aimed at improving
operational aspects of the framework. For example, we discussed Funding
only problems with two agents. It would be relevant to deal The authors acknowledge the support of the National Science Foundation
with multiple agents, including cases in which agents on attack under Grant DMS-1638521 to the Statistical and Applied Mathematical Sci-
or defense attempt to cooperate. There is also potential in new ence Institute (SAMSI), NC, USA. RN acknowledges the support of CUNEF
University. DRI is supported by the AXA-ICMAT Chair and the Spanish
algorithmic approaches. Exploring gradient-based techniques
Ministry of Science program PID2021-124662OB-I00. VG acknowledges
for bi-level optimization problems arising in AML is a fruitful support from grant FPU16-05034. This work is supported by the Severo
line (Naveiro and Insua 2019). More efficient MCMC samplers Ochoa Excellence Programme CEX-2019-000904-S, the European Union’s
can be adopted in the proposed workflow (Gallego and Insua Horizon 2020 Research and Innovation Programme under Grant Agree-
2018). Recall that our framework essentially goes through sim- ment No. 101021797 (Starlight) and 815003 (Trustonomy), the US NSF
grant DMS-1638521 and a grants from the FBBVA (Amalfi), EOARD
ulating from the attacker problem to forecast attacks and then
(FA8655-21-1-7042) and AFOSR (FA-9550-21- 1-0239).
optimizing for the defender to find her optimal decision. This
may be computationally demanding and we could explore single
stage approaches, such as augmented probability simulation ORCID
(Ekin et al. 2022).
Roi Naveiro [Link]
As mentioned in Section 3.2, adversarial versions of various
ML problems have been studied. However, further research is
required in unsupervised learning, including clustering meth- References
ods, dynamic linear models (Naveiro 2021), natural language
processing models (Wang et al. 2019), and in RL, including Alfeld, S., Zhu, X., and Barford, P. (2016), “Data Poisoning Attacks Against
Autoregressive Models,” in Proceedings of the 30th AAAI Conference
policy gradient (Lin et al. 2017) and extensions to semi-Markov
Artificial Intelligence, pp. 1452–1458. [2198]
Decision Processes (Du, Futoma, and Doshi-Velez 2020). Athalye, A., Carlini, N., and Wagner, D. (2018), “Obfuscated Gradients
Applications, such as those presented in Comiter (2019) Give a False Sense of Security: Circumventing Defenses to Adversarial
and Ciancaglini et al. (2020), are abound. We mention four of Examples,” in International Conference on Machine Learning, pp. 274–
direct interest to us: (i) the development of defenses against fake 283. [2197]
Banks, D., Gallego, V., Naveiro, R., and Ríos Insua, D. (2020), “Adversarial
news; (ii) the development of robust ADS algorithms (Caballero, Risk Analysis: An Overview,” Wiley Interdisciplinary Reviews: Computa-
Ríos Insua, and Banks 2021); (iii) the use of AML for improving tional Statistics, 14, e1530. [2197]
 JOURNAL OF THE AMERICAN STATISTICAL ASSOCIATION 2205

Biggio, B., Fumera, G., and Roli, F. (2014), “Security Evaluation of Pattern Goodfellow, I., Shlens, J., and Szegedy, C. (2015), “Explaining and Harness-
Classifiers Under Attack,” IEEE Transactions on Knowledge and Data ing Adversarial Examples,” in International Conference on Learning Rep-
Engineering, 26, 984–996. [2198] resentations, Available at [Link] [2195,2203]
Biggio, B., Pillai, I., Rota Bulò, S., Ariu, D., Pelillo, M., and Roli, F. (2013), Gowal, S., Dvijotham, K., Stanforth, R., Bunel, R., Qin, C., Uesato, J.,
“Is Data Clustering in Adversarial Settings Secure?” in Proceedings of the Arandjelovic, R., Mann, T., and Kohli, P. (2018), “On the Effectiveness
2013 ACM Workshop on Artificial Intelligence and Security, pp. 87–98, of Interval Bound Propagation for Training Verifiably Robust Models,”
ACM. [2198] arXiv:1810.12715. [2197]
Biggio, B., and Roli, F. (2018), “Wild Patterns: Ten Years after the Rise of Hargreaves-Heap, S., and Varoufakis, Y. (2004), Game Theory: A Critical
Adversarial Machine Learning,” Pattern Recognition, 84, 317–331. [2196] Introduction, New York: Routledge. [2195]
Bishop, C. (2006), Pattern Recognition and Machine Learning, New York: Johansson, F., Shalit, U., and Sontag, D. (2016), “Learning Representations
Springer. [2199] for Counterfactual Inference,” in International Conference on Machine
Brown, T. B., Mané, D., Roy, A., Abadi, M., and Gilmer, J. (2017), “Adver- Learning, pp. 3020–3029, Available at [Link]
sarial Patch,” arXiv:1712.09665. [2199] [Link] . [2204]
Brückner, M., Kanzow, C., and Scheffer, T. (2012), “Static Prediction Joseph, A., Nelson, B., Rubinstein, B., and Tygar, J. (2019), Adversarial
Games for Adversarial Learning Problems,” Journal of Machine Learning Machine Learning, Cambridge, UK: Cambridge University Press. [2196]
Research, 13, 2617–2654. [2198] Joshi, C., Aliaga, J. R., and Insua, D. R. (2021), “Insider Threat Modeling: An
Brückner, M., and Scheffer, T. (2011), “Stackelberg Games for Adversarial Adversarial Risk Analysis Approach,” IEEE Transactions on Information
Prediction Problems,” in Proceedings of the 17th ACM SIGKDD Interna- Forensics and Security, 16, 1131–1142. [2197]
tional Conference, pp. 547–555. [2198] Kadane, J. B., and Larkey, P. D. (1982), “Subjective Probability and the
Buşoniu, L., Babuška, R., and De Schutter, B. (2010), “Multi-Agent Rein- Theory of Games,” Management Science, 28, 113–120. [2199]
forcement Learning: An Overview,” in Innovations in Multi-Agent Sys- Kannan, H., Kurakin, A., and Goodfellow, I. (2018), “Adversarial Logit
tems and Applications - 1, eds. D. Srinivasan and L. C. Jain, pp. 183–221, Pairing,” arXiv:1803.06373. [2199]
Berlin, Heidelberg: Springer-Verlag. [2199] Kim, J.-H. (2009), “Estimating Classification Error Rate: Repeated Cross-
Caballero, W. N., Ríos Insua, D., and Banks, D. (2021), “Decision Support Validation, Repeated Hold-Out and Bootstrap,” Computational Statistics
Issues in Automated Driving Systems,” International Transactions in and Data Analysis, 53, 3735–3745. [2196]
Operational Research, 30, 1216–1244. [2195,2196,2204] Kos, J., Fischer, I., and Song, D. (2018), “Adversarial Examples for Generative
Carlini, N., Athalye, A., Papernot, N., Brendel, W., Rauber, J., Tsipras, Models,” in 2018 IEEE Security and Privacy Workshops (SPW), IEEE, pp.
D., Goodfellow, I., and Madry, A. (2019), “On Evaluating Adversarial 36–42. [2198]
Robustness,” arXiv:1902.06705. [2199] LeCun, Y., Cortes, C., and Burges, C. (1998), “The MNIST Database
Carlini, N., and Wagner, D. (2017), “Towards Evaluating the Robustness of of Handwritten Digits,” Available at [Link]
Neural Networks,” in 2017 IEEE Symposium on Security and Privacy (SP), [2196]
Lee, D., Moon, S., Lee, J., and Song, H. O. (2022), “Query-Efficient and
pp. 39–57. [2202]
Scalable Black-Box Adversarial Attacks on Discrete Sequential Data via
Ciancaglini, C. G., Sancho, D., McCarthy, O., Eira, M., Amann, P., Klayn,
Bayesian Optimization,” in International Conference on Machine Learn-
A., McArdle, R., Beridze, I., and Amann, P. (2020), “Malicious uses
ing, pp. 12478–12497, PMLR. [2201]
and Abuses of Artificial Intelligence,” Trend Micro Research, Available
Lin, Y.-C., Hong, Z.-W., Liao, Y.-H., Shih, M.-L., Liu, M.-Y., and Sun, M.
at [Link]
(2017), “Tactics of Adversarial Attack on Deep Reinforcement Learning
[Link] . [2203,2204]
Agents,” in Proceedings of the 26th International Joint Conference on
Clanuwat, T., Bober-Irizar, M., Kitamoto, A., Lamb, A., Yamamoto, K.,
Artificial Intelligence, IJCAI’17, p. 3756–3762, AAAI Press. [2199,2204]
and Ha, D. (2018), “Deep Learning for Classical Japanese Literature,”
Liu, X., Li, Y., Wu, C., and Hsieh, C.-J. (2018), “ADV-BNN: Improved
arXiv:1812.01718. [2203]
Adversarial Defense through Robust Bayesian Neural Network,”
Comiter, M. (2019), “Attacking Artificial Intelligence,” Available at arXiv:1810.01279. [2199]
[Link] Lowd, D., and Meek, C. (2005), “Adversarial learning,” in Proceedings of
[Link] . [2195,2203,2204] the 11th ACM International Conference on Knowledge Discovery in Data
Couce-Vieira, A., Insua, D. R., and Kosgodagan, A. (2020), “Assessing Mining, KDD’05, pp. 641–647. [2198]
and Forecasting Cybersecurity Impacts,” Decision Analysis, 17, 356–374. Madry, A., Makelov, A., Schmidt, L., Tsipras, D., and Vladu, A. (2018),
[2196] “Towards Deep Learning Models Resistant to Adversarial Attacks,” in
Dalvi, N., Domingos, P., Mausam, Sumit, S., and Verma, D. (2004), “Adver- International Conference on Learning Representations [2198,2199]
sarial Classification,” in Proceedings of the 10th ACM SIGKDD Interna- Menache, I., and Ozdaglar, A. (2011), “Network Games: Theory, Models,
tional Conference, KDD ’04, pp. 99–108. [2195,2197,2198,2199,2203] and Dynamics,” Synthesis Lectures on Communication Networks, 4, 1–
Dasgupta, P., and Collins, J. B. (2019), “A Survey of Game Theoretic 159. [2195]
Approaches for Adversarial Machine Learning in Cybersecurity Tasks,” Miller, J. W., and Dunson, D. B. (2019), “Robust Bayesian Inference via
AI Magazine, 40, 31–43. [2196] Coarsening,” Journal of the American Statistical Association, 114, 1113–
Du, J., Futoma, J., and Doshi-Velez, F. (2020), “Model-based Reinforce- 1125. [2204]
ment Learning for Semi-Markov Decision Processes with Neural ODEs,” Naveiro, R. (2021), “Adversarial Attacks against Bayesian Forecasting
arXiv:2006.16210. [2204] Dynamic Models,” in 22nd European Young Statisticians Meeting, p. 66,
Dua, D., and Graff, C. (2017), “UCI Machine Learning Repository,” Avail- Available at [Link] . [2198,2204]
able at [Link] [2196] Naveiro, R., and Insua, D. R. (2019), “Gradient Methods for Solving Stackel-
Ekin, T., Naveiro, R., Insua, D. R., and Torres-Barrán, A. (2022), “Aug- berg Games,” in International Conference on Algorithmic DecisionTheory,
mented Probability Simulation Methods for Sequential Games,” Euro- pp. 126–140, Springer. [2198,2204]
pean Journal of Operational Research, 306, 418–430. [2204] Naveiro, R., Redondo, A., Ríos Insua, D., and Ruggeri, F. (2019a), “Adversar-
Fan, J., Ma, C., and Zhong, Y. (2021), “A Selective Overview of Deep ial Classification: An Adversarial Risk Analysis Approach,” International
Learning,” Statistical Science, 36, 264–290. [2195] Journal of Approximate Reasoning, 113, 133–148. [2196,2197,2203]
Gallego, V., and Insua, D. R. (2018), “Stochastic Gradient MCMC with Naveiro, R., Rodríguez, S., and Ríos Insua, D. (2019b), “Large-Scale Auto-
Repulsive Forces,” in Bayesian Deep Learning Workshop, Neural Informa- mated Forecasting for Network Safety and Security Monitoring,” Applied
tion and Processing Systems, Available at [Link] Stochastic Models in Business and Industry, 35, 431–447. [2197]
2018/papers/[Link] . [2204] Papernot, N., Faghri, F., Carlini, N., Goodfellow, I., Feinman, R., Kurakin,
Gallego, V., Naveiro, R., Redondo, A., Insua, D. R., and Ruggeri, F. A., Xie, C., Sharma, Y., Brown, T., Roy, A., Matyasko, A., Behzadan, V.,
(2020), “Protecting Classifiers From Attacks. A Bayesian Approach,” Hambardzumyan, K., Zhang, Z., Juang, Y.-L., Li, Z., Sheatsley, R., Garg,
arXiv:2004.08705. [2200,2201,2202] A., Uesato, J., Gierke, W., Dong, Y., Berthelot, D., Hendricks, P., Rauber,
Goodfellow, I., Bengio, Y., Courville, A., and Bengio, Y. (2016), Deep Learn- J., and Long, R. (2018), “Technical Report on the CleverHans v2.1.0
ing, Cambridge, MA: MIT Press. [2196] Adversarial Examples Library,” arXiv:1610.00768. [2199]
2206 D. R. INSUA ET AL.

Papernot, N., McDaniel, P., Swami, A., and Harang, R. (2016), “Crafting Tuyls, K., and Weiss, G. (2012), “Multiagent Learning: Basics, Challenges,
Adversarial Input Sequences for Recurrent Neural Networks,” in 2016 and Prospects,” AI Magazine, 33, 41–41. [2199]
IEEE Military Communications Conference, IEEE, pp. 49–54. [2198] Vorobeychik, Y., and Kantarcioglu, M. (2018), “Adversarial Machine Learn-
Park, T., and Casella, G. (2008), “The Bayesian Lasso,” Journal of the Amer- ing,” Synthesis Lectures on Artificial Intelligence and Machine Learning,
ican Statistical Association, 103, 681–686. [2196] 12, 1–169. [2196]
Rios, J., and Insua, D. R. (2012), “Adversarial Risk Analysis for Counterter- Wang, C., Bunel, R., Dvijotham, K., Huang, P.-S., Grefenstette, E., and
rorism Modeling,” Risk Analysis: An International Journal, 32, 894–915. Kohli, P. (2019), “Knowing When to Stop: Evaluation and Verification of
[2200,2202] Conformity to Output-Size Specifications,” in Proceedings of IEEE Com-
Rios Insua, D., Banks, D., and Rios, J. (2016), “Modeling Opponents in puter Society Conference on Computer Vision and Pattern Recognition, pp.
Adversarial Risk Analysis,” Risk Analysis, 36, 742–755. [2201] 12260–12269. [2204]
Rios Insua, D., Couce-Vieira, A., Rubio, J. A., Pieters, W., Labunets, K., Welling, M., and Teh, Y. W. (2011), “Bayesian Learning via Stochastic
and G. Rasines, D. (2021), “An Adversarial Risk Analysis Framework for Gradient Langevin Dynamics,” in Proceedings of the 32nd International
Cybersecurity,” Risk Analysis, 14, 16–36. [2197] Conference on Machine Learning, pp. 681–688. [2202]
Rios Insua, D., Naveiro, R., and Gallego, V. (2020), “Perspectives on Adver- Xiao, H., Rasul, K., and Vollgraf, R. (2017), “Fashion-MNIST: A Novel
sarial Classification,” Mathematics, 8. [2202] Image Dataset for Benchmarking Machine Learning Algorithms,”
Rios Insua, D., Rios, J., and Banks, D. (2009), “Adversarial Risk Analysis,” arXiv:1708.07747. [2203]
Journal of the American Statistical Association, 104, 841–854. [2195,2199] Ye, N., and Zhu, Z. (2018), “Bayesian Adversarial Learning,” in Proceedings
Rios Insua, D., and Ruggeri, F. (2000), Robust Bayesian Analysis, of the 32nd International Conference on Advances in Neural Information
Lecture Notes in Statistics (Vol. 152), New York: Springer. [2204] Processing Systems, pp. 6892–6901, Red Hook, NY: Curran Associates
Ríos Insua, D., González-Ortega, J., Banks, D., and Ríos, J. (2018), “Concept Inc. [2201]
Uncertainty in Adversarial Statistical Decision Theory,” in The Mathe- Zeager, M. F., Sridhar, A., Fogal, N., Adams, S., Brown, D. E., and Beling,
matics of the Uncertain, eds. E. Gil, E. Gil, J. Gil, and M. Á. Gil, pp. 527– P. A. (2017), “Adversarial Learning in Credit Card Fraud Detection,” in
542, Cham: Springer. [2197] Systems and Information Engineering Design Symposium (SIEDS), 2017,
Schölkopf, B., Locatello, F., Bauer, S., Ke, N. R., Kalchbrenner, N., Goyal, IEEE, pp. 112–116. [2195]
A., and Bengio, Y. (2021), “Toward Causal Representation Learning,” Zhang, C., Zhang, K., and Li, Y. (2020a), “A Causal View on Robustness of
Proceedings of the IEEE, 109, 612–634. [2204] Neural Networks,” in Advances in Neural Information Processing Systems
Silver, D., Schrittwieser, J., Simonyan, K., Antonoglou, I., Huang, A., Guez, (Vol. 33), pp. 289–301. [2204]
A., Hubert, T., Baker, L., Lai, M., Bolton, A., et al. (2017), “Mastering the Zhang, W. E., Sheng, Q. Z., Alhazmi, A., and Li, C. (2020b), “Adversarial
Game of Go Without Human Knowledge,” Nature, 550, 354–359. [2199] Attacks on Deep-Learning Models in Natural Language Processing: A
Sutton, R. S., and Barto, A. G. (1998), Introduction to Reinforcement Learning Survey,” ACM Transactions on Intelligent Systems and Technology (TIST),
(Vol. 2), Cambridge, MA: MIT Press. [2199] 11, 1–41. [2198]
Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, Zhang, Y., Gong, M., Liu, T., Niu, G., Tian, X., Han, B., Schölkopf, B.,
I., and Fergus, R. (2014), “Intriguing Properties of Neural Networks,” in and Zhang, K. (2021), “Adversarial Robustness through the Lens of
International Conference on Learning Representations. [2196,2197] Causality,” arXiv:2106.06196. [2195,2204]