See discussions, stats, and author profiles for this publication at: [Link]

net/publication/348153427

A Step-by-Step Tutorial on Active Inference and its Application to Empirical

Data

Preprint · January 2021

DOI: 10.31234/[Link]/b4jm6

CITATIONS READS
35 3,409

3 authors:

Ryan Smith Karl J Friston

The Laureate Institute for Brain Research (LIBR) University College London
222 PUBLICATIONS 5,338 CITATIONS 1,228 PUBLICATIONS 284,398 CITATIONS

SEE PROFILE SEE PROFILE

Christopher Whyte
The University of Sydney
29 PUBLICATIONS 436 CITATIONS

SEE PROFILE

All content following this page was uploaded by Ryan Smith on 06 December 2021.

The user has requested enhancement of the downloaded file.

Active Inference Tutorial

Revised 12/4/2021

A Step-by-Step Tutorial on Active Inference and its Application to

Empirical Data

Ryan Smith1*, Karl J. Friston2, Christopher J. Whyte3*

*These authors contributed equally.

Author affiliations:

1Laureate
Institute for Brain Research, Tulsa, OK, USA.
2Wellcome Centre for Human Neuroimaging, Institute of Neurology, University College London,
WC1N 3AR, UK.
3MRC Cognition and Brain Sciences Unit, University of Cambridge, Cambridge, UK.

Corresponding author:
Ryan Smith
Laureate Institute for Brain Research
6655 S Yale Ave, Tulsa, OK 74136, USA
Email: rsmith@[Link]

1
Active Inference Tutorial

Abstract

The active inference framework, and in particular its recent formulation as a partially observable
Markov decision process (POMDP), has gained increasing popularity in recent years as a useful
approach for modelling neurocognitive processes. This framework is highly general and flexible in
its ability to be customized to model any cognitive process, as well as simulate predicted neuronal
responses based on its accompanying neural process theory. It also affords both simulation
experiments for proof of principle and behavioral modelling for empirical studies. However, there
are limited resources that explain how to build and run these models in practice, which limits their
widespread use. Most introductions assume a technical background in programming, mathematics,
and machine learning. In this paper we offer a step-by-step tutorial on how to build POMDPs, run
simulations using standard MATLAB routines, and fit these models to empirical data. We assume a
minimal background in programming and mathematics, thoroughly explain all equations, and
provide exemplar scripts that can be customized for both theoretical and empirical studies. Our
goal is to provide the reader with the requisite background knowledge and practical tools to apply
active inference to their own research. We also provide optional technical sections and multiple
appendices, which offer the interested reader additional technical details. This tutorial should
provide the reader with all the tools necessary to use these models and to follow emerging
advances in active inference research.

Keywords: Active Inference; Computational Neuroscience; Bayesian Inference; Learning; Decision-

Making; Machine Learning

2
Active Inference Tutorial

Introduction

Active inference, and in particular its recent application to partially observable Markov decision
processes (POMDPs; defined below), offers a unified mathematical framework for modelling
perception, learning, and decision making (Da Costa, Parr, et al., 2020; Friston, Parr, & de Vries,
2017; Friston, Rosch, Parr, Price, & Bowman, 2018; Parr & Friston, 2018b). This framework treats
each of these psychological processes, and their interactions, as interdependent forms of inference.
Namely, decision-making agents are assumed to infer the probability of different external states
and events in the environment – including their own actions – by combining prior beliefs with
sensory input. Unlike “passive” perceptual inference processes (e.g., inferring the presence of an
external object based on patterns of light impinging on the retina), the inferences underlying
decision-making are “active” in the sense that the agent infers the actions most likely to generate
preferred sensory input (e.g., inferring that eating some food will reduce a feeling of hunger).
Agents also infer the actions most likely to reduce uncertainty and facilitate learning (e.g., inferring
that opening the fridge will reveal available food options). This leads decision-making to favor
actions that optimize a trade-off between maximizing reward and information gain. The resulting
patterns of perception and behavior predicted by active inference match well with those observed
empirically (e.g., see (Smith, Kirlic, Stewart, Touthang, Kuplicki, McDermott, et al., 2021; Smith,
Kirlic, Stewart, Touthang, Kuplicki, Khalsa, et al., 2021; Smith, Kuplicki, Feinstein, et al., 2020; Smith,
Kuplicki, Teed, Upshaw, & Khalsa, 2020; Smith, Mayeli, et al., 2021; Smith, Schwartenbeck, Stewart,
et al., 2020)). The neural process theory associated with active inference has also successfully
reproduced empirically observed neural responses in multiple research paradigms and generated
novel, testable predictions (Friston, FitzGerald, Rigoli, Schwartenbeck, & Pezzulo, 2017;
Schwartenbeck, FitzGerald, Mathys, Dolan, & Friston, 2015; Whyte & Smith, 2020). Due to these and
other considerations, this framework has become increasingly influential in recent years within
psychology, neuroscience, and machine learning.

Over the last decade there have been many articles that offer either 1) broad intuitions about the
workings and potential implications of active inference (e.g., (Badcock, Friston, Ramstead, Ploeger,
& Hohwy, 2019; A. Clark, 2013, 2015; J. E. Clark, Watson, & Friston, 2018; Hohwy, 2014; Pezzulo,
Rigoli, & Friston, 2015, 2018; Smith, Badcock, & Friston, 2020)), or 2), technical presentations of
the mathematical formalism and how it continues to evolve (e.g., (Da Costa, Parr, et al., 2020;
Friston, FitzGerald, et al., 2016; Friston, FitzGerald, et al., 2017; Friston, Parr, et al., 2017; Hesp,
Smith, Allen, Friston, & Ramstead, 2020; Parr & Friston, 2018b)). However, for those first becoming
acquainted with this field, the former class of articles does not provide sufficient detail to instill a
thorough understanding of the framework, leading to potential misunderstanding and potentially
inaccurate empirical predictions. At the other extreme, the latter class of articles is highly technical
and requires considerable mathematical expertise, familiarity with notational conventions, and the
broader ability to translate the mathematical formalism into empirical predictions relevant to a
given field of study. This has made the active inference literature less accessible to a broader
audience who might otherwise benefit from engaging with it. To date, there are also relatively few
materials available for students seeking to gain the practical skills necessary to build active
inference models and apply them to their own research aims (although some very helpful material
has been prepared by others; e.g., Philipp Schwartenbeck [[Link]

3
Active Inference Tutorial

and Oleg Solopchuk [[Link]

30edcf50f5dc]).

The goal of this paper is to provide an accessible tutorial on the POMDP formulation of active
inference that is easy to follow for readers without upper-level undergraduate/graduate-level
training in mathematics and machine learning, while simultaneously offering basic mathematical
understanding—as well as the practical tools necessary to build and use active inference models for
their own purposes. We review the conceptual and formal foundations and provide a step-by-step
guide on how to use code in MATLAB (provided in the appendices and supplementary code) to
build active inference (POMDP) models, run simulations, fit models to empirical data, perform
model comparison, and perform further steps necessary to test hypotheses using both simulated
and fitted empirical data (all supplementary code can also be found at:
[Link] we note here that there is also a
recently developed python implementation of active inference that can be found at:
[Link] We have tried to assume as little as possible about the
reader’s background knowledge, in hope of making these methods accessible to researchers (e.g.,
psychologists and neuroscientists) without a strong background in mathematics or machine
learning. However, we have also included sections that provide additional technical detail, which
the pragmatic reader can safely skip over and still follow the practical tutorial aspects of the paper.
We have also provided additional material in appendices and supplementary code with:

1) Definitional material to help the non-expert reader who would like to attempt the technical
sections.
2) Additional mathematical detail for interested readers with a stronger technical background.
3) Pencil-and-paper exercises that help build an intuition for the behavior of these models.
4) A stripped down but well commented version of the most commonly used model inversion
script (described below) for running simulations, which can serve as a springboard for
readers seeking a deeper understanding of the code that implements these models.

Throughout the article, we will refer to the associated MATLAB code, assuming the reader is
working through the paper and the code in parallel.

While we assume as little mathematical background as possible, some limited knowledge of

probability theory, calculus, and linear algebra will be necessary to fully appreciate some sections of
the tutorial. Building models in practice also requires some basic familiarity with the MATLAB
programming environment. We realize that this background knowledge is nontrivial. However, to
minimize these potential hurdles, we 1) provide thorough explanations when presenting the
mathematics and programming (with further expansion within optional technical sections and in
Appendix 1), 2) include hands on examples/exercises in the companion MATLAB code, and 3)
provide pencil-and-paper exercises (see Appendix 2 and 𝐏𝐞𝐧𝐜𝐢𝐥_𝐚𝐧𝐝_𝐩𝐚𝐩𝐞𝐫_𝐞𝐱𝐞𝐫𝐜𝐢𝐬𝐞_𝐬𝐨𝐥𝐮𝐭𝐢𝐨𝐧𝐬. 𝐦
code) that readers can work through themselves. In total, this tutorial should offer the reader the
necessary resources to:

1) Acquire a basic understanding of the mathematical formalism.

4
Active Inference Tutorial

2) Build generative models of behavioral tasks and run simulations of both behavioral and
neural responses.
3) Fit models to behavioral data and recover model parameters on an individual basis, which
can then be used for subsequent (e.g., between-subjects) analyses.

Our hope is that this will increase the accessibility and use of this framework to a broader
audience. Note, however, that our focus is specifically on the POMDP formulation, which models
time in discrete steps and treats beliefs and actions as discrete categories (referred to as “discrete
state-space” models with “discrete time”). This means that we do not cover a number of other
topics associated with active inference and the broader free energy principle from which it is
derived. For example, we do not cover “continuous state-space” models, which can be used to
model perception of continuous variables (e.g., brightness; for a tutorial, see (Bogacz, 2017)) as
well as motor control processes (e.g., controlling continuous levels of muscle contraction; see
(Adams, Shipp, & Friston, 2013; Buckley, Sub Kim, McGregor, & Seth, 2017)). Nor do we cover
“mixed” models, in which discrete and continuous state-space models can be linked – allowing
decisions to be translated into motor commands (e.g., see (Friston, Parr, et al., 2017; Millidge, 2019;
Alexander Tschantz et al., 2021)). We also do not cover work on free energy minimization in self-
organizing systems or the basis of the free energy principle in physics. The most thorough technical
introduction to the physics perspective can be found in (Friston, 2019); a less technical (but still
rigorous) introduction is presented in (Andrews, 2020)1. Thus, the focus of this tutorial is
somewhat narrow and practical. Our aim is to equip the reader with the understanding and tools
necessary to build models in practice and apply them in their own research.

The paper is organized as follows. In Part 1, we introduce the reader to the terms, concepts, and
mathematical notation used within the active inference literature, and present the minimum
mathematics necessary for a basic understanding of the formalism (as applied in a practical
experimental setting). In Part 2, we introduce the reader to the concrete structure and elements of
POMDPs and how they are solved. In Part 3, we provide a step-by-step description of how to build
a generative model of a behavioral task (a variant on commonly used explore-exploit tasks), run
simulations using this model, and interpret the outputs of those simulations. In Part 4, we
introduce the reader to learning processes in active inference. In Part 5, we introduce the reader to
the neural process theory associated with active inference and walk the reader through generating
and interpreting the outputs of neural simulations that can be used to derive empirical predictions.

1
This other work appeals to a number of common constructs discussed in the free energy principle literature that
are also not covered here, but which the reader may have come across previously. One such construct is a
“Markov blanket”, which is a mathematical way of describing the boundary that separates the internal states of an
organism from the external environment (although note that this term is sometimes used in different ways; see
(Bruineberg, Dolega, Dewhurst, & Baltieri, 2020)). Another related construct is a “non-equilibrium steady state
(NESS) density”, which describes the states an organism must have a high probability of occupying if it is to
maintain its existence – where this can be understood as maintaining the integrity of its Markov blanket (i.e.,
keeping the boundary intact that separates an organism from its environment). The POMDP scheme described in
this tutorial does not explicitly appeal to these constructs; however, one can think of an agent’s preferred
observations in POMDPs as those that keep it within the high-probability states consistent with its continued
existence (i.e., those that would keep its Markov blanket intact).

5
Active Inference Tutorial

In Part 6, we introduce hierarchical models and illustrate how, based on the neural process theory,
they can be used to simulate established electrophysiological responses in a commonly used
auditory mismatch paradigm. Finally, in Part 7 we describe how to fit behavioral data to a model
and derive individual-level parameter estimates and how they can be used for further group-level
analyses.

1. Basic Terminology, Concepts, and Mathematics

1.1 Mathematical Foundations: Bayes’ Theorem and Active Inference

The active inference framework is based on the premise that perception and learning can be
understood as minimizing a quantity known as variational free energy (𝑉𝐹𝐸), and that action
selection, planning, and decision-making can be understood as minimizing expected free energy
(𝐸𝐹𝐸), which quantifies the 𝑉𝐹𝐸 of various actions based on expected future outcomes. To motivate
the use and derivation of these quantities, we need to first introduce the reader to Bayesian
inference and explore its relation to the notion of active inference. We will cover these foundational
principles here. By the end of this subsection, the reader should have a working knowledge of the
basic building blocks of active inference. This includes understanding what a model is, how rules
within probability theory can be used to perform inference within a model, and how this inference
process can be extended to perform action selection.

As an initial note to readers with less mathematical background, a full understanding of the
equations presented below will not be necessary to begin building models and applying them to
behavioral data. Often, building and working with models in practice is a great way to get an
intuitive grasp of the underlying mathematics. So, if some of the equations below have unfamiliar
notation and become hard to follow, do not get discouraged. An intuitive grasp of the concepts
described in this section will be enough to learn the practical applications in the subsequent
sections. That said, we also explain the equations and notation in this section assuming minimal
mathematical background.

We start by highlighting that the term ‘active inference’ is based on two concepts. The first is the
idea that organisms actively engage with (e.g., move around in) their environments to gather
information, seek out ‘preferred’ observations (e.g., food, water, shelter, social support, etc.), and
avoid non-preferred observations (e.g., tissue damage, hunger, thirst, social rejection, etc.). The
second concept is Bayesian inference, a statistical procedure that describes the optimal way to
update one’s beliefs (understood as probability distributions) when making new observations (i.e.,
receiving new sensory input) based on the rules of probability (for a brief introduction to the rules
of probability, see Appendix 1). Specifically, beliefs are updated in light of new observations using
Bayes’ theorem, which can be written as follows:

𝑝(𝑜|𝑠, 𝑚)𝑝(𝑠|𝑚)
𝑝(𝑠|𝑜, 𝑚) = (1)
𝑝(𝑜|𝑚)

6
Active Inference Tutorial

Starting on the right-hand side of the equation, the term 𝑝(𝑠|𝑚) indicates the probability (𝑝) of
different possible states (𝑠) under a model of the world (𝑚). This ‘prior belief’ (the ‘prior’)
encodes a probability distribution (‘Bayesian belief’) with respect to 𝑠 before making a new
observation (𝑜). In general, the concept of a ‘state’ is abstract and can refer to anything one might
have a belief about. For example, 𝑠 might refer to the different possible shapes of an object, such as
a square vs. a circle vs. a triangle, and so forth. The term 𝑝(𝑜|𝑠, 𝑚) is the ‘likelihood’ term and
encodes the probability within a model that one would make a particular observation if some state
were the true state (e.g., observing a straight line is consistent with a square shape but not with a
circular shape). The symbol (|) means ‘conditional on’ and is also often read as ‘given’ (e.g., the
probability of 𝑜 given 𝑠). The term 𝑝(𝑜|𝑚) is the ‘model evidence’ (also called the ‘marginal
likelihood’) and indicates how consistent an observation is with a model of the world in general
(i.e., across all possible states). Finally, the term 𝑝(𝑠|𝑜, 𝑚) is the ‘posterior’ belief, which encodes
what one’s new belief (i.e., adjusted probability distribution over possible states) optimally should
be after making a new observation.

In essence, Bayes rule describes how to optimally update one’s beliefs in light of new data.
Specifically, to arrive at a new belief (your posterior), you must: 1) take what you previously
believed (your prior), 2) combine it with what you believe about how consistent a new observation
is with different possible states (your likelihood), and 3) consider the overall consistency of that
observation with your model (i.e., how likely that observation is under any set of possible states
included in your model; the model evidence, 𝑝(𝑜|𝑚)). The last step (i.e., dividing by 𝑝(𝑜|𝑚))
ensures that your posterior belief remains a proper probability distribution that sums to 1 (i.e., it
accomplishes ‘normalization’). For a simple numerical example of Bayesian inference in the context
of perception see Figure 1.

7
Active Inference Tutorial

Figure 1. Simple example of perception as Bayesian inference (based on (Ramachandran, 1988)).

Please note that, while we have not explicitly conditioned on a model (𝑚) in the expression of
Bayes’ rule shown in the left panel (as in the text), this should be understood as implicit (i.e., priors
and likelihoods are always model-dependent). In this example, we take a ‘brain’s eye view’ and
imagine that we are presented with the shaded gray disk (the ‘observation’) in the left panel of the
figure. Due to the shading pattern, the central portion of the disk is typically perceived as concave
(as shown in the bottom-left), but it can also be perceived as convex (and typically is perceived as
convex if rotated 180o). This is because the brain is equipped with a strong (unconscious) belief
that light sources typically come from above. Given this assumption, the apparent shadow in the
upper portion of the disk is much more likely to arise from a concave surface. To capture this
mathematically, on the right we consider ‘concave’ and ‘convex’ as the two possible hidden states or
‘causes’ of sensory input (i.e., the shadow on the gray disk). We want to know whether the shadow
pattern on the disk (observation) is caused by a concave or convex surface. The optimal way to
infer the hidden state (concave or convex) is to use Bayes’ theorem. For the sake of this example,
assume we believe the chances of observing a concave vs. convex surface in general are almost
equal, with a slight bias toward expecting a convex surface (i.e., encoded in the prior distribution
shown above). The likelihood is a different story. The apparent shadow is much more consistent
with a concave surface if light is coming from above (i.e., encoded in the likelihood distribution). To
infer the posterior probability, we multiply the likelihood and prior probabilities, giving us the joint
distribution. We then sum the probabilities in the joint distribution, yielding the total probability of
the observation across the possible hidden states (i.e., the marginal likelihood). Finally, we divide
the joint distribution by the marginal likelihood to reach the posterior. The posterior tells us that
the most probable hidden state is a concave surface (i.e., corresponding to what is most often
perceived). Thus, even though the two-dimensional gray disk alone is equally consistent with a
convex or concave surface, the assumption that light is coming from above (encoded in the
likelihood) most often leads us to perceive a concave 3-dimensional shape.

In this tutorial, the concept of a model is key. As briefly introduced above, we here focus specifically
on generative models, which are models of how observations (sensory inputs) are generated by
objects and events outside of the brain that cannot be known directly (typically termed ‘hidden
states’ or ‘hidden causes’; e.g., a baseball generating a specific pattern of activation on the retina).
In simple generative models (i.e., not yet incorporating action), the necessary variables correspond
to those presented within Bayes’ theorem above (although note that conditioning on the model
variable 𝑚 is often left implicit, as we will also do going forward). That is, each model includes a set
of possible hidden states (𝑠), priors over those states 𝑝(𝑠), a set of possible observations (𝑜; also
called ‘outcomes’), and a likelihood that specifies how states generate observations 𝑝(𝑜|𝑠). The
notion of hidden or unobservable states causing observable outcomes illustrates how inference can
be seen as a type of model inversion. Namely, updating one’s beliefs from prior to posterior beliefs
is like inverting the likelihood mapping – that is, moving from 𝑝(𝑜|𝑠) to 𝑝(𝑠|𝑜). In other words,
starting with a mapping from causes to consequences and then using it to infer the causes from
consequences.

8
Active Inference Tutorial

Importantly, models can also include multiple types/sets of states (i.e., different state spaces). For
example, one set of states could encode possible shapes, while another set of states could encode
possible object locations. When different sets of states are independent in this way, each set is
called a different ‘hidden state factor’. Similarly, models can include multiple types/sets of
observable outcomes. For example, one set of possible observations could come from vision, while
another set of possible observations could come from audition. When sets of observable outcomes
are independent in this way, each set is called a different ‘outcome modality’. Once all sets of
possible states and observations are specified, the generative model is defined in terms of the joint
distribution 𝑝(𝑜, 𝑠) – that is, the probability distribution over all possible combinations of states
and observations. Based on the product rule in probability theory (see Appendix 1), this can be
decomposed into the separate terms just mentioned:

𝑝(𝑜, 𝑠) = 𝑝(𝑜|𝑠)𝑝(𝑠) (2)

If there is only one set of states and observations, this joint distribution is a 2-dimensional
distribution. If there are more sets, it becomes a higher-dimensional distribution that, while harder
to visualize (and more time consuming to compute), can be treated in the same way.

A crucial point to keep in mind at this point is the distinction between a generative model and the
generative process (see Figure 2). A generative model, as discussed above, is constituted by
beliefs about the world and can be inaccurate (sometimes referred to as ‘fictive’). In other words,
explanations for (i.e., beliefs about) how observations are generated do not have to represent a
veridical account of how they are actually generated. Indeed, explanations for sensory data within
models are often simpler than the true processes generating those data. In contrast, the generative
process refers to what is actually going on out in the world – that is, it describes the veridical
‘ground truth’ about the causes of sensory input. For example, a model might hold the prior belief
that the probability of seeing a pigeon vs. a hawk while at a city park is [.9 .1], whereas the true
probability in the generative process may instead be [.7 .3]. This distinction is important in practical
uses of modelling when one wants to simulate behavior under false beliefs and unexpected
observations (e.g., when modelling delusions or hallucinations).

9
Active Inference Tutorial

Figure 2. Visual depiction of the distinction between the generative process and the generative
model, as well as their implicit coupling in the perception-action cycle. The generative process
describes the true causes of the observations that are received by the generative model, which
inform posterior beliefs about those causes (i.e., perception). In active inference, a generative model
also includes policies (𝜋), where each policy is a possible sequence of actions (𝑢) that could be
selected. The policy with the highest posterior probability (given preferred outcomes) is typically
chosen, which couples the agent back to the generative process by changing the true state of world
through action.

While Bayesian inference represents the optimal way to infer posterior beliefs within a generative
model, Bayes theorem is computationally intractable for anything but the simplest distributions.
This is because evaluating 𝑝(𝑜|𝑚) – the marginal likelihood (denominator) in Bayes’ theorem –
requires us to sum the probabilities of observations under all possible states in the generative
model (i.e., based on the sum rule of probability; see Appendix 1 and Figure 1). For discrete
distributions, as the number of dimensions (and possible values) increases, the number of terms
that must be summed increases exponentially. In the case of continuous distributions, it requires
the evaluation of integrals that do not always have closed-form (analytic) solutions. As such,
approximation techniques are required to solve this problem. This is where 𝑉𝐹𝐸 is crucial, as it
provides a computationally tractable quantity that allows for approximate inference. One common
way this is explained requires the introduction of an information-theoretic quantity known as self-
information or surprisal (often also just called ‘surprise’, but we avoid this term here to minimize
confusion with psychological surprise). Surprisal reflects a deviation between observed outcomes
and those predicted by a model. It is typically written as the negative log-probability of that
observation, −ln𝑝(𝑜|𝑚), where ln is the natural logarithm. Consistent with the intuitive notion of
surprise, lower probability events generate higher surprisal values (e.g., −ln(. 5) = 0.69, while −

10
Active Inference Tutorial

ln(.9) = 0.1). It therefore follows that minimizing surprisal is equivalent to maximizing the
evidence an observation provides for a model; i.e., 𝑝(𝑜|𝑚). As will be demonstrated in the next
section, 𝑉𝐹𝐸 is always greater than or equal to surprisal, which means that minimizing 𝑉𝐹𝐸 is also
a means of maximizing model evidence. This gets around the problem of computational
intractability mentioned above and allows for inference of posterior beliefs over states. Figure 3
provides an example of inference using Bayes’ theorem and using minimization of 𝑉𝐹𝐸 (see the
next section for an introduction to the relevant mathematics).

Figure 3. Simple example of exact versus approximate Bayesian inference. As with the example in
Figure 1, we are given a prior belief over states 𝑝(𝑠) and the likelihood of a new observation
𝑝(𝑜|𝑠), and we wish to infer the posterior probability over states given that new observation
𝑝(𝑠|𝑜). Exact inference requires the evaluation of the marginal likelihood 𝑝(𝑜), which, for anything
but the simplest distributions, is either computationally intensive or intractable. Instead,
variational inference minimizes 𝑉𝐹𝐸 (here denoted by 𝐹), which scores the difference between an
(initially arbitrary) approximate posterior distribution 𝑞(𝑠) and a target distribution (here the
exact posterior; for an introduction to variational inference, see Appendix 1). By iteratively
updating the approximate posterior to minimize 𝐹 (usually via gradient descent, see main text for
details), a distribution can be found that approximates the exact posterior. That is, 𝑞(𝑠) will
approximate the true posterior when it produces a minimum value for 𝐹. Here, we have shown an
example of iterative updating for the simplest distribution possible to illustrate the concept. As
shown in the bottom-left, in this example we start the agent with an approximate posterior
distribution 𝑞(𝑠) = 𝑝(𝑠) = [. 5 .5]T – which can be thought of as an initial guess about what the true

11
Active Inference Tutorial

posterior belief 𝑝(𝑠|𝑜) should be after making a new observation (𝑜). We then define a generative
model with the joint probability, 𝑝(𝑜, 𝑠), where the true posterior we wish to find is 𝑝(𝑠|𝑜) =
[. 8 .2]T for the observation 𝑜 (as calculated using exact inference in the top panel). On the bottom-
right, under ‘Initial F’, we first solve for 𝐹 using our initial 𝑞(𝑠). Under ‘Update 1’, we then find (by
searching neighboring values) a nearby value for 𝑞(𝑠) that leads to a lower value for 𝐹, and we
repeat this process in ‘Update 2’. In ‘Update 3’, 𝑞(𝑠) is equal to 𝑝(𝑠|𝑜), which corresponds to finding
a minimum value for 𝐹 (i.e., where the remaining value above zero corresponds to surprisal,
−ln𝑝(𝑜)). The fact that 𝐹 has reached a minimum can be seen in ‘Update 4’, where continuing to
change 𝑞(𝑠) causes 𝐹 to again increase in value. Thus, by finding the posterior 𝑞(𝑠) over states that
generates the minimum for 𝐹, that 𝑞(𝑠) will also best approximate the true posterior. In the
supplementary code we have included a script 𝐕𝐅𝐄_𝐜𝐚𝐥𝐜𝐮𝐥𝐚𝐭𝐢𝐨𝐧_𝐞𝐱𝐚𝐦𝐩𝐥𝐞. 𝐦 that will allow you
to define your own priors, likelihoods, and observations and calculate 𝐹 for different 𝑞(𝑠) values.

Although Bayesian inference is used to model perception and learning in related frameworks (e.g.,
predictive coding; see (Bogacz, 2017)), the active inference approach covered in this tutorial
extends this application of Bayesian inference in two ways. First, it models categorical inference
(e.g., the presence of a cat vs. a dog), as opposed to continuous inference (i.e., variables that take a
continuous range of values, such as speed, direction of motion, brightness, etc.). Second, it models
the inference of optimal action sequences during decision-making (i.e., inferring a probability
distribution over possible action options, which can be thought of as encoding the estimated
probability of achieving one’s goals if each action were chosen). In planning, possible sequences of
actions (called ‘policies’)2 are denoted by the Greek letter pi (π), so the generative model is
extended to:

𝑝(𝑜, 𝑠, 𝜋) = 𝑝(𝑜|𝑠, 𝜋)𝑝(𝑠|𝜋)𝑝(𝜋) (3)

We will return to the prior over policies 𝑝(𝜋) later. For now, we simply note that active inference
models can include additional elements that control (for example) how much randomness is
present in decision-making and how habits can be acquired and influence decisions. They can also
be extended to include learning. We will return to these extensions in later sections.

To make decisions, an agent requires a means of assigning higher value to one policy over another.
This in turn requires that some observations are preferred over others. One of the more
(superficially) counterintuitive aspects of active inference is the way it formalizes preferences. This
is because there are no additional variables labelled as ‘rewards’ or ‘values’. Instead, preferences
are encoded within a specific type of prior probability distribution – which is often called a ‘prior
preference distribution’. This distribution is often simply denoted as 𝑝(𝑜); however, the term

2
It is important to note that the term ‘policy’ in active inference is used in a different way than in model-free
reinforcement learning. As stated in the main text, a policy in active inference refers to an allowable sequence of
actions (e.g., a plan to move to state 1, then to state 2, then to state 3). In contrast, a policy in model-free
reinforcement learning typically consists of a mapping from states to actions. That is, a policy specifies the action
that should be chosen for each possible state an agent might occupy (e.g., if in state 1, move to state 2; if in state
3, move to state 1, etc.).

12
Active Inference Tutorial

𝑝(𝑜) is also used in other ways, which can be a source of confusion. Therefore, we will instead
represent this distribution as 𝑝(𝑜|𝐶), where the variable 𝐶 denotes the agent’s preferences. In this
distribution, observations with higher probabilities are treated as more rewarding. Note that this is
distinct from priors over states, 𝑝(𝑠), which encode beliefs about the true states of the world (i.e.,
irrespective of what is preferred).

The value of each policy in active inference is also specified within a probability distribution, where
a higher value corresponds to a higher probability of being selected. This probability is based on
𝐸𝐹𝐸 (i.e., lower 𝐸𝐹𝐸 indicates higher value) and reflects beliefs about how likely each policy is to
generate preferred observations (and how effective it is expected to be at maximizing information
gain; discussed further below). In one sense, the use of probability distributions to encode
preferences and policy values can simply be considered a kind of mathematical ‘trick’ to bring all
elements of action selection within the domain of Bayesian belief updating—a kind of planning as
inference (H Attias, 2003; Botvinick & Toussaint, 2012; Kaplan & Friston, 2018). However, many
articles have considered the possibility (or use language suggesting) that the formalism may have
deeper implications. Specifically, the active inference literature often discusses how prior
preferences may be thought of as encoding the observations that are implicitly ‘expected’ by an
organism in virtue of its phenotype (i.e., the observations an organism must seek out to maintain its
survival and/or reproduction). For example, consider body temperature. Humans can only survive
if body temperatures continue to be observed within the range of 36.5 – 37.5 degrees Celsius. Thus,
the human phenotype implicitly entails a high prior probability of making such observations. If a
human agent perceives (i.e., infers) that body temperature has (or is going to) deviate from
‘expected’ temperatures, it will infer which policies are most likely to minimize this deviation (e.g.,
seeking shelter when it is cold or is expected to become cold). In this sense, body temperatures
within survivable ranges (for the human phenotype) are the least ‘surprising’. In formal terms, the
variable 𝐶 in prior preferences 𝑝(𝑜|𝐶) can therefore be thought of as standing in for a model of an
organism’s phenotype, where this model predicts specific (internal and external) observations
consistent with that phenotype and motivates actions expected to maintain those observations.

However, it is important to highlight that this formal treatment of preferences and values as
‘Bayesian beliefs’ (i.e., probability distributions) need not be understood as a psychological
description, nor must it be if one wishes to use active inference models in practice. In other words,
not all beliefs at the mathematical level of description need to be equated with beliefs at the
psychological level; some Bayesian beliefs in the formalism can instead correspond to rewarding or
desired outcomes at the psychological level (Smith, Ramstead, & Kiefer, 2021). Similarly, the notion
of ‘surprise’ with respect to prior preferences is not equivalent to the conscious experience of
surprise; minimizing the type of ‘phenotypic surprise’ discussed in active inference is better
mapped onto psychological states associated with achieving one’s goals. Traditional beliefs (in the
psychological sense) can instead be identified with priors/posteriors over states 𝑝(𝑠) and with the
observations expected given policies, 𝑝(𝑜|𝜋). However, regardless of the way one views the
mapping between mathematical and psychological levels of description, active inference can more
broadly (and less controversially) be seen as suggesting that the brain just is (or that it ‘implements’
or ‘entails’) a generative model of the body and external environment of the organism.

13
Active Inference Tutorial

1.2 Non-Technical Introduction to Solving Partially Observable Markov Decision Processes via Free
Energy Minimization

In this subsection, we expand on the need for approximate inference in cases where Bayes’ theorem
cannot be computed directly and explain how this motivates the use of 𝑉𝐹𝐸 and 𝐸𝐹𝐸. By the end of
this section, the reader should have a basic understanding of how 𝑉𝐹𝐸 can be used to perform
approximate Bayesian inference within a generative model and of how 𝐸𝐹𝐸 extends this approach
to infer optimal choices.

The specific type of generative model used here is a partially observable Markov decision
process (POMDP). A Markov decision process describes beliefs about abstract states of the world,
how they are expected to change over time, and how actions are selected to seek out preferred
outcomes or rewards based on beliefs about states. This class of models assumes the ‘Markov
property’, which simply means that beliefs about the current state of the world are all that matter
for an agent when deciding which actions to take (i.e., that all knowledge about past states is
implicitly ‘packed into’ beliefs about the current state). The agent then uses its model, combined
with beliefs about the current state, to select actions by making predictions about possible future
states. To be ‘partially observable’ means that the agent can be uncertain in its beliefs about the
state of the world it is in. In this case, states are referred to as ‘hidden’ (as introduced above). The
agent must infer how likely it is to be in one hidden state or another based on observations (i.e.,
sensory input) and use this information to select actions.

In active inference, these tasks are solved using a form of approximate inference known as
variational inference ((Hagai Attias, 2000; Beal, 2003; Parr, Markovic, Kiebel, & Friston, 2019); for
a brief introduction, see Appendix 1). Broadly speaking, the idea behind variational inference is to
convert the intractable sum or integral required to perform model inversion into an optimization
problem that can be solved in a computationally efficient manner. This is accomplished by
introducing an approximate posterior distribution over states (denoted 𝑞(𝑠)) that makes
simplifying assumptions about the nature of the true posterior distribution. For example, it is
common to assume that hidden states under the approximate distribution do not interact (i.e., are
independent), which gives the approximate distribution a much simpler mathematical form. Such
assumptions are often violated, but the approximation is usually good enough in practice. Note that,
despite its aim of approximating the true posterior 𝑝(𝑠|𝑜), the approximate posterior 𝑞(𝑠) is
typically not written as being conditioned on observations. This is because it does not directly
depend on observations – it is simply an (initially arbitrary) distribution over states that is
iteratively updated to match the true posterior distribution as closely as possible (described
below).

After introducing 𝑞(𝑠), the next step in variational inference is to measure the similarity between
this distribution and the generative model, 𝑝(𝑜, 𝑠), using a measure called the Kullback–Leibler
(KL) divergence. We will discuss the KL divergence in more detail in the following (technical)
section. For now, it is sufficient to think of the KL divergence as a measure of the dissimilarity
between two distributions. It is zero when the distributions match, and it gets larger the more
dissimilar the distributions become. 𝑉𝐹𝐸 corresponds to the surprisal we want to minimize plus

14
Active Inference Tutorial

the KL divergence between the approximate distribution and the posterior distribution. In
variational inference, we systematically update 𝑞(𝑠) until we find the value that minimizes 𝑉𝐹𝐸, at
which point 𝑞(𝑠) will approximate the true posterior, 𝑝(𝑠|𝑜). In Figure 3 we provide a simple
example of calculating 𝑉𝐹𝐸 under different values for 𝑞(𝑠) to provide the reader with an intuition
for how this works (this example can also be reproduced and customized in the
𝐕𝐅𝐄_𝐜𝐚𝐥𝐜𝐮𝐥𝐚𝐭𝐢𝐨𝐧_𝐞𝐱𝐚𝐦𝐩𝐥𝐞. 𝐦 script provided in the supplementary code). For ease of
calculation, this figure uses the following expression for 𝑉𝐹𝐸 (note that 𝑉𝐹𝐸 = 𝐹 when presented
in equations):

𝑞(𝑠)
𝐹 = ∑ 𝑞(𝑠)ln (4)
𝑝(𝑜, 𝑠)
𝑠∈𝑆

However, this expression does not make it obvious how minimizing 𝑉𝐹𝐸 will lead 𝑞(𝑠) to
approximate the true posterior. As discussed further in the technical sections, this can be seen more
clearly by algebraically manipulating 𝑉𝐹𝐸 into the following form, which is more often seen in the
active inference literature:

𝑞(𝑠)
𝐹 = E𝑞(𝑠) [ln ] − ln𝑝(𝑜) (5)
𝑝(𝑠|𝑜)

For details on how we move between different expressions for 𝐹, see the optional technical section
(Section 1.3). Here the E𝑞(𝑠) term indicates the expected value or ‘expectation’ of a distribution
𝑞(𝑠)
and is equivalent to the ∑𝑠∈𝑆 𝑞(𝑠) term in equation 4. It indicates that 𝑞(𝑠)[ln 𝑝(𝑠|𝑜)] is evaluated for
each value of 𝑞(𝑠) and then the resulting values are summed (see numerical example in Figure 3).
Based on this form of the equation, we can see that, because ln𝑝(𝑜) does not depend on 𝑞(𝑠), the
value of 𝐹 will become smaller as the value of 𝑞(𝑠) approaches the value of the true posterior,
𝑝(𝑠|𝑜) – since the former is divided by the latter and the log of one is zero.

Within the active inference framework, the task of both perception and learning is to minimize 𝑉𝐹𝐸
in order to find (approximately) optimal posterior beliefs after each new observation. Perception
corresponds to posterior state inference after each new observation, while learning corresponds to
more slowly updating the priors and likelihood distributions in the model over many observations
(which facilitates more accurate state inference in the long run). It is important to note, however,
that minimizing 𝑉𝐹𝐸 is not simply a process of finding the best-fitting approximation on every trial.
Sensory input is inherently noisy, and simply finding the best-fitting posterior on each trial would
lead to fitting noise, which would result in exaggerated and metabolically costly updates. In
statistics, this is known as overfitting. Fortunately, 𝑉𝐹𝐸 minimization naturally avoids this problem.
Put into words, 𝑉𝐹𝐸 measures the complexity of a model minus the accuracy of that model. Here,
the term ‘accuracy’ refers to how well a model’s beliefs predict sensory input (i.e., the goodness of
fit), while the term ‘complexity’ refers to how much beliefs need to change to maintain high
accuracy when new sensory input is received (i.e., 𝑉𝐹𝐸 remains higher if beliefs need to change a
lot to account for new sensory input). Perception therefore seeks to find the most parsimonious

15
Active Inference Tutorial

(smallest necessary) changes in beliefs about the causes of sensory input that can adequately
explain that input.

Analogously, the task of action selection and planning is to select policies that will bring about
future observations that minimize 𝑉𝐹𝐸. The problem is, of course, that future outcomes have not
yet been observed. Actions must therefore be selected such that they minimize expected free energy
(𝐸𝐹𝐸). Crucially, 𝐸𝐹𝐸 scores the expected cost (i.e., a lower value indicates higher reward) minus
the expected information gain of an action. This means that decisions that minimize 𝐸𝐹𝐸 seek to
both maximize reward and resolve uncertainty. When beliefs about states are very imprecise or
uncertain, actions will tend to be information-seeking. Conversely, selected actions will tend to be
reward-seeking when confidence in beliefs about states is high (i.e., when there is no more
uncertainty to resolve and the agent is confident about what to do to bring about preferred
outcomes).

However, as we will see later, if the magnitude of expected reward is sufficiently high (i.e., if a
preference distribution is highly precise), actions that minimize 𝐸𝐹𝐸 can become ‘risky’ – in that
they seek out reward in the absence of sufficient information (i.e., reward value outweighs
information value). In general, the imperative to minimize 𝐸𝐹𝐸 is especially powerful in accounting
for commonly observed behaviors in which, instead of seeking immediate reward, organisms first
gather information and then maximize reward when they are confident about states of the world
(e.g., turning on a light before trying to find food). It can also capture interesting behaviors that
occur in the absence of opportunities for reward, where organisms appear to act simply out of
‘curiosity’ (Barto, Mirolli, & Baldassarre, 2013; Oudeyer & Kaplan, 2007; Schmidhuber, 2006).
Further, variations in the precision of preferences during 𝐸𝐹𝐸 minimization can capture interesting
individual differences in behavior, as exemplified in the example of ‘risky’ behavior just mentioned.
Note that this crucial aspect of active inference effectively addresses the ‘explore-exploit dilemma’
(discussed further below), because the imperatives for exploration (information-seeking) and
exploitation (reward-seeking) are just two aspects of expected free energy, and whether
exploratory or exploitative behaviors are favored in a given situation depends on current levels of
uncertainty and the level of expected reward.

As we shall see in later sections, the posterior over policies is informed by both 𝑉𝐹𝐸 and 𝐸𝐹𝐸. For
now, we simply note that this is because 𝑉𝐹𝐸 is a measure of the free energy of the present (and
implicitly the past), while 𝐸𝐹𝐸 is a measure of the free energy of the future. This is important,
because while some policies may lead to a minimization of free energy in the future, they may not
have led to a minimization of free energy in the past (and are therefore suboptimal policies when
evaluated overall). In other words, 𝐸𝐹𝐸 scores the likelihood of pursuing (i.e., the value assigned to)
a particular course of action based upon future outcomes, while 𝑉𝐹𝐸 reflects the likelihood of (i.e.,
the value assigned to) a course of action based upon past outcomes. This means the posterior
distribution over policies is a function of both 𝑉𝐹𝐸 and 𝐸𝐹𝐸, where these quantities respectively
furnish retrospective and prospective policy evaluations. At the psychological level, one can
intuitively think of 𝑉𝐹𝐸 as asking “how good has this action plan turned out so far?”, while 𝐸𝐹𝐸
asks “how good do I expect things to go if I continue to follow this action plan?”.

16
Active Inference Tutorial

When viewed from the perspective of potential neuroscientific applications, another crucial benefit
of 𝑉𝐹𝐸 and 𝐸𝐹𝐸 is that they can be computed in a biologically plausible manner. This has inspired
neural process theories that specify ways in which sets of neuron-like nodes (e.g., neuronal
populations), with particular patterns of synapse-like connection strengths, can implement
perception, learning, and decision-making through the minimization of these quantities. These
neural process theories postulate several neuronal populations whose activity represents:

1) Categorical probability distributions (a special case of the multinomial distribution)

over the possible states of the world. For those without background in probability
theory, these distributions assign one probability value to each possible interpretation
of sensory input, and all these probability values must add up to a value of 1. In other
words, this assumes the world must be in this state or that state, but not both at the
same time, and assigns one probability value to the world being in each possible state.
(Note: in other papers you may come across the notation 𝐶𝑎𝑡(𝑥), which simply indicates
that a distribution 𝑥 is a categorical distribution).
2) Prediction-errors, which signal the degree to which sensory input is inconsistent with
current beliefs. Prediction errors drive the system to find new beliefs – that is, adjusted
probability distributions – so that they are more consistent with sensory input, and
therefore minimize these error signals.
3) Categorical probability distributions over possible policies the agent might choose.

These neural process theories also include simple, coincidence-based learning mechanisms that
can be understood in terms of Hebbian synaptic plasticity (i.e., which involve adjusting the strength
of the connections between two neurons when both neurons are activated simultaneously; (T. H.
Brown, Zhao, & Leung, 2010)). They further incorporate message passing algorithms (discussed
in detail below) that can model the connectivity and firing rate patterns of neuronal populations
organized into cortical columns. Such theories afford precise quantitative predictions that can be
tested using neuroimaging and other electrophysiological measures of neuronal activation during
specific experimental paradigms.

1.3 Technical Introduction to Variational Free energy and Expected Free Energy (optional)

In the previous subsections we introduced two quantities, EFE and VFE, which were described in
largely qualitative terms. In this section, we consider the formal details behind the above
descriptions. By the end of this subsection, the reader should have a working understanding of the
different ways that 𝑉𝐹𝐸 and 𝐸𝐹𝐸 are often expressed in the literature, how these expressions are
derived, and the theoretical insights that each expression provides. Readers without strong
mathematical background can safely skip much of this section (if so desired) and move on to the
next section without significant loss of understanding. That is, they should still be able to follow the
rest of the paper. For those who choose to read this section, we provide accessible explanations of
each equation in the hope that as many people as possible will be able to follow and learn from the
material.

17
Active Inference Tutorial

Before formally defining 𝑉𝐹𝐸 and 𝐸𝐹𝐸, it will be helpful to first familiarize the reader with the
relevant machinery. We will first expand on the KL divergence (𝐷𝐾𝐿 ; also sometimes called
relative entropy), which was briefly introduced in the previous subsection. To remind the reader,
this is a measure of the similarity, or dissimilarity, between two distributions. The KL divergence
between two distributions, 𝑞(𝑥) and 𝑝(𝑥), is written as follows:

𝑞(𝑥)
𝐷𝐾𝐿 [𝑞(𝑥)||𝑝(𝑥)] = ∑ 𝑞(𝑥)ln (6)
𝑝(𝑥)
𝑥∈𝑋

This equation states that the KL divergence is found by taking each value of 𝑥 in range 𝑋 (for which
𝑝 and 𝑞 assign values), calculating the value of the right-hand quantity, and then summing the
resulting values (for a concrete numerical example, see the calculation of 𝐹 in Figure 3). From the
perspective of information theory, the KL divergence can be thought of as scoring the amount of
information we would need to reconstruct 𝑝(𝑥) given full knowledge of 𝑞(𝑥). Note that
‘information’ is here measured in a quantity called nats, because we are working with the natural
log, as opposed to log base 2 where information would be quantified in bits.

As mentioned above, because calculating model evidence is generally not possible, we instead
minimize 𝑉𝐹𝐸, which is constructed to be an upper bound on negative (log) model evidence. As we
noted above, this is also called ‘surprisal’ in information theory: −ln𝑝(𝑜). In other words, by
minimizing 𝑉𝐹𝐸, one can minimize the negative model evidence – or, more intuitively, one can
maximize model evidence (i.e., by finding beliefs in a model for which observations provide the
most evidence).

Before turning to the definition of 𝑉𝐹𝐸, it will be helpful to clarify some notational conventions and
concepts. Specifically, using the sum rule of probability we can express model evidence in terms our
generative model as: 𝑝(𝑜) = ∑𝑠,𝜋 𝑝(𝑜, 𝑠, 𝜋). That is, 𝑝(𝑜) is the sum of the probabilities of
observations for every combination of states and policies in the model. Next, one can multiply and
divide the joint distribution, 𝑝(𝑜, 𝑠, 𝜋), by the (initially arbitrary) approximate distribution 𝑞(𝑠, 𝜋).
By definition, multiplying and then dividing by 𝑞(𝑠, 𝜋) does not change the value of this distribution.
However, this trick ends up being quite useful as we will see. For mathematical convenience, one
can take the negative logarithm of the resulting term, leading to:

𝑝(𝑜, 𝑠, 𝜋)𝑞(𝑠, 𝜋) 𝑝(𝑜, 𝑠, 𝜋)

−ln𝑝(𝑜) = −ln ∑ = − lnE𝑞(𝑠,𝜋) [ ] (7)
𝑞(𝑠, 𝜋) 𝑞(𝑠, 𝜋)
𝑠,𝜋

As briefly mentioned in the previous subsection, E𝑞(𝑠,𝜋) denotes the expected value or expectation
of a distribution. This can be thought as a kind of weighted average, where each value of one
distribution (here 𝑞(𝑠, 𝜋)) is multiplied by the associated value in another distribution (here
𝑝(𝑜,𝑠,𝜋)
[ 𝑞(𝑠,𝜋) ]), and each of the resulting values is summed to get the expected value of the latter
distribution. Note that, although written as a summation, the calculation of F in Figure 3 represents
a numerical example. Readers should note the formal similarity between the KL divergence and the
expectation value. This is no accident, as the KL divergence is simply the expected difference of the

18
Active Inference Tutorial

log of two distributions, where the expectation is taken with respect to the distribution in the
𝑥 𝑦
numerator. Note that, because ln 𝑦 = −ln 𝑥, the distributions in the numerator are often swapped
around, as we have done below. In the active inference literature, it is very common to move
between KL divergence notation and expectation notation. As such, it can be useful to keep the
following identities in mind:

𝑞(𝑥) 𝑞(𝑥) 𝑝(𝑥)

𝐷𝐾𝐿 [𝑞(𝑥)||𝑝(𝑥)] = E𝑞(𝑥) [ln ] = ∑ 𝑞(𝑥)ln = − ∑ 𝑞(𝑥)ln (8)
𝑝(𝑥) 𝑝(𝑥) 𝑞(𝑥)
𝑥∈𝑋 𝑥∈𝑋

With a clear idea of expectation values and the KL divergence now in mind, we move onto the
definition of 𝑉𝐹𝐸. Specifically, leveraging the mathematical result referred to as Jensen’s
inequality (Kuczma & Gilá nyi, 2009) – which states that the expectation of a logarithm is always
less than or equal to the logarithm of an expectation – we arrive at the following inequality:

𝑝(𝑜, 𝑠, 𝜋) 𝑝(𝑜, 𝑠, 𝜋)
−ln𝑝(𝑜) = −lnE𝑞(𝑠,𝜋) [ ] ≤ −E𝑞(𝑠,𝜋) [ln ]=𝐹 (9)
𝑞(𝑠, 𝜋) 𝑞(𝑠, 𝜋)

On the right-hand side of this equation is 𝑉𝐹𝐸 (always denoted by F in equations), which is defined
in terms of the KL divergence – that is, expected difference of the respective logs – between the
generative model 𝑝(𝑜, 𝑠, 𝜋) and the approximate posterior distribution 𝑞(𝑠, 𝜋). When the
approximate posterior distribution and the generative model match, 𝑉𝐹𝐸 is equal to zero (i.e., when
p(o,s,𝜋)
𝑞 = 𝑝, ln ( q(s,𝜋) ) = 0). To understand the equality on the left-hand side, consider that the
𝑝(𝑜,𝑠,𝜋)
expectation, −lnE𝑞(𝑠,𝜋) [ 𝑞(𝑠,𝜋)
], entails summing over all values of 𝑠 and 𝜋; that is,
𝑝(𝑜,𝑠,𝜋)
−ln ∑𝑠,𝜋 𝑞(𝑠, 𝜋) [
𝑞(𝑠,𝜋)
]. This removes 𝑠 and 𝜋 from the expression in both the denominator and the
numerator, leaving −ln𝑝(𝑜). From this, we can see that, by minimizing 𝑉𝐹𝐸, we minimize an upper
bound on negative log model evidence (here, with respect to states under each policy). This means
that 𝑉𝐹𝐸 will always be greater than or equal to −ln𝑝(𝑜), which entails that by minimizing the
value of 𝑉𝐹𝐸, the model evidence 𝑝(𝑜) will either increase or remain the same (i.e., it will be
maximized, as the logarithm is a monotonically increasing function).

Therefore, all that is needed to perform approximate Bayesian inference in perception, learning,
and decision-making is a tractable approach to finding the value of 𝑠 (i.e., the approximate posterior
distribution over 𝑠) that minimizes 𝑉𝐹𝐸. This can be accomplished by performing a gradient
descent on 𝑉𝐹𝐸. Gradient descent is a technique that starts by picking some initial value for 𝑠 and
then calculates 𝑉𝐹𝐸 for this value. It then calculates 𝑉𝐹𝐸 for neighboring values of 𝑠 and identifies
the neighboring values for which 𝑉𝐹𝐸 decreases most. It then samples from values of 𝑠 that
neighbor those values and continues to do so iteratively until a minimum 𝑉𝐹𝐸 value is found (i.e.,
where 𝑉𝐹𝐸 no longer decreases for any neighboring values). At this point, an approximation to the
optimal beliefs for 𝑠 has been found (given some set of observations 𝑜). On a final terminological
note, because 𝑉𝐹𝐸 is a function that is defined in terms of probability distributions, which are
themselves functions, 𝑉𝐹𝐸 is sometimes referred to as a ‘functional’, which is simply the
mathematical term for a function of a function.

19
Active Inference Tutorial

Note that in active inference we calculate 𝑉𝐹𝐸 with respect to each available policy individually
(denoted by 𝐹𝜋 ). This is because different policies, through their impact on hidden states in the
generative process, make certain observations more likely than others. For example, consider a
situation where I believe there is a chair to my left and a table to my right. Conditional on having
chosen to look left, it is more likely that I will observe a chair than a table. This means that
observing the chair acts as evidence that I have chosen the policy of looking left. Because
observations provide evidence for policies in this way, both the approximate posterior 𝑞(𝑠|𝜋), and
the generative model 𝑝(𝑜, 𝑠|𝜋), are conditioned on policies. This may be useful in some cases where,
for example, one is considering the possibility that an agent could have false beliefs about the
actions they are carrying out or could be surprised when their intended policy does not match the
true observed actions. Going forward, 𝑉𝐹𝐸 will be presented in terms of 𝐹𝜋 (as shown in the
following paragraph).

As we will discuss below, one way in which the brain may accomplish gradient descent on 𝑉𝐹𝐸
during perception is through the minimization of prediction error. The reason for this can be
brought out by doing some algebraic rearrangement to express 𝑉𝐹𝐸 as a measure of complexity
minus accuracy (i.e., as touched upon earlier):

𝑞(𝑠|𝜋)
𝐹𝜋 = E𝑞(𝑠|𝜋) [ln ] (10)
𝑝(𝑜, 𝑠|𝜋)

= Ε𝑞(𝑠|𝜋) [ln 𝑞(𝑠|𝜋) − ln 𝑝(𝑜, 𝑠|𝜋)] L2

= Ε𝑞(𝑠|𝜋) [ln 𝑞(𝑠|𝜋) − ln 𝑝(𝑠|𝜋)] − Ε𝑞(𝑠|𝜋) [ln 𝑝(𝑜|𝑠, 𝜋)] L3

= 𝐷𝐾𝐿 [𝑞(𝑠|𝜋)||𝑝(𝑠|𝜋)] − E𝑞(𝑠|𝜋) [ln𝑝(𝑜|𝑠)] L4

The first line expresses 𝑉𝐹𝐸 in terms of the expected log difference (KL divergence) between the
approximate posterior and the generative model. In the second line we use log algebra to express
𝑥
the division as a subtraction (ln 𝑦 = ln 𝑥 − ln y). In the third line we use the product rule of
probability (𝑝(𝑜, 𝑠|𝜋) = 𝑝(𝑠|𝜋)𝑝(𝑜|𝑠, 𝜋)) to take the likelihood term out of the first expectation
term. The fourth line re-expresses the third line, but uses more compact notation for the first term
and drops the dependency on policies in the second term (i.e., because we assume here that the
likelihood mapping does not depend on policies). The first term in line 4 is the KL divergence
between prior and posterior beliefs. This value will be larger if one needs to make larger revisions
to one’s beliefs, which is the measure of complexity introduced earlier. A greater complexity means
there is a greater chance of changing beliefs to explain random aspects of one’s observations, which
can reduce the future predictive power of a model (analogous to ‘overfitting’ in statistics). The
second term in line 4 reflects predictive accuracy (i.e., the probability of observations given model
beliefs about states). The brain will therefore minimize 𝑉𝐹𝐸 if it minimizes prediction error
(maximizing accuracy) while not changing beliefs more than necessary (minimizing complexity).

Another common way that 𝑉𝐹𝐸 is expressed is in terms of placing a bound on surprisal:

20
Active Inference Tutorial

𝐹𝜋 = Ε𝑞(𝑠|𝜋) [ln 𝑞(𝑠|𝜋) − ln 𝑝(𝑠|𝑜, 𝜋)] − ln 𝑝(𝑜|𝜋) (11)

This equation rearranges line 1 in equation 10 (again using the product rule: 𝑝(𝑜, 𝑠|𝜋) =
𝑝(𝑠|𝑜, 𝜋)𝑝(𝑜|𝜋)) to show that 𝑉𝐹𝐸 is always greater than surprisal (i.e., is an upper bound on
surprisal) with respect to a policy (i.e., greater than − ln 𝑝(𝑜|𝜋)). In machine learning, the sign of
𝑉𝐹𝐸 is usually switched, so that it becomes an evidence lower bound, also known as an ELBO.
Maximizing the ELBO is a commonly used optimization approach in machine learning (Winn &
Bishop, 2005).

It is also worth briefly highlighting here how the gradients of 𝑉𝐹𝐸 leveraged in active inference can
always be expressed as a mixture of prediction errors. This is because complexity is the average
difference between posterior and prior beliefs, while accuracy is the difference between predicted
and observed outcomes. This licenses a description of active inference as prediction error
minimization (Burr & Jones, 2016; A. Clark, 2017; Fabry, 2017; Hohwy, Paton, & Palmer, 2016),
corresponding to the minimization of these two differences (minimizing 𝑉𝐹𝐸 through prediction
error minimization is described in more detail in Section 2.4 and Section 5).

However, active inference is not solely concerned with minimizing prediction error in perception. It
is also a model of action selection. When inferring optimal actions, one cannot simply consider
current observations, because actions are chosen to bring about preferred future observations. As
described informally above, this means that, to infer optimal actions, a model must predict
sequences of future states and observations for each possible policy, and then calculate the
expected free energy (𝐸𝐹𝐸) associated with those different sequences of future states and
observations. As a model of decision-making, 𝐸𝐹𝐸 also needs to be calculated relative to
preferences for some sequences of observations over others (i.e., how rewarding or punishing they
will be). In active inference, this is formally accomplished by equipping a model with prior
expectations over observations, 𝑝(𝑜|𝐶), that play the role of preferences3. For an initial intuition of
how this works, consider two possible policies that correspond to two different sequences of states
and observations, where one sequence of observations is preferred more than the other. Since
‘preferred’ here formally translates to ‘expected by the model’, then the policy expected to produce
preferred observations will be the one that maximizes the accuracy of the model (and hence
minimizes 𝐸𝐹𝐸). This means that the probability (or value) of each policy can be inferred based on
how much expected observations under a policy will maximize model accuracy (i.e., match
preferred observations). When preferred observations are treated as implicit expectations
definitive of an organism’s phenotype (e.g., those consistent with its survival, such as seeking
warmth when cold, or water when thirsty) this has also been described as ‘self-evidencing’ (Hohwy,
2016).

3
Note that in some papers, preferences are formulated over states instead of observations. In this case, one might
wonder how an agent can have two priors over states at the same time (one for beliefs and one for preferences).
Although the technical details are beyond the scope of this paper, in this case one must think more explicitly in
terms of an agent having two models - one of true states of the world and one of preferred states (cast as priors in
each model, respectively). Policy selection then attempts to minimize the divergence between the two by bringing
true states to match preferred states (for details, see (Da Costa, Parr, et al., 2020)).

21
Active Inference Tutorial

To score each possible policy in this way, 𝐸𝐹𝐸 (denoted 𝐺𝜋 in equations) can be expressed as
follows:

𝐺𝜋 = Ε𝑞(𝑜,𝑠|𝜋) [ln 𝑞(𝑠|𝜋) − ln 𝑝(𝑜, 𝑠|𝜋)] (12)

= Ε𝑞(𝑜,𝑠|𝜋) [ln 𝑞(𝑠|𝜋) − ln 𝑝(𝑠|𝑜, 𝜋)] − Ε𝑞(𝑜|𝜋) [ln𝑝(𝑜|𝜋)] L2

≈ Ε𝑞(𝑜,𝑠|𝜋) [ln 𝑞(𝑠|𝜋) − ln 𝑞(𝑠|𝑜, 𝜋)] − Ε𝑞(𝑜|𝜋) [ln𝑝(𝑜|𝐶)] L3

= −Ε𝑞(𝑜,𝑠|𝜋) [ln𝑞(𝑠|𝑜, 𝜋) − ln𝑞(s| 𝜋)] − Ε𝑞(𝑜|𝜋) [ln𝑝(𝑜|𝐶)] L4

The first line expresses 𝐸𝐹𝐸 as the expected difference between the approximate posterior and the
generative model. Note that because 𝐸𝐹𝐸 is calculated with respect to expected outcomes that (by
definition) have not yet occurred, observations enter the expectation operator Ε𝑞 as random
variables. The second line uses the product rule of probability, 𝑝(𝑜, 𝑠|𝜋) = 𝑝(𝑠|𝑜, 𝜋)𝑝(𝑜|𝜋) to
rearrange 𝐸𝐹𝐸 into two terms that can be associated with information-seeking and reward-seeking.
To make this clear, the third line does two things. First, it replaces the true posterior (ln𝑝(𝑠|𝑜, 𝜋))
with an approximate posterior (ln𝑞(𝑠|𝑜, 𝜋)). Second, it drops the conditionalization on 𝜋 in the
second term and instead conditions on the variable 𝐶 described above that encodes preferences
(i.e., Ε𝑞(𝑜|𝜋) [ln𝑝(𝑜|𝜋)] → Ε𝑞(𝑜|𝜋) [ln𝑝(𝑜|𝐶)]). This is a central move within active inference. Namely,
𝑝(𝑜|𝐶) is used to encode preferred observations, and the agent seeks to find policies expected to
produce those observations. The agent’s preferences can be independent of the policy being
followed, which allows us to drop the conditionalization on 𝜋. As mentioned earlier, in most papers
on active inference prior preferences are simply written as Ε𝑞(𝑜|𝜋) [ln𝑝(𝑜)]; however, to clearly
distinguish this from the ln𝑝(𝑜) term within 𝑉𝐹𝐸 (i.e., where 𝑜 is an observed variable), we write
the term here as explicitly conditioned on 𝐶.

The first term on the right-hand side of line 3 is commonly referred to as the epistemic value, or the
expected information gain of a state when it is conditioned on expected observations. The second
term is commonly referred to as pragmatic value, which, as just mentioned, scores the agent’s
preferences for particular observations. To make the intuition behind epistemic value more
apparent, the fourth line flips the terms inside the first expectation so that it becomes prefixed with
a negative sign (i.e., 𝑝(𝑥)[ln𝑝(𝑥) − ln𝑞(𝑥)] = −𝑝(𝑥)[ln𝑞(𝑥) − ln𝑝(𝑥)]). Because the epistemic
value term is now subtracted from the total, it is clear that to minimize 𝐸𝐹𝐸 overall an agent must
maximize the value of this term by selecting policies that take it into states that maximize the
difference between prior and posterior beliefs; that is, maximize the difference between ln𝑞(𝑠|𝑜, 𝜋)
and ln𝑞(𝑠|𝜋). In other words, the agent is driven to seek out observations that reduce uncertainty
about hidden states (Parr & Friston, 2017a). For example, if an agent were in a dark room, the
mapping between hidden states and observations would be entirely ambiguous, so it would be
driven to maximize information gain by turning on a light before seeking out preferred
observations (i.e., as it would be unclear how to bring about preferred outcomes before the light
was turned on).

Another very common expression of 𝐸𝐹𝐸 in the active inference literature is:

22
Active Inference Tutorial

𝐺𝜋 = 𝐷𝐾𝐿 [𝑞(𝑜|𝜋)||𝑝(𝑜|𝐶)] + E𝑞(𝑠|𝜋) [H[𝑝(𝑜|𝑠)]] (13)

For a full description of how you get from line 1 of equation 12 to this decomposition, see
Appendix 1. The first term on the right-hand side of this equation scores the anticipated difference
(KL divergence) between 1) beliefs about the probability of some sequence of outcomes given a
policy, and 2) preferred outcomes (i.e., those expected a priori within the model). This term is
sometimes referred to as ‘risk’ (or expected complexity), but it can more intuitively be thought of as
beliefs about the probability of reward for each choice one could make. That is, the lower the
expected divergence between preferred outcomes and those expected under a policy, the higher the
chances of attaining rewarding outcomes if one chose that policy. The second term one the right-
hand side of the equation is the expected value of the entropy (H) of the likelihood function, where
H[𝑝(𝑜|𝑠)] = − ∑ 𝑝(𝑜|𝑠)ln𝑝(𝑜|𝑠). Entropy is a measure of the dispersion of a distribution, where a
flatter (lower precision) distribution has higher entropy. A higher-entropy likelihood means there
are less precise predictions about outcomes given beliefs about the possible states of the world.
This term is therefore commonly referred to as a measure of ‘ambiguity’. Policies that minimize
ambiguity will try to occupy states that are expected to generate the most precise (i.e., most
informative) observations, because those observations will provide the most evidence for one
hidden state over others. Putting the risk and ambiguity terms together means that minimizing 𝐸𝐹𝐸
will drive selection of policies that maximize both reward and information gain (for simple
numerical examples of calculating the risk and ambiguity terms, see discussion of ‘outcome
prediction errors’ in Section 2.4). Typically, seeking information will occur until the model is
confident about how to achieve preferred outcomes, at which point it will choose reward-seeking
actions. Importantly, as briefly mentioned earlier, the expression for 𝐸𝐹𝐸 above entails that
stronger (more precise) preferences for one outcome over others will have the effect of down-
weighting the value of information, leading to reduced information-seeking (and vice-versa if
preferences are too weak or imprecise). This affects how a model resolves the ‘explore-exploit
dilemma’ (Addicott, Pearson, Sweitzer, Barack, & Platt, 2017; Friston, Lin, et al., 2017; Parr &
Friston, 2017a; Schwartenbeck et al., 2019; Wilson, Geana, White, Ludvig, & Cohen, 2014) – that is,
the difficult judgement of whether or not one “knows enough” to trust their beliefs and act on them
to seek reward or whether to first act to gather more information (see example simulations below).
For a more detailed description and step-by-step derivation of the most common formulations of
𝐸𝐹𝐸 in the active inference literature, see Appendix 1.

2. Building and Solving POMDPs

2.1 Formal POMDP Structure

In this first subsection, we introduce the reader to the abstract structure and elements of an active
inference POMDP, which is the standard modelling approach in active inference research at
present. In a POMDP one is given a specific type of generative model, including observations, states,
and policies, and the goal is to infer posterior beliefs over states and policies when conditioning on
observations. By the end of this subsection, the reader should be able to identify and interpret each
type of variable in these models and understand the role they play in performing inference. A
warning: upon initial exposure, gaining a full understanding of this abstract structure can feel

23
Active Inference Tutorial

daunting. However, after we put together a model of a concrete behavioral task (in Section 3), this
structure – and how to practically use it – tends to become much clearer. The task we will model is
an ‘explore-exploit’ task similar to commonly used multi-armed bandit tasks employed in
computational psychiatry research (see Figure 4). In this variant, there are two slot machines with
unknown probabilities of paying out. A participant can simply guess, resulting in either a large
reward or no reward, or they can ask for a hint (which may or may not be accurate). If they get it
right after taking the hint, they receive a smaller reward. This allows for competition between an
information-seeking drive and a reward-seeking drive. This task will be described in detail in
Section 3, but we will use parts of this broad-strokes description below to exemplify uses of the
more abstract elements making up POMDPs.

Figure 4. Depiction of the explore-exploit task example. Note that the states and outcomes shown
on the right are only examples. Table 1 and Section 3 list all states, outcomes, and policies required
to build a generative model for this task.

The term POMDP denotes two major concepts. As described above, the first is partial observability,
which means that observations may only provide probabilistic information about hidden states
(e.g., observing a hint may indicate that one or another slot machine is more likely to pay out). The
second is the Markov property, which simply means that, when making decisions, all relevant
knowledge about distant past states is implicitly included within beliefs about the current state.
This assumption can be violated, but it allows modelling to be more tractable and is ‘good enough’
in many cases. When dealing with violations of the Markovian assumption – such as when

24
Active Inference Tutorial

modelling memory – it is necessary to model several interconnected Markovian processes that

evolve over different timescales. We will see related examples in later sections covering both
hierarchical models and how the parameters of a POMDP can be learned through repeated
observations. Here we start with a simple, single-level POMDP where Markovian assumptions are
not violated. In the presentation below, note that vectors (i.e., single rows or columns of numbers)
are denoted with italics, while matrices (i.e., multiple rows and columns of numbers) are not
italicized and denoted with bold.

A POMDP includes both trials and time points (tau; 𝜏) within each trial (sometimes called ‘epochs’).
An important thing to note here is that 𝜏 indexes the time points about which agents have beliefs.
This is distinct from the variable 𝑡, which denotes the time points at which each new observation is
presented. This is a common (and understandable) source of confusion for those new to the active
inference literature (perhaps exacerbated by the fact that 𝑡 and τ look so similar). To appreciate the
need for this distinction, consider cases in which an observation in the present can change one’s
beliefs about the past. For example, imagine that you start out in one of two rooms (a green room or
a blue room), but you do not know what color the walls are. Later, when you open your eyes and
find out the room is painted blue, you will change your belief now about where you were earlier
before you opened your eyes (i.e., you had been in the blue room the whole time). In a formal
model, this would be a case in which beliefs about one’s state at time 𝜏 = 1 change after making a
new observation at time 𝑡 = 2. Thus, the inclusion of both 𝑡 and τ in active inference entails that the
agent updates its beliefs about states at all time points τ with new observations at each time point 𝑡.
This allows for retrospective inference, as in the previous example, as well as for prospective
inference, in which an agent updates beliefs about the future (e.g., τ = 3), when making new
observations in the present (e.g., 𝑡 = 2). This would be the case in the explore-exploit task example,
where observing a hint at one time point could update beliefs about which slot machine will be
better at the next time point. This is an important distinction to keep in mind when trying to
understand simulation results (e.g., in terms of working memory for the past and future; i.e.,
postdiction and prediction).

In practice, this is accomplished by having entries of 0 for all elements of an observation vector
when 𝑡 < 𝜏. To illustrate this formally, we will use the simpler example of being in one of two
rooms described above. In this case, there will be a ‘color’ observation modality where observations
could be ‘blue’ or ‘green’ (i.e., a vector with one element for each color). At time 𝑡 = 1, the observed
color for time 𝜏 = 2 has not yet occurred. So, at 𝑡 = 1:

0
𝑜𝜏=2 = [ ]
0

If blue were then observed at 𝑡 = 2, the observation for 𝜏 = 2 would be updated to:

1
𝑜𝜏=2 = [ ]
0

This vector would then remain unchanged for all future time points 𝑡 > 2 (i.e., the observation is
never ‘forgotten’ once it has taken place). The same thing would then occur for all subsequent

25
Active Inference Tutorial

0
observations (e.g., 𝑜𝜏=3 = [ ] at 𝑡 = 1 and 2; but if green were observed at 𝑡 = 3 then the vector
0
0
would be updated to 𝑜𝜏=3 = [ ] and remain that way for 𝑡 > 3, etc.). This allows beliefs about
1
states for all time points to be updated at each time point 𝑡 when these observation vectors are
updated.

Having now clarified time indexing, we will move on to other model elements. At the first time point
in a trial (𝜏 = 1), the model starts out with a prior over categorical states, 𝑝(𝑠𝜏=1 ), encoded in a
vector denoted by 𝐷 – one value per possible state (e.g., which slot machine is more likely to pay
out). When there are multiple state factors, there will be one 𝐷 vector per factor. As touched on
above, multiple state factors are necessary to account for multiple types of beliefs one can hold
simultaneously. One common example is holding separate beliefs about an object’s location and its
identity. In the explore-exploit task example, this could include beliefs about which slot machine is
better and beliefs about available choice states (e.g., the state of having taken the hint).

At each subsequent time point, the model has prior beliefs about how one state will evolve into
another depending on the chosen policy, 𝑝(𝑠𝜏+1 |𝑠𝜏 , 𝜋), encoded in a ‘transition matrix’ denoted by
𝐁𝜋,𝜏 – one column per state at 𝜏 and one row per state at 𝜏 + 1. If transitions for a given state factor
are identical across policies, they can be represented by a single matrix. When transitions for a state
factor are policy-dependent, there will be one 𝐁𝜋,𝜏 matrix per possible action (i.e., one for each
possible state transition under a policy). In other words, the combination of a policy and a time
specifies a transition matrix (i.e., encoding the action that would be taken under that policy at that
point in time; described further below). In the explore-exploit task example, this could include
transitioning to the state associated with getting the hint or transitioning to the state associated
with selecting one of the two machines (depending on the policy).

The likelihood function, 𝑝(𝑜𝜏 |𝑠𝜏 ), is encoded in a matrix denoted by 𝐀 – one column per state at 𝜏
and one row per possible observation at 𝜏. When there are multiple outcome modalities, there will
be one 𝐀 matrix per outcome modality. As touched upon above, multiple outcome modalities are
necessary to account for parallel channels of sensory input (e.g., one for possible visual inputs and
one for possible auditory inputs). In the explore-exploit task example, this could include one
modality for observing the hint and another modality for observing reward vs. no reward.

Preferred outcomes, ln𝑝(𝑜𝜏 |𝐶), are specified using a matrix denoted by 𝐂 – one column per time
point and one row per possible observation. When there are multiple outcome modalities, there
will be one 𝐂 matrix per modality. In the explore-exploit task example, this could encode a strong
prior preference for a large reward, a moderate preference for a small reward, and low preference
for no reward.

Prior beliefs about policies 𝑝(𝜋) are encoded in a (column) vector 𝐸 (one row per policy) –
increasing the probability that some policies will be chosen over others (i.e., independent of
observed/expected outcomes). This can be used to model the influence of habits. For example, if an
agent has chosen a particular policy many times in the past, this can lead to a stronger expectation
that that policy will be chosen again. In the explore-exploit task example, 𝐸 could be used to model

26
Active Inference Tutorial

a simple choice bias in which a participant is more likely to choose one slot machine over another
(independent of previous reward learning). However, it is important to distinguish between this
type of prior belief and the initial distribution over policies from which actions are sampled before
making an observation (𝜋0 ). As explained further below (and in Table 2), this latter distribution
depends on 𝐸, 𝐺, and 𝛾, where the influences of habits and expected future outcomes each have an
influence on initial choices.

Each allowable action (𝑢) is encoded as a possible state transition (one of several 𝐁 matrices that
can be chosen for a state factor). In this case, each possible action is encoded in a vector 𝑈, and the
possible sequences of actions (where each allowable sequence defines a policy) are encoded in a
matrix denoted by 𝐕 (one row per time point, one column per policy, and a third dimension for
state factor). In the explore-exploit task example, 𝑈 could include the choice to take the hint and the
choice to select each of the two machines, while 𝐕 could include the possible action sequences, such
as, for example, taking the hint and then choosing the left machine vs. taking the hint and then
choosing the right machine. Note that the possible actions encoded in the vector 𝑈 are also
sometimes referred to as ‘control states’ in the active inference literature.

As we have done in previous sections, the free energy and expected free energy for each policy are
denoted by vectors 𝐹 and 𝐺, respectively. The degree to which 𝐺 controls policy selection is
modulated by a further parameter 𝛾 (a single number; i.e., a scalar). This parameter is a precision
estimate for the expected free energy over policies. It can be thought of as encoding a prior belief
about the confidence with which policies can be inferred (i.e., how reliable beliefs about the best
policy are expected to be). It is often called the ‘prior policy precision’ parameter; however, it is
important to note that this is not the same thing as the precision of posterior beliefs over policies
(𝜋). This is because 𝜋 also depends on the vectors 𝐸 (habits) and 𝐹 (shown in Table 2 further
below) – which means, for example, that 𝜋 could be precise even if 𝛾 were low (Hesp et al., 2020).
For this reason, it is better to think of 𝛾 as an ‘expected free energy (G) precision’ parameter as
opposed to a policy precision parameter per se. If no habits are present (i.e., if 𝐸 is a flat
distribution), lower 𝛾 values lead to more randomness in policy selection. In the presence of strong
habits, lower 𝛾 values increase how much habits influence policy selection, because the influence of
𝐺 is reduced relative to 𝐸 (see equation for 𝜋 in Table 2 further below). You can simulate these
dynamics yourself by specifying values for 𝛾, 𝐸, 𝐹, and 𝐺 within the 𝐄𝐅𝐄_𝐏𝐫𝐞𝐜𝐢𝐬𝐢𝐨𝐧_𝐔𝐩𝐝𝐚𝐭𝐢𝐧𝐠. 𝐦
code provided in the supplementary code.

In some models we will discuss below with multi-step policies, the prior value for 𝛾 is updated into
a posterior 𝛾 estimate (via updates to its hyperparameter 𝛽). Table 2 and Figure 9 explain this in
detail. Briefly, the value of this precision parameter is increased after each observation if the
variational free energy over policies is consistent with the expected free energy over policies prior
to that observation. If these free energies are inconsistent, this precision is decreased (i.e., the agent
becomes less confident in its estimates of 𝐺). However, it is important to note that there are
situations in which policies are only considered from the current time step into the future (such as
‘shallow’ or ‘short-sighted’ policies that, at each time step, only ‘look ahead’ one time step to
consider the immediate consequences of different actions). In such cases, previous observations do
not inform the relative probabilities of policies (i.e., they are just ‘reset’ at each time step) – and the

27
Active Inference Tutorial

expected precision reduces to the prior value for 𝛾 (and is not updated). (Note: below, and in the
supplementary code, we show how ‘shallow’, one-step policies vs. ‘deep’, multi-step policies can
be included by specifying policies with the variable 𝑈 vs. 𝑉).

All the model variables are summarized in Table 1. Solutions for inference in the POMDP are
shown in Table 2 at the end of this section. In each of these tables we also provide a description of
the way each model element can be used to implement the explore-exploit task example, which we
build in Section 3.

Table 1. Model variables

Model General definition Model specification for explore-exploit

variable* task (described in detail Section 3)

𝑜𝜏 Observable outcomes at time 𝜏. Outcome modalities:

1. Hints (no hint, hint-left, hint-right)

2. Reward (start, lose, win)

3. Observed behavior (start, take hint,

choose left, choose right)

𝑠𝜏 Hidden states at time 𝜏. One vector of Hidden state factors:

possible state values for each state
factor (i.e., each independent set of 1. Context (left machine is better vs.
states; e.g., visual vs. auditory states). right machine is better)

2. Choices (start, take hint, choose left,

choose right)

𝜋 A vector encoding the distribution over Allowable policies include the decision to:
policies reflecting the predicted value of
each policy. Each policy is a series of 1. Stay in the start state
allowable actions in a vector 𝑈, where 2. Get the hint and then choose the left
actions correspond to different state machine
transitions (i.e., different 𝐁𝜋,𝜏 matrices) 3. Get the hint and then choose the right
that can be chosen by the agent for each machine
state factor. Policies are chosen by 4. Immediately choose the left machine
sampling from this distribution. (and then return to the start state)
5. Immediately choose the right
machine (and then return to the start

28
Active Inference Tutorial

state)

𝐀 matrix A matrix encoding beliefs about the Encodes beliefs about the relationship
relationship between hidden states and between:
observable outcomes at each time point
𝜏 (i.e., the probability that specific 1. Probability that the hint is accurate in
𝑝(𝑜𝜏 | 𝑠𝜏 ) each context
outcomes will be observed given specific
hidden states at specific times). Note 2. Probability of reward in each context
that in the POMDP structure typically 3. Identity mapping between choice
used in the active inference literature states and observed behavior
(and which we describe in this tutorial),
the likelihood is assumed to remain
constant across time points in a trial,
and hence will not differ at different
values for 𝜏 (although one could adjust
this if so desired). The likelihood is also
assumed to be identical across policies,
and so there is no indexing with respect
to 𝜋.

When there is more than one outcome

modality, there is one 𝐀 matrix per
outcome modality. When there is more
than one state factor, these matrices
become high-dimensional and are
technically referred to as tensors. For
example, a second state factor
corresponds to a third matrix
dimension, a third state factor
corresponds to a fourth matrix
dimension, and so forth.

𝐁𝜋,𝜏 matrix A matrix encoding beliefs about how Encodes beliefs that:
hidden states will evolve over time
(transition probabilities). For states that 1. Context does not change within a trial
are under the control of the agent, there 2. Transitions from any choice state to
𝑝(𝑠𝜏+1 | 𝑠𝜏 , 𝜋) are multiple 𝐁 matrices, where each any other are possible, depending on
𝜋,𝜏
matrix corresponds to one action (state the policy.
transition) that the agent may choose at
a given time point (if consistent with an
allowable policy). When there is more
than one hidden state factor, there is one

29
Active Inference Tutorial

or more 𝐁𝜋,𝜏 matrices per state factor

(depending on policies).

𝐂 matrix A matrix encoding the degree to which Encodes the stronger preference for wins
some observed outcomes are preferred than losses. Wins are also more preferred at
over others (technically modeled as the second time point than the third time
prior expectations over outcomes). point.
𝑝(𝑜𝜏 |𝐶)
When there is more than one outcome
modality, there is one 𝐂 matrix per
outcome modality. Rows indicate
possible observations; columns indicate
time points. Note that each column of
values in C is passed through a softmax
function (transforming it into a proper
probability distribution) and then log-
transformed (using the natural log).
Thus, preferences become log-
probabilities over outcomes.

𝐷 vector A vector encoding beliefs about (a The agent begins in an initial state of
probability distribution over) initial maximal uncertainty about the context state
hidden states. When there is more than (prior to learning), but complete certainty
one hidden state factor, there is one 𝐷 that it will start in the ‘start’ choice state.
𝑝(𝑠1 )
vector per state factor.

𝐸 vector A distribution encoding beliefs about The agent has no initial habits to choose one
what policies will be chosen a priori (a slot machine or another (prior to learning).
prior probability distribution over
policies, implemented as a vector
𝑝(𝜋)
assigning one value to each policy),
based on the number of times different
actions have been chosen in the past.

*While, for consistency, we have used the standard notation found in the active inference literature, it is
important to note that it does not always clearly distinguish between distributions and the possible values
taken by random variables under those distributions. For example, 𝜋 refers to the distribution over policies,
but when used as a subscript it indexes each individual policy (e.g., 𝐁𝜋,𝜏 indicates a distinct matrix for each
different policy). This same convention holds for 𝑠 and 𝑜.

2.2 Graphical Models

In many papers in the active inference literature, POMDPs are represented using graphical
models. We will now discuss these representations and what their role is in active inference. By the

30
Active Inference Tutorial

end of this subsection, the reader should be able to interpret graphical models and understand the
different benefits they provide.

Graphical models, such as the graphs shown in Figures 5-7, are a useful method for visually
depicting how variables in a model depend on one another. When models include probability
distributions over variables, these graphs can be used to represent the conditional relationships
between these variables. These types of probabilistic graphical models are particularly useful in the
context of active inference because they provide a clear visual summary of the computational
architecture of the models, and the way (biologically plausible) message passing algorithms
(described below) can be used to update beliefs. Here we consider two types of graphical models –
Bayesian networks (or ‘Bayes nets’, see Figure 5; and for an introduction, see chapter 8 of
(Bishop, 2006)) and Forney-style (normal) factor graphs (Dauwels, 2007; Loeliger, 2004). For
readers interested in a more detailed introduction to the use of Forney-style factor graphs in active
inference, we recommend the excellent tutorial introduction by de Vries and Friston (de Vries &
Friston, 2017).

Figure 5. Bayesian network representations of state estimation (perception) and policy selection.
Each graph depicts a generative model of the causes of observations, which can be inverted to
perform inference. Top left: Static perception with a worked example. Variables: 𝑠 = states, 𝑜 =
observations, A = likelihood mapping between states and outcomes, D = initial state priors. This is
equivalent to exact Bayesian inference. Bottom left: Dynamic perception. Transition matrices (𝐁𝜏 )
have been added to describe (beliefs about) how states change over time. Subscripts for
observations and states correspond to time point in a trial (denoted by tau; 𝜏). Importantly, when
𝜏 > 1, the 𝐁𝜏 matrix from the previous 𝜏 (i.e., 𝐁𝜏−1) functions as an empirical prior, playing the
same role as the D vector at 𝜏 = 1. Top right: Dynamic perception with policy selection. Each policy
(𝜋) entails a different sequence of actions, which corresponds to different transitions between
states (i.e., different 𝐁𝜋,𝜏 matrices). Based on expected free energy (𝐺; which in turn depends on

31
Active Inference Tutorial

prior preferences, C), the highest probability will be assigned to policies expected to minimize
uncertainty over states and maximize the probability of preferred observations. Bottom right:
Dynamic perception with flexible policy selection. This final model includes an expected free energy
precision term 𝛾 = E[γ], where 𝑝(γ) corresponds to a gamma distribution (Γ) with a ‘shape’
parameter equal to 1 and a ‘rate’ parameter specified by 𝛽. Note that the non-italicized γ in the
generative model is a random variable, whereas the italicized 𝛾 is a statistic (expected value) of the
gamma distribution (i.e., a fixed scalar) that is updated based on the equations shown in the figure.
The value of 𝛾 encodes the agent’s confidence in policy selection and adjusts the contribution of 𝐺
to the posterior distribution over policies (𝜋). This precision value is also optimized by updating the
value of 𝛽 after new observations, based on the variational free energy (𝐹) over policies associated
with those observations. In short, when a new observation is inconsistent with prior beliefs about
policies (𝜋0 ; i.e., based on 𝐺), the agent assigns a lower expected precision (𝛾) to 𝐺 when arriving at
posteriors over policies (𝜋; see Figure 9 and Table 2 for more details). A prior over policies (𝐸) is
also included, which can be used to model habit formation. A lower 𝛾 (i.e., less confidence in model-
based beliefs about 𝐺) also entails a stronger influence of the habits encoded in 𝐸 on policy
selection. Note that the dependency of 𝜋 on 𝐹 has been omitted from the graphical depiction of the
generative model in this panel. See main text for further variable descriptions. See Table 2 for
further explanation of these equations.

When depicting active inference models with Bayes nets (Figure 5), the circles (‘nodes’)
correspond to variables (e.g., observations, hidden states, and policies), while the arrows
connecting nodes (‘edges’) show the dependencies between variables represented by each node.
For example, the arrows in Figure 5 going from the ‘𝑠𝜏 node’ (i.e., states at time 𝜏) to the ‘𝑜𝜏 node’
(i.e., outcomes at time 𝜏) means that the value of 𝑜𝜏 depends on the value of 𝑠𝜏 . This entails that if
one knows something about observations, then one can infer something about the hidden states
that cause them (i.e., the hidden states that generate those observations in the generative model).
Readers familiar with Bayesian networks will note that the form of the graphical model shown in
Figure 5 is slightly unusual, as squares denoting the factors that mediate the conditional
relationships have been placed on top of the edges (e.g., the 𝐀 and 𝐁𝜋,𝜏 matrices).

This graphical model serves as a concise visual depiction of the relationships between model
elements covered in detail in the previous subsection. It illustrates how observations at each time
step (purple) are generated by hidden states (green) via a mapping encoded by the likelihood
matrix 𝐀. The 𝐁𝜋,𝜏 matrix is shown as mediating the dependencies between states at different time
points (i.e., encoding beliefs about how states change over time). The probability over states at the
first time point is shown to depend on the 𝐷 vector. State transitions (𝐁𝜋,𝜏 ) are shown to depend
upon policies (𝜋). The probability distribution over policies in turn depends on learned priors over
policies (𝐸) and the 𝐸𝐹𝐸 of each allowable policy (𝐺). The 𝐸𝐹𝐸 is shown to depend on the prior
distribution over observations (𝐂), which encodes the agent’s preferences for some observations
over others (i.e., this dependency entails that policies with the lowest 𝐸𝐹𝐸 will be those expected to
generate the most preferred observations). The influence of 𝐸𝐹𝐸 on policies also depends upon its
precision term (𝛾), which encodes confidence in current 𝐸𝐹𝐸 estimates. This in turn depends on the
value of 𝛽 (an initial prior over 𝛾 that can subsequently be updated; see Table 2 for a description).

32
Active Inference Tutorial

To help readers gain an intuition for inference using graphical models, Figure 5 builds up a full
POMDP starting from a graphical representation of perception at a single time point using Bayes
theorem (with a worked example). It then adds the evolution of states over time, followed by their
dependence on policies, and the dependence of policies on the variables just described.

The defining characteristic of a generative model such as that shown in Figure 5 is that it can be
used to generate data (i.e., observations). The conditional dependencies depicted in the Bayesian
network in the bottom-right of this figure show how observations are generated by a POMDP.
Starting at the top of the network, a policy (𝜋) is first selected via a softmax (normalized
exponential) function (𝜎) of the aforementioned variables (for an introduction to the softmax
function, see Appendix 1). The initial distribution over policies prior to receiving an observation is
denoted 𝜋0 = 𝜎(ln 𝐸 −𝛾𝐺 ), while the posterior over policies after receiving an observation also
incorporates the 𝑉𝐹𝐸: 𝜋 = 𝜎(ln 𝐸 −𝐹 − 𝛾𝐺 ). Next, policy-dependent transition probabilities
encoded in the 𝐁𝜋,𝜏 matrix (or the D vector at 𝜏 = 1) generate hidden states, which in turn generate
observations at each time point. The likelihood (𝐀) matrix determines which observations are
generated by each hidden state.

Recall that to perform inference we must invert the generative model (i.e., infer the most likely
states and policies given each new observation). This is where normal factor graphs (Figure 6) are
crucial, as they can be leveraged to both derive and visualize a suite of message passing algorithms
(see below) for Bayesian inference. Normal factor graphs are made up of square nodes and edges
(lines connecting square nodes). Square nodes can be thought of as functions (i.e., factors or
conditional probability distributions; see below) that take in some input (e.g., sufficient statistics4 of
beliefs over states or observations) and transform that information in some way to produce an
output (e.g., the sufficient statistics of the conditional probabilities specified by the factor). These
inputs and outputs are called ‘messages’ and are represented by the edges connecting the square
nodes. When an edge connects to only one square node it is called a half edge, and it only carries
messages to and from that node. When an edge connects two square nodes, this indicates that these
nodes exchange messages and that each contributes to the value represented on that edge. When
three square nodes exchange messages, the graph is adjusted to contain three edges (one connected
to each square node) that converge onto an equality node (a small square node with an ‘=’ sign),
which combines the messages in a specific manner (described in the following technical section). As
described further below, the messages represented on each edge correspond to the variables
represented by the circular nodes in the Bayes’ net depiction in Figure 5, while the square nodes in
a factor graph correspond directly to the square nodes in this figure. Note that, in some cases, the
edges in factor graphs are also supplemented by including circle nodes to represent (the sufficient
statistics of) hidden variables (as in Figure 5), but we depict them without circle nodes in Figures
6 and 7 to give the reader some familiarity with this commonly presented form. Also note that,
unlike Bayes net depictions, normal factor graphs have undirected edges, which highlights the
bidirectional nature of message passing (see light purple arrows in the bottom portion of Figure 6).

4
Note that, for the categorical distributions we use in this tutorial, the sufficient statistics will correspond to the
probability of each possible value of a random variable (e.g., the probability of each possible state or observation).

33
Active Inference Tutorial

More technically, normal factor graphs represent a factorization of the generative model. Recall that
generative models are formally defined as the joint probability distribution over observations,
states, and policies of the POMDP across time, 𝑝(𝑜1:𝑇 , 𝑠1:𝑇 , 𝜋). Factorization means that this joint
probability can be defined as the product of several conditionally independent distributions. In
POMDPs, the factorization assumes that each state only depends on the state at the previous time
step and policy (i.e., the so-called Markov property). This is described by the following equation,
which shows a factorization of the joint distribution into prior distributions over states and
policies, and the distributions representing the likelihood and state transitions.

T T

𝑝(𝑜1:𝑇 , 𝑠1:𝑇 , 𝜋) = 𝑝(𝑠1 )𝑝(𝜋) ∏ 𝑝(𝑜𝜏 |𝑠𝜏 ) ∏ 𝑝(𝑠𝜏 |𝑠𝜏−1 , 𝜋) (14)

𝜏=1 𝜏=2

T T L2
𝑝(𝑜1:𝑇 , 𝑠1:𝑇 |𝜋) = 𝑝(𝑠1 ) ∏ 𝑝(𝑜𝜏 |𝑠𝜏 ) ∏ 𝑝(𝑠𝜏 |𝑠𝜏−1 , 𝜋)
𝜏=1 𝜏=2

T T L3
= 𝑠1 ∙ 𝐷 ∏ 𝑜𝜏 ∙ 𝐀𝑠𝜏 ∏ 𝑠𝜏 ∙ 𝐁𝜋,𝜏 𝑠𝜏−1
𝜏=1 𝜏=2

For the unfamiliar reader, please note that the symbol ∏𝑇𝜏(∙) indicates taking the product of each of
the distributions to the right of it for each time point 𝜏 to the final time point 𝑇. Line 2 shows the
form of the distribution conditioned on policies. Line 3 rewrites line 2 in matrix form by replacing
each categorical distribution with the above-described matrices/vectors whose columns contain
the parameters of the respective distributions. Specifically, 𝑝(𝑠1 ) = 𝑠1 ∙ 𝐷, 𝑝(𝑜𝜏 |𝑠𝜏 ) = 𝑜𝜏 ∙ 𝐀𝑠𝜏 , and
𝑝(𝑠𝜏 |𝑠𝜏−1 , 𝜋) = 𝑠𝜏 ∙ 𝐁𝜋,𝜏 𝑠𝜏−1. Here 𝑜𝜏 and 𝑠𝜏−1 are vectors of zeros with a one placed in the element
corresponding to the state/observation of interest. Their role is simply to select out the elements of
the 𝐀 or 𝐁 matrix corresponding to the relevant state-outcome pair or current state-previous state
pair. Once in matrix form, it is easy to see the direct correspondence between the factorized
distribution shown in the equation above and the factors included in the normal factor graph in
Figure 6. Each of these distributions is associated with a factor node. Each edge represents the
probability distribution over the variable that needs to be inferred (i.e., the approximate posteriors
over states 𝑠𝜋,𝜏 , and policies 𝜋). Edges connect factors that exchange messages with the same
variables (e.g., D, A, and 𝐁𝜋,𝜏 are all connected by the variable 𝑠𝜋,1 ).

34
Active Inference Tutorial

Figure 6. Top: Equations specifying the factors that constitute the factorized generative model.
Numbers in the green squares highlight the correspondence between the equations and the factors
in the generative model that are represented within the normal factor graph in the bottom panel.
Here, 𝐶𝑎𝑡() indicates a categorical distribution. Middle: Belief update equation for approximate
posteriors over states that is derived from variational message passing (note the difference
between this message passing scheme and the marginal message passing approach described in the
text). Purple numbers indicate the correspondence between terms within the update equation and
the messages passed between each factor shown in the factor graph in the bottom panel. Bottom:
Normal factor graph representation of the factorized POMDP. In contrast to the Bayes net
representation shown in Figure 5, nodes (large white boxes) represent factors, whereas the edges

35
Active Inference Tutorial

(lines connecting each box) represent the sufficient statistics of approximate posteriors, which are
passed as messages between factors (i.e., edges represent the common variables that participate in
the factors they connect, such as posteriors over states under each policy for each time point, 𝑠𝜋,𝜏 ).
Adapted from (Friston, Parr, et al., 2017).

2.3 Technical Section on Variational and Marginal Message Passing (optional)

In this subsection we will introduce variational message passing (Winn & Bishop, 2005), which is
foundational to the way active inference performs approximate inference of posteriors over states.
By the end of this subsection, the reader should understand the general steps for performing
variational message passing and how they can be carried out using the factor graph representation
of POMDPs in active inference. For readers with less mathematical background, this section can also
be safely skipped without compromising the ability to understand the rest of the tutorial. Although,
as always, we have made efforts to fully explain all equations. For readers with specific interest in
this topic, we also note here that more recent implementations of active inference have used a
refined algorithm – called marginal message passing – that is more robust to problems of
overconfidence that arise with variational message passing (i.e., where posterior beliefs can
become too precise too quickly; see (Parr et al., 2019)). However, understanding marginal message
passing requires us to first understanding variational message passing. Therefore, we will focus on
this approach here, and return to how it has been refined at the end of the section.

To invert the model (i.e., condition on observations to infer approximate posteriors over states and
policies) via the minimization of 𝑉𝐹𝐸, some simplifying assumptions need to be made (i.e., since
exact inference is intractable in most real-word cases). Variational message passing is based on the
mean-field approximation, which assumes that the approximate posterior factorizes into the
product of (independent) distributions (Bishop, 2006). This approximation often works well in
practice, but it has the limitation of ignoring possible pairwise (or more complex) interactions
between variables. In the POMDPs under discussion here, the mean-field approximation assumes
that the approximate posterior factorizes into a prior distribution over policies and the
distributions over states expected under each policy at each time point:

𝑝(𝑠1:𝑇 |𝑜1:𝑇 , 𝜋) ≈ 𝑞(𝑠1:𝑇 , 𝜋) = 𝑞(𝜋) ∏ 𝑞(𝑠𝜏 |𝜋) (15)

𝜏

Note that, by convention, approximate posterior distributions are denoted with the variable 𝑞. Also
again recall that 𝑇 corresponds to the final time point in a trial, such that this posterior distribution
is over the values of states across time points under each policy – and this distribution itself evolves
over time with each new observation. This means that an observation at a later time can change
posterior beliefs about states at earlier times (i.e., retrospective inference).

With this factorization in hand, we can employ variational message passing to infer the
approximate posterior 𝑞(𝑠𝜏 |𝜋) at each edge of the graph, and then combine them into a global

36
Active Inference Tutorial

posterior 𝑞(𝑠1:𝑇 |𝜋) using the equation just presented. Variational message passing can be
summarized in terms of the following steps:

1. Initialize the values of the approximate posteriors 𝑞(𝑠𝜋,𝜏 ) for all hidden variables (i.e., all
edges) in the graph.
2. Fix the value of observed variables (here, 𝑜𝜏 ).
3. Choose an edge (𝑉) corresponding to the hidden variable you want to infer (here, 𝑠𝜋,𝜏 ).
4. Calculate the messages, 𝜇(𝑠𝜋,𝜏 ), which take on values sent by each factor node connected to
𝑉.
5. Pass a message from each connected factor node 𝑁 to 𝑉 (often written as 𝜇𝑁→𝑉 ).
6. Update the approximate posterior represented by 𝑉 according to the following rule:
𝑞(𝑠𝜋,𝜏 ) ∝ 𝜇⃗(𝑠𝜋,𝜏 )𝜇⃖(𝑠𝜋,𝜏 ). The arrow notation here indicates messages from two different
factors arriving at the same edge.
a. Normalize the product of these messages so that 𝑞(𝑠𝜋,𝜏 ) corresponds to a proper
probability distribution.
b. Use this new 𝑞(𝑠𝜋,𝜏 ) to update the messages sent by connected factors (i.e., for the
next round of message passing).
7. Repeat steps 4-6 sequentially for each edge.
8. Steps 3-7 are then repeated until the difference between updates converges to some
acceptably low value (i.e., resulting in stable posterior beliefs for all edges).

For unfamiliar readers, the ‘∝’ symbol in step 6 denotes proportionality, meaning that the ratio
between variables is always constant. We can change from the proportionality sign to an equals
(‘=’) sign by explicitly introducing a constant (𝑘) into the equation, so 𝑥 ∝ 𝑦 becomes 𝑥 = 𝑘 × 𝑦. For
probability distributions, the constant is the normalization factor that ensures a distribution sums
to 1. Also note that, while the arrows above each 𝜇 in step 6 are used to distinguish messages
conveyed from two different factor nodes onto the same edge, the factor graphs in active inference
models require three factor nodes to exchange messages. As mentioned earlier, when more than
two factors exchange messages, this requires edges from each factor node to converge onto an
equality node. In this case, the message conveyed to each edge is the product of the messages from
the other connected factors: 𝜇⃗(𝑠𝜋,𝜏 ) ∝ ⃗⃗⃗⃗⃗(𝑠
𝜇1 𝜋,𝜏 )𝜇
⃗⃗⃗⃗⃗(𝑠
2 𝜋,𝜏 ) … ⃗⃗⃗⃗⃗⃗(𝑠
𝜇𝑁 𝜋,𝜏 ).

For hidden states 𝑠𝜋,𝜏 , each message conveys the exponentiated expected log value of each factor
𝜇⃗(𝑠𝜋,𝜏 ) ∝ exp E𝑞 [ln 𝑔(𝑠𝜋,𝜏 )], where 𝑔(𝑠𝜋,𝜏 ) denotes the function represented by each factor
(Dauwels, 2007). For observed variables, the message simply conveys the known value of the
factor, which is easily calculated (e.g., in the POMDPs considered in this tutorial, the message is
simply 𝐀T o). When combined, these messages allow for approximation of the posterior represented
by the associated edge. The posterior at each edge 𝑞(𝑠𝜋,𝜏 ) is normalized by applying a softmax
function prior to the next round of message passing.

Using these update rules, we arrive at the following update equations for approximate posteriors
over states in our POMDP models, which we will call messages 1-3 (i.e., performed in log-space;

37
Active Inference Tutorial

represented by purple circles on the factor graph at each location in Figure 6 where edges meet
factors):

Message 1: ln𝜇⃗𝐁𝜋,𝜏−1 →𝑠𝜋,𝜏 = E𝑞(𝑠𝜏−1 |𝜋) [ln𝑝(𝑠𝜏 |𝑠𝜏−1 , 𝜋)]

Message 2: ln𝜇⃖𝐁𝜋,𝜏→𝑠𝜋,𝜏 = E𝑞(𝑠𝜏+1 |𝜋) [ln𝑝(𝑠𝜏 |𝑠𝜏+1 , 𝜋)]

Message 3: ln𝜇𝐀→𝑠𝜋,𝜏 = ln𝑝(𝑜𝜏 |𝑠𝜏 )

Note the straightforward relation between these messages and Bayes’ theorem. As depicted in
Figure 6, message 1 corresponds to the prior from the previous time point (denoted by the right-
facing arrow). Message 2 corresponds to prior information from the future time point (denoted by
the left-facing arrow; e.g., allowing retrospective inference about the state at time point 1 after
receiving an observation at time point 2). Message 3 corresponds to the likelihood of an
observation at the current time point (no arrow notation; here assumed to be the same for all
values of 𝜋). So, for example, if we take the edge corresponding to the posterior for 𝑠𝜋,2 (in the
middle of the graph), this posterior will then correspond to integrating priors (𝐁𝜋,𝜏−1 and 𝐁𝜋,𝜏 )
with the likelihood (𝐀) and then normalizing to convert back to a proper probability distribution
(i.e., as in Bayes’ theorem). When adopting the matrix notation for these messages, belief updating
can be written as:

T
𝑠𝜋,𝜏 = 𝜎(ln 𝐁𝜋,𝜏−1 𝑠𝜋,𝜏−1 + ln𝐁𝜋,𝜏 𝑠𝜋,𝜏+1 + ln 𝐀T 𝑜𝜏 ) (16)

Note that 𝐁𝜋,𝜏−1 𝑠𝜋,𝜏−1 is replaced by the prior over initial states 𝐷 for 𝜏 = 1. As described in the
following section, these updates (or their marginal message passing counterparts) can also be
reformulated in terms of prediction errors that illustrate the biological plausibility of this message
passing scheme. To help the reader get a concrete sense of the dynamics of message passing, we
have provided some simple example code within the accompanying MATLAB scripts
(𝐌𝐞𝐬𝐬𝐚𝐠𝐞_𝐩𝐚𝐬𝐬𝐢𝐧𝐠_𝐞𝐱𝐚𝐦𝐩𝐥𝐞. 𝐦).

Inferring policies can be thought of as making use of an analogous process. Although it should be
emphasized that posterior inference over policies in the current implementation
(𝐬𝐩𝐦_𝐌𝐃𝐏_𝐕𝐁_𝐗_𝐭𝐮𝐭𝐨𝐫𝐢𝐚𝐥. 𝐦) is not explicitly done in this manner. Still, we will stay with the
message passing notation for didactic purposes. Recall that, under active inference, policies are
selected based on their (expected) ability to generate preferred observations and maximize
information gain. In addition, recall from the previous two sections that preferred observations are
formally treated as being more probable a priori. State transitions under a policy can then be seen
as more probable if they maximize the probability of current observations, ln𝑝(𝑜𝜏 |𝜋), and the
expected probability of future observations conditioned upon the policy in question,
E𝑞(𝑜𝜏,𝑠𝜏|𝜋) [ln𝑝(𝑜𝜏 |𝜋)]. Notice that future observations are here treated as random variables that
need to be inferred (i.e., because they have not yet been given to the model). Also notice the
similarity between this and the expression for 𝐸𝐹𝐸. This similarity is no accident, as we shall see
shortly (Parr & Friston, 2018a). Inferring these distributions requires us to evaluate partition
functions of the normal factor graph. This means summing over the variables (i.e., probabilities)

38
Active Inference Tutorial

represented by the edges enclosed in the red dotted lines in Figure 7. This operation is also
sometimes called “closing the box” (Loeliger, 2004). For example, to obtain the probability of
current observations conditioned upon policies, and that of expected future observations
conditioned upon policies, we must evaluate the following summations:

ln𝑝(𝑜𝜏 |𝜋) = ln ∑ 𝑝(𝑜𝜏 , 𝑠𝜏 | 𝜋) (17)

𝑠

E𝑞(𝑜𝜏>𝑡 ,𝑠𝜏>𝑡 |𝜋) [ln𝑝(𝑜𝜏>𝑡 |𝜋)] = E𝑞(𝑜𝜏>𝑡 ,𝑠𝜏>𝑡 |𝜋) [ ln ∑ 𝑝(𝑜𝜏>𝑡 , 𝑠𝜏>𝑡 | 𝜋)] (18)
𝑠

As we have seen, however, such summations are often intractable. Instead, we evaluate the free
energy functionals 𝑉𝐹𝐸 and 𝐸𝐹𝐸, as they approximate the required probabilities (as we saw in
Section 1) and can be computed efficiently:

− ln𝑝(𝑜𝜏 |𝜋) ≈ 𝐹𝜋,𝜏 (19)

− E𝑞(𝑜𝜏>𝑡 ,𝑠𝜏>𝑡 |𝜋) [ln𝑝(𝑜𝜏>𝑡 |𝜋)] ≈ 𝐺𝜋,𝜏 (20)

The posterior over policies can then be computed in a similar manner as the posterior over states.
Specifically, we can express the messages sent from the 𝐁𝜋,𝜏−1 and 𝐁𝜋,𝜏 matrix factor nodes, and
the 𝐸 vector factor node, to the edges representing the posterior over policies as follows (messages
1-3 shown in Figure 7).

𝑞(𝜋) ∝ 𝜇𝐸→𝜋 ∙ 𝜇⃗𝐁𝜋,𝜏−1 →𝜋 ∙ 𝜇⃖𝐁𝜋,𝜏→𝜋 (21)

Message 1: ln𝜇𝐸→𝜋 = ln𝐸

Message 2: ln𝜇⃗𝐁𝜋,𝜏−1 →𝜋 = 𝐹𝜋,𝜏

Message 3: ln𝜇⃖𝐁𝜋,𝜏→𝜋 = 𝐺𝜋,𝜏

Here again, messages from past and future time points are denoted with right-pointing and left-
pointing arrows (respectively), while the message conveying priors over policies is denoted
without arrow notation. Once these messages are passed, if we normalize the result by applying a
softmax function, we arrive at an expression for the posterior over policies that (suppressing the
precision term 𝛾) corresponds to the equation shown in Table 2:

𝜋 = 𝜎(ln𝐸 − 𝐹 − 𝐺) (22)

It is important to note, however, that unlike the state inference process shown in Figure 6, there is
no need for iterative message passing in this case. A single round of message passing is equivalent
to the equation above. Thus, while inference over policies can be heuristically viewed in terms of
message passing (for illustrative consistency with variational message passing in state inference), it

39
Active Inference Tutorial

need not be described in this manner (and there are differences; e.g., the messages are not
bidirectional).

Figure 7. This figure reproduces the same graph shown in Figure 6 to illustrate the link between
message passing and policy selection in active inference. Top: As in Figure 6, these equations
specify the factors that constitute the factorized generative model, and the numbers in the green
squares highlight the correspondence between the equations and the factors represented by the
normal factor graph below. Middle: Belief update equation for inferring the posterior over policies.
Purple numbers indicate the correspondence between terms within the update equation and the
messages passed between each factor shown in the factor graph. Bottom: Normal factor graph

40
Active Inference Tutorial

representation of message passing in the context of inference over policies. Red dotted lines show
partition functions of the graph, which are used to construct the free energy approximations to the
probability of current observations conditioned on policies, − ln𝑝(𝑜𝝉 |𝜋) ≈ 𝐹𝝅,𝝉 , and the expected
probability of future observations conditioned on policies, − E𝑞(𝑜𝜏>𝑡 ,𝑠𝜏>𝑡 |𝜋) [ln𝑝(𝑜𝜏>𝑡 |𝜋)] ≈ 𝐺𝜋,𝜏 . The
factors 𝐹𝝅,𝝉 and 𝐺𝜋,𝜏 then become the messages (shown by the purple arrows) sent from the two
transition probability factors (𝐁𝜋,𝜏−1 and 𝐁𝜋,𝜏 ) that converge on the equality constraint node
(connecting the 𝐁𝜋,𝜏 nodes and the 𝐸 node). When combined with the message sent from 𝐸, and
after the application of a softmax function, this becomes the posterior over policies (adapted from
(Friston, Parr, et al., 2017; Parr & Friston, 2018a)). As noted in the text, this representation of
inference over policies as message passing is heuristic and only meant as an analogy to message
passing with respect to posteriors over states. This is because it is not carried out iteratively (i.e.,
the posterior is arrived at using a single iteration), the messages are not bidirectional, and 𝐹𝜋,𝜏 and
𝐺𝜋,𝜏 are not factors. This graphical representation also cannot illustrate all dependencies with
respect to 𝐺𝜋,𝜏 . This is because 𝐺𝜋,𝜏 depends on two different types of predicted future observations
– 𝑝(𝑜𝜏>𝑡 |𝜋) and 𝑝(𝑜𝜏>𝑡 |𝐶) – only the first of which is depicted here (i.e., with respect to 𝑜3 ). For a
proposed scheme for carrying out iterative message passing with respect to inference over policies,
see (Champion, Grześ, & Bowman, 2021).

At this juncture, we return to the more recent development of marginal message passing that was
mentioned at the beginning of this section. Because variational message passing and the mean-field
approximation have known limitations, this improved algorithm has been adopted, and is
incorporated into the most recent software implementation (𝐬𝐩𝐦_𝐌𝐃𝐏_𝐕𝐁_𝐗. 𝐦; as well as in the
tutorial version included as supplementary code: 𝐬𝐩𝐦_𝐌𝐃𝐏_𝐕𝐁_𝐗_𝐭𝐮𝐭𝐨𝐫𝐢𝐚𝐥. 𝐦). Briefly, marginal
message passing represents a type of compromise between the computational efficiency of
variational message passing and another widely used algorithm – called belief propagation – that is
more computationally expensive but can perform exact (as opposed to approximate) inference
under suitable conditions (for details, see (Parr et al., 2019)). A full explanation of marginal
message passing is beyond the scope of this tutorial, as it first requires a more thorough
introduction to both variational message passing and belief propagation. Here, we chose to
introduce the reader to the mean-field approximation in combination with variational message
passing due to its simplicity and wide usage within the active inference literature, and to provide
the interested reader with a foundation to build from when pursuing more details on these topics
elsewhere.

However, the major resulting adjustment under marginal message passing is that the posterior over
states becomes:

1 † (23)
𝑠𝜋,𝜏 = 𝜎 ( (ln 𝐁𝜋,𝜏−1 s𝜋,𝜏−1 + ln𝐁𝜋,𝜏 𝑠𝜋,𝜏+1 ) + ln 𝐀T 𝑜𝜏 )
2

As above, note that 𝐁𝜋,𝜏−1 𝑠𝜋,𝜏−1 is replaced by the prior over initial states 𝐷 for 𝜏 = 1. The result of
adding the ½ to scale the influence of transition beliefs (𝐁𝜋,𝜏 ) is that the precision of transition

41
Active Inference Tutorial

probabilities is reduced. This prevents overestimation of the precision of posteriors – something

†
that can occur with variational message passing. Also note that 𝐁𝜋,𝜏 denotes the transpose of 𝐁𝜋,𝜏
with normalized columns (i.e., columns that sum to 1). As presented here, this modification may
come across somewhat ad hoc. However, as with variational message passing, the update equations
for marginal message passing can be derived in a principled manner (described in (Parr et al.,
2019)).

2.4 Prediction Error Formulation

One strength of active inference is that it comes equipped with a biologically plausible instantiation
in terms of prediction-error minimization. In this subsection, we will introduce the reader to the
different types of prediction errors described in the active inference literature and how their
minimization affords state inference and policy selection. We will also provide explicit numerical
examples. By the end of this subsection, the reader should understand the basis of these prediction
errors, the role of the different terms within their respective equations, and how they relate to 𝑉𝐹𝐸
and 𝐸𝐹𝐸.

There are two types of prediction errors described in active inference – ‘state’ and ‘outcome’
prediction errors – based on the equations for 𝐹𝜋 and 𝐺𝜋 , respectively (see Table 2). State
prediction errors drive belief updating with respect to states and are based on message passing
algorithms. Outcome prediction errors drive policy selection. They are not based on an explicit
message passing algorithm, but they illustrate how 𝐺𝜋 can be formulated within the same type of
biologically plausible error-minimization scheme. We will now cover each of these types of
prediction errors in turn.

State prediction errors track how 𝐹𝜋 changes over time as beliefs about states 𝑠𝜋,𝜏 are updated (i.e.,
reductions in 𝐹𝜋 correspond to reductions in state prediction error). These prediction errors are
based on the marginal message passing scheme described in the previous technical section. For
those who skipped this section, please briefly review equation 23 above, which corresponds to
message passing between the square nodes and edges in the factor graph for an active inference
POMDP (shown in the bottom panel of Figure 6). As we now describe, it is this equation that can be
reformulated in terms of a state prediction-error signal that the brain seeks to minimize in order to
infer posteriors over states 𝑠𝜋,𝜏 , using three types of messages (i.e., message 1: ln 𝐁𝜋,𝜏−1 s𝜋,𝜏−1,
†
message 2: ln𝐁𝜋,𝜏 𝑠𝜋,𝜏+1 , and message 3: ln 𝐀T 𝑜𝜏 ). This is part of a more general mapping proposed
between the variables included in active inference and both neuronal and synaptic activity.

In the proposed mapping, firing rates in specific neuronal populations represent the continually
updated posteriors over states 𝑠𝜋,𝜏 – corresponding to edges in the factor graph shown in Figure 6
or circle nodes in the Bayes’ net depiction in Figure 5. Patterns of synaptic connection strengths
implement factors (i.e., the square nodes within graphs), such as the 𝐀 and 𝐁𝜋,𝜏 matrices, that
implement functions and transform the incoming messages encoded within firing rates; see (Parr &
Friston, 2018a). To simulate neuronal dynamics, one can set up an ordinary differential equation,
based on marginal message passing, that performs a gradient descent on 𝑉𝐹𝐸 by introducing the
state prediction error (𝜀𝜋,𝜏 ) as an auxiliary variable. This prediction error scores the difference

42
Active Inference Tutorial

between the log prior probability of each hidden state (i.e., the posterior from the previous time
step) and the log probability of each hidden state following a round of message passing (i.e., when a
new observation has been received). As described in the previous section, with each observation
there will be many rounds (iterations) of message passing (i.e., the message passing equation will
be repeated many times) until posterior beliefs over states converge to a stable value. All of this is
conditioned upon a specific policy (denoted by the subscript 𝜋), because the agent is trying to infer
the states it will occupy if it chooses one policy vs. another. To arrive at empirical predictions about
measurable neural responses, we can then substitute in a ‘depolarization’ or ‘voltage’ variable, 𝑣𝜋,𝜏 ,
to stand in for the log posterior over states; 𝑣𝜋,𝜏 = ln𝑠𝜋,𝜏 . The resulting state prediction error
equation and belief updating are then written as follows.

State Prediction Errors:

1 †
𝜀𝜋,𝜏 ← (ln(𝐁𝜋,𝜏−1 𝑠𝜋,𝜏−1 ) + ln(𝐁𝜋,𝜏 𝑠𝜋,𝜏+1 )) + ln𝐀T 𝑜𝜏 − ln𝑠𝜋,𝜏 (24)
2

𝑣𝜋,𝜏 ← 𝑣𝜋,𝜏 + 𝜀𝜋,𝜏 (25)

𝑠𝜋,𝜏 ← 𝜎(𝑣𝜋,𝜏 ) (26)

†
For those who skipped the technical section, note that 𝐁𝜋,𝜏 in equation 24 denotes the transpose of
𝐁𝜋,𝜏 with normalized columns (i.e., columns that sum to 1). In this equation, the combination of the
two 𝐁 matrices (combined with state beliefs) correspond to priors, whereas the 𝐀 matrix
(combined with observations) corresponds to the likelihood. The arrow notation indicates updates
to the value of a variable at each iteration. Equation 25 states that the change in level of
depolarization 𝑣𝜋,𝜏 with each iterative update corresponds to the prediction error 𝜀𝜋,𝜏 . Note that
𝜕𝐹𝜋
this error term corresponds to the rate of change in 𝑉𝐹𝐸; 𝜀𝜋,𝜏 = − . The updated value of 𝑣𝜋,𝜏 is
𝜕𝑠𝜋,𝜏
subsequently put through a softmax function (𝜎) in equation 26 to return an updated posterior
distribution over states 𝑠𝜋,𝜏 . The key aspect of this set of update equations is that the value of 𝑠𝜋,𝜏
continues to change (i.e., the equations are continually repeated) until the value of the state
prediction error term 𝜀𝜋,𝜏 is minimized. In other words, the equations are set up such that they
change the value of 𝑠𝜋,𝜏 (in the direction of steepest descent) until this produces the lowest value of
𝜀𝜋,𝜏 , at which point the resulting value of 𝑠𝜋,𝜏 will correspond to an approximate posterior over
states. This is because 𝜀𝜋,𝜏 = 0 is the attracting fixed point, meaning that the system tends to evolve
towards 𝜀𝜋,𝜏 = 0, and that once it reaches this value it will remain there. This leaves us with a
biologically plausible prediction-error minimization scheme that can perform posterior inference
over states and can be instantiated in a relatively simple neural network (see Section 5; for more
details, see (Parr & Friston, 2018a)). That is, by finding posterior beliefs over states 𝑠𝜋,𝜏 (on the far
right of equation 24) that minimize 𝜀𝜋,𝜏 , 𝐹𝜋 is minimized and 𝑠𝜋,𝜏 becomes a stable posterior belief.

As described in more detail below (and elsewhere; (Da Costa, Parr, Sengupta, & Friston, 2021)), the
variable 𝑣𝜋,𝜏 is used to model the average voltage or membrane potential of a neural population

43
Active Inference Tutorial

(i.e., by taking either positive or negative values), where one population is assumed to encode
information about each state factor (i.e., the probability of each state within that factor for each
time 𝜏 under each policy). The state variable 𝑠𝜋,𝜏 then corresponds to the firing rates of that
population, which are driven by their membrane potential. This is because 𝑠𝜋,𝜏 is the softmax (𝜎) of
the voltage and therefore, similar to a firing rate, takes only non-negative values (i.e., between 0 and
1). This follows from the assumption made in mean-field models of neural dynamics that the
average firing rate of a population can be treated as a sigmoid function of the average membrane
potential (Breakspear, 2017; Da Costa et al., 2021). Local field potentials (LFPs) and event-related
potentials (ERPs) in electroencephalography (EEG) research are then modelled as the time
derivative (rate of change) in the firing rates 𝑠𝜋,𝜏 .

To make these equations more concrete, consider a worked example with the following generative
model entailed by a policy, combined with a specific observation and initialized value of the
approximate posterior:

.8 .4 .9 .2 .2 .3 1 .5
𝐀=[ ]; 𝐁𝜋,𝜏−1 = [ ]; 𝐁𝜋,𝜏 = [ ]; 𝑜𝜏 = [ ]; 𝑠𝜋,𝜏 = [ ];
.2 .6 .1 .8 .8 .7 0 .5
−.6931
𝑣𝜋,𝜏 = ln𝑠𝜋,𝜏 = [ ]
−.6931

As can be seen here, the likelihood (𝐀) matrix indicates that outcome 1 (row 1) is more likely (i.e.,
𝑝 = .8) under state 1 (column 1). Under the policy being considered, the agent believes it will most
likely remain in its current state in the first state transition (i.e., p = .9 and .8 on the diagonal in
𝐁𝜋,𝜏−1 ; columns indicate states at time 𝜏 − 1, rows indicate state at time 𝜏) and more likely to move
to state 2 during the second state transition (i.e., p = .8 and .7 in the bottom row in 𝐁𝜋,𝜏 ). Further,
the agent observes outcome 1 at time 𝜏 (𝑜𝜏 ) and has a prior expectation that both states are equally
likely (𝑠𝜋,𝜏 and 𝑣). Here we will also set 𝑠𝜋,𝜏−1 = 𝑠𝜋,𝜏 = 𝑠𝜋,𝜏+1, which is a common initialization at
the beginning of a trial. However, the values of these three variables will often not remain equal
after new observations as a trial progresses. In this case, the error signal will be:

1 .9 .2 .5 . 475 . 525 . 5 .8 .2 1 .5
𝜀𝜋,𝜏 ← (ln ([ ] [ ]) + ln ([ ] [ ])) + ln ([ ] [ ]) − ln [ ]
2 .1 .8 .5 . 525 . 475 . 5 .4 .6 0 .5
1 . 55 .5 .8 .5
= (ln [ ] + ln [ ]) + ln [ ] − ln [ ]
2 . 45 .5 .4 .5

−.6455 . 4700
=[ ]+[ ]
−.7458 −.2231

−.1755
=[ ]
−.9690

This error signal will then update beliefs over states through the depolarization variable 𝑣𝜋,𝜏 as
follows:

−.6931 −.1755 −.8686

𝑣𝜋,𝜏 ← 𝑣𝜋,𝜏 + 𝜀𝜋,𝜏 = [ ] + [ ]= [ ]
−.6931 −.9690 −1.6621

44
Active Inference Tutorial

𝑒 −.8686
−.8686 + 𝑒 −1.6621 0.6886
𝑠𝜋,𝜏 ← 𝜎(𝑣𝜋,𝜏 ) = 𝑒 −1.6621 =[ ]
𝑒 0.3114
[𝑒 −.8686 + 𝑒 −1.6621 ]

Notice that in this example the variational update (i.e., single step of gradient descent) results in a
−.1755
negative value for the state prediction error term (i.e., 𝜀𝜋,𝜏 ← [ ]. As can be seen, this shifts
−.9690
the approximate posterior such that it will better minimize prediction error in the next variational
update (i.e., here increasing the probability of occupying state 1).

In contrast to state prediction errors, the outcome prediction errors mentioned above track how 𝐺𝜋
changes over time as beliefs about policies are updated (i.e., reductions in 𝐺𝜋 correspond to
reductions in outcome prediction error). In other words, when this type of prediction error is
minimized, policies are identified that minimize both uncertainty over states (i.e., ambiguity) and
the expected difference between predicted and preferred outcomes. However, as noted above, it is
important to clarify that, unlike state prediction errors, outcome prediction errors are not directly
tied to the message passing schemes described above – and the currently available routines in SPM
for performing active inference do not explicitly calculate outcome prediction errors. The current
implementation instead calculates 𝐺𝜋 directly. When this prediction error formulation has been
presented in previous literature, it has largely been for illustrative purposes with respect to
demonstration of biological plausibility (Parr & Friston, 2018a). However, calculating outcome
predictions errors could feasibly be added if one were interested in modelling the associated
neuronal responses predicted by this aspect of the process theory (for one recently proposed
scheme for inferring policies through message passing, see (Champion et al., 2021)). In contrast, the
current routines do calculate state prediction errors, which can be used without modification for
purposes of empirical prediction.

The update equation for outcome prediction error is as follows.

Outcome Prediction Errors:

𝜍𝜋,𝜏 = 𝐀𝑠𝜋,𝜏 ⋅ (ln 𝐀𝑠𝜋,𝜏 − ln 𝐂𝜏 ) − 𝑑𝑖𝑎𝑔(𝐀T ln 𝐀) ⋅ 𝑠𝜋,𝜏 (27)

This prediction error is best understood as a mixture of two types of expected predictions errors.
The first term, 𝐀𝑠𝜋,𝜏 ⋅ (ln 𝐀𝑠𝜋,𝜏 − ln 𝐂𝜏 ), corresponds to the expected difference between preferred
outcomes (i.e., the probability distribution encoding preferences over outcomes specified by 𝐂𝜏 )
and the outcomes expected under a policy (i.e., 𝐀𝑠𝜋,𝜏 corresponds to the observations expected
under a policy, o𝜋,𝜏 ). This can therefore be thought of as the expected prediction error (under each
policy) with respect to the observations predicted by prior preferences. The second term,
𝑑𝑖𝑎𝑔(𝐀T ln 𝐀) ⋅ 𝑠𝜋,𝜏 , corresponds to how much observations are expected to update beliefs if
adopting a particular policy (i.e., it is the entropy term, where lower entropy entails greater
information gain). Therefore, as with state prediction error, minimizing this term minimizes
uncertainty – but in this case it is uncertainty with respect to policies. Note that the 𝑑𝑖𝑎𝑔() function
simply takes the diagonal elements of a matrix and places them in a row vector. Note also that,
unlike with state prediction errors, we have not used the update (←) notation for outcome
prediction errors. This is because, in the current formulation of active inference, outcome
prediction errors are not iteratively minimized; they are simply computed once for each policy.

45
Active Inference Tutorial

Those who read the technical section on 𝑉𝐹𝐸 and 𝐸𝐹𝐸 will recognize these two terms as the matrix
forms of the risk (𝐷𝐾𝐿 [𝑞(𝑜𝜏 |𝜋)||𝑝(𝑜𝜏 )] ≈ 𝐀𝑠𝜋,𝜏 ⋅ (ln 𝐀𝑠𝜋,𝜏 − ln 𝐂𝜏 ), and ambiguity
(𝐸𝑞(𝑠|𝜋) [H[𝑝(𝑜𝜏 |𝑠𝜏 )] ≈ −𝑑𝑖𝑎𝑔(𝐀 ln 𝐀) ⋅ 𝑠𝜋,𝜏 ) terms in 𝐸𝐹𝐸.
T

Again, to make this more concrete we provide a worked example of each term under two possible
policies, assuming the following variable values:

.9 .1 1 .9 .5
𝐀=[ ]; 𝐂𝜏 = [ ]; 𝑠𝜋=1,𝜏 = [ ]; 𝑠𝜋=2,𝜏 = [ ]
.1 .9 0 .1 .5

In other words, the agent prefers outcome 1 (row 1), and the likelihood (𝐀) matrix indicates that
state 1 (column 1) is more likely to generate outcome 1 (i.e., 𝑝 = .9). Further, state beliefs under
policy 1 (𝑠𝜋=1,𝜏 ) entail a higher probability of being in state 1 (i.e., 𝑝 = .9) than state beliefs under
policy 2 (𝑠𝜋=2,𝜏 ; i.e., 𝑝 = .5). We can first calculate the risk (reward-seeking) term within the
outcome prediction error for each policy:

Policy 1:

. 82
𝑜𝜋=1,𝜏 = 𝐀𝑠𝜋=1,𝜏 = [ ]
. 18

𝐀𝑠𝜋=1,𝜏 ⋅ (ln 𝐀𝑠𝜋=1,𝜏 − ln 𝐂𝜏 ) =

. 82 . 82 1
[ ] ⋅ (ln [ ] − ln [ ]) = 2.4086
. 18 . 18 0

Policy 2:

.5
𝑜𝜋=2,𝜏 = 𝐀𝑠𝜋=2,𝜏 = [ ]
.5

𝐀𝑠𝜋=2,𝜏 ⋅ (ln 𝐀𝑠𝜋=2,𝜏 − ln 𝐂𝜏 ) =

.5 .5 1
[ ] ⋅ (ln [ ] − ln [ ]) = 7.3069
.5 .5 0

Note that a negligibly small number (here, 𝑒−16 ) is added to the values in 𝐂𝜏 because ln(0) is
undefined. As expected, because the approximate posterior over states for policy 1 makes the
generation of preferred observations more likely, policy 1 has lower values for the risk term (i.e.,
leading to lower outcome prediction error, all else being equal).

Moving onto the ambiguity (information-seeking term), consider another example with the
following variables:

.4 .2 .9 .1
𝐀=[ ]; 𝑠𝜋=1,𝜏 = [ ]; 𝑠𝜋=2,𝜏 = [ ]
.6 .8 .1 .9

In this case, the likelihood (𝐀) matrix indicates that state 2 (column 2) has a more precise
distribution than state 1 (column 1). In other words, observations are expected to provide more
precise information about states when in state 2. As such, we expect outcome prediction errors to

46
Active Inference Tutorial

drive selection of the policy that will lead the agent toward state 2. In this case, policy 2 assigns a
higher probability to state 2 (i.e., = .9 in row 2). We can confirm this by calculating the ambiguity
term for each policy as follows:

Policy 1:

.4 .6 .4 .2 .9
−𝑑𝑖𝑎𝑔(𝐀T ln 𝐀) ⋅ 𝑠𝜋=1,𝜏 = −𝑑𝑖𝑎𝑔 ([ ] ln [ ]) ∙ [ ]
.2 .8 .6 .8 .1

−.67 −.78 .9 −.67 . 9

= −𝑑𝑖𝑎𝑔 ([ ]) ∙ [ ] = [ ]∙[ ]
−.59 −.50 .1 −.50 . 1

= −(−.66) = .66

Policy 2:

.4 .6 .4 .2 .1
−𝑑𝑖𝑎𝑔(𝐀T ln 𝐀) ⋅ 𝑠𝜋=2,𝜏 = −𝑑𝑖𝑎𝑔 ([ ] ln [ ]) ∙ [ ]
.2 .8 .6 .8 .9
−.67 −.78 .1 −.67 . 1
= −𝑑𝑖𝑎𝑔 ([ ]) ∙ [ ] = [ ]∙[ ]
−.59 −.50 .9 −.50 . 9

= −(−.52) = .52

As expected, because the outcomes generated by state 1 are more ambiguous (i.e., less informative),
and policy 2 assigns a higher probability to state 2 than policy 1, policy 2 better minimizes
ambiguity.

It is important to stress that the risk and ambiguity terms for outcome prediction errors work
synergistically, and one often has policies that minimize both risk and ambiguity. As can be seen in
the outcome prediction error equation above, subtracting the ambiguity term from the risk term
corresponds to adding (i.e., note the double negative) the entropy of the likelihood mapping under a
policy to the risk of the policy, which we have calculated separately here. This drives selection of
policies that maximize both reward- and information-seeking by minimizing the overall resulting
error.

While these example calculations may appear somewhat involved (even for a single policy), an
intuitive way to think about these two prediction errors is that minimizing state prediction error
maximizes confidence in posterior beliefs, while minimizing outcome prediction error maximizes
confidence in how to achieve goals or desires. To reproduce the worked examples above and allow
the reader to calculate state and outcome prediction errors under different model parameters, we
have provided the 𝐏𝐫𝐞𝐝𝐢𝐜𝐭𝐢𝐨𝐧_𝐞𝐫𝐫𝐨𝐫_𝐞𝐱𝐚𝐦𝐩𝐥𝐞. 𝐦 script in the supplementary code.

Table 2. Matrix formulation of equations used for inference

Model Update equation Explanation Model-specific

update description for
component explore-exploit task
(described in detail

47
Active Inference Tutorial

Section 3)

Updating First equation: The Updating beliefs about:

beliefs variable 𝜀𝜋,𝜏=1 is the
about initial 1 † state prediction error 1. Whether the left
𝜀𝜋,𝜏=1 ← (ln𝐷 + ln(𝐁𝜋,𝜏 𝑠𝜋,𝜏+1 ))
states 2 with respect to the vs. right slot
expected + ln 𝐀T 𝑜𝜏 machine is
first time point in a
under each − ln𝑠𝜋,𝜏=1 more likely to
trial. Minimizing this
allowable error corresponds to pay out on a
policy. minimizing 𝑉𝐹𝐸 (via given trial.
gradient descent) and 2. The initial
1
𝑠𝜋,𝜏=1 = 𝜎 ( (ln𝐷 is used to update choice state
2
† posterior beliefs over (here, always
+ ln(𝐁𝜋,𝜏 𝑠𝜋,𝜏+1 ))
states. The term the ‘start’ state).
+ ln𝐀T 𝑜𝜏 )
(ln𝐷 +
†
ln(𝐁𝜋,𝜏 𝑠𝜋,𝜏+1 ))
corresponds to prior
beliefs in Bayesian
inference, based on
beliefs about the
probability of initial
states, 𝐷, and the
probability of
transitions to future
states under a policy,
†
ln (𝐁𝜋,𝜏 𝑠𝜋,𝜏+1 ). The
term 𝐀 𝑜𝜏
T

corresponds to the
likelihood term in
Bayesian inference,
evaluating how
consistent observed
outcomes are with
each possible state.
The term ln𝑠𝜋,𝜏=1
corresponds to
posterior beliefs over
states (for the first
time point in a trial)
at the current update
iteration.

Second Equation: We

48
Active Inference Tutorial

move to the solution

for the posterior
𝑠𝜋,𝜏=1 by setting
𝜀𝜋,𝜏=1 = 0, solving for
ln𝑠𝜋,𝜏=1 , and then
taking the softmax
(normalized
exponential) function
(denoted 𝜎) to ensure
that the posterior
over states is a proper
probability
distribution with non-
negative values that
sums to 1. This
equation is described
in more detail in the
main text. A
numerical example of
the softmax function
is also shown in
Appendix 1.

Updating First equation: The Updating beliefs about:

beliefs variable 𝜀𝜋,𝜏>1 is the
about all 1 state prediction error 1. Whether the left
𝜀𝜋,𝜏>1 ← (ln(𝐁𝜋,𝜏−1 𝑠𝜋,𝜏−1 )
states after 2 with respect to all vs. right slot
†
the first + ln(𝐁𝜋,𝜏 𝑠𝜋,𝜏+1 )) time points in a trial machine is
time point + ln𝐀T 𝑜𝜏 after the first time more likely to
in a trial − ln𝑠𝜋,𝜏>1 point. Minimizing this pay out on a
that are error corresponds to given trial.
expected minimizing 𝑉𝐹𝐸 (via 2. Beliefs about
under each gradient descent) and choice states
1
allowable 𝑠𝜋,𝜏>1 = 𝜎 ( (ln(𝐁𝜋,𝜏−1 s𝜋,𝜏−1 ) is used to update after the initial
2 time point
policy. † posterior beliefs over
+ ln(𝐁𝜋,𝜏 𝑠𝜋,𝜏+1 )) (here, this
states. The term
+ ln 𝐀T 𝑜𝜏 ) (ln(𝐁𝜋,𝜏−1 s𝜋,𝜏−1 ) + depends on the
† choice to take
ln (𝐁𝜋,𝜏 𝑠𝜋,𝜏+1 ))
the hint or to
corresponds to prior
select one of the
beliefs in Bayesian
slot machines).
inference, based on
beliefs about the
probability of

49
Active Inference Tutorial

transitions from past

states,
ln(𝐁𝜋,𝜏−1 s𝜋,𝜏−1 ), and
the probability of
transitions to future
†
states, ln(𝐁𝜋,𝜏 s𝜋,𝜏+1 ),
under a policy. The
term ln𝐀T 𝑜𝜏
corresponds to the
likelihood term in
Bayesian inference,
evaluating how
consistent observed
outcomes are with
each possible state.

Second Equation: As
in the previous row,
we move to the
solution for the
posterior, 𝑠𝜋,𝜏>1 , by
setting 𝜀𝜋,𝜏>1 = 0,
solving for ln𝑠𝜋,𝜏>1 ,
and then taking the
softmax function (𝜎).
This equation is
described in more
detail in the main
text.

Probability The initial Updating overall beliefs

of selecting distribution over about whether the best
each 𝜋0 = 𝜎(ln 𝐸 − 𝛾𝐺) policies before course of action is to
allowable making any take the hint and/or to
policy observations (𝜋0 ), choose the left vs. right
and the posterior slot machine.
𝜋 = 𝜎(ln 𝐸 − 𝐹 − 𝛾𝐺)
distribution over
policies after an
observation (𝜋). The
initial distribution is
made up of the
learned prior over
policies encoded in

50
Active Inference Tutorial

the 𝐸 vector
(reflecting the
number of times a
policy has previously
been chosen) and the
expected free energy
of each allowable
policy (𝐺). The
posterior distribution
is determined by 𝐸, 𝐺,
and the variational
free energy (𝐹) under
each policy after
making a new
observation. The
influence of 𝐺 is also
modulated by an
expected precision
term (𝛾), which
encodes prior
confidence in beliefs
about 𝐺 (described
further in the main
text; also see Figure
9). See row 1 for an
explanation of the
function of the 𝜎
symbol.

We note, however,
that incorporation of
𝐸, 𝐹, and/or 𝛾 when
computing 𝜋 is a
modelling choice.
These need not be
included in all cases
(e.g., see top-left
portion of Figure 5;
also see (Da Costa,
Parr, et al., 2020)). In
some contexts, one
might choose to
include some of these

51
Active Inference Tutorial

terms but not others,

or to only include 𝐺.
This depends on the
research question.
(e.g., 𝐸 will be useful
if task behavior is
influenced by habits,
while 𝐹/𝛾 can be
useful when there are
many possible deep
policies to choose
from). See the row in
this table on
“Expected free energy
precision” for more
details about
inference over
policies when 𝐹/𝛾 are
included. This is also
discussed further in
the main text.

Expected The first equation The ‘risk’ term –

free energy reproduces the ‘risk + 𝐷𝐾𝐿 [𝑞(𝑜|𝜋)||𝑝(𝑜|𝐶)] =
of each 𝐺𝜋 ambiguity’ expression 𝐀𝑠𝜋,𝜏 ⋅ (ln 𝐀𝑠𝜋,𝜏 − ln 𝐂𝜏 )
allowable = 𝐷𝐾𝐿 [𝑞(𝑜|𝜋)||𝑝(𝑜|𝐶)] for the expected free – drives the agent to
policy + E𝑞(𝑠|𝜋) [H[𝑝(𝑜|𝑠)]] energy of each policy select the slot machine
(𝐺𝜋 ) that is explained expected to be most
in the main text. The likely to pay out. If the
second equation value of winning money
𝐺𝜋
shows this same in 𝐂𝜏 is high enough
= ∑(𝐀𝑠𝜋,𝜏 ⋅ (ln 𝐀𝑠𝜋,𝜏 − ln 𝐂𝜏 ) expression in terms of (i.e., if 𝑝(𝑜|𝐶) is
𝜏 the elements in the sufficiently precise),
− 𝑑𝑖𝑎𝑔(𝐀T ln𝐀) ⋅ 𝑠𝜋,𝜏 ) POMDP model used in this will deter the agent
this tutorial (i.e., in from choosing to ask
matrix notation). for the hint.

Expected free energy The ‘ambiguity’ term –

evaluates the value of E𝑞(𝑠|𝜋) [H[𝑝(𝑜|𝑠)]] =
each policy based on −𝑑𝑖𝑎𝑔(𝐀T ln𝐀) ⋅ 𝑠𝜋,𝜏 –
their expected ability drives the agent to
to: 1) generate the
minimize uncertainty
most desired

52
Active Inference Tutorial

outcomes, and 2) by asking for the hint.

minimize uncertainty
about hidden states.
Achieving the most
desired outcomes
corresponds to
minimizing the KL
divergence between
preferred
observations,
𝑝(𝑜|𝐶) = 𝐂𝜏 , and the
observations
expected under each
policy, 𝑞(𝑜|𝜋) =
𝐀𝑠𝜋,𝜏 = 𝑜𝜋,𝑡 .
Minimizing
uncertainty
corresponds to
minimizing the
expected entropy of
the likelihood
(E𝑞(𝑠|𝜋) [H[𝑝(𝑜|𝑠)]] =
−𝑑𝑖𝑎𝑔(𝐀T ln𝐀) ⋅
𝑠𝜋,𝜏 ). Note that the
𝑑𝑖𝑎𝑔() function
simply takes the
diagonal elements of
a matrix and places
them in a row vector.
This is simply a
convenient method
for extracting and
operating on the
correct matrix entries
to calculate the
entropy, H[𝑝(𝑜|𝑠)] =
− ∑ 𝑝(𝑜|𝑠)ln𝑝(𝑜|𝑠), of
the distributions
encoded within each
column in 𝐀. For
simple numerical
examples of
calculating the risk

53
Active Inference Tutorial

and ambiguity terms,

see discussion of
‘outcome prediction
errors’ in Section 2.4.

Marginal The first equation This would encode the

free energy shows the marginal amount of surprise
of each 𝐹𝜋 (as opposed to (given a choice of
allowable = Ε𝑞(𝑠|𝜋) [ln 𝑞(𝑠|𝜋) variational) free policy) when observing
policy energy, which is now a hint or a win/loss
1
− Ε𝑞(𝑠𝜏−1 |𝜋) [ln𝑝(𝑠𝜏 |𝑠𝜏−1 , 𝜋)] used in the most after selecting a specific
2
1 recent slot machine.
− Ε𝑞(𝑠𝜏+1 |𝜋) [ln𝑝(𝑠𝜏 |𝑠𝜏+1 , 𝜋)] implementations of
2
active inference. The
− ln𝑝(𝑜𝜏 |𝑠𝜏 )]
second equation
shows this same
𝐹𝜋 expression in terms of
= ∑ 𝑠𝜋,𝜏 the elements in the
𝜏 POMDP model used in
∙ (ln𝑠𝜋,𝜏 this tutorial (i.e., in
1 matrix notation).
− (ln(𝐁𝜋,𝜏−1 𝑠𝜋,𝜏−1 ) Marginal free energy
2
† has a sightly different
+ ln (𝐁𝜋,𝜏 𝑠𝜋,𝜏+1 )) − ln 𝐀T 𝑜𝜏 )
form than the
expressions for 𝑉𝐹𝐸
that are also shown in
the text (and which
have been used in
many previous
papers in the active
inference literature).
This updated form
improves on certain
limitations of the
message passing
algorithms derived
from minimization of
𝑉𝐹𝐸 (see Section 2.3;
also see (Parr et al.,
2019)).

54
Active Inference Tutorial

Marginal free energy

evaluates the
evidence that inferred
states provide for
each policy (based on
new observations at
each time point). See
the first two rows in
this table on updating
beliefs about states
for an explanation of
how each term in the
equation relates to
Bayesian inference.

Expected The 𝛽 term, and its A higher value for 𝛽

free energy prior value 𝛽0 , is a would reduce an
precision 𝑝(γ) = Γ(1, 𝛽) hyperparameter on agent’s confidence in
the expected free the best policy based on
E[γ] = 𝛾 = 1/𝛽
energy precision term the values in 𝐺. This
(𝛾). Specifically, 𝛽 is might lead the agent to
the 'rate' parameter select a slot machine
Iterated to convergence: of a gamma more randomly or
distribution (Γ) with a based to a greater
𝜋0 ← 𝜎(ln 𝐸 − 𝛾𝐺) ‘shape’ parameter extent on its past
value of 1. The choices (i.e., if it has a
𝜋 ← 𝜎(ln 𝐸 − 𝐹 − 𝛾𝐺) expected value of this precise prior over
distribution, E[γ] = 𝛾, policies in the vector
𝐺𝑒𝑟𝑟𝑜𝑟 ← (𝜋 − 𝜋0 ) ⋅ (−𝐺) is equal to the 𝐸).
reciprocal of 𝛽. Note
𝛽𝑢𝑝𝑑𝑎𝑡𝑒 ← 𝛽 − 𝛽0 + 𝐺𝑒𝑟𝑟𝑜𝑟
that we use the non-
𝛽 ← 𝛽 − 𝛽𝑢𝑝𝑑𝑎𝑡𝑒 /𝜓 italicized γ to refer to
the random variable
𝛾 ← 1/𝛽 and use the italicized
𝛾 to refer to the scalar
value of that variable.
This scalar is what is
subsequently updated
based on the
equations shown
here.

The 𝛾 term controls

55
Active Inference Tutorial

the precision of 𝐺,
based on the agent’s
confidence in its
estimates of expected
free energy. This
confidence changes
when new
observations are
consistent or
inconsistent with 𝐺.
More specifically, 𝛾
modulates the
influence of 𝐺 on
policy selection based
upon a 𝐺 prediction
error (𝐺𝑒𝑟𝑟𝑜𝑟 ). This is
calculated based on
the difference
between the initial
distribution over
policies (𝜋0 ) and the
posterior distribution
after making a new
observation (𝜋). The
difference between
these terms reflects
the extent to which
new observations
(scored by 𝐹) make
policies more or less
likely. If the vector
encoding the
posterior over
policies increases in
magnitude in
comparison to the
prior, and still points
in the same direction,
the difference vector
between the posterior
and the prior will
point in the same
direction as the −𝐺

56
Active Inference Tutorial

vector (i.e., less than a

90° angle apart; see
Figure 9). If so, the
value of 𝛾 will
increase, thereby
increasing the impact
of 𝐺 on policy
selection. In contrast,
if the difference
vector between the
posterior and the
prior does not point
in the same direction
as the −𝐺 vector (i.e.,
greater than a 90°
angle apart), 𝛾 will
decrease and thereby
reduce the impact of
𝐺 on policy selection
(i.e., as the agent’s
confidence in its
estimates of expected
free energy has
decreased).

Note that the 𝛽𝑢𝑝𝑑𝑎𝑡𝑒

term mediating these
updates technically
corresponds to the
gradient of free
energy with respect
to 𝛾 (∇𝛾 𝐹). The
subsequent update in
the value of 𝛾 is such
that 𝐺 contributes to
the posterior over
policies in an optimal
manner. 𝛽 and 𝐺𝑒𝑟𝑟𝑜𝑟
are often discussed in
relation to dopamine
in the active inference
literature.

57
Active Inference Tutorial

Note that 𝛽0 is the

initial prior (which is
not updated), and 𝛽 is
the initial posterior,
which is subsequently
updated to provide a
new estimate for 𝛾 =
1/𝛽. The variable 𝜓
is a step size
parameter that
reduces the
magnitude of each
update and promotes
stable convergence to
final values of 𝛾. For a
derivation of these
equations, see
Appendix in (Sales,
Friston, Jones,
Pickering, & Moran,
2019).

†
Table note: The term 𝐁𝜋,𝜏 denotes the transpose of 𝐁𝜋,𝜏 with normalized columns (i.e., columns that sum to
1). Note that you may commonly see the dot (∙) notation used in the active inference literature to denote
transposed matrix multiplication, such as 𝐀 ∙ 𝑜𝜏 , which means 𝐀T 𝑜𝜏 (we use the latter notation here). When 𝐀
matrices have more than two dimensions (i.e., when they are tensors), the transpose is applied to the two-
dimensional matrix associated with each value of the other dimensions. The 𝜎 symbol indicates a softmax
operation (for an introduction see Appendix 1), which transforms vector values to make up a proper
probability distribution (i.e., with non-negative values that sum to 1). Italicized variables indicate vectors (or
single numbers [scalars] in the case of 𝛽 and 𝛾). Bolded, non-italicized variables indicate matrices. Subscripts
indicate conditional probabilities; e.g., 𝑠𝜋,𝜏 = 𝑝(𝑠𝜏 |𝜋).

3. Building Specific Task Models

3.1 Explore-Exploit Task

To make the structure of a POMDP more concrete, in this and subsequent sections we will build
models of specific behavioral tasks commonly used in empirical studies. This will provide the
reader with the necessary tools to build their own models and use them in both simulation work
and empirical studies. This will also allow us to concretely demonstrate some of the unique
resources offered by active inference when modelling behavior in a simple reinforcement learning
context. To be sure, in many task contexts (e.g., when there is no uncertainty about states) active
inference models can perform similarly to reinforcement learning models, and they do not always

58
Active Inference Tutorial

generate optimal behavior (Da Costa, Sajid, Parr, Friston, & Smith, 2020; Markovic, Stojic,
Schwoebel, & Kiebel, 2021)5. However, as discussed above, when tasks involve various types of
uncertainty (e.g., about task condition or reward probabilities), active inference offers a unique
approach for modelling information-seeking behavior that can lead to superior performance in
some cases (Markovic et al., 2021; Sajid, Ball, Parr, & Friston, 2021). Another resource offered by
active inference, even when it performs similarly to other types of models, is its associated neural
process theory (i.e., describing how neural signaling might implement variational or marginal
message passing; see Section 5). The task models we build in this tutorial will further allow us to
illustrate how active inference can be used to make testable empirical predictions about neural
responses. As we will see, because active inference models integrate perception, learning, and
decision-making within a single model architecture, this affords the generation of predictions about
neural responses across a wide range of perceptual tasks in addition to reinforcement learning and
decision-making tasks.

In this subsection, we will build a model of the explore-exploit task briefly described in the previous
section. Every step we outline in this section for building the explore-exploit task model is laid out
in the accompanying MATLAB code (Step_by_Step_AI_Guide.m). This code is included in the
supplementary code files and can also be found at: [Link]
Inference-Tutorial-Scripts. While going through this section, we encourage the reader to work
through this code in parallel. Here we will use non-bolded italics when presenting the general
mathematical notation and subsequently show the associated MATLAB syntax in bold.

In the beginning of the explore-exploit task, the participant is told that on each trial one machine
will tend to pay out more often, but they will not know which one. They are also told that the better
machine will not always be the same on each trial. They can choose to select one right away and
possibly win $4. Or they can choose to press a button that gives them a hint about which slot
machine is better on that trial. However, if they choose to take the hint, they can only win $2 if they
pick the correct machine. Over many trials, the participant can learn which slot machine tends to
pay out more often and either make safe or risky choices (i.e., take the hint or not).

To model this task, it can be helpful to start by specifying the sets of possible hidden states (state
factors). In this case, one state factor corresponds to whether the left or right slot machine is more
likely to win (‘left-better context’, ’right-better context’). The second state factor corresponds to the
choice state (‘start state’, ‘asking for the hint’, ‘choosing the left machine’, ‘choosing the right
machine’). Moving to MATLAB code, we can set these up by specifying the priors over initial states
with a set of vectors D, with one vector per state factor (i.e., where the factor number is specified in
brackets). The general structure for these vectors is:

𝑓𝑎𝑐𝑡𝑜𝑟
𝑝(𝑠𝜏=1 )=

5
Although note that, even in task contexts where they perform similarly, active inference models and
reinforcement learning models make decisions in a different way. Specifically, while reinforcement learning
models seek to maximize a reward signal, active inference models instead seek to reach a target distribution that is
treated as rewarding (i.e., the distribution encoding the agent’s preferred observations).

59
Active Inference Tutorial

𝑫{𝐬𝐭𝐚𝐭𝐞 𝐟𝐚𝐜𝐭𝐨𝐫}(𝐬𝐭𝐚𝐭𝐞, 𝟏) = [𝐯𝐞𝐜𝐭𝐨𝐫]

In this case, we specify:

𝑐𝑜𝑛𝑡𝑒𝑥𝑡 )
𝑝(𝑠𝜏=1 =

𝑫{𝟏} = [𝟎. 𝟓 𝟎. 𝟓]′

𝑐ℎ𝑜𝑖𝑐𝑒
𝑝(𝑠𝜏=1 )=

𝑫{𝟐} = [𝟏 𝟎 𝟎 𝟎]′

Note that, to match MATLAB syntax, we use the apostrophe (‘) to indicate a transpose. This says
that the participant begins with the belief that the ‘left-better’ and ‘right-better’ contexts have equal
probability (left and right entries, respectively), and with a fully precise belief that he/she will start
the trial in the start state (from left to right: ‘start’, ‘get hint’, ‘choose left’, and ‘choose right’ states).

It is important to briefly note, however, that things change slightly if we wish to simulate learning
as opposed to just inference, because we need to separate the generative process from the
generative model. In this case, capital 𝐷 stands for initial state probabilities in the generative
process, while lowercase 𝑑 stands for the initial state priors in the generative model (which are
learned). For example, one could specify 𝐷{1} = [1 0]’ and 𝑑{1} = [.5 .5]’, which would mean the
true context is ‘left-better’ but the agent believes each context is equally likely. In the
supplementary code accompanying this section, we do this as a way of controlling which context
we want to simulate. The same capital vs. lowercase letter convention holds for all other
matrices/vectors used here. We return to this in the section on learning further below (Section 4).

Moving forward, we must next specify the (in this case three) sets of possible observations
(outcome modalities). Here, the first set of observations corresponds to the hint (‘no hint’,
‘machine-left hint’, ‘machine-right hint’). The second set of observations corresponds to decision
outcomes (‘start’, ‘lose’, and ‘win’). Finally, the participant also observes their own behavior;
namely, the observed action (‘start’, ‘asking for the hint’, ‘choosing the left machine’, ‘choosing the
right machine’). This last outcome modality can be important in active inference models because
choice states must be inferred just like any other state. Observing their own behavior therefore
allows a participant to be more confident about whether their intended actions have been
successfully carried out. In MATLAB, we can set these up by specifying the likelihood (𝐀) matrices.
There will always be one 𝐀 matrix for each outcome modality. Rows correspond to outcomes,
columns correspond to the states in the first state factor, and there is an additional dimension
for each additional state factor (note that, as mentioned in Table 1, the 𝐀 matrices are more
correctly referred to as tensors if they include more than two dimensions). The general structure is
therefore:

𝑚𝑜𝑑𝑎𝑙𝑖𝑡𝑦 𝑓𝑎𝑐𝑡𝑜𝑟
𝑝(𝑜𝜏 |𝑠𝜏 )=

𝐀{𝐨𝐮𝐭𝐜𝐨𝐦𝐞 𝐦𝐨𝐝𝐚𝐥𝐢𝐭𝐲}(𝐨𝐮𝐭𝐜𝐨𝐦𝐞, 𝐟𝐚𝐜𝐭𝐨𝐫 𝟏, 𝐟𝐚𝐜𝐭𝐨𝐫 𝟐, … , 𝐟𝐚𝐜𝐭𝐨𝐫 𝐍) = [𝐦𝐚𝐭𝐫𝐢𝐱]

60
Active Inference Tutorial

In this case, only the ‘get hint’ state (state 2) in state factor 2 generates a hint observation, so for the
third dimension we specify a ‘2’ as follows:

𝑐𝑜𝑛𝑡𝑒𝑥𝑡,𝑐ℎ𝑜𝑖𝑐𝑒=𝑔𝑒𝑡 ℎ𝑖𝑛𝑡
𝑝(𝑜𝜏ℎ𝑖𝑛𝑡 |𝑠𝜏 )=

𝟎 𝟎
𝐀{𝟏}(: , ∶, 𝟐) = [𝟏 𝟎]
𝟎 𝟏

Here, columns from left to right correspond to the ‘left-better’ and ‘right-better’ states, while rows
from top to bottom correspond to the ‘no hint’, ‘machine-left hint’, ‘machine-right hint’
observations. This matrix indicates that the hint is accurate with a probability of 1 (100%
accuracy). For example, a ‘machine-left hint’ observation (row two) will be generated by the ‘left-
better context’ (column one) with probability = 1. Here each column must add up to 1.

For the other dimensions of state factor 2 (i.e., matrix dimension 3):

𝑐𝑜𝑛𝑡𝑒𝑥𝑡,𝑐ℎ𝑜𝑖𝑐𝑒=𝑠𝑡𝑎𝑟𝑡,𝑐ℎ𝑜𝑜𝑠𝑒 𝑙𝑒𝑓𝑡,𝑐ℎ𝑜𝑜𝑠𝑒 𝑟𝑖𝑔ℎ𝑡

𝑝(𝑜𝜏ℎ𝑖𝑛𝑡 |𝑠𝜏 )=

𝒇𝒐𝒓 𝒊 = 𝟏, 𝟑, 𝟒:

𝟏 𝟏
𝐀{𝟏}(: , ∶, 𝒊) = [𝟎 𝟎]
𝟎 𝟎

𝒆𝒏𝒅

This indicates that all other choice states will generate the ‘no hint’ observation (i.e., a hint will
never be observed in those states).

For the second outcome modality, the ‘start’ and ‘get hint’ states generate ‘start’ observations (row
1):

𝑐𝑜𝑛𝑡𝑒𝑥𝑡,𝑐ℎ𝑜𝑖𝑐𝑒=𝑠𝑡𝑎𝑟𝑡,𝑔𝑒𝑡 ℎ𝑖𝑛𝑡
𝑝(𝑜𝜏𝑤𝑖𝑛 |𝑠𝜏 )=

𝒇𝒐𝒓 𝒊 = 𝟏, 𝟐:

𝟏 𝟏
𝐀{𝟐}(: , ∶, 𝒊) = [𝟎 𝟎]
𝟎 𝟎

𝒆𝒏𝒅

We will now specify the probability of winning in the ‘left-better context’ vs. the ‘right-better
context’ depending on choice state. First, we specify that choosing the left machine (i.e.,
transitioning to the ‘choose left machine’ state; state 3 in factor 2) will lead to a win 80% of the time
(row 3) if in the ‘left-better context’ (column 1) and lead to a win 20% of the time if in the ‘right-

61
Active Inference Tutorial

better context’ (column 2), with inverse probabilities for a loss (row 2), and a probability of 0 of
continuing to observe the ‘start’ observation (row 1):

𝑐𝑜𝑛𝑡𝑒𝑥𝑡,𝑐ℎ𝑜𝑖𝑐𝑒=𝑐ℎ𝑜𝑜𝑠𝑒 𝑙𝑒𝑓𝑡
𝑝(𝑜𝜏𝑤𝑖𝑛 |𝑠𝜏 )=

𝟎 𝟎
𝐀{𝟐}(: , ∶, 𝟑) = [. 𝟐 . 𝟖]
.𝟖 .𝟐

We will then specify that the probabilities of winning are reversed if choosing the right machine
(i.e., transitioning to the ‘choose right machine’ state; state 4 in factor 2):

𝑐𝑜𝑛𝑡𝑒𝑥𝑡,𝑐ℎ𝑜𝑖𝑐𝑒=𝑐ℎ𝑜𝑜𝑠𝑒 𝑟𝑖𝑔ℎ𝑡
𝑝(𝑜𝜏𝑤𝑖𝑛 |𝑠𝜏 )=

𝟎 𝟎
𝐀{𝟐}(: , ∶, 𝟒) = [. 𝟖 . 𝟐]
.𝟐 .𝟖

Remember, the first column is the context where the left machine is better, and the second column
is the context where the right machine is better. It is the third dimension that corresponds to choice
states (in this case, choice state 3 and 4, corresponding to choosing the left vs. right machine).

Finally, for the third observation modality (observed action), states simply map 1-to-1 to outcomes
(across all other state combinations):

𝑝(𝑜𝜏𝑜𝑏𝑠𝑒𝑟𝑣𝑒𝑑 𝑎𝑐𝑡𝑖𝑜𝑛 |𝑠𝜏𝑐𝑜𝑛𝑡𝑒𝑥𝑡,𝑐ℎ𝑜𝑖𝑐𝑒=𝑠𝑡𝑎𝑟𝑡 ) =

𝟏 𝟏
𝟎 𝟎
𝐀{𝟑}(: , ∶, 𝟏) = [ ]
𝟎 𝟎
𝟎 𝟎
𝑐𝑜𝑛𝑡𝑒𝑥𝑡,𝑐ℎ𝑜𝑖𝑐𝑒=𝑔𝑒𝑡 ℎ𝑖𝑛𝑡
𝑝(𝑜𝜏𝑜𝑏𝑠𝑒𝑟𝑣𝑒𝑑 𝑎𝑐𝑡𝑖𝑜𝑛 |𝑠𝜏 )=

𝟎 𝟎
𝟏 𝟏
𝐀{𝟑}(: , ∶, 𝟐) = [ ]
𝟎 𝟎
𝟎 𝟎
𝑐𝑜𝑛𝑡𝑒𝑥𝑡,𝑐ℎ𝑜𝑖𝑐𝑒=𝑐ℎ𝑜𝑜𝑠𝑒 𝑙𝑒𝑓𝑡
𝑝(𝑜𝜏𝑜𝑏𝑠𝑒𝑟𝑣𝑒𝑑 𝑎𝑐𝑡𝑖𝑜𝑛 |𝑠𝜏 )=

𝟎 𝟎
𝟎 𝟎
𝐀{𝟑}(: , ∶, 𝟑) = [ ]
𝟏 𝟏
𝟎 𝟎
𝑐𝑜𝑛𝑡𝑒𝑥𝑡,𝑐ℎ𝑜𝑖𝑐𝑒=𝑐ℎ𝑜𝑜𝑠𝑒 𝑟𝑖𝑔ℎ𝑡
𝑝(𝑜𝜏𝑜𝑏𝑠𝑒𝑟𝑣𝑒𝑑 𝑎𝑐𝑡𝑖𝑜𝑛 |𝑠𝜏 )=

62
Active Inference Tutorial

𝟎 𝟎
𝟎 𝟎
𝐀{𝟑}(: , ∶, 𝟒) = [ ]
𝟎 𝟎
𝟏 𝟏

This simply allows the individual to infer what their choice was with complete certainty (rows top
to bottom: ‘start’, ‘asking for the hint’, ‘choosing the left machine’, ‘choosing the right machine’
observations).

Now that we have the likelihood, the next step is to specify the (policy-dependent) state transition
(𝐁𝜋,𝜏 ) matrices. The general structure for these matrices is:

𝑓𝑎𝑐𝑡𝑜𝑟 𝑓𝑎𝑐𝑡𝑜𝑟
𝑝(𝑠𝜏+1 |𝑠𝜏 , 𝑈) =

𝐁{𝐬𝐭𝐚𝐭𝐞 𝐟𝐚𝐜𝐭𝐨𝐫}(𝐬𝐭𝐚𝐭𝐞 𝐚𝐭 𝐭𝐢𝐦𝐞 𝝉 + 𝟏, 𝐬𝐭𝐚𝐭𝐞 𝐚𝐭 𝐭𝐢𝐦𝐞 𝝉, 𝐚𝐜𝐭𝐢𝐨𝐧 𝐧𝐮𝐦𝐛𝐞𝐫) = [𝐦𝐚𝐭𝐫𝐢𝐱]

Here the vector 𝑈 contains indices specifying the action number assigned to each matrix, where
each policy 𝜋 subsequently specifies a sequence of these action numbers (described further below).

Because we have two state factors, we need two sets of matrices. The first matrix is for the context
factor. In this case, because a context remains the same within each trial, this is simply an identity
matrix that says states at time 𝝉 (columns) remain the same at 𝝉 + 𝟏 (rows):

𝑐𝑜𝑛𝑡𝑒𝑥𝑡 𝑐𝑜𝑛𝑡𝑒𝑥𝑡
𝑝(𝑠𝜏+1 |𝑠𝜏 , 𝑈) =

𝟏 𝟎
𝐁{𝟏}(: , ∶, 𝟏) = [ ]
𝟎 𝟏

Columns from left to right and rows from top to bottom both correspond to the ‘left-better’ and
‘right-better’ states. There is only one ‘action’ (possible transition from each state) for this factor, so
the third dimension is a 1 and has a length of 1; i.e., there is no B{1}(: , ∶, 2).

In contrast, the second state factor is choice state, where four different transitions (actions) are
possible at each time step. In this case, each matrix below indicates that one could move from any
state to the chosen state:

𝑐ℎ𝑜𝑖𝑐𝑒 𝑐ℎ𝑜𝑖𝑐𝑒
𝑝(𝑠𝜏+1 |𝑠𝜏 , 𝑈 = 𝑠𝑡𝑎𝑟𝑡) =

𝟏 𝟏 𝟏 𝟏
𝟎 𝟎 𝟎 𝟎
𝐁{𝟐}(: , ∶, 𝟏) = [ ]
𝟎 𝟎 𝟎 𝟎
𝟎 𝟎 𝟎 𝟎
𝑐ℎ𝑜𝑖𝑐𝑒 𝑐ℎ𝑜𝑖𝑐𝑒
𝑝(𝑠𝜏+1 |𝑠𝜏 , 𝑈 = 𝑔𝑒𝑡 ℎ𝑖𝑛𝑡) =

63
Active Inference Tutorial

𝟎 𝟎 𝟎 𝟎
𝟏 𝟏 𝟏 𝟏
𝐁{𝟐}(: , ∶, 𝟐) = [ ]
𝟎 𝟎 𝟎 𝟎
𝟎 𝟎 𝟎 𝟎
𝑐ℎ𝑜𝑖𝑐𝑒 𝑐ℎ𝑜𝑖𝑐𝑒
𝑝(𝑠𝜏+1 |𝑠𝜏 , 𝑈 = 𝑐ℎ𝑜𝑜𝑠𝑒 𝑙𝑒𝑓𝑡) =

𝟎 𝟎 𝟎 𝟎
𝟎 𝟎 𝟎 𝟎
𝐁{𝟐}(: , ∶, 𝟑) = [ ]
𝟏 𝟏 𝟏 𝟏
𝟎 𝟎 𝟎 𝟎
𝑐ℎ𝑜𝑖𝑐𝑒 𝑐ℎ𝑜𝑖𝑐𝑒
𝑝(𝑠𝜏+1 |𝑠𝜏 , 𝑈 = 𝑐ℎ𝑜𝑜𝑠𝑒 𝑟𝑖𝑔ℎ𝑡) =

𝟎 𝟎 𝟎 𝟎
𝟎 𝟎 𝟎 𝟎
𝐁{𝟐}(: , ∶, 𝟒) = [ ]
𝟎 𝟎 𝟎 𝟎
𝟏 𝟏 𝟏 𝟏

In other words, action 1, B{2}(: , ∶, 1), entails moving to the ‘start’ state from any other state, action
2, B{2}(: , ∶, 2), entails moving to the ‘get hint’ state from any other state, and so forth. The third
dimension labels these as action numbers 1, 2, 3, and 4.

Next, we need to specify preferences over each set of outcomes (𝐂), with one matrix per outcome
modality. Here, rows indicate observations (same order as in the corresponding A matrices) and
columns indicate time points in a trial from left to right. In other words:

𝐶 𝑚𝑜𝑑𝑎𝑙𝑖𝑡𝑦 =

𝐂{𝐨𝐮𝐭𝐜𝐨𝐦𝐞 𝐦𝐨𝐝𝐚𝐥𝐢𝐭𝐲}(𝐨𝐮𝐭𝐜𝐨𝐦𝐞, 𝐭𝐢𝐦𝐞 𝐩𝐨𝐢𝐧𝐭) = [𝐦𝐚𝐭𝐫𝐢𝐱]

In this case, the model has no direct preference for getting a hint (i.e., preferences for outcome
modality 1: 𝐶{1}) or for observing the choice of a particular action (i.e., preferences for outcome
modality 3: 𝐶{3}). So:

𝐶 ℎ𝑖𝑛𝑡 =

𝟎 𝟎 𝟎
𝐂{𝟏} = [𝟎 𝟎 𝟎]
𝟎 𝟎 𝟎

𝐶 𝑜𝑏𝑠𝑒𝑟𝑣𝑒𝑑 𝑎𝑐𝑡𝑖𝑜𝑛 =

𝟎 𝟎 𝟎
𝟎 𝟎 𝟎
𝐂{𝟑} = [ ]
𝟎 𝟎 𝟎
𝟎 𝟎 𝟎

64
Active Inference Tutorial

Here, columns from left to right indicate 𝜏 = 1, 2, 3 in the trial. The model does have preferences for
winning and losing (outcome modality 2), which are specified as follows:

𝐶 𝑤𝑖𝑛 =

𝟎 𝟎 𝟎
𝐂{𝟐} = [𝟎 −𝟏 −𝟏]
𝟎 𝟒 𝟐

This indicates that, at the second and third time points, a win (row 3) has a value of 4 and 2,
respectively. Remember, the value is less at the third time point (third column) because the
individual wins less money if they instead choose to take the hint at the second time point (i.e., and
thus continues to observe the ‘start’ outcome in row 1 for this outcome modality). The values of -1
in row 2 indicate a preference against observing a loss at time points 2 and 3.

Note that we only initially specify the C matrix values in this form for convenience. These
preference distributions are passed through a softmax (normalized exponential) function (𝜎) such
that each column (i.e., the preference distribution for each time 𝜏) in the C matrix encodes a proper
probability distribution of non-negative values that sums to 1, at which point a natural log is
applied. This means that the values are transformed into log-probabilities as follows (i.e., less
negative indicates more preferred):

𝑚𝑜𝑑𝑎𝑙𝑖𝑡𝑦 𝑚𝑜𝑑𝑎𝑙𝑖𝑡𝑦 𝑚𝑜𝑑𝑎𝑙𝑖𝑡𝑦

ln𝑝(𝑜𝜏 |𝐶𝜏 ) = ln(𝜎(𝐶𝜏 ))

For example, in the case of the preferences for 𝐶 𝑤𝑖𝑛 specified above:

ln𝑝(𝑜𝜏𝑤𝑖𝑛 |𝐶𝜏𝑤𝑖𝑛 ) =

−𝟏. 𝟏 −𝟒. 𝟎 −𝟐. 𝟏

ln(𝜎(𝐂{𝟐}) = [−𝟏. 𝟏 −𝟓. 𝟎 −𝟑. 𝟐]
−𝟏. 𝟏 −𝟎. 𝟎𝟐 −𝟎. 𝟐

To be clear, in the line immediately above, the softmax is applied to each column separately (i.e.,
corresponding to the preference distribution over outcomes at each time point).

Next, we need to specify allowable policies. There are three time points in a trial for this task, which
means a policy will consist of two actions. If we want to include ‘shallow’ policies, where the
model only looks one step ahead, we need to specify a set of vectors 𝑈 that index each action (as
already referred to above). Technically, this set of vectors is specified as a matrix including one row,
one column for each allowable action, and a third dimension specifying each state factor.
Thus, the structure is:

𝑈𝑓𝑎𝑐𝑡𝑜𝑟 =

𝑼(𝟏 , 𝐚𝐜𝐭𝐢𝐨𝐧 𝐧𝐮𝐦𝐛𝐞𝐫, 𝐬𝐭𝐚𝐭𝐞 𝐟𝐚𝐜𝐭𝐨𝐫) = [𝐯𝐞𝐜𝐭𝐨𝐫]

65
Active Inference Tutorial

In this case, we can include all actions:

𝑈 𝑐𝑜𝑛𝑡𝑒𝑥𝑡 =

𝑼(: , ∶ , 𝟏) = [𝟏 𝟏 𝟏 𝟏]

𝑈 𝑐ℎ𝑜𝑖𝑐𝑒 =

𝑼(: , ∶ , 𝟐) = [𝟏 𝟐 𝟑 𝟒]

Entries for 𝑈(: , ∶ , 2) allow all four possible transitions (actions) between choice states (factor 2) at
each time point. The entries for 𝑈(: , ∶ , 1) are all ones because there is only one possible ‘action’ –
that is, one transition matrix 𝐁𝜋,𝜏 – for state factor 1. There still needs to be four ones within 𝑈(: , ∶
, 1) to match the number of actions in 𝑈(: , ∶ , 2). In other words, each overall action option
corresponds to the combined entries in a given column for both state factors.6

If we instead want to include ‘deep’ policies, where the simulated participant plans ahead until the
end of the trial, this means that we need to specify one column for each allowable policy (with
each entry indicating an action number) in a matrix 𝐕, with one row per time point and a third
dimension specifying each state factor. Thus, the general structure is:

𝜋 𝑓𝑎𝑐𝑡𝑜𝑟 =

𝐕(𝐭𝐢𝐦𝐞 𝐩𝐨𝐢𝐧𝐭 , 𝐩𝐨𝐥𝐢𝐜𝐲, 𝐬𝐭𝐚𝐭𝐞 𝐟𝐚𝐜𝐭𝐨𝐫) = [𝐦𝐚𝐭𝐫𝐢𝐱]

In this case, we might reasonably include five policies:

𝜋 𝑐𝑜𝑛𝑡𝑒𝑥𝑡 =

𝟏 𝟏 𝟏 𝟏 𝟏
𝐕(: , ∶ , 𝟏) = [ ]
𝟏 𝟏 𝟏 𝟏 𝟏

𝜋 𝑐ℎ𝑜𝑖𝑐𝑒 =

𝟏 𝟐 𝟐 𝟑 𝟒
𝐕(: , ∶ , 𝟐) = [ ]
𝟏 𝟑 𝟒 𝟏 𝟏

6
Although not shown in detail here, this also affords the possibility of multidimensional policies in more complex
models. For example, if there were multiple possible actions (transition matrices) for two different state factors,
one might specify that action 2 for state factor 1 can be chosen together with action 2 for state factor 2, but that
action 2 for state factor 1 cannot be chosen together with action 3 for state factor 2 (simply by including a column
that has entries of 2 for both state factors but no column that has an entry of 2 for state factor 1 and an entry of 3
for state factor 2).

66
Active Inference Tutorial

As with 𝑈, all policies here do not change state factor 1 at either time point (hence, all entries are
ones7). For state factor 2, we have included policies in which the model chooses to remain in the
‘start state’ (i.e., choosing action 1 twice; column 1), chooses to take the hint (action 2) and then
select either of the slot machines (i.e., actions 3 or 4; columns 2-3), or decides to choose a slot
machine right away (columns 4-5; note, these policies subsequently return to state 1, since it is not
possible to win twice in one trial). We will use deep policies in the simulations below.

If so desired, one can also specify a fixed prior over policies 𝐸 to incorporate a bias or ‘habit’
to select some policies over others. This is simply a column vector with one entry per policy that
encodes the probability of that policy. Here we will not include such a bias, which means that 𝐸 will
simply be a flat distribution over our 5 allowable policies in 𝐕:

𝑝(𝜋) =

𝟏 𝟏 𝟏 𝟏 𝟏
𝑬=[ ]′
𝟓 𝟓 𝟓 𝟓 𝟓

Finally, there are several scalar (single-value) parameters one can set. One parameter is
beta (𝛽), which is the prior on the expected free energy precision term 𝛾 discussed above (which
encodes the precision estimate for the expected free energy over policies). A low 𝛽 value (around 1)
indicates high expected precision, whereas higher values (e.g., 3, 5, 10) indicate lower expected
precision. Higher 𝛽 values will increase randomness in policy selection and also make policy
selection more influenced by habits encoded in the 𝐸 vector (for an example of these dynamics, see
(Smith, Khalsa, & Paulus, 2021)). This follows from the fact that it implies less confidence that
model beliefs will generate preferred outcomes (Hesp et al., 2020). Another parameter is alpha (𝛼),
which is a standard ‘inverse temperature’ (or ‘action precision’) parameter that controls
randomness (e.g., motor stochasticity) in action selection under a chosen policy (higher values
indicate less randomness; typical range is between around 1 – 32, but very high values can be
chosen to remove choice stochasticity).

𝑃(𝐴𝑐𝑡𝑖𝑜𝑛|𝛼) = 𝜎(𝛼 × ln 𝑃(𝐴𝑐𝑡𝑖𝑜𝑛|𝜋)) (28)

Here, sigma (𝜎) indicates a softmax function that transforms the quantity on the right into a proper
probability distribution that sums to 1 (see Appendix 1 for more detail). Both 𝛽 and 𝛼 must be
positive numbers. Here we will make 𝛽 = 1 and 𝛼 = 32, specifying reasonable amounts of
indeterminacy in action selection.

3.2 Running and Plotting Simulations

We have now specified a generative model and are ready to run single-trial simulations. To do so in
MATLAB, we will assign each of our variables to a structure called mdp (for Markov decision
process). Concretely, this means assigning 𝐦𝐝𝐩. 𝐃 = 𝐃, 𝐦𝐝𝐩. 𝐕 = 𝐕, 𝐦𝐝𝐩. 𝐛𝐞𝐭𝐚 = 𝐛𝐞𝐭𝐚, and so

7
However, note that they need not all be ones in a more complex model with multidimensional policies, as also
described for U in the previous footnote.

67
Active Inference Tutorial

forth for all the scalars, vectors, and matrices constructed above. We can then run this structure
through the standard active inference estimation function 𝐬𝐩𝐦_𝐌𝐃𝐏_𝐕𝐁_𝐗. 𝐦 (available in the
DEM toolbox of the most recent versions of SPM academic software:
[Link] This just means entering:

𝐌𝐃𝐏 = 𝐬𝐩𝐦_𝐌𝐃𝐏_𝐕𝐁_𝐗(𝐦𝐝𝐩)

However, because SPM software is often updated, we include a specific version for this tutorial. So
here you should run:

𝐌𝐃𝐏 = 𝐬𝐩𝐦_𝐌𝐃𝐏_𝐕𝐁_𝐗_𝐭𝐮𝐭𝐨𝐫𝐢𝐚𝐥(𝐦𝐝𝐩)

This function will simulate behavior based on an POMDP structure (i.e., it runs the equations in
Table 2), and the output MDP (capital letters) structure will contain the simulation results. As we
have specified it here, it assumes the generative process and generative model are identical (see
section on learning below where we remove this assumption). It will thus generate outcomes based
on the generative process and simulate the subsequent inference and decision dynamics within the
generative model when observing those outcomes. Because the above-mentioned simulation script
is quite complex, we also direct readers interested in the details of how the belief updating scheme
is implemented to the 𝐒𝐢𝐦𝐩𝐥𝐢𝐟𝐢𝐞𝐝_𝐬𝐢𝐦𝐮𝐥𝐚𝐭𝐢𝐨𝐧_𝐬𝐜𝐫𝐢𝐩𝐭. 𝐦 script included in the supplementary
code, which is a stripped down (but heavily commented) version of the standard model inversion
scheme used in 𝐬𝐩𝐦_𝐌𝐃𝐏_𝐕𝐁_𝐗. 𝐦. For clarity, this additional tutorial script inverts the same
generative model of the explore-exploit task introduced above.

Single-trial behavior can be plotted with some default plotting routines. The primary single-trial
plotting routine available in SPM can be run in MATLAB by entering:

𝐬𝐩𝐦_𝐟𝐢𝐠𝐮𝐫𝐞(′𝐆𝐞𝐭𝐖𝐢𝐧′, ′𝐅𝐢𝐠𝐮𝐫𝐞 𝟏′); 𝐜𝐥𝐟; 𝐬𝐩𝐦_𝐌𝐃𝐏_𝐕𝐁_𝐭𝐫𝐢𝐚𝐥(𝐌𝐃𝐏); 𝐬𝐮𝐛𝐩𝐥𝐨𝐭(𝟑, 𝟐, 𝟑)

This plotting routine can also take additional optional inputs:

𝐬𝐩𝐦_𝐌𝐃𝐏_𝐕𝐁_𝐭𝐫𝐢𝐚𝐥(𝐌𝐃𝐏, 𝐆𝐟, 𝐆𝐠).

Gf: state factors to plot.

Gg: outcome modalities to plot.

For example, 𝐬𝐩𝐦_𝐌𝐃𝐏_𝐕𝐁_𝐭𝐫𝐢𝐚𝐥(𝐌𝐃𝐏, 𝟏: 𝟐, 𝟐: 𝟑) would plot the first two state factors and the
second and third outcome modalities.

At this point, the reader is encouraged to set the variable 𝑺𝒊𝒎 in the first section of the
accompanying tutorial code (i.e., 𝐒𝐭𝐞𝐩_𝐛𝐲_𝐒𝐭𝐞𝐩_𝐀𝐈_𝐆𝐮𝐢𝐝𝐞. 𝐦, line 51) to 𝑺𝒊𝒎 = 𝟏 and then click
‘Run’, which will run the model and this plotting script. Before running this script, remember to
make sure SPM12 is installed and that the ‘DEM’ folder within the SPM folder structure is added as
a path in MATLAB (…spm12\toolbox\DEM).

68
Active Inference Tutorial

Based on the current model specification, a representative plot of simulation results is shown in
Figure 8A. This and similar plots are generated from specific output fields in the MDP structure
(Table 3 describes each output field). The two panels in the top-left of Figure 8A show posteriors
over states at the end of the trial (i.e., the states the model believes it was in at each time point 𝜏
when at the last time point 𝑡). Here time goes from left to right, darker indicates higher probability,
and the cyan dots denote the true states. Here, the model believes it was in the ‘left-better context’
and that it chose to take the hint and then chose the left slot machine. The top-right shows the
action probabilities and true actions. Here the agent is highly confident that taking the hint and
choosing the left machine were the best choices (and cyan dots indicate that these were also the
actual actions taken). The left-middle panel just shows the different possible two-step action-
sequences specified in the model (from left to right). Note that lighter shades in this panel just
indicate higher action numbers (e.g., action 1 is black, action 2 is dark gray, etc.). The right-middle
panel shows the evolution of the posterior distribution over policies over time (from left to right).
Here, it can be seen that at the second time point the model became highly confident in policy 2 (i.e.,
the ‘take the hint and then choose the left slot machine’ policy). The three panels in the bottom-left
show the outcomes (cyan dots) and preference distributions. The first ‘hint’ modality shows that
the model received the hint at time point 2. The plot is gray because there is no preference for one
observation over others. This is also the case for the third ‘observed action’ plot, which simply
confirms what the model chose. The second ‘win/lose’ modality shows that a win was observed at
the third time point. The preference distribution here indicates the strong preference for the win at
time points 2 and 3 (darker value), and a preference not to lose (lighter value). The ‘null’ (starting)
outcome in the top row is an intermediate gray at time point 2 (no preference for or against this
outcome); the distribution becomes darker gray at the third time point because the value of the win
at time point 3 was relatively less (i.e., $2 vs. $4) and so the overall distribution over outcomes at
the third time point is less precise.

As mentioned earlier, however, optimal information-seeking (in the sense of maximizing preferred
outcomes in the long-run) depends on having the right balance of reward value and information
value. To illustrate this, Figure 8B shows simulations in which the magnitude of the preference
distribution for a win has been multiplied by 2: C{2}(3, ∶) = [0 8 4]. As can be seen there, the model
instead decided to make a guess right away about which machine will win (in this case, choosing
right) and unfortunately observed a loss (bottom-left, middle sub-panel). As can be seen in the
upper right, its confidence in the left vs. right action is equal (equally gray over each). As can be
seen in the upper left, the model’s posterior over states shows high confidence that it was in fact in
the ‘left-better’ context at every time point, because (retrospectively) this was most likely the case if
it lost after choosing the machine on the right.

69
Active Inference Tutorial

Figure 8. Example simulation plots that can be generated with the code included in this tutorial. A
detailed walk-through is provided in the main text. Panel A: Example simulation of a risk-averse
agent performing the explore-exploit task. The agent takes the hint, then chooses the left slot
machine, and observes a win. Panel B: Example simulation of a risk-seeking agent performing the
explore-exploit task. The agent foregoes the hint and immediately chooses the right slot machine,
observes a loss, and then returns to the start state. Panel C: Example neuronal simulations based
on the risk-averse agent in Panel A. This illustrates the neuronal firing rates, local field potentials,
and dopamine responses predicted by the neural process theory. For all panels, note that darker
colors indicate higher probability values for beliefs about states, actions, and policies over time. For
outcomes, darker values indicate stronger prior preferences. The ‘allowable policies’ plots in the
first two panels simply display the action sequences corresponding to each policy (darker indicates
lower numbers, where each number denotes an available action). The dopamine response plots
(lower-right plots in each panel) correspond to updates in the expected precision of the 𝐸𝐹𝐸
distribution over policies (𝛾); cyan lines indicate 𝛾 values while black spikes correspond to their
rate of change. In the upper- and lower-left plots (‘contexts’ and ‘firing rates’) of panel C, each
column (moving left to right along the x-axis) corresponds to beliefs about context states at the
time when an observation was received (t), while rows from top to bottom on the y-axis
correspond to the time point for which beliefs are updated (tau; τ). For example, the top-right
quadrant corresponds to beliefs at time t = 3 about time τ = 1 (note that, unfortunately, this
standard SPM plotting routine inappropriately labels each row with ts instead of τs). Firing rates
(upper-right) correspond to the magnitude of posteriors over each state (in this case, the states in
the ‘context’ state factor), while local field potentials (middle-right) correspond to their rate of
change (in both cases, there is one line plotted for each row in the plots in the upper- and lower-
left). See main text for interpretation of time frequency response plots and their motivation. These
simulations can be reproduced by running the 𝑺𝒊𝒎 = 𝟏 option in the supplementary

70
Active Inference Tutorial

𝐒𝐭𝐞𝐩_𝐛𝐲_𝐒𝐭𝐞𝐩_𝐀𝐈_𝐆𝐮𝐢𝐝𝐞. 𝐦 code (although note that, because outcomes are sampled from
probability distributions, results will not be identical each time).

Finally, the bottom-right plots in these panels display predictions about dopamine responses in the
neural process theory, which we have not yet discussed. These responses are based on changes in
confidence in expected free energy estimates after receiving new observations (i.e., the updates to
the expected free energy precision parameter 𝛾; explained in the bottom row of Table 2). In this
case, the large ‘dopamine spike’ shown at the second time step is because 𝐸𝐹𝐸 at that time step
favored policy 2 (i.e., taking the hint and then choosing the left machine) and 𝑉𝐹𝐸 at that time step
(based on observations) provided support for policies 2 and 3 (i.e., after observations at the second
time step, only the two policies that included taking the hint remained plausible). Because the
policy favored by 𝐸𝐹𝐸 was supported, the precision estimate for 𝐸𝐹𝐸 increased (corresponding to
the positive dopamine spike).

As a numerical example to help offer an intuition for how these updates operate, we can plug the
𝑉𝐹𝐸 and 𝐸𝐹𝐸 values at time point 2 in this simulation into the policy distribution and precision
update equations shown in Table 2. In these equations, 𝛾 is the expected free energy precision
term, 𝛽0 is the initial prior for this precision at the start of a trial (𝛾 = 1/𝛽), and 𝛽 is the posterior
value that is continuously updated over time by a term we label 𝛽𝑢𝑝𝑑𝑎𝑡𝑒 . This term technically
reflects the gradient of free energy with respect to 𝛾 (∇𝛾 𝐹) and is informed by a value scoring the
level of (dis)agreement between expected free energy and observed (variational) free energy after
making a new observation – which can be thought of as a type of prediction error (𝐺𝑒𝑟𝑟𝑜𝑟 ; with
proposed associations with emotion; see (Hesp et al., 2020)). For the sake of illustration, we can set
𝛾, 𝛽0 , and 𝛽 equal to one and specify the distributions over policies as follows8:

𝐸 = [1 1 1 1 1]T

𝐺 ≈ [12.505 9.51 12.5034 12.505 12.505]T

𝐹 ≈ [17.0207 1.7321 1.7321 17.0387 17.0387]T

As can be seen here, the agent has no habit-like prior expectations over policies (i.e., the 𝐸
distribution is flat), and the expected free energy over policies (𝐺) favors policy 2 (i.e., entry 2 has
the lowest value). The variational free energy after a new observation (𝐹) provides precise
evidence for policies 2 and 3 (values much closer to 0, indicating that the new observation is
inconsistent with policies 1, 4, and 5). Given this setup, one round of iterative updating would be:

𝜋0 ← 𝜎(ln 𝐸 − 𝛾𝐺) = [0.0417 0.8332 0.0418 0.0417 0.0417]T

𝜋 ← 𝜎(ln 𝐸 − 𝐹 − 𝛾𝐺) = [0 . 9523 . 0477 0 0]T

𝐺𝑒𝑟𝑟𝑜𝑟 ← (𝜋 − 𝜋0 ) ⋅ (−𝐺) = .3567

8
Note that the spm_MDP_VB_X.m script (and the tutorial version here) works with negative free energies, and so
these F and G values are made negative in the MDP output structure in MATLAB.

71
Active Inference Tutorial

𝛽𝑢𝑝𝑑𝑎𝑡𝑒 ← 𝛽 − 𝛽0 + 𝐺𝑒𝑟𝑟𝑜𝑟 = .3567

𝛽 ← 𝛽 − 𝛽𝑢𝑝𝑑𝑎𝑡𝑒 ⁄𝜓 = 1 − .3567/2 = .8216

1
𝛾← = 1.2171
𝛽

Here we have included a step size parameter of 𝜓 = 2, which reduces the magnitude of each
update and promotes stable convergence. Note that, while we have here shown an example of a
single round of updating, there will be many rounds of updating to convergence for each new
observation. Notice that the increase in the probability of policy 2 and 3 between the prior and
posterior over policies is driven by 𝐹, which scores the evidence afforded each policy given current
observations. Policy 2 and 3 better minimize 𝐹 and are therefore more plausible. Taking the dot
product between the vector encoding the difference between the prior and posterior over policies
(𝜋 − 𝜋0) and the −𝐺 vector is equivalent to scaling each element of the difference vector by the
associated 𝐺 value and then summing the results, which in this case creates a positive update. This
is because the difference vector is pointing in roughly the same direction as the −𝐺 vector. This is
apparent in that 𝐺 initially indicated the highest probability for policy 2, and 𝐹 also provided
evidence for policy 2. As a result, the prediction error (𝐺𝑒𝑟𝑟𝑜𝑟 ) is positive and the updated 𝛾 value
increases the impact of 𝐺 on the posterior over policies (i.e., because the agent is now more
confident in its beliefs about 𝐺). In contrast, if the policies favored by 𝐺 were not supported by 𝐹 (as
in Figure 8B), the 𝐺𝑒𝑟𝑟𝑜𝑟 term would be negative and the updated value of 𝛾 would decrease the
impact of 𝐺 on the posterior over policies, as the agent has lost confidence in its beliefs about 𝐺. For
a derivation of these update equations, see Appendix in (Sales et al., 2019).

Note that the cyan line in the dopamine plot corresponds to the stable expected free energy
precision value (𝛾; with a hypothesized link to tonic dopamine levels), as opposed to the rate of
change in this precision (in black; with a hypothesized link to phasic dopamine responses). To help
the reader gain a better intuition for the dynamics of these updates, we have provided
supplementary code (𝐄𝐅𝐄_𝐏𝐫𝐞𝐜𝐢𝐬𝐢𝐨𝐧_𝐔𝐩𝐝𝐚𝐭𝐢𝐧𝐠. 𝐦), which allows the reader to specify the number
of policies, the values for the vectors 𝐸, 𝐹, and 𝐺, and the value of 𝛽0 , and then simulate these
updates. Figure 9 also illustrates a helpful geometric interpretation of the factors that determine
the direction of 𝛽 updates. Namely, when the difference vector (𝜋 − 𝜋0 ) and the −𝐺 vector point in
a similar direction (i.e., an angle of less than 90° apart), the dot product of the two will result in an
increase in 𝛾. The fact that these vectors point in the same direction is a way to visualize how new
observations (through 𝐹) provide evidence supporting the reliability of 𝐺, and therefore increase
its precision weighting. In contrast, when these vectors point in different directions (i.e., the angle
separating them is greater than 90°), this suggests that 𝐺 is less reliable; its precision (𝛾) is
therefore reduced and it contributes less to the posterior distribution over policies (𝜋).

72
Active Inference Tutorial

Figure 9. Illustration of a geometric interpretation of the factors contributing to updates in

expected free energy precision estimates (𝛾). In these examples we include two policies and specify
the values of priors over policies (𝐸), expected free energy over policies (𝐺), and the variational
free energy over policies following a new observation (𝐹). Based on these, a prior and posterior
over policies (𝜋0 and 𝜋) are computed (calculations shown in each of the top panels). Updates are
then computed for 𝛽, where 𝛾 = 1/𝛽, constrained by a fixed prior value 𝛽0 and a step size
parameter (𝜓) that promotes convergence to a stable posterior across iterations. The direction and
magnitude of the resulting update in 𝛾 is driven by the dot product between the difference vector
for prior and posterior policy beliefs (𝜋 − 𝜋0 ) and the −𝐺 vector. This dot product can be thought of
as a prediction error (𝐺𝑒𝑟𝑟𝑜𝑟 ) reflecting the level of (dis)agreement between expected free energy
and the variational free energy of a subsequent observation. The plots in the top panels show cases
where 𝛾 is positively updated (left) and negatively updated (right). In the case on the left, the two
vectors (𝜋 − 𝜋0 and −𝐺) point in a similar direction (i.e., less than 90° apart), which represents a
way to visualize how new observations (through 𝐹) provide evidence for the reliability of 𝐺

73
Active Inference Tutorial

(leading to an increase in its precision weighting 𝛾). In the case on the right, the vectors are greater
than 90° apart, providing evidence against the reliability of 𝐺 (leading to a decrease in its precision
weighting 𝛾). Note that, for reasons of clarity, the endpoints of the vectors shown here are not the
actual values of 𝜋 − 𝜋0 and −𝐺; they instead correspond to scaled values of these vectors, which
makes them similar in length and more clearly illustrates the angle separating them. The middle
panels show 16 iterations of 𝛾 updating, as is done per time point (i.e., observation) in a trial in the
supplementary code (and in the standard SPM routines) until a stable posterior estimate is
reached. This is similar to prediction error minimization dynamics for the state and outcome
predictions errors described earlier. The bottom panels show the rate of change in 𝛾, which bears
some similarity to prediction error responses. The updates shown in this figure have been
associated with dopamine in the neural process theory accompanying active inference. These
simulations can be reproduced using the EFE_Precision_Updating.m code provided in the
supplementary code.

Table 3. Output fields for spm_MDP_VB_X_tutorial.m simulation script

MDP Model Element Structure Description

Field

MDP.F Negative variational Rows = policies. Negative variational free

free energy of each energy of each policy at
policy over time. Columns = time points. each time point in the trial.
For example, if there are 2
policies and 6 time points
there will be a 2x6 matrix
containing the negative
variational free energy of
each policy at each point in
the trial.

MDP.G Negative expected Rows = policies. Negative expected free

free energy of each energy of each policy at
policy over time. Columns = time points. each time point in the trial.
For example, if there are 2
policies and 6 time points
there will be a 2x6 matrix
containing the negative
expected free energy of
each policy at each point in
the trial.

74
Active Inference Tutorial

MDP.H Total negative Columns = time points. Total negative variational

variational free free energy averaged
energy over time. across states and policies at
each time point. For
example, if there are 8 time
points there will be a 1x8
row vector containing the
total negative free energy
at each time point.

[Link] [Link] is the Columns = one per outcome KL divergence between the
[Link] negative free energy modality or hidden state factor parameters of the
[Link] of parameter ‘a’ (if (i.e., depending on the specific matrix/vector that is being
… learning 𝐀 matrix). parameters being learned). If learned at the beginning of
There are also the agent is learning parameters each trial and at the end of
analogous fields if of a single vector (e.g., 𝐸), this each trial. Each column in
learning other will be a single column. the vector may represent
matrices/vectors an outcome modality (i.e.,
(e.g., [Link] for in the case of the 𝐀 matrix),
learning the a hidden state factor (i.e., in
parameters of the 𝐷 the case of the 𝐁 matrix and
vector, etc.). 𝐷 vector), or any other
vector (e.g., the 𝐸 vector).

MDP.O Outcome vectors Rows = outcome modalities. Vectors (one per cell)
specifying the outcomes for
Columns = time points. each modality at each time
point. Observed outcomes
are encoded as 1s, with 0s
otherwise.

MDP.P Probability of Rows = one per controllable The probability of emitting

emitting an action state factor. each particular action,
expressed as a softmax
Columns = actions. function of a vector
containing the probability
Third dimension = time point.
of each action summed
over each policy. For
example, assume that there
are two possible actions,
with a posterior over
policies of [.4 .4 .2], with
policy 1 and 2 leading to
action 1, and policy 3

75
Active Inference Tutorial

leading to action 2. The

probability of action 1 and
2 is therefore [.8 .2]. This
vector is then passed
through another softmax
function controlled by the
inverse temperature
parameter 𝛼, which by
default is extremely large
(𝛼 = 512). Actions are then
sampled from the resulting
distribution, where higher
𝛼 values promote more
deterministic action
selection (i.e., by choosing
the action with the highest
probability).

MDP.Q Posteriors over 1 cell per state factor. Posterior probability of

states under each each state conditioned on
policy at the end of Rows = states. each policy at the end of the
the trial. trial after successive
Columns = time points.
rounds of updating at each
time point.
Third dimension = policy
number.

MDP.R Posteriors over Rows = policies. Posterior over policies at

policies. each time point.
Columns = time points.

MDP.X Overall posteriors 1 cell per state factor. This means taking a
over states at the weighted average of the
end of the trial. Rows = states. posteriors over states
These are Bayesian under each policy, where
Columns = time points.
model averages of the weighting is
the posteriors over determined by the
states under each posterior probability of
policy. each policy.

[Link] Neuronal encoding 1 cell per policy dimension. Simulated neuronal

of policies encoding of the posterior
Rows = policies. probability of each policy at
each iteration of message
Columns = iterations of message

76
Active Inference Tutorial

passing (16 per time point). For passing.

example, 16 iterations, and 8
time points gives a vector with
128 columns).

[Link] Neuronal encoding 1 cell per state factor. Bayesian model average of
of state prediction state prediction errors at
errors. Rows = iterations of message each iteration of message
passing (16 per time point). passing (weighted by the
posterior probability of the
Columns = states.
associated policies).
Third Dimension: time point the
belief is about (𝜏).

Fourth Dimension: time point

the belief is at (𝑡).

[Link] Neuronal encoding 1 cell per state factor. Bayesian model average of
of hidden states. normalized firing rates,
Rows = iterations of message which reflect posteriors
passing (16 per time point). over states at each iteration
of message passing
Columns = states.
(weighted by the posterior
probability of the
Third Dimension: time point the
associated policies).
belief is about (𝜏).

Fourth Dimension: time point

the belief is at (𝑡).

[Link] Neuronal encoding Rows = number of iterative This reflects the value of
of tonic dopamine, updates (16 per time point). For the expected precision of
reflecting the example, if there were two time the expected free energy
current value of 𝛾. points in a trial this would be 1 over policies (𝛾) at each
column with 32 rows. iteration of updating.

[Link] Neuronal encoding Rows = number of iterative This variable reflects the
of phasic dopamine updates (16 per time point). For rate of change in the
responses, example, if there were two time expected precision of
reflecting the rate of points in a trial this would be 1 expected free energy over
change in 𝛾. column with 32 rows. policies (𝛾) at each
iteration of updating.

77
Active Inference Tutorial

[Link] Simulated reaction Columns = time points. Computation time (i.e., time
times. to convergence) for each
round of message passing
and action selection.

Now that we have gone through an example of these dynamics, an important question concerns the
settings in which they may be useful. Here, it is important to highlight that 𝐸, 𝐹, and 𝛽/𝛾 can be
viewed as optional elements (e.g., they are not incorporated in the policy selection model within the
upper right portion of Figure 5, or in other examples of active inference (Da Costa, Parr, et al.,
2020)). For example, incorporating 𝐸 may not be useful unless modelling a task in which you
suspect that participants enter a study with a particular choice bias or that habitual choice behavior
could be learned over time. Incorporating 𝐹 and 𝛽/𝛾 updating is only useful in the context of deep
policies. As mentioned elsewhere, 𝛽/𝛾 can optimize the relative influence of goals and habits (𝐸 and
𝐺). Among others, one benefit of incorporating 𝐹 emerges when there are a large number of deep
policies to choose from. This is because it allows observations to render some policies highly
implausible early in a trial, which has the effect of narrowing the search space for the optimal
policy. This typically works in conjunction with an “Occam’s window” parameter that removes
policies from the search space if their probability becomes too low relative to the most probable
policy (this parameter is specified as 𝐦𝐝𝐩. 𝐳𝐞𝐭𝐚 and explained in more detail within the
accompanying tutorial code Step_by_Step_Hierarchical_Model.m.

4. Modelling Learning

In this section we will discuss how learning is implemented in active inference and how this can be
used to model multi-trial behavioral data. As a concrete example, we will return to the explore-
exploit task model and allow the agent to learn prior beliefs (i.e., within the vector 𝐷) about how
often the left slot machine and right slot machine tend to pay out. More generally, we will discuss
how any set of model parameter values – such as those encoding distributions within the likelihood
(𝐀), transition beliefs (𝐁𝜋,𝜏 ), or priors over policies (𝐸) – can be learned over repeated trials. This is
based on updating prior beliefs over these parameters within a class of distributions called
Dirichlet distributions. We first provide a technical introduction to the general mathematical
foundations of Dirichlet distributions. Then we discuss learning in more intuitive terms, provide
numerical examples, and demonstrate how to run simulations in practice.

4.1 Technical Introduction to Dirichlet Priors (optional)

In this subsection we provide a technical introduction to the Dirichlet distribution used to

implement learning in active inference. After completing this section, the reader should have an
understanding of how the parameters in a Dirichlet distribution can: 1) act as priors on the
categorical distributions used in the POMDP models covered above, and 2) be updated based on
posterior beliefs at the end of a trial. For readers less interested in these technical details, this
section can be skipped. As mentioned above, we will provide a more intuitive conceptual

78
Active Inference Tutorial

introduction in the next subsection. We encourage all readers to consider the formal details below,
but a complete understanding of this subsection will not be required to follow subsequent sections.

Learning in active inference is formulated in terms of a Dirichlet-categorical model. Specifically,

Bayesian inference is performed using a categorical distribution (which was discussed earlier) as
the likelihood, and a Dirichlet distribution as the prior. The Dirichlet distribution is a distribution
defined over a vector of values that sit on the interval [0, 1], and sum to 1. That is, the values of the
vector have the same properties as a probability distribution. As such, the Dirichlet distribution is
often described as a distribution over a distribution. In this case, it can be used to encode beliefs
about model parameters (e.g., confidence in parameters in the likelihood or transition matrices of a
POMDP).

The Dirichlet distribution is used as the prior over the parameters of the categorical distribution
because it is the conjugate prior for the categorical distribution. This means that if we multiply a
categorical distribution by a Dirichlet distribution, and then normalize to obtain the posterior
distribution over the parameters of the categorical distribution, we end up with another Dirichlet
distribution – allowing it to be used as a prior in the next round of inference. This allows active
inference agents to sequentially update their beliefs about model parameters as they receive new
observations. The Dirichlet distribution, denoted 𝐷𝑖𝑟(𝜃|𝛼), is defined as follows:

𝐾 (29)
Γ(∑𝐾
𝑘=1 𝛼𝑘 ) 𝛼 −1
𝑝(𝜽|𝜶) = 𝐷𝑖𝑟(𝜽|𝜶) = 𝐾 ∏ 𝜃𝑘 𝑘
∏𝑘=1 Γ( 𝛼𝑘 )
𝑘=1

Γ(∑𝐾 𝛼 )
Where ∏𝐾 𝑘=1 𝑘
Γ(𝛼 )
is a normalization constant that ensures the distribution sums to 1, and Γ denotes
𝑘=1 𝑘

the gamma function (for a brief introduction to the gamma function, see Appendix 1). The variable
𝜽 = (𝜃1 , … , 𝜃𝐾 ) is a vector of length 𝐾 containing the parameters of a categorical distribution, and
𝜶 = (𝛼1 , … , 𝛼𝐾 ) is the set of concentration parameters of the Dirichlet distribution, which satisfy
the condition that 𝛼𝑘 > 0. The gamma function is used in the normalization constant to account for
the combinatorics of drawing a random variable from a categorical distribution. That is, it counts
the number of ways in which we can place the variable 𝛼 in one of 𝐾 mutually exclusive states.
Similarly, the categorical distribution is defined as:

𝐾 (30)
1 𝑥
𝑝(𝒙|𝜽) = 𝐶𝑎𝑡(𝒙|𝜽) = ∏ 𝜃𝑘 𝑘
𝑥1 ! 𝑥2 ! … 𝑥𝐾 !
𝑘=1

Where 𝒙 = (𝑥1 , … , 𝑥𝐾 ) is a categorical variable that occupies one of 𝐾 mutually exclusive states
(e.g., 𝒙 = [0 1 0]T ). Here 𝜽 = (𝜃1 , … , 𝜃𝐾 ) are the parameters of the distribution and satisfy the
1
conditions 𝜃𝑘 ≥ 0, and ∑𝑘 𝜃𝑘 = 1. The term 𝑥 is the normalization constant.
1 !𝑥2 !…𝑥𝐾 !

If we multiply the Dirichlet and categorical distributions to arrive at the posterior distribution over
the parameters 𝜽 = (𝜃1 , … , 𝜃𝐾 ) of the categorical distribution, we obtain the following (ignoring the
normalization constant for the sake of brevity):

79
Active Inference Tutorial

𝐾 (31)
𝛼 +𝑥𝑘 −1
𝑝(𝜽|𝒙, 𝜶) = 𝐷𝑖𝑟(𝜽|𝒙, 𝜶 + 𝒙) ∝ ∏ 𝜃𝑘 𝑘
𝑘=1

Notice that this has exactly the same form as the prior defined above, except that we have added a
‘count’ (i.e., 𝑥𝑘 ) of 1 to the concentration parameters corresponding to the observed variable, while
a ‘count’ of 0 is added to those corresponding to the non-observed variables. It is the concentration
parameters of the Dirichlet distributions in the POMDP structure that are updated during learning.
The exact way they are updated depends on the model element in question (e.g., 𝐀 or 𝐁 matrix, or 𝐷
vector) which we discuss in more intuitive terms below. For a more detailed introduction to the
Dirichlet-categorical model, see (Tu, 2014).

4.2 Non-Technical Continuation on Dirichlet Priors

In this subsection we will introduce learning in more concrete and intuitive terms. This will build
on what was presented in the previous technical section, but it does not require an understanding
of the details presented there. At the end of this subsection, readers should have a practical
understanding of what changes in a model during learning and what causes these changes to occur.
Although the form of the Dirichlet distribution shown in the previous section can seem complex, the
resulting learning process turns out to be quite intuitive. Essentially, it just involves adding counts
to a vector or matrix based on posterior beliefs, where larger numbers of counts indicate higher
confidence. To illustrate this, we will first consider a Dirichlet distribution for initial state priors
over two possible states. As is standard notation in the active inference literature, we represent
Dirichlet distributions with the lowercase letters associated with each vector or matrix. For
example, we will denote the Dirichlet (𝐷𝑖𝑟) distribution for the initial state prior 𝐷 as 𝑑. Expressed
formally:

𝑝(𝐷) = 𝐷𝑖𝑟(𝑑) (32)

𝑑 = 𝑝(𝑠𝜏=1 ) = [𝑑1 𝑑2 ]T (33)

Here, the concentration parameters for 𝐷 – denoted by lowercase 𝑑 = 𝑑1 , 𝑑2 – are the individual
parameters that will change during learning. In other words, the process of adding counts
mentioned above will apply to the values of these variables. This is based on the following learning
equation:

𝑑𝑡𝑟𝑖𝑎𝑙+1 = 𝜔 × 𝑑𝑡𝑟𝑖𝑎𝑙 + 𝜂 × 𝑠𝜏=1 (34)

The eta (𝜂) parameter is a learning rate (scalar from 0-1), which controls how much the values in
𝑑 change after each trial. The omega (𝜔) parameter is a forgetting rate (scalar from 0-1), which
influences how quickly learning in recent trial can ‘overwrite’ the changes in 𝑑 that occurred in
earlier trials. We will return to these below (for now we will assume they are both equal to 1).

To get an intuition for how this ‘learning by counting’ process works, consider a case where you
start out with an initial state prior of 𝑑 = [0.5 0.5]T on the first trial, and your posterior belief at the

80
Active Inference Tutorial

end of that trial is that you were in state 1 (with probability = 1). In this case, your prior on the
second trial would become 𝑑 = [1.5 0.5]T . In other words, a count of 1 was added to the first entry
(i.e., the entry for state 1). If this happened 3 more times, then it would become 𝑑 = [4.5 0.5]T. In
cases of uncertainty, you instead add proportions of counts. For example, if you start out with an
initial state prior of 𝑑 = [1 1]T on the first trial, and your posterior belief at the end of the trial was
𝑠 = [0.7 0.3]T, then your prior would be updated on the second trial to be 𝑑 = [1.7 1.3]T. During
within-trial inference, these distributions are put through a softmax function so that they retain
their same shape but again add up to 1. However, larger numbers indicate greater confidence in the
shape of the distribution. This can be seen by comparing 𝑑 = [1 1]T to 𝑑 = [50 50]T . While both
distributions have the same (in this case flat) shape, it would take many more (new) observations to
meaningfully change the shape of the second distribution compared to the first. For example, after
one further trial with a precise posterior over state 1, the resulting shape of the distribution 𝑑 =
[2 1]T has changed quite a bit more than 𝑑 = [51 50]T. This is an important aspect of active
learning because it means that the initial (prior) counts determine how ‘open’ an agent is to new
experience. Typically, in a novel environment or task, the initial counts are set to very low values
(e.g., .25) – so that experience has a substantial effect on an agent’s prior beliefs. This makes
inference and planning more context-sensitive, as opposed to an agent with high initial counts who
is ‘stuck in its ways’ and would require much more evidence to ‘change its mind’.

As mentioned above, the agent also has a learning rate 𝜂. This controls how quickly it gets ‘stuck in
its ways’ during learning (this also influences how quickly the agent ceases to select information-
seeking policies; see below for more details). For example, if 𝜂 = 0.5, then an update from 𝑑 =
[1 1]T after inferring state 1 would not lead to 𝑑 = [2 1]T as shown above. Instead, it would be 𝑑 =
1 × [1 1]T + 0.5 × [1 0]T = [1.5 1]T . Thus, counts (and hence confidence) will increase more slowly
after each trial.

As also mentioned above, learning can be further modulated (multiplied) by a forgetting rate (𝜔).
This parameter controls how strongly recent experience is able to ‘overwrite’ what one has learned
in the more distant past. A value of 𝜔 = 1 indicates no forgetting (i.e., recent experience is unable to
overwrite what has been learned previously), while values less than 1 allow increasing levels of
forgetting (essentially, with each new observation the agent becomes less confident in what it has
previously learned). This is important because, as counts increase during learning, an agent’s
beliefs can become rigid and resistant to change, which is suboptimal in changing environments.
Higher counts also reduce information-seeking, because the agent is highly confident in its beliefs
(described in more detail below), which further hinders the opportunity to learn that (and how) the
environment has changed. As such, if an agent believes that the environment is volatile (e.g., that
the probabilities of rewards under each policy can change over time), then it is appropriate to adopt
a low value for 𝜔 (i.e., a high forgetting rate). In other words, a low value for 𝜔 can be understood as
encoding an agent’s prior belief that the contingencies in the world are unstable. It is worth noting
that inferences about changes in the environment (and about the volatility of the environment)
could also be implemented in a more principled manner in a hierarchical model (one example of a
model with dynamically updated beliefs about environmental volatility is the Hierarchical Gaussian

81
Active Inference Tutorial

Filter; (Mathys et al., 2014)). However, including the simpler forgetting rate parameter described
here could be sufficient for modelling task behavior in many cases.

To get an intuition for how this works, glance back at the equation for learning at the end of the
previous section (equation 34) and then consider a case where 𝑑 = [50 50]T and 𝜂 = 1. If 𝜔 = 1,
and the agent infers that it is in state 2, then the update will be 𝑑 = 1 × [50 50]T + 1 × [0 1]T =
[50 51]T. In contrast, if 𝜔 = 0.1, then the update will be 𝑑 = 0.1 × [50 50]T + 1 × [0 1]T = [5 6]T . In
this latter case, the agent therefore becomes much less confident in its prior beliefs, and the shape
of the posterior (Dirichlet) distribution is changed to a greater degree at the end of that trial.

As a slightly more complex example of learning, the updates for an example 𝐀 matrix become:

𝑝(𝐀) = 𝐷𝑖𝑟(𝒂) (35)

𝑎1 𝑎2 (36)
𝒂 = 𝑝(𝑜𝜏 | 𝑠𝜏 ) = [𝑎3 𝑎4 ]
𝑎5 𝑎6

𝒂𝑡𝑟𝑖𝑎𝑙+1 = 𝜔 × 𝒂𝑡𝑟𝑖𝑎𝑙 + 𝜂 × ∑ 𝑜𝜏 ⊗ 𝑠𝜏 (37)

𝜏

Here, the concentration parameters for 𝐀 – denoted by lowercase 𝒂 = 𝑎1 . . , 𝑎6 – are the individual
parameters in the matrix to be updated. The ⊗ symbol indicates the Kronecker (i.e., outer) product.
This again just involves accumulating (proportions of) counts, modulated by a learning rate and a
forgetting rate. But in this case, what is being counted are coincidences between states and
observations. For example, assume you have a posterior over states of 𝑠 = [1 0]T and you made the
observation associated with row 1, 𝑜 = [1 0 0]T . Because you believed you were in state 1 when
you observed outcome 1, this indicates that their association in 𝒂 should increase. That is, a count
should be added to 𝑎1 (i.e., the intersection of state 1 and outcome 1) before the subsequent trial. If
you instead have a posterior over states of 𝑠 = [0.7 0.3]T and you made the observation associated
with row 2, 𝑜 = [0 1 0]T , this indicates that their association in 𝒂 should increase proportionally.
That is, updates of 𝑎3 + 0.7 and 𝑎4 + 0.3 should occur before the next trial. Analogous update rules
apply to the other model parameters (𝐁, 𝐂, 𝐸). This general type of ‘coincidence detection’ learning
is analogous to Hebbian synaptic plasticity, where neurons with coincident firing rates increase
their synaptic connection strengths (T. H. Brown et al., 2010). As discussed further below, the
neural process theory associated with active inference proposes that each concentration parameter
can therefore be associated with the strength of a synaptic connection.

Another important change when learning is incorporated is that the expected free energy gains an
extra term, depending on which parameter is being learned. This is because learning is also based
on minimizing expected free energy. For example, if learning 𝐀, the equation for expected free
energy becomes:

𝐺𝜋 = 𝐷𝐾𝐿 [𝑞(𝑜𝜏 |𝜋)||𝑝(𝑜𝜏 )] + E𝑞(𝑠𝜏|𝜋) [H[𝑝(𝑜𝜏 |𝑠𝜏 )]] − E𝑝(𝑜𝜏|𝑠𝜏)𝑞(𝑠𝜏|𝜋) [𝐷𝐾𝐿 [𝑞(𝐀|𝑜𝜏 , 𝑠𝜏 )||𝑞(𝐀)]] (38)

82
Active Inference Tutorial

≈ ∑(𝐀𝑠𝜋,𝜏 ⋅ (ln 𝐀𝑠𝜋,𝜏 − ln 𝐂𝜏 ) − 𝑑𝑖𝑎𝑔(𝐀T ln 𝐀) ⋅ 𝑠𝜋,𝜏 − 𝐀𝑠𝜋,𝜏 ⋅ 𝐖𝑠𝜋,𝜏 ) (39)

𝜏

1 ⨀(−1) (40)
𝐖 ∶= (𝒂⨀(−1) − 𝒂𝑠𝑢𝑚𝑠 )
2

Note that the ∶= symbol just means that two things are defined to be equivalent. The ⨀ symbol
indicates the element-wise power (i.e., separately raising each element in a matrix to the power of
some number). The term 𝒂𝑠𝑢𝑚𝑠 is a matrix of the same size as 𝒂 where each entry within a column
. 25 1
corresponds to the sum of the values of the associated column in 𝒂. For example, if 𝒂 = [ ],
. 75 3
. 25 + .75 1 + 3 1 4
then 𝒂𝑠𝑢𝑚𝑠 = [ ]=[ ].
. 25 + .75 1 + 3 1 4

Although this updated equation for 𝐸𝐹𝐸 may appear complex, it simply adds one additional term –
often called the ‘novelty’ term (i.e., E𝑝(𝑜𝜏|𝑠𝜏)𝑞(𝑠𝜏|𝜋) [𝐷𝐾𝐿 [𝑞(𝐀|𝑜𝜏 , 𝑠𝜏 )||𝑞(𝐀)]] in equation 38, or 𝐀𝑠𝜋,𝜏 ⋅
𝐖𝑠𝜋,𝜏 in the matrix formulation in equation 39). This term scores how much beliefs within the 𝐀
matrix are expected to change after receiving a new observation. Because the novelty term is a
positive value (and subtracted from the total value), this entails that minimizing expected free
energy will now also drive information-seeking about parameter values in the 𝐀 matrix (i.e., as
opposed to simply seeking out information about states). In other words, the agent will also seek
out observations to increase confidence in its beliefs about 𝑝(𝑜𝜏 |𝑠𝜏 ). To do this, the agent will seek
out state-observation pairings that will maximize the difference in concentration parameters
between posterior and prior distributions over 𝐀. This difference quantifies the drive or epistemic
affordance of finding out ‘what would happen if I do that?’. Although we do not show them explicitly
here, similar terms can also be added to the 𝐸𝐹𝐸 if the agent is learning any of the other matrices or
vectors in the model (e.g., learning the transition probabilities in 𝐁𝜋,𝜏 ).

Note that the value of the novelty term is inversely related to concentration parameter values.
When the concentration parameters have large values, this term will have a small value, and when
the concentration parameters have small values, this term will have a large value. Therefore, when
the concentration parameter values are high (i.e., novelty is low), the model will become primarily
reward-seeking, as it will be highly confident in its beliefs. In contrast, the agent will be
information-seeking when concentration parameter values are low. Analogous dynamics occur
when updating concentration parameters for other model parameters.

To make this more concrete, we show a worked example of the novelty term for two 𝐀 matrices.
One with small concentration parameter values (i.e., low confidence in beliefs about the outcomes
generated by hidden states), and the other with large concentration parameter values (i.e., high
confidence in beliefs about the outcomes generated by hidden states).

Small concentration parameter values (low confidence):

. 25 1 . 25 . 5 .9
𝒂=[ ]; 𝐀 = σ(𝒂) = [ ]; 𝑠𝜋,𝜏 = [ ];
. 75 1 . 75 . 5 .1

83
Active Inference Tutorial

. 25 + .75 1 + 1 1 2
𝒂𝑠𝑢𝑚𝑠 = [ ]=[ ];
. 25 + .75 1 + 1 1 2

. 275
𝐀𝑠𝜋,𝜏 = [ ]
. 725
1
𝐖 ∶= (𝒂⨀(−1) − 𝒂⨀(−1)
𝑠𝑢𝑚𝑠 )
2
1 . 25−1 1−1 ] − [1−1 2−1 ]) = 1 ([ 4 1 1 .5 1 3 .5
𝐖= ([ ]−[ ]) = ([ ])
2 . 75−1 1−1 1−1 2−1 2 1.3333 1 1 .5 2 . 3333 . 5

1.5 . 25
= [ ]
. 1667 . 25

1.5 . 25 . 9 1.375
𝐖𝑠𝜋,𝜏 = [ ] [ ]=[ ]
. 167 . 25 . 1 . 175

. 275 1.375
𝑁𝑜𝑣𝑒𝑙𝑡𝑦 = 𝐀𝑠𝜋,𝜏 ⋅ 𝐖𝑠𝜋,𝜏 = [ ]⋅[ ] = .505
. 725 . 175

Large concentration parameter values (high confidence):

25 100 . 25 .5 .9
𝒂=[ ]; 𝐀 = σ(𝒂) = [ ]; 𝑠𝜋,𝜏 = [ ];
75 100 . 75 .5 .1

25 + 75 100 + 100 100 200

𝒂𝑠𝑢𝑚𝑠 = [ ]=[ ];
25 + 75 100 + 100 100 200

. 275
𝐀𝑠𝜋,𝜏 = [ ]
. 725

1 ⨀(−1)
𝐖 ∶= (𝒂⨀(−1) − 𝒂𝑠𝑢𝑚𝑠 )
2
1 25−1 100−1 ] − [100−1 200−1 ]) = 1 ([ . 04 . 01 . 01 . 005
𝐖= ([ ]−[ ])
2 75−1 100−1 100−1 200 −1 2 . 0133 . 01 . 01 . 005
1 . 03 . 005 . 015 . 0025
= ([
2 . 0033
]) = [ ]
. 005 . 0017 . 0025

. 015 . 0025 . 9 . 01375

𝐖𝑠𝜋,𝜏 = [ ] [ ]=[ ]
. 00167 . 0025 . 1 . 00175

. 275 . 01375
𝑁𝑜𝑣𝑒𝑙𝑡𝑦 = 𝐀𝑠𝜋,𝜏 ⋅ 𝐖𝑠𝜋,𝜏 = [ ]⋅[ ] = .00505
. 725 . 00175

In both examples, the policy assigns high probability to occupying state 1 (p = .9). The normalized
shape of the distribution for each column in 𝐀 is also the same in both examples. However, the
novelty term is larger in the first example where the associated Dirichlet prior 𝒂 has smaller

84
Active Inference Tutorial

concentration parameter values (which, when subtracted from the total, will lead to a lower 𝐸𝐹𝐸).
This means the agent will learn more (i.e., change its beliefs more) when moving to states where it
is less confident in its beliefs (encoded as smaller concentration parameter values). To get a more
intuitive sense for these computations, you can reproduce these results and adjust the
concentration parameter values in the supplementary script 𝐄𝐅𝐄_𝐥𝐞𝐚𝐫𝐧𝐢𝐧𝐠_𝐧𝐨𝐯𝐞𝐥𝐭𝐲_𝐭𝐞𝐫𝐦. 𝐦.

4.3 Simulating Learning

In this subsection we will build on the explore-exploit task model we specified above and
demonstrate how it can also be used to simulate learning. By the end of this section, the reader
should be equipped to run these simulations independently, and to plot and interpret their results.
With the explore-exploit task model in place, we only need a few additions to the code. First, a
lowercase version of the to-be-learned model element must be created. For example, to enable
learning within the 𝐀 matrix, one must specify an 𝐦𝐝𝐩. 𝐚 with the same dimensions as 𝐦𝐝𝐩. 𝐀. The
same goes for other parameters (𝐦𝐝𝐩. 𝐛, 𝐦𝐝𝐩. 𝐝, etc.). Typically, the initial concentration
parameters would be set to low-confidence (i.e., low-magnitude), flat distributions before learning
begins; for example: 𝐷𝑖𝑟(𝑑) = 𝑑{1} = [.25 .25]′. Note here that the generative process continues to
correspond to the capital-letter matrices (i.e., which will generate the patterns of observations),
while the lowercase-letter matrices are now the generative model. Next, we can specify a learning
rate and a forgetting rate by setting 𝐦𝐝𝐩. 𝐞𝐭𝐚 and 𝐦𝐝𝐩. 𝐨𝐦𝐞𝐠𝐚 equal to values between 0 and 1.
Finally, we need to replicate the 𝐦𝐝𝐩 structure to include many trials; for example, using code such
as:

𝑵_𝑻𝒓𝒊𝒂𝒍𝒔 = 𝟑𝟎;

[𝒎𝒅𝒑(𝟏: 𝑵_𝑻𝒓𝒊𝒂𝒍𝒔)] = 𝒅𝒆𝒂𝒍(𝒎𝒅𝒑);

Then, we can simply run the 𝐦𝐝𝐩 structure through the 𝐬𝐩𝐦_𝐌𝐃𝐏_𝐕𝐁_𝐗_𝐭𝐮𝐭𝐨𝐫𝐢𝐚𝐥. 𝐦 function as
before. Here, we will simulate two different versions of the task. In the first version, there are 30
trials, and the better slot machine is the same for all trials (left machine). We will allow the
simulated agent to learn prior expectations about which context is more likely (i.e., whether the left
or right machine tends to lead to wins more often). To enable this type of learning, we will include
𝐦𝐝𝐩. 𝐝. As shown in the 𝐒𝐭𝐞𝐩_𝐛𝐲_𝐒𝐭𝐞𝐩_𝐀𝐈_𝐆𝐮𝐢𝐝𝐞. 𝐦 code, we specify low confidence in initial state
priors, 𝑑{1} = [.25 .25]′. We also set the learning rate to 𝐦𝐝𝐩. 𝐞𝐭𝐚 = . 𝟓 and the forgetting rate to
𝐦𝐝𝐩. 𝐨𝐦𝐞𝐠𝐚 = 𝟏 (i.e., no forgetting). We then define a “risk-seeking” (𝑅𝑆) parameter that
corresponds to how precise the preference is to win the higher amount of money in the 𝐂 matrix:

𝐶 𝑤𝑖𝑛 =

𝟎 𝟎 𝟎

𝐂{𝟐} = 𝟎 −𝟏 −𝟏
𝑹𝑺
[𝟎 𝑹𝑺 𝟐 ]

85
Active Inference Tutorial

In our first simulation, we set 𝑅𝑆 = 3 and in our second we set 𝑅𝑆 = 4. We expect the agent will be
less information-seeking, and more risk-seeking, in the latter case. As can be seen in Figure 10
(top-left), the agent with 𝑅𝑆 = 3 chooses to take the hint on the first several trials, and slowly
begins to forego the hint on later trials (with some choice stochasticity). Unexpected losses, shown
in the panel just below, also cause the agent to return to ‘playing it safe’ and again ask for the hint.
In contrast, the agent with 𝑅𝑆 = 4 chooses to take the hint only once (Figure 10, top-right) and
then takes that as sufficient evidence that the left machine must be the better one (and continues to
choose this one throughout, despite occasional losses). The lower panels in the figure show
simulated event-related potentials (ERPs), simulated dopamine responses, and how beliefs change
over time regarding which context is more likely (darker = higher probability). When the 𝑺𝒊𝒎
variable is set to 𝑺𝒊𝒎 = 𝟐 in the 𝐒𝐭𝐞𝐩_𝐛𝐲_𝐒𝐭𝐞𝐩_𝐀𝐈_𝐆𝐮𝐢𝐝𝐞. 𝐦 code, you can reproduce these
simulations (and adjust the 𝑅𝑆 parameter to other possible values).

Figure 10. Simulated learning on the explore-exploit task and predicted neuronal responses. Blue
circles in the top panel indicate chosen actions (i.e., the agent’s first choice on each trial); darker
shading indicates a higher action probability. Wins/losses, free energies, simulated neuronal
responses, and changes in prior beliefs about context over time (darker = stronger prior belief) are
shown in the lower panels. See main text for more details. Left: Example simulation of a risk-averse
agent (i.e., defined by a moderately precise preference distribution; 𝑅𝑆 = 3) learning from repeated
trials of the explore-exploit task. Here the agent slowly gains confidence that the left machine will
always be better and begins to choose that option without taking the hint. This agent often returns
to taking the hint after unexpected losses. Right: Example simulation of a risk-seeking agent (i.e.,
defined by a highly precise preference distribution; 𝑅𝑆 = 4) learning from repeated trials of the

86
Active Inference Tutorial

explore-exploit task. This agent only required seeing the hint one time before attempting to pick the
correct option directly (and win more money). It did not return to taking the hint with occasional
unexpected losses. Note that both agents also show some stochasticity in choice. These simulations
can be reproduced by running the 𝑺𝒊𝒎 = 𝟐 option in the supplementary
𝐒𝐭𝐞𝐩_𝐛𝐲_𝐒𝐭𝐞𝐩_𝐀𝐈_𝐆𝐮𝐢𝐝𝐞. 𝐦 code (although note that, because outcomes are sampled from
probability distributions, results will not be identical each time).

Next, we simulate a reversal learning paradigm. Here, there are 32 trials in total. Unbeknownst to
the agent, in the first 4 trials the left machine will be better, but in the rest of the trials the right
machine will be better. Again, we examine an agent with 𝑅𝑆 = 3 and 𝑅𝑆 = 4. As shown in Figure
11, the 𝑅𝑆 = 3 agent chose to take the hint on all trials in this simulation. In contrast, the 𝑅𝑆 = 4
agent quickly locked on to the left machine, but it then returned to taking the hint after the reversal.
After several trials of again choosing the hint, it becomes confident in directly choosing the right
machine in the final trials. When the 𝑺𝒊𝒎 variable is set to 𝑺𝒊𝒎 = 𝟑 in the
𝐒𝐭𝐞𝐩_𝐛𝐲_𝐒𝐭𝐞𝐩_𝐀𝐈_𝐆𝐮𝐢𝐝𝐞. 𝐦 code, you can reproduce these simulations (and adjust the 𝑅𝑆
parameter to other possible values).

Figure 11. Simulated reversal learning on the explore-exploit task and predicted neuronal
responses. Here, the better machine was on the left for the first 4 trials and then on the right for all
subsequent trials (without the agent expecting this). Blue circles in the top panel indicate chosen
actions (i.e., the agent’s first choice on each trial); darker shading indicates a higher action
probability. Wins/losses, free energies, simulated neuronal responses, and changes in prior beliefs
about context over time (darker = stronger prior belief) are shown in the lower panels. See main

87
Active Inference Tutorial

text for more details. Left: Example simulation of a risk-averse agent (i.e., defined by a moderately
precise preference distribution; 𝑅𝑆 = 3), who always chose to take the hint. Right: Example
simulation of a risk-seeking agent (i.e., defined by a highly precise preference distribution; 𝑅𝑆 = 4),
who quickly became confident in choosing the left machine without taking the hint. After the
unexpected reversal, it decided to again take the hint for many trials before becoming confident in
choosing the right machine directly. These simulations can be reproduced by running the 𝑺𝒊𝒎 = 𝟑
option in the supplementary 𝐒𝐭𝐞𝐩_𝐛𝐲_𝐒𝐭𝐞𝐩_𝐀𝐈_𝐆𝐮𝐢𝐝𝐞. 𝐦 code (although note that, because
outcomes are sampled from probability distributions, results will not be identical each time).

Each of these examples is meant primarily to give the reader a sense of how to work with these
types of simulations. But there are many other parameters that could be manipulated. In the
accompanying MATLAB code, the reader can easily re-run these simulations while changing the
learning rate, forgetting rate, action precision, or any other parameters in the model. We encourage
the reader to do so to get a sense of the unique influences of different parameters on task behavior.
For examples of papers that model learning using active inference, see (Friston, FitzGerald, et al.,
2016; Friston, Lin, et al., 2017; Schwartenbeck et al., 2019; Smith, Mayeli, et al., 2021; Smith, Parr, &
Friston, 2019; Smith, Schwartenbeck, Parr, & Friston, 2020; Smith, Schwartenbeck, Stewart, et al.,
2020; A. Tschantz, Seth, & Buckley, 2020).

5. Neural Process Theory

In many active inference papers, one sees figures similar to Figure 12. These figures typically
depict a series of update equations, several columns of ‘ball’ neurons, a specific pattern of synaptic
connections, and labels assigning model variables to those neurons and synaptic connections. Such
figures are meant to depict one possible neural implementation of active inference, which serves as
a concrete illustration of the general biological plausibility of the theory. This type of biological
plausibility is considered an important strength of active inference models, due to the resulting
ability to make empirical predictions about neurophysiological responses. In turn, these predictions
allow one to assess the evidence for distinct message passing algorithms and possible neuronal
implementations (Parr et al., 2019). Some of the simulation outputs in Table 3 contain predicted
neuronal responses that can be assigned to distinct neuronal populations or synaptic connections
(some examples are plotted in Figure 8). To be clear, some neurophysiological predictions in active
inference are not specific to a single neural implementation (i.e., they are based only on the generic
prediction error minimization and precision updating equations described above); and these types
of predictions have been successfully associated with neural responses observed in previous
functional magnetic resonance imaging [fMRI] and EEG studies (e.g., see (Schwartenbeck et al.,
2015; Smith, Mayeli, et al., 2021; Whyte, Hohwy, & Smith, 2021; Whyte & Smith, 2020)). However,
cortical column proposals – such as that shown in Figure 12 – can also motivate targeted research
using methods (such as laminar fMRI) that have been applied to test similar columnar
implementations proposed for predictive coding (K. E. Stephan et al., 2019). To prepare the reader
for understanding – and potentially contributing to – this important area of active inference
research, we will walk the reader through Figure 12 step-by-step. This will also be important when

88
Active Inference Tutorial

we discuss hierarchical models in the next section (Section 6), in which simulated EEG responses
occur over different timescales and are predicted to occur at distinct levels of processing within the
brain.

Figure 12. Common depiction of the neural process theory associated with active inference. This
includes the update equations on the left and an example neural network that could implement
them on the right (only exemplar synaptic connections are shown to avoid visual clutter). See main
text for an in-depth walk-through. Note that while most equations are presented in the same form
as in the main text, we have updated the outcome prediction error (𝜍𝜋,𝜏 ) equation to include the
‘novelty’ term (𝐀𝑠𝜋,𝜏 ⋅ 𝐖𝑠𝜋,𝜏 ) within 𝐺𝜋 that was introduced in the previous section on learning. We
also depict iterative prediction error minimization and precision updating processes using arrows
indicating the order of repeated updating until convergence.

In the depicted neural network, each column of neurons (in this case, 3 columns) represents beliefs
and prediction errors about each point in time (from left to right, indicated by subscripts for 𝜏 =
1,2,3). With each new observation, beliefs about all time points (i.e., about the past, present, and
future) are updated, corresponding to changes in neural activation across all neurons. The upward
arrows from observations (purple nodes at the bottom) to layer 3 (denoted by 𝜀𝜋,𝜏 for state
prediction errors) are depicted as conveying excitatory (red) observation signals (e.g., sensory
input) to granular cells in each cortical column, where these observations can differ at each time
point. The receiving (pink) state prediction error neurons calculate their prediction errors by
combining observation signals with prediction signals from the state representations in the cyan
neurons of layer 2 (supragranular neurons denoted by 𝑠𝜋,𝜏 for state representations). Note that
excitatory (red) downward signals from these neurons to layer 3 are conveyed both forward (from
the 𝜏 = 1, left neurons) and backward (from the 𝜏 = 3, right neurons) – indicating both prospective
and retrospective predictive influences on state representations about a time point. In contrast,
inhibitory (blue) signals are conveyed by these state representation neurons to layer three neurons

89
Active Inference Tutorial

for the current time point, leading to minimization of prediction error when predictions from state
representations match observation signals.

Note next that each of these state and state prediction error representations are calculated in
parallel for each policy (denoted by one example neural column in front of another). The top (red)
layer 1 neurons, however, do not have another set of neurons behind them. This is because they
perform a Bayesian model average as an overall best guess about states. They do this by taking
state representations for each policy, multiplying them by the probability of that policy, and then
averaging them to get a final posterior over states. This is accomplished in conjunction with the
policy representation (𝜋) neuron on the left (meant to represent a subcortical neural population),
the signals from which multiply (via the green modulatory connection) the excitatory (red) signals
from the state representation neurons for each policy in layer 2. After each new observation,
activity in these policy representation neurons also promotes some actions (𝑢) over others.

Policy representation (𝜋) neurons are in turn activated by neurons encoding habits (𝐸) and
inhibited by those encoding expected free energy (𝐺; i.e., greater expected free energy reduces the
probability of a policy). The influence of 𝐺 on 𝜋 is modulated by the expected free energy precision
term (𝛾) – depicted here as being conveyed by subcortical dopamine neurons. The activity of 𝐺
neurons is increased by outcome prediction errors (𝜁𝜋,𝜏 ) in layer 5, multiplied (green connections)
by the probability of those outcomes under each policy (𝑜𝜋,𝜏 , cyan neurons in layer 4; note that
these are calculated from 𝐀𝑠𝜋,𝜏 ). These outcome prediction errors reflect the expected difference
between preferred outcomes and predicted outcomes under each policy (from layer 4; downward
excitatory connections between layers 4 and 5), and where those predicted outcomes are in turn
based on state representations in layer 2 (excitatory red connections from layer 2 to layers 4 and
5). Note that the matrix 𝐖 (i.e., calculated based on the synaptic strengths encoded in 𝐀) also
influences 𝐺 through interaction with activity in the neurons encoding 𝑠𝜋,𝜏 , but these connections
are not explicitly shown to minimize visual clutter.

Based on this description, we assume the reader should be able to follow the equations on the left
to identify each associated connection in the network on the right. Note that this figure only shows
example connections, assuming two policies and three time points. However, the basic idea is that,
if each excitatory connection corresponds to addition, each inhibitory connection corresponds to
subtraction, and each modulatory connection corresponds to multiplication, then each of the
update equations on the left (in the ‘Belief updating’, ‘Policy inference and expected free energy
precision’, ‘Action selection’, and ‘Learning’ boxes) can be implemented in a straightforward
manner within a relatively simple neural network. It is worth mentioning that these update
equations are not always presented in identical form in such figures across the literature. However,
these variations are either algebraically equivalent or have been presented with or without certain
elements (e.g., with or without learning, with or without expected free energy precision, etc.),
depending on the goals of the paper.

One other aspect of the neural process theory that we will not discuss in detail – but that we would
like to briefly point the interested reader toward – pertains to the idea that Bayesian model
reduction (i.e., comparing models to find the simplest one that can account for available data) is

90
Active Inference Tutorial

implemented by homeostatic synaptic adjustment processes during sleep and resting wakefulness
(Bucci & Grasso, 2017; Friston, Lin, et al., 2017; Hobson & Friston, 2012; Hobson, Hong, & Friston,
2014; Smith, Schwartenbeck, Parr, et al., 2020; Tononi & Cirelli, 2014). In short, this literature
suggests that, during sleep/rest, the brain can also minimize VFE by finding simpler models (with
fewer parameters) that can successfully account for previous experience. This can be accomplished
in part via a ‘synaptic downscaling’ process – known to occur during sleep – in which synaptic
connections that have gotten stronger during recent learning are subsequently attenuated. Synaptic
downscaling can be helpful in removing any small synaptic changes driven by noise (e.g.,
uninformative coincidences in the presence of multiple stimuli), leaving only the larger synaptic
changes needed to account for consistent patterns in recent experience. Although we do not cover it
here, we note that the 𝐬𝐩𝐦_𝐌𝐃𝐏_𝐕𝐁_𝐗. 𝐦 script (and our tutorial version) does have an additional
‘BMR’ option that can be turned on, which will implement Bayesian model reduction by calling a
further SPM script written to simulate this proposed function of sleep/rest
(𝐬𝐩𝐦_𝐌𝐃𝐏_𝐕𝐁_𝐬𝐥𝐞𝐞𝐩. 𝐦). In this case, the entries in each matrix (e.g., the 𝐀 matrix) are assumed to
represent synaptic connections, and model reduction involves eliminating any changes in the
concentration parameters for those matrices during learning that did not improve the explanatory
power of the model (i.e., in terms of the accuracy/complexity trade-off).

To facilitate the reader’s ability to use the neural process theory in their own research, the
𝐬𝐩𝐦_𝐌𝐃𝐏_𝐕𝐁_𝐗_𝐭𝐮𝐭𝐨𝐫𝐢𝐚𝐥. 𝐦 script automatically generates simulated responses for several of the
neuronal populations described above (see Table 3 for specific output descriptions). This includes
the simulated firing rates of neuronal populations representing posteriors over states (i.e., whose
predicted location in the brain would depend on the task). Here, distinct firing rates are generated
for the states within each state factor of a model (e.g., in the explore-exploit task, firing rates in one
population would encode the probability of the ‘left-better context’, while those in another
population would encode the probability of the ‘right-better context’). The script also generates
vectors encoding the associated state prediction errors discussed above. Simulated
electrophysiological responses are based on the rates of change (derivatives) in firing rates during
prediction-error minimization (interpreted as local field potentials [LFPs] or event-related
potentials [ERPs], depending on the context). Separate neural populations are postulated to encode
posteriors over states and prediction errors for each time point 𝜏 (i.e., each of which are updated
with new observations over time).

Another set of potentially useful outputs generated by the 𝐬𝐩𝐦_𝐌𝐃𝐏_𝐕𝐁_𝐗_𝐭𝐮𝐭𝐨𝐫𝐢𝐚𝐥. 𝐦 script

corresponds to the 𝑉𝐹𝐸 of changes in concentration parameters (i.e., higher = greater surprise
during learning; see Table 3). Recall here that another important aspect of the neural process
theory is that synaptic inputs with different strengths are postulated to carry the prior and
conditional probabilities encoded in each of the vectors/matrices that define a model (e.g., the
probabilities in the 𝐀 or 𝐁𝜋,𝜏 matrices, or in the 𝐷 or 𝐸 vectors). More specifically, the value of each
entry in particular matrices or vectors can be thought of as corresponding to the strength of a
synaptic connection between two neurons. Updating concentration parameters over repeated trials
(i.e., over a slower timescale than perception) can therefore be linked to synaptic plasticity – where
the specific changes involved depend on the pattern of observations, belief updates, and/or policies
chosen on each trial). The 𝑉𝐹𝐸 of changes in concentration parameters can therefore be used to

91
Active Inference Tutorial

quantify the magnitude of change in a model on a trial by trial basis, which could be used to identify
neural correlates of such changes. The predicted dynamics of within-trial prediction errors and
belief updates will also be modulated by these synaptic changes, allowing for predicted time
courses that could be used for similar experimental purposes. In addition, synaptic strength
changes will also inform the matrix 𝐖 – dynamically adjusting motivation to seek information
about model parameters. Thus, several experimentally useful outputs are provided to test the
neural process theory.

An available SPM function for plotting single-trial neural simulations can be run by inputting the
following into MATLAB:

𝐬𝐩𝐦_𝐟𝐢𝐠𝐮𝐫𝐞(′𝐆𝐞𝐭𝐖𝐢𝐧′, ′𝐅𝐢𝐠𝐮𝐫𝐞 𝟐′); 𝐜𝐥𝐟; 𝐬𝐩𝐦_𝐌𝐃𝐏_𝐕𝐁_𝐋𝐅𝐏(𝐌𝐃𝐏); 𝐬𝐮𝐛𝐩𝐥𝐨𝐭(𝟑, 𝟐, 𝟑)

Setting the variable 𝑺𝒊𝒎 in the accompanying tutorial code (i.e., 𝐒𝐭𝐞𝐩_𝐛𝐲_𝐒𝐭𝐞𝐩_𝐀𝐈_𝐆𝐮𝐢𝐝𝐞. 𝐦, line
51) to 𝑺𝒊𝒎 = 𝟏 will also generate simulation plots using this function.

Based on the current model specification, a representative plot of simulation results is shown in
Figure 8C, based on the single trial depicted in Figure 8A. The top-left panel depicts the belief
updates at each time point 𝑡 (columns) about each time point 𝜏 (rows). As before, darker indicates
higher probability. In this case, after presentation of the first observation (𝑡 = 1, column 1), the
model is fully uncertain about current or future states (each row). After presentation of the second
observation (𝑡 = 2, column 2), when the model receives the hint, it becomes highly confident that it
was, currently is, and will continue to be in the ‘left-better’ state (i.e., based on its transition beliefs
that this state does not change within a trial). These beliefs remain stable when it receives the third
observation (column 3; i.e., upon observing the expected win). The top-right panel depicts these
belief updates (i.e., changes in beliefs over time about the state at each time point; 2 possible states
for each of the 3 time points). These belief updates are depicted as traces of changes in neural firing
rates, with 2 firing rates per distribution (i.e., encoding the probability of the left- vs. right-better
context), resulting in 6 firing rate traces in total (note that, due to overlap, not all 6 traces are
clearly visible in this example). The bottom-left plot depicts this same information, but here
displayed in terms of a simulated raster plot (i.e., one tick per action potential per neuron in a
simulated population). The middle-right panel depicts predicted local field potentials (or event-
related potentials), which reflect the rate of change in the simulated firing rates. The middle-left
panel depicts the neural responses associated with context state beliefs before (dotted line) and
after (solid line) filtering at 4 Hz, superimposed on a time-frequency decomposition of the local
field potential (averaged over all simulated neurons). This type of plot has been used in previous
work to explain how/why simulated depolarization in specific frequency ranges may coincide with
specific stimulus-induced neural responses (Friston, FitzGerald, et al., 2017). The bottom-right
panel depicts simulated dopamine responses after each new observation (as also depicted in a
slightly different way in Figure 8A). These were covered in detail in Section 3 (see Figure 9 and the
last row of Table 2).

This plotting function also has several options as additional function entries and outputs as follows:

92
Active Inference Tutorial

[𝐮, 𝐯] = 𝐬𝐩𝐦_𝐌𝐃𝐏_𝐕𝐁_𝐋𝐅𝐏(𝐌𝐃𝐏, 𝐔𝐍𝐈𝐓𝐒, 𝐅𝐀𝐂𝐓𝐎𝐑, 𝐒𝐏𝐄𝐂𝐓𝐑𝐀𝐋).

UNITS: a matrix with 2 rows and one or more columns. The first row indicates which hidden
state(s) to plot over time for the specified state factor. The second row specifies the time point
being represented. For example: 𝑼𝑵𝑰𝑻𝑺 = [𝟏 𝟐 𝟏 𝟐; 𝟏 𝟏 𝟑 𝟑] would plot firing rates for the first
two hidden states of the selected state factor over time with regard to beliefs about time points 1
and 3. By default, all units are selected.

FACTOR: a single number denoting which state factor to plot (default = 1).

SPECTRAL: either a 0 or 1 (default = 0). If 1, the top-left plot is replaced by a plot of the power of
simulated neural responses in different frequency ranges.

The optional outputs u and v correspond to vectors encoding simulated event-related potentials
and firing rates, respectively (for selected units). As was mentioned above, the event-related
potentials correspond to the temporal derivative of the firing rates, while the firing rates reflect the
magnitude of posterior beliefs over each state at each iteration of marginal message passing.

6. Building Hierarchical Models

6.1 Hierarchical Model Structure

Now that we have a clear idea of how to specify a generative model of a behavioral task, how to
interpret the relevant outputs, and how we can generate testable predictions regarding neural
responses, we will now extend this foundation by building a hierarchical or ‘deep temporal’ model
and demonstrating how it can be used to reproduce established neurophysiological results (for
examples, see (Friston et al., 2018; Parr & Friston, 2017b; Smith, Lane, Parr, & Friston, 2019; Whyte
et al., 2021; Whyte & Smith, 2020)). Specifically, we will reproduce the results of experiments
examining ERPs in a commonly used auditory mismatch paradigm designed to study the neural
basis of perceptual learning and expectation violation. This should help the reader to generalize
their understanding by seeing how to build a model with a different structure. It will also
demonstrate the versatility and wide range of applications of active inference models. For example,
while reinforcement learning models are often used to solve tasks like the slot machine task, such
models are not readily applicable to perceptual tasks, like the auditory mismatch paradigm, that do
not include rewards or produce notable variability in behavior (e.g., when performance tends to be
near ceiling across participants). By the end of this section, the reader should have a solid
understanding of both how hierarchical models work and how to implement them.

The steps for building hierarchical models are quite similar to what we’ve already covered, because
this primarily just involves building two models, and then placing one below the other. The further
step is figuring out how to link the two models together. This is because, in hierarchical models, the
states at the lower level exchange information with states at the higher level in a very specific
bidirectional manner (see Figure 13). First, for each time point in a second-level trial, the hidden
{2}
states at the second level (𝑠𝜏 ; superscript indicates hierarchical level) provide prior beliefs over
the initial states of a first-level trial (i.e., 𝐷 {1} = 𝐀{2}T 𝐷{2} for the first first-level trial; 𝐷 {1} =

93
Active Inference Tutorial

{2}
𝐀{2}T 𝑠𝜏,𝑡=𝜏−1 for all subsequent first-level trials). In turn, posterior beliefs over initial states at the
{1}
end of a first-level trial (i.e., 𝑠𝜏=1,𝑡=𝑇 ) are treated as observations at a time point in a second-level
trial (𝑜𝜏Error! Bookmark not defined. ). This means that the second-level 𝐀 matrix (likelihood mapping)
mediates the ascending and descending messages between hierarchical levels. This structure also
entails that the second-level model must operate at a slower timescale than the first-level model,
because each observation in the second-level model (i.e., each time point in a second-level trial)
corresponds to the results of (i.e., posterior beliefs after) a complete trial in the first-level model.
Thus, there are as many first-level trials as there are time points in a second-level trial. For
example, if there are four time points in a second-level trial, this means there will need to be a
corresponding sequence of four first-level trials (i.e., where each first-level trial could itself have
several time points). This is why such models are often called deep temporal models.

This type of model architecture is essential for capturing perceptual phenomena with nested
dynamics, or where objects must be recognized before regularities in the behavior of those objects
can be detected. For example, to perceive a baseball flying in a leftward direction, a lower-level
model would first need to infer the baseball’s identity and position (i.e., one inferred position per
lower-level trial), and a higher-level model would then need to accumulate evidence for a leftward
trajectory of motion based on how the baseball’s inferred position changes across several lower-
level trials. As another example, to recognize a melody, a lower-level model would be needed to
infer the presence of each note, and a higher-level model would then be needed to accumulate
evidence for a specific melody, based on a specific sequence of inferred notes over time. A further
intuitive example is reading, where the first level infers single words, while the second level infers
the narrative entailed by sequences of words (Friston et al., 2018). Note, as soon as we start to use
deep or hierarchical generative models, we are essentially relaxing the Markovian assumption by
introducing a separation of temporal scales to produce what are known as semi-Markovian models.
These are essential for inferring narratives, language, or any deeply structured sequence of state
transitions.

Aside from these examples, the hierarchical POMDP setup is quite flexible and can be used to model
a wide range of temporally structured phenomena. For example, a policy space could be included at
either level alone, or both levels, depending on the target phenomenon to be modelled (e.g., verbal
report at a higher level vs. reflexive behavior at a lower level). One could also specify several time
points in each lower-level trial, such that higher-level states generate sequences or trajectories of
state transitions at the lower level (i.e., within-trial). In previous work, hierarchical models have
been used to model working memory, reading, visual consciousness, and emotional awareness,
among other phenomena (Friston, Parr, et al., 2017; Friston et al., 2018; Hesp et al., 2020; Parr &
Friston, 2017b, 2018b; Sandved-Smith et al., 2021; Smith, Lane, et al., 2019; Whyte et al., 2021;
Whyte & Smith, 2020). Hierarchical POMDPs also afford further opportunities for simulating
neuronal processes. To date, simulations associated with the faster and slower timescales of belief
updating have been shown to reproduce an impressive number of task-based electrophysiological
findings. For example, empirically observed patterns of ERPs associated with specific cognitive and
perceptual processes, such as the P300 and mismatch negativity (MMN), emerge naturally in
simulations of different experimental paradigms, which supports the face validity of both the model

94
Active Inference Tutorial

structure and the neural process theory (e.g., (Friston, FitzGerald, et al., 2017; Parr & Friston,
2017b; Whyte et al., 2021; Whyte & Smith, 2020)).

Figure 13. Top Left: Bayesian network depiction of a 2-level POMDP. Observations depend on

95
Active Inference Tutorial

hidden states at the first level. In turn, hidden states at the first level depend on hidden states at the
second level. Specifically, second-level hidden states provide prior beliefs over initial states at the
first level, while posterior beliefs over initial states at the first level are treated as observations by
the second level. In the example shown here, the first level has two state transitions per trial. This
entails that the first level has two state transitions for every one state transition at the second level.
Thus, beliefs at the second level evolve over a slower timescale. Top Right: Example neural
network implementing the hierarchical POMDP shown on the left. Bottom: Illustration of the
auditory mismatch paradigm for which we build a generative model and run simulations in this
section (Section 6). In the model specified in the main text, beliefs about single tones are encoded at
the first level, while beliefs about sequences of tones are encoded at the second level. Across trials,
the agent then builds up prior expectations through presentation of repeated tones and tone
sequences. In line with empirical results, the post-learning model simulations shown in Figures 14-
16 predict earlier ERPs for unexpected tones (local deviations) and later ERPs for unexpected tone
sequences (global deviations).

6.2 Building a Hierarchical Model

As a concrete, empirically relevant example, we will now demonstrate how one could build a
hierarchical model to simulate a simplified version of the auditory mismatch ‘local-global’ paradigm
introduced in (Bekinschtein et al., 2009); see bottom panel of Figure 13. In our simplified version
of this paradigm, each trial consists of a sequence of four tones (with either low or high frequency),
the first three have the same frequency, and the fourth tone either conforms to the predicted
pattern (e.g., high-high-high-high; local standard) or violates the predicted pattern by presenting a
different tone (e.g., high-high-high-low; local deviation). During EEG, local deviations elicit a
mismatch negativity component in ERPs (i.e., a negative component obtained by subtracting ‘local
standard’ trials from ‘local deviation’ trials), which appears after approximately 130 ms.
Importantly, the sequence of local standard and local deviation trials establishes a global pattern
that can itself be confirmed (global standard) or violated (global deviation). For example, this could
include several local deviation trials in a row followed by an unexpected local standard trial. Global
deviations are known to elicit a P300 ERP component (i.e., a positive component that appears after
approximately 300 ms). Unlike other auditory mismatch paradigms, this design also allows local
and global violation responses to be dissociated. That is, the factorial design leads to four conditions
local standard + global standard, local standard + global deviation, local deviation + global standard,
and local deviation + global deviation. For brevity, here we only simulate two of the four possible
combinations, local deviation + global deviation, and local deviation + global standard. In our
simulated version of the task, we presented an active inference agent with sequences of 4 tones,
where each tone could be either low or high, in an analogous manner to the empirical task. We then
had the agent report whether the last stimulus on each trial was the same as, or different from, the
established pattern.

At this point, the reader is encouraged to open the accompanying MATLAB script and follow along
in parallel (Step_by_Step_Hierarchical_Model.m). As with the previous model, we will start by
setting up the priors over initial states for each hidden state factor at the first level:

96
Active Inference Tutorial

𝑡𝑜𝑛𝑒 )
𝑝(𝑠𝜏=1 =

𝑫{𝟏} = [𝟏 𝟏]′

𝐷𝑖𝑟(𝑑) =

𝒅=𝑫

The specification for 𝐷{1} means that there is an equal probability of a high or a low tone (left and
right entries, respectively). Note that, because the columns of all the vectors and matrices of the
generative process are run through a softmax function in the 𝐬𝐩𝐦_𝐌𝐃𝐏_𝐕𝐁_𝐗_𝐭𝐮𝐭𝐨𝐫𝐢𝐚𝐥. 𝐦 script,
𝐷{1} = [1 1]′ is equivalent to 𝐷{1} = [. 5 .5]′ . As the simulation involves learning, we also need to
separate the generative process from the generative model by including the lowercase 𝑑 for the
generative model. Here we simply set 𝑑 = 𝐷, as the agent will also start out with the belief that a
high and a low tone are equally probable (but with fairly low confidence). However, including 𝑑 will
allow the agent to accumulate concentration parameters (changing the shape of its initial state
priors) over trials based on patterns in its observations.

Next, we must specify the likelihood mappings for the first level in the 𝐀 matrix.

𝑝(𝑜𝜏𝑡𝑜𝑛𝑒 |𝑠𝜏𝑡𝑜𝑛𝑒 ) =

𝟏 𝟎
𝐀{𝟏} = [ ]
𝟎 𝟏

This is simply an identity matrix indicating that the tone states correspond 1-to-1 with tone
observations (columns [left to right]: high tone, low tone states; rows [top to bottom]: high tone,
low tone observations). However, in the generative model we may want to introduce some noise
into tone perception. One convenient way to do this is to first specify:

𝐷𝑖𝑟(𝒂) =

𝒂=𝐀

Then, we can use a softmax function to control the expected precision of the state-observation
mapping with a 𝑝𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛 parameter:

𝒑𝒓𝒆𝒄𝒊𝒔𝒊𝒐𝒏 = 𝟐;

𝒂{𝟏} = 𝒔𝒑𝒎_𝒔𝒐𝒇𝒕𝒎𝒂𝒙(𝒑𝒓𝒆𝒄𝒊𝒔𝒊𝒐𝒏 ∗ 𝒍𝒐𝒈(𝐀{𝟏} + 𝒆𝒙𝒑(−𝟒)));

Note that the 𝑒𝑥𝑝(−4) is simply a very small number added to A{1} to prevent the possibility of
𝑙𝑜𝑔(0), which is undefined (also note that, while −4 is a reasonable value, other values could be
chosen). Depending on the value of the 𝒑𝒓𝒆𝒄𝒊𝒔𝒊𝒐𝒏 parameter (higher = more precise), this will
result in a likelihood mapping that specifies different amounts of sensory noise. For example:

𝐷𝑖𝑟(𝒂) =

97
Active Inference Tutorial

. 𝟗𝟐 . 𝟎𝟖
𝐚{𝟏} = [ ]
. 𝟎𝟖 . 𝟗𝟐

Note that, for clarity, this example shows a lower precision than what results from setting
𝒑𝒓𝒆𝒄𝒊𝒔𝒊𝒐𝒏 = 𝟐 in the accompanying tutorial code.

As we are mainly interested in simulating the learning of prior expectations (𝐷 vector), we also
multiply the generative model parameter for 𝐚 by 100 (an arbitrary large number) to effectively
prevent learning in this parameter. This is because we want the level of sensory noise to remain
consistent across trials.

Next, we can specify transition probabilities in the 𝐁 matrix as identity matrices9, as tones do not
change within a lower-level trial.

𝑡𝑜𝑛𝑒 𝑡𝑜𝑛𝑒 )
𝑝(𝑠𝜏+1 |𝑠𝜏 =

𝟏 𝟎
𝐁{𝟏} = [ ]
𝟎 𝟏

Here columns (left to right) are high tone and low tone states at time 𝜏, while rows (top to bottom)
are high tone and low tone states at time 𝜏 + 1. Here there is no need to separate the generative
process from the generative model, so we do not specify a separate 𝐛 matrix.

We do not include preferences or policies at this level, so we now simply assign each variable to the
mdp structure. For convenience when later linking this to the higher-level model below, we will
denote this structure with an ‘_1’ as follows:

𝐦𝐝𝐩_𝟏. 𝐃 = 𝐃

𝐦𝐝𝐩_𝟏. 𝐝 = 𝐝

𝐦𝐝𝐩_𝟏. 𝐀 = 𝐀

𝐦𝐝𝐩_𝟏. 𝐚 = 𝐚

𝐦𝐝𝐩_𝟏. 𝐁 = 𝐁

For consistency with how we link first- and second-level models below, we then set MDP_1 =
mdp_1 and clear mdp_1.

9
As a brief general note, the columns of all matrices are put through a softmax function in the
spm_MDP_VB_X_tutorial.m script after the addition of negligibly low values to each entry to prevent the problem
that log(0) is undefined. As such, even specification of identity matrices for transition beliefs will not completely
rule out the possibility that states could change over time in the face of very strong observational evidence to the
contrary.

98
Active Inference Tutorial

Now we move on to specifying the second-level model. To keep the variables separate, we will
denote each model variable with a ‘_2’ for this level.

Here, we will include one hidden state for each possible sequence of tones:

𝑠𝑒𝑞𝑢𝑒𝑛𝑐𝑒
𝑝(𝑠𝜏=1 )=

𝑫_𝟐{𝟏} = [𝟏 𝟏 𝟏 𝟏]′

This indicates that initially there is an equal probability of each tone sequence (left to right: ‘all
high’, ‘all low’, ‘high-low’, ‘low-high’). Note again that using four 1s here is just for convenience, as
this vector will subsequently be run through a softmax function and each 1 will become a .25.

Here we must also include a second hidden state factor that encodes beliefs about the time point
within a trial (e.g., ‘first tone’, ‘second tone’, etc.). This includes (from left to right) time points for 4
tones, a delay period, and then a reporting period:

𝑡𝑖𝑚𝑒
𝑝(𝑠𝜏=1 )=

𝑫_𝟐{𝟐} = [𝟏 𝟎 𝟎 𝟎 𝟎 𝟎]′

This indicates that the agent always starts in the ‘time 1’ state.

We also include a reporting state factor, corresponding to the agent either not yet reporting (left
entry), reporting ‘same tone’ (middle entry), and reporting ‘different tone’ (right entry) at the end
of the trial:

𝑟𝑒𝑝𝑜𝑟𝑡
𝑝(𝑠𝜏=1 )=

𝑫_𝟐{𝟑} = [𝟏 𝟎 𝟎]′

This indicates that the agent always starts in a state of not yet having made a report.

Finally, we allow the agent to build up prior beliefs over time with repeated trials. In this case,
because the agent’s beliefs initially match the generative process, we can simply set:

𝐷𝑖𝑟(𝑑) =

𝒅_𝟐 = 𝑫_𝟐

Next, we must specify the likelihood mappings for the second-level 𝐀 matrix. Because time in trial is
a state factor, this becomes somewhat more complex. Specifically, we are now required to specify
the type of tone at each time point that is expected under each sequence. To do so, we can specify the
matrices as follows. For convenience, we can first specify:

𝑠𝑒𝑞𝑢𝑒𝑛𝑐𝑒,𝑡𝑖𝑚𝑒,𝑟𝑒𝑝𝑜𝑟𝑡
𝑝(𝑜𝜏𝑙𝑒𝑣𝑒𝑙 1 𝑡𝑜𝑛𝑒 𝑠𝑡𝑎𝑡𝑒 |𝑠𝜏 )=

99
Active Inference Tutorial

𝒇𝒐𝒓 𝒊 = 𝟏: 𝟔

𝒇𝒐𝒓 𝒋 = 𝟏: 𝟑

𝟏 𝟎 𝟏 𝟎
𝐀_𝟐{𝟏}(: , : , 𝒊, 𝒋) = [ ]
𝟎 𝟏 𝟎 𝟏

𝒆𝒏𝒅

𝒆𝒏𝒅

This says that for the first six time points (𝑖 = 1: 6), and for all three choice states (𝑗 = 1: 3), the
first and third sequence states (i.e., ‘all high tones’ and ‘high tones followed by low tone’; columns 1
and 3) are associated with the ‘high tone’ observation (top row), whereas the second and fourth
sequence states (i.e., ‘all low tones’ and ‘low tones followed by high tone’; columns 2 and 4) are
associated with the ‘low tone’ observation (bottom row). Then, we can adjust this so that the
deviation sequences (‘low tones followed by high tone’ and ‘high tones followed by low tone’;
columns 3 and 4) are associated with the opposite tone mapping at the fourth time point (𝑖 = 4):

𝑠𝑒𝑞𝑢𝑒𝑛𝑐𝑒,𝑡𝑖𝑚𝑒=4,𝑟𝑒𝑝𝑜𝑟𝑡
𝑝(𝑜𝜏𝑙𝑒𝑣𝑒𝑙 1 𝑡𝑜𝑛𝑒 𝑠𝑡𝑎𝑡𝑒 |𝑠𝜏 )=

𝒇𝒐𝒓 𝒊 = 𝟒

𝒇𝒐𝒓 𝒋 = 𝟏: 𝟑

𝟏 𝟎 𝟎 𝟏
𝐀_𝟐{𝟏}(: , : , 𝒊, 𝒋) = [ ]
𝟎 𝟏 𝟏 𝟎

𝒆𝒏𝒅

𝒆𝒏𝒅

The second outcome modality at the higher level (which does not correspond to a lower-level state
factor) is feedback about whether a chosen report was correct or incorrect. Here, we need to
specify that the agent will observe ‘correct’ feedback (at the final time point) in cases where its
report matches the appropriate sequence (row 3), and ‘incorrect’ feedback otherwise (row 2). To
do this, we can initially specify that no feedback (‘null’; row 1) will be observed across all time
points:

𝑓𝑒𝑒𝑑𝑏𝑎𝑐𝑘 𝑠𝑒𝑞𝑢𝑒𝑛𝑐𝑒,𝑡𝑖𝑚𝑒,𝑟𝑒𝑝𝑜𝑟𝑡
𝑝(𝑜𝜏 |𝑠𝜏 )=

𝒇𝒐𝒓 𝒊 = 𝟏: 𝟔

𝒇𝒐𝒓 𝒋 = 𝟏: 𝟑

100
Active Inference Tutorial

𝟏 𝟏 𝟏 𝟏
𝐀_𝟐{𝟐}(: , : , 𝒊, 𝒋) = [𝟎 𝟎 𝟎 𝟎]
𝟎 𝟎 𝟎 𝟎

𝒆𝒏𝒅

𝒆𝒏𝒅

Then we can specify that at time point 𝑖 = 6, if the agent reports “same” (𝑗 = 2), then it will
receive correct feedback when it is one of the first two (standard) sequences and incorrect for
either of the second two (deviant) sequences:

𝑓𝑒𝑒𝑑𝑏𝑎𝑐𝑘 𝑠𝑒𝑞𝑢𝑒𝑛𝑐𝑒,𝑡𝑖𝑚𝑒=6,𝑟𝑒𝑝𝑜𝑟𝑡="𝑠𝑎𝑚𝑒"
𝑝(𝑜𝜏 |𝑠𝜏 )=

𝒇𝒐𝒓 𝒊 = 𝟔

𝒇𝒐𝒓 𝒋 = 𝟐

𝟎 𝟎 𝟎 𝟎
𝐀_𝟐{𝟐}(: , : , 𝒊, 𝒋) = [𝟎 𝟎 𝟏 𝟏]
𝟏 𝟏 𝟎 𝟎

𝒆𝒏𝒅

𝒆𝒏𝒅

The we specify the opposite mapping if the agent reports “different” (𝑗 = 3):

𝑓𝑒𝑒𝑑𝑏𝑎𝑐𝑘 𝑠𝑒𝑞𝑢𝑒𝑛𝑐𝑒,𝑡𝑖𝑚𝑒=6,𝑟𝑒𝑝𝑜𝑟𝑡="𝑑𝑖𝑓𝑓𝑒𝑟𝑒𝑛𝑡"
𝑝(𝑜𝜏 |𝑠𝜏 )=

𝒇𝒐𝒓 𝒊 = 𝟔

𝒇𝒐𝒓 𝒋 = 𝟑

𝟎 𝟎 𝟎 𝟎
𝐀_𝟐{𝟐}(: , : , 𝒊, 𝒋) = [𝟏 𝟏 𝟎 𝟎]
𝟎 𝟎 𝟏 𝟏

𝒆𝒏𝒅

𝒆𝒏𝒅

As with the first-level model, to control the precision of the mapping between first- and second-
level states in the generative model, we can use a 𝑝𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛_2 parameter. To do so, we can specify:

𝐷𝑖𝑟(𝒂) =

𝒂_𝟐 = 𝑨_𝟐

101
Active Inference Tutorial

𝒑𝒓𝒆𝒄𝒊𝒔𝒊𝒐𝒏_𝟐 = 𝟐;

𝒂_𝟐{𝟏} = 𝒔𝒑𝒎_𝒔𝒐𝒇𝒕𝒎𝒂𝒙(𝒑𝒓𝒆𝒄𝒊𝒔𝒊𝒐𝒏_𝟐 ∗ 𝒍𝒐𝒈(𝐀_𝟐{𝟏} + 𝒆𝒙𝒑(−𝟒)));

This results in a minor amount of noise in the messages passed between levels. As with the first
level, we also multiply this parameter by 100 to (effectively) prevent learning.

Next, we must specify the transition matrices for the second level. In this case, the sequence type is
stable within a trial, so this should be an identity matrix (columns: states at time 𝜏, rows: states at
𝜏 + 1):
𝑠𝑒𝑞𝑢𝑒𝑛𝑐𝑒 𝑠𝑒𝑞𝑢𝑒𝑛𝑐𝑒
𝑝(𝑠𝜏+1 |𝑠𝜏 )=

𝟏 𝟎 𝟎 𝟎
𝟎 𝟏 𝟎 𝟎
𝐁_𝟐{𝟏} = [ ]
𝟎 𝟎 𝟏 𝟎
𝟎 𝟎 𝟎 𝟏

Time in trial should progress forward (e.g., ‘Time 1’ should ‘transition to ‘Time 2’, and so forth. As
such:

𝑡𝑖𝑚𝑒 𝑡𝑖𝑚𝑒
𝑝(𝑠𝜏+1 |𝑠𝜏 ) =

𝟎 𝟎 𝟎 𝟎 𝟎 𝟎
𝟏 𝟎 𝟎 𝟎 𝟎 𝟎
𝐁_𝟐{𝟐} = 𝟎 𝟏 𝟎 𝟎 𝟎 𝟎
𝟎 𝟎 𝟏 𝟎 𝟎 𝟎
𝟎 𝟎 𝟎 𝟏 𝟎 𝟎
[𝟎 𝟎 𝟎 𝟎 𝟏 𝟏]

Finally, report states are under control of the agent. In this case, there are three actions:

𝑟𝑒𝑝𝑜𝑟𝑡 𝑟𝑒𝑝𝑜𝑟𝑡
𝑝(𝑠𝜏+1 |𝑠𝜏 , 𝑈 = 𝑛𝑜 𝑟𝑒𝑝𝑜𝑟𝑡) =

𝟏 𝟏 𝟏
𝐁_𝟐{𝟑}(: , : , 𝟏) = [𝟎 𝟎 𝟎]
𝟎 𝟎 𝟎
𝑟𝑒𝑝𝑜𝑟𝑡 𝑟𝑒𝑝𝑜𝑟𝑡
𝑝(𝑠𝜏+1 |𝑠𝜏 , 𝑈 = 𝑟𝑒𝑝𝑜𝑟𝑡 "𝑠𝑎𝑚𝑒") =

𝟎 𝟎 𝟎
𝐁_𝟐{𝟑}(: , : , 𝟐) = [𝟏 𝟏 𝟏]
𝟎 𝟎 𝟎
𝑟𝑒𝑝𝑜𝑟𝑡 𝑟𝑒𝑝𝑜𝑟𝑡
𝑝(𝑠𝜏+1 |𝑠𝜏 , 𝑈 = 𝑟𝑒𝑝𝑜𝑟𝑡 "𝑑𝑖𝑓𝑓𝑒𝑟𝑒𝑛𝑡") =

102
Active Inference Tutorial

𝟎 𝟎 𝟎
𝐁_𝟐{𝟑}(: , : , 𝟑) = [𝟎 𝟎 𝟎]
𝟏 𝟏 𝟏

These three matrices (from 1-3 in dimension 3) correspond to the actions (𝑈) of moving (from any
state) to the ‘no report’ state, ‘report same’ state, and ‘report different’ state, respectively. Next, we
must specify the allowable sequences of actions (i.e., policies). In this case, we include two policies
(two columns) and one row per time point. There are no actions for the first state factor, so
(number = action, column = policy, row = time point):

𝜋 𝑠𝑒𝑞𝑢𝑒𝑛𝑐𝑒 =

𝟏 𝟏
𝟏 𝟏
𝐕_𝟐(: , ∶ , 𝟏) = 𝟏 𝟏
𝟏 𝟏
[𝟏 𝟏]

There are also no actions for the second state factor:

𝜋 𝑡𝑖𝑚𝑒 =

𝟏 𝟏
𝟏 𝟏
𝐕_𝟐(: , ∶ , 𝟐) = 𝟏 𝟏
𝟏 𝟏
[𝟏 𝟏]

For the third state factor, the agent must wait until the last time point and then either select the
‘report same’ or ‘report different’ actions:

𝜋 𝑟𝑒𝑝𝑜𝑟𝑡 =

𝟏 𝟏
𝟏 𝟏
𝐕_𝟐(: , ∶ , 𝟑) = 𝟏 𝟏
𝟏 𝟏
[𝟐 𝟑]

Lastly, we must provide the agent with preferences that will motivate accurate reporting. For the
first outcome modality (tones), the agent has no preferences (columns = time point, rows [top to
bottom] = high tone, low tone observation):

𝐶 𝑙𝑒𝑣𝑒𝑙 1 𝑡𝑜𝑛𝑒 𝑠𝑡𝑎𝑡𝑒 =

𝟎 𝟎 𝟎 𝟎 𝟎 𝟎
𝐂_𝟐(: , ∶ , 𝟏) = [ ]
𝟎 𝟎 𝟎 𝟎 𝟎 𝟎

103
Active Inference Tutorial

For the second outcome modality (accuracy feedback), the agent prefers to receive ‘correct’
feedback at the last time point (column 6, row 3) and finds ‘incorrect’ feedback to be aversive at the
last time point (column 6, row 2):

𝐶 𝑓𝑒𝑒𝑑𝑏𝑎𝑐𝑘 =

𝟎 𝟎 𝟎 𝟎 𝟎 𝟎
𝐂_𝟐(: , ∶ , 𝟐) = [𝟎 𝟎 𝟎 𝟎 𝟎 −𝟏]
𝟎 𝟎 𝟎 𝟎 𝟎 𝟏

As already mentioned when building the explore-exploit task model, each column in this matrix is
put through a softmax function and then converted into log-probabilities. Having now specified the
second-level model, we will place each of these matrices into its own mdp structure:

𝐦𝐝𝐩. 𝐃 = 𝐃_𝟐

𝐦𝐝𝐩. 𝐝 = 𝐝_𝟐

𝐦𝐝𝐩. 𝐀 = 𝐀_𝟐

𝐦𝐝𝐩. 𝐚 = 𝐚_𝟐

𝐦𝐝𝐩. 𝐁 = 𝐁_𝟐

𝐦𝐝𝐩. 𝐂 = 𝐂_𝟐

𝐦𝐝𝐩. 𝐕 = 𝐕_𝟐

We then need to connect the lower-level model with the higher-level model as follows:

𝐦𝐝𝐩. 𝐌𝐃𝐏 = 𝐌𝐃𝐏_𝟏

Next, we need to provide a matrix specifying which outcome modalities at the second level
(columns) corresponds to which state factors at the lower level (rows) within a ‘link’ field. Here, the
first outcome at the second level (‘tones’) corresponds to the first state factor at the first level:

𝐦𝐝𝐩. 𝐥𝐢𝐧𝐤 = [𝟏 𝟎]

In this case, the matrix only has a single row because there is only one state factor at the lower
level.

Lastly, we need to set the value of the ERP ‘reset’ or ‘decay’ parameter 𝐦𝐝𝐩. 𝐞𝐫𝐩, which at the start
of every epoch of gradient descent is used to reset the posterior over states by dividing the
posterior by the value of the parameter (i.e., higher values = more resetting). Setting the value of
the parameter is entirely up to the discretion of the modeler, depending on assumptions about the
particular neurocognitive process under study. In empirical work, this parameter could be fit to
observed ERP responses. In the experimental paradigm we simulate here, the tones are played to

104
Active Inference Tutorial

the participant in quick succession, so we assume that the posterior at each time step carries over
and does not decay between presentations. As such, we set 𝐦𝐝𝐩. 𝐞𝐫𝐩 = 𝟏. If, however, we were
trying to model a task with longer time periods between updating (e.g., a subject navigating a
maze), some degree of decay could be appropriate (𝐦𝐝𝐩. 𝐞𝐫𝐩 = 𝟒 is the default value in the current
𝐬𝐩𝐦_𝐌𝐃𝐏_𝐕𝐁_𝐗. 𝐦 script).

As before, we can now run this structure through the standard routine to generate simulated
behavioral and neuronal responses of an example trial.

𝐌𝐃𝐏 = 𝐬𝐩𝐦_𝐌𝐃𝐏_𝐕𝐁_𝐗_𝐭𝐮𝐭𝐨𝐫𝐢𝐚𝐥(𝐦𝐝𝐩);

To simulate our two conditions of interest, local deviation + global deviation, and local deviation +
global standard, we will simulate 10 sequential trials for each condition. For brevity, we will not
describe the code that implements this here. Instead, we direct readers to the
Step_by_Step_Hierarchical_Model.m script, which has detailed comments describing each of the
necessary steps. For both conditions, the first nine trials consist of three high tones and a fourth low
tone. On the tenth trial of the local deviation + global standard condition, the trial again consists of
three high tones and a fourth low tone. On the tenth trial of the local deviation + global deviation
condition, the tenth trial instead consists of four high tones, thereby violating the global regularity.
Figure 14 shows plots of second-level belief updating and policy selection, analogous to the single-
level model plots shown in Figure 8. The model performed at ceiling with 100% accuracy when
classifying the last stimulus in the sequence as same or different.

105
Active Inference Tutorial

Figure 14. Simulated beliefs and behavior in the hierarchical POMDP (analogous to the single-level

106
Active Inference Tutorial

model plots show in Figure 8) described in the main task. The top and bottom panels simulate the
two task conditions described in the main text both before and after learning (Trial 1 and Trial 10,
respectively). These simulations demonstrated that the agent performed the task appropriately.
The three panels in the top-left of each plot show posteriors over states at the end of the trial. That
is, the states the model believes it was in at each time point 𝜏 at the last time point 𝑡 (i.e., after
receiving the last observation). Here, time goes from left to right, darker indicates higher
probability, and the cyan dots denote the true states. The top-right panels in each plot show the
action probabilities and true actions. The left-middle panel in each plot just shows the different
possible action-sequences/policies possible in the specified model (encoded numerically from left
to right). A darker color indicates a lower number. The right-middle panel in each plot shows the
progression of posterior beliefs in each policy over time (from left to right, darker = higher
confidence). Policies (rows) line up with the action sequences in the plot on the middle-left. The two
panels in the bottom-left of each plot display the outcomes in cyan dots and the agent’s preference
for each outcome, where darker colors indicate a greater preference (i.e., higher prior probability).
Lastly, the bottom-right plot displays predictions about dopamine responses based on the neuronal
process theory (i.e., encoding changes in expected precision of 𝐸𝐹𝐸; see last row of Table 2 and
Figure 9). In terms of behavior, notice that all models selected the correct actions and received
“correct” feedback, as indicated in the lower outcome plot. These simulations can be reproduced
using the Step_by_Step_Hierarchical_Model.m script included as supplementary code. Note that,
due to random sampling, results may not be identical each time.

The next step is to visualize the resulting simulations in order to make empirical predictions about
behavior and neuronal responses.

6.3 Plotting Hierarchical Models

To visualize the resulting belief updating and simulated neuronal responses for this hierarchical
model, we have provided a modified version of the plotting script provided in the freely available
SPM routines:

𝐌𝐃𝐏 = 𝐬𝐩𝐦_𝐌𝐃𝐏_𝐕𝐁_𝐄𝐑𝐏_𝐭𝐮𝐭𝐨𝐫𝐢𝐚𝐥(𝐌𝐃𝐏);

This plotting script shows the simulated firing rates associated with belief updating at each level. It
also shows the simulated ERPs (summed first derivative of the firing rates) that would be expected
to be generated during the simulated task. Figure 15 shows the first and tenth trial from each
condition (i.e., before and after building up prior beliefs favoring some states over others). Notice
that on the first trial of both the local deviation + global deviation condition and the local deviation +
global standard condition, the high firing rates at the first level reflect a high posterior confidence in
the tone each time it is presented. In contrast, after hearing the first tone, the firing rate is evenly
spread between both the “high” and “high-low” hidden states at the second level, reflecting the
agent’s uncertainty about which type of sequence is being presented (i.e., since both sequences
predict “high” tones for the first three time steps). Note, however, that the fourth tone on the first
trial generates an increased firing rate for one of the hidden states, and the firing rate drastically
decreases for the other state (depending on whether a “high” or “low” tone is presented). By the

107
Active Inference Tutorial

tenth trial, the agent has a high prior expectation that it will experience a “high-low” sequence,
reflected in the high firing rates for this second-level hidden state from early time points. In the
local deviation + global standard condition, this expectation is confirmed, leading to little change in
second-level firing rates. Importantly, however, in the local deviation + global deviation condition,
this expectation is violated, because the agent is presented with four high tones. This creates a
rapid switch in posterior confidence from the “high-low” hidden state to the “high” hidden state at
the fourth time step, generating a very strong and rapid shift in second-level beliefs.

108
Active Inference Tutorial

Figure 15. Simulated ERPs and firing rates extracted from the hierarchical POMDP model of the

109
Active Inference Tutorial

auditory mismatch paradigm during each task condition (top vs. bottom) both before and after
learning (Trial 1 vs. Trial 10). As described in the main text (and illustrated further in Figure 16),
these simulations reproduce ERP results observed in previous empirical studies (e.g., stronger
short-latency ERPs in response to local deviations and stronger longer-latency ERPs in response to
global deviations). The unit response plots show the posterior probability over states (𝑠𝜋,𝜏 ) at each
level of the model (as usual, darker colors = higher posterior = higher firing rates). As described in
the neuronal process theory section, normalized firing rates are generated by passing the
depolarization variable 𝑣𝜋,𝜏 through a softmax function 𝑠𝜋,𝜏 = 𝜎(𝑣𝜋,𝜏 ). The ERP plots show the rate
of change (first derivative) of posterior beliefs over states summed over all states at each level of
the model (analogous to the aggregate signal measured at the level of the scalp by EEG). Note that
each increment of 0.5 along the x-axis corresponds to a trial at the lower level, and to a time point at
the higher level (i.e., 6 time points in a higher-level trial, with 6 corresponding lower-level trials).
For a detailed description of each plot and its meaning in relation to the task, see the main text.
These simulations can be reproduced using the Step_by_Step_Hierarchical_Model.m script
included as supplementary code. Note that, due to random sampling, results may not be identical
each time.

This brings us to simulated ERPs, which reflect the rate at which posterior beliefs change within
each epoch of belief updating, summed over hidden states at each level of the model (i.e., similar to
the aggregate signal measured by EEG). At the first level, the repeated presentation of high tones
generates small ERPs, whereas deviations from this pattern at the fourth time step (local
deviations) generate a larger amplitude mismatch response because posterior beliefs about the
tone change rapidly. Importantly, when the local deviation response is subtracted from the local
standard response, we can reproduce the classic mismatch negativity effect (MMN; see Figure 16).
At the second level, the repeated occurrence of three high tones and one low tone for the first nine
trials creates a strong prior expectation (through the increase in concentration parameters in the 𝐷
vector) for the “high-low” sequence state. When the model is unexpectedly presented with four
high tones on the tenth trial, this expectation violation generates a rapid change in beliefs and a
correspondingly large second-level ERP resembling the P300 (see Figures 15 and 16). Figure 16
shows custom-made ERP plots, which isolate the contrasts of interest. Again, for the sake of brevity,
we will not describe the code that generates this plot in the main text, but direct interested readers
to the Step_by_Step_Hierarchical_Model.m script for more details.

110
Active Inference Tutorial

Figure 16. Custom ERP plots generated from the simulated auditory mismatch paradigm. On the
left we have the “raw” first- and second-level ERP waveforms centered on the fourth time step of
the tenth trial from each condition. The right side of the figure shows how the subtraction of the
deviant trials from the standard trials reproduces both the MMN and P300. At both the first and
second level, “deviant” ERPs have a substantially larger amplitude than “standard” ERPs. Note that,
although the relative differences in timing between simulated ERPs (and the relative differences in
timing between ERPs at different levels of the model) are meaningful, the units we ascribe to time
are (usually) somewhat arbitrary. So, for clarity, we have not included any units of time on the x-
axis. These simulations can be reproduced using the Step_by_Step_Hierarchical_Model.m script
included as supplementary code. Note that, due to random sampling, results may not be identical
each time.

7. Fitting Models to Behavior

So far, we have focused on simulating behavior and establishing the face validity of active inference
using canonical examples from the decision-making and electrophysiological literature. In this
section, we demonstrate how active inference models can be used in empirical studies. More
specifically, we will describe how one can estimate the model parameter values that best explain
participant behavior during an experimental task. This approach has been employed in several
recent studies that have used active inference models to account for behavior during tasks designed
to study a wide range of phenomena – such as attention, risk-taking, approach-avoidance conflict,

111
Active Inference Tutorial

explore-exploit behavior, and interoception (Mirza, Adams, Mathys, & Friston, 2018;
Schwartenbeck et al., 2015; Smith, Kirlic, Stewart, Touthang, Kuplicki, McDermott, et al., 2021;
Smith, Kirlic, Stewart, Touthang, Kuplicki, Khalsa, et al., 2021; Smith, Kuplicki, Feinstein, et al., 2020;
Smith, Kuplicki, Teed, et al., 2020; Smith, Mayeli, et al., 2021; Smith, Schwartenbeck, Stewart, et al.,
2020). In each of these studies, a model was used to evaluate the prior beliefs that participants
most likely held when performing a task (i.e., the prior beliefs that would have generated their
behavior in a model). All necessary steps for carrying out this approach are described below, with
the goal of preparing the reader to use active inference models in their own empirical studies. By
the end of this section, the reader should understand how to fit a model to participant behavior,
how to perform a number of diagnostic checks to ensure the validity of parameter estimates, and
how to use parameter estimates within group-level Bayesian models.

As stated above, empirical applications of active inference require fitting a task model to participant
behavior. When fitting a model to behavior, one would like to find the parameters that maximize
the posterior probability of a model given that behavior, 𝑝(𝑚𝑜𝑑𝑒𝑙|𝑝𝑎𝑟𝑡𝑖𝑐𝑖𝑝𝑎𝑛𝑡 𝑏𝑒ℎ𝑎𝑣𝑖𝑜𝑟).
Assuming a flat prior belief over models, this posterior is proportional to the likelihood term,
𝑝(𝑝𝑎𝑟𝑡𝑖𝑐𝑖𝑝𝑎𝑛𝑡 𝑏𝑒ℎ𝑎𝑣𝑖𝑜𝑟|𝑚𝑜𝑑𝑒𝑙). Thus, many fitting approaches (estimation algorithms) try to find
the parameters that maximize this likelihood – referred to as maximum likelihood estimation (MLE).
In some cases, one might also have reason to expect that certain models are more likely than others
a priori. In this case, one can also incorporate an informative prior belief over models, 𝑝(𝑚𝑜𝑑𝑒𝑙)10.
Regardless of the specific approach, the goal of any fitting procedure is to find the set of parameters
that would best reproduce/predict the actual behavior of a participant (i.e., with the highest
probability) when running simulations using a model (while in some cases also incorporating any
prior knowledge one might have).

To do this, one needs to feed the trial-by-trial observations made by participants (e.g., cues,
wins/losses, etc.) into the model and look at the actions predicted by the model (i.e., posterior
probabilities over actions). One can then compare these predictions to the actions a participant
actually chose. Under some sets of parameter values (e.g., prior expectations, precisions, etc.), the
model’s predictions may not match behavior well (i.e., the probability of a participant’s actions
under the model may be low). However, by searching through different possible combinations of
parameter values, the best combination can be found for a given participant. Note, however, that
this will only be the best combination possible for that model. This does not mean that the model
has high explanatory power (e.g., the best parameter combination for one model might lead to an
average action probability of 0.4, while another model might reach 0.7, etc.). This is why it is also
important to compare the explanatory power of different models.

Throughout this section, we encourage the reader to follow along in the companion MATLAB script
(Step_by_Step_AI_Guide.m). This can be found in the supplementary code files, as well as at:
[Link] At the top of this script, if you set

10
An example of this approach is maximum a posteriori (MAP) estimation, in which an algorithm tries to find the
parameter values (i.e., point estimates) that maximize the value of the posterior as opposed to the likelihood.

112
Active Inference Tutorial

𝑺𝒊𝒎 = 𝟒 it will perform parameter estimation on a single set of simulated behavioral data from the
explore-exploit task model (see the top panels and bottom-left panel of Figure 17 for example
outputs). If you set 𝑺𝒊𝒎 = 𝟓 it will simulate behavioral data from two models (one with two
parameters and one with three parameters; see below) for a few synthetic participants and then
perform model comparison. It will also assess how well the estimated parameter values match with
the true parameter values used to generate the simulated task behavior (i.e., it will output
correlation matrices and associated p-values; bottom-middle panel of Figure 17). This is important
to check (as described below in relation to parameter recoverability) to ensure that parameters can
be estimated reliably (this is sometimes referred to as model identifiability). All the code for
performing these steps is included and described in the script comments. All examples here will be
based on the explore-exploit task model, when simulating behavior during the case of reversal
learning used above (see Figure 11). We will estimate the alpha parameter (𝛼) encoding action
precision (i.e., inverse temperature) as well as a risk-seeking parameter (described below). For
model comparison, we will also estimate learning rate (𝜂) in a second model.

As mentioned above, finding the optimal combination of parameter values to account for a
participant’s behavior requires a model fitting (parameter estimation) procedure. There are many
different procedures (estimation algorithms) that are available, each with their strengths and
limitations. The simplest approach is called a grid search, where each possible combination of
parameter values (within some specified range of values) is tried one-by-one, and then the one that
best reproduces (e.g., maximizes the likelihood of) behavior is chosen. However, this approach is
limited to fairly simple models with a small number of parameters, and it can also lead to
overfitting (i.e., finding parameter values that reproduce random aspects of behavior). In more
complex models with a larger number of parameters, exhaustive search of the parameter space
becomes intractable. Thus, a number of other algorithms have been developed. In this section, we
will focus on an estimation technique called variational Bayes (i.e., a specific variant of this
approach called variational Laplace; (Friston, Mattout, Trujillo-Barreto, Ashburner, & Penny,
2007)), which is based on exactly the same free energy minimization ideas described above11.
Namely, variational Bayes starts with a prior distribution over parameter values, which acts as the
starting value of an approximate posterior (or ‘proposal’) distribution. Via gradient descent on
variational free energy, this approximate posterior is adjusted until convergence to a stable
minimum value, providing posterior estimates of model parameter values. However, the detailed
mathematics underlying this technique are quite advanced and beyond the scope of this tutorial.
Therefore, our focus here is on practical applications, with the aim of equipping readers without
advanced mathematical training to still be able to make use of such approaches appropriately. We
will now go over a set of concrete steps that can be taken to use variational Bayes for model fitting.

11
Another important class of estimation algorithms uses Markov Chain Monte Carlo (MCMC) methods. These
methods involve sequentially sampling from a distribution according to specific sets of rules that try to find
locations under that distribution with high probability. In this case, the probability of a participant’s actions would
be evaluated under a sequence of parameter value combinations (with an initially random starting point) until a
sufficient approximation to the true distribution was generated.

113
Active Inference Tutorial

As we are focusing on the explore-exploit task model, it may help to glance back at the model/task
description in earlier sections before moving on.

The first step is to place trial-by trial-observations and participant actions into an 𝑚𝑑𝑝 structure. In
the explore-exploit task model, for each trial this would mean specifying the observations (𝑚𝑑𝑝. 𝑜)
a participant made for the three outcome modalities at the three time points in each trial, and then
specifying the two actions taken (𝑚𝑑𝑝. 𝑢). For example, if on trial #1 the participant chose to take
the hint and then chose the left machine, this would be:

𝟏 𝟏
𝒎𝒅𝒑(𝟏). 𝒖 = [ ]
𝟐 𝟑

𝟏 𝟐 𝟏
𝒎𝒅𝒑(𝟏). 𝒐 = [𝟏 𝟏 𝟑]
𝟏 𝟐 𝟑

Here, time goes from left to right. For 𝑚𝑑𝑝. 𝑢, actions for each state factor correspond to each row
from top to bottom. For 𝑚𝑑𝑝. 𝑜, time also goes from left to right and rows correspond to outcome
modalities. The parenthetical “(1)” for the 𝑚𝑑𝑝 denotes trial #1. Here, for 𝑚𝑑𝑝. 𝑢, remember there
is only one “action” for the first state factor (i.e., the participant does not have control of which
machine will most likely lead to a win). So, the top row is simply a 1 for both time points. For the
second state factor (choice), the participant first chose the hint (action #2) and then chose the left
machine (action #3). For 𝑚𝑑𝑝. 𝑜, the first row indicates “null” (observation 1), “left hint”
(observation #2), and then back to “null”. The second row indicates two “null” observations
followed by a “win” (observation #3). The last row simply corresponds to the choices (“start”, “take
hint”, “choose left”). Observations and behavior for each trial need to be inserted in this same way
(e.g., 𝑚𝑑𝑝(2), 𝑚𝑑𝑝(3), etc. until the final trial number).

The second step is to choose the model parameters you want to estimate and which you want to
hold fixed. Ultimately, one may want to try estimating different models and/or different numbers of
parameters and then compare them to find which best accounts for behavior, as we describe
further below. In this example, we will first consider estimating two parameters: learning rate (𝜂)
and “risk-seeking” (𝑅𝑆). The latter corresponds to how strong the preference is to win money in the
𝐂 matrix:

𝐶 𝑤𝑖𝑛 =

𝟎 𝟎 𝟎

𝐂{𝟐} = 𝟎 −𝟏 −𝟏
𝑹𝑺
[𝟎 𝑹𝑺 𝟐 ]

As reviewed above, learning rate scales the size of the ‘count’ that is added to the (Dirichlet)
concentration parameters after a new observation. The 𝑅𝑆 parameter controls the explore-exploit
trade-off. In the simulations shown in Figure 10 and Figure 11, we saw that, as 𝑅𝑆 values go up,

114
Active Inference Tutorial

the probability that a participant will take the hint goes down. That is, they will tend to ‘risk it’ and
simply guess left or right in hopes of winning the larger amount of money.

When using variational Bayes, the next step is to choose a set of estimation priors. These are not
the prior beliefs of a participant, but the initial parameter values that are evaluated during model
fitting. Estimation priors include both a prior mean and a prior variance, each of which can be set
within the supplementary “Estimate_parameters.m” script, as described further below. A simple
way of thinking about the variational Bayes algorithm is that it starts from the chosen estimation
priors (i.e., the prior means) and slowly moves away from them to find the combination of
parameter values that best explains a participant’s behavior, while also balancing the associated
cost of increased model complexity that is incurred when the parameters move too far away from
the prior means. More specifically, variational Bayes is accomplished by performing a gradient
descent on 𝑉𝐹𝐸 (i.e., the same process active inference models use to accomplish perception,
learning, and action selection). In this case, the gradient descent starts with the prior means, and
then evaluates the log-probability of a participant’s actions (e.g., sequence of choices). This log-
probability is evaluated using the posterior beliefs about action that the subject would have had,
given the priors in question and the outcomes they observed. The scheme then evaluates
neighboring parameter values and continues in the direction of increasing likelihood, until a
combination is found with no neighboring values that improve the fit.

However, because we seek a 𝑉𝐹𝐸 minimum, which includes both complexity and accuracy terms,
the algorithm will not simply maximize model accuracy. It will also try to minimize how far
parameter estimates move away from the mean values of the estimation priors. In other words,
parameter estimates for a given participant will represent values that accurately reproduce that
participant’s behavior while moving as little as possible from the prior values chosen by the
experimenter. Importantly, complexity minimization also depends on the prior variances that are
chosen. Setting a small prior variance will lend strong weight to complexity minimization; in
contrast, setting a large variance for estimation priors will lead a stronger weight to be placed on
maximizing accuracy (but with a greater chance for overfitting). How to set the prior variance can
be a crucial choice in variational Bayes, because posterior parameter estimates can in some cases
be sensitive to the choice of prior means. When available, prior means can be based on previous
studies. However, if previous literature is not available to draw from, one might consider specifying
a large prior variance to reflect this uncertainty (in which case the complexity cost will play little-
to-no role). This issue of setting appropriate estimation priors, and diagnosing when they may be
inappropriate, is one of several important model checking procedures that should be taken to
confirm the validity of parameter estimates, which we return to below.

Before moving forward, however, it is important to highlight a specific subtlety in the above
scheme. Namely, we are using variational Bayes to estimate the parameters that underwrite
variational inference within the brain of a study participant. In other words, there are two
generative models: first, a subjective model, which is the POMDP we assume the participant is
using to plan their responses. Second, there is an objective model that the experimenter specifies
in terms of the estimation priors on parameters of the subjective model. A key conceptual point
here is that the parameters of the subjective model can always be interpreted as priors; either

115
Active Inference Tutorial

explicitly or implicitly in terms of the structure or form of the subjective model. This is important
because of something called the complete class theorem (L. D. Brown, 1981; Wald, 1947). This
theorem says that for any pair of reward functions (i.e., preferences) and choice behavior, there
exists some prior beliefs that render the choices Bayes optimal. This means that any behavior can
be described, under ideal Bayesian assumptions, given the right set of prior beliefs. It is these prior
beliefs that provide a theoretically complete characterization of any given participant, in any given
experimental paradigm.

After choosing a final set of estimation priors, and obtaining parameter estimates for each
participant, it is also important to confirm what is called parameter recoverability (also
sometimes referred to as testing whether the model is identifiable or invertible). What this means is
that you need to be sure that, if simulated behavior were generated from a model with a given set of
parameter values, that the estimation algorithm would provide reasonably precise estimates that
approach the parameter values that generated the behavior. For example, assume a given
participant’s posterior parameter estimates were 𝑅𝑆 = 3.2 and 𝛼 = 3.7 (e.g., using estimation
priors of 𝑅𝑆 = 4 and 𝛼 = 2). Assessing recoverability would require: 1) putting the values of these
posterior parameter estimates into the task model, 2) generating simulated behavior using those
values, 3) feeding that simulated behavior into the fitting algorithm, and 4) examining how similar
the resulting parameter estimates are to the parameter values that you used to generate the
simulated behavior in the first place. If the algorithm returns estimates that move in the right
direction from the estimation priors and approach the values that generated the data (e.g., 𝑅𝑆 = 3.3
and 𝛼 = 3.5), then we can be more confident that the participant’s estimates are capturing
something meaningful and reliable about their decision process. The plot on the top-right of Figure
17 illustrates a case in which posterior estimates move from estimation priors toward the true
values (this plot can be reproduced by setting 𝑺𝒊𝒎 = 𝟒 when you run the
𝐒𝐭𝐞𝐩_𝐛𝐲_𝐒𝐭𝐞𝐩_𝐀𝐈_𝐆𝐮𝐢𝐝𝐞. 𝐦 code). If the fitting algorithm instead returned estimates that were
farther away and moved in the wrong direction from the estimation priors (e.g., 𝑅𝑆 = 4.9 and 𝛼 =
1.3), this would suggest that the parameter estimates for that participant may not be reliable (e.g., a
different combination of parameter values might reproduce their behavior equally well).

Thus, before interpreting parameter estimates in real participants – and using them in subsequent
group analyses – it is important to confirm in simulated data that the estimates provided by the
fitting algorithm match well with (e.g., are significantly correlated with) the parameters used to
generate the simulated behavior in the first place. This should be checked using the combinations of
parameter estimates found when fitting a model to the true participant data in a study. For
example, if you had three participants with estimated parameter values of [𝑅𝑆, 𝛼] =
[2.8, 4.1], [3.5, 6.2], [3.8, 8.1], then these three combinations could be used to generate simulated
behavior, and this simulated behavior could then be fed into the fitting algorithm to see if it
returned results similar to the generative values (e.g., if estimated values were highly correlated
with those used to generate the simulated behavior).

It is important to keep in mind, however, that the products of Bayesian model inversion (here,
parameter estimation) will, generally speaking, never be the same as the parameters used to
generate data. This is because there is usually a simpler way of generating any given set of data,

116
Active Inference Tutorial

which the model inversion will identify. In other words, in a certain sense there are no ‘true’
parameters – only the best explanation in the sense of Occam's principle.

If the 𝑺𝒊𝒎 variable is set to 𝑺𝒊𝒎 = 𝟓 when you run the 𝐒𝐭𝐞𝐩_𝐛𝐲_𝐒𝐭𝐞𝐩_𝐀𝐈_𝐆𝐮𝐢𝐝𝐞. 𝐦 code, simulated
behavior will be generated under several parameter values for the explore-exploit task model and it
will run correlations between the estimated parameter values and the parameter values that
actually generated the simulated behavior. In a representative example simulation, recoverability
appeared high for 𝑅𝑆 (r = .95) and 𝛼 (r = .94). In a model that also included learning rate (𝜂), this
parameter appeared recoverable as well (r = .75). The bottom-middle panel of Figure 17 shows a
scatterplot illustrating the correlation between generative and estimated parameter values for 𝛼,
which the above-mentioned code will reproduce.

Before assessing parameter recoverability (i.e., model identifiability), however, it is important to

first understand more about the concrete steps for fitting. As mentioned above, if you set 𝑺𝒊𝒎 = 𝟒
in the 𝐒𝐭𝐞𝐩_𝐛𝐲_𝐒𝐭𝐞𝐩_𝐀𝐈_𝐆𝐮𝐢𝐝𝐞. 𝐦 script, it will generate simulated explore-exploit task behavior
for a single participant. It will feed this simulated behavior into a ‘DCM’ (for Dynamic Causal
Modeling) data structure in the appropriate format for model fitting. It then calls the
supplementary 𝐄𝐬𝐭𝐢𝐦𝐚𝐭𝐞_𝐩𝐚𝐫𝐚𝐦𝐞𝐭𝐞𝐫𝐬. 𝐦 script we have provided, which takes the DCM structure
as input, runs the variational Bayes routine in SPM12 (𝐬𝐩𝐦_𝐧𝐥𝐬𝐢_𝐍𝐞𝐰𝐭𝐨𝐧. 𝐦), and calculates the
log-likelihood – that is, the sum of the log-probabilities of chosen actions under the model. In our
script, the DCM structure includes the following:

𝐃𝐂𝐌. 𝐌𝐃𝐏 (generative model)

𝐃𝐂𝐌. 𝐔 (participant observations)

𝐃𝐂𝐌. 𝐘 (participant actions)

𝐃𝐂𝐌. 𝐟𝐢𝐞𝐥𝐝 (specifies the parameters to be fit)

Within the Estimate_parameters.m file, you will see locations (starting on line 47) where the
script searches for the to-be-fit parameter names. Here, you can enter the estimation priors (means
and variances) for each parameter. Note that some parameters (e.g., learning rate) need to be
between 0 – 1. For this reason, they are here transformed into logit-space so that the estimation
routine does not assess values outside of that range. Similarly, parameters that can only take on
positive values can be transformed into log-space to preclude negative values. For illustration, we
have assumed that 𝑅𝑆 can only take positive values and is therefore log-transformed. Farther down
in the script (starting on line 133), these values are re-transformed out of logit- or log-space when
they are fed into the model and output as final estimates. Starting on line 188, the log-likelihood
function loops through each trial, takes the probability of a participant’s actions in the model
(stored in 𝐌𝐃𝐏. 𝐏; see Table 3), log-transforms it, and then sums the log-probabilities. The closer
this value is to 0, the better the model fits the behavior.

When you run the 𝐒𝐭𝐞𝐩_𝐛𝐲_𝐒𝐭𝐞𝐩_𝐀𝐈_𝐆𝐮𝐢𝐝𝐞. 𝐦 script (which calls the others automatically) with
𝑺𝒊𝒎 = 𝟒, a display will also appear (as in the top-left panel of Figure 17) that shows how the free

117
Active Inference Tutorial

energy changes over iterations. Note that these values increase because they are negative and will
approach 0 as model fit improves. If these values steadily increase, this suggests that fitting is
converging to a reliable estimate. If they instead fluctuate up and down inconsistently, this could
suggest one or more problems that need to be addressed before model estimates are considered
reliable (e.g., poor choice of estimation priors, problems with parameter values remaining in valid
ranges, estimates getting stuck in locally [but not globally] optimal values). A failure to converge is
usually read as a useful diagnostic that the estimation priors are somehow mis-specified. In other
words, if model inversion does not converge gracefully within a few tens of iterations, you may
want to think about whether your priors are appropriate – or whether you are trying to fit too
many parameters to rather sparse data. This type of model checking is crucial to ensure that
parameter estimates are valid and informative.

The estimates resulting from initially chosen prior means can also sometimes offer guidance in this
regard. For example, if estimates for all participants tend to move far from the chosen prior means
in the same direction, this could suggest that these estimation priors were poorly chosen. If you
initially set a prior mean of 𝑅𝑆 = 1, for example, and then notice that estimates for all participants
tend to move up to between 3 and 5, this could indicate that a prior of 𝑅𝑆 = 4 would be more
appropriate (as it appears to be a better prior for the group as a whole) and might help prevent
over-weighted complexity costs that could hinder identification of individual differences. It is worth
noting that letting data guide the way one chooses estimation priors in this way could be viewed as
suspect under some interpretations of Bayesian statistics. However, it is also possible to view
estimation priors in variational Bayes as simply being starting values for estimation that can be
optimized (and may need to be in cases where some starting values will lead gradient descent to get
stuck in suboptimal local minima). Note also that, although we do not go into detail here, there are
also ‘empirical Bayes’ methods that can be used to rigorously estimate group-level priors and then
re-estimate individual-level parameters based on those group-level priors (Carlin & Louis, 1998;
Friston, Litvak, et al., 2016).

After convergence to the best estimates, the output 𝐃𝐂𝐌 structure will contain additional fields.
The following are relevant to our current use:

𝐃𝐂𝐌. 𝐄𝐩: posterior mean estimates (i.e., expectations) for each parameter.

𝐃𝐂𝐌. 𝐂𝐩: posterior covariance matrix (with posterior parameter variances on the diagonal).

𝐃𝐂𝐌. 𝐅: the final free energy value of the best fit model.

The free energy values for each participant will later be used for model comparison. The covariance
values (i.e., the off-diagonals in 𝐃𝐂𝐌. 𝐂𝐩) should be checked to make sure they are not too high
(e.g., > .8), or it would suggest they were not independently estimated and that each estimate may
not carry unique/reliable information. This is the same kind of check you would apply to an
experimental design – to ensure explanatory variables are orthogonal and the model parameters
can be estimated efficiently. The bottom-left panel of Figure 17 plots an example co-variance

118
Active Inference Tutorial

matrix (which will be reproduced by the accompanying tutorial scripts; 𝑺𝒊𝒎 = 𝟒, as with previous
plots).

When testing hypotheses about which of several models best explains your data, one must fit each
model and then compare how well they each fit behavior. For Bayesian model comparison using
variational Bayes, this means comparing the free energies at the group level. One common
approach is to use a random effects model, which can be done using the 𝐬𝐩𝐦_𝐁𝐌𝐒. 𝐦 function
(available within SPM12). This function takes as input a matrix containing the free energies for each
participant for each model (one column per model). For the details of this implementation of
Bayesian model selection, see (Rigoux, Stephan, Friston, & Daunizeau, 2014; K. Stephan, Penny,
Daunizeau, Moran, & Friston, 2009). An important output of this function is the protected
exceedance probability (𝒑𝒙𝒑) of each model. In this case, the model with the highest 𝒑𝒙𝒑 has the
most evidence. Often there will be a clear winner, with 𝑝𝑥𝑝 = 1 for a single model and 0s for the
others. In cases where there is no clear winner (e.g., 𝑝𝑥𝑝 = [.48 .52]), this will be important to note,
as it may reflect insufficient evidence for a “best” model. If parameters in both models are
recoverable, one may wish to consider the parameters of each model in further between-subjects
analyses (e.g., parameters in one model may have higher explanatory value for specific theoretical
questions). As mentioned above, setting 𝑺𝒊𝒎 = 𝟓 in the 𝐒𝐭𝐞𝐩_𝐛𝐲_𝐒𝐭𝐞𝐩_𝐀𝐈_𝐆𝐮𝐢𝐝𝐞. 𝐦 script will
generate/estimate behavior for a few simulated participants for models with and without the
learning rate parameter. It will then do Bayesian model comparison (this will take several minutes).
This will output the 𝑝𝑥𝑝 for each model, as well as the model probabilities and a few other
diagnostic outputs we will not cover in detail here; for further information, see (Rigoux et al., 2014;
K. Stephan et al., 2009) as well as the documentation within the 𝐬𝐩𝐦_𝐁𝐌𝐒. 𝐦 function. As one
example, you could input the free energies for the 2-parameter (𝐅_𝟐_𝐩𝐚𝐫𝐚𝐦𝐬) and 3-parameter
(𝐅_𝟑_𝐩𝐚𝐫𝐚𝐦𝐬) models as follows:

[alpha,exp_r,xp,pxp,bor] = spm_BMS([F_2_params F_3_params])

If the output were 𝑝𝑥𝑝 = [1 0], this would mean that the probability is equal to 1 that the 2-
parameter model is a better fit than the 3-parameter model. In a representative example
simulation, the 𝑝𝑥𝑝 = [.37 .63], weakly favoring the 3-parameter model.

Once (recoverable) parameters for a winning model have been estimated, one simple approach for
group-level analysis would be to analyze the posterior parameter means across participants using
standard frequentist analyses. However, one advantage of using variational Bayes is that it also
provides information about posterior parameter variances for each individual (i.e., as opposed to
only the posterior means). This allows for between-subject Bayesian analyses that take the
variances into account. One approach that takes the output 𝐃𝐂𝐌 structures for each participant as
input is parametric empirical Bayes (PEB), described in detail in (Friston, Litvak, et al., 2016;
Zeidman et al., 2019). PEB uses a general linear model and effectively down-weights the
contribution of individual subject parameter estimates when those estimates have larger posterior
variances (i.e., those with greater uncertainty). PEB can be run using the 𝐬𝐩𝐦_𝐝𝐜𝐦_𝐩𝐞𝐛. 𝐦 and
𝐬𝐩𝐦_𝐝𝐜𝐦_𝐩𝐞𝐛_𝐛𝐦𝐜. 𝐦 functions. These functions estimate and compare group models,
respectively. In short, they allow a test of whether there is evidence for a model that does include

119
Active Inference Tutorial

group differences or whether there is more evidence for a model that does not include those
differences. When including covariates (e.g., age), it also allows comparing the evidence for or
against a relationship between parameters and those covariates. The 𝐬𝐩𝐦_𝐝𝐜𝐦_𝐩𝐞𝐛_𝐫𝐞𝐯𝐢𝐞𝐰. 𝐦
function can be used to inspect the results returned from the PEB scripts.

Example code to implement such empirical Bayesian (random effects) analyses is also included in
the accompanying MATLAB code 𝐒𝐭𝐞𝐩_𝐛𝐲_𝐒𝐭𝐞𝐩_𝐀𝐈_𝐆𝐮𝐢𝐝𝐞. 𝐦. The code shows examples about
(with commented descriptions of) how to set up inputs to PEB. In our example, the code sets
several parameters to default values (for more information on these, see (Friston, Litvak, et al.,
2016; Zeidman et al., 2019)) and then inputs a matrix (𝑴. 𝑿) for a general linear model, with a
column for the mean, a column separating participants into two groups (here, with low vs. high
learning rates), and a column with randomly generated participant age values). This will allow us to
assess the evidence for models including effects of group and/or age (and the strength of these
effects).

To run PEB, set 𝑺𝒊𝒎 = 𝟓 and set 𝑷𝑬𝑩 = 𝟏. This may take some time, as it will first generate and
estimate parameters for several (six) simulated participants before feeding them into the PEB
scripts (note that the script will save the outputs of 𝑺𝒊𝒎 = 𝟓 so that this does not need to be
repeated each time you want to practice using PEB). In beginning of section 10 of the script, you can
also specify whether you want to use PEB on the 2-parameter or 3-parameter model. When
complete, some output plots and the PEB results viewer window will appear. Example outputs are
shown in the bottom-right panel of Figure 17). The figure legend describes how to interpret these
outputs.

To review, we have covered several steps for model fitting, model checking, and subsequent
analysis. These steps are as follows:

1. Select estimation priors, including prior means and variances.

2. Run variational Bayes on participant data for each model under consideration.
a. Check that gradient descent shows smooth convergence toward a free energy
minimum (Figure 17, top-left panel). Note that this is displayed as maximization
because negative free energies are used.
3. Perform Bayesian model comparison to confirm the best model.
4. Generate simulated behavior in the best model using the parameter values estimated in
your participants, and then run variational Bayes to estimate parameters using this
simulated behavior.
a. Confirm that posterior estimates approach the parameter values used to generate
the simulated behavior (Figure 17, top-right panel).
b. Confirm that posterior co-variances are not high (Figure 17, bottom-left panel).
c. Confirm that generative and estimated parameter values are strongly correlated
(Figure 17, bottom-middle panel).
5. Use PEB to run regression or group comparison analyses on posterior means and variances
at the between-subjects level (Figure 17, bottom-right panel).

120
Active Inference Tutorial

Figure 17. Illustration of different steps in model fitting, model checking, and subsequent analysis
methods. Top Left: Single-subject parameter estimates generated by the scripts provided with this
tutorial. This example includes estimation of two parameters: alpha (action precision) and 𝑅𝑆 (risk-
seeking). The left subplot shows the trajectory of parameter values as fitting progressed (from red
to blue). The middle subplot shows how evidence for the model (given participant data) increased
during estimation (via gradient descent on free energy; here shown as ascent on negative free
energy). The right subplot depicts how parameters moved from estimation priors to posterior
values (pink lines indicate 95% Bayesian confidence intervals). Note that, for clarity, prior values
for all parameters in these plots are given a common reference value of 0, such that posterior values
are shown as deviations from those priors. For example, the prior values for alpha and 𝑅𝑆 were 16
and 5, respectively. The plots therefore show that posteriors for alpha deviated to a value below 16
and that those for 𝑅𝑆 deviated to a value above 5. Top Right: Prior and posterior mean estimates
for alpha and 𝑅𝑆 for a single simulated participant. This shows how posterior estimates for both
parameters move toward the true parameter values. Bottom Left: Posterior variances and co-
variances for parameters at the single-subject level, here showing that both variances and co-
variances were low. These results can be reproduced by running the 𝑺𝒊𝒎 = 𝟒 option in the
supplementary 𝐒𝐭𝐞𝐩_𝐛𝐲_𝐒𝐭𝐞𝐩_𝐀𝐈_𝐆𝐮𝐢𝐝𝐞. 𝐦 code. Bottom Middle: Example of a recoverability
analysis for alpha in 6 simulated participants. This plot shows the (in this case strong) correlation
between alpha values estimated from simulated behavior in each participant and the true
parameter values used to generate that simulated behavior. Bottom Right: Output of group-level
parametric empirical Bayes’ (PEB) analyses (scripts provided in supplementary code). This
example included estimation of three parameters for six simulated participants, where parameters

121
Active Inference Tutorial

1, 2, and 3 correspond to alpha (action precision), 𝑅𝑆 (risk-seeking), and eta (learning rate),
respectively. The top row (left to right) shows the evidence (for each parameter) for differences
from 0, differences between two groups (simulated as having different 𝑅𝑆 values, i.e., parameter 2),
and relationships to (arbitrarily specified) age values in models assuming effects for all parameters.
The bottom row shows analogous results for reduced models that have greater evidence than the
full models (here, no group difference in alpha [parameter 1] or eta [parameter 3], and some effects
of age on alpha and eta). Recall that these results are based on a very small sample of only 6
simulated participants, which will generally be unreliable. They are for illustration only, and the
identified relationships with the arbitrary ‘age’ values should not be taken seriously. Note that we
have omitted some additional plots that are also generated. This is because these are the automatic
output of scripts originally designed for dynamic causal modelling in neuroimaging, and not all of
them are useful in the present context. These results can be reproduced by running the 𝑺𝒊𝒎 = 𝟓
option in the supplementary 𝐒𝐭𝐞𝐩_𝐛𝐲_𝐒𝐭𝐞𝐩_𝐀𝐈_𝐆𝐮𝐢𝐝𝐞. 𝐦 code, while also setting the option 𝑷𝑬𝑩 =
𝟏 (note that, because outcomes are sampled from probability distributions, results will not be
identical each time).

8. Concluding Remarks

This concludes the tutorial. For readers seeking more hands-on practice, we have also provided
pencil-and-paper exercises in Appendix 2 (as well as solutions to check your work; see
𝐏𝐞𝐧𝐜𝐢𝐥_𝐚𝐧𝐝_𝐩𝐚𝐩𝐞𝐫_𝐞𝐱𝐞𝐫𝐜𝐢𝐬𝐞_𝐬𝐨𝐥𝐮𝐭𝐢𝐨𝐧𝐬. 𝐦 code). In our experience, doing a few practice problems
of this sort, and working with the code, are the best way to gain useful intuitions about the
dynamics of these models and how they can be tailored for specific studies. We note that, while we
have strived to be comprehensive, there are many new directions in active inference (and
associated functionality in the standard SPM routines) incorporating, for example, multi-agent
interactions, deep parametric models (e.g., 2nd-order parameters on habits, preferences, precisions,
etc.), and mixed models that link POMDPs to continuous motor control processes, among others
(e.g., see (Friston, Parr, et al., 2017; Hesp et al., 2020)). We hope that working through the materials
provided here will offer a “launching point” that will provide the reader with a sufficient foundation
to independently extend their work with advances in the field as they emerge.

Acknowledgments: The authors would like to thank Lance Da Costa, Thomas Parr, Giuseppe
Pagnoni, and Conor Heins for offering useful comments and suggestions during preparation of the
manuscript. The authors would also like to acknowledge Samuel Taylor for additional suggestions
that improved the clarity of the manuscript.

Conflict of Interest: The authors have no conflicts of interest to disclose.

Funding: R. S. is supported by the William K. Warren Foundation, the Stewart G. Wolf Fellowship,
and the National Institute of General Medical Sciences (P20GM121312). C. W. is supported by the
University of Cambridge Harding Distinguished Postgraduate Scholars Programme.

122
Active Inference Tutorial

Appendices

1. Additional mathematical details

Introduction to the rules of probability

There are two basic rules in the mathematics of probability that will underpin much of the material
covered in this tutorial. The first rule is called the sum rule:

𝑝(𝑎) = 𝑝(𝑎, 𝑏) + 𝑝(𝑎, ~𝑏) (A.1)

Here we use the tilde (~) symbol to indicate negation. So this states that the probability of 𝑎 being
true, 𝑝(𝑎), is equal to the joint probability that both 𝑎 and 𝑏 are true, 𝑝(𝑎, 𝑏), plus the joint
probability that 𝑎 is true and 𝑏 is false, 𝑝(𝑎, ~𝑏). If the variable 𝑏 can take on several values (i.e.,
more than just true and false), then calculating 𝑝(𝑎) requires summing 𝑝(𝑎, 𝑏) for each possible
value of 𝑏. Note that 𝑝(𝑎) is often referred to as a marginal probability in this context when
calculated by summing over all the values of another variable in a joint probability distribution. In
the context of Bayesian inference, it is also sometimes referred to as a prior probability when used
to describe beliefs about 𝑎 before making a new observation (described further below).

The second rule is called the product rule:

𝑝(𝑎|𝑏)𝑝(𝑏) = 𝑝(𝑎, 𝑏) = 𝑝(𝑏|𝑎)𝑝(𝑎) (A.2)

This says that the joint probability of 𝑎 and 𝑏 is equivalent to the conditional probability of 𝑎
given 𝑏, 𝑝(𝑎|𝑏), multiplied by the probability of 𝑏, 𝑝(𝑏). Here, the joint probably indicates the
likelihood that 𝑎 and 𝑏 will occur together (e.g., that it is cloudy and that it is raining). The
conditional probability indicates how likely 𝑎 is if we are told 𝑏 (e.g., how likely it is to rain if we
know that it is cloudy). The marginal probability, 𝑝(𝑏), indicates how likely something is to occur in
general (e.g., how often it is cloudy overall).

Symmetrically, the product rule says that 𝑝(𝑎, 𝑏) is also equivalent to the conditional probability of
𝑏 given 𝑎, 𝑝(𝑏|𝑎), multiplied by the marginal probability of 𝑎, 𝑝(𝑎). We can also use the product rule
to do standard algebraic manipulations. For example, we can take 𝑝(𝑎, 𝑏) = 𝑝(𝑏|𝑎)𝑝(𝑎) and then
divide both sides by 𝑝(𝑎) to get the conditional probability of 𝑏 given 𝑎, 𝑝(𝑏|𝑎):

𝑝(𝑎, 𝑏) (A.3)
𝑝(𝑏|𝑎) =
𝑝(𝑎)

123
Active Inference Tutorial

We can then derive Bayes’ theorem by simply replacing 𝑝(𝑎, 𝑏) in equation A.3 with 𝑝(𝑎|𝑏)𝑝(𝑏),
where this equivalence is shown in equation A.2:

𝑝(𝑎|𝑏)𝑝(𝑏) (A.4)
𝑝(𝑏|𝑎) =
𝑝(𝑎)

In scientific practice, we can represent observational data with the variable 𝑎 and then use the
variable 𝑏 to represent some theory. In this case, 𝑝(𝑏) is referred to as the prior probability of the
theory being true before observing the data. Equation A.4 then means that if we get some new data
𝑎, then we can use Bayes’ theorem to infer whether it increases the probability that some theory 𝑏
is true. In other words, we can infer whether the posterior probability of the theory given the
data, 𝑝(𝑏|𝑎), is higher than the prior probability that the theory was true before observing the new
data, 𝑝(𝑏). This just requires that we know how strongly the theory predicts that new piece of data
– typically referred to as the likelihood that the data would be observed if the theory were true,
𝑝(𝑎|𝑏).

The use of Venn diagrams is often helpful for gaining intuitions about probabilistic inference. We
illustrate this in Figure A1 by showing one circle on the left that contains all situations in which 𝑎 is
true (solid outline) and another circle on the right containing all situations in which 𝑏 is true
(dashed outline). The joint probability that both 𝑎 and 𝑏 are true corresponds to the shaded blue
area in the middle where the circles overlap. When we infer the posterior probability, 𝑝(𝑏|𝑎), we
first simply remove the circle for 𝑏 (i.e., imagine erasing the dashed outline and the separate 𝑏
variable, since we know from our new data that we are in a situation where 𝑎 is true). However, we
retain the blue shaded region within the circle for 𝑎 corresponding to when 𝑎 and 𝑏 are both true.
Then we look at how large the proportion of the circle is for 𝑎 that corresponds to when 𝑏 is also
true (i.e., how large the blue shaded area is). If this area represents a large portion of the circle for
𝑎, then this means the posterior probability, 𝑝(𝑏|𝑎), will be high. This proportion is what is
𝑝(𝑎,𝑏) 𝑝(𝑎|𝑏)𝑝(𝑏)
captured by (remember this is equivalent to in Bayes’ theorem; see equation A.2).
𝑝(𝑎) 𝑝(𝑎)

124
Active Inference Tutorial

Figure A1. Venn diagram depiction of a joint probability distribution and how it is used to infer a
conditional probability (such as a posterior probability in Bayesian inference). The solid circle
corresponds to all situations where 𝑎 is true. The dashed circle corresponds to all situations where
𝑏 is true. The blue area where the circles overlap represents the situations where both 𝑎 and 𝑏 are
true. To infer 𝑝(𝑏|𝑎), we simply remove the circle for b (since we are conditioning on 𝑎 being true).
Then we evaluate how much of the circle for 𝑎 is occupied by the blue region where both 𝑎 and 𝑏
are true. If the blue region covers the majority of the circle for 𝑎, then 𝑝(𝑏|𝑎) will be large, whereas
if the blue region only covers a small part of the circle for 𝑎, then 𝑝(𝑏|𝑎) will be small. Thus, 𝑝(𝑏|𝑎)
can be thought of as the proportion of situations where 𝑎 is true in which 𝑏 is also true.

A further fact to keep in mind about the rules of probability is that they do not change if other
constants are included. For example, the sum rule remains the same if all terms are conditioned on
some other variable 𝑐:

𝑝(𝑎|𝑐) = 𝑝(𝑎, 𝑏|𝑐) + 𝑝(𝑎, ~𝑏|𝑐) (A.5)

This also holds for the product rule:

𝑝(𝑎, 𝑏|𝑐) = 𝑝(𝑎|𝑏, 𝑐)𝑝(𝑏|𝑐) (A.6)

125
Active Inference Tutorial

As a numerical illustration of these rules, we will let 𝑎 = "𝑟𝑎𝑖𝑛𝑖𝑛𝑔" and 𝑏 = "𝑐𝑙𝑜𝑢𝑑𝑦 " and use
Table A1 to indicate the following prior and conditional probabilities (i.e., the probability of it
raining if it is cloudy):

Table A1. Prior and conditional probabilities

𝒄𝒍𝒐𝒖𝒅𝒚 (𝒑 =. 𝟕) 𝒏𝒐𝒕 𝒄𝒍𝒐𝒖𝒅𝒚 (𝒑 =. 𝟑)

𝒓𝒂𝒊𝒏𝒊𝒏𝒈 𝑝(𝑟𝑎𝑖𝑛𝑖𝑛𝑔|𝑐𝑙𝑜𝑢𝑑𝑦) 𝑝(𝑟𝑎𝑖𝑛𝑖𝑛𝑔|𝑛𝑜𝑡 𝑐𝑙𝑜𝑢𝑑𝑦)

= .6 = .01

𝒏𝒐𝒕 𝒓𝒂𝒊𝒏𝒊𝒏𝒈 𝑝(𝑛𝑜𝑡 𝑟𝑎𝑖𝑛𝑖𝑛𝑔|𝑐𝑙𝑜𝑢𝑑𝑦) 𝑝(𝑛𝑜𝑡 𝑟𝑎𝑖𝑛𝑖𝑛𝑔|𝑛𝑜𝑡 𝑐𝑙𝑜𝑢𝑑𝑦)

= .4 = .99

Note that while the values of each column sum to 1, the values of the rows do not, which is why the
likelihood term in Bayes’ theorem is not technically considered a proper probability distribution
(i.e., if one wanted to treat it as such, the rows would need to be normalized so that they kept the
same proportions but did sum to 1).

To get the joint probabilities, we simply multiply each prior by each conditional (i.e., the product
rule):

Table A2. Joint probabilities

𝒄𝒍𝒐𝒖𝒅𝒚 𝒏𝒐𝒕 𝒄𝒍𝒐𝒖𝒅𝒚 Marginal probabilities

𝒓𝒂𝒊𝒏𝒊𝒏𝒈 𝑝(𝑟𝑎𝑖𝑛𝑖𝑛𝑔, 𝑐𝑙𝑜𝑢𝑑𝑦) 𝑝(𝑟𝑎𝑖𝑛𝑖𝑛𝑔, 𝑛𝑜𝑡 𝑐𝑙𝑜𝑢𝑑𝑦) 𝑝(𝑟𝑎𝑖𝑛𝑖𝑛𝑔)

= .7 × .6 = .42 = .3 × .01 = .003 = .42 + .003 = .423

𝒏𝒐𝒕 𝒓𝒂𝒊𝒏𝒊𝒏𝒈 𝑝(𝑛𝑜𝑡 𝑟𝑎𝑖𝑛𝑖𝑛𝑔, 𝑐𝑙𝑜𝑢𝑑𝑦) 𝑝(𝑛𝑜𝑡 𝑟𝑎𝑖𝑛𝑖𝑛𝑔, 𝑛𝑜𝑡 𝑐𝑙𝑜𝑢𝑑𝑦) 𝑝(𝑛𝑜𝑡 𝑟𝑎𝑖𝑛𝑖𝑛𝑔)

= .7 × .4 = .28 = .3 × .99 = .297 = .28 + .297 = .577

In this case, all the joint probabilities sum to 1, as these four cells (i.e., in the first and second
columns) describe all possible outcomes. As shown in the third column, summing across each row
gives us the marginal probabilities for each possible value of 𝑝(𝑎) (i.e., raining vs. not raining),

126
Active Inference Tutorial

based on the sum rule. These are the values that end up “in the margin” (which is why they are
called marginal probabilities) when summing over the probabilities under each possible value of 𝑏
(i.e., cloudy vs. not cloudy).

We can then find the posterior probabilities (Table A3) by dividing the joint probabilities (i.e., the
value in each cell within the first and second columns of Table A2) by the marginal probabilities
(i.e., the value of the associated row within the third column in Table A2):

Table A3. Posterior probabilities

𝒄𝒍𝒐𝒖𝒅𝒚 𝒏𝒐𝒕 𝒄𝒍𝒐𝒖𝒅𝒚

𝒓𝒂𝒊𝒏𝒊𝒏𝒈 𝑝(𝑐𝑙𝑜𝑢𝑑𝑦|𝑟𝑎𝑖𝑛𝑖𝑛𝑔) 𝑝(𝑛𝑜𝑡 𝑐𝑙𝑜𝑢𝑑𝑦|𝑟𝑎𝑖𝑛𝑖𝑛𝑔)

. 42 . 003
= = .993 = = .007
. 423 . 423

𝒏𝒐𝒕 𝒓𝒂𝒊𝒏𝒊𝒏𝒈 𝑝(𝑐𝑙𝑜𝑢𝑑𝑦|𝑛𝑜𝑡 𝑟𝑎𝑖𝑛𝑖𝑛𝑔) 𝑝(𝑛𝑜𝑡 𝑐𝑙𝑜𝑢𝑑𝑦|𝑛𝑜𝑡 𝑟𝑎𝑖𝑛𝑖𝑛𝑔)

. 28 . 297
= = .485 = = .515
. 577 . 577

This tells us, for example, that if you look out your window and see that it is raining, then you can
confidently infer that it is cloudy.

A final rule that will be useful to know for this tutorial pertains to logarithmic (log)
transformations. Specifically, log transformations of probabilities allow multiplication and
division to be expressed in terms of addition and subtraction (respectively). For example, using the
natural logarithm (ln), the product rule can be expressed as follows:

ln𝑝(𝑎, 𝑏) = ln𝑝(𝑏|𝑎) + ln𝑝(𝑎) (A.7)

ln𝑝(𝑏|𝑎) = ln𝑝(𝑎, 𝑏) − ln𝑝(𝑎) (A.8)

Performing these transformations is often beneficial in practice because it simplifies the necessary
computations and can prevent the need to work with very small probabilities (which can happen
when many probabilities must be multiplied together in complex real-world problems). Log-

127
Active Inference Tutorial

probabilities can also be easily converted back into standard probabilities by exponentiating them.
That is, 𝑒 ln𝑝(𝑎) = 𝑝(𝑎), where 𝑒 ≈ 2.71828 (often called Euler's number).

This concludes our brief introduction to the rules of probability. Many derivations in this tutorial
work primarily based on using these rules in combination with algebraic manipulation.

Introduction to Variational Inference

The typical goal of Bayesian inference is to find the posterior distribution 𝑝(𝑠|𝑜) – that is, to infer
how the states of the world (𝑠) have changed based on new observations (𝑜). However, this
requires one to calculate the marginal likelihood 𝑝(𝑜), which often involves intractable sums (or
integrals in the case of states/observations with continuous values). The key idea behind
variational inference is to convert this inference problem into an optimization problem. To do so,
instead of evaluating the marginal likelihood, we optimize an auxiliary distribution 𝑞(𝑠)
(sometimes called the recognition distribution, or variational posterior) to approximate the true
posterior 𝑝(𝑠|𝑜). This is done by using the KL divergence as a measure of the relative difference (in
the information-theoretic unit nats – the natural log equivalent of bits) between the two
distributions:

𝑞(𝑠) (A.9)
𝐷𝐾𝐿 (𝑞(𝑠)||𝑝(𝑠|𝑜)) = ∑ 𝑞(𝑠) ln
𝑝(𝑠|𝑜)
𝑠

The KL divergence sums over the states of the two distributions so the output is always greater
than or equal to zero. When the recognition distribution and the true posterior match, the KL
divergence is zero (i.e., when 𝑞(𝑠) = 𝑝(𝑠|𝑜), 𝐷𝐾𝐿 (𝑞(𝑠)||𝑝(𝑠|𝑜)) = 0). Although, as we do not know
the true posterior distribution, this sum also cannot be evaluated. Crucially, however, working from
𝑝(𝑜,𝑠)
the definition of conditional probabilities, 𝑝(𝑠|𝑜) = 𝑝(𝑜)
, we can introduce a quantity that can be
1 1 y
evaluated directly (noting that = x / y
= x ):

𝑞(𝑠)
𝐷𝐾𝐿 (𝑞(𝑠)||𝑝(𝑠|𝑜)) = ∑ 𝑞(𝑠) ln
𝑝(𝑜, 𝑠) (A.10)
𝑠
𝑝(𝑜)

𝑞(𝑠)𝑝(𝑜)
= ∑ 𝑞(𝑠) ln L2
𝑝(𝑜, 𝑠)
𝑠

𝑞(𝑠)
= ∑ 𝑞(𝑠) ln + ln (𝑝(𝑜)) L3
𝑝(𝑜, 𝑠)
𝑠

128
Active Inference Tutorial

Equation A.10 substitutes the alternate definition of the posterior distribution into equation A.9.
With some minor rearrangement, we see that the KL divergence between our approximate
posterior and the true posterior is now equal to the KL divergence between our approximate
posterior and 𝑝(𝑜, 𝑠) – which can be viewed as a generative model of how states of the world
generate observations – plus the log probability of observations (i.e., the log of the marginal
likelihood). This is the critical move. Because we are free to specify the generative model, and 𝑞(𝑠)
is the variable we seek to optimize (and can thus be initially set to an arbitrary value; see below),
we have access to both the quantities we need to compute the KL divergence. We now introduce a
new quantity, variational free energy (𝑉𝐹𝐸), denoted 𝐹, and define it in terms of this KL divergence:
𝑞(𝑠)
𝐹 ≔ ∑𝑠 𝑞(𝑠) ln 𝑝(𝑜,𝑠) . The value of the approximate posterior 𝑞(𝑠) that minimizes 𝑉𝐹𝐸 will then be
the 𝑞(𝑠) that best approximates the true posterior distribution.

𝐷𝐾𝐿 (𝑞(𝑠)||𝑝(𝑠|𝑜)) = 𝐹 + ln(𝑝(𝑜)) (A.11)

Equation A.11 rewrites line 3 of equation A.10, but substitutes in 𝑉𝐹𝐸 as an explicit variable. From
this vantage point, we see that when 𝑉𝐹𝐸 is minimized the KL divergence between the approximate
posterior and the true posterior is also minimized, meaning that the approximate posterior is close
to the true posterior. Hence, minimizing 𝑉𝐹𝐸 allows a tractable means of performing approximate
Bayesian inference. One way to do this is using gradient descent. That is, one can start 𝑞(𝑠) at an
arbitrary value and then test neighboring values to find the one that reduces 𝑉𝐹𝐸 most. Then one
can move to that value, search neighboring values again (etc.), and repeat this process until a value
for 𝑞(𝑠) is found for which no neighboring values further reduce 𝑉𝐹𝐸.

Expected Free Energy

Active inference reverses the usual logic of action selection. Instead of asking “which sequence of
actions will bring about my preferred outcomes?”, it formally asks, “given the assumption that I will
achieve my preferred outcomes, what course of action am I most likely to pursue?”. Within active
inference, the answer to this question is the policy (𝜋; i.e., action sequence) that best minimizes a
quantity termed expected free energy (𝐸𝐹𝐸). Here we show the most common decompositions of
𝐸𝐹𝐸 that appear in the active inference literature and describe the workings and the intuition
behind each decomposition. 𝐸𝐹𝐸 is defined in terms of the expected difference between the log of
the generative model 𝑝(𝑜, 𝑠|𝜋) and the log of the approximate posterior given a choice of policy
𝑞(𝑠|𝜋). Equation A.12 shows the decomposition of the 𝐸𝐹𝐸 of each policy (𝐺𝜋 ) into terms often
referred to as epistemic and pragmatic value (intuitively, expected information gain and reward
probability under each policy, respectively). It also shows another common decomposition into
terms referred to as risk and ambiguity (similarly corresponding to expected reward and
uncertainty minimization under each policy). Note below that, because 𝐸𝐹𝐸 is calculated with
respect to expected outcomes that (by definition) have not yet occurred, observations enter the
expectation operator Ε𝑞 as random variables.

129
Active Inference Tutorial

𝐺𝜋 = Ε𝑞(𝑜,𝑠|𝜋) [ln 𝑞(𝑠|𝜋) − ln 𝑝(𝑜, 𝑠|𝜋)] (A.12)

= Ε𝑞(𝑜,𝑠|𝜋) [ln 𝑞(𝑠|𝜋) − ln 𝑝(𝑠|𝑜, 𝜋)] − Ε𝑞(𝑜|𝜋) [ln𝑝(𝑜|𝜋)] L2

≈ Ε𝑞(𝑜,𝑠|𝜋) [ln 𝑞(𝑠|𝜋) − ln 𝑞(𝑠|𝑜, 𝜋)] − Ε𝑞(𝑜|𝜋) [ln𝑝(𝑜|𝐶)] L3

= −Ε𝑞(𝑜,𝑠|𝜋) [ln𝑞(𝑠|𝑜, 𝜋) − ln𝑞(s| 𝜋)] − Ε𝑞(𝑜|𝜋) [ln𝑝(𝑜|𝐶)] L4

= Ε𝑞(𝑜,𝑠|𝜋) [ln 𝑞(𝑜|𝜋) − ln 𝑞(𝑜|𝑠, 𝜋)] − Ε𝑞(𝑜|𝜋) [ln𝑝(𝑜|𝐶)] L5

= 𝐷𝐾𝐿 (𝑞(𝑜|𝜋)||𝑝(𝑜|𝐶)) + E𝑞(𝑠|𝜋) [𝐻[𝑝(𝑜|𝑠)]] L6

The second line uses the product rule of probability, 𝑝(𝑜, 𝑠|𝜋) = 𝑝(𝑠|𝑜, 𝜋)𝑝(𝑜|𝜋), to rearrange 𝐸𝐹𝐸
into the epistemic and pragmatic value terms described in the main text. In the third line, the
dependence on policies is dropped from the pragmatic value term so that it can be used to encode
preferences (i.e., this is a key move in active inference). Note that, in most papers on active
inference, this is simply written as Ε𝑞(𝑜|𝜋) [ln𝑝(𝑜)]; however, to clearly distinguish this from the
ln𝑝(𝑜) term within 𝑉𝐹𝐸 (i.e., where 𝑜 is an observed variable), we write the pragmatic value term
here as explicitly conditioned on a preference variable 𝐶. Line 3 also replaces the true posterior
(ln𝑝(𝑠|𝑜, 𝜋)) with an approximate posterior (ln𝑞(𝑠|𝑜, 𝜋)). Line 4 offers a clearer intuition for
epistemic value by flipping the terms inside the first expectation so that it becomes prefixed with a
negative sign (i.e., 𝑝(𝑥)[ln𝑝(𝑥) − ln𝑞(𝑥)] = −𝑝(𝑥)[ln𝑞(𝑥) − ln𝑝(𝑥)]). Because the epistemic
value term is now subtracted from the total, it is clear that, to minimize 𝐸𝐹𝐸 overall, an agent must
maximize the value of this term by selecting policies that take it into states that maximize the
difference between ln 𝑞(𝑠|𝑜, 𝜋) and ln 𝑞(𝑠|𝜋). In other words, the agent is driven to seek out
observations that reduce uncertainty about hidden states (i.e., maximize the change from prior to
posterior beliefs after a new observation). For example, if you are in a dark room, then the mapping
between hidden states and observations is entirely ambiguous. The best way to minimize
uncertainty is to turn a light on.

𝑞(𝑠|𝜋)
Moving from the expression in line 3, line 5 uses Bayes rule in the denominator 𝑞(𝑠|𝑜,𝜋)
=
𝑞(𝑠|𝜋)𝑞(𝑜|𝜋) 𝑞(𝑜|𝜋)
= to express the same epistemic imperative, but with the conditional
𝑝(𝑜|𝑠,𝜋)𝑞(𝑠|𝜋) 𝑝(𝑜|𝑠,𝜋)
𝑞(𝑜|𝑠,𝜋)𝑞(𝑠|𝜋)
probabilities flipped. Note here that, although algebraically 𝑞(𝑠|𝑜, 𝜋) = , in this case
𝑞(𝑜|𝜋)
𝑞(𝑜|𝑠, 𝜋) and 𝑝(𝑜|𝑠, 𝜋) refer to the same distribution. The epistemic value terms in lines 3, 4, and 5
are formally equivalent since they each express the mutual information between hidden states
and observations. First noting that 𝐻[𝑝(𝑥)] denotes the entropy of a distribution 𝑝(𝑥), where
𝐻[𝑝(𝑥)] = − ∑𝑥 𝑝(𝑥) ln𝑝(𝑥) = −E𝑝(𝑥) [𝑙𝑛𝑝(𝑥)], mutual information can be written as 𝐼(𝑥, 𝑦) =
𝐻[𝑝(𝑥)] − 𝐻[𝑝(𝑥|𝑦)] = 𝐻[𝑝(𝑦)] − 𝐻[𝑝(𝑦|𝑥)]. This quantity 𝐼(𝑥, 𝑦) is symmetric and scores the
reduction in uncertainty (entropy) about the value of a variable 𝑥 afforded by knowledge of another
variable 𝑦. If two variables are independent, mutual information is zero. Finally, line 6 expresses
𝐸𝐹𝐸 in the form shown in the main text as risk plus ambiguity. The first term, risk, is the KL

130
Active Inference Tutorial

divergence between the observations expected under a policy and prior preferences. The second
term, ambiguity, scores the uncertainty in the likelihood mapping between states and observations.
Minimizing 𝐸𝐹𝐸 thus requires agents to select policies that minimize the difference between
expected observations and preferred observations (e.g., seeking warmth when it is cold,
maximizing reward, etc.) and to take actions that reduce uncertainty (i.e., ambiguity) about the
mapping between hidden states and observations (i.e., another way of expressing the drive to
maximize information gain). The work to move between the fifth and the sixth line is somewhat
convoluted, so equation A.13 shows it step by step:

𝑞(𝑜|𝜋) (A.13)
𝐺𝜋 = ∑ 𝑝(𝑜|𝑠)𝑞(𝑠|𝜋)(ln ) − ∑ 𝑝(𝑜|𝑠)𝑞(𝑠|𝜋)ln𝑝(𝑜|𝐶)
𝑝(𝑜|𝑠, 𝜋)
𝑜,𝑠 𝑜,𝑠

𝑞(𝑜|𝜋) L2
= ∑ 𝑝(𝑜|𝑠)𝑞(𝑠|𝜋)(ln )
𝑝(𝑜|𝐶)𝑝(𝑜|𝑠, 𝜋)
o,s

𝑞(𝑜|𝜋) L3
= ∑ 𝑞(𝑜, 𝑠|𝜋)(ln ) − ∑ 𝑞(𝑠|𝜋) ∑ 𝑝(𝑜|𝑠) ln𝑝(𝑜|𝑠, 𝜋)
𝑝(𝑜|𝐶)
o,s s o

𝑞(𝑜|𝜋) L4
= ∑ 𝑞(𝑜|𝜋)(ln ) − ∑ 𝑞(𝑠|𝜋) ∑ 𝑝(𝑜|𝑠) ln𝑝(𝑜|𝑠)
𝑝(𝑜|𝐶)
o s o

𝑞(𝑜|𝜋) L5
= ∑ 𝑞(𝑜|𝜋)(ln ) + ∑ q(s|π)H[ 𝑝(𝑜|𝑠)]
𝑝(𝑜|𝐶)
o s

= DKL (𝑞(𝑜|𝜋)||𝑝(𝑜|𝐶)) + E𝑞(s|π) [H[𝑝(𝑜|𝑠)]] L6

Line 1 of equation A.13 rewrites line 5 of equation A.12, but makes the summations implied by the
expectation operators explicit. Line 2 moves 𝑝(𝑜|𝐶) back into the same expectation. Line 3
expresses the expectation in the first term in terms of an approximate joint distribution
𝑝(𝑜|𝑠)𝑞(𝑠|𝜋) = 𝑞(𝑜, 𝑠|𝜋), and in the second term separates out 𝑝(𝑜|𝑠, 𝜋). In the first term of line 4
we evaluate the summation over states in the joint distribution ∑o,s 𝑞(𝑜, 𝑠|𝜋) = ∑o 𝑞(𝑜|𝜋),
leaving the fraction inside the log untouched as it does not depend on states. In the second term of
line 4 we drop the dependency on policies since the likelihood mapping is constant across choices
of policy. In line 5 this then allows us to express − ∑o 𝑝(𝑜|𝑠) ln𝑝(𝑜|𝑠) in terms of entropy 𝐻[𝑝(𝑜|𝑠)],
𝑞(𝑜|𝜋)
and rewrite ∑𝑜 𝑞(𝑜|𝜋)𝑙𝑛 in terms of the KL divergence (defined in the main text) between
𝑝(𝑜|𝐶)
prior preferences and observations expected under each policy 𝐷𝐾𝐿 (𝑞(𝑜|𝜋)||𝑝(𝑜|𝐶)). Finally,
because entropy is a negative quantity, we swap the sign between the two terms, leaving us with
the canonical form of 𝐸𝐹𝐸 as risk plus ambiguity in line 6 – where lower risk indicates a higher
probability of preferred outcomes under a policy and lower ambiguity indicates more precise
(informative) observations expected under a policy.

131
Active Inference Tutorial

It is important to highlight here that generative models in active inference also maintain confidence
estimates for the model parameters themselves, via a form of distribution called a Dirichlet
distribution that encodes priors over these parameters (see main text for an introduction to this
type of distribution). Dirichlet distributions contain what are called concentration parameters,
where higher concentration parameter values indicate lower uncertainty in the parameters of each
distribution. The above expressions of 𝐸𝐹𝐸 assume that the concentration parameters are
saturated (i.e., that they are maximally precise), and hence that there is no uncertainty. However,
when there is uncertainty in parameters (as is the case for real organisms), agents must also learn
the values for those parameters via the selection of appropriate policies. This means that parameter
uncertainty now enters the equation for 𝐸𝐹𝐸. For example, equation A.14 shows the form of 𝐸𝐹𝐸
when the parameters of the likelihood 𝑝(𝑜|𝑠) must be learned. Note that this likelihood is termed
the 𝐀 matrix in active inference models (see Table 1).

𝐺𝜋 = Εq [ln 𝑞(𝑠, 𝐀|𝜋) − ln 𝑝(𝑜, 𝑠, 𝐀|𝜋)] (A.14)

= Εq [ ln𝑞(𝑠|𝜋) + ln𝑞(𝐀) − ln𝑝(𝐀|𝑠, 𝑜, 𝜋) − ln𝑝(𝑠|𝑜, 𝜋) − ln𝑝(𝑜|𝜋)] L2

≈ Εq [ ln𝑞(𝑠|𝜋) + ln𝑞(𝐀) − ln𝑞(𝐀|𝑠, 𝑜, 𝜋) − ln𝑞(𝑠|𝑜, 𝜋) − ln𝑝(𝑜|𝜋)] L3

= Εq [ ln𝑞(𝑠|𝜋) + ln𝑞(𝐀) − ln𝑞(𝐀|𝑠, 𝑜, 𝜋) − ln𝑞(𝑠|𝑜, 𝜋)] − Εq [ln𝑝(𝑜|𝐶)] L4

= Εq [ ln𝑞(𝑠|𝜋) − ln𝑞(𝑠|𝑜, 𝜋)] + Εq [ln𝑞(𝐀) − ln𝑞(𝐀|𝑠, 𝑜, 𝜋)] − Εq [ln𝑝(𝑜|𝐶)] L5

= −Εq [ ln𝑞(𝑠|𝑜, 𝜋) − ln𝑞(𝑠|𝜋)] − Εq [ln𝑞(𝐀|𝑠, 𝑜, 𝜋) − ln𝑞(𝐀)] − Εq [ln𝑝(𝑜|𝐶)] L6

Here 𝑞 = 𝑞(𝑜, 𝑠, 𝐀|𝜋). Line 1 shows the form of 𝐸𝐹𝐸 when 𝐀 is treated as a random variable. Line 2
breaks the approximate posterior and generative model into separate terms using the product rule
of probability 𝑝(𝑜, 𝑠, 𝐀|𝜋) = 𝑝(𝐀|𝑠, 𝑜, 𝜋)𝑝(𝑠|𝑜, 𝜋)𝑝(𝑜|𝜋). It also uses the mean-field approximation –
which assumes that the approximate posterior factorizes into the product of independent marginal
distributions – to express the approximate joint distribution as 𝑞(𝑠, 𝐀|𝜋) = 𝑞(𝑠|𝜋)𝑞(𝐀). Line 3
approximates line two, replacing the exact posteriors 𝑝(𝐀|𝑠, 𝑜, 𝜋) and 𝑝(𝑠|𝑜, 𝜋) with approximate
posteriors 𝑞(𝐀|𝑠, 𝑜, 𝜋) and 𝑞(𝑠|𝑜, 𝜋). Line 4 takes pragmatic value Εq [ln𝑝(𝑜|𝜋)] out of the first
expectation term and then (as above) conditions on the preference variable 𝐶 instead of 𝜋; i.e.,
Εq [ln𝑝(𝑜|𝐶)]. Line 5 breaks the first expectation in line 4 into two quantities. The first,
Εq [ ln𝑞(𝑠|𝜋) − ln𝑞(𝑠|𝑜, 𝜋)]) is the epistemic value term that we saw in the previous expression of
𝐸𝐹𝐸 without uncertainty in the parameters. As we now have two types of epistemic value, to
distinguish them we call the first salience and the second novelty. Salience scores the reduction of
uncertainty about states afforded by observations (driving “state exploration” behavior), while
novelty Εq [ln𝑞(𝐀) − ln𝑞(𝐀|𝑠, 𝑜, 𝜋)] scores the reduction in uncertainty about parameters of the
generative model afforded by states and observations (driving “parameter exploration” behavior;
see (Schwartenbeck et al., 2019)). As with line 4 in equation A.12, line 6 here flips the terms inside
the first and second expectation to make the salience and novelty terms negative (i.e., such that
maximizing these terms brings their value closer to zero). This makes it clearer why maximizing the

132
Active Inference Tutorial

difference between prior and posterior beliefs – here about both states and parameters – will
minimize 𝐸𝐹𝐸. The parameters of the model are the sufficient statistics of Dirichlet distributions,
which, in the case of learning 𝐀, essentially count the number of times a particular categorical state
is inferred when a particular outcome is observed (proportional to the posterior probability over
each state). Like the epistemic value term in the previous expression of 𝐸𝐹𝐸, to minimize 𝐸𝐹𝐸 here
agents must maximize both salience and novelty by seeking out observations that 1) reduce
uncertainty about hidden states, and/or 2) reduce uncertainty about parameters. The reduction of
uncertainty of the parameters via the maximization of novelty encourages agents to explore novel
parts of the state space that are less familiar (i.e., states that have low concentration parameters).

The Softmax Function

The softmax (or normalized exponential) function, denoted by σ, takes a vector 𝑥 of length 𝑘 and
normalizes the vector such that the elements 1) have a monotonic relationship with the elements of
the input vector, and 2) sum to 1 and can thus be treated as a categorical probability distribution
over 1 to k mutually exclusive states. Importantly, the vector is weighted by a precision parameter
denoted by 𝛾, which controls the extent to which differences between the elements are amplified or
dampened by the exponential.

eγxi (A.15)
σ(x) =
∑k eγxk

For example, for 𝑥 = [1 2 3 4]T , when 𝛾 = 1, σ(x) = [0.0321 0.0871 0.2369 0.6439]T .

When 𝛾 = 0.1, σ(x) = [0.2138 0.2363 0.2612 0.2887]T.

When 𝛾 = 2, 𝜎(𝑥) = [0.0021 0.0158 0.1171 0.8650]T.

The Gamma Function

The gamma function (denoted by Γ) is a generalization of the factorial function that, unlike the
factorial function (whose domain is restricted to positive integers), is well defined for complex and
real valued (i.e., non-integer) inputs (except for the negative integers). For positive, real-valued, and
complex numbers, the gamma function is defined by the following definite integral.

∞
(A.16)
Γ(𝑧) = ∫ 𝑥 𝑧−1 𝑒 −𝑥 𝑑𝑥
0

For the positive integers, the gamma function reduces to the factorial function (i.e., n! =
n × (n − 1) × (n − 2) × (n − 3) × … × 3 × 2 × 1) but is shifted by 1.

133
Active Inference Tutorial

Γ(n) = (𝑛 − 1)! (A.17)

Γ(2) = (2 – 1)! = 1 × 1 = 1 L2

Γ(3) = (3 – 1)! = 2 × 1 = 2 L3

Γ(4) = (4 – 1)! = 3 × 2 × 1 = 6 L4

To gain an intuition for how the gamma function relates to the factorial function , we will use
𝑎 𝑎
integration by parts (i.e., ∫𝑏 𝑓(𝑥)𝑔′ (𝑥)𝑑𝑥 = [𝑓(𝑥)𝑔(𝑥)]𝑎𝑏 − ∫𝑏 𝑓′(𝑥)𝑔(𝑥)𝑑𝑥) to show that
Γ(n + 1) = (𝑛)!.

∞ ∞
Γ(𝑧 + 1) = ∫ 𝑥 𝑧 𝑒 −𝑥 𝑑𝑥 = [−𝑥 𝑧 𝑒 −𝑥 ]∞ 𝑧−1 −𝑥 (A.18)
0 + ∫ 𝑧𝑥 𝑒 𝑑𝑥
0 0

∞
= lim (−𝑥 𝑧 𝑒 −𝑥 ) − (−0𝑒 −0 ) + 𝑧 ∫ 𝑥 𝑧−1 𝑒 −𝑥 𝑑𝑥 L2
𝑥→∞ 0

Because 𝑒 −𝑥 grows faster than 𝑥 𝑧 , the first term is sent to zero leaving us with the following.

∞
Γ(𝑧 + 1) = 𝑧 ∫ 𝑥 𝑧−1 𝑒 −𝑥 𝑑𝑥 (A.19)
0

= 𝑧 Γ(𝑧) L2

If we plug in some examples, we see immediately that this is equivalent to the factorial function.

Γ(2 + 1) = 2 𝛤(2) = 2 (2 − 1)! = 2 = 2! (A.20)

Γ(3 + 1) = 3 𝛤(3) = 3 (3 − 1)! = 6 = 3! L2

Γ(4 + 1) = 4 Γ(4) = 4 (4 − 1)! = 24 = 4! L3

In the context of the Dirichlet distribution, the gamma function is used to define a normalization
constant that accounts for the combinatorics of the concentration parameters, which are positive
real-valued numbers (i.e., they can take non-integer values), most of which are not defined when
using the factorial function, which is why the gamma function is employed. We can think of the
gamma function as interpolating between the values of the factorial function (i.e., because it is

134
Active Inference Tutorial

defined with respect to non-negative real numbers, while the factorial function is only defined with
respect to the non-negative integers). In other words, when the factorial function and gamma
function are given the same inputs, they produce consistent outputs (shifted by one), but the
gamma function outputs values between the integer outputs given by the factorial function.

2. Pencil and Paper Exercises

The purpose of this supplementary section is to provide a set of exercises that can be worked
through using pencil and paper, with the aim of building an intuition for how active inference
models operate. The first exercise is a simple example of static perception. The second example
extends this by modelling how states (and the observations they generate) change across time,
which is an example of a hidden Markov model (HMM). HMMs are the perceptual component of
partially observable Markov decision processes (POMDPs). We have not included an example of
policy selection, as performing the calculations for even two policies involves too many
computations to reasonably expect readers to perform them by hand.

To help the reader build a conceptual bridge between the update equations and their
implementation in code, we have also included MATLAB code
(𝐏𝐞𝐧𝐜𝐢𝐥_𝐚𝐧𝐝_𝐩𝐚𝐩𝐞𝐫_𝐞𝐱𝐞𝐫𝐜𝐢𝐬𝐞_𝐬𝐨𝐥𝐮𝐭𝐢𝐨𝐧𝐬. 𝐦) that has solutions to each of the exercises shown
below. Finally, readers should note that, to ensure that the exercises can be solved by hand, we have
excluded a key aspect of active inference models as they are usually implemented. Namely, instead
of inferring the posterior over hidden states using gradient descent, we use only a single round of
marginal message passing. For readers seeking to understand how message passing and policy
selection operate in the model inversion procedure implemented in 𝐬𝐩𝐦_𝐌𝐃𝐏_𝐕𝐁_𝐗. 𝐦, please see
the stand-alone MATLAB script 𝐒𝐢𝐦𝐩𝐥𝐢𝐟𝐢𝐞𝐝_𝐬𝐢𝐦𝐮𝐥𝐚𝐭𝐢𝐨𝐧_𝐬𝐜𝐫𝐢𝐩𝐭. 𝐦 that is also provided, which is a
stripped down, but thoroughly commented, version of the model inversion scheme used in
𝐬𝐩𝐦_𝐌𝐃𝐏_𝐕𝐁_𝐗. 𝐦.

Exercises

Static Perception

For this first example we will keep things as simple as possible.

Example 1

Update Equation

s = σ(ln𝐷 + ln𝐀T o)

Generative Model and Observation

.5 .9 .3 1
𝐷 = [ ]; 𝐀 = [ ]; o = [ ]
.5 .1 .7 0

135
Active Inference Tutorial

Model Inversion

.5 .9 .1 1 .5 .9
s = σ (ln [ ] + ln [ ] [ ]) = σ (ln [ ] + ln [ ])
.5 .3 .7 0 .5 .3

. 5 × .9 . 45
= σ (ln [ ]) = σ (ln [ ])
. 5 × .3 . 15

𝑒 ln (.45) . 45
ln (.45) ln (.15) . 75
= 𝑒 +𝑒 = [. 45 + .15] = [ ]
𝑒 ln (.15) . 15 . 25
[ 𝑒 ln (.45) + 𝑒 ln(.15) ] . 45 + .15

Exercise 1

Based on the update equation in example 1 and the observation listed below, invert the following
generative model:

. 75 .8 .2 1
𝐷 = [ ]; 𝐀 = [ ]; o = [ ]
. 25 .2 .8 0

Dynamic Perception

In this second example we move from a static environment to a dynamic one in which hidden
states, and the observations generated by these states, change across time. In addition to including
a 𝐁 matrix that encodes the transition probabilities, we must initialize the approximate posteriors
for each tau (𝜏) before starting model inversion (recall, tau references a time about which one has
beliefs, not a time at which one updates beliefs with a new observation). It is also important to note
that, because the log of zero is not defined, the model inversion procedure implemented in the code
adds a very small number (𝑒 −16 = 0.00000011253) to all inputs which turns the log of zero into the
log of a very small number. Since we expect readers to be able to do this exercise by hand, we
approximate this by adding 0.01 to the input of all logs.

Example 2

Update Equations

1
𝑠𝜏=1 = 𝜎 ( (ln 𝐃 + ln𝐁𝜏† 𝑠𝜏+1 ) + ln 𝐀T 𝑜𝜏 )
2

1
𝑠1<𝜏<𝑇 = 𝜎 ( (ln 𝐁𝜏−1 s𝜏−1 + ln𝐁𝜏† 𝑠𝜏+1 ) + ln 𝐀T 𝑜𝜏 )
2

1
𝑠𝜏=𝑇 = 𝜎 ( (ln 𝐁𝜏−1 s𝜏−1 ) + ln 𝐀T 𝑜𝜏 )
2

136
Active Inference Tutorial

*Recall that 𝐁† denotes the transpose of 𝐁 with normalized columns (i.e., columns that sum to 1).
Also note that because the example below only includes 2 time points, only the first and third
equations will apply.

Generative Model and Observations

. 75 .8 .2 0 1 1 0
𝐷 = [ ]; 𝐀 = [ ]; 𝐁 = [ ]; 𝑜𝜏=1 = [ ]; 𝑜𝜏=2 = [ ]
. 25 .2 .8 1 0 0 1

Initialize Approximate Posteriors

.5 .5
s𝜏=1 = [ ]; s𝜏=2 = [ ]
.5 .5

Model Inversion: Time Step 1

1 . 75 1 0 1 .5 .8 .2 1
s𝜏=1 = σ (2 ln [ ] + 2 ln [ ] [ ] + ln [ ] [ ])
. 25 1 0 .5 .2 .8 0

−.1372 −.3367 −.2107

= σ ([ ]+[ ]+[ ])
−.6735 −.3367 −1.5606

−.6846
= σ ([ ])
−2.5709

. 8683
=[ ]
. 1317

1 0 1 . 8683 .8 .2 0
s𝜏=2 = σ ( ln [ ][ ] + ln [ ] [ ])
2 1 0 . 1317 .2 .8 0

−.9771 −4.6052
= σ ([ ]+[ ])
−.0649 −4.6052

−5.5823
= σ ([ ])
−4.6700

. 2865
=[ ]
. 7135

Model Inversion: Time Step 2

1 . 75 1 0 1 . 2865 .8 .2 1
s𝜏=1 = σ ( ln [ ] + ln [ ][ ] + ln [ ] [ ])
2 . 25 2 1 0 . 7135 .2 .8 0

−.1372 −.1619 −.2107

= σ ([ ]+[ ]+[ ])
−.6735 −.6078 −1.5606

−.5098
= σ ([ ])
−2.8420

137
Active Inference Tutorial

. 9115
=[ ]
. 0885

1 0 1 . 9115 .8 .2 0
s𝜏=2 = σ (2 ln [ ][ ] + ln [ ] [ ])
1 0 . 0885 .2 .8 1

−1.1589 −1.5606
= σ ([ ]+[ ])
−.0409 −0.2107

−2.7195
= σ ([ ])
−0.2516

. 0781
=[ ]
. 9219

Exercise 2

Using the equations presented in example 2, and the observations listed below, invert the following
generative model. Note again that because this example only includes 2 timepoints, only the first
and third equations will apply.

.5 .9 .1 1 0 1 1
𝐷 = [ ]; 𝐀 = [ ]; 𝐁 = [ ]; 𝑜𝜏=1 = [ ]; 𝑜𝜏=2 = [ ]
.5 .1 .9 0 1 0 0

Answers

Answer: Exercise 1

. 75 .8 .2 1 . 75 .8
s = σ (ln [ ] + ln [ ] [ ]) = σ (ln [ ] + ln [ ])
. 25 .2 .8 0 . 25 .2

. 75 × .8 .6
= σ (ln [ ]) = σ (ln [ ])
. 25 × .2 . 05

𝑒 ln (.6) .6
ln (.6) ln (.05) . 9231
= 𝑒 +𝑒 = [. 6 + .05] = [ ]
𝑒 ln (.05) . 05 . 0769
[ 𝑒 ln (.6) + 𝑒 ln(.05) ] . 6 + .05

Answer: Exercise 2

Model Inversion: Time Step 1

1 .5 1 1 0 .5 .9 .1 1
s𝜏=1 = σ ( ln [ ] + ln [ ] [ ] + ln [ ] [ ])
2 .5 2 0 1 .5 .1 .9 0

−.3367 −.3367 −.0943

= σ ([ ]+[ ]+[ ])
−.3367 −.3367 −2.2073

−.7677
= σ ([ ])
−2.8806
138
Active Inference Tutorial

. 8922
=[ ]
. 1078

1 1 0 . 8922 .9 .1 0
s𝜏=2 = σ ( ln [ ][ ] + ln [ ] [ ])
2 0 1 . 1078 .1 .9 0

−0.0515 −4.6052
= σ ([ ]+[ ])
−1.0692 −4.6052

−4.6567
= σ ([ ])
−5.6744

. 7345
=[ ]
. 2655

Model Inversion: Time Step 2

1 .5 1 1 0 . 7345 .9 .1 1
s𝜏=1 = σ ( ln [ ] + ln [ ][ ] + ln [ ] [ ])
2 . 5 2 0 1 . 2655 .1 .9 0

−.3367 −.1475 −.0943

= σ ([ ]+[ ]+[ ])
−.3367 −.6446 −2.2073

−.5785
= σ ([ ])
−3.1886

. 9315
=[ ]
. 0685

1 1 0 . 9315 .9 .1 1
s𝜏=2 = σ ( ln [ ][ ] + ln [ ] [ ])
2 0 1 . 0685 .1 .9 0

−0.0301 −0.943
= σ ([ ]+[ ])
−1.2724 −2.2073

−0.1244
= σ ([ ])
−3.4797

. 9663
=[ ]
. 0337

139
Active Inference Tutorial

References

Adams, R., Shipp, S., & Friston, K. (2013). Predictions not commands: active inference in the motor
system. Brain Structure and Function, 218, 611-643.
Addicott, M., Pearson, J., Sweitzer, M., Barack, D., & Platt, M. (2017). A Primer on Foraging and the
Explore/Exploit Trade-Off for Psychiatry Research., 42, 1931-1939.
Andrews, M. (2020). The Math is not the Territory: Navigating the Free Energy Principle.
pittphilsci:18315.
Attias, H. (2000). A variational baysian framework for graphical models.
Attias, H. (2003). Planning by Probabilistic Inference. Paper presented at the Proc. of the 9th Int.
Workshop on Artificial Intelligence and Statistics.
Badcock, P. B., Friston, K. J., Ramstead, M. J. D., Ploeger, A., & Hohwy, J. (2019). The hierarchically
mechanistic mind: an evolutionary systems theory of the human brain, cognition, and behavior. Cogn
Affect Behav Neurosci, 19(6), 1319-1351.
Barto, A., Mirolli, M., & Baldassarre, G. (2013). Novelty or Surprise? Frontiers in Psychology, 4.
Beal, M. J. (2003). Variational algorithms for approximate Bayesian inference: university of London
London.
Bekinschtein, T., Dehaene, S., Rohaut, B., Tadel, F., Cohen, L., & Naccache, L. (2009). Neural signature of
the conscious processing of auditory regularities. Proceedings of the National Academy of Sciences of the
United States of America, 106, 1672-1677.
Bishop, C. M. (2006). Pattern recognition and machine learning. New York: Springer.
Bogacz, R. (2017). A tutorial on the free-energy framework for modelling perception and learning.
Journal of mathematical psychology, 76, 198-211.
Botvinick, M., & Toussaint, M. (2012). Planning as inference. Trends Cogn Sci., 16(10), 485-488.
Breakspear, M. (2017). Dynamic models of large-scale brain activity. Nat Neurosci, 20(3), 340-352.
Brown, L. D. (1981). A Complete Class Theorem for Statistical Problems with Finite-Sample Spaces.
Annals of Statistics, 9(6), 1289-1300.
Brown, T. H., Zhao, Y., & Leung, V. (2010). Hebbian plasticity. In Encyclopedia of Neuroscience (pp. 1049-
1056).
Bruineberg, J., Dolega, K., Dewhurst, J., & Baltieri, M. (2020). The Emperor’s New Markov Blankets.
pittphilsci:18467.
Bucci, A., & Grasso, M. (2017). Sleep and dreaming in the predictive processing framework. In T.
Metzinger & W. Wiese (Eds.), Philosophy and Predictive Processing): Johannes Gutenberg-Universität
Mainz.
Buckley, C. L., Sub Kim, C., McGregor, S., & Seth, A. K. (2017). The free energy principle for action and
perception: A mathematical review. Journal of Mathematical Psychology, 81, 55-79.
Burr, C., & Jones, M. (2016). The body as laboratory: Prediction-error minimization, embodiment, and
representation. Philosophical Psychology, 29(4), 586-600.
Carlin, B. P., & Louis, T. A. (1998). Bayes and empirical Bayes methods for data analysis. Boca Raton:
Chapman & Hall/CRC.
Champion, T., Grześ, M., & Bowman, H. (2021). Realising Active Inference in Variational Message
Passing: the Outcome-blind Certainty Seeker. Neural Computation.
Clark, A. (2013). Whatever next? Predictive brains, situated agents, and the future of cognitive science.
The Behavioral and brain sciences, 36, 181-204.
Clark, A. (2015). Surfing uncertainty: Prediction, action, and the embodied mind. New York: Oxford
University Press.

140
Active Inference Tutorial

Clark, A. (2017). How to Knit Your Own Markov Blanket. In T. K. Metzinger & W. Wiese (Eds.), Philosophy
and Predictive Processing). Frankfurt am Main: MIND Group.
Clark, J. E., Watson, S., & Friston, K. J. (2018). What is mood? A computational perspective. Psychological
Medicine, 1-8.
Da Costa, L., Parr, T., Sajid, N., Veselic, S., Neacsu, V., & Friston, K. J. (2020). Active inference on discrete
state-spaces: A synthesis. Journal of Mathematical Psychology, 99, 102447.
Da Costa, L., Parr, T., Sengupta, B., & Friston, K. (2021). Neural dynamics under active inference:
Plausibility and efficiency of information processing. Entropy, 23(4), 454.
Da Costa, L., Sajid, N., Parr, T., Friston, K. J., & Smith, R. (2020). The relationship between dynamic
programming and active inference: the discrete, finite-horizon case. arXiv, arXiv:2009.08111.
Dauwels, J. (2007). On variational message passing on factor graphs. IEEE International Symposium on
Information Theory, 2546-2550.
de Vries, B., & Friston, K. J. (2017). A Factor Graph Description of Deep Temporal Active Inference. Front
Comput Neurosci, 11, 95.
Fabry, R. E. (2017). Transcending the evidentiary boundary: Prediction error minimization, embodied
interaction, and explanatory pluralism. Philosophical Psychology, 30(4), 395-414.
Friston, K. J. (2019). A free energy principle for a particular physics. arXiv:1906.10184.
Friston, K. J., FitzGerald, T., Rigoli, F., Schwartenbeck, P., O Doherty, J., & Pezzulo, G. (2016). Active
inference and learning. Neuroscience and biobehavioral reviews, 68, 862-879.
Friston, K. J., FitzGerald, T., Rigoli, F., Schwartenbeck, P., & Pezzulo, G. (2017). Active Inference: A
Process Theory. Neural Computation, 29, 1-49.
Friston, K. J., Lin, M., Frith, C., Pezzulo, G., Hobson, J., & Ondobaka, S. (2017). Active Inference, Curiosity
and Insight. Neural Computation, 29, 2633-2683.
Friston, K. J., Litvak, V., Oswal, A., Razi, A., Stephan, K. E., van Wijk, B. C. M., et al. (2016). Bayesian
model reduction and empirical Bayes for group (DCM) studies. NeuroImage, 128, 413-431.
Friston, K. J., Mattout, J., Trujillo-Barreto, N., Ashburner, J., & Penny, W. (2007). Variational free energy
and the Laplace approximation. NeuroImage, 34, 220-234.
Friston, K. J., Parr, T., & de Vries, B. (2017). The graphical brain: Belief propagation and active inference.
Network Neuroscience, 1, 381-414.
Friston, K. J., Rosch, R., Parr, T., Price, C., & Bowman, H. (2018). Deep temporal models and active
inference. Neuroscience & Biobehavioral Reviews, 90, 486-501.
Hesp, C., Smith, R., Allen, M., Friston, K. J., & Ramstead, M. J. D. (2020). Deeply Felt Affect: The
Emergence of Valence in Deep Active Inference. Neural Computation, 1-49.
Hobson, J., & Friston, K. (2012). Waking and dreaming consciousness: neurobiological and functional
considerations. Progress in neurobiology, 98, 82-98.
Hobson, J., Hong, C.-H., & Friston, K. (2014). Virtual reality and consciousness inference in dreaming.
Frontiers in Psychology, 5, 1133.
Hohwy, J. (2014). The Predictive Mind.
Hohwy, J. (2016). The Self-Evidencing Brain. Noûs, 50, 259-285.
Hohwy, J., Paton, B., & Palmer, C. (2016). Distrusting the present. Phenomenology and the Cognitive
Sciences, 15(3), 315-335.
Kaplan, R., & Friston, K. J. (2018). Planning and navigation as active inference. Biol Cybern, 112(4), 323-
343.
Kuczma, M., & Gilányi, A. (2009). An introduction to the theory of functional equations and inequalities :
Cauchy's equation and Jensen's inequality (2nd ed.). Basel ; Boston, MA: Birkäeuser.
Loeliger, H. A. (2004). An introduction to factor graphs. IEEE Signal Processing Magazine, 21(1), 28-41.
Markovic, D., Stojic, H., Schwoebel, S., & Kiebel, S. (2021). An empirical evaluation of active inference in
multi-armed bandits. arXiv:2101.08699.

141
Active Inference Tutorial

Mathys, C. D., Lomakina, E. I., Daunizeau, J., Iglesias, S., Brodersen, K. H., Friston, K. J., et al. (2014).
Uncertainty in perception and the Hierarchical Gaussian Filter. Front Hum Neurosci, 8, 825.
Millidge, B. (2019). Combining Active Inference and Hierarchical Predictive Coding: A Tutorial
Introduction and Case Study. PsyArXiv.
Mirza, M. B., Adams, R. A., Mathys, C., & Friston, K. J. (2018). Human visual exploration reduces
uncertainty about the sensed world. PLOS ONE, 13, e0190429.
Oudeyer, P.-Y., & Kaplan, F. (2007). What is intrinsic motivation? a typology of computational
approaches. Frontiers in Neurorobotics, 1, 6.
Parr, T., & Friston, K. J. (2017a). Uncertainty, epistemics and active inference. Journal of the Royal
Society, Interface, 14.
Parr, T., & Friston, K. J. (2017b). Working memory, attention, and salience in active inference. Scientific
Reports, 7, 14678.
Parr, T., & Friston, K. J. (2018a). The Anatomy of Inference: Generative Models and Brain Structure.
Frontiers in Computational Neuroscience, 12, 90.
Parr, T., & Friston, K. J. (2018b). The Discrete and Continuous Brain: From Decisions to Movement—and
Back Again. Neural Computation, 1-29.
Parr, T., Markovic, D., Kiebel, S., & Friston, K. J. (2019). Neuronal message passing using Mean-field,
Bethe, and Marginal approximations. Scientific Reports, 9, 1889.
Pezzulo, G., Rigoli, F., & Friston, K. J. (2015). Active Inference, homeostatic regulation and adaptive
behavioural control. Progress in neurobiology, 134, 17-35.
Pezzulo, G., Rigoli, F., & Friston, K. J. (2018). Hierarchical Active Inference: A Theory of Motivated
Control. Trends in Cognitive Sciences, 22, 294-306.
Ramachandran, V. S. (1988). Perceiving shape from shading. Scientific American, 259(2), 76–83.
Rigoux, L., Stephan, K. E., Friston, K. J., & Daunizeau, J. (2014). Bayesian model selection for group
studies - revisited. Neuroimage, 84, 971-985.
Sajid, N., Ball, P., Parr, T., & Friston, K. (2021). Active Inference: Demystified and Compared. Neural
Computation, 1-39.
Sales, A. C., Friston, K. J., Jones, M. W., Pickering, A. E., & Moran, R. J. (2019). Locus Coeruleus tracking of
prediction errors optimises cognitive flexibility: An Active Inference model. PLoS Comput Biol, 15(1),
e1006267.
Sandved-Smith, L., Hesp, C., Mattout, J., Friston, K., Lutz, A., & Ramstead, M. J. (2021). Towards a
computational phenomenology of mental action: modelling meta-awareness and attentional control
with deep parametric active inference. Neuroscience of consciousness, 2021(2), niab018.
Schmidhuber, J. (2006). Developmental robotics, optimal artificial curiosity, creativity, music, and the
fine arts. Connection Science, 18(2), 173-187.
Schwartenbeck, P., FitzGerald, T., Mathys, C., Dolan, R., & Friston, K. J. (2015). The Dopaminergic
Midbrain Encodes the Expected Certainty about Desired Outcomes. Cerebral Cortex, 25, 3434-3445.
Schwartenbeck, P., Passecker, J., Hauser, T. U., FitzGerald, T. H., Kronbichler, M., & Friston, K. J. (2019).
Computational mechanisms of curiosity and goal-directed exploration. Elife, 8.
Smith, R., Badcock, P. B., & Friston, K. J. (2020). Recent advances in the application of predictive coding
and active inference models within clinical neuroscience. Psychiatry and Clinical Neurosciences.
Smith, R., Khalsa, S. S., & Paulus, M. P. (2021). An Active Inference Approach to Dissecting Reasons for
Nonadherence to Antidepressants. Biol Psychiatry Cogn Neurosci Neuroimaging, 6(9), 919-934.
Smith, R., Kirlic, N., Stewart, J., Touthang, J., Kuplicki, R., McDermott, T., et al. (2021). Long-term Stability
of Computational Parameters During Approach-avoidance Conflict in a Transdiagnostic Psychiatric
Patient Sample. Scientific Reports, 11.

142
Active Inference Tutorial

Smith, R., Kirlic, N., Stewart, J. L., Touthang, J., Kuplicki, R., Khalsa, S. S., et al. (2021). Greater decision
uncertainty characterizes a transdiagnostic patient sample during approach-avoidance conflict: a
computational modeling approach. Journal of Psychiatry & Neuroscience, 46(1), E74-E87.
Smith, R., Kuplicki, R., Feinstein, J., Forthman, K. L., Stewart, J. L., Paulus, M. P., et al. (2020). A Bayesian
computational model reveals a failure to adapt interoceptive precision estimates across depression,
anxiety, eating, and substance use disorders. PLoS Computational Biology, 16(12), e1008484.
Smith, R., Kuplicki, R., Teed, A., Upshaw, V., & Khalsa, S. S. (2020). Confirmatory evidence that healthy
individuals can adaptively adjust prior expectations and interoceptive precision estimates. In T.
Verbelen, P. Lanillos, C. Buckley & C. De Boom (Eds.), Active Inference. IWAI 2020.): Springer, Cham.
Smith, R., Lane, R. D., Parr, T., & Friston, K. J. (2019). Neurocomputational mechanisms underlying
emotional awareness: Insights afforded by deep active inference and their potential clinical relevance.
Neurosci Biobehav Rev, 107, 473-491.
Smith, R., Mayeli, A., Taylor, S., Al Zoubi, O., Naegele, J., & Khalsa, S. S. (2021). Gut inference: A
computational modelling approach. Biol Psychol, 108152.
Smith, R., Parr, T., & Friston, K. J. (2019). Simulating Emotions: An Active Inference Model of Emotional
State Inference and Emotion Concept Learning. Front Psychol, 10, 2844.
Smith, R., Ramstead, M. J. D., & Kiefer, A. (2021). Active inference models do not contradict folk
psychology. PsyArXiv: [Link]
Smith, R., Schwartenbeck, P., Parr, T., & Friston, K. J. (2020). An Active Inference Approach to Modeling
Structure Learning: Concept Learning as an Example Case. Front Comput Neurosci, 14, 41.
Smith, R., Schwartenbeck, P., Stewart, J. L., Kuplicki, R., Ekhtiari, H., Investigators, T., et al. (2020).
Imprecise Action Selection in Substance Use Disorder: Evidence for Active Learning Impairments When
Solving the Explore-exploit Dilemma. Drug and Alcohol Dependence, 215, 108208.
Stephan, K., Penny, W. D., Daunizeau, J., Moran, R. J., & Friston, K. J. (2009). Bayesian model selection
for group studies. Neuroimage, 46(4), 1004-1017.
Stephan, K. E., Petzschner, F. H., Kasper, L., Bayer, J., Wellstein, K. V., Stefanics, G., et al. (2019). Laminar
fMRI and computational theories of brain function. Neuroimage, 197, 699-706.
Tononi, G., & Cirelli, C. (2014). Sleep and the Price of Plasticity: From Synaptic and Cellular Homeostasis
to Memory Consolidation and Integration. Neuron, 81, 12-34.
Tschantz, A., Barca, L., Maisto, D., Buckley, C. L., Seth, A. K., & Pezzulo, G. (2021). Simulating
homeostatic, allostatic and goal-directed forms of interoceptive control using Active Inference. bioRxiv,
2021.2002.2016.431365.
Tschantz, A., Seth, A. K., & Buckley, C. L. (2020). Learning action-oriented models through active
inference. PLoS Comput Biol, 16(4), e1007805.
Tu, S. (2014). The dirichlet-multinomial and dirichlet-categorical models for bayesian inference. Berkeley,
CA: Computer Science Division, UC Berkeley.
Wald, A. (1947). An Essentially Complete Class of Admissible Decision Functions. The Annals of
Mathematical Statistics, 549-555.
Whyte, C., Hohwy, J., & Smith, R. (2021). An Active Inference Model of Conscious Access with and
without Report. PsyArXiv: [Link]
Whyte, C., & Smith, R. (2020). The Predictive Global Neuronal Workspace: A Formal Active Inference
Model of Visual Consciousness. Progress in Neurobiology, 2020.2002.2011.944611.
Wilson, R., Geana, A., White, J., Ludvig, E., & Cohen, J. (2014). Humans use directed and random
exploration to solve the explore-exploit dilemma. Journal of experimental psychology. General, 143,
2074-2081.
Winn, J., & Bishop, C. M. (2005). Variational message passing. Journal of Machine Learning Research, 6,
661-694.

143
 Active Inference Tutorial

Zeidman, P., Jafarian, A., Seghier, M. L., Litvak, V., Cagnan, H., Price, C. J., et al. (2019). A guide to group
effective connectivity analysis, part 2: Second level analysis with PEB. Neuroimage, 200, 12-25.

144

View publication stats