iScience ll

OPEN ACCESS

Review
How context changes the neural
basis of perception and language
Roel M. Willems1,2,3,4,* and Marius V. Peelen1,4,*

SUMMARY
Cognitive processes—from basic sensory analysis to language understanding—
are typically contextualized. While the importance of considering context for un-
derstanding cognition has long been recognized in psychology and philosophy, it
has not yet had much impact on cognitive neuroscience research, where cognition
is often studied in decontextualized paradigms. Here, we present examples of
recent studies showing that context changes the neural basis of diverse cognitive
processes, including perception, attention, memory, and language. Within the
domains of perception and language, we review neuroimaging results showing
that context interacts with stimulus processing, changes activity in classical
perception and language regions, and recruits additional brain regions that
contribute crucially to naturalistic perception and language. We discuss how
contextualized cognitive neuroscience will allow for discovering new principles
of the mind and brain.

INTRODUCTION
Observing two people shouting at each other would usually evoke a strong emotional reaction in the
observer; this depends however on the context in which the event takes place: observing the same social
interaction in the context of a film shoot may evoke a very different reaction. The effect of context is perva-
sive and present at multiple levels of processing—from visual perception to language. For example, a
distant object will be perceived as a boat when situated on a lake but as a car when situated on a road;
and the sentence ‘‘sometimes it is better to talk less’’ can be a general advice in a presentation course
or a request to shut up in a meeting.

In addition to qualitatively changing cognitive processing, context also facilitates processing, allowing us
to rapidly understand complex scenes and events. For example, context allows for predicting what some-
one will do next, facilitating the processing of that action when it is executed. The ease with which we move
around and get things done in our familiar environments attests to how well suited our brains are to navi-
gate seemingly complex cognitive spaces. Clearly, the brain is well adapted and intricately coupled to its
rich natural habitat. Because of the strong influence of context, we argue that cognitive neuroscience needs
to be (re-)contextualized: considering natural context should be the starting point of cognitive neurosci-
ence research. 1Donders Institute for Brain,
Cognition and Behaviour,
Radboud University,
When browsing articles in cognitive neuroscience journals, it becomes clear that this is rarely done. Instead, Nijmegen, the Netherlands
the focus is on understanding the neural mechanisms of highly artificial and decontextualized cognitive 2Centre for Language
tasks, such as detecting oriented gratings in rapid visual streams, memorizing multiple simple stimuli Studies, Radboud University,
across brief delays, and speeded reading of letter sequences (Figure 1). The contrast with cognition as it Nijmegen, the Netherlands
3Max Planck Institute for
operates in real life is stark. In real life, we perceive, read, think, and act in rich familiar environments,
Psycholinguistics, Nijmegen,
set within a particular time, place, and culture. the Netherlands
4These authors contributed
These considerations raise the question of how well cognition can be approximated by decontextualized equally
laboratory tasks. Do the cognitive processes isolated by these tasks operate similarly in the real world? Is it *Correspondence:
valid to reduce complex behavior to a collection of isolated components? Can we study cognition without [Link]@[Link] (R.M.W.),
[Link]@[Link]
considering context? Some of these issues have featured prominently in psychology and philosophy (Clark, (M.V.P.)
1997; Gibson, 1979; Hutchins, 1996; Neisser, 1976; Varela et al., 1991), but they have not yet significantly [Link]
influenced the field of cognitive neuroscience. 2021.102392

iScience 24, 102392, May 21, 2021 ª 2021 The Author(s). 1

This is an open access article under the CC BY license ([Link]
 ll iScience
OPEN ACCESS Review

Figure 1. Contextualizing cognitive neuroscience

Can traditional research stimuli be used to understand contextualized cognition?
(A) Artificial visual search displays (upper left) are often used to study the neural basis of visual attention (‘‘find the red –’’).
It is unclear, however, if or how neural mechanisms involved in artificial visual search are similarly implicated in searching
for objects in natural scenes (e.g. bicyclists in the upper right).
(B) The lexical decision task (lower left) is used to study the neural basis of language understanding. The link with language
understanding in the real world (e.g. when reading a book, lower right) is not clear. The crucial question is whether the
brain processes investigated with decontextualized paradigms (left) can be translated to contextualized settings (right). In
the paper, we argue that the cognitive and neural processes involved in contextualized cognition can be qualitatively
different than those in decontextualized cognition. Brain icon from [Link]

What do we mean by context? According to dictionaries, context is ‘‘The situation within which something
exists or happens, and that can help explain it’’ (Cambridge Dictionary), or ‘‘The interrelated conditions in
which something exists or occurs’’ (the Merriam-Webster Dictionary). Although these definitions are admit-
tedly broad, they help point out how context implies situatedness. Context can relate to situatedness at
different spatial and temporal scales. For example, context can be of short or long duration, forms that
are unlikely to be comparable: the context of a social setting is not comparable to the context of a personal
lifetime history. Taken to the extreme, one may argue that the specific context in which a cognitive act takes
place is always different, precluding empirical study. In practice however, specific contexts often share
structure that makes them generalizable. In this paper, we will show examples of context that have effects
that generalize across specific instances and that can thus be studied experimentally.

We now turn to the main argument of the paper: Context can qualitatively change cognitive processing in a
way that makes it hard to compare to seemingly similar processes studied in a decontextualized manner. It
is common in cognitive neuroscience to consider ‘‘context’’ as an add-on to core cognitive processing.
Context is traditionally considered like a Maggi seasoning, ‘‘spicing up’’ the core of cognitive processing.
This is unlikely to be correct. Quite on the contrary, context is not something we add on to cognitive op-
erations, it is core to cognition. Without context, cognitive tasks are often difficult and time consuming.
With context, many of these tasks become effortless and fast. Context thus both facilitates and changes
cognitive processing, as we will see in later examples.

In this paper, we show why incorporating context provides an excellent opportunity for making progress in
cognitive neuroscience. We argue that contextualizing research will increase the relevance of research find-
ings for understanding the neural mechanisms of cognition. Relevance is to be understood here in relation
to so-called representative design (Brunswik, 1955): Is the research environment representative of the envi-
ronment the researcher wants to generalize to? For example, findings from research that investigates visual
attention in environments that are far removed from real-world environments may not necessarily gener-
alize to real-world conditions and may thus not be optimally relevant for understanding attention.

After brief theoretical considerations, the main part of the paper consists of examples of how taking
context into account has changed our views on the neural basis of cognition. Our examples will be focused

2 iScience 24, 102392, May 21, 2021

iScience ll
Review OPEN ACCESS

on visual perception and language since these are our fields of expertise. We believe the paper to be of
interest to (cognitive) neuroscientists at large since similar examples could be given regarding other sub-
fields (we refer to a few examples later in the paper). It should be clear that our examples are not meant to
be an exhaustive list; they are meant to illustrate our larger point. Finally, in the discussion section, we take
stock and describe how our approach relates to other positions. We end the paper by giving recommen-
dations on how contextualization can be used to move cognitive neuroscience forward.

RELATION TO THE TRADITIONAL APPROACH

In order to make the case for a contextualized approach, we will first describe how it deviates from the tradi-
tional approach to research in cognitive neuroscience. The traditional approach is to isolate small subcom-
ponents of a cognitive process and study those in a decontextualized manner. The overarching rationale of
this approach is that the subcomponents can be combined in order to reach understanding of the com-
plete process. In this approach, each subcomponent is a building block, comparable to a piece of Lego
that when correctly put together forms a unity, just like separate Lego pieces can form a Lego castle. How-
ever, as we will see, context typically interacts with cognitive and neural processing in ways that are not pre-
dicted from studying components in isolation, making this assumption unlikely to be correct. This approach
has its historical roots in the logic of pure insertion (Donders, 1869). Despite the fact that it has long been
recognized that this logic is untenable (Friston et al., 1996; Sternberg, 1969), the traditional approach con-
tinues to be the leading paradigm in cognitive neuroscience.

The popularity of the traditional approach is based on the advantage it is believed to have for experimen-
tation: if subcomponents can be studied in isolation, context can be ignored, which makes experimentation
easier. An example of how this can be problematic comes from work on the cortical motor system in the
macaque monkey. A classical view of the motor cortex is that it is organized somatotopically. This means
that each effector (each finger, each hand, each foot, etc.) is controlled by a distinct part of the motor cor-
tex. There is ample evidence for such an organization. Interestingly, however, Graziano and colleagues
observed that the classical somatotopy of the motor cortex is variable and state dependent (Graziano
and Aflalo, 2007). By varying the timing of cortical stimulation (a factor which was not prominent in earlier
work), it was observed that the mapping of actions in the motor cortex does not consist of a neat delinea-
tion according to effector but is better characterized by a mapping according to behaviorally relevant ac-
tions, like ‘‘reach to grasp’’ or ‘‘hand to mouth’’. This example shows that the functional organization of the
motor cortex will not be correctly understood when mapping each effector separately and assuming that a
combination of these mappings together describes the neural basis of complex movements.

A perspective opposed to the traditional approach comes from dynamical systems theory, a general math-
ematical framework used to describe dynamical systems. In a dynamical system, there is a strong depen-
dence on time for a system to move from being in one state to the other (Chiel and Beer, 1997; Kiverstein
and Miller, 2015; Thelen and Smith, 1994; Turkheimer et al., 2021; Ward, 2002). In a dynamical systems
framework, it is uninformative to consider components in isolation. The reason for this is that subcompo-
nents of a system are inherently coupled so that they interact in complex, time-dependent nonlinear ways.
What is important for the present paper is that when viewed as a dynamical system, it is not informative to
investigate neural processing outside of a context since context will have an immediate and complex effect
on neural processing. We merely describe the dynamical systems approach as one of several theoretical
viewpoints that argue against the traditional building blocks approach to studying cognition. This spirit
is also part of enacted and embodied cognition to which we will turn toward the end of the paper.

Now that we have set the stage, we will move to the heart of the paper. We will give examples of how study-
ing cognition within a naturally occurring context leads to a different understanding of the brain basis of
cognition as compared to when studying it outside of a context.

CONTEXT IN PERCEPTION, ATTENTION, AND MEMORY

Object perception
The dominant approach in the field of object perception is to measure neural responses evoked by individ-
ual isolated objects (or object features) presented at central fixation. While these studies are relevant to our
understanding of basic visual processing, they miss important contextual contributions to object recogni-
tion. Indeed, object processing is influenced by the identity and position of surrounding objects (Quek

iScience 24, 102392, May 21, 2021 3

 ll iScience
OPEN ACCESS Review

A B

Figure 2. Context effects in object perception

(A) An example of a poorly visible (degraded) object that is unrecognizable when presented outside of scene context (top
picture) but easily recognizable when presented within scene context (bottom picture).
(B) fMRI data showing that activity patterns in object-selective visual cortex (region shown in top) provide more
information (i.e., better decoding accuracy) about the category of objects when presented within scenes (orange bar) vs
outside of scenes (light blue bar). Furthermore, the above chance decoding accuracy for object-in-scene is more than the
sum of the decoding accuracies of the object and scene shown separately, indicating an interaction between scene
context and object processing. Error bars indicate standard error of the mean. **p<0.01. Figure adapted from Brandman
and Peelen (2017) under CC BY 4.0 license.

and Peelen, 2020) as well as the global scene context. For example, objects presented in a coherent scene
context (e.g., a car on a road) are more easily recognized than objects presented in an incoherent scene
context (e.g., a car on a lake; Bar, 2004; Oliva and Torralba, 2007). This effect can be so strong that objects
that would be unrecognizable when viewed in isolation become readily recognizable when viewed in
context (Figure 2). These results imply the existence of a context-based route to object recognition, indi-
cating that isolated object processing is only one pathway to object recognition. A recent study showed
that functional magnetic resonance imaging (fMRI) and magnetoencephalography activity patterns in the
visual cortex evoked by degraded objects in context closely resembled activity patterns evoked by clearly
visible objects presented in isolation, despite the lack of object features in the degraded objects (Brandman
and Peelen, 2017). These effects could not be explained by additive (parallel) processing of the degraded
object and scene when presented separately (Figure 2). This provides evidence for an interaction between
simultaneously presented objects and scenes, whereby objects are recognized through the integration of
context-based expectations and isolated object features.

Visual attention
Context is also crucial for attentional selection, for example, during visual search. Visual search is the task of
finding relevant objects among irrelevant objects. This is a common task in daily life, for example, when we
search for a friend at the airport, for our keys on the table, or for coffee in the supermarket. The neural
mechanisms of visual search have mostly been studied using highly simplified displays, typically consisting
of arrays of artificial shapes (see Figure 1 for an example). These studies have been fundamentally impor-
tant for revealing basic neural mechanisms involved in perception and attention. However, later empirical
work showed that these studies do not fully explain how the brain rapidly selects familiar objects from com-
plex but meaningful natural scenes (Peelen and Kastner, 2014). One of the reasons for this is that natural
scenes give a powerful context that provides information about likely target locations as well as likely target
properties, thereby strongly constraining visual search: we start looking for a pen on top of a desk not on
the floor (Castelhano and Krzys, 2020; Oliva and Torralba, 2007; Võ et al., 2019); and we expect objects to be

4 iScience 24, 102392, May 21, 2021

iScience ll
Review OPEN ACCESS

large when nearby and small when far away. Targets are easily missed when these regularities are violated,
for example, when the size of targets is inconsistent with the scene context (Eckstein et al., 2017). Recent
studies have started to investigate the neural basis of visual search in more naturalistic conditions (e.g. Çu-
kur et al., 2013; Nardo et al., 2011; Peelen and Kastner, 2011; Preston et al., 2013; Summerfield et al., 2006).
One of the findings from this research is that attention to high-level object categories (e.g., people) mod-
ulates neural responses to natural scenes across the visual field, independently of spatial attention (Peelen
et al., 2009). Such spatially global attention effects were previously considered to be unique to low-level
features such as orientation, color, or motion direction (Carrasco, 2011). This is an example of how the brain
is optimized to perform naturalistic tasks (e.g., detecting people in scenes), efficiently directing attention to
complex objects based on high-level object representations and scene context.

Visual memory
The presence of scene context is equally important for visual memory. Research has shown that we encode
and remember only very few details of a natural scene at any given moment, such that even large (but un-
expected) changes across views are easily missed (change blindness; Rensink et al., 1997). Despite this
objectively poor encoding of scenes, our daily life experience is of a rich, complete, and continuous visual
world. This experience may be an illusion, reflecting our ability to access information in the scene as
‘‘external memory’’ at any moment in time (O’Regan, 1992). In the real world, scenes are typically stable
and continuously present, such that there is usually no need to encode everything in memory. Rather,
the observer only needs to have a coarse representation of the scene to know where relevant information
can be found, such that attention and eye movements can be directed to relevant information when
needed. Indeed, research has shown that participants make frequent eye movements rather than encode
items in working memory when performing naturalistic tasks (Ballard et al., 1995; Draschkow et al., 2021).
This implies that the limited capacity of visual working memory (to 3–4 items), a major field of study in cogni-
tive neuroscience, may not often limit performance in daily life (Van der Stigchel, 2020). The neural basis of
interactions between internal and external memory remains largely unexplored, opening up interesting
research questions with great relevance for understanding memory deficits in neuropsychological disor-
ders and cognitive aging.

Summary
These examples highlight how visual perception, attention, and memory depend on the presence of natu-
ralistic scene context. Visual, attention, and memory systems have developed and evolved to optimally
perform real-world tasks, as also reflected in the remarkable speed of natural scene perception (Li et al.,
2002; Potter, 1976; Thorpe et al., 1996). It is thus increasingly appreciated that the brain makes use of a
wide range of available information (e.g., scene context, statistical regularities) to support naturalistic
perception, attention, and memory (Kaiser et al., 2019). The neural basis of these processes in contextual-
ized settings is still unclear; we expect that much progress can be made in this area in the coming years.

CONTEXT IN LANGUAGE COMPREHENSION

Decades of cognitive neuroscience research has investigated the neural basis of decontextualized lan-
guage comprehension, focusing on the understanding of single words and sentences. Naturally, during
actual language comprehension, words and sentences can be combined to form larger pieces of discourse
such as narratives. In this section, we will show that studying language at the level of decontextualized
words and sentences gives an incomplete picture of the neural basis of language. Moreover, context
not only extends the brain regions involved in language, it can also alter the neural processes involved
in understanding language.

Context extends the language network in the brain

The traditional neural network for language consists of areas in the middle and anterior temporal cortex,
inferior frontal regions, and the inferior parietal lobe (e.g., Binder et al., 2009; Fedorenko and Thompson-
Schill, 2014; Hagoort, 2013). Interestingly, once language is processed at the level beyond single senten-
ces, other regions become involved. Examples include information extraction over longer periods of time
(e.g., paragraphs within a narrative), which is associated with activation in posterior midline structures such
as the precuneus (Jacobs and Willems, 2018; Lerner et al., 2011; Zacks et al., 2010). Another example is the
role of anterior medial prefrontal regions thought to be involved in understanding the intentions and be-
liefs of others (Ferstl et al., 2008; Hagoort and Levinson, 2014; Mason and Just, 2006; Willems and Varley,

iScience 24, 102392, May 21, 2021 5

 ll iScience
OPEN ACCESS Review

Figure 3. Context effects in language comprehension

A text understood within a valid context involves different brain regions than understanding the same text within an invalid context. Participants read a
description of an activity, in this example a description of how to change a flat car tire. The description could be preceded by a picture that fitted the
description (someone changing a tire; upper left) or not (someone sitting at a fireplace; lower left). The written description is much easier to understand when
preceded by a matching picture (valid trials), as compared to when preceded by a non-matching picture. Note that the sentence in the figure was only the
start of the description, and the actual stimulus was much longer. Areas of the traditional language network (e.g. middle temporal and inferior frontal
regions), as well as areas not traditionally thought to be involved in language comprehension, such as the posterior cingulate cortex (PCC) and anterior
medial prefrontal cortex (mPFC), are sensitive to this manipulation, as shown here through analyses of inter-subject similarity of fMRI activity. ISC = inter-
subject correlation; A = anterior; P = posterior; A1+ = primary auditory cortex; L = left; SMG = supramarginal gyrus; STS = superior temporal sulcus; AG =
angular gyrus. Figure adapted from Ames et al. (2015), with permission from MIT Press Journals.

2010). In contextualized language comprehension, understanding the intentions of others is so important

that it becomes hard to see how the process is not part and parcel of natural language. Put differently, the
picture of the language network in the brain changes substantially when contextualized language is studied
instead of single words and sentences.

An example case in point is a study in which discourse coherence was investigated (Ames et al., 2015). Par-
ticipants read short descriptions (several sentences long) of basic activities (e.g. change a tire on a car; see
Figure 3). When the text was preceded by a picture of someone changing a tire, the description was easy to
understand. When presented with a different, unrelated picture, the description was hard to understand. It
was found that traditional language areas (e.g., middle temporal and inferior frontal areas) were activated
in both conditions (Figure 3). The presence or absence of context did not modulate activity in these areas.
By contrast, posterior and anterior midline structures (anterior medial prefrontal cortex and posterior
cingulate cortex) were more strongly activated when the content of the description could be more easily
understood because of the presence of valid context cues (Ames et al., 2015; see also Heidlmayr et al.,
2020). These studies show that adding a meaningful context to even a relatively simple piece of language
changes the neural areas involved. Some have hypothesized that it is therefore better to either extend the
language network to include these additional regions (Ferstl et al., 2008) or to let go of the notion of a lan-
guage network altogether (Hasson et al., 2018). Either way, these findings make clear that context funda-
mentally changes the neural basis of language.

Contextualized grammar
Next to changing the areas involved in language comprehension, context can modulate activation in
‘‘core’’ language areas. Typically, context makes language comprehension easier. An interesting illustra-
tion is the context sensitivity of syntactic comprehension (‘‘grammar’’), a part of language comprehension
considered crucial by many. In an fMRI study, participants were presented with subject-initial sentences
(‘‘He noticed her’’) and object-initial sentences (‘‘Her, he noticed’’; Kristensen et al., 2014). When presented
on their own, the object-initial reading is most difficult (both are grammatically correct in Danish). The sen-
tences were presented in two conditions: one absent of context and one where a brief preceding context
rendered the (more complex) object-initial sentence more acceptable. For instance, when preceded by the

6 iScience 24, 102392, May 21, 2021

iScience ll
Review OPEN ACCESS

sentence ‘‘Peter overlooked all the shoplifters except Anne’’, it is the object-initial continuation ‘‘Her, he
noticed’’ that is more natural (see also Mak et al., 2008). The authors found that activity in the left inferior
frontal gyrus was reduced when context licensed the object-initial continuation, as compared to when the
sentences were presented on their own. These results illustrate that findings from single-sentence, out-of-
context studies of language do not necessarily extend to findings obtained with language materials that
are contextualized, even if the context is rather minimal.

Context makes language understanding easier

The example of ‘‘difficult’’ sentences (‘‘Her, he noticed’’) becoming easy to understand within context is
likely to be a general rule. In a compelling conceptual analysis, Van Duijn and colleagues argue that complex
multiple mental embeddings that are very difficult to understand outside of a contextual embedding are
actually the norm in famous works of fiction (Van Duijn et al., 2015). To illustrate, take the following sentence:

‘‘I know that Ann believes that Peter suspects that Mary realizes that Bill understood that Will is
a sailor.’’ (Van Duijn et al., 2015)

This sentence shows a mental embedding of 5 degrees. It is very difficult to understand: Try to quickly say what
Peter is suspecting about Bill? Knowing who is thinking or believing what about whom is like a puzzle in this
example, taking considerable effort. It has been found that areas in the inferior frontal cortex are sensitive to
these multiple embeddings (Grodzinsky and Santi, 2008), and it has been argued that this is evidence for the
region’s functional role in syntax. However, in works of fiction, this kind of embedding is paramount. For
example, in Shakespeare’s Othello, there are points in the play in which the number of embeddings exceeds
five (character 1 thinks of character 2 that she believes that character 3 suggests, etc.; Van Duijn et al., 2015).
Spectators not only find such embeddings easy to understand in the context of Shakespeare, they might
even be attracted to them, spending time and money to see the play. Therefore, what is difficult outside of
context becomes easy to understand and actually enjoyable within a context.

A similar case can be made for (linguistic) perspective switching. Perspective switching (e.g. seeing the
world from your own versus from another person’s point of view) is cognitively costly, as evidenced by re-
action times (e.g. Vukovic and Williams, 2015). However, within a narrative context, perspective shifts occur
repeatedly. Moreover, these are not just switches from one perspective to another but are rather complex,
multiple person switches occurring within one paragraph, even in ordinary newspaper articles (van Krieken
et al., 2019). Details left aside, the point is that context eases the cognitive load of these processes.

This begs the question whether it is justifiable to study processes such as mental embeddings and perspec-
tive switching outside of context. What is the relevance of knowing what happens when we read isolated
sentences with difficult embedding when we, under normal circumstances, have context to make compre-
hension so much easier?

We end this section with a note about another core process in language: word comprehension. Many studies
have sampled people’s reactions to single words, either using brain measures, reaction times, or ratings, to
understand how words are processed by the brain (Meteyard and Vigliocco, 2018). On the one hand, insights
and ratings gained from these studies correlate with reading measures in more contextualized settings (e.g.,
lexical frequency during natural reading of narratives; Mak and Willems, 2019). On the other hand, eye move-
ment measures (e.g., gaze duration) are strikingly different when reading the same words within a novel
versus within short paragraphs (Dirix et al., 2019). The shared variance between reading under these
different circumstances was as low as 1%–14%, depending on the eye movement measure used (Dirix
et al., 2019). This finding suggests that the overall context in which words (the same words) are read has a
much larger influence on reading times than word characteristics of the words in isolation. Studies that mea-
sure responses to words in isolation by definition miss out on these important contextual effects.

Summary
Language in context is understood and processed differently than decontextualized language. The lan-
guage network in the brain as established from decontextualized language stimuli is different from the
network of areas involved in understanding language at the discourse level (at the level of multi-sentence
units such as narratives). Not only do additional brain regions play a role during contextualized language

iScience 24, 102392, May 21, 2021 7

 ll iScience
OPEN ACCESS Review

comprehension, it also changes the response of language areas. Phenomena that are difficult to under-
stand without context, such as syntactically complicated sentences, become easy to process once
embedded in a context. Taking context seriously will change the way we think about the neural basis of
language (Hasson et al., 2018; Willems et al., 2020). It provides ample opportunities to make future research
on language and the brain more relevant to understanding this uniquely human capacity.

DISCUSSION
What are the implications for cognitive neuroscience of taking context seriously? In the remainder, we show
how our position can help to guide future experimentation. Our hope is that doing our science differently
will increase the relevance of cognitive neuroscience for understanding the workings of the brain in its nat-
ural habitat.

4E philosophy
Although not mainstream in cognitive neuroscience, thinking of cognition in context has a strong theoretical
basis in philosophy of mind. The central tenet of so-called 4E (embedded, embodied, extended, enacted)
cognition is that cognition should be thought of as resulting from the interplay between brain, body, and envi-
ronment. Proponents of 4E cognition consider it a conceptual mistake to understand cognition solely as the pro-
cessing of internalized models by a brain. Instead, cognition is what happens when the brain and body act in the
rich environment that the brain and body are used to act in. As a result, the right unit of analysis is not the brain
but the coupled system brain-body-environment (e.g. Chemero, 2009; Newel et al., 2018).

One example of this coupling is cognitive off-loading, in which the environment is used as an external memory
(see also the section on visual memory above). The thesis underlying these examples is that of the extended
mind, part of 4E cognition. Clark and Chalmers (1998) described ‘‘cognitive off-loading’’ as follows:

‘‘In these cases, the human organism is linked with an external entity in a two-way interaction, creating a
coupled system that can be seen as a cognitive system in its own right. All the components in the system
play an active causal role, and they jointly govern behavior in the same sort of way that cognition usually
does.’’ (Clark and Chalmers, 1998)

Other examples of cognitive off-loading are the use of devices (computers, smartphones) to remember or
organize information or to navigate one’s environment. Here, we see that cognition is not constrained to
the brain and body but is actively extended to the environment (Slors, 2020). Another telling example of the
influence of cognitive off-loading comes from research on aging. It is well established that tasks requiring
cognitive control and memory decline with age. Interestingly, however, this decline is much less pro-
nounced—or even absent—when older adults are tested in contextualized settings (Phillips et al., 2006).
An example of a standard cognitive control task is the Tower of London task (requiring complicated plan-
ning), whereas an example of a situated task is buying items in a grocery store (and plan other actions, im-
plemented in multiple errands test; Alderman et al., 2003; Phillips et al., 2006). Indeed, performance on the
decontextualized task declines with age, whereas performance on the contextualized task does not. This
shows that the older participants can use context to compensate for the apparent decline in cognitive func-
tion. The outcome is that they perform better in a contextualized setting than in a setting without context.
Decontextualized tasks give an unrealistic impression of how cognitive processes are impacted by aging,
making them unsuitable for producing a better insight into these processes.

It is beyond the scope of this paper to go deeper into 4E philosophy. We point interested readers to several
excellent contributions (Bruineberg and Rietveld, 2019; Chemero, 2009; Clark, 1997; Gallagher, 2005; Newel
et al., 2018; Wheeler, 2005). It is worthwhile to notice that although the 4E movement is mainly conceptual,
at least two fields in the broader landscape of cognitive science have benefited greatly from it. Incorporating
the idea of environment-agent coupling led to the shift in robotics to situated robotics (Pfeifer and Bongard,
2007). Similarly, animal cognition research has made progress by putting the animal-environment coupling cen-
ter stage (Barrett, 2011). Cognitive neuroscientists can similarly use this theoretical basis to their advantage.

Relationship with previous calls for ecological validity

Our call for including context in cognitive neuroscience research is related to recent calls for increasing
ecological validity and moving toward ‘‘real-world’’ neuroscience (e.g. Matusz et al., 2019; Nastase et al.,

8 iScience 24, 102392, May 21, 2021

iScience ll
Review OPEN ACCESS

2020; Shamay-Tsoory and Mendelsohn, 2019), which in turn build on older discussions of the topic (e.g.
Gibson, 1979; Neisser, 1976). Several methodological advances make ecologically valid research more
feasible than in the past, including computational advances (Armeni et al., 2017; Kriegeskorte and Douglas,
2018), virtual reality (Peeters, 2019; Reggente et al., 2018), and the possibility of measuring neural re-
sponses in more naturalistic situations (mobile imaging: Parada and Rossi, 2020; bringing objects in the
scanner: Snow et al., 2011). We see great promise in these developments because they will allow for
including increasingly naturalistic context, allowing researchers to address new questions and discover
new principles of the mind and brain. However, our call is not a general call for increasing ecological val-
idity. Rather, we more specifically point to the role of context in cognitive neuroscience. The dogma that
taking context into account necessarily leads to a decrease in experiment control is countered by actual
experimental practice. As the examples in the previous sections show, context can be incorporated or
manipulated in tightly controlled experiments (see also Peeters, 2019).

Guidelines for future research

What are the implications for cognitive neuroscientists? First and foremost, we want to stress that we do not
believe that there is a magic formula for how to ‘‘best’’ do research. What we do suggest is one golden rule
that we believe can help our field move forward. The rule is simple yet influential: Define the goal of your
research, and make sure that the implementation stays close to that goal. Defining the goal of research is an
explicit formulation of the phenomenon that the researcher wants to explain (Willems, 2011). This position
is reminiscent of the one formulated by Brunswik (1955), who used the term ‘‘representative design’’ to
make the case:

‘‘. representative design means that researchers first need to define a reference class of stim-
uli, tasks, and situations to which they intend to generalize a result. By sampling from this pre-
defined set of conditions, the set of conditions toward which the generalization is intended is
part of the experimental design.’’ (Brunswik, 1955; discussed in Holleman et al., 2020; see also
Nastase et al., 2020).

As a concrete example, if a researcher is interested in syntactic language comprehension during reading,

the next step would be to assess how syntactic comprehension naturally takes place. If syntactic compre-
hension almost never occurs in the form of multiple recursive embeddings (Verhagen, 2009), then it follows
that using such recursive language stimuli is not representative of the phenomenon under study since it will
not allow generalization to the actual circumstances in which syntactic comprehension takes place (Wil-
lems, 2011). Similarly, if one’s topic of interest is to study how people find objects in daily life, it is important
to consider the role of familiarity and context in visual search, as discussed above.

There is a useful analogy with cognitive neuroscience research done in animals. Animal researchers are
often asked to defend why they believe animal results translate to humans. How are insights into
spatial cognition in rats informative for how humans navigate? Why does having a neural model of
macaque’s working memory help us understand how memory works in humans? We argue that, simi-
larly, cognitive neuroscientists need to describe why they believe their experimental design allows for
generalization to cognition as it occurs in daily life. Why is it conceivable that the insights gained from
a simplified experiment are informative to the part of cognition under study? We believe that being
explicit about this link will help cognitive neuroscientists take full potential of the opportunities that
a context-centered approach offers and will lead to relevant and translatable insights into the neural
basis of cognition.

ACKNOWLEDGMENTS
This project has received funding from the European Research Council (ERC) under the European Union’s
Horizon 2020 research and innovation programme (grant agreement No 725970), as well as from the
Netherlands Organization for Scientific Research (NWO-Vidi 276-89-007). The funders had no role in study
design, data collection and analysis, decision to publish, or preparation of the manuscript. We thank Surya
Gayet and Chuanji Gao for comments on an earlier version of the paper.

AUTHOR CONTRIBUTIONS
Conceptual analysis, R.W. and M.P.; writing, R.W. and M.P.

iScience 24, 102392, May 21, 2021 9

 ll iScience
OPEN ACCESS Review

DECLARATION OF INTERESTS
The authors declare no competing interests.

REFERENCES
Alderman, N., Burgess, P.W., Knight, C., and Clark, A. (1997). Being There: Putting Brain, Body Neurosciences, Fifth edition (MIT press),
Henman, C. (2003). Ecological validity of a and World Together Again (MIT press). pp. 667–674.
simplified version of the multiple errands
shopping test. J. Int. Neuropsychol. Soc. 9, 31–44, Clark, A., and Chalmers, D. (1998). The extended Hasson, U., Egidi, G., Marelli, M., and Willems,
[Link] mind. Analysis 58, 7–19. R.M. (2018). Grounding the neurobiology of
language in first principles: the necessity of non-
Ames, D.L., Honey, C.J., Chow, M.A., Todorov, Çukur, T., Nishimoto, S., Huth, A.G., and Gallant, language-centric explanations for language
A., and Hasson, U. (2015). Contextual alignment J.L. (2013). Attention during natural vision warps comprehension. Cognition 180, 135–157, https://
of cognitive and neural dynamics. J. Cogn. semantic representation across the human brain. [Link]/10.1016/[Link].2018.06.018.
Neurosci. 27, 655–664, [Link] Nat. Neurosci. 16, 763–770, [Link]
jocn_a_00728. 1038/nn.3381. Heidlmayr, K., Weber, K., Takashima, A., and
Hagoort, P. (2020). No title, no theme: the joined
Armeni, K., Willems, R.M., and Frank, S.L. (2017). Dirix, N., Brysbaert, M., and Duyck, W. (2019). neural space between speakers and listeners
Probabilistic language models in cognitive How well do word recognition measures during production and comprehension of multi-
neuroscience: Promises and pitfalls. Neurosci. correlate? Effects of language context and sentence discourse. Cortex 130, 111–126, https://
Biobehav. Rev. [Link] repeated presentations. Behav. Res. Methods 51, [Link]/10.1016/[Link].2020.04.035.
neubiorev.2017.09.001. 2800–2816, [Link]
1158-9.
Holleman, G.A., Hooge, I.T.C., Kemner, C., and
Ballard, D.H., Hayhoe, M.M., and Pelz, J.B. (1995). Hessels, R.S. (2020). The reality of ‘‘real-life’’
Memory representations in natural tasks. J. Cogn. Donders, F.C. (1869). On the speed of mental
neuroscience: a commentary on Shamay-Tsoory
Neurosci. 7, 66–80, [Link] processes. Acta Psychol. (Amst.) 30, 412–431.
and Mendelsohn (2019). Perspect. Psychol. Sci.
1995.7.1.66. [Link]
Draschkow, D., Kallmayer, M., and Nobre, A.C.
Bar, M. (2004). Visual objects in context. Nat. Rev. (2021). When natural behavior engages working
memory. Curr. Biol. 31, 869–874.e5, [Link] Hutchins, E. (1996). Cognition in the Wild (MIT
Neurosci. 5, 617–629, [Link] Press).
nrn1476. org/10.1016/[Link].2020.11.013.

Eckstein, M.P., Koehler, K., Welbourne, L.E., and Jacobs, A.M., and Willems, R.M. (2018). The
Barrett, L. (2011). Beyond the Brain: How Body
Akbas, E. (2017). Humans, but not deep neural fictive brain: neurocognitive correlates of
and Environment Shape Animal and Human
networks, often miss giant targets in scenes. Curr. engagement in literature. Rev. Gen. Psychol. 22,
Minds, First edition (Princeton University Press).
Biol. 27, 2827–2832.e3, [Link] 147–160, [Link]
cub.2017.07.068.
Binder, J.R., Desai, R.H., Graves, W.W., and
Kaiser, D., Quek, G.L., Cichy, R.M., and Peelen,
Conant, L.L. (2009). Where is the semantic
Fedorenko, E., and Thompson-Schill, S.L. (2014). M.V. (2019). Object vision in a structured world.
system? A critical review and meta-analysis of 120
Reworking the language network. Trends Cogn. Trends Cogn. Sci. 23, 672–685, [Link]
functional neuroimaging studies. Cereb. Cortex
Sci. 18, 120–126, [Link] 10.1016/[Link].2019.04.013.
N. Y. N. 19, 2767–2796, [Link]
2013.12.006.
cercor/bhp055.
Kiverstein, J., and Miller, M. (2015). The
Ferstl, E.C., Neumann, J., Bogler, C., and von embodied brain: towards a radical embodied
Brandman, T., and Peelen, M.V. (2017).
Cramon, D.Y. (2008). The extended language cognitive neuroscience. Front. Hum. Neurosci. 9,
Interaction between scene and object processing
network: a meta-analysis of neuroimaging studies [Link]
revealed by human fMRI and MEG decoding.
on text comprehension. Hum. Brain Mapp. 29,
J. Neurosci. 37, 7700–7710, [Link]
581–593. Kriegeskorte, N., and Douglas, P.K. (2018).
1523/JNEUROSCI.0582-17.2017.
Cognitive computational neuroscience. Nat.
Friston, K.J., Price, C.J., Fletcher, P., Moore, C., Neurosci. 21, 1148–1160, [Link]
Bruineberg, J., and Rietveld, E. (2019). What’s Frackowiak, R.S., and Dolan, R.J. (1996). The
inside your head once you’ve figured out what 1038/s41593-018-0210-5.
trouble with cognitive subtraction. NeuroImage
your head’s inside of. Ecol. Psychol. 31, 198–217, 4, 97–104, [Link]
[Link] Kristensen, L.B., Engberg-Pedersen, E., and
0033.
Wallentin, M. (2014). Context predicts word order
Brunswik, E. (1955). Representative design and processing in Broca’s region. J. Cogn. Neurosci.
Gallagher, S. (2005). How the Body Shapes the
probabilistic theory in a functional psychology. 26, 2762–2777, [Link]
Mind (Oxford University Press).
Psychol. Rev. 62, 193–217, [Link] jocn_a_00681.
1037/h0047470. Gibson, J.J. (1979). The Ecological Approach to
Visual Perception (Houghton, Mifflin and Lerner, Y., Honey, C.J., Silbert, L.J., and Hasson,
Carrasco, M. (2011). Visual attention: the past 25 Company). U. (2011). Topographic mapping of a hierarchy of
years. Vis. Res 51, 1484–1525, [Link] temporal receptive windows using a narrated
1016/[Link].2011.04.012. Graziano, M.S.A., and Aflalo, T.N. (2007). story. J. Neurosci. Off. J. Soc. Neurosci. 31, 2906–
Mapping behavioral repertoire onto the cortex. 2915, [Link]
Castelhano, M.S., and Krzys, K. (2020). Rethinking Neuron 56, 239–251, [Link] 10.2011.
space: a review of perception, attention, and neuron.2007.09.013.
memory in scene processing. Annu. Rev. Vis. Sci. Li, F.F., VanRullen, R., Koch, C., and Perona, P.
6, 563–586, [Link] Grodzinsky, Y., and Santi, A. (2008). The battle for (2002). Rapid natural scene categorization in the
vision-121219-081745. Broca’s region. Trends Cogn. Sci. 12, 474–480, near absence of attention. Proc. Natl. Acad. Sci.
[Link] 99, 9596–9601, [Link]
Chemero, A.P. (2009). Radical Embodied 092277599.
Cognitive Science (MIT Press). Hagoort, P. (2013). MUC (memory, Unification,
control) and beyond. Front. Psychol. 4, 416, Mak, M., and Willems, R.M. (2019). Mental
Chiel, H.J., and Beer, R.D. (1997). The brain has a [Link] simulation during literary reading: individual
body: adaptive behavior emerges from differences revealed with eye-tracking. Lang.
interactions of nervous system, body and Hagoort, P., and Levinson, S.C. (2014). Cogn. Neurosci. 34, 511–535, [Link]
environment. Trends Neurosci. 20, 553–557. Neuropragmatics. In The Cognitive 1080/23273798.2018.1552007.

10 iScience 24, 102392, May 21, 2021

iScience ll
Review OPEN ACCESS

Mak, W.M., Vonk, W., and Schriefers, H. (2008). Peeters, D. (2019). Virtual reality: a game- Thorpe, S., Fize, D., and Marlot, C. (1996). Speed
Discourse structure and relative clause changing method for the language sciences. of processing in the human visual system. Nature
processing. Mem. Cognit. 36, 170–181. Psychon. Bull. Rev. 26, 894–900, [Link] 381, 520–522.
10.3758/s13423-019-01571-3.
Mason, R.A., and Just, M.A. (2006). Neuroimaging Turkheimer, F.E., Rosas, F.E., Dipasquale, O.,
contributions to the understanding of discourse Pfeifer, R., and Bongard, J.C. (2007). How the Martins, D., Fagerholm, E.D., Expert, P., Vása, F.,
processes. In Handbook of Psycholinguistics, M. Body Shapes the Way We Think (MIT press). Lord, L.-D., and Leech, R. (2021). A complex
Traxler and M.A. Gernsbacher, eds. (Elsevier), systems perspective on neuroimaging studies of
pp. 765–799. Phillips, L.H., Kliegel, M., and Martin, M. (2006). behavior and its disorders. The Neuroscientist.
Age and planning tasks: the influence of [Link]
Matusz, P.J., Dikker, S., Huth, A.G., and Perrodin, ecological validity. Int. J. Aging Hum. Dev. 62,
C. (2019). Are We Ready for real-world 175–184, [Link] Van Duijn, M.J., van Sluiter, I., and Verhagen, A.
neuroscience? J. Cogn. Neurosci. 31, 327–338, TMLM-WW8X. (2015). When narrative takes over: the
[Link] representation of embedded mindstates in
Potter, M.C. (1976). Short-term conceptual Shakespeare’s Othello: Lang. Lit. [Link]
memory for pictures. J. Exp. Psychol. [Hum. 10.1177/0963947015572274.
Meteyard, L., and Vigliocco, G. (2018). Lexico-
Learn.] 2, 509–522.
semantics. In The Oxford Handbook of
Psycholinguistics (Oxford University Press). K. van Krieken, J. Sanders, and E. Sweetser, eds.
Preston, T.J., Guo, F., Das, K., Giesbrecht, B., and (2019). Time and Viewpoint in Narrative
[Link]
Eckstein, M.P. (2013). Neural representations of Discourse; Special Issue Cognitive Linguistics
9780198786825.013.4.
contextual guidance in visual search of real-world (Cognitive Linguistics).
scenes. J. Neurosci. 33, 7846–7855, [Link]
Nardo, D., Santangelo, V., and Macaluso, E. org/10.1523/JNEUROSCI.5840-12.2013.
(2011). Stimulus-driven orienting of visuo-spatial Varela, F.J., Thompson, E., and Rosch, E. (1991).
attention in complex dynamic environments. The Embodied Mind (MIT Press).
Quek, G.L., and Peelen, M.V. (2020). Contextual
Neuron 69, 1015–1028, [Link] and spatial associations between objects
neuron.2011.02.020. interactively modulate visual processing. Cereb. Van der Stigchel, S. (2020). An embodied account
Cortex N. Y. N. 30, 6391–6404, [Link] of visual working memory. Vis. Cogn. 28, 414–419,
Nastase, S.A., Goldstein, A., and Hasson, U. 1093/cercor/bhaa197. [Link]
(2020). Keep it real: rethinking the primacy of
experimental control in cognitive neuroscience. Reggente, N., Essoe, J.K.-Y., Aghajan, Z.M., Verhagen, A. (2009). What do you think is proper
NeuroImage 222, 117254, [Link] Tavakoli, A.V., McGuire, J.F., Suthana, N.A., and place of recursion? Conceptual and empirical
1016/[Link].2020.117254. Rissman, J. (2018). Enhancing the ecological issues. In Recursion and Human Language, H. van
validity of fMRI memory research using virtual der Hulst, ed. (Mouton de Gruyter).
Neisser, U. (1976). Cognition and Reality: reality. Front. Neurosci. 12, [Link]
Principles and Implications of Cognitive 3389/fnins.2018.00408. Võ, M.L.-H., Boettcher, S.E., and Draschkow, D.
Psychology (W. H. Freeman). (2019). Reading scenes: how scene grammar
Rensink, R.A., O’Regan, J.K., and Clark, J.J. guides attention and aids perception in real-
Newel, A., De Bruin, L., and Gallagher, S. (2018). (1997). To see or not to see: the need for attention world environments. Curr. Opin. Psychol. 29,
The Oxford Handbook of 4E Cognition, the to perceive changes in scenes. Psychol. Sci. 8, 205–210, [Link]
Oxford Handbook of 4E Cognition (Oxford 368–373. 03.009.
University Press). [Link]
oxfordhb/9780198735410.001.0001. Shamay-Tsoory, S.G., and Mendelsohn, A. (2019). Vukovic, N., and Williams, J.N. (2015). Individual
Real-life neuroscience: an ecological approach to differences in spatial cognition influence mental
Oliva, A., and Torralba, A. (2007). The role of brain and behavior research. Perspect. Psychol. simulation of language. Cognition 142, 110–122,
context in object recognition. Trends Cogn. Sci. Sci. 14, 841–859, [Link] [Link]
11, 520–527, [Link] 1745691619856350.
09.009. Ward, L.M. (2002). Dynamical Cognitive Science
Slors, M. (2020). From notebooks to institutions: (MIT press).
O’Regan, J.K. (1992). Solving the ‘‘real’’ mysteries the case for symbiotic cognition. Front. Psychol.
of visual perception: the world as an outside 11, [Link] Wheeler, M. (2005). Reconstructing the Cognitive
memory. Can. J. Psychol. 46, 461–488. World (MIT press).
Snow, J.C., Pettypiece, C.E., McAdam, T.D.,
McLean, A.D., Stroman, P.W., Goodale, M.A., and
Parada, F.J., and Rossi, A. (2020). Perfect timing: Willems, R.M. (2011). Re-appreciating the why of
Culham, J.C. (2011). Bringing the real world into
mobile brain/body imaging scaffolds the 4E- cognition: 35 years after Marr and Poggio. Front.
the fMRI scanner: Repetition effects for pictures
cognition research program. Eur. J. Neurosci. Psychol. 2, 244, [Link]
versus real objects. Sci. Rep. 1, 130, [Link]
[Link] 2011.00244.
org/10.1038/srep00130.
Peelen, M.V., Fei-Fei, L., and Kastner, S. (2009). Sternberg, S. (1969). The discovery of processing Willems, R.M., Nastase, S.A., and Milivojevic, B.
Neural mechanisms of rapid natural scene stages: Extensions of Donders’ method. Acta (2020). Narratives for neuroscience. Trends
categorization in human visual cortex. Nature Psychol. (Amst.) 30, 276–315, [Link] Neurosci. 43, 271–273, [Link]
460, 94–97, [Link] 1016/0001-6918(69)90055-9. tins.2020.03.003.

Peelen, M.V., and Kastner, S. (2014). Attention in Summerfield, J.J., Lepsien, J., Gitelman, D.R., Willems, R.M., and Varley, R. (2010). Neural
the real world: toward understanding its neural Mesulam, M.M., and Nobre, A.C. (2006). insights into the relation between language and
basis. Trends Cogn. Sci. [Link] Orienting attention based on long-term memory communication. Front. Hum. Neurosci. 4, 203,
[Link].2014.02.004. experience. Neuron 49, 905–916, [Link] [Link]
10.1016/[Link].2006.01.021.
Peelen, M.V., and Kastner, S. (2011). A neural Zacks, J.M., Speer, N.K., Swallow, K.M., and
basis for real-world visual search in human Thelen, E., and Smith, L.B. (1994). A dynamic Maley, C.J. (2010). The brain’s cutting-room floor:
occipitotemporal cortex. Proc. Natl. Acad. Sci. U. systems approach to the development of segmentation of narrative cinema. Front. Hum.
S. A. 108, 12125–12130, [Link] cognition and action. In MIT Press/Bradford Book Neurosci. 4, [Link]
pnas.1101042108. Series in Cognitive Psychology (MIT Press). 00168.

iScience 24, 102392, May 21, 2021 11