Language and Executive Control Overview
Language and Executive Control Overview
Module - 08
Part - 1
Lecture - 18
Language and executive control
Welcome to the Module 8th Part 1. In the last module we have looked at the attentional
mechanism and its relationship with language through various language processing, and
so on. Now, we will move on to the domain of executive control, executive function,
executive control, cognitive control there are many ways to look at it. So, we will look at
this particular mental mechanism and its relationship with language, as in language
processing.
Before we move on to the language aspect of the mechanism, let us look at the function
itself. What is executive function? Executive function is also called cognitive control. So,
what is it, what is executive control? What is executive function? This is what we will look
at in the beginning.
And then what are its components? What is it comprised of, what are its, you know, smaller
parts, what are its sub subdivisions subparts and so on. Is it a single undifferentiated
cognitive process or is it made up of separate parts that make it a whole process? And then
we will move on to the brain regions responsible for the for carrying out this function in
the in humans.
And then we will look at the role of language and then we will move on to the more
complex aspects of language and executive function relationship.
So, first and foremost, the importance of executive functions. To till now, we have looked
at many other mental functions, most recently we have looked at the attentional
mechanism. Executive function and attentional mechanism are interconnected. They are
part of one superimposed bigger category which is called by various names.
393
Sometimes it is called executive function; sometimes it is called cognitive control. So, it
is considered a part of a very very important part of intelligent behavior, to the extent it is
so important that sometimes it is considered the core of intelligent behavior in humans.
What is intelligence? If we ask this simple question what is intelligence and can we break
it down to smaller parts as to what are the behavioral outputs, what are the negotiations
that the human mind engages with the surroundings and environment and so on.
If we look at it deeply, we will see that executive function lies at the core of many of our
intelligent behavior. So, that is why it is sometimes considered the core. It is also important
to look at executive function in terms of when there is a compromised EF in certain
patients.
So, executive function is not only looked at and understood and studied in normal people.
But also in case of patients, because in many various kinds of psychiatric problems,
disorders as well as neurological disorders, executive function gets compromised, meaning
that is that it gets affected.
Just like we have seen that brain damage affects language functions, brain damage creates
language disorders, similarly executive function also gets affected in certain cases of
neurological disorder or psychiatric problem and so on and so forth. That is why this is
also understood, this is also looked at from the from clinical literature perspective as well.
Another interesting fact about executive function is that it varies. It is not something that
we are born with and it stays the same all throughout our lifespan, it changes. A child’s
small child’s executive function mechanism and the controlling factors and so on and so
forth are not the same as that of an adult.
So, as a child grows the executive function mechanism also grows and reaching it peak its
peak somewhere around young adulthood and then it goes on like that. So, this is yet
another way of looking at executive function. It also affects executive function affects
performance in challenging task conditions. Why is it important? Because executive,
function is at the core of human behavior, human intelligent behavior.
And that is what decides how we behave what is the output of our behavior or any other
kind of verbal output and so on, in a challenging task condition. Given various kinds of
394
task demands, how a human being really negotiates those challenges is at the core of it.
So, this is understanding executive function also may takes us to the domain of how
humans really negotiate a challenging path.
For example, in real life one very challenging job is the job of a pilot, aircraft pilot or a
normal passenger pilot or in a or a fighter aircraft and so on and so forth. So, the job inside
the cockpit is extremely challenging, there can be many tasks simultaneously demanding
the attention of the pilot. So, how does one handle those things?
So, these are also studied. So, as you can see there are many dimensions of the same thing
same thing called executive function and that is why it has been studied by various domains
various you know experts from various domains like cognitive psychologists,
neuroscientists, linguists, education researchers and so on. So, what is this whole thing
about?
Let us see what the trajectory has been. So, the notion of executive function has been
around for a very long time. We have seen when we discussed Luria, that it has been
around since the time of Luria and even before. So, psychologists have known the concept
of executive function for a very long time. In fact, it goes back to the late 19th century, to
Bianchi and then of course, to Luria.
So, this has been this notion of executive function of a higher mental function, a kind of a
control system, that modulates our behavioral output has been around for some time within
the domain of psychology. And when cognitive psychology came into being, then initial
stages of initial research in cognitive psychology also use this term very heavily as I have
mentioned a few important papers here.
However, like all things this the domain also has been changing a very rapidly in the very
rapidly through last many decades. So, in the in the later years primarily due to the
advances made in the communications and control systems and computer science, AI and
so on, a new term started replacing the term executive function which is cognitive control.
So, cognitive control has come to replace executive function in many domains though not
entirely. There are still some places where we still use executive function, but largely
cognitive control has come to replace the term executive function. So, it basically owes its
origin to cybernetics and later cybernetics and then of course, AI and so on.
395
One important seminal book by Miller and his group, George Miller and his group was the
first to bring to explicit notice of the academic world, the where they brought out cognitive
control, human mental mechanisms, hierarchical structures, all these things together.
So, cybernetics on the one hand was on its on peak, the control system education; education
academia was rather interested in control systems and the research was going on in full
swing. So, at that time Miller’s book actually got actually connected control theory, goal
directed human behavior and human cognition, hierarchical structure of the cognitive
mechanisms and so on, through this book.
Subsequently, there were other researchers, most importantly Posner and his group that
brought out very very important three seminal papers. Almost all of these around they
revolve around the same topic and basically these papers along with this book quickly and
firmly established the construct called the ‘cognitive control’. So, these three papers most
important papers are listed here I have added them in the references.
So, what are these papers talking about? These articles focused on differences between
automatic and controlled processes, because that is what they were looking at that time.
So, control processes as well as control systems and human cognition and so on.
So, the difference between automatic process and controlled processes, what are the
differences, how are they different, what are the finer nuances and so on is the entire gamut
of things that these papers talked about.
So, they propose that control processes are, there were three main proposals. One was,
controlled processes are slower as opposed to the automatic processes, controlled
processes can suffer from interference of the automatic processes and thirdly, it is
dependent on a limited capacity mechanism. These were the first important proposals
given by those given by Posner, Snyder and Shriffin.
So, this is what basically brings us brings to the forefront the difference between automatic
process versus controlled process. Automatic processes are those processes that are well
learned, that has been established over a period of time. We know, for example, the
396
moment we see a written word we read it, it is not a controlled process, it is it does not
require our selective attention, it just happens. This is why it is called an automatic process.
Automatic reading is an automatic process for most of us because it has been established
well established, well learned process. So, these are the differences.
So, one of the important ways in which Posner started to show this difference is the famous
color-word stroop task. In fact, stroop task is so relevant even today.
So, what they did was they were they showed that through that color-word stroop task they
showed that adults are universally faster to read out the word than to name its color. Notice
here that we have the word ‘blue’ written in three different colors, blue is written in red,
blue is written in blue and red is written in green.
Now, all of you as you are looking at this slide, will have no difficulty whatsoever to read
the word. So, blue, blue and red, it does not matter in which ink which color ink it is
written. However, because reading is automatic process, reading the word is automatic
process.
However, if we ask you to name the color in which the word is written, then there is a
problem because that is a controlled process. It needs to be controlled, it needs to be
regulated. So, there it becomes a problem because, it is written blue here, but the ink is
red. So, there is a conflict in this particular case. However, here, there is no conflict blue
is written in blue and again there is a conflict here red is written in green.
So, the primary finding that Posner and his group showed is that reading a word is
automatic process and it is extremely quick, it does not take time. And it is also not affected
by the color of the ink in which it is written. So, whether it is written in a blue is written
in green or red it does not really matter.
However, on the other hand the process of naming the color is not an automatic process it
is a controlled process and hence it gets affected by the word. So, if you have to name the
color blue and it is written in blue it is slightly faster, but when it is written in when you
have to mention the color, but it is written in the other color is written.
397
So, it is for example, here it is green you have to say green, but the word says red there is
a conflict and in these cases the naming is impaired. So, naming a color is impaired if done
with another task as well.
So, there is so controlled processes are open to interferences. In one case it is interference
by the word itself, in another cases it is interfered by if there is an another task that is given
to the participant which does not happen, if it is an automatic process.
So, these are some of the initial studies, initial landmark studies that showed the difference
between automatic versus a controlled process. Of course because these are initial ideas so
many of these notions have been challenged later and there are lots of debates and
controversies.
For example, one big controversy is around the limited capacity mechanism, the notion of
limited capacity mechanism as well as the automatic versus controlled process because the
automatic versus control process may not be a fixed sort of a dichotomy.
Some processes which are controlled now, which need to be controlled now, may become
an automatic process over a period of time if you where if you learn it well. If it becomes
a continuous process, if it is regularly you know if you get habituated to it; it might become
an automatic process and so on, similarly, the notion of flexibility.
However, that is that is how all theoretical constructs are, there are of course, debates and
update on every theoretical construct, but the basic idea of cognitive control of difference
between automatic and controlled processes remains.
So, these are the primary components of executive function. One is flexible thinking or it
is also called cognitive flexibility, as in you know being adaptable to the changed scenario,
we will see it in a short while. And then the second most important aspect of executive
function is working memory and then comes inhibitory control and of course, attention.
Attention because we have already looked at before in detail, we will not go into it here,
we will look at the other aspects.
398
So, then what then is it Is executive function? it is a high level cognitive process. There
are two levels of cognitive process of course, it is not airtight or and also there is a lot of
dialogue between the processes, but there is a hierarchy, in the in the mental mechanisms.
Some are considered high level mental mechanisms, some are considered low level mental
mechanism.
So, the low level mechanisms are correspond to the automatic processes. So, this is
executive function on the other hand, is a high level cognitive process and this higher level
process modulates the lower level processes. For example, reading the word ‘blue’. So,
the reading aspect, as soon as you see the word blue you read, it is an automatic process
and that is why it is also called a lower level process, this is almost automatic.
Now, this process of reading can be guided by various other modalities, other variables let
us call them. So, this can be guided by the goal: are you supposed to read the word or are
you suppose to do something else? So, there are many other things like current goal,
environments and so on. Now, this guiding part of the lower mechanism, lower processes
is the higher part of the process.
So, depending on our current goal, we modulate our lower level perceptual analysis and
our vocal output. Sometimes it is a vocal output, sometimes not, but this is how it works.
So, the local lower level processes are guided by the higher level processes, which is
basically taking into account the goal, current goal as well as the environment within which
the entire process is set.
So, what do we mean by the current goal? Sometimes we may ignore the word may in fact,
in experimental setup, we tell the participants to ignore the word and just look at the color
and tell us the name of the color. So, that is what exactly is the task here. Sometimes the
task is to read it aloud and so on.
So, even though the stimulus remains the same, participant remain the same, environment
is same, but the goal might change, goal is at that point the current goal. Thus, even though
the perceptual cue is same the task can be different and hence the response can be different.
This control is the core of what we call executive function.
So, controlling your output, controlling your response, keeping in mind the current goal
and the environment is what is the crux of the matter. So, it basically helps us in devising
399
new ways of behavior and thus it allows for flexibility, right? So, it is not that just the
moment you see the word you have to read it, there are also other possibilities. There are
many ways of looking at the same word blue.
So, thus this also is very very strongly related to the flexibility aspect. As a result of which
this then leads to optimizing one’s reaction to unfamiliar circumstances. In life there are
two kinds of circumstances; one is familiar, the other is unfamiliar. Familiar circumstances
automatically gives rise to the automatic responses, we already have a stimulus response
connection set.
So, in this particular scenario the moment you see the word you automatically read and so
on and so forth. However, there are also situations when circumstances demand a different
reaction; different reaction not only because the circumstances have changed, but maybe
the circumstances are still same, but the current goal has changed.
This is not only true about experimental setup, but also about life in general. So, that is
why executive function is extremely useful for non routine behavior and responses. And
all of these lead us to adapting a novel approach adapting to novel or changing situations
and tailoring our responses, keeping in mind both the situation and our goal.
Now, executive function has many sub-parts of it, it is comprised of many stages. Logan
for example, has listed these four stages, four parts of executive function of the human
cognitive system. The first is the choice of strategy as to what has to be done, given this
scenario, given this kind of a goal what kind of strategy is to be used.
And then instantiation of the chosen strategy, to initiate that particular strategy to be
utilized and then of course, comes the execution and maintenance of the both are important
execution as well as maintenance of the strategy to achieve that goal. So, you choose your
strategy, then start utilizing it, execute the strategy, execute your particular job that you
have to do.
And as long as this is required you have to maintain that strategy and then inhibition of the
strategy in face of a changed goal or scenario. This is you can see this is in many ways
similar to our attentional engagement and disengagement loop that we talked about.
400
So, we orient our attention, we engage our attention and then disengage and then the cycle
goes on. That is why it is important to understand executive function, also keeping in mind
the attentional network. So, these are the four functions of human executive functions of
human cognitive system as proposed by Logan.
Of course we can go on and on, this is a very very broad area of research and it has really
smaller parts and many smaller parts and finer nuances, but we need to sum it up because
our primary goal is to look at it from a linguistic perspective. So, how do we sum up?
General this is a general process executive function is a general process that regulate and
guide cognitive processes in sensory, memory and motor systems, alongside an internal
goal.
So, this is a this is a process that helps regulate our cognitive processes. In what domain?
In sensory, in memory and in motor system. So, motor system will also include language,
in the sense of linguistic output verbal output. So, this is composed of at least two
components: conflict monitoring and conflict resolution.
If you if you really simplify the whole thing it ultimately boils down to conflict monitoring.
So, first you have to identify there is a conflict, there is an interference you monitor and
then there is a resolution aspect of the of the same conflict and then it also includes
inhibition. So, not only you have to take action on one particular aspect, you have to also
stop your action from on another aspect of the same scenario. Hence you need inhibition.
So, let us get into a slight detail about conflict monitoring. Conflict monitoring refers to
the processes that monitor the occurrence of conflicts in information processing. Now, this
information processing can be of any domain, it can be the language related or it can be
any other sensory motor kind of information processing.
The conflict monitoring process evaluate current levels of conflicts and trigger and then it
will trigger compensatory adjustments of processing pathways. What has to be done, first
you monitor the scenario, identify the problem, monitor the problem and then trigger the
processing pathway, as to how to deal with it and then comes the resolution. So, conflict
resolution, this is a process that aim at eliminating the sources of conflicts.
The conflict resolution process enhances task relevant information and suppresses task
irrelevant information. This is one of the most important aspects of executive function,
401
because they are sometimes in the environment there might be too many stimuli present,
some stimuli might be more salient, there they might be more visible, they might be more
you know more salient more kind of important to on the surface. However, they are not
part of your goal at that moment. Hence, you have to suppress that particular aspect of the
scenario and focus your attention on the one that is directly related to the goal. So, this is
what we mean by enhancing task relevant information and suppressing task irrelevant
information in the environment.
And this of course, needs organizing of behavior, along the internal goals. Inhibition is of
course, the processes that suppresses an active representation or action tendency. So, even
if, for example, if you are let us say if you are going to the library and your goal is to go
to the library, sit there and study for 4-5 hours, but on the way there is a pizza shop.
So, now, pizza shop is more salient because that is more inviting, let us say. So, you have
to eat now you need to your goal is to go and study and not eat a full snacks and then go
to sleep. So, the goal is to go to the library and study, hence you have to suppress your
tendency to go walk into the pizza shop. So, this is what we mean by inhibition. Inhibition
is a very very important component of executive function in order to achieve.
So, these are the some simple examples to show you what exactly it is all about. We have
we have said a lot of thing, but how does it really work. So, in terms of working memory,
what is it what we are we looking at in terms of working memory?
This is the ability to hold information in mind while thinking either about it or to do
something about it or thinking some about something else. So, juggling many things at the
same time while everything is keeping it keeping all the information intact. So, basically,
theoretically speaking, it refers to holding multiple things in mind while thinking of most
suitable response, keeping track of all the thoughts.
You need you need all this information to be worked on right now. So, they are all in the
in your working memory and, but you must keep track of all of them, in order to work on
them, in order to find the most suitable response for that given scenario. For example, in a
real life scenario practically speaking, let us say in a conversation, you have to remember
others opinions and ideas while formulating your own.
402
Let us say there is a discussion going on, this is commonly seen in group discussions. So,
everybody has their own opinions. So, another participant in the discussion has raised a
significant point and then you have to keep those points in mind while formulating your
own idea.
So, this is not something you come prepared for, here is where your working memory,
your functioning of the working memory, it plays a very important role and this is part of
your executive function. Similarly, inhibitory control, inhibitory control in fact, a lot of
research is now going on inhibitory control among adolescents and smaller children, both
in typical as well as atypical children to see how much of self regulation is in place and
how it can be enhanced and so on.
So, basically this refers to the suppression of an automatic response. Something sometimes
some stimulus you know elicits a knee jerk reaction from us, we just listen and
immediately react. So, suppressing that kind of a reaction given the scenario is what
inhibitory control refers to. So, for example, not leaping to the first possible response to a
situation and assess the situation for different possible solutions.
A particular scenario given a particular a group of people might demand different kind of
solution in different with different goals in mind, so this is where inhibitory control plays
out. For example, not bursting the balloons in somebody else’s party, not blowing out the
candle on somebody else’s birthday cake and not blurting out ‘how stupid’ even when you
know it is, the situation is stupid or the person is stupid, but you have to control yourself
not to blurt it out, skipping the pizza in favor of the sprouts and so on.
So, skipping the pizza in favor of sprouts basically is in keeping with your long term goal.
So, we are basically skipping the instant gratification in terms of our long term goal to get
healthy, to become healthy and so on.
So, inhibitory control is again a very very important aspect of executive function and then
of course, flexible thinking. Flexible thinking is nothing, but is a fancy name, for a
possibility that you can look at the same problem from the different possible angles, every
problem has multiple solutions. So, typically we are more commonly we are used to take
the same route again and again, but flexible mind will take different possible angles to
look at the same problem. That is all there is about flexible thinking.
403
So, ability to notice when our original approach to a problem does not work and it needs a
it needs to be reworked, the same problem can be reworked to find a new solution. So, this
is what basically refers to in a real life situation as well, looking at a problem from multiple
perspectives and choosing the best possible, best suited response.
Now, what is best suited is of course, we you can go back. Best suited response means best
suited for the current goal and the environment, this is very important. So, what is suitable
for a particular scenario may not be. So, that kind of flexibility and adaptability is what
executive function is all about.
So, in terms of visible, behavioral output, executive control refers to adaptability and
flexibility in the face of conflict. There is a saying: ‘adapt or perish’ this is basically what
it comes down to. People who often we see some people cannot adapt to changing scenario
the world is changing at a very very rapid pace.
And if we cannot change ourselves, change the way we look at the world, change the way
we value things or you know the value systems and so on and so forth, there are so many
things to look at it. So, the more you adapt, the more flexible you are the more you will be
able to adjust to the changes. So, this is what is ultimately about executive control.
So, adaptability and flexibility in the face of conflict. It is the ability of conflict resolution
using flexible approach, using mental mechanisms of selective, sustained attention. We
did talk about selective and sustained attention which is a very very important crucial
aspect of executive function and of course, working memory, inhibition of undesirable
response and so on.
So, you see executive function is not only crucial in case of challenging challenging task
conditions, but also it is an important part of social cognition. What to say when and what
not to say when, what not to do in what scenario are also an important part of our behavior
in the world, in the in the in the in face of conflict.
So, this is these are all the important things in terms of behavioral output, this is what is
visible to us. So, executive control mechanism, executive function, cognitive control
whatever you call it, there are many facets to it. One facet, the most visible one, is the
behavioral part of it, this is what we see.
404
At the at and it lies at the core of intelligence and therefore, often success. Often we see
there is a deluge of motivational speakers these days. What all they talk about is how to
become successful, how what not to do and what to do to be successful and so on.
Notice how often they underline the same things, many of them. And more often they are
not you will see they all will ultimately take you to flexibility and adaptability. So, this is
what is at the core of intelligence and of course, success.
So, now this till here we are talking about what is visible, what we see people doing and
all people not doing and so on and so forth. But this is the face of a larger mechanism, a
very fine mechanism, a very complex mechanism that plays out in the background. That
background mechanism happens in the brain therefore, we now go to the neurophysiology
of the cognitive control.
Neuroscience as we have seen in the beginning has made huge strides in the recent times,
huge strides it has it has developed very well, it has established itself as a very advanced
domain of and domain of academic academic enterprise and so on.
So, therefore, we now have a lot of different methods, very very objective and very subtle
methods to look at how the brain really works, in terms of various kinds of functions, in
terms of even at the level of single cell recording. So, we now have been able to map the
brain and its functioning in terms of even executive functions.
So, as the brain is busy doing taking part in some kind of an executive function task, we
can actually map. So, techniques like neuroimaging, neuropsychological, computational
modeling, single cell recording and meta analytic techniques etcetera so on and so forth,
have enhanced the understanding of cognitive function, executive function at the brain
level, at the level of brain anatomy, as to what region is busy doing what. The components
of the executive function and the discrete brain regions responsible for this process.
There is a lot of agreement on this that primarily, most of the executive functions are
located in the frontal lobe of the brain, not all of course, but most of the functions are
located in the frontal lobe of the brain, particularly in the prefrontal cortex, it is called PFC.
So, prefrontal cortex is very very important when it comes to executive function, because
not only because it has rich interconnections with its sub regions, within its sub regions.
405
So, prefrontal cortex has a strong relationship with the sub regions, lot of connections and
it also receives signal from all sensory and motor cortical system.
It is connected to all the sensory and motor systems. It also has strong connection with
subcortical regions, like basal ganglia and so on. So, patterns of activity within prefrontal
cortex represent the current behavioral context and the current goals and intentions. So,
basically, what with what this means is that various kinds of tasks, when the participants
are busy in carrying out then we see a lot of activation in various domains within the
prefrontal cortex, as per depending on the current goal, the environment and the task and
so on and so forth.
So, but the problem is PFC is a large area, it is not a small brain region, it is a large region
and hence it has been now divided into various sub domains in terms of the functions. So,
it is possible to look at the PFC in terms of various functions that is located in different
parts of the same thing same PFC. So, it has it is a large system with different
cytoarchitecture and connectivity.
Remember when we talked about brain and its parts and how they are organized, how the
structure is, how organized in terms of connections to the other brain regions as well as in
terms of the density of cell and so on and so forth. We did talk about cytoarchitectonic
grouping of various areas, so that PFC also has it is a large region, which has very different
cytoarchitecture as well as connectivity level.
So, roles of different PFC regions can thus differ. Thus the PFC can be divided into sub
regions, depending on the cytoarchitectonic as well as functional properties.
So, depending on how the structure is, how the morphology of particular domain is as well
as what kind of function, within executive function what kind of which part of executive
function it is busy doing, what it does and how it carries out those that part of the work.
So, one of them is the Ventrolateral PFC, in short VLPFC, this is involved in
comparatively simple tasks, such as short term maintenance of information that cannot
currently be perceived in working memory.
For example, memorizing a phone number you have just been told, before keying the
number into a telephone. So, you have you have just heard the phone number and now you
406
are keying in the number in the phone. So, that part of the work that, keeping it in mind so
called, is basically what ventrolateral PFC is does.
There is also a slightly controversial many of the proposals are actually controversial
because there is always data referring to different aspects of the same problem and often
contradictory results also. So, one controversial proposal is that different parts of the
ventrolateral PFC are used to store different types of information.
So, the again further breaking down the VLPFC also into further smaller parts, each part
has different kinds of storage. One of them can be sound of a word, another can be the
meaning of a word and so on and so forth. This has this is a slightly controversial claim,
but this is one of the claims that have been made about the working of VLPFC.
So, basically rearranging the digits we have I tell you a number phone number dorsolateral
PFC may not remember the number, but it may be active in helping me typing that number
in a reverse order.
So, there are, in fact, this is just one of the many possibilities of dorsolateral PFC. Each
small act that we do for example, hearing a phone number and keying it in actually goes
through a lot of smaller, finer nuanced processes, sub processes and each of these processes
are taking place in the in different distinct.
Sometimes distinct areas, sometimes it needs a collaboration between different areas and
so on. So, this is why we see such small little details here. Similarly, there is yet another
very important area which is Anterior Cingulate Cortex, ACC in short, this is also this also
plays a very important part in executive function mechanism.
So, part of the medial PFC has a strong projection to and from DLPFC, this is part of the
medial PFC and it has strong connections between dorsolateral PFC, this that this region
407
detects the need for control. So, control the doing the control, needing understanding that
there is a need for control, the telling the other regions that the to carry out the control
mechanism and so on and so forth can be some smaller aspects of the control mechanism.
So, this particular this particular part, ACC detects the need for control. For example, there
is a competition between two ways of behaving in a certain scenario, both of which may
be triggered by events in our environment, it will require a top down input to resolve the
conflict, what should be done given that particular scenario.
So, it ACC comes into this picture, then it does not do the controlling per se, but it rather
tells PFC when control is required, as per the claims of certain some researchers. However,
there had it has been challenged as we have seen in many other cases.
And in yet another area is the rostral PFC or RPFC. So, this is this is one of the largest,
but most mysterious, sub region of the PFC. Patients with damage restricted to this area
often perform well on standard neuropsychological tests, they do not, they do not falter
there. However, they have difficulty in real life multitasking. So, this is one a very
interesting aspect of RPFC.
Neuroimaging results show that there are distinct functions associated with different parts
of RPFC again you see. So, we have already seen that the PFC is now broken down to 3/4
major areas and each of those measure areas can again be broken down into smaller parts,
this is a very rich area of research.
In fact, as we speak, many groups are busy finding out the finer very very fine detail of
each of these sub parts, different parts of each of these smaller regions also. But, for our
purpose, we just we need to have a broader idea about how, what are the main areas that
are responsible for executive function and these are those.
So, also a very important, much cited paper in 2007 Cole and Schneider, points to a
different to the different distinct regions of the brain that are part of a network. So, now,
as we have seen the PFC has this many parts and then there are also certain parts in the
other part of the brain that are all connected.
So, now there is the proposal by this by researchers that this they all form a network and
they call it ‘cognitive control network’, Cole and Schneider call it cognitive control
408
network. There are different regions which are distinct part of the brain and they together
form a network, which is responsible for the carrying out executive function mechanism.
All kinds of cognitive control related functions are carried out by this, not all of them are
active all the time, but this is the network.
So, these regions as they have listed are some of them have already we have seen ACC is
there and then there is pre-supplementary motor area, this we have already seen. Again
another area is inferior frontal junction, anterior insular cortex and this we have already
seen and then posterior parietal cortex.
So, these are some of the areas of course, it is getting updated every very very frequently,
but this is the kind of the network that the brain cognitive control network that is
responsible for or that is directly connected to all the executive function mechanisms or
cognitive control mechanisms in case of humans.
This is this is one of the latest depiction of the various regions that we have just now talked
about. So, inferior frontal gyrus, you can see this is the back of the head and this is the
front. So, this is rostral PFC, this is ventrolateral PFC and then we have all the regions are
mentioned here very clearly. This is from a paper by this Cristofori et al 2019 paper, where
they have given this clear cut distinction of the areas.
And then we have basal ganglia here, which is also something that is. So, mostly there is
a lot of agreement among the neuroscientists that these are the areas where various aspects
of executive control or cognitive control takes place. So, you can see working memory,
inhibitory control, reasoning, maintenance of information that we have just seen and then
manipulation of information in working memory. So, this is roughly the map of the brain
in terms of those areas that are directly linked to executive control, cognitive control
mechanism
However still a lot more needs to be learnt that is that is as I said there is research going
on in all these aspects and we are constantly getting updated. Various regions and their
workings have now been found that, but the ‘how’ part, but necessarily not how we do not
really necessarily still understand, how this connection really works.
409
That is this is a very very nuanced area, this is a very complex area of research and then is
of course. So, the reason why I am mentioning this is that this is not the final, work is still
going on.
Now, we have seen what is executive control or cognitive control, how do we see this in
terms of behavior. Then we saw how there are very different brain regions that are directly
responsible for carrying out those control mechanisms.
And now, we will see the various behavioral tasks that are utilized in experimental setup
to really check it in controlled environment. So, there are various kinds of tasks that are
utilized to figure out how a particular participant, given on given various kinds of
variables, how they perform in executive control tasks in various dimensions.
So, there are some of them are here, not all of them I have listed, but these are the most
commonly utilized ones. So, Simon task, Flanker, Card sorting, Stroop task, working
memory and so on.
Let us see them one by one. This is Simon task one of the most commonly utilized one.
Simon task is basically, it is a behavioral task as I just mentioned.
So, this is this looks at the capacity of conflict resolution in a participant. Now, depending
on what the research question, is we might connect, we might like to connect it to various
kinds of population. For example, children versus adults, typical versus atypical children
and so on and so forth; also language can be brought in as a variable and so on.
The task is like this, if the participants are told if they there is a red dot on the red circle it
can be a square or anything, there is a red circle on if they see a red circle they have to
press the right key, if they see a blue one they will have to press the left key. That is the
only instruction given nothing else is told, right?
Now, so the right key is connected to the red, left key is connected to blue. Now, the
manipulation here is there are three possible manipulation, one is called the congruent
condition. Congruent condition is when remember this is the right key, red is connected to
right key. So, in the congruent condition the red circle appears on the right side of the
screen.
410
So, the red dot is on the right side of the screen, you have to press the right key. In the
incongruent condition, the red circle appears on the left side of the screen and you have to
still press the right key, because the position of the circle does not matter. The only cue for
the participant is to press right for red, left for blue.
Similarly, the blue can blue’s position can also be manipulated and this is called neutral
because it has nothing to do with the horizontal, left/right kind of a coordination. Now,
what is happening here is this on the face looks very simple. You see a red circle or a red
dot or whatever, you press the right key. But the conflict is here, you have to press the
right key, but the circle appears on the left side of the screen.
This is a conflict that often results in slower response latency, meaning the participant
takes slightly longer to react. They might still get it correct, but they will take slightly
longer time, also errors will be higher in this incongruent condition.
So, what is happening here is if you are, if you are able to this shows basically the reaction
time that the participant shows, the time the time to react one and how accurate your
responses are: these are the two output, that we look at and this shows how much how
quickly and how ably the participant has been able to resolve the conflict.
This is a very very effective and very productive task, that is utilized for executive to check
executive control mechanism in participants. Now, the depending on what our research
question is, let us say we want to see if a bilingual person is better at conflict resolution as
opposed to a monolingual person or a child typical child as opposed to atypical child you
can that that all that will depend on the research question, but this is how the task looks.
This is the conflict that is built into the system and we look at the conflict resolution by
the participants.
Similarly, there is this is Eriksen Flanker task. Flanker task also revolves around conflict
resolution and selective attention, sustained selective attention, it is a behavioral task. So,
the target participant is required to respond to the target stimulus that is flanked by other
stimuli. So, this tests as I said selective attention.
So, the task typically is to say which way this is one of the manifestations, but there can
be many other ways of flanker task. A simple flanker task may have just simple arrows
411
like this. So, what happens in a flanker task is that the participant has to look at the central
arrowhead or in this case the fish send the central this is the target.
And these are the ones that are flanking the target. So, this is the target and these are the
flankers. These have no other role except disturbing the participant’s attention. The
participant has to keep his eyes constantly on the central target stimuli, whatever that is an
arrow or a leaf or a fish basically it has a, it points towards one of the directions.
Now, you see in this particular this is called congruent condition, in this case it is easier to
identify the central target because it is flanked by other stimuli that are all pointing to the
same direction, hence it is called congruent. So, the central target as well as the flankers
are all pointing to the same direction.
In the incongruent condition, there is a problem here. This one is pointing this way, but
the central is pointing this way, again this is pointing that way. So, this is creating a conflict
for the participant.
This is again the there is no distraction, these are all oriented differently this is vertically
oriented, this is horizontally oriented; this can also have no flankers at all only the. So,
these are various ways of presenting the stimulus, but the target stimulus remains the same.
The target stimulus is the central one. Sometimes it is flanked by similar type of stimuli,
sometimes it is flanked by different. So, the idea is to check how much, under what
circumstances the participant fails to detect the central arrowhead or the fish or whatever.
So, basically this is a test of selective attention and response inhibition.
More often than not, what will happen is in many cases if this might distract, the flankers
might distract the target stimuli. So, who gets distracted under what conditions, what task
conditions, what kind of variables in terms of the participants and so on, will be the
research question, but this is how the task looks.
Wisconsin card sorting task is again a very commonly utilized one, to check cognitive
control and this also has a very crucial aspect which is updating, which is flexibility,
adaptability. So, this is a game; this game revolves around choosing cards, sorting cards.
Now, the cards are different, there are many ways of playing the card. There are many
412
possible sets for children the sets are small lesser in number, for adults they are larger in
number.
So, the cards are different in terms of you can you can make it as complex as you want. In
this particular case the example that we have they are different in terms of color, in terms
of shapes and in terms of the number of objects that are present, right? So, here there are
two and there is this is red and then there are these are triangles this is this is star and so
on and so forth.
So, the cards here in this particular set are different in terms of three different variables, so
color, shape and the number. Now, the game starts like this, the experimenter plays starts
playing and then starts sorting the cards as per some rule and then the other participants
will have to subjects, that are participants who are participating in the experiment will have
to play along.
So, they notice the experimenter and then they play along. Without telling the participants,
the experimenter changes the rule after a few let us say 6 or 7 times of playing, after that
the experimenter changes the rule and then the participant is supposed to follow suit. Now,
how quickly you can adapt to the changed rule of the game is what Wisconsin card sorting
task basically looks at.
So, let us say you start the game with color, sorting on the basis of color and then after 10
games you just change it into shape and after 5 games, you change it into the number of
objects present and so on and so forth. So, this can change randomly number of tiles may
not be same and so on and so forth.
So, that success in this particular game of the participants depends entirely on how flexible
you are, how quickly you adapt to the new rule that has been brought into the game and so
on.
So, basically you updating of relevant information and adaptability as part of the changed
goal is what Wisconsin card sorting tasks task looks at. This is used for both children as
well as adults, in various task conditions. So, this is again a very important aspect of
executive control.
413
This is the famous Stroop task this is what we talked about in the beginning, that Posner
and his group were talking about. Stroop task looks like this, stroop task has many different
variations, there are many different types now. Nowadays we have many different
manifestations, but this is the classic stroop task.
Here you notice the word purple is written, but it is written in red color similarly yellow
and red and so on. So, the automatic response here will be to read them as purple, yellow,
red, but the task might require you to say red, blue and green. So, this is the, this is the
classic stroop task, this is called the color word stroop task. There can be picture and words
stroop task and many other types.
But, stroop task basically looks at how you can suppress the automatic response.
Automatic response is to automatic processing is to read the word. But our task is not to
read the word, our task is to name the color of the word.
So, this is what it looks at, suppressing the irrelevant cue and basically with respect to
selective attention. This is one of the most well known psychological tests and processing
speed is often one of the outputs of this test.
There are many others also, there are tasks based on working memory like N-back task
and so on and so forth. We will see some of them when we discuss the discuss various
experiments. And of course, in today, these are the classic types of tasks behavioral tasks
and today of course, we have some many new types and which also we will see. For
example, interlocutor the presence of another person in the picture, or in presence of a
culturally sensitive cue and so on and so forth.
So, now what happens in an experimental setup is that as the participant takes part in
various kinds of behavioral tasks, sometimes simultaneously there is also a brain mapping
that goes hand in hand. So, you might have a stroop task and an EEG, or a stroop task and
an FMRI going side by side and that is how we know that which brain area is responsible
for which kind of conflict resolution and so on.
Sometimes the behavioral tasks are utilized on their own, as a standalone experimental
setup as well. We will see some of them as we go ahead. Now, so this is the background
of cognitive control or executive function as we as we have just seen.
414
Now, cognitive control with respect to language is what we are primarily interested in.
Now, why language? One of the basic assumptions is that communication itself is a
complicated process. This is a not simple thing, with though we take it for granted as I
have mentioned in the introduction that we communication, speaking, being understood
these are taken for granted, but often it does not work that way.
We do not notice, but it is a very very complicated process. So, communication itself is a
complicated process that needs organizing our thoughts, organizing the lexical outputs,
organizing the sentence structure and so on and so forth. An action in accordance with a
goal for example, choosing the right word. So, what is the right word in a friends’ group
is not a right word in a formal setup for example, for the same topic, for the same
participants and everything.
So, choosing a right word, making sense of anomalous or ambiguous sentences and so on
and so forth are some of the very simple problems we face in communication. Sensory and
motor systems typically provide clues about other people, realities of the world and so on.
If you notice in a communicative scenario we automatically update ourselves continuously
on the feedback that we get, non verbal feedback.
So, how you know that is the sensory motor in input that we are getting. So, as I speak in
a class let us say when we when we talk in a class or in a group of people. You are
constantly taking in the feedback of the other people and then you accordingly change
your way of speaking or you change the way you are the kind of words you are using or
you use simpler words, simple sentences or whatever.
It can it is entirely dependent on the participants and the your goal and the environment.
So, this is so communication happens within this kind of a complex system, even though
we do not really we are not always conscious of it, but this is what actually goes on. So,
we do, we do update, we do some play adjustments in our speak speech, even in a simple
scenario.
So, the conflicts thus can arise. So, an internal state caused by incompatibility of multiple
representations or opposition of action tendencies, this is what conflict is all about. In the
domain of language processing, conflicts usually appear between simultaneously activated
lexical items or sentential representations within or across languages.
415
So, a lot of literature, lot of research in terms of connecting language with executive
control, basically revolves around these domains. So, we have we activate multiple lexical
candidates at the same time, as we speak or as we even just hear that we have already seen
in some cases before.
So, the simultaneously activated lexical items that already creates a conflict scenario that
has to be resolved and it does get resolved without any delay. So, as we speak, let us say
as we speak, sometimes the word that I was supposed to, I had in my mind I decide not to
use it at the last moment and choose a different word.
I do not have to really pause, it automatically comes because that act that similar sounding
or similar meaning word is already activated. In a in a normal scenario, in a normal
communicative scenario we need to choose between all these possible candidates.
Sometimes even sentential representation, if you are a bilingual or a multilingual and so
on and so forth.
So, you see a simple communicative scenario also can create a lot of conflict. In fact, it
does create a lot of conflict only thing is that we do not see it, we resolve it automatically,
the brain resolves it automatically. It is for dealing with such confusions; confusions as in
competitions among linguistic candidates, their relationship with worldly realities and so
on, as well as interference; interference in case of bilinguals more importantly, of course
it also happens in monolinguals.
So, you might have different word candidates, what we call synonyms. Synonyms are
actually not an exact replica of each other, they are slightly different in meaning, but they
are similar in meaning. So, for a monolingual the conflict is choosing the right word among
the synonyms for a different, for a particular purpose, for a particular goal, for a particular
environment.
So, what is suitable for a legal document, is not suitable for a normal scenario, even if you
are talking about the same property with the same people. So, legal documents reads very
different. Similarly, if this problem gets more complicated if you are a bilingual. So, there
are too many candidates, too many lexical candidates, too many possibilities to choose
from and that creates interference.
416
So, your language one might interfere with your language two or vice versa depending on
various parameters. So, these are the some of the some of the problems, some of the
conflict that we are talking about in terms of language; language as in language processing.
So, that is why the conflict, conflict monitoring and conflict resolution is part of language
processing as well. And inhibition basically refers to the inhibiting the interferences from
the other candidates, whether it is a lexical candidate or a sentential candidate.
So, that is where we bring in executive control in terms of language processing. So,
language processing when we talk about language processing and executive control, this
is where we are going. Why do we need executive control mechanism? Because we need
to monitor and resolve conflict. Where is conflict in language? There is conflict in
language in these kind of domains.
So, these are simple things, but then there are many other complex scenario as well with
respect to language use, whether you are comprehending or producing language. So, these
are the domains that we will look at in detail in the coming sections and with respect to
various kinds of mechanisms.
Similarly, you can the speaker may use executive function also to select the right word
from competing alternate. So, these are some of the references that we have in many of
these domains.
So, one is to select the right word among computations, another is to switch from one
language to another. This is another very important area of research now in bilingualism
when we switch from one language to another what happens, what are the sub processes
that are part of it and how executive control mechanism, cognitive control is a part of that;
that is also yet another area. So, these are the some of the references.
The listener may use executive functions. So, here this is when you are producing sentence,
here when you are listening right. Select among competing interpretations according to the
417
communication goal. So, to grasp what the speaker really says and ignore what we already
the person already knows.
So, when you are listening to somebody speaking as also you need to resolve a lot of
conflicts. Sometimes you might already know, you already have a an idea that is
established in your mind, but the speaker might mean completely different things. So, that
also leads to conflict resolution.
So, this is the introduction to this particular segment. In the next part, we will look into
more detail look at look in more detail into each of these aspects of relationship between
language processing and executive control mechanism.
Thank you.
418
Language, Culture and Cognition: An Introduction
Prof. Bidisha Som
Department of Humanities and Social Sciences
Indian Institute of Technology, Guwahati
Module - 08
Part - 02
Lecture - 19
Role of executive function in language processing
Welcome to the part 2 of module 8. In part 1 of this module we have discussed in detail
about the executive control or the cognitive control mechanism in humans. We have
looked at how the different brain regions are responsible for controlling various sort of
various kinds of human output in case of, in terms of more sensory motor output as well
as linguistic output. And we have had a detailed discussion on various aspects of it.
Now, in this part, we will look at those relationship of those cognitive control networks
with respect to language functions, as in, how various types of language processing are
interrelated with the control cognitive control mechanism, in terms of the brain regions
that are responsible for those control systems in place.
So, the structure of this part is like this. First, we will discuss the relationship between
language processing and cognitive control or executive control mechanism, through
various experimental research, various data from experimental research spanning various
types of language processing.
And then we will go ahead on the assumption that general purpose brain function are
related to language related processing as well. And then there is also a controversial, this
is not entirely a straight jacketed idea, there are controversies there are debates on this. So,
the counterview of this assumption will also be discussed.
Now, when we look at language processing with respect to executive control, we will take
various types within that. So, we will start with sentence processing, word processing,
bilingual advantage and so on.
So, let us start. So, the language and executive control link has been established by a
number of, great number of research output primarily from the neuroscientific
experiments. So, recent findings have suggested that mechanisms of executive control may
419
be probably involved in language processing in these three domains, domains of sentence
processing, word production and as well as bilingual language processing.
So, these are the three domains that have been extensively researched on, extensively
worked upon and this is the data that we will look at.
So, this linking was possible due to a large amount of data coming in primarily from
neuroscience experiments as we have seen before as well, in the in part 1 of this module.
So, that the large number of data that we are talking about here, that we are referring to
here come from neuroscience.
So, these findings point out that brain areas that are responsible for general purpose
executive control are also activated during language tasks. So, this is the fundamental
thing. In terms of executive control mechanism, we have seen that there are certain brain
areas we have also talked about those brain areas. So, those certain brain areas are
responsible for general purpose cognitive control mechanism in human beings.
Now, when we link this function to the language processing task, the primary objective is
to find whether those brain regions are activated while processing language. While
processing language in conditions where there is some kind of a conflict, some kind of a
competition, which needs to be resolved. Remember we talked about cognitive control
which is activated when there is a conflict.
So, conflict monitoring and conflict resolution are the fundamental aspects of cognitive
control and executive control. So, this is what we are looking at, whether the same regions
are activated by language processing when there is a conflict. So, what are those areas? we
have already seen, but just to remind you these are the left inferior frontal gyrus and then
dorsal medial PFC as well as dorsal anterior. So, the ACC, dACC dorsal ACC we have
already looked at these areas.
So, now this means that the fundamental thing that we are driving trying to drive home
here is the mechanisms of executive control that are recruited to resolve competition
between representations in language; so, it can be phonological representation, it can be
semantic representation, syntactic representation and so on and so forth. So, when there
are competitions between different representations, between different linguistic this
representations in language processing, it may be similar to those that are recruited to
420
resolve competition between perception and attentional responsible representations as
well. So, what are we looking at here? We are looking at the competitions that may arise
during language processing, be it in any domain of language processing are we looking at
the same brain regions or not, is the question.
So, in perception and attention the general cognitive executive functions are mediated by
a network of frontal, parietal and sub-cortical structures, which we have already seen
which are those areas.
So, now let us move on with some data. This area of research is vast, it is really very rich.
So, there are number of groups working on these even you know, nuances on these in each
of these domains. So, we will discuss only some representative experiments, but I will
include a lot more references in the list for you to read. So, one of the, one of the very well
cited and very well known studies looked at this particular question as to whether there is
a similarity in conflict resolution in linguistic scenario with respect to the non-linguistic
scenario.
So, this study by Ye and Zhou explored what to what extent the neural correlates of control
processes in sentence processing are similar to those in perception and attention. So, they
used a fMRI study along with a behavioural study. In the behavioural task, they had a
sentence comprehension task, the flanker task and a color-word Stroop task.
Color-word Stroop task and flanker task both are designed to look at conflict resolution
mechanism. And sentence comprehension task is the linguistic task thereby they can
compare whether the sentence comprehension task had anything to do, any similarities to
the non linguistic task. The tasks were carried out on the same group of participants.
The linguistic task was like this. It was a sentence comprehension task and the participants
were asked to indicate the meaning of a sentence. This was the task. The meaning of the
sentence, but the problem was, the manipulation was this: sentences described an event
either consistent or inconsistent with the real world knowledge. There were simply some
sentences and they had to give the meaning.
So, sentence reading and then consequently they had to give the meaning of the sentence.
The problem that was manipulated, the problem that was underlying the task was: some
sentences had a scenario which is not consistent with the real world information. For
421
example, ‘the dog bit the man’. Typically we do not do that. Humans do not bite the; sorry
‘dog bit the man’, but inconsistently ‘man bit the dog’ though that does not happen. So,
one is consistent and one is inconsistent.
So, in this particular case this is the target scenario. In case of ‘dog bit the man’ that is
normal, that is there is no conflict to resolve, but in the inconsistent situation, there is a
conflict. What is the conflict? That there is a there is a meaning to the sentence, that the
‘man bit the dog’.
Simply there is a meaning lexical the meaning of each of the words and thereby the
meaning of the sentence, but that meaning does not really match with the real world
scenario. This was the linguistic task.
They also had to take part in non linguistic task. One was color-word Stroop task, which
we have already seen what it means. So, in this case also it was the similar task; participants
were asked to name the ink colour or the color word. The meaning of which was consistent.
So, in case of congruent, word red in red ink or inconsistent, in case of inconsistent or
incongruent: red in green ink, that we have already discussed in the previous segment,
previous part. So, what is a congruent and what is an incongruent condition in a Stroop
task.
Similarly, they also had a flanker task, participants were asked to judge the direction of
the central arrow, which was flanked on both sides by arrows in the either in the same or
in the opposite direction on the way usual simple flanker task. So, what they did was they
had these three behavioural task and simultaneously fMRI was also carried out.
So, now we have data from both the fMRI as well as from the behavioural output. Now,
our primary interest is to check whether the same general purpose cognitive control
networks are activated during both types of task. So, what they find out is observations
are, this is medial PFC, left vlPFC and left lateral parietal cortex, all of which are
implicated in executive general purpose executive control, were recruited to monitor and
resolve competitions in the sentential representation.
So, when they were trying to do the task of sentence representation, sentential
representation that is sentence comprehension task in the incongruent condition, or
condition that was incongruent with real world knowledge, they find these areas these
422
brain regions which are well known for general purpose cognitive control mechanism to
be activated. So, within this network, even they find even finer representations, they see
that dorsal medial PFC as well as vlPFC, within which this particular region has been
activated, as well as inferior parietal cortex. So, this was consistently found to be activated
for the control process in sentence comprehension as well as in the non-linguistic tasks.
So, this is one rather famous example of both language task, language processing task as
well as non linguistic task activating the same general network that is responsible for
cognitive control.
Yet another kind of sentence processing that is very commonly utilized is the category of
ambiguous sentences. Now, ambiguity can be of various types. In one particular kind, the
referential processes are looked at. So, for example, referential or syntactic process
themselves sometimes generate two possible interpretations, causing longer reading time
and more comprehension errors. For example, this sentence “Ronald told Frank that he
had a positive attitude towards life”. Now, the problematic part is this, who does this ‘he’,
the pronoun ‘he’ referred to this could refer to Frank ,this could also refer to Ronald.
So, the pronoun ‘he’ could refer to either of the two mentioned characters leading to
potential confusion. So, this is one kind of ambiguity and in while reading these kind of
sentences, what we find that both medial PFC as well as bilateral angular gyrus are
activated. Because there is a conflict, there is a conflict which means there has to be a
resolution to that conflict.
Similarly, another kind of ambiguity can arise which is considered temporary ambiguity
in case where part of the particular target word can have two different kinds of
interpretation. For example, executive functions for example, you in a sentence like in a
verb like ‘assert’ might be followed by two possible structures. One is: the direct object
assert what? do you assert? ‘I assert this’ that is one. Another could be that a subordinate
clause these are these both of these are possible.
However, more common more readily acceptable is the direct object, that would
immediately follow the word ‘assert’. So, in a sentence like this, where you have verbs
that can be followed by two different kinds of, two or more different kinds of subsequent
portions of the sentence, there is a conflict. So, in case of direct object, “the diligent
disciple asserted the belief readily”.
423
So, here ‘asserted the belief’ is the object. On the other hand, asserted the belief on the
other hand, subordinate clause can also be there ‘asserted the belief would be justified’
right. so till here this is the this is the sentences are the same, but the moment this part
appears there is a problem.
So, in a context like ‘the worried friar asserted the belief’ till here there is no problem. The
belief is temporarily it becomes temporarily ambiguous at that point when either the direct
object or the subordinate clause will be activated. Now, participants usually prefer the
direct object equivalent, direct object preference.
So, they prefer usually the direct object interpretation because it is highly frequent, this is
how mostly sentences would be formed. However, once you read till ‘would be justified’
you need a re-look at the sentence that no, this is not how it is supposed to be. So, it is
against the direct object interpretation.
So, you cannot interpret it in the usual way. So, you need a re-look and then participant
obviously, had to return to the subordinate clause interpretation and inhibit the preferred
interpretation. So, there are two possible interpretation, till you have read the ‘worried friar
asserted the belief’, after that the ambiguity starts and then that because of this competition,
control processes need to be activated.
So, what do they find? The control processes are were supported by the same areas that
we have been seeing till now, left vlPFC and dorsolateral PFC. So, these are the areas that
were found to be activated by this kind of a confusion, this kind of a competition within a
sentential context leading to ambiguity. So, these are the some of the references that have
reported these findings. This is not only one group, but there are many groups who have
find reported similar findings.
Yet another cases, another type of cases, include strongly containing sentence constraining
sentence completed by a plausible, but unexpected word. For example, ‘the children went
outside to look’. This is, this creates a constraint. Typically, a sentence like this outside till
here then you do not expect the word ‘to look’ is not expected as much as ‘to play’ would
be expected.
A similar typical sentence would ‘the children went outside to play’ not ‘outside to look’.
So, this is where again you need an inhibition of the preferred interpretation thereby there
424
is a kind of a conflict that needs to be monitored. So, once one the what we have seen in
the last three experiments is that three types of experiments is that the there are different
kinds of interpretations of a sentence. Sometimes on a one particular interpretation is more
common, it is more high frequency word or some or something of in that line. So, then,
but the experimental paradigm is such that the non frequent interpretation was activated.
Now, this resulted in a strong inhibition, this had to be inhibited, this automatic
interpretation had to be inhibited in order to facilitate the task, given the given task. And
in all these cases what we see is that a particular network that are responsible for executive
control are activated.
Similarly, we find the same things here as well, posterior positivity or an anterior
negativity in ERP, this is not an fMRI study, this was an ERP study.
Thus, what we see here is that sentence processing that was dependent upon conflict
resolution and inhibition of irrelevant information, has been consistently found to be
involving the general purpose cognitive control network of the brain
Word production has various types. Now, speaking itself includes searching and picking
the right word that is appropriate for a particular given , that we already have discussed
before. So, sometimes the right words are automatically activated as we have just seen that
the children went out to play, it is automatically activated. The word play is automatically
already there for you to be used, to be retrieved.
However in certain situations, speakers must voluntarily retrieve the right words from
semantic memory and sustain them against competing alternatives. So, in the previous
case, we have seen that the children went out to look now to look. Because they are reading
the sentence as that, they had to consistently keep it activated and have a constant
inhibition of the ‘to play’ candidate.
So, this is what happens in case of word retrieval. So, of in a famous case in a famous
paper by Thompson and Schill et al., 1997, they carried out a study, where participants
were asked to generate a verb related to a noun, which was associated with many items,
without any clearly dominant response. What does it mean? Sometimes, some nouns are
425
connected to many verbs or a verb is connected to many nouns were automatically
connected, automatically generated, they are high frequency associations.
So, for example, ‘hanging a coat’ or ‘singing a song’ and something like this. So, these
connections are these associations are automatic. Sometimes there are some associations
which are not that automatic which are kind of weakly associated. So, what happens then,
this is what they try to find out.
So, if you have a case where nouns and verbs are having an associate that is comparatively
weak, then there this leads to high selection demand, because due to higher competition.
Because there are too many words that are connected and, but all of them are weak
connections.
So, now you have to go search for one that is apt or if it is given to you already, for your
attentional mechanism to be sustained on that you need to continuously suppress the more
strong connections. So, what they found in this case was again the left middle vfc was
activated. Not only that, it was also found that patients who had a focal lesion in this
particular region, could not make a response when the cue noun had many associates.
So, this shows that not only in normal participants, normal subjects they had a control
process in place when weak associations were processed. They also found that in patients
who had a lesion in those areas, that was found to be activated if you have a lesion in those
places the patients could not form the associations at all. They could not make a response
to with when the cue noun had many associates.
So, not only in retrieval, but also in access in lexical items, the control process in accessing
lexical items have also been found to utilize similar kind of network. For example, in a
task in a yet another famous study, in this was an association task again. So, participants
were asked to select target which was associated with the cue.
Now in one case for example, ‘candle’ is more strongly associated with ‘flame’ rather than
with ‘bald’, similarly candle is weakly associated with halo over ‘exist’. So, this is the
background within which they carried out the study.
Now, if the association between the cue and the target is weak again automatic retrieval
processes are insufficient. So, it cannot be, if the target cue and target pair are weakly
426
associated, automatic retrieval will not work. In that case there has to be a voluntary search
operation for the participant to go back in semantic memory and retrieve.
So, that control that there is a control mechanism at that level as well and this, as expected,
activation was found in the left anterior vlPFC. So, what we see is that very often most
consistently vlPFC has been found to be activated in many such studies, many such
experiments, whether it is sentential or at word level processing, word level processing
when there is a conflict.
Semantic processing as well have found the same kind of same kind of activation level.
So, again what we see is the VLPFC is critical for performance of tasks that demand access
to an evaluation of semantic knowledge. In fact, this is connected to the study we just
mentioned. This particular group as Badre et al., has have done a number of series of
studies on this in this particular domain. And in all cases what we see is that the consistent
finding is that ventrolateral PFC is almost always found to be activated.
So, ventrolateral PFC is critical for various kinds of tasks demanding semantic knowledge,
retrieval, word retrieval and access to word retrieval in the semantic memory and so on.
So, left VLPFC control mechanisms are critical when a subset of knowledge that is task
relevant, must be selected over a competing task subset of irrelevant knowledge.
Irrelevant and it is it becomes more competitive when the irrelevant subset of knowledge
is more, is higher frequency word or more commonly associated and so on. So, that is the
that is the scenario when inhibition has to be stronger. So, control mechanism has to be
stronger and that is when we find this kind of activation of various general purpose
cognitive control network.
Now, even though we have enough already a lot of evidence coming from various types
of language processing, be it sentential level or a word level or at the level of meaning and
so on, the largest amount of data, largest amount of work has been actually coming from
bilingualism.
A lot of interesting work has happened. So, as a result of which, large amount of literature
actually exist in connecting language and executive control in this domain. The reason is
that both the linguistic systems of a bilingual are active when even when only one is in
use. This automatically creates a state of competition that needs to be dissolved.
427
A bilingual is already bilingual mind is already all the time in a state of competition
because the two languages of a bilingual, or many languages of a multilingual, are all the
time active even though there is no demand on the other languages. Even when only one
language is being utilized, other languages are simultaneously active.
Now, this is already well known knowledge, it is already has been has all been established
for a over a long period of time now. So, what we know in this domain already is that both
orthographic and phonological properties of the words of the two languages are active and
they influence performance, even when the bilingual is highly proficient.
One would expect that a bilingual, who is not very high proficient in the second language,
would be dominated by or would be would have more interference from his first language
on the second language. But that is not the case. The findings suggest that even in case of
very high proficient bilinguals, the properties, the orthographic, the semantic, the
phonological properties of both languages impact each other in case of any kind of
bilingual language processing, even when only one language is utilized.
And so, work initial work focused on cognates and inter lingual homographs to prove this,
but now a large range of data also point the same situation. So, what is a cognate? Cognate
is are those words that have form and meaning converging. So, in case of for example, in
the in case of Dutch and English pair the word ‘hotel’ is a cognate. Because in both English
and Dutch they look the same, they are pronounce the same way and they mean the same
thing.
So, the form and meaning are same, but in some words they are even though they look
similar, but they are not the same. So, the word ‘room’ in English is of course, we all know
it is a room what room means, but in case of Dutch the same word means cream. So, this
is an inter-lingual homograph. So, they are written similarly, but they do not mean the
same thing.
So, experimental paradigms that have used this kind of words that have, that give us a
large amount of data as to how this interference really works. So, both spoken and visual
word recognition prove that bilinguals do not function like two monolinguals. Meaning
that bilinguals’ both languages interact with one another and as a result this is not a
bilingual is not two monolinguals put together.
428
So, when the when a bilingual speaks in language 1, only language 1 is active, when he
speaks in language 2, only when language 2 is active, it does not work that way, all the
languages are active all the time. And this effect of interference from both languages on
each other are seen. So, it is also this effect is also found in case of languages that have
different script. One may argue that languages having same script like Dutch and English
have some similarities, so that there is interference.
But it also found in case of different script languages like Japanese English and even in
bimodal bilinguals. Bimodal bilinguals are those bilinguals that use two different
languages two different types of language using different modalities. So, one is the verbal
language another is the sign language.
So, in case of bilingual who speak one language and sign another even in that scenario we
find interference of one language on to another. This is also seen to the extent this effect
is so strong that this is also seen when speakers hear unrelated L 1 as background noise
while speaking in their L 2.
So, many of these many of many researchers many groups have looked at these issues all
of these issues and in their finer nuances and thereby using tasks that require bilinguals to
suppress the irrelevant language. So, there is a there is a task in Dutch going on in the
English is not to be utilized, but the experiment still finds impact of English language on
the Dutch language performance and so on.
So, in all those cases in all these different kinds of studies, what we find is that the
suppressing the in irrelevant language and sustained activation of the target language are
also are controlled by or at least supported by the left dlPFC as well as the supplementary
motor area.
Similar kind of control mechanism are found in language switching as well. Language
switching experiments are very fascinating in terms of bilingual language processing
429
language switching tasks require participants to switch between languages, so on the basis
of a cue.
So, they have they are participating in, let us say, a picture naming task. So, there is a
picture on the screen a line drawing basically and they have to name the picture. So, for
some time they will name the picture in language 1 and then there is a cue on the screen
and they have to shift to language 2 and now use language 2. There are many such many
manipulations of this basic design.
It can be a picture naming task, it can be a comprehension task. Basically switching the
task would require you to utilize one language at a time. And then after that there is a cue
they then you have to switch. So, there are many manipulations, many permutations and
combinations of such tasks.
Sometimes there will be after each after two three tasks there will be a change. Sometimes
there are single language groups and in between there is a mixed language group followed
by another single language group and so on. So, language switching tasks have various
paradigms, but primarily the task is to switch between two languages on the basis of a cue.
Sometimes the switching can also happen within language, in that case the task is switched
not the language.
So, if it is within language switch the task is different. So, in case in one case identifying
noun, in another case identifying verb, but both within the English language, let us say.
Similarly, but in case of within language between language switching, you are naming the
word in one language and then naming the same thing in a different language; these tasks
are aimed at finding the switching cost.
Now, switching cost is a very very important concept in bilingual research. And basically
it shows how much time is taken how much time is let us say how much cost in terms of
time you pay when you switch. So, switching from L 1 to L 2 versus switching from L 2
to L 1, what is the time lapse which is compared to the single language block.
So, if you have a one block where you have to name only in English, how much time you
take for how, what is the average reaction time as opposed to when you switch. So,
switching cost is an indicator of many things in bilingual bilingualism literature. Typically,
switching cost reduces when the proficiency of the bilingual in L2 goes up.
430
But in any case, switching cost basically is what we find try to find out and how it is
modulated across variables. There can be many variables. Proficiency is one, but there can
be many other variables. Now, this particular task will show this particular switching cost
shows the language control ability of bilinguals in various scenarios, in various conditions,
various experimental conditions that manipulate different kinds of variables.
So, participants were asked to name pictures of objects in English, if it was cued with the
word ‘say’. So, basically there will be a there will be a cue either written or auditorily the
word ‘say’ comes and then they will say the name of the object, they will name the object.
And if it is or if it is in Spanish [diga] then they will say it in Spanish. So, this was the cue
that is how the switching happened.
So, in some cases the word ‘say’ will appear and then they have to name the picture in
English and when it is the Spanish version they will have to speak in Spanish. So, there
was a switch on this ground.
This experiment also had within language condition. In case of within language condition,
participants were asked to name a picture if in English as the action depicted. In one case
they were supposed to name the action that is the word if the or if it was cued with the
word ‘to’. So, if there is a cue said ‘to’ now you have to mention the verb what is happening
in the picture.
So, in this particular case the boy is eating a hotdog. So, what is happening? What is the
verb here? How do you know you have to say the verb? By the cue by using the cue ‘to’.
So, when the word ‘to’ comes participant had to say ‘eat’. However, on the other hand if
the cue word was ‘the’ then they have to name the object what was being eaten. So, in this
case, ‘the hotdog’. This was the this was how the experiment was conducted.
So, they were both within language and between language switching on either on task or
on language.
431
What they find is that for both the between language and the within language conditions,
the switching processes were sub served by again left dlPFC, left vlPFC and we already
know anterior cingulate cortex which is ACC and the left caudate. These are the regions
that were found to be activated during the process of resolving the conflict in both cases,
resolving during the switching process.
The bilingual advantage question or the debate has been a very a hot topic within
bilingualism research. This basically talks about a particular kind of advantage that the
bilinguals may have over monolinguals. Now, the findings from bilingual as well as
monolingual language processing point to the fact that co-activation happens between
language processing and general purpose cognitive control neural networks.
So, this led to the belief that bilingualism that led to the belief that bilingualism give rise
to better cognitive control. So, in case of a monolingual also, there is you find whenever
there is a conflicting scenario, the general purpose cognitive control network gets
activated.
In case of bilinguals, that kind of a competition is all the time present just by virtue of
being a bilingual, we are supposed to be juggling between two different systems. Two
different systems as in, in terms of phonology, morphology, syntax you name it all kinds
of. So, there is always a process of control in place, as for the researcher. So, being a
bilingual automatically means suppressing the task irrelevant language.
So, right now I am speaking in English, but all the other languages that I speak, are also
simultaneously activated or they are active to be utilized. But I have to, so my brain has to
suppress those languages constantly and keep my target language activated as long as this
is relevant.
432
So, this particular idea has given rise to the notion, given rise to the hypothesis that because
the bilingual is constantly employing the control mechanism just by being a bilingual, just
by even when they are speaking in only one language this might, this probably could have
a spill-over effect on the general purpose cognitive control mechanism itself.
For example, there are many various kinds of cognitive tasks various kinds of challenging
demanding cognitive tasks that have been found to be having a spill-over effect on other
types of executive mechanism. For example, when you lot of research has gone into
meditation. So, by meditating you can train your brain. Similarly, by training hard to, even
body builders by training hard they are basically training their selective attention
mechanism this might have a spill-over effect on other domains.
Similarly, people who play video game for long period of time will have trained their
selective attention and sustained attention for a very long time, which has been already
found to be having positive effect on many other skills. So similarly, bilinguals because
they are constantly they are constantly employing that cognitive control mechanism, this
might this could actually lead to having a higher cognitive control capacity in the bilingual.
This is famously called the bilingual advantage.
So, research in this domain has been very very fascinating and also rather controversial.
Now, let us start with the developmental point of view. A lot of research has focused on
how being a bilingual makes us smarter, makes people smarter or whether does it make it
make people smarter. So, a large chunk of research actually focused on children. So,
bilingual children versus monolingual children and how they function in various kinds of
tasks that require response inhibition?
So, basically how well established the cognitive control mechanism is or how whether one
has a higher or superior cognitive control mechanism. So, but to because for a bilingual
we have to constantly see, to choose which system to use and which to stop from
interfering and we have to also learn to ignore irrelevant information. This has the
consequence of creating more general expertise in resolving conflict.
Now, Ellen Bialystok’s research of many decades have shown that bilingual children
outperform their monolingual counterpart in simple tasks that require cognitive control,
that require in irrelevant information to be ignored. She has used, she and her group has
433
used, various kinds of tasks Wisconsin card sorting task and various simpler tasks for
children to show, to find out whether bilingual children have a higher response inhibition
system in place. And more or a more less more often than not the findings point to that
possibility that bilingual children, children who are learning 2 languages or who speak 2
languages, more commonly have outperformed their monolingual peers in tasks that
require inhibition, in tasks that require irrelevant information to be ignored.
Yet another domain which is again very controversial, but some findings point to the fact
that a bilingual people, a bilingual person probably has an advantage over monolingual
peers in terms of the onset of the various kinds of diseases like Alzheimer’s and so on. So,
there has been. So, this has led to the belief that the bilingual people have, in the elderly
population bilingual people have an kind of a protection against cognitive and attentional
deficit, decline that typically affects people in the age after 60 years.
So, Alzheimer’s and Parkinson’s and so on, the large number of data has been collected in
this domain to show. In fact, a large a number of data actually point up to the fact that they
have later onset of the diseases which has been of course, challenged and there are lot of
work going on in that domain till now. So, these findings prove that bilinguals are able to
manage their attention to a complex set of rapidly changing task demands which is better
than their monolingual peers.
So, Bialystock’s one particular study we will mention here of Bialystock; this was in 2004.
So, she had taken both old and middle aged monolingual and bilingual subjects on several
variations of Simon task. Remember Simon task, that is there are that is the that is the
special correspondence task between source and target. So, the task in this case was to
press right shift key with right index finger on seeing a brown box and left shift key with
left index finger on seeing a blue box.
So, we have already seen this task, how it goes. The findings shows that response was
faster and more accurate when there is a special correspondence, as a as expected, between
source and target. And when there is a there is the correspondence is not there when it is
an incongruent condition of course, response was slower. But more important finding is
that the bilinguals outperform monolinguals in these studies.
Now, this kind of findings and many other similar kinds of findings have now taken have
been now taken to a step further. And one of them one of the very important in very and
434
an influential hypothesis in this domain is the ‘adaptive control hypothesis’ of Green and
Abutalebi in a famous paper in 2013. They have talked about that bilingualism has the way
the kind of bilingualism that we practice, may have different kinds of reaction, they may
have different kinds of effect on the different kinds of cognitive control processes.
So, what they basically talk about is the interactional context of bilinguals. Interactional
context they define it in three different ways. So, there is a three way difference between
in terms of switching language. So, language switching, we have just seen that language
switching and task switching are similar in terms of response inhibition and so on.
So, depending on that they have a three way difference single language, dual language and
dense code switching context. And they say that depending on what context on the
bilingual actually comes from will have difference different kinds of effect on these
various types of cognitive control process; goal maintenance, conflict monitoring,
interference, separation salient cue detection.
So, basically they further segment the cognitive control processes in these sub processes
and they say that each of them might get differently affected by the kinds of interactional
context that the bilingual comes from. So, adaptive changes also will happen in the neural
regions and circuits associated with specific control processes.
So, this is a very powerful hypothesis and there is a lot of support for adaptive control
hypothesis from literature as well. So, I have just quoted a few. So, interleaved picture
word comprehension and Flanker task in Chinese-English bilinguals find this find the
support.
Similarly, language switching task with English-Mandarin bilinguals. Similarly it was also
reported that bilinguals with balanced usage of both languages showed lower switch cost.
This has been also found out in by Albert Costa in case of language switching. So, if your
proficiency goes higher, your switching cost goes down. So, they find that lower switch
cost in case of balanced usage of both the languages in trial making tasks than bilinguals
who spoke only one language at home.
So, people basically it is the usage of language of a bilingual that also. So, you see it is not
only just being a bilingual that affects your cognitive control mechanism, but also the kind
of bilingual you are, in terms of using those languages. Whether it is a single language
435
context, dual language or a dense code switching and they find different fascinating results
with that.
So, there is a lot of support. A similar kind of result was also reported from Stroop and
delay gratification task in yet another group’s result. Of course, there are also findings that
do not support the adaptive control hypothesis and as a result there is a relook at the finer
aspects of this hypothesis.
More recently, bilingual’s cultural context has also been investigated with this within this
paradigm. So, bilinguals what kind of cultural context and whether it gets, whether it does
affect, whether a bilingual’s being a high proficient bilingual or whether it is in the person
is a bicultural bilingual or just a bilingual without the cultural aspect attest and so on.
So, basically various subtle very subtle nuances within bilingualism is now being looked
at to understand cognitive control mechanism that is connected, that is considered to be
connected with bilingual bilingualism. So, the level of control in the presence of
incongruent culture cue has been found to be different in different kinds of bilinguals.
So, the famous studies there are some famous examples from these Chinese-English
bilinguals in language production study where incongruent phases created a delay in the
response in the L 2 in the English language. So, pointing this particularly points to the
cultural angle to language cognitive control relationship.
So, it is not just the fact of bilingual not all bilinguals show the same kind of control
mechanism to be in place. So, what is happening more, what is it only the abstract kind of
just speaking two languages or it is certain other factors that are part of know being part
of a community, speech community part of speaking a language and so on that is now
getting investigated.
In one such study in by Roychoudhury et al., in 2016 showed the impact of culture cue on
naming. So, this was this study was carried out on Bengali-English bilinguals. So, the
cultural cue here is the Howrah Bridge and then they had to name, this was a picture
naming task. So, they see the picture of the Howrah Bridge. So, automatically the moment
you see Howrah Bridge your cultural aspect that is activated is the Bengali culture.
436
So, then there is a there is a gap and then there is the picture emerges and then the task was
to name the picture. And so, in whether when this was this is an incongruent condition.
You see a Bengali picture and you have to name the picture in English then there was an
incongruency effect in congruency effect in the sense of inhibition. So, response was
slower in when there was an incongruent picture.
Basically, this refers to the fact that the cultural background of a particular language does
have an impact on the other language’s cognitive control capacity. So, how much you are
able to inhibit those responses? So, in this case that inhibition was not found, it was found
that the cultural cue was actually affecting. So, it was not only active, it was interfering
with the second language output
So, these two words mean the same thing. The task was to say whether the second word is
a translation of the first word or not; manipulation was this picture. So, in between two
pictures there was a culturally sensitive image, cartoonized image of a person wearing the
culturally specific dress.
So, in this case and, so, when the task was to see if it irrelevant incongruent condition
results in higher reaction time or more errors. So, in this case however, the results do not
show any kind of inhibition, whether it is a congruent picture with this language or not,
the results do not show any kind of inhibition.
So, basically this shows that this is the particular kind of bilingual community, who do not
get. So, basically there is a higher response inhibition that is present in this particular
community that was studied.
So, basically this refers to this takes us to the possibility that the kind of bilinguals that are
studied will have to be looked at a very finer level. But overall, we can sum up the results
by saying that the studies mentioned show a relationship between general purpose
cognitive mechanisms of the brain and language processing. This is a kind of this is the
437
general outcome of the all the research that have been just now talked about, basically the
relationship between language processing.
This view has been proposed by Evelina Fedorenko and her group and there are also some
other groups who are now working on the finer nuances into this. Now, there are new
evidence that are emerging from the studies of these various groups of studies, groups of
neuroscientists that the overlapping of general purpose neural network for control and
language function is too simplistic.
What has happened in the past as per Fedorenko and others is that, the finding that have
been reported is too simplistic because they are reporting always a large area. In terms of
the brain, in terms of cortical region we have to be sure about you know locating a certain
functional domain in a very very small, tiny area. The just saying vlPFC is active is very
simplistic. So, this is the this is the standpoint that they are taking.
So, vlPFC fine, but what area within vlPFC is what they are talking about. So, general you
can just mapping them together is not enough. And they have proposed this is the result of
many years of research by these groups and now they propose that there are actually two
networks in the brain. There is one general area and one language related area. So, they
name them like this ‘language related area’ and a ‘multiple demand network’.
So, there are two networks that are there in the brain that are responsible for different kinds
of control mechanisms. So, and these areas lie side by side. So, basically finer
methodological interventions is what they propose, that are needed in order to find out the
subtle differences between these two areas.
So, what is language area? Language area is basically, they propose is only in the left
hemisphere and it shows robust activation in language comprehension and production. So,
all kinds of language processing that needs control mechanisms, that is there in the left
hemisphere.
438
So, these are the what are the areas? Frontal, temporal and parietal regions of the left
hemisphere. So, they have mapped out a particular network within these three areas that
are in the left hemisphere and that show a very strong activation for language related
processing. This is particularly selective for language processing over non-linguistic task.
So, what does this area do? This supports word meaning and combinatorial processing.
So, various studies that needed a resolving, meaning related or semantic related
competitions, similarly various kinds of combinatorial processing as well. Combinatorial
as in terms of both syntactic and semantic processing of within language processing that
is what they found was to be this was this area to be responsible for.
Studies show that this region is not activated while performing non linguistic executive
functions. Remember if we go back a few slides we have seen that various studies have
pointed out that participants when they participate in resolving conflict in a linguistic
scenario versus when they are resolving a conflict in a non-linguistic scenario the similar
kind of brain regions or sometimes overlapping brain regions have been found to be
activated.
However, research by these various groups they show that this language related area,
language area let us call it, that language area is not activated when non-linguistic tasks
are, non=linguistic executive function related tasks are carried out. So, in case of let us say
Flanker or Simon task or any such Stroop task, whatever they are carrying out the this
language area do not show activation.
What they show? These areas show only strong activation for language processing. So,
they have used finer mechanisms in order to find out the differences
On the other hand, what they call multiple demand network, which is called MD network
in short; multiple demand network is present in the brain bilaterally. What does this mean?
Whereas, on the one hand the language related area is only there in the left hemisphere the
MD network, which is the multiple demand network, is present bilaterally that is in both
sides of the brain, both the hemispheres and this is there in the frontal and parietal cortex.
Of course there are the smaller areas that they have mentioned, but I am just giving you a
brief overview. Now, this area, what they find out is this area is responsible for different
439
kinds of demanding tasks be it arithmetic, deductive reasoning, problem solving so on and
so forth, but not language.
So, various kinds of demanding tasks that requires a control mechanism to be in place they
find the this particular area to be active. And this is also sensitive to the difficulty level of
the task. So, depending on how difficult a maths problem is that will be again that will
again show a different kind of activation level. Similarly, deductive reasoning, problem
solving and so on. So, this particular area is found to support goal directed behaviour,
cognitive control, attention so on and so forth.
We, this is what? These are the group these are the things that are grouped together as fluid
intelligence we have already talked about. So, executive control mechanism, executive
function or cognitive control or whatever you call it they are basically the hallmark of an
of intelligence.
So, you have a goal directed behavior here, cognitive control all these are supported by the
multiple demand network. So, it can the you can have various kinds of demands on this
network that can that are non-linguistic in nature. So, it can be anything that requires this
kind of control.
So, as a result of which goal directed behaviour, cognitive control in short a fluid
intelligence is controlled by this particular area.
So, the finding of these groups is that language and the MD networks are functionally
distinct. So, whereas, language area is responsible for conflict resolution in case of
language processing, on the other hand MD area is responsible for conflict processing,
gold directed behavior and so on and so forth, but not language processing. And they also
find out that MD network do not support core aspects of language interpretation.
However, this has to be taken with a pinch of salt, because Fedorenko herself has pointed
out in one of her latest papers that there is some there seems to be slight activation of this
network in certain kinds of language processing, certain kinds and they are still working
on this. But largely, on overall, the MD network do not support the core aspects at least of
language interpretation.
440
Now, what do they call the key signature of linguistic processing? This is very important,
what is considered a key signature of linguistic processing and what is a peripheral
processing. So, stronger response to meaningful and structured linguistic representation,
this is what they take to be the key signature of linguistic processing. These are the core
linguistic processing, language processing related areas and this in terms of these in
representations MD network do not have a role to play, but language area does have.
More evidence come from different other researchers as well. So, in case by using signal
fluctuations between the brain regions in using both within and between systems they also
show no overlap between the two networks. Also some researchers have pointed out that
the time course of development of language and executive control mechanism in children,
from childhood to adulthood also differ.
So, when a 3 year old child can have the language system in place perfectly fine in a
normally developing child, the executive control mechanism gets properly kind of settled
by adolescence or young adulthood. So, even in terms of the developmental aspects of
these two functions are different. So, how can they be how can there be an overlap, because
language function is developed far more quickly than executive control functions and so
on and so forth.
So, there are evidence coming up from various domains of research and it is still a very a
richly debated claim. So, one question that arises is that, why did not the previous
researchers find this out? Why do we have such a large number of research output pointing
to the fact that general purpose cognitive mechanism is also responsible for language
processing?
The reasons that Fedorenko gives is that the it is primarily a methodological issue.
Methodological issue because the typically the problem arises because there is an
averaging of brain areas in common space. So, basically we talk about the you know the
DLPFC or the mPFC and so on and so forth.
This is the large area. And basically we average the there is an averaging of various
individuals and then finding out, but then there are, that cannot be done because across
individuals precise location for language area may vary. So, this is what actually they are
working on. They are looking at individual level rather than at an averaging kind of a
study.
441
So, averaging brain in a common space and then voxel based performance, these are the
find these are the problems that she refers to why we have the kind of result we have had.
So basically, the methodology has to be changed there has to be more fine tune methods
to be utilized and also taking yet another very important aspect into the consideration
which is the inter-subject differences.
So, each subject, each human has slightly different regions for language functions. So, that
has not been taken into account by most of these studies and that is why it is kind of
simplistic in her terms. And also because these regions lie side by side, they are very close
to each other. And so, there is always a possibility of finding of you know confusing one
with the other.
So, there are control mechanisms in the brain that are sensitive to language processing of
course, there are. But these are different and distinct from the general purpose cognitive
control network. So, this is how it looks like at this point of time; this also points to a
possibility that linguistic mechanisms are more closely linked to social mechanisms.
This is a claim that Fedorenko and her group has made that the reason why we have a
different distinct network and the way it behaves, the way it the control mechanism really
spells out is probably because language is more connected to social mechanisms. And thus
that is why we find we have to give a more focus on meaning and communication.
In fact, that is what her most of her tasks also do, that the that what they have developed.
So, meaning and communication are the core of language mechanisms and that is why it
is also connected to the social mechanism. So, this field is growing every day and of
course, newer evidence is emerging.
Till few years back the received wisdom was that language functions, language processing
and general purpose cognitive control mechanism are the same; they use the same control
network neural network. But now there is a challenge to this and there is lot of data coming
out from these different groups. And we are still learning, what is what the story is still
unfolding gradually.
So, this is where we will end the part 2 of this. There are lots of references, I will add at
least a couple of reference for each type of studies. And that will give you a good enough
442
idea about how the research, what are the directions that the research has taken place and
what are the primary findings. In the next segment we will take up simulation.
Thank you.
443
Language, Culture and Cognition: An Introduction
Prof. Bidisha Som
Department of Humanities and Social Sciences
Indian Institute of Technology, Guwahati
Module - 08
Part - 03
Lecture - 20
Mental Simulation and its relation with language
Hello and welcome, this is the 3rd part of module 8. So, this is essentially the last part of
this course. In this course, in this part of the course, we will look at Mental Simulation and
its relationship with language. So far, we have seen language in its various facets in
structure as well as in processing, and how various kinds of dynamic relationships actually
play out while we process language and so on.
So, today we will look at mental simulation. Now what is mental simulation? Mental
simulation refers to reconstructing, recreating an a kind of an event or an object or an
understanding of an event and so on. So, once we have we have encountered a situation in
life that gets represented in the mind in terms of the various modalities that are part of it.
And then this is something we have looked at through image schema and its ramifications.
Now, what happens when we face similar situations again, either directly or indirectly?
That same experience gets recreated and that is how we try to understand or we do
understand those events. So, that is simulation, that is mentally a mental simulation, we
simulate a particular scenario.
And so this is how we will this is what we will look at in this part. The structure of the
lectures like this. So, first we will see the position of mental simulation in terms of
cognition. And then we will look at various experimental studies, various evidence that we
have gathered so far and how that what is the relationship of those evidence with the
theoretical standpoint. And then because this is the last part of the course, we will also
summarize the entire course.
444
Let us go back a little bit and let us refresh our memory a bit. We talked about we started
this course with the question as to what is cognition and then we discussed various theories
or in terms of what constitutes cognition.
So, in one of the most important and in fact, the most dominant theories of cognition talked
about a kind of an abstract, amodal kind of a interpretation of cognition. So, this particular
theoretical point says that human mind manipulates abstract, arbitrary and amodal
symbols.
And this combination of manipulation and symbolic representation is what is the basic
understanding of what cognition is. One of the most important aspects of this theory is that
the mind has symbols which are abstract, arbitrary and essentially amodal. This we have
discussed in the first part of the course.
So, and as a result of this, since the mind stores various kinds of symbols and then
depending on the kind of action necessary that there is a manipulation and various kinds
of combination of manipulations, and that ultimately results in what we call cognition. So,
that is basically the early view of what is called the disembodied cognition.
However, the problem with this set this idea of cognition is that abstract, amodal and
arbitrary symbols are not really connected to the real experience. They are abstract, they
are the abstraction of what has happened. So, an image schema is just the skeleton of a of
an experience. So, that does not have much connection with the real experience.
Real experience is far more varied, far more it has many more angles to it, many more
modalities attached to it and so on. Hence abstract, arbitrary, symbol manipulation cannot
be said to have a direct relationship with the real experience. This is beautifully put forward
by Zwaan, in this ‘they are floating free in some mental ether and are therefore, essentially
meaningless’.
So, it cannot be. So, it is essentially not tenable. So, the other view on this is that, modal
representations play central role in knowledge. So, modal representations basically talk
about the sensory motor representation; various modalities of visual, auditory, tactile and
various other, all the five different sensory modalities, do play a significant role in creating
cognition, in understanding, in cognizing per se.
445
So, starting from the ancient philosophers to empiricists, to nativist philosophers, many
believed that images of experiences play a role in knowledge representation. So, it is it
cannot be amodal, it has to be modal and there because the images, the entire gamut of
feelings associated with it sensory motor input associated with it are also part of cognition.
So, as per this standpoint the brain captures modal states during perception, action. These
are the two very important notions that we have been seeing till now. So, while perceiving
something, as well as while being indulged in some kind of an action. So, the brain
perceives those modal states in terms of those actions and perceptions. And then simulates
the same to represent knowledge.
So, when one comes across similar situation again the same multi modal states that is the
same multimodal sensory motor states are reactivated, and then an appropriate action is
followed. So, we understand a new scenario by simulating a similar situation that we have
already encountered. And then we already know what to do.
So, if you have if you have already played with rubik’s cube and you find a new rubik’s
cube which is slightly different, but you still know you can still simulate the notion of how
to play rubiks cube, what are the steps to be followed, how is the problem to be solved,
then you simulate that knowledge while playing a new rubik’s cube.
This is also true even when you are not doing, you are not indulging in the action yourself,
but you see somebody else doing the same thing. For example, you watch your friend
playing rubik’s cube. So, you understand his moves by simulating the moves that you had
earlier utilized.
So, this is how simulation really works. So, similar reactivation can also be used for
inference, recollection and language; this is what our primary focus is. So, simulation is
important not only from a perspective of understanding and inference, but also recollection
and more importantly for our purpose in case of language.
A related notion with the aspect of simulation is that knowledge representation is situated.
So, it is not only not abstract, not only it is a multimodal kind of a storage where we have
all the information necessary, for visual, tactile, auditory and all other things, but also the
background information.
446
So, the context and the context contextual relationship with those inputs are also important.
So, there are two primary aspects to this: one is that knowledge representation is modal, is
multi modal actually. And then that representation is simulated at a later instance for the
purpose of understanding, recollecting, inferencing or using language. Simultaneously this
representation is also contextualized.
In this regard, we have already discussed Harnad’s theory about the ‘symbol grounding
problem’. So, let us just quickly recollect that Harnad argued that ‘serious theories of
cognition should look at symbols that are grounded in perception and action’, that is what
the symbol ground the very well known symbol grounding problem proposed by Harnad
is.
So, symbols or any good theory of cognition needs to take into account the grounding, as
in, the ‘rootedness’ of those experiences. Experiences and their understandings cannot be
devoid of the real experience, of the grounding in reality that is what his theory essentially
talks about. After Harnad, many other researchers have echoed the same theory and they
have looked at various experimental domains to find proof for that.
So, one of the one of the most important names in this regard is Lawrence Barsalou. His
focus has primarily been on language. So, and he argued that language comprehension
should not be viewed as a disembodied manipulations of a symbol like language, this is
the theory that was also given by Searle.
So, you cannot understand just because you can manipulate certain symbols does not mean
you really understand language. So, same thing Barsalou also said that language
understanding or comprehending language even production of language is dependent upon
a manipulation of symbols that is related to the perception and action.
It is not like a computer manipulating symbols. So, after a Harnad and Barsalou’s theories,
a lot of studies have taken place a couple of decades now and there have been studies from
both psychology and neuroscience as well as linguistics that have strengthened this
domain.
So, what does Barsalou really mean, why how what does it, how does it really work that
comprehenders need to activate or simulate the real life scenario real life scenario of a
situation? He gives this example now famous: ‘the pencil is in the cup’ and ‘the pencil is
447
in the drawer’. There are two ways that you can that he has given two possible scenarios,
one is the pencil is in the cup and the other is pencil in the drawer.
So, when you read the first sentence the pencil is in the cup you look at this you think of
pencils in this way, that this is how the pencils are arranged. Pencils in the drawer this is
how you in some manipulation of this kind. So, basically the difference between these two
is that the pencil remains the same.
However, in the first case, the pencil is in vertical position, in the second, it is in the
horizontal position. As per Barsalou, this is this particular part of the information about
the orientation of the pencil is an integral part of us comprehending the two sentences
differently. We perfectly understand this sentence because we know what it is what it refers
to.
Similarly, what it refers to and this understanding is takes into account the relative
orientation of the pencil. This is what basically Barsalou talks about. So, he argued that a
pencil’s orientation as well as other spatial details should be an ‘automatic by product’ of
a mental simulation. If you understand the sentence pencils are in the box or the on the in
the cup, you automatically are able to simulate the orientation of the pencils, this is what
he says.
So, understanding language and in terms of mental simulation, what are the primary what
are the main important factors? One is language comprehenders construct mental
simulations of the content of the utterances’. Remember, we have always focused in this
course in the on the meaning part of it, on the conceptualization aspect of language, not so
much on the structural part of it.
So, here also he talks about the mental simulation of the ‘content’ of the utterance not
necessarily how the sentence syntax was formed. So, language is understood through
sensory-motor simulations of the actions and events being described. Just as I just
mentioned that we understand. So, ‘he was playing football’.
The playing football the event that you are talking about is understood perfectly fine by
anybody who listens to the sentence, because we are capable of simulating the same
experience. So, that is what basically talks about that actions and events all are getting
448
simulated as we listen. So, data from various behavioral experiments as well as brain
imaging studies have found proof for this assertion.
Now, what are these? There are mental simulation is not a homogeneous entity, it has
various finer nuances within it. So, some of these finer perceptual details include motion,
shape, orientation, distance and location and so on and so forth. So, when you are
simulating an event or simulating even an object, simple object, that object may have its
orientation, it has its shape. Similarly, it has a location you know close or far and this kind
of locations and so on. So, these are the finer aspects of a particular scenario, that could be
simulated that have been looked at through various experimental paradigms.
In this backdrop, the idea of an ‘immersed experiencer’ has been given by researchers.
The idea of ‘immersed experiencer’ is primarily this that the person, comprehenders
anybody who is listening to a sentence or a word or anything, who is comprehending a
linguistic input, they construct mental simulation in which they virtually place themselves
inside the described events.
For example, if you were reading a story or even imagine yourself watching a movie, any
kind any movie, the reason why we empathize with the characters in a movie is because
we almost consider ourselves transported into the scene. Similarly, for stories when we are
reading about reading a story we given the descriptions of the surroundings of the
protagonists, we almost consider ourselves as part of the scene as part of the event, so that
is what is the idea about immersed experiencer. So, when we are comprehending, we are
taking the we are considering ourselves inside the event described event, as a result of
which we can be called immersed experiencers.
So, the person the comprehender is ‘immersed’ in that experience. This immersed
experiencer view argues that ‘understanding language about a described scene is akin to
perceptually and motorically experiencing the same scene as a participant in it’. So, this is
basically what. So, the simulation and the immersion of the experiencer inside that
simulation is what the whole theory is about.
So, you are not only simulating the experience, but also simulating in such a way that the
comprehender considers himself or herself inside that event. Of course, there are various
facets to it we will see, which this is also modulated depending on the linguistic input. So,
as a result of that immersion, objects mentioned in the sentences should be, in this view,
449
are mentally simulated as having perceptual properties reflecting the viewpoint of
someone immersed in the same scene would take, reflecting for instance angle distance
and so on.
So, if the protagonist in a scene is facing the sea, let us say, so, then the reader also reader
of the story also is considering himself facing the sea. So, the same viewpoint is the reader
takes the same viewpoint as the character. So, they will reflect the same kind of angle,
same kind of distance, same kind of viewpoint, which side of the object is visible to him
or her and so on. So, for example, reading a story makes us see the surroundings to the
protagonist’s eyes as I have just said and impacts how we perceive the scene.
So, there are many experimental paradigm that have found proof for this. There are two
primary predictions of this immersed experiencer theory within the mental simulation idea.
One is, that the reader or the comprehender is taking a perspective as the same as the
protagonist in the sentence or a story or whatever the case may be. And secondly, it is that
experience is multimodal, the mental simulation is multimodal. So, it not only includes
visual, but also auditory characteristics of the objects, as we will see; visual as well as
auditory characteristics of the objects.
So, after this introduction now let us go ahead with some of the findings in this regard. So,
findings are of course, there is a large amount of literature, large number of data that have
come out from the last couple of decades or more. So, we will divide those, it is not
possible to do justice to all of the finding, but we will look at the major findings in this
regard.
So, we will talk look at visual simulation, motor simulation, auditory simulation and even
the use of personal pronouns having a an impact on the perspective of the simulation. So,
these are the primarily these are the areas. So, there are the these are the different
modalities. So, perception these two talk about perception, these talks about action and
then even at the domain of pronouns we will see the impact.
450
Visual simulation: a large number of data actually come from visual simulation,
comparatively less data we have from the domain of auditory simulation.
So, in terms of visual simulation there are many different aspects to it. If you just think
about a scene that you were looking at, you can imagine how many different facets it can
have. You can think about the orientation of an object, you can think about the shape of
an object, the distance, the color and so on and so forth.
So, there are so many types. So, we will just try to look at some of these, this some of these
finer aspects within the visual simulation. One of the most well known and quite famous
studies in this regard about implied orientation and shape of object that is implied in the
sentence structure, in terms of visual imagery is this.
Basically what this do is this study is primary focus was on trying to find out if the sentence
can modulate your simulation. So, the way a sentence is projected, the way a sentence is
created, does it have an impact on the way we actually visualize that particular object, that
was the idea. If so then we will see some interaction between these two things.
So, the study went like this. So, they had a sentence and image matching task. Sentences
were like this. There were many pairs of sentences, one was like ‘the man hammered the
nail on to the floor’ and ‘the man hammered the nail into the wall’. Similarly, there were
many other manipulations of this particular structure.
After this, an image of the manipulated object was presented, which is either compatible
or incompatible. Let me explain what is happening here. First, the subjects will read a
sentence like this ‘the man hammered the nail into the floor’, and then there will be a
picture of an image and the subjects had to decide whether that particular object was there
in the sentence or not.
So, for example, when you listen ‘the man hammered the nail on to the floor’. And then
you look at the picture of a nail, you will the answer will be ‘yes’, yes the nail was
mentioned in the sentence, but if you see the picture of a tree he will say no the picture
was not. So, this is the task, but the task over task is this.
But, the manipulation is in this case that the manipulations are the ‘yes’ trials, yes trials
are those where there is a match. There is a match, as in, that the sentence mentions that
451
particular object. Now, what was the basic hypothesis in this case is that compatible
conditions will have less reaction time than the incompatible ones. What does this mean?
Let us think of the sequence. So, first they read the sentence and then they look at the
object and then they there is a key press yes/no kind of a reaction. So, they say yes or no.
Now, compatible condition will be let us say this is the sentence and the image represented
here is that of a nail.
So, it will be an yes trial this will be a yes trial. Now, the interesting manipulation in this
case was they will have this sentence and this will come once with the nail, once with the
nail that is oriented vertically, in another case the nail will be oriented horizontally. The
hypothesis was that if the orientation of the object aligns with the way the sentence talks
about it, then they will take the participants will take less time to say ‘yes’ to that.
However, if the orientation is not compatible with the sentence, then they will take longer
to say yes. In both cases the yes trial is the target trial. And because this is an experimental
set up, there is also a equal or known number of ‘no’ trials, but the no trials are not the
target trials, the target trials are yes one is compatible condition, the other is incompatible.
Compatibility basically refers to similarity between what the sentence says and what the
object really represents. So, in case of when they are the same orientation, subjects took
less time, RT is for Reaction Time, that is how long you take to give your answer. So, the
subjects took considerably lesser time in when the object was in the same orientation as
the sentence simulated.
So, this is one of the earliest studies of language helping us, helping comprehenders
simulate a scenario. So, the when the man have when we read a sentence like the man
hammered the nail into the floor, we are automatically simulating the orientation of the
nail. Had it not been the case, it would not impact the judgment on the object, this is the
hypothesis if we if the sentence does not simulate, for example, if you if it has nothing to
do with how we look at the nail, how the nail was it should not have any relationship
whatsoever with the judgment of how the nail whether it was a nail was present or not, but
they did find a strong reaction, strong compatibility effect, which means that if they are
similar they took less time which proved the other the point that actually we are simulating
the subjects were simulating the scene in their mind, and that is how they understood.
452
So, results verify this hypothesis. Similarly, there are many other similar ones for example,
‘the eagle was in the sky’ versus ‘the eagle was in the nets nest’. In both cases the same
object the bird was the same, in both cases there were the there were this was the eagle.
But, in one case when you say that the eagle was in the sky is you already imagined the
eagle with the wings spread.
In the other case, the eagle in the nest will have the wings drawn in and then you show
them the picture of eagles in different, either with the wings open or closed and there will
be the same kind of reaction time differences were found. So, participants in each of these
sentence picture matching task were asked to just see the object.
They were not asked to do anything else just look at the object and then say whether it was
there in the sentence or not. So, even when you are not actively helping them simulate, we
are still simulating this is what the major point of this line of research is.
So, language mediates simulation, by simply changing one part of the sentence we can
simulate an entirely different scenario. And the fact that we do simulate, is reflected in this
kind of findings. So, in this case also the critical manipulations were ‘yes’ trials match and
mismatch were in terms of shape and orientation.
There are many other studies also in the same line, which all found the same kind of
answer. So, basically the first the sentence comes and then the object matching, object
which has to be matched comes. So, it is a sequential trial, it is one thing follows after
another, so the prime and then the target.
Yet another domain of research within visual simulation is the condition of visibility,
whether it is clearly visible or not clearly visible and so on. A very interesting study by
Zwaan and his group demonstrated that language comprehenders mentally simulate the
visibility conditions of a described scene.
So, it is not only the orientation or the shape of the object, but also the visibility condition,
which means even a finer detail about the object is simulated. So, for example, they use
sentences like ‘through the fogged goggles, the skier could hardly identify the moose’ and
there were many other sentences like this.
453
And they what they found was, participant responded more quickly to a blurred image than
to a high resolution image of the mentioned entity. You see, it is a very very significant
finding in the sense that the sentence says that the skier could not see because his goggles
were fogged, he could not see the moose clearly.
If the comprehender, if the reader or the hearer of the sentence, is not putting himself in
the skier’s shoes there should be no difference in him identify. In fact, the high resolution
image should be easier to identify than a than a blurred image; however, the reverse
happened, reverse happened after reading a sentence that talks about a blurred image, the
participants actually found the blurred image far more quickly. They responded far more
quickly to a blurred image, this is what is a proof of simulation.
Similarly, yet another important group and they studied that the experiment in what they
did was their experiment explored knowledge retrieval during use of words to think of an
object.
So, so far we have seen about shape, orientation as well as visibility, how clearly visible
or not clearly visible the object is. In this case they talk about, they say if a thinking of a
word makes you simulate the whole thing, because conceptual knowledge is rooted in
perceptual and action based representations.
So, one such aspect of knowledge is the spatial information, as that is something very very
important, that information is extremely important for us to help, to decide how to interact
with an object, how far it is you know how closely you can see the thing and what the other
such aspects of the scene.
So, they did a ‘inside versus outside’ perspective study and its role on object recognition.
They use sentences like this, there were three perspectives: one is inside, one is outside,
one is a mixed perspective. Inside perspective use sentences like this: ‘you are eating in a
restaurant’. So, what is happening here is you are eating in a restaurant automatically
simulates a scenario where you were inside that scene.
Outside perspective, ‘you were waiting outside the restaurant’ and then mixed perspective
they are used, ‘we are walking toward and entering a restaurant’. So, this is a mixed
perspective. What was the task? Task was to identify is the if the probe word, that is the
target word for us, is a part of the object.
454
So, is table a part of restaurant? is sign a part of restaurant? and so on. So, first the sentence
that basically creates a scene within which the subject is either inside or outside of the
scene and then they have to simply decide if the probe word is part of that
So, if the object if the way linguistically the scene is described makes it has an impact on
how you retrieve information about that particular object, then we will see an interaction.
So, by putting you inside the restaurant, does it make you look closely, does it create a
closer interaction between the subject and the objects inside that scene or does it have no
impact.
This is what they tried to find out, simply by using language sentences in different ways.
Nothing is there was no other the task given to participant was simply to say ‘ok you read
this sentence and then you see the word and you may tell us whether this object was this
word is part of that object or not’.
The result showed effect of perspective on the availability of conceptual knowledge. So,
if you have read a sentence, you were eating in a restaurant and then you were given a
word like table you were quicker to identify. However, if you are eating inside a restaurant
and the probe word is a sign, then there is a the inhibition, there is a longer time that you
take.
So, by this, they have proved they proved that the way linguistically mediated conceptual
information is simulated, it affects the way you perceive words and their connections.
Similarly, there is yet another interesting study in this regard: Horton and Rapp they look
at shift in accessibility as a result of occlusion in the narrative.
455
This is how the idea, this is what the question that they asked. So, the perceptual
availability is the question here perceptual availability hypothesis talks about if the object
is perceptually available to the protagonist, the reader will readily identify. So, this is again
taking us back to the experiencer immersed experiencer. So, the if a particular object is
perceptually available to the story the characters in the story, then the reader also will have
the same kind of understanding.
They created stories this particular experiment was carried out by using stories when the
subjects read the story line by line. So, these are the five till here the story goes like this.
So, Russ was in the hospital recovering from minor surgery. In the bed next to him was an
older man named Marty. A television was attached to the ceiling between them. At Marty’s
side was a tall vase of flowers. Till here, the story goes like this and then it has it takes two
turns, one is called the blocked version, blocked story, the other is the unblocked story.
There are two types of continuations; the story had two parts till the 5th sentence, Russ’s
friends had only given him a get well card. So, this till here it is the same story and then
there are two types one is the blocked story continuation, where a nurse came in and drew
the curtain around, a nurse came in and attached a monitor to Marty’s bed, right?
There is a problem here this is the second sentence this should be the one a nurse came in
and basically the nurse comes in and puts a curtain around, and then because she needs to
take some do some to do some to she wanted privacy, while she did an examination. In the
other scenario the unblocked scenario is, the nurse comes in and attaches a monitor to his
bed and then she needs to take blood pressure.
So, this basically there is a scene somebody is in the bed, hospital bed and somebody else
has come there. And now, because there is there are two possible scenarios; in one scenario
the nurse comes in and blocks the view of the protagonist, in another case the nurse does
not block the view.
Now, the target question was did Marty have a vase of flowers, was there a vase of flowers?
Now, if the reader of the story is viewing the scene as Marty is, then in the first case if the
scene is blocked they will not be able to the viewer will also think of a blocked scenario.
So, readers were as a result of this, the readers were slowest to respond in the blocked
456
story, because Marty cannot see if the scenario was blocked because the nurse came in and
pulled the curtain.
The readers were also slow to respond whether the vase was there, because their situation
models reflected decreased perceptual availability; because the protagonist in the story had
a decreased perceptual availability, similarly the readers also behaved the same way.
Similarly, visual distance is yet another aspect that has been that has been that has been
worked on by Bodo Winter and Bergen, Benjamin Bergen.
In this case, the task was to decide whether the object was or not mentioned in the sentence.
Again it is a ‘yes’ trial, but the manipulation was in the sentence itself. So, reading a
sentence would lead the reader to automatically perform a mental simulation of the event.
The more similar a subsequent picture is to the readers mental situation the more the
reaction will be facilitated.
So, these are the kind of sentences that they had. ‘A frisbee in your hand; versus ‘a frisbee
in the sky’, in the first case frisbee is closer. So, the distance is being manipulated here.
So, in this case it is closer, in this case it is far. Similarly, you were looking at the milk
bottle across the supermarket or this is a far condition or you are looking at the milk bottle
in the fridge which is the near condition.
So, they manipulated various sentence pairs that basically looked at the same object, but
in two different distance condition far and near. So, basically pictures showed objects in
far and near condition. The situation sentences created either far or near conditions and
then the pictures came in, they also had a far and near condition, which was basically
manipulated by redoing their sizes.
Now, one important thing to be remembered here is that they the researchers in this case
looked at objects that are token invariant. Token invariant basically refers to objects in the
real world that display relatively little variation in sizes across exemplars, basically those
objects that do not typically differ in sizes in the real world.
So, a cricket bat remains the same shape and size wherever you go in the world. A football
the ball used in the football game remains the same size. So, basically if you decrease the
size of a football, then it creates the understanding of a distance. So, you were looking at
457
the football from a distance versus you were looking at it from a close up then it is near to
you that is the were the pictures were manipulated.
Again, but here what happened was mismatch condition had higher response latency as
well, but in the match condition you have a lower response latency when there was a match
like we have seen before.
So, if you have a if you read a sentence that talks about far distance, like you have ‘you
see the milk bottle across the supermarket’ and then you see an object the same object that
the bottle of a milk in a in small size then you recognize it much more quickly compared
to if it is far if it is near to you.
So, if there is a match the time is less. So, this shows when reading sentences about distant
objects, comprehender simulate smaller objects. So, when you think of distance you think
of the objects in a smaller size. So, this is also in terms in keeping with the earlier findings.
Now, let us move on to auditory simulation. So, we not only simulate the visual aspects of
a scene that the sentence talks about, but also the auditory information, how is that?
Experimentally, speaking how is that carried out in case of both simulation of motion as
well as other spatial features, like distance. In case of auditory simulation these are the two
domains that have been studied primarily.
So, in one study while the study was basically reading and making sensibility judgments
whether the sentences are meaningful or not. So, the participant listened to an auditory
stimuli, conveying motion towards, up away and four kinds of motions were integrated in
the sentence.
And then the auditory stimuli were bands of white noise manipulated to create the
impression of motion. There were two things that are happening simultaneously. One is
the participant listened to the white noise, there was through their earphone, through their
headphone, they listen to a noise that the noise was modulated, it was doctored in order to
make it sound make it sound like there were four different motions that were depicted.
One is the noise is going away from the participant, noise is coming closer basically by
manipulating the amplitude. Similarly, they also manipulated it to make it sound like the
sound is going upward or the sound is going downward. So, basically manipulating the
458
sound , auditory input through the headphone, which simulates which creates an imagery
of four kinds of different motions. Simultaneously, there were sentences that conveyed
motion in four directions, all similar same four directions.
They had to do a sensibility judgment task ,simply read the sentence and say whether the
sentence is meaningful or not. In this case, participants were faster in the mismatch
condition.
This is a very interesting finding, in this case we do not see a facilitation of the match
condition. So, when they are hearing a noise that is coming closer to them and they are
also reading a sentence that mentions something moving closer to the subject, to the agent,
then there is a there is a problem, then there is a higher reaction time as opposed to the
mismatch condition.
In the mismatch condition, they hear a sound going away from them, but the sentence
depicts a movement towards them, this is the mismatch condition. In this mismatch
condition they were much faster in reacting. Now, the how the researchers have explained
for this, explanation for this finding is that conflict arises when auditory perception system
is required to process two stimuli at the same time.
What is happening in this case? Till now, we saw there was a there was a prime a sentence
will be displayed first and then after that they have to do a judgment task. So, in that case
we see a facilitation, that if you have already simulated a ‘near’ object and then you see an
object that appears to be near even quicker to react.
However, in this case, both the stimuli were given simultaneously. So, you are making the
perceptual system busy with two different tasks at the same time. Now, when that happens
there is a conflict and that conflict is seen in the reaction time. So, the sound of the motion
described in the sentence and the sound of the auditory motion stimulus are in conflict with
each other, because they both are using the same perceptual system.
Yet another study of by Winter and Bergen 2014 talked about this is again a sentence-
sound matching task. What they did was participants read sentences and subsequently
heard sounds of objects or animals that were either mentioned in the sentence or not,
similar kind of design. So, the participant’s task was to verify whether the sound they heard
was of an entity in the mentioned in the sentence.
459
Just like the picture and sentence matching task before. So, you listen to the you read the
sentences and then hear a sound and then say then you have to say whether the sound was
present in the sentence or not. Manipulations, again the same kind of manipulation
sentence mention two kinds of distances, sound volume basically refers to the again the
distance. If it is loud it is nearer, if it is not loud it is very far.
So, sentences were like this, this is just an example there were many such sets. So,
‘someone fires a handgun in the distance’. So, this is a ‘far’ condition. ‘Right next to you
someone fires a handgun’. So, these two kinds of sentences is creating two different is
simulating two different kinds of perception of the sound.
If somebody fires a handgun right next to you, the sound will be much higher much louder
as opposed to when it is far away. What is the result showing? Matched condition yields
significantly shorter response latency. Again in this case, one thing follows another the
sentence follow sentence precedes the sound.
So, when they listen to a sentence ‘right next to you somebody fires a handgun’ and then
you hear a very loud noise of a handgun you say yes this was there, but if you have the
reaction time is much shorter, but if you hear a very faint kind of sound of the handgun
you take longer, this is what the finding says.
So, facilitation effect was seen. This study points out that simulation is multimodal. So,
not only it is visible in terms of simulation is a impactful only when in the visual domain,
but also in the auditory domain.
So, it is basically multimodal. So, simulation is not only a present not only is simulation
proven through these experiments, but also that the simulation is multimodal
So, these are the hypotheses that we started with. So much so, that even the use of personal
pronouns can modulate the way simulation really works. So, several studies have found
that personal pronoun such as ‘you’ or ‘she’ can modulate the perspective of a mental
simulation. In one such study readers mentally embody an actor’s perspective,
alternatively readers might also mentally simulate the events from an onlooker perspective.
So, what perspective you take? See till now we were looking at the perspective taken by
the protagonist being the same as the perspective taken by the reader. This particular study
460
actually manipulated that by manipulating the pronoun system as used in the story. So, in
this case they in the use two experiments examine the role of pronouns by manipulating
that.
In one case, ‘you’ is or ‘I’ is used in another case he is used. So, when third person pronoun
is used the perspective taken is that of an onlooker, but when you use a first person or a
second person pronoun, then the perspective taken is that of the protagonist. So, even a
simple change of pronouns shows that language mediated simulation actually is activated
when you comprehend language.
So, language actually the structure of language the sentences, in this case they mediate the
simulation, mental simulation of the event not only in terms of visual, auditory, but also in
terms of the perspective taken by simply manipulating the pronominal system.
And then comes the motor simulation, so visual simulation auditory and so on. Now, we
come to the motor simulation and in fact, in simulation literature, a large number of data
actually comes from motor simulation and very interesting findings too. So, in a series of
sensible in this particular study, series of sensible and nonsense sentences were used and
the participants were asked to determine, as quickly as possible, whether each sentence
made sense this was the task.
So, they read sentences and then the task was to read the sentence and just say whether it
is meaningful or not whether it is sensible or not; obviously, that is the task there are two
kinds of sentences used meaningful and meaningless. What are the sentences? But the
sentences that were used were manipulated in terms of movement.
One is ‘open the drawer’, ‘put your finger under your nose’ and so on. In this case the
implication is that there is an action towards the body, when you open the drawer and open
a drawer there is a movement of the hand towards your body towards the body of the agent
the speaker.
Put your finger under your nose and many such other possibilities that imply that simulate
a scenario where the movement is closer to the body.
461
In other sentences which they call ‘away sentences’, ‘close the drawer. So, when you
close the drawer, your hand moves away from the body, ‘put your finger under the faucet’
and so on.
Similarly, they also had these meaningless sentences because the task was to judge the
sensibility of the sentence. So, the meaningless sentences were like boil the air and so on.
The manipulation here was the button press which was very cleverly manipulated. So,
once first they had the sentences and then they had a button box. Button box had various
different buttons, but the manipulation was that in certain cases the critical is in this case
also the critical is the ‘yes’ trials. So, the meaningful sentences are the critical trials not
the meaningless sentences this is just a filler. So, meaningless sentences were a filler, they
are distracters.
But, the critical trial were the meaningful sentences. Now, the test was to see if you read a
sentence that implies a movement toward the body and then you have to, subsequently,
perform an action that is either similar to the sentence or opposite to the opposite to the
movement that is mentioned in the sentence, is there an interaction.
So, if a sentence says that you close the drawer which means the sentence is implying a
movement away from the body and then you have to perform a task for which you have to
move your hand away from the body also, then there is a compatibility these are similar
kind of actions.
So, you are already simulating an away movement and then you are carrying out an away
movement. So, then there should be a compatibility. So, this is how cleverly the response
button was created. So, the buttons that you in they were manipulated in two ways. In one
case in some of the cases the ‘yes’ button was near to the participant, in some cases the
yes button was.
It was they used a button box and in let us just imagine something like this and there was
in one in some cases the yes button was this, in some cases the yes button was this. And
the middle button they had used for moving the sentences. So, they started the experiment
by moving the middle button.
462
The result showed the action-sentence compatibility effect. Meaning, if they say if they
read a sentence that talked about and movement away from the body and they had to press
the far yes is far button, then there was compatibility, meaning they took lesser time. This
was a facilitation.
So, in now we have seen a lot of data and lot of studies both in terms of visual as well as
auditory as well as motor movements that are that can be simulated through language.
Now, we have seen that there are two possible kinds of impact. Simulation is there, but
does simulate simulation mental simulation facilitate the effect, facilitate then subsequent
action or does it inhibit the subsequent action.
So, there are two possible effects of that simulation. So, the studies represented so far show
a dynamic relationship, as you as the sentence unfolds the simulation also unfolds. So,
between depicted aspects in the linguistic input and the reaction of the participants, this is
dynamic. Often, we see a facilitation effect and we describe the findings in the light of
activation of the same networks.
However, there are also findings that show a reverse pattern. So, often we see facilitation,
but also sometimes we see a reverse pattern. These two different kinds of findings can be
understood in terms of these two effects, one is called the compatibility effect, the other is
called the interference effect.
Two things happen in case of simulation and the reaction of the participants, the way they
interact it is dynamic. And this can have two different types, two different manifestations,
one is the compatibility effect, the other is the interference effect.
So, compatibility effect is that when that we have already seen a sentence depicts a
scenario and then something similar has to be processed, we see a compatibility effect.
So, in order to perform a motor action, one must activate neural motor structures
responsible for that type of action. If understanding a sentence leads to increased or
decreased activation of the same neural structures, then the this should result in quicker
compatibility actions. Performing an action as well as perceiving images depicting actions
should be facilitated by either facilitated or inhibited by language.
463
So, action-sentence compatibility effect we as you have just seen the opening the drawer
versus the closing the drawer, there are many other such studies as well. So, action
sentence compatibility effect or it is also called ACE. This effect shows the extent to which
motor representations are activated for language understanding.
The main idea behind this is that, if language understanders perform motor imagery, using
neural structure dedicated to motor control, then understanding sentences would actually
facilitate, that we have already seen.
So, this is the explanation for that why do we see the compatibility effect? Because reading
a sentence, understanding a sentence has already activated the motor perceptual domains
in the neural networks in the brain that is already active. So, subsequently if we have to
perform an action it shows compatibility effect.
So, similar studies where there in another study. So, there is sentences are like this ‘Andy
handed you the pizza’, ‘you handed Andy the pizza’. So, in two cases the movement of
the hand with respect to the body is opposite. Similarly, they have they just like the drawer
example, we have also they had equal number of meaningless , similar kind of study
because it was a sensibility judgment and we see compatibility effect just like the previous
study. So, what is basically happening is that understanding language depicting an imagery
or action, activates the neural structure
The opposite also happens, when does it happen? This is how opposite of inter
compatibility effect works out. This is, however, this is closely related to the compatibility
effect, this is not something different, it is not a separate function at all.
This is the same part of the same process, but the crucial difference is that, this is the
crucial difference. The same neural structures are required to do various tasks at the same
time, when we have that prime sentence first and then the target object follows, there is a
facilitation.
However, if you are making the same neural structure busy, at the same time for doing two
different things, then you see interference. Either way it proves the fact that you are
simulating, perceivers or comprehenders simulate the scenario and subsequent as a result
of which, the neural network is active at that time doing processing exactly that.
464
So, like understanding language and performing the perceptual or motor task require the
same neural structures to perform different task at the same time. So, when do we see
interference effect? When you give the same task to somebody who is do already doing
something else. So, this is simple if you were eating and talking, often people choke, same
kind of things we see.
So, if you are making the neural network do the same at the same time, we are making
them do two different things we will see interference.
Interference effect again it has been seen in various domains. One of the most interesting
is the domain of visual interference effect. The study of visual interference actually goes
back a long way, Perky effect was seen in 1910 probably. So, what is Perky effect?
Perky effect shows that, in her experiment she showed that visual imagery affects visual
perception. So, while you were imaging imagining something and there is simultaneously
the object present in the screen, you will think you are still imagining this is what Perky’s
finding was.
So, the experiment was like this: the participants subjects were asked to imagine
something, while staring at the blank screen and the blank screen that was presented in
front of the participants. So, they were told to imagine a leaf or a banana or something.
As the participant is busy imagining looking at the blank screen, the object that they have
been asked to visualize, is actually projected on the screen, gradually, first under the
threshold of perception and gradually it gets higher resolution and becomes more and more
clear. However, the interesting aspect of this is that the participant thinks even when the
object is clearly visible in the screen, he or she thinks that he or she is still imagining the
object.
So, this is what the famous ‘Perky effect’ is visual imagery can affect visual perception.
This is the same paradiagm that has been utilized in many subsequent research
experiments, many permutations were later on carried out. So, these findings show that
perceptual system, in particular the visual system consciously engaged in process of, in
engaged in the process of natural language processing.
465
In this regard there is a rather famous study by Richardson et al in 2001. What they showed
was language processing language about concrete or abstract motion along different
trajectories in the visual field. Now, we can talk about visual imagery that a sentence
depicts.
So, ‘the Frisbee is in the sky’, frisbee in the hand and so on, but you can also depict notion
both in terms of a concrete sentence as well as in terms of an abstract sentence you can
actually depict kind of a motion.
For example, sentences like ‘the poacher hunts the deer’ or ‘the ship sinks in the ocean’.
Sentences like these simulate a scenario that has an orientation that has a movement across
either on the horizontal plane or on the vertical plane.
So, what they do is, that the sentences that create abstract motions along different
trajectories in the visual field. Sentences were given some subjects heard sentences whose
content had implied visual characteristics. So, they hear these sentences, while you hear
this sentence the simulation will help you imagine. So, there is a visual imagery already
accompanying the understanding of the sentence and immediately they had to do a task.
The task was like this. So, first they hear the sentence and immediately after that they had
to do a judgment task that whether the object on the screen is a square or a circle.
This was the simple task, there is no connection whatsoever between the sentence or on
with the image on the surface, but the manipulation here is very crucial. So, in one case
the poacher hunts the deer this implies the horizontal arrows move like this, arrows do not
usually do not typically, arrows do not go in the vertical scenario, vertical orientation.
Now, if you have if you have heard this sentence and then subsequently you have to do a
judgment task, you have to do a identification task, whether it is a circle or a square. Now,
the crucial manipulation was that if you have heard the sentence ‘a poacher hunting a deer’
and then these two objects appear on the horizontal plane they will take more time to for
the judgment.
If you have this kind of orientation after the sentence then time taken will be less and this
is exactly what they found. So, in a sentence like ‘offended’ even abstract notions like
offense offended, ‘lifted’ so on and so forth. Also simulate kind of a movement either on
466
the horizontal or on the vertical plane. And then subsequently, immediately after that, if
they are required to utilize the same visual field for a different task then there is an
interference effect that is seen. Similar kind of interference we also see in terms of motor
activities, motor interference effect
This is in case of cross model methodology, this is a rather well known very famous study
by Benjamin Bergen.
And he has done many versions of this; this is a image-verb matching pair as similar. So,
there are different kinds of possibilities. So, there are stick figures, doing various things
three kinds of and they call it affector. They use different kinds of affector: hands, legs and
mouth and the there are pairs, there is one image and then there is an a word, there is a
verb.
The task was to say whether there is a match. So, this is a match scenario this is a match
scenario for example, the character is hopping. So, if you see a character hopping and then
you see the word hop it will be an yes answer. But, the critical answers were actually the
‘no’ answers the mismatch was the main target here.
So, they had two different kinds of mismatch condition, one is the same affector is being
used, but they do a different task. So, in this picture is showing hopping, but the word says
‘punt’. There are two different things that the affector same affector is doing.
Similarly in this picture the character is scratching his hands, but if you give the word a
‘juggle’ then there is a mismatch. What they found was, that subjects took longer in cases
of mismatch when they shared the affector, rather than when they did not. So, for example,
both this case and this case are mismatch conditions; however, similarly this.
So, this is a mismatch because the person is screaming this is also a mismatch. So, you see
the picture of the stick figure screaming and the word appearing after that is ‘kick’ there
is absolutely no problem. The subjects took considerably less time to say that they do not
match; however, most critical was this, this particular pair.
This particular pair, because in this case the affector in this case the mouth, in this case the
hand, in this case the legs the affector remains the same. So, that in the image it is doing
one thing, in the word it is doing another thing.
467
There is a conflict, there is a tussle and as a result of which we see an interference effect
in terms of longer reaction time. So, all these studies, whether it is compatibility effect or
it is interference effect, we see that sentences or words basically simulate we simulate a
scenario.
And on the basis of that the simulation, automatically leads to a neural activation and then
depending on what kind of task has been given. If it is task is subsequent to the prime, then
we see a facilitation and if it is simultaneously happening or happening in quick succession
then we see an interference effect.
So, either way, we understand that comprehenders simulate the object or event presented
in the sentences and evidence is both from compatibility and interference effect are
evidence of the same hypothesis. So, this language driven mental simulation is
multimodal, as seen through visual and auditory modalities.
So, this is about simulation and its relationship with the with language, language
comprehension.
So, this is the last part of the course. So, let us go through a quick summary of what we
have done so far from the beginning. So, module 1 we talked about the background, the
theories of cognition and cognitive exploration into the nature of language and knowledge.
2nd module talked about conceptualization as a process and within that process we looked
at the relationship between language and the mental functions, the expressions.
Module 4 looked at again looked at the embodiment hypothesis in terms of image schema
and how it interrelates with the language structure and language understanding.
Module 5, we looked at language acquisition, language learning and how this process also
like the previous parts, this aspect of language also has relationship with various non
linguistic factors. So, primarily the social cognition and other such modalities.
468
Module 6 talked exclusively about the brain relationship of language with the brain and
language functions.
Module 7 looked at language and attention in terms of how the two processes drive. So,
this is a this is a two-way traffic how attentional mechanisms and language has a dynamic
relationship during processing.
And last, the module 8 has executive functions. We looked at executive functions through
various language systems. We also talked about various critical debates, and today we
talked about simulation in terms of language comprehension, how simulation is an integral
part of., so, basically when we talked about language having relationship with brain, with
social cognition, with attention and executive control, all these things ultimately basically
lead to what we call simulation. So, simulation basically means recreating the entire
experience and you see this recreating of experience has such an important role to
understand language.
And thereby we can conclude, safely, that language is a process that can be fully
understood if and only if we take into account the background cognition. So, all these
processes refer to the, what is famously called, the ‘background cognition’ of language.
So, we need to understand language with respect to or at the same time looking at the
background cognition.
Thank you very much for your attention, this is where we close the course.
469
THIS BOOK
IS NOT FOR
SALE
NOR COMMERCIAL USE