Introducing Multimodality

This accessible introduction to multimodality illuminates the potential of multimodal

research for understanding the ways in which people communicate.
Readers will become familiar with the key concepts and methods in various domains
while learning how to engage critically with the notion of multimodality. Now fully
revised to engage with new research, include new case studies and present a more global
outlook, the book challenges widely held assumptions about language and presents the
practical steps involved in setting up a multimodal study, including:

■ formulating research questions

■ collecting research materials
■ assessing and developing methods of transcription
■ considering the ethical dimensions of multimodal research

With a wide range of examples, clear practical support and a glossary of terms,
Introducing Multimodality is the ideal reference for undergraduate and postgraduate
students in multimodality, semiotics, applied linguistics and media and communication
studies. Online materials including an updated study guide, exercises and links to relevant
resources are available on the Student and Instructor website at [Link]/cw/
jewitt and the Routledge Language and Communication Portal.

Carey Jewitt is Professor of Technology and Learning in the Department of Culture,

Communication and Media at the Faculty of Education and Society, University College
London. She is founding co-editor of the journal Multimodality & Society and has
authored/co-authored and edited several books and journal articles in this area, including
The Routledge Handbook of Multimodal Analysis and most recently Digital Touch.

Jeff Bezemer is Professor of Communication and Head of the Department of Culture,

Communication and Media at the Institute of Education, University College London.
He regularly publishes on multimodal interaction in healthcare, and is the co-author of
Multimodality, Learning and Communication.

Kay O’Halloran is Chair Professor and Head of the Department of Communication and
Media in the School of the Arts at the University of Liverpool. She is the series editor
of Routledge Studies in Multimodality and is co-editor of Multimodal Studies: Exploring
Issues and Domain.
Introducing Multimodality
Second Edition

Carey Jewitt, Jeff Bezemer and Kay O’Halloran

Designed cover image: Getty Images | Wisaad
Second edition published 2025
by Routledge
4 Park Square, Milton Park, Abingdon, Oxon, OX14 4RN
and by Routledge
605 Third Avenue, New York, NY 10158
Routledge is an imprint of the Taylor & Francis Group, an informa business
© 2025 Carey Jewitt, Jeff Bezemer and Kay O’Halloran
The right of Carey Jewitt, Jeff Bezemer and Kay O’Halloran to be identified as authors
of this work has been asserted in accordance with sections 77 and 78 of the Copyright,
Designs and Patents Act 1988.
All rights reserved. No part of this book may be reprinted or reproduced or utilised in any
form or by any electronic, mechanical, or other means, now known or hereafter invented,
including photocopying and recording, or in any information storage or retrieval system,
without permission in writing from the publishers.
Trademark notice: Product or corporate names may be trademarks or registered trademarks, and
are used only for identification and explanation without intent to infringe.
First edition published by Routledge 2016
British Library Cataloguing-in-Publication Data
A catalogue record for this book is available from the British Library
Library of Congress Cataloging-in-Publication Data
Names: Jewitt, Carey, author. | Bezemer, Josephus Johannes, 1976- author. |
O’Halloran, Kay L., author.
Title: Introducing multimodality / Carey Jewitt, Jeff Bezemer and Kay
O’Halloran.
Description: Second edition. | Abingdon, Oxon ; New York, NY : Routledge,
2025. | Includes bibliographical references and index. | Identifiers:
LCCN 2024046169 (print) | LCCN 2024046170 (ebook) |
ISBN 9781032845388 (paperback) | ISBN 9781032845357 (hardback) |
ISBN 9781003513698 (ebook)
Subjects: LCSH: Modality (Linguistics) | Cognitive grammar. | Discourse
analysis. | Psycholinguistics.
Classification: LCC P299.M6 B49 2025 (print) | LCC P299.M6 (ebook) |
DDC 415--dc23/eng/20241230
LC record available at [Link]
LC ebook record available at [Link]

ISBN: 978-1-032-84535-7 (hbk)

ISBN: 978-1-032-84538-8 (pbk)
ISBN: 978-1-003-51369-8 (ebk)
DOI: 10.4324/9781003513698
Typeset in Frutiger
by KnowledgeWorks Global Ltd.
Access the Instructor and Student Resources: [Link]/cw/jewitt
Contents

List of tables vi
List of figures vii
About this book viii
Acknowledgementsx
1 Navigating a diverse field 1
2 Why engage with multimodality? 11
3 Systemic functional linguistics 23
4 Social semiotics 45
5 Conversation analysis 70
6 More approaches to multimodality 88
7 Designing a multimodal study 108
Glossary  129
Self-study guide  134
References166
Index179
Tables

1.1 Mapping three approaches to multimodality: SFL, social semiotics

and conversation analysis 8
3.1 SF-MDA text and image systems 34
7.1 Summary of the aims and theory of meaning of multimodal approaches 109
7.2 Examples of research questions in multimodal approaches 111
7.3 Summary of the empirical focus of multimodal approaches 113
7.4 Summary of the methodology of multimodal approaches 114
Figures

2.1 Impression of a job interview. 21

3.1 SF-MDA approach to the WHO Ebola website (O’Halloran et al., 2019). 35
3.2 Reporting section of the WHO Ebola website. 37
3.3 Reporting section of the WHO Ebola home page. 39
3.4a Screenshot of sample analysis in Multimodal Analysis Video (O’Halloran
et al., 2017). 41
3.4b Screenshot of State Machine in Multimodal Analysis Video (O’Halloran
et al., 2017). 42
4.1 Beanie baby toys in hospital, Singapore, submitted October 2021. Play
Observatory (2024), PL175A1/S001/p1, [Link]
data.21198142, CC BY-NC-ND 4.0. 51
4.2a Reproduction of an image from Salters GCSE Science: Y11 student book,
Heinemann Educational Publishers. 61
4.2b The digestive system: columns reversed (reprinted from Gunther Kress,
Multimodality (2010), p. 90, with permission from author). 62
4.2c The digestive system: rearranged (reprinted from Gunther Kress,
Multimodality (2010), p. 90, with permission from author). 62
5.1 Surgeons cutting a suture. 74
5.2 Sample transcript from Gail Jefferson’s ‘Nixon’ tapes study
(Jefferson, 2004). 79
5.3 Transcript of an exchange between a doctor and a patient (from Heath,
Hindmarsh & Luff, 2010). 80
5.4 Transcript of interaction between a surgical trainee and his supervisor. 85
6.1 Simultaneous higher-level actions (reprinted from Sigrid Norris, Analyzing
Multimodal Interaction (2004), p. 102). 95
7.1 Multimodal transcript of the teacher’s framing of the task (page 146).
Reprinted from Mavers, D. (2009). Student Text-Making as Semiotic
Work. Journal of Early Childhood Literacy, 9 (2), 145–155. Reprinted
with permission from SAGE Publications and the author. 123
About this book

This book is about multimodality. Taking stock of three decades of theorizing and research-
ing, we sketch the history and current state of affairs in a fast growing academic field. Our
focus is on three distinct traditions in which multimodality has been taken up: systemic
functional linguistics, social semiotics and conversation analysis. We outline the history,
key concepts and methods of each; point to seminal studies and critiques; and reflect on
their synergies and differences. We also identify ways in which these focal approaches
have been transformed through amalgamation with or incorporation into other research
traditions, and future directions for multimodality. Throughout the book, we illustrate
concepts and methods with examples from a variety of different studies, reflecting the
breadth of domains that have now been explored from a multimodal perspective.
The book is written for anyone who is interested in language, communication and
meaning, including undergraduate and postgraduate students in applied and sociolin-
guistics. No prior specialist knowledge in linguistics or semiotics is required or assumed.
The aim of the book is to help you to:

■■ Engage critically with the notion of multimodality, with reference to seminal studies
on multimodality
■■ Recognize similarities and differences in theoretical and methodological positions
■■ Identify and describe how multimodality has been taken up in systemic functional
linguistics, social semiotics and conversation analysis
■■ Assess the quality of multimodal research, notably aptness of fit between research
questions, theories and methods
■■ Reflect on the potentialities, challenges and future directions of multimodal research
■■ Understand how to go about designing a multimodal study

The book consists of seven chapters.

In Chapter 1, ‘Navigating a diverse field’, we discuss the history of the notion of
multimodality and formulate key premises that can be used to differentiate between dif-
ferent types of multimodal studies and consider one’s own position in the diverse field of
multimodality.
In Chapter 2, ‘Why engage with multimodality?’, we provide a rationale for doing
multimodal research. We challenge widely held assumptions about language and review
the arguments for taking a multimodal approach.
In Chapters 3 to 5, we set out three distinct approaches to multimodality.
Chapter 3 introduces systemic functional linguistics, Chapter 4 introduces social semiotics,
 About this book ix

and Chapter 5 conversation analysis. In each of these chapters, we outline the key prin-
ciples, concepts and methods of the respective approaches. We also illustrate their areas
of application, as well as their limitations.
In Chapter 6, ‘More approaches to multimodality’, we explore the ways in which the
three focal approaches discussed in Chapters 3 to 5 have been taken forward by combin-
ing them with other traditions. Thus we present four more approaches to multimodal-
ity: geo-semiotics; multimodal (inter)actional analysis; multimodal ethnography; and a
corpus-based approach to multimodality. We introduce each of these approaches with
attention to their origins, methods used and their distinct contribution to multimodality.
We also discuss new directions for multimodality through its intersection with translan-
guaging and sensory studies.
In Chapter 7, ‘Designing a multimodal study’, we present the steps involved in setting
up a multimodal study. This includes deciding on a theoretical frame for your study, for-
mulating research focus and questions, collecting research materials, assessing methods
of transcription and considering the ethical dimensions of multimodal research.
In addition to these seven chapters, the book includes a self-study guide, designed
as an optional stand-alone resource or basis for a short course. You can use this resource
as a guide to reading the book and engaging with the issues raised by it. It consists of
seven comprehensive study units: one for each chapter. Each unit provides the following
set of resources: a chapter overview, with chapter topics and a summary; a set of study
questions; a series of ‘hands-on’ exercises; tips; and suggested resources, including read-
ings and online resources. If you consider using the self-study guide, we suggest you first
read the book, and then work through the self-study guide units in the order that it is
presented.
Acknowledgements

We would like to acknowledge the following copyright holders for permission to reprint
the following material:

■■ The World Health Organization for the reproduction of three images from the WHO
Ebola website in Figures 3.1b, 3.2a and 3.3a. Available from [Link]
health-topics/ebola#tab=tab_1 [cited 19 June 2015]. The WHO website operated
during a certain period of Ebola outbreak, but it is no longer operational. For up-to-
date WHO information on Ebola, please refer to the following website: [Link]
[Link]/csr/disease/ebola/
■■ The Play Observatory Project for the reproduction of the image in Figure 4.1 https://
[Link]/
■■ Heinemann Educational Publishers for the reproduction of an image from Salters
GCSE Science: Y11 student book in Figure 4.2a.
■■ Gunther Kress for the reproduction of his reorganized images of the digestive system
in Figures 4.2b and 4.2c. Reprinted from Gunther Kress (2010).
■■ Christian Heath and colleagues for the reproduction of a transcript in Figure 5.3.
■■ Sigrid Norris for the reproduction of a transcript in Figure 6.1.
■■ Diane Mavers and SAGE Publications for the transcript in Figure 7.1.

Every effort has been made to contact copyright holders. Please advise the publisher of
any errors or omissions, and these will be corrected in subsequent editions.
CHAPTER 1

Navigating a diverse field

WHAT IS MULTIMODALITY?

‘Multimodality’ is a term that is widely used in the academic world. The number of pub-
lication titles featuring the term has grown exponentially since it was first coined in the
mid-1990s. Since then, a myriad of conferences, monographs, edited volumes and other
academic discussion forums have been produced that are dedicated to multimodality.
Book series, journals and textbooks defining their focus in terms of multimodality are
further signs of a maturing field. These and many other outlets inviting contributions in
the area of multimodality provide platforms for scholars working in different disciplines,
including semiotics, linguistics, media studies, new literacy studies, education, sociology
and psychology, addressing a wide range of different research questions.
With the term being used so frequently and widely, it may seem as though a shared
phenomenon of interest has been recognized and a common object of study identified.
Indeed, we can, in relatively generic terms, describe that phenomenon, or object of inter-
est, as something like, ‘We make meaning in a variety of ways,’ or, ‘We communicate in a
variety of ways.’ Yet we must immediately add that ‘multimodality’ (and related concepts,
including ‘mode’/’modality,’ ‘[semiotic] resource’) is differently construed. Exactly how
the concept is articulated and ‘operationalized’ varies widely, both across and within the
different disciplines and research traditions in which the term is now commonly used.
Therefore, it is very difficult and potentially problematic to talk about multimodality with-
out making explicit one’s theoretical and methodological stance.
Before going any further, we turn to those who first used the term and explore
what it was that they were trying to draw attention to. As far as we can reconstruct, the
term first appeared in the 1990s in different parts of the world. It is used, for instance,
in what were to become seminal articles by Gunther Kress (1993) and Charles Goodwin
(2000). These scholars started using the term more or less independently of each other,
with Kress (then) in the UK in the tradition of social semiotics, focusing on visual media,
and Goodwin in the US working in the tradition of ethnomethodology and conversa-
tion analysis, focusing on action and embodiment. Around this same time, O’Halloran,
working (then) in Australia and drawing on earlier work by O’Toole (1994) and Kress and
van Leeuwen (1996), began to use the term ‘multisemiotic’ to describe the multimodal
character of mathematics texts (see, for instance, O’Halloran, 1999b).
If a ‘means for making meaning’ is a ‘modality,’ or ‘mode,’ as it is usually called, then we
might say that the term ‘multimodality’ was used to highlight that people use multiple means
of meaning making. But that formulation alone does not accurately describe the conceptual
shift these scholars were trying to mark and promote. After all, disciplines such as linguistics,

DOI: 10.4324/9781003513698-1
2 INTRODUCING MULTIMODALITY

semiotics and sociology have studied different forms of meaning making since well before
the term ‘multimodality’ was introduced. Indeed, Ferdinand de Saussure (1857–1913), writ-
ing in the early 20th century, already suggested that ‘linguistics’ was a ‘branch’ of a more
general science he called semiology. For a long time, the branches of that imaginary science
continued to specialize in the study of one or a small set of means for making meaning:
linguistics on speech and writing, psychologists on gesture and facial expression, semiotics
on image and film, musicology on music. Subdisciplines emerged, highlighting an interest in
visual phenomena, such as visual sociology and visual anthropology. New fields were defined
dedicated to the study of new multimodal forms, such as games and comic books.
Multimodality questions a strict ‘division of labour’ along the lines of modes, on the
grounds that in the world we’re trying to account for, different means of meaning making
are not separated but almost always appear together: image with writing, speech with
gesture, maths symbolism with writing and so forth. It is that recognition of the need
for studying how different kinds of meaning-making are combined into an integrated,
multimodal whole that scholars attempted to highlight when they started using the term
‘multimodality.’ It was a recognition of the need to move beyond the established empirical
boundaries of linguistics and other disciplines and develop theories and methods that can
account for the ways in which we combine gesture, inscription, speech and other means
of communication. This fact only became more noticeable with the introduction of digital
technologies, which enable people to combine means of making meaning that were more
difficult or impossible to produce and disseminate before—for the majority of people any-
way (the moving image being one pertinent example). So that is how the introduction of
the notion of multimodality marks a significant turn in theorizing and analysing meaning.
What the early adopters of the term recognized was not only the need to look at the co-
occurrence and interplay of different means of making meaning but also that each ‘mode’
offers distinct possibilities and constraints. It had often been argued (e.g. by Saussure and
Vygotsky) that language has, ultimately, the highest ‘reach,’ that it can serve the widest
range of communicative functions or that it enables the highest, most complex forms of
thinking and is therefore the ‘most important.’ Others, including Goodwin, Kress, van Leeu-
wen and others who first introduced the notion of multimodality, have pointed out that all
resources that are being used contribute to the meanings being made, albeit in different
ways, given their material, structural and historical differences. The same point was made
by O’Halloran, who in her definition of ‘multisemiotic’ emphasized the significance of the
combination of different resources, each with their own potential. Thus multimodality marks
a departure from the traditional opposition of ‘verbal’ and ‘non-verbal’ communication,
which gives primacy to the verbal and does not separate between ‘other’ means of making
meaning. It also challenges the suggestion that the verbal carries the essence of meaning,
and that the non-verbal is not essential for (understanding or studying) communication.
We can now formulate three key premises of multimodality:

1 People combine semiotic resources to make meaning.

2 Each semiotic resource makes meaning in distinct ways.
3 If we want to study meaning, we need to attend to all semiotic resources being
used to make a complete whole.
 Navigating a diverse field 3

We should add four important footnotes to this. First, not everyone working in multimo-
dality uses the notion of meaning-making. Depending on their disciplinary background
and focus, they might say that they are interested in ‘multimodal communication,’ ‘mul-
timodal discourse’ or ‘multimodal interaction.’ We will use the term ‘meaning-making’
unless we are writing about a specific approach to multimodality. Nor does everyone
working in multimodality use the term ‘mode’: some prefer to talk about ‘resource,’
or ‘semiotic resource,’ and generally avoid drawing strong boundaries between differ-
ent resources, highlighting instead the significance of the multimodal whole (‘Gestalt’).
Indeed, for that very reason, some scholars whose work we subsume under the heading
of ‘multimodality’ do not use that term themselves, while otherwise committing to the
three key premises we just presented.
Second, scholarly interest in the connections between different means of making
meaning predates the notion of multimodality. For instance, the study of gesture and
its relation to speech, gaze and the built environment has a long history in linguistic
anthropology, interactional sociology and other disciplines (see, e.g. Goffman, 1981;
Kendon, 2004a; Mehan, 1980); the relation between image and writing has been
studied in semiotics (e.g. Barthes, 1977 [1964]) and so on. These early contributions
have produced important insights into what we now call multimodality. At the same
time, we should note that the potential empirical scope of multimodality goes further
still. We can see a development from an exclusive interest in language to an interest
in language and its relations to other means of making meaning, to an interest in
making meaning more generally, without a clear base point, whether language or any
other mode.
Third, while those using the term ‘multimodality’ generally aim to develop a frame-
work that accounts for the ways in which people combine distinctly different kinds of
meaning making, their epistemological perspectives (i.e. their perspective on how we can
know the world) are different. As we shall see later on in this chapter, in some approaches
to multimodality the assumption is that it is possible and indeed necessary to develop
an integrated theoretical and methodological framework for some kinds of meaning
making, for instance for the study of speech, gesture, gaze and the material environ-
ment. In other approaches, the assumption is that it is possible and necessary to develop
an encompassing theoretical and methodological framework to account for all kinds of
meaning making—whether in image or in gesture or in writing or in any other mode. So
researchers who adopt the notion of multimodality (or whose work is treated by others
as being part of the field of multimodality) still draw different boundaries around what
it is in the empirical world that they aim to account for. This is not a matter of ambition
but a matter of epistemology: some argue that the differences between, say, image and
speech are too great to handle within one and the same framework; others argue that,
notwithstanding the differences, it is still possible, at a more general level, to establish
common principles of meaning-making.
Fourth, when exploring how the notion of multimodality has been and is being devel-
oped along diverse lines and schools of thought, it is important to recognize that multi-
modality has focused on the visible and audible world. We are now witnessing growing
attention to the role of senses other than sight and hearing, with studies exploring how
meaning is made through taste, smell and touch (Mondada, 2023; Jewitt & Price, 2024).
As with any horizon widening, insights from these studies might well lead to revisions of
multimodal concepts and analytical procedures.
4 INTRODUCING MULTIMODALITY

WHAT MAKES A STUDY ‘MULTIMODAL’?

When reviewing literature or when planning your own study, it is important to clarify
what makes a study multimodal. The following sets of questions about aims, theory and
method can help you assess the centrality (or marginality) of multimodality in a study:

1. Aims and research questions: Does it address research questions about meaning,
communication, discourse or interaction? Is one aim of the study to contribute to the
development of a theory of multimodality? For instance, you might find questions
such as, ‘What is the semiotic relation between objects displayed in museums and
their captions? What is the role of gaze in turn-taking?’
2. Theory: What is the place of multimodality in the theoretical framework of the study?
Is it a central concept, or is it referenced but not expanded on? It may also be that
a theory is presented that could be described as multimodal even though it is not
described as such by the authors/researchers themselves.
3. Method: What empirical materials are collected and analysed, and how? Do the col-
lected materials include documentation of human artefacts and social interactions?
Do the researchers attend to all (or at least a number of different) means of meaning
making that can be reconstructed from the collected materials? Do they give equally
systematic attention to all?

Considering the place of multimodality on these dimensions, we can distinguish between:

■■ Doing multimodality: Designing a study in which multimodality is central to aims/

research questions, theory and method;
■■ Adopting multimodal concepts: Designing a study in which multimodality concepts
(such as mode, semiotic resource) are applied to explore an object of study other than
multimodality.

When adopting multimodal concepts, you can draw selectively from approaches to multi-
modality such as the ones we discuss in the book to shed light on a social phenomenon of
interest in your own discipline or field (say, consumer behaviour). But picking and mixing
can be a tricky approach. When selecting concepts from the frameworks and connecting
them to concepts derived from other frameworks, it is important to reflect on their ‘com-
patibility.’ Drawing on a theory raises expectations about methods used. For example,
claiming to ‘use’ a theory from one of the approaches discussed in this book raises the
expectation (among others, as we shall see in the next section) that you will analyse
human artefacts or social interactions. So if you choose to combine that theory with the
method of the interview, you are likely to be seen as having produced an incoherent
framework. If you believe there are good reasons to use the interview as a method, you
need to make a case for it (alternatively, you could treat the interview not as a method
but as an object of study and analyse it multimodally).
Making explicit what the place of multimodality is in one’s study along these lines can
be a way of setting appropriate expectations about the coherence of the research design.
When you submit a research paper to a journal and suggest that the study you present is
multimodal, some reviewers will expect multimodality to be central throughout the paper.
When you explain that you adopt selected multimodal concepts, reviewers are more likely
 Navigating a diverse field 5

to assess the ‘fit’ between those concepts and the theoretical and methodological frame
within which you integrate them. We will elaborate on the issue of mixing approaches in
Chapter 6.

THREE APPROACHES TO MULTIMODAL RESEARCH

In Chapters 3, 4 and 5 of this book, we discuss three approaches to doing multimodality.

We will elaborate on how elements of the three approaches have been incorporated into
other approaches in Chapter 6. Each is grounded in a distinct discipline, with a distinct
theoretical and methodological outlook: conversation analysis, systemic functional lin-
guistics and social semiotics. Not all scholars working in these originating disciplines are
interested in multimodality. For instance, many conversation analysts or systemic func-
tional linguists continue to focus on the study of ‘talk’ or ‘speech.’ Yet within each of the
three disciplines, we can identify a substantial and well-established body of literature and
a community of scholars engaging with multimodal research. It is these bodies of work
that we will focus on.
While there are significant differences between them, they share a number of impor-
tant features:

■■ They draw on disciplines that originally focused on language in use, that is disciplines
concerned with what people do with language in their everyday life, notably how
they construe the social world through language.
■■ They have a preference for collecting and analysing observable traces of meaning
making, notably those found on human artefacts and video recordings of social
interaction.
■■ They aim to describe, transcribe, annotate and analyse materials at a micro level, that
is with attention to the fine-grained detail of form and meaning.
■■ Through micro-analysis, they have produced rich, detailed metalanguages for theo-
rizing about the social world.
■■ They have all branched out, maintaining the focus on the social and incorporating
means of meaning making beyond speech and writing in their theoretical and meth-
odological frameworks.

As the last bullet point suggests, the approaches that we focus on in this book have
developed a more encompassing multimodal frame, largely by expanding their original
frame: all had developed sophisticated toolkits to investigate language in use and then
branched out, as it were, to explore meaning made with other means—gesture, for
instance, or image. We should point out from the outset that the risk of branching out is
that the new territory is described in terms of the originating discipline. Indeed, this is a
common critique of all three approaches, and one that we will attend to throughout the
book. When expanding the traditional scope, it is important to keep a close eye on what
is typical of a mode or semiotic resource and what may count as a more general principle
of meaning making, making sure that linguistic categories are not imposed onto other
modes. Every time the frame is expanded, old terms and categories need to be revisited
and re-evaluated, in the light of the wider range of empirical cases being considered.
So we might ask, ‘What would the counterpart be of a verb in image?’ But we can ask
6 INTRODUCING MULTIMODALITY

that only if we then immediately add, ‘Maybe image doesn’t have anything like the verb.
Maybe it has categories unlike anything language has.’
The same can be said about the names of the originating disciplines. The terms ‘con-
versation analysis’ or ‘systemic functional linguistics’ no longer match the scope of the
disciplines they describe. Alternative terms have from time to time been coined to mark
the changing scopes of these disciplines. As none of them ever settled, we will continue
to use some of the old names and use new names if they are widely used within the com-
munity they represent. Thus we use the term ‘systemic functional multimodal discourse
analysis’ (SF-MDA) but not, for instance, ‘multimodal conversation analysis.’
We will discuss the three approaches at length in Chapters 3, 4 and 5, respectively.
Here we summarize them by briefly introducing their aims, history, theory of meaning,
concept of mode, empirical focus and methodology. We also present a typical research
question for each approach. If you have problems understanding some of the bullet
points at this point, rest assured that we will come back to all of them.

Systemic functional linguistics

■■ Aim: To understand the ways in which language is organized and used to fulfil a
range of social functions. From there, the focus shifts to other semiotic resources.
■■ History: Originally developed by Michael Halliday in the 1960s in the UK, influenced
by European functionalism. O’Toole, van Leeuwen, Martin and O’Halloran and oth-
ers have taken SFL procedures to explore what is now often called multimodal dis-
course, resulting in systemic functional multimodal discourse analysis (SF-MDA).
The approach aims to understand and describe the functions of different semiotic
resources as systems of meaning and to analyse the meanings that arise when semi-
otic choices combine in multimodal phenomena over space and time.
■■ Theory of meaning: Language is conceptualized as a social semiotic resource for creat-
ing meaning. The meaning potential of language is reflected in its underlying organiza-
tion, which is modelled as interrelated systems of meaning. The systems are ‘networks
of interlocking options’ (i.e. choice between different forms), and ‘text’ is a process and
product of selection (and materialization) from that potential (e.g. Halliday, 2008). This
provides the basis for conceptualizing other semiotic resources as systems of meaning
from which choices are made in multimodal texts, interactions and events.
■■ Concept of mode: The notion of semiotic resource is central to multimodal studies
grounded in SFL. Semiotic resources are seen to fulfil four main social functions: (1) to
construct our experience of the word, (2) to make logical connections in that world,
(3) to enact social relations and (4) to organize the message. Semiotic resources have
different metafunctional strengths which have evolved according to their use.
■■ Empirical focus: ‘Artefacts’ of all kinds, including print and digital texts, videos and
three-dimensional objects and sites. Usually these artefacts are readily available as
popular media (advertisements, TV programmes, websites, social media), educational
media (e.g. textbooks and other education materials) and arts and crafts (e.g. sculp-
tures, buildings).
■■ Methodology: Detailed transcription and analysis of selected fragments of the
texts, as well as the analysis of larger corpora (e.g. Bateman, 2008; O’Toole, 2011;
O’Halloran, Tan & E, 2014; O’Halloran et al., 2022).
 Navigating a diverse field 7

■■ Typical research question: Unsworth (2007), building on Martinec and Salway (2005),
explores the nature of text–image relations in school textbooks and other educational
materials by developing a classification system that documents the types of logical
relations established between the text and images. In this SF approach, the questions
that are asked include: ‘What is the nature of the text–image relations? Are these
compatible with the communicative purpose of the educational materials? What, if
any, challenges do they represent for young learners?’

Social semiotics

■■ Aim: To recognize the agency of social actors and social/power relations between them.
■■ History: Pioneered by Gunther Kress and Bob Hodge in the early 1980s in Australia,
building on critical linguistics, SFL, semiotics and social theory. Van Leeuwen brought
inspiration from music and film studies.
■■ Theory of meaning: Based on the notion of the motivated sign (Kress, 1993), which
holds that the relation between signifier and signified is always motivated.
■■ Concept of mode: Central to social semiotic theory. Indeed most theorizing on what
counts as mode comes from social semiotics. A short definition would be ‘a set of
socially and culturally shaped resources for making meaning’ that has distinct ‘affor-
dances’ (cf. Kress, 2014).
■■ Empirical focus: Initially focused on ‘artefacts’ (especially print media, film and
games—both ‘professional,’ e.g. an advert in a magazine, and ‘vernacular,’ e.g. a
child’s drawing), it then also began to account for social interaction recorded on
video through fieldwork.
■■ Methodology: Typically detailed analysis of selected small fragments (e.g. a drawing
or a small set of them), sometimes involving historical comparisons; is often com-
bined with ethnography.
■■ Typical research question: Björkvall, Van Meerbergen and Westberg (2023) explored
how the semiotic resources of the built environment and interior design are used by
international airports to create affective atmospheres for travellers. They demonstrate
how the same spatial resources can be used differently to signify both notions of
freedom and surveillance, of being in control and being controlled. For example, ceil-
ing height and permeability of wall material may be used to suggest varying levels of
boundedness/openness.

Conversation analysis

■■ Aim: To explore how people engage in social interaction in different domains and
types of activity.
■■ History: Originally developed in the US in the early 1960s by Schegloff, Sacks and
Jefferson, influenced by interactionism and ethnomethodology. The majority of early
work focused on talk-in-interaction. Interest within CA in multimodality started to
accelerate in the early 2000s.
■■ Theory of meaning: Based on the notion of sequentiality: Action unfolds in time, one
action after another. Each social action is understood in relation to the action that
8 INTRODUCING MULTIMODALITY

preceded and followed. That principle provides a basis for making claims about the
meanings that people make.
■■ Concept of mode: While it is recognized that people use a range of different (‘semi-
otic’) ‘resources’ and that these resources are ‘mutually elaborating’, the term ‘mode’
is rarely used.
■■ Empirical focus: Video recordings of ‘naturally occurring’ social encounters (i.e.
encounters that were not initiated by the researcher), obtained through fieldwork,
showing all participants involved in an activity. Activities include those in which speech
is only used occasionally, such as when two people assemble a piece of furniture.
■■ Methodology: Typically involves detailed transcription and analysis of (collections of)
small fragments or strips of interaction illustrating a phenomenon of interest.
■■ Typical research question: Goodwin and Tulbert (2011) looked at toothbrushing in
family houses. In ‘plain English,’ their question can be formulated as: ‘How do par-
ents get their children to brush their teeth?’ In a CA framework, such formulations
get translated into questions, such as: ‘How do members of a community (in this
case, families) organize their routine activities? How do they use their bodies, objects
and the built environment as resources for the accomplishment of these activities?
How do they achieve a joint focus of attention? How do they jointly “build” the activ-
ity up? How are the activities related?’

Throughout the book, we will cross-reference and point out differences and similarities
among these three focal approaches. The main differences are summarized in Table 1.1.
We want to highlight two significant differences here: one theoretical and one
methodological.

Table 1.1 Mapping three approaches to multimodality: SFL, social semiotics and conversation
analysis

SFL Social semiotics CA

Aims Recognition of social Recognition of power Recognition of social

functions of forms and agency order in interaction
Theory of meaning Meaning as choice Motivated sign Sequentiality
History European SFL, critical linguistics, American interactionism,
functionalism semiotics ethnomethodology
Conceptualization Semiotic resource, Mode, semiotic resource (Semiotic) resource
of ‘means for mode
making meaning’
Example O’Toole, Martin, Kress, van Leeuwen, Goodwin, Heath,
representatives Unsworth, O’Halloran, Adami, Diamantopoulou, Hindmarsh, Mondada,
Lim Fei, Djonov, Ravelli Björkvall, Archer Lindwall, Kendrick, Gibson
Empirical focus Artefacts, including Artefacts, mostly texts Researcher-generated
texts and objects video recordings of
interaction
Method of Micro-analysis Micro-analysis of Micro-analysis of
analysis of selected short selected short fragments, (collections of) selected
fragments, corpus historical analysis, corpus short video fragments
analysis analysis
 Navigating a diverse field 9

The theoretical point is, first of all, an issue of naming. The three approaches have dif-
ferent terminological preferences, coupled with different conceptualizations of what we
have described so far as ‘means for making meaning.’ In SS and SFL, the terms ‘mode’ and
‘semiotic resource’ are both used, and definitions have been proposed that make a distinc-
tion between the two. In CA, ‘(semiotic) resource’ is used, but ‘mode’ is not, or very rarely,
and some attempts at defining ‘(semiotic) resource’ have been made. Yet none of these
definitions is (as yet) widely and consistently used beyond those who proposed them.
There is, put simply, much variation in the meanings ascribed to mode and (semi-
otic) resource. Gesture and gaze, image and writing seem plausible candidates, but what
about colour or layout? And is photography a separate mode? What about facial expres-
sion and body posture? Are action and movement modes? You will find different answers
to these questions not only between different research publications but also within. To
avoid potential confusion, it is important to make a deliberate decision on what cat-
egories and terms to use when engaging with multimodal research. It will be helpful to
formulate some ‘working definitions,’ drawing on the ones already put forward by the
approach you adopt. Even though the working definition is unlikely to be entirely satis-
factory, it is important to strive for maximum conceptual clarity and consistency. We will
discuss the definitions proposed within our focal approaches in the respective chapters.
The methodological point is this. CA is primarily interested in meanings made in
situ, in dynamic, (technology-mediated) face-to-face interactions. It looks at artefacts
only insofar as these artefacts are being oriented to in observed interactions. So, for
instance, Charles Goodwin (2000) looked at the Munsell chart, a tool used to determine
the colour of soil by the archaeologists participating in the interactions he had video-
recorded. In social semiotics, artefacts have been explored in situ—for instance, the use
of three-dimensional models in the science classroom (Kress & van Leeuwen, 2001)—
but in other social semiotic work, artefacts have also been studied away from specific
situated interactions. For instance, Bezemer and Kress (2008) studied textbooks. Their
focus was on meanings made by the makers of textbooks (including authors and graphic
designers), not on the meanings of those who engage with textbooks, such as teachers
and students. In SFL, a similar position is usually taken, recognizing that it is possible to
reconstruct meanings from (collections of) artefacts. Thus SS and SFL generally cover a
wider empirical scope than CA. For instance, the architectural design of a building would
normally fall outside the scope of CA.
There is, of course, significant variation in the degree to which scholars stay close
(some might say ‘faithful’) to the principles put forward by the founders of the originating
disciplines. Indeed there is a tension between staying faithful to concepts as they were
originally defined and the need to revise old concepts in the light of the changing world.
After all, the world we live in now looks very different from what it looked like when the
originating disciplines appeared. Social, cultural and technological changes constantly
challenge old notions.
There are close connections between scholars working with the different frame-
works, and indeed some are active members in both communities. The closest links are
between SFL and SS; there is less interaction between representatives of CA, on the
one hand, and SFL and SS, on the other. CA is closely linked with interactional (socio)
linguistics and linguistic anthropology, and this connection is reflected in early work on
the role of, for example, gaze in classroom interaction (see the work of Ray McDermott
and Frederick Erickson). SFL and SS are also closely linked with critical discourse analysis,
10 INTRODUCING MULTIMODALITY

which developed as a separate branch of critical linguistics. That is visible, for instance,
in the joint work of Gunther Kress and Theo van Leeuwen (e.g. 2021).
In many studies, selected elements of one of the three approaches have been adopted
and brought into connection with concepts and methods derived from other disciplines,
such as psychology. For instance, you could use technology that tracks the movement
of eyes, hands, other body parts or the whole body to test hypotheses from multimodal
case studies about how people are likely to engage with and respond to certain signs
(cf. Holsanova, 2012). Other work has attempted to bring together concepts from social
semiotics with ethnographic methods of data collection and analysis. We will elaborate
on how elements of the three approaches have been incorporated into other approaches
in Chapter 6.
References
Abousnnouga, G. & Machin, D. (2011). The Changing Spaces of War Commemoration: A Multimodal
Analysis of the Discourses of British Monuments. Social Semiotics, 21 (2), 175–196.
Adami, E. (2009). ‘We/YouTube’: Exploring Sign-Making in Video-Interaction. Visual Communication, 8,
379–399.
Adami, E. , Diamantopoulou, S. & Lim, F.V. (2022). Design in Gunther Kress's Social Semiotics. London
Review of Education, 20 (1), 41.
Andersen, T.H. , Boeriis, M. , Maagerø, E. & Tønnessen, E.S. (2015). Social Semiotics: Key Figures, New
Directions. London; New York: Routledge, Taylor & Francis Group.
Antaki, C. (Ed.). (2011). Applied Conversation Analysis: Intervention and Change in Institutional Talk.
Basingstoke: Palgrave Macmillan.
Archer, A. (2014). Power, Social Justice, and Multimodal Pedagogies. In: C. Jewitt (Ed.), The Routledge
Handbook of Multimodal Analysis (2nd ed.). Milton Park, Abingdon, Oxon: Routledge, pp. 189–198.
Archer, A. (2022). A Multimodal Approach to English for Academic Purposes in Contexts of Diversity. World
Englishes, 41 (4), 545–553.
BAAL (2006). BAAL Recommendations on Good Practice. [Link]
content/uploads/2017/08/goodpractice_stud.pdf
Baldry, A. & Thibault, P. (2006). Multimodal Transcription and Text Analysis . London: Equinox.
Barthes, R. (1977 [1964]). Rhetoric of the Image. In: Image Music Text. London: Fontana.
Bateman, J. (2008). Multimodality and Genre: A Foundation for the Systematic Analysis of Multimodal
Documents. Hampshire: Palgrave Macmillan.
Bateman, J. (2014a). Looking for What Counts in Film Analysis: A Programme of Empirical Research. In: D.
Machin (Ed.), Visual Communication. Berlin: De Gruyter Mouton, pp. 301–329.
Bateman, J. (2014b). Text and Image: A Critical Introduction to the Visual/Verbal Divide. London; New York:
Routledge.
Bateman, J. (2014). Using Multimodal Corpora for Multimodal Research. In: C. Jewitt (Ed.), The Routledge
Handbook of Multimodal Analysis (2nd ed.). Milton Park, Abingdon, Oxon: Routledge, pp. 238–252.
Bateman, J. (2022a). Growing Theory for Practice: Empirical Multimodality Beyond the Case Study.
Multimodal Communication, 11 (1), pp. 63–74.
Bateman, J. (2022b). Multimodality, Where Next? Some Meta-Methodological Considerations. Multimodality
& Society, 2 (1), 41–63.
Bateman, J. , Delin, J. & Henschel, R. (2004). Multimodality and Empiricism: Preparing for a Corpus-based
Approach to the Study of Multimodal Meaning-Making. In: E. Ventola , C. Charles & M. Kaltenbacher (Eds.),
Perspectives on Multimodality. Amsterdam: John Benjamins, pp. 65–87.
Bateman, J. & O'Donnell, M. (2015). Computational Linguistics: The Halliday Connection. In: J.J. Webster
(Ed.), The Bloomsbury Companion to M. A. K. Halliday. London: Bloomsbury.
Bateman, J. & Schmidt, K.-H. (2012). Multimodal Film Analysis: How Films Mean. London; New York:
Routledge.
Bell, P. & Milic, M. (2002). Goffman's Gender Advertisements Revisited: Combining Content Analysis with
Semiotic Analysis. Visual Communication, 1, 203–222.
Benveniste, Émile (1969). The Semiology of Language. In: Robert E. Innis (Ed.) (1986). Semiotics: An
Introductory Reader. London: Hutchinson, pp. 228–246.
Bernad-Mechó, E. (2023). Multimodal (Inter)action Analysis for the Study of Lectures: Active and Passive
Uses of Metadiscourse. Multimodal Communication, 12 (1), 7–21.
Bezemer, J. (2014). Multimodal Transcription: A Case Study. In: S. Norris & C.D. Maier (Eds.), Interactions,
Images and Texts: A Reader in Multimodality. Berlin; München; Boston: De Gruyter Mouton, pp. 155–170.
Bezemer, J. (2023). What Modes Can and Cannot Do: Affordance in Gunther Kress's Theory of Sign
Making. Text & Talk, 44 (4), 493–510.
Bezemer, J. & Cowan, K. (2021). Exploring Reading in Social Semiotics: Theory and Methods. Education,
3–13, 49(1), 107–118.
Bezemer, J. , Cope, A. , Faiz, O. & Kneebone, R. (2012). Participation of surgical residents in operations:
challenging a common classification. World Journal of Surgery, 36 (9), 2011–2014.
Bezemer, J. , Cope, A. , Kress, G. & Kneebone, R. (2014). Holding the Scalpel: Achieving Surgical Care in a
Learning Environment. Journal of Contemporary Ethnography, 43 (1), 38–63.
Bezemer, J. & G. Kress (2008). Writing in Multimodal Texts: A Social Semiotic Account of Designs for
Learning. Written Communication, 25 (2), 166–195.
Bezemer, J. & Kress, G. (2014). Touch: A Resource for Meaning Making. Australian Journal of Language
and Literacy, 37, 77–85.
Bezemer, J. & Kress, G. (2016). Multimodality, Learning and Communication. A Social Semiotic Frame.
London: Routledge.
Bezemer, J. & Mavers, D. (2011). Multimodal Transcription as Academic Practice: A Social Semiotic
Perspective. International Journal of Social Research Methodology, 14, 191–206.
Blommaert, J. (2005). Discourse. Cambridge: Cambridge University Press.
Bloor, T. & Bloor, M. (2013). The Functional Analysis of English: A Hallidayan Approach (3rd ed.). New York:
Routledge.
Björkvall, A. , Van Meerbergen, S. & Westberg, G. (2023). Feeling Safe While Being Surveilled: The Spatial
Semiotics of Affect at International Airports. Social Semiotics, 33 (1), 209–231.
British Sociological Association (BSA) (2017). Digital Research Ethics Annex.
[Link]/media/24309/bsa_statement_of_ethical_practice_annexe.pdf
Broth, M. , E. Laurier & L. Mondada (Eds.) (2014). Studies of Video Practices: Video at Work. New York:
Routledge/Taylor & Francis Group, pp. 1–29.
Burawoy, M. (2009). The Extended Case Method: Four Countries, Four Decades, Four Great
Transformations, and One Theoretical Tradition. Berkley, CA: University of California Press.
Butler, C. (2003a). Structure and Function: A Guide to Three Major Structural-Functional Theories, Part 1:
Approaches to the Simplex Clause. Amsterdam; Philadelphia: John Benjamins.
Butler, C. (2003b). Structure and Function: A Guide to Three Major Structural-Functional Theories, Part 2:
From Clause to Discourse and Beyond. Amsterdam; Philadelphia: John Benjamins.
Caldwell, D. , Knox, J.S. & Martin, J.R. (Eds.) (2022). Appliable Linguistics and Social Semiotics: Developing
Theory from Practice. London: Bloomsbury Academic.
Campbell, S. & Roberts, C. (2007). Migration, Ethnicity and Competing Discourses in the Job Interview:
Synthesising the Institutional and Personal. Discourse and Society, 18 (3), 243–271.
Canale, G. (2023). A Multimodal and Ethnographic Approach to Textbook Discourse. Abingdon, Oxon:
Routledge.
Chandler, D. (2007). Semiotics (2nd ed). Abingdon: Routledge.
Cook, G. (2003). Applied Linguistics. Cambridge: Cambridge University Press.
Cope, B. & Kalantzis, M. (2020). Making Sense: Reference, Agency and Structure in a Grammar of
Multimodal Meaning. Cambridge: Cambridge University Press.
Cowan, K. (2014). Multimodal Transcription of Video: Examining Interaction in Early Years Classrooms.
Classroom Discourse, 5 (1), 6–21.
Cowan, K. (2018). Visualising Young Children's Play: Exploring Multimodal Transcription of Video-Recorded
Interaction. Doctoral Thesis, London: UCL Institute of Education.
Cowan, K. & Flewitt, R. (2021). Moving from Paper-Based to Digital Documentation in Early Childhood
Education: Democratic Potentials and Challenges. International Journal of Early Years Education, 31 (4),
888–906.
Cowan, K. , van Leeuwen, T. & Selander, S. (2024). Pandemic Playthings: A Multimodal Perspective on
Toys and Toy Play in the Time of COVID-19. Global Studies of Childhood, 14 (1), 62–80.
Crescenzi, L. , Jewitt, C. & Price, S. (2014). The Role of Touch in Pre-school Children's Learning Using iPad
Versus Paper Interaction. Australian Journal of Language and Literacy, 37, 86–95.
Dattatreyan, E. (2024). Multimodal Ethnography, Oxford Bibliographies.
[Link]
Davis, J. (2010). Architecture of the Personal Interactive Homepage: Constructing the Self through
MySpace. New Media & Society, 12 (7).
Deppermann, A. (2019). Intersubjectivity and Other Grounds for Action-Coordination in an Environment of
Restricted Interaction: Coordinating with Oncoming Traffic When Passing an Obstacle. Language &
Communication, 65, 22–40.
Deppermann, A. & R. Schmidt (2007). Koordination. Zur Begründung eines neuen
Forschungsgegenstandes. In: R. Schmitt (Ed.), Koordination. Studien zur multimodalen Interaktion.
Tübingen: Gunter Narr, pp. 15–54.
De Stefani, E. , Broth, M. & Depperman, A. (2019). On the Road: Communicating Traffic. Language and
Communication, 65, 1–148.
Diamantopoulou, S. & Christidou, D. (2019). Museum Encounters: A Choreography of Visitors’ Bodies in
Interaction. Museum Management and Curatorship, 34 (4), 344–361.
Dicks, B. (2014). Action, Experience, Communication: Three Methodological Paradigms for Researching
Multimodal and Multisensory Settings. Qualitative Research, 14 (6), 656–674.
[Link]
Dicks, B. , Soyinka, B. & Coffey, A. (2006). Multimodal Ethnography. Qualitative Research, 6 (1), 77–96.
Djonov, E. & van Leeuwen, T. (2022). Semiotic Software Through the Lens of Systemic Functional Theory.
In: D. Caldwell , J.S. Knox & J.R. Martin (Eds.), Appliable Linguistics and Social Semiotics: Developing
Theory From Practice. London: Bloomsbury Academic, pp. 421–435.
Domingo, M. (2014). Transnational Language Flows in Digital Platforms: A Study of Urban Youth and Their
Multi- Modal Text Making. Pedagogies, 9 (1), 7–25.
Dowling, P. (2004). Mythologising and Organising. [Link]/publications/mo/[Link]
Dreyfus, S. , Hood, S. & Stenglin, M. (Eds.). (2011). Semiotic Margins: Meaning in Multimodalities. London;
New York: Continuum.
Eggins, S. (2005). An Introduction to Systemic Functional Linguistics (2nd ed.). London: Continuum.
Erickson, F. (2004). Talk and Social Theory: Ecologies of Speaking and Listening in Everyday Life.
Cambridge: Polity.
Erickson, F. (2007). Ways of Seeing Video: Toward a Phenomenology of Viewing Minimally Edited Footage.
In: R. Goldman (Ed.), Video Research in the Learning Sciences. Mahwah, NJ: Lawrence Erlbaum
Associates, pp. 145–158.
Eriksson, G. & Kenalemang, L.M. (2023). How Cosmetic Apps Fragmentise and Metricise the Female Face:
A Multimodal Critical Discourse Analysis. Discourse & Communication, 17 (3), 278–297.
Flewitt, R. (2011). Bringing Ethnography to a Multimodal Investigation of Early Literacy in a Digital Age.
Qualitative Research, 11, 293–310.
Flewitt, R. , Hampel, R. , Hauck, M. & Lancaster, L. (2014). What are Multimodal Data and Transcription? In:
Jewitt, C. (Ed.), The Routledge Handbook of Multimodal Analysis (2nd ed.). Milton Park, Abingdon, Oxon:
Routledge, pp. 44–59.
Garfinkel, H. (1967). Studies in Ethnomethodology. Cambridge: Polity.
Garfinkel, H. & Livingston, E. (2003). Phenomenal Field Properties of Order in Formatted Queues and Their
Neglected Standing in the Current Situation of Inquiry. Visual Studies, 18 (1), 21–28.
Gibson, W. (2024). Flirting and Winking in Tinder Chats: Emoji, Ambiguity, and Sequential Actions. Internet
Pragmatics 7(2): 249–271.
Gibson, W. & vom Lehm, D. (2021). The Senses in Social Interaction. Symbolic Interaction, 44 (1), 3–9.
Goffman, E. (1981). Forms of Talk. Oxford: Blackwell.
Goffman, E. (1983). The Interaction Order: American Sociological Association, 1982 Presidential Address.
American Sociological Review, 48 (1), 1–17.
Goldman, R. (Ed.) (2007). Video Research in the Learning Sciences. Mahwah, NJ: Lawrence Erlbaum
Associates.
González-Martínez, E. (2024). Senior Staff Member Walks Ahead, Nursing Intern Follows: Mobility Practices
in Hospital Corridors. In: S. Keel (Ed.), Medical and Healthcare Interactions. Members' Competence and
Socialization. London and New York: Routledge, pp. 73–101.
Goodings, L. (2012). Understanding Social Network Sites: Lessons from MySpace. Visual Communication,
11, 485–510.
Goodwin, C. (1979). The Interactive Construction of a Sentence in Natural Conversation. In: G. Psathas
(Ed.), Everyday Language: Studies in Ethnomethodology. New York: Irvington. pp. 97–121.
Goodwin, C. (1994). Professional Vision. American Anthropologist, 96 (3), 606–633.
Goodwin, C. (2000). Action and Embodiment Within Situated Human Interaction. Journal of Pragmatics, 32,
1489–1522.
Goodwin, C. , Goodwin, M.H. & Olsher, D. (2009). Producing Sense with Nonsense Syllables: Turn and
Sequence in Conversations with a Man with Severe Aphasia. In: P. Griffiths , A.J. Merrison & A. Bloomer
(Eds.), Language in Use: A Reader. London, Routledge, pp. 272–285.
Goodwin, M. & Tulbert, E. (2011). Choreographies of Attention: Multimodality in a Routine Family Activity. In:
J. Streeck , C. Goodwin & C.D. LeBaron (Eds.), Embodied Interaction: Language and Body in the Material
World, in the series Learning in Doing: Social, Cognitive and Computational Perspectives. New York:
Cambridge University Press, pp. 79–92.
Govender, N. (2020). Critical Transmodal Pedagogies: Student Teachers Play with Genre Conventions.
Multimodal Communication, 9 (1).
Gumperz, J. & Hymes, D. (Eds). (1964). The Ethnography of Communication. American Anthropologist, 66
(6), part 2.
Hak, T. (1999). ‘Text’ and ‘Con-text’: Talk Bias in Studies of Health Care Work. In: S. Sarangi & C. Roberts
(Eds.), Talk, Work and Institutional Order: Discourse in Medical, Mediation and Management Settings. Berlin:
Mouton de Gruyter.
Hall, E.T. (1990). The Silent Language. New York: Anchor Books.
Halliday, M.A.K. (1976 [1956]). Grammatical Categories in Modern Chinese. Transactions of the Philological
Society, 1956, 180–202.
Halliday, M.A.K. (1978). Language as Social Semiotic: The Social Interpretation of Language and Meaning.
London: Edward Arnold.
Halliday, M.A.K. (1985a). An Introduction to Functional Grammar (1st ed.). London: Edward Arnold.
Halliday, M.A.K. (1985b). Spoken and Written Language. Geelong, Victoria: Deakin University Press
[Republished by Oxford University Press 1989].
Halliday, M.A.K. (1994). An Introduction to Functional Grammar (2nd ed.). London: Arnold.
Halliday, M.A.K. (2002). A Personal Perspective. In: J.J. Webster (Ed.), The Collected Works of M. A. K.
Halliday. Volume 1. On Grammar. London: Continuum.
Halliday, M.A.K. (2003 [1977]). Ideas about Language. In: J.J. Webster (Ed.), The Collected Works of M. A.
K. Halliday. Volume 3. On Language and Linguistics. London and New York: Bloomsbury, pp. 92–115.
Halliday, M.A.K. (2004). Introduction: How Big is a Language? On the Power of Language. In: J.J. Webster
(Ed.), The Language of Science: Collected Works of M. A. K. Halliday (Volume 5). London and New York:
Bloomsbury, pp. xi–xxiv.
Halliday, M.A.K. (2006). The Collected Works of M. A. K. Halliday. Volume 5. The Language of Science.
London & New York: Continuum.
Halliday, M.A.K. (2008). Complementarities in Language. Beijing: Commercial Press.
Halliday, M.A.K. (2015). The Influence of Marxism. In: J.J. Webster (Ed.), The Bloomsbury Companion to M.
A. K. Halliday. London: Bloomsbury, pp. 94–100.
Halliday, M.A.K. & Fawcett, R.P. (1987). Introduction. In: M.A.K. Halliday & R.P. Fawcett (Eds.), New
Developments in Systemic Linguistics Vol. 1: Theory and Description. London: Frances Pinter, pp. 1–13.
Halliday, M.A.K. & Hasan, R. (1985). Language, Context, and Text: Aspects of Language in a Social-
Semiotic Perspective. Geelong, Victoria: Deakin University Press [Republished by Oxford University Press
1989].
Halliday, M.A.K. & Matthiessen, C.M.I.M. (2014). Halliday's Introduction to Functional Grammar (4th ed,
revised by C.M.I.M. Matthiessen (Ed.)). London; New York: Routledge.
Hammersley, M. & Traianou, A. (2012). Ethics in Qualitative Research: Controversies and Contexts.
Thousand Oaks, CA: SAGE Publications.
Han, J. & Zappavigna, M. (2024). Multimodal Rhythm in TikTok Videos: Exploring a Recontextualization of
the Gillard ‘Misogyny Speech.’ Multimodality & Society, 4 (1), 58–79.
Hasan, R. (2015). Systemic Functional Linguistics: Halliday and the Evolution of a Social Semiotic. In: J.J.
Webster (Ed.), The Bloomsbury Companion to M. A. K. Halliday. London: Bloomsbury, pp. 101–134.
Haviland, J. (2011). Musical Spaces. In: J. Streeck , C. Goodwin & C. LeBaron (Eds.), Embodied Interaction.
Language and Body in the Material World. Cambridge: Cambridge University Press, pp. 289–304.
Heath, C. (1986). Body Movement and Speech in Medical Interaction. Cambridge: Cambridge University
Press.
Heath, C. , Luff, P. & Hindmarsh, J. (2010). Video in Qualitative Research. London: Sage.
Heritage, J. (1997). Conversation Analysis and Institutional Talk: Analyzing Data. In: David Silverman (Ed.),
Qualitative Research: Theory, Method and Practice. London: Sage, pp. 161–182.
Herrlitz, W. (1994). Spitzen der Eisberge. Vorbemerkungen zu einer vergleichenden Analyse metonymischer
Struktur in Unterricht der Standardsprache. Osnabrücker Beiträge zur Sprachtheorie, 48, 13–52.
Hiippala, T. (2016). The Structure of Multimodal Documents: An Empirical Approach, in the series Routledge
Studies in Multimodality. New York: Routledge.
Hiippala, T. (2021). Distant Viewing and Multimodality Theory: Prospects and Challenges. Multimodality &
Society, 1 (2), 134–152.
Hiippala, T. (2023). Corpus-Based Insights into Multimodality and Genre in Primary School Science
Diagrams. Visual Communication, 0(0).
Hiippala, T. & Bateman, J.A. (2020). Introducing the Diagrammatic Semiotic Mode. arXiv, 2001.11224.
Hiippala T. & Bateman, J.A. (2022a). Introducing the Diagrammatic Semiotic Mode. In: V. Giardino et al.
(Eds.), Diagrammatic Representation and Inference: 13th International Conference (Diagrams 2022),
Lecture Notes in Computer Science, 13462. Cham, Switzerland: Springer, pp. 3–19.
Hiippala T. & Bateman J.A. (2022b). Semiotically-Grounded Distant View of Diagrams: Insights From Two
Multimodal Corpora. Digital Scholarship in the Humanities, 37 (2): 405–425.
Hiippala, T. , Alikhani, M. , Haverinen, J. , Kalliokoski, T. , Logacheva, E. , Orekhova, S. , Tuomainen, A. ,
Stone, M. & Bateman, J. A. (2021). AI2D-RST: A Multimodal Corpus of 1000 Primary School Science
Diagrams. Language Resources & Evaluation, 55, 661–688.
Hindmarsh, J. & Heath, C. (2007). Video-Based Studies of Work Practice. Sociology Compass, 1 (1),
156–173.
Hindmarsh, J. & A. Pilnick (2002). The Tacit Order of Teamwork: Collaboration and Embodied Conduct In
Anesthesia. The Sociological Quarterly, 43 (2), 139–164.
Hindmarsh, J. & A. Pilnick (2007). Knowing Bodies at Work: Embodiment and Ephemeral Teamwork in
Anaesthesia. Organization Studies, 28 (9), 1395–1416.
Ho, W.Y.J. & Feng, D. (2022). Orchestrating Multimodal Resources in English Language Teaching: A Critical
Study of an Online English Teaching Video. Pedagogies: An International Journal, 17 (4), 368–388.
Hodge, B. (2024). Motivated Signs and Multimodal Analysis in Gunther Kress's Semiotics. Text & Talk, 44
(4).
Hofstetter, E. & Robles, J. (2023). Metagaming and Multiactivity: How Board Game Players Deal With
Progressivity. In: P. Haddington , T. Eilittä , A. Kamunen , L. Kohonen-Aho , I. Rautiainen & A. Vatanen
(Eds.), Complexity of Interaction: Studies in Multimodal Conversation Analysis. Cham: Palgrave Macmillan,
pp. 65–97.
Holmberg, P. (2021). Reading Runes with the Sun. A Geosemiotic Analysis of the Rök Runestone.
Multimodality & Society, 1 (4), 455–473.
Holsanova, J. (2012). New Methods for Studying Visual Communication and Multimodal Integration. Visual
Communication, 11, 251–257.
Hutchins, E. (1995). Cognition in the Wild. Cambridge, MA: MIT Press.
Ibrahim, S. , Clarke, M. , Vasalou, A. & Bezemer, J. (2024). Common Ground in AAC: How Children Who
Use AAC and Teaching Staff Shape Interaction in the Multimodal Classroom. Augmentative and Alternative
Communication, 40 (2), 74–85.
Iedema, R. (2003). Multimodality, Resemiotization: Extending the Analysis of Discourse as a Multisemiotic
Practice. Visual Communication, 2 (1), 29–57.
Insulander, E. & Thorsén, D. (2023). Teaching Through Objects and Collections: The Case of Strängnäs
Secondary Grammar School and School Museum 1830–1960. Multimodality & Society, 3 (4), 336–365.
Israel, M. (2015). Research Ethics and Integrity for Social Scientists: Beyond Regulatory Compliance.
London; Thousand Oaks, CA: Sage.
Jakobson, R. (1968). Selected Writings: Word and Language. The Hague: Mouton and Co.
Jefferson, G. (2004). Glossary of Transcript Symbols with an Introduction. In G. H. Lerner (Ed.),
Conversation Analysis: Studies from the First Generation. Amsterdam: John Benjamins, pp. 13–31.
Jewitt, C. (1997). Images of Men: Male Sexuality in Sexual Health Leaflets and Posters for Young People.
Sociological Research Online, 2.
Jewitt, C. (2002). Move from Page to Screen. Visual Communication, 1 (2), 171–196.
Jewitt, C. (2005a). Multimodal ‘Reading’ and ‘Writing’ On Screen. Discourse, 26 (3), 315–332.
Jewitt, C. (2005b). Classrooms & the Design of Pedagogic Discourse. Culture & Psychology, 11 (3),
309–320.
Jewitt, C. , Bezemer, J. , Jones, K. & Kress, G. (2009). Changing English? The Impact of Technology and
Policy in the 21st Century. English Teaching: Practice and Critique, 8 (3), 21–40.
Jewitt, C. , Chubinidze, D. , Price, S. , Yiannoutsou, N. & Barker, N. (2021). Making Sense of Digitally
Remediated Touch in Virtual Reality Experiences. Discourse, Context, Media, 41.
Jewitt, C. & Leder Mackley, K. (2019). Methodological Dialogues Across Multimodality and Sensory
Ethnography: Digital Touch Communication. Qualitative Research, 19 (1), 90–110.
[Link]
Jewitt, C. , Leder Mackley, K. & Price, S. (2021). Digitally-Mediated Parent–Baby Touch and the Formation
of Subjectivities. Visual Communication, 20 (4), 577–599.
Jewitt, C. & Kress, G. (2003). Multimodal Literacy. New York: Peter Lang.
Jewitt, C. & Price, S. (2024). Digital Touch. Cambridge, UK: Polity Press.
Jones, C. & Ventola, E. (Eds.). (2008). From Language to Multimodality: New Developments in the Study of
Ideational Meaning. London: Equinox.
Kalantzis, M. & Cope, B. (2020). Adding Sense: Context and Interest in a Grammar of Multimodal Meaning.
Cambridge, UK: Cambridge University Press.
Katz, Jerrold J. (1972). Semantic Theory. New York: Harper & Row.
Kendon, A. (1990). Conducting Interaction: Patterns of Behavior in Focused Encounters. Cambridge:
Cambridge University Press.
Kendon, A. (2004a). Gesture: Visible Action as Utterance. Cambridge: Cambridge University Press.
Kendon, A. (2004b). On Pointing. In: Gesture: Visible Action as Utterance. Cambridge: Cambridge University
Press, pp. 199–224.
Kendrick, K. & Drew, P. (2016). Recruitment: Offers, Requests, and the Organization of Assistance in
Interaction. Research on Language and Social Interaction, 49 (1), 1–19.
Kendrick, K. , Holler, J. & Levinson, S. (2023). Turn-Taking in Human Face-To-Face Interaction is
Multimodal: Gaze Direction and Manual Gestures Aid the Coordination of Turn Transitions. Philosophical
Transactions of the Royal Society, B (378), 20210473.
Kenner, C. & Kress, G. (2003). The Multisemiotic Resources of Biliterate Children. Journal of Early
Childhood Literacy, 3, 179–202.
Knoblauch, H. , Soeffner, H.-G. , Raab, J. & Schnettler, B. (Eds.) (2012). Video Analysis: Methodology and
Methods; Qualitative Audiovisual Data Analysis in Sociology, 3., rev. ed. Frankfurt am Main: Lang.
Koffka, K. (1935). Principles of Gestalt Psychology. London: Lund Humphries.
Kong, K. C. (2013). A Corpus-Based Study in Comparing the Multimodality of Chinese- and English-
Language Newspapers. Visual Communication, 12, 173–196.
Kress, G. R. (1993). Against Arbitrariness: The Social Production of the Sign as a Foundational Issue in
Critical Discourse Analysis. Discourse & Society, 4, 169–191.
Kress, G. R. (1997). Before Writing: Rethinking the Paths to Literacy. London; New York: Routledge.
Kress, G. R. (2003). Literacy in the New Media Age. London: Routledge.
Kress, G. R. (2010). Multimodality: A Social Semiotic Approach to Contemporary Communication. London;
New York: Routledge.
Kress, G.R. (2014). ‘What is Mode?’ In: C. Jewitt (Ed.), The Routledge Handbook of Multimodal Analysis
(2nd ed.). London: Routledge, pp. 60–75.
Kress, G.R. & Hodge, B. (1979). Language as Ideology. London; Boston: Routledge & Kegan Paul.
Kress, G.R. & Hodge, R. I. V. (1988). Social Semiotics. Cambridge: Polity Press.
Kress, G.R. , Jewitt, C. , Jones, K. , Franks, A. & Hardcastle, J. (2005). English In Urban Classrooms: A
Multimodal Perspective on Teaching and Learning. London; New York: Routledge Falmer.
Kress, G.R. , Jewitt, C. , Ogborn, J. & Tsatsarelis, C. (2014). Multimodal Teaching And Learning: The
Rhetorics of the Science Classroom (2nd ed.) London: Bloomsbury Press.
Kress, G.R. & van Leeuwen, T. (1996). Reading Images: The Grammar of Visual Design. London:
Routledge.
Kress, G.R. & van Leeuwen, T. (2001). Multimodal Discourse: The Modes and Media of Contemporary
Communication. New York; London: Arnold and Oxford University Press.
Kress, G. R. & van Leeuwen, T. (2006). Reading Images: The Grammar of Visual Design (2nd ed.). London:
Routledge.
Kress, G. R. & van Leeuwen, T. (2021). Reading Images: The Grammar of Visual Design (3rd ed.). London:
Routledge.
Lebaron, C. & J. Streeck (1997). Built Space and the Interactional Framing of Experience During a Murder
Interrogation. Human Studies, 20, 1–25.
Ledin, P. & Machin, D. (2018). Doing Critical Discourse Studies With Multimodality: From Metafunctions to
Materiality. Critical Discourse Studies, 16 (5), 497–513.
Lemke, J. (1998). Multiplying Meaning: Visual and Verbal Semiotics in Scientific Text. In: J. I. M. Martin & R.
Veel (Eds.), Reading Science. London: Routledge, pp. 87–113.
Levinson, S. (1983). Pragmatics. Cambridge: Cambridge University Press.
Lim, F. V. (2021a). Designing Learning with Embodied Teaching Perspectives from Multimodality. London:
Routledge.
Lim, F. V. (2021b). Investigating Intersemiosis: A Systemic Functional Multimodal Discourse Analysis of the
Relationship between Language and Gesture in Classroom Discourse. Visual Communication, 20, 34–58.
Literat, I. , Conover, A. , Herbert-Wasson, E. , Kirsch Page, K. , Riina-Ferrie, J. , Stephens, R. & Vasudevan,
L. (2017). Toward Multimodal Inquiry: Opportunities, Challenges and Implications of Multimodality for
Research and Scholarship. Higher Education Research & Development, 37 (3), 565–578.
Liu, Y. & O'Halloran, K. L. (2009). Intersemiotic Texture: Analyzing Cohesive Devices between Language
and Images. Social Semiotics , 19 (4), 367–387.
Lotherington, H. & Jenson, J. (2011). Teaching Multimodal and Digital Literacy in L2 Settings: New
Literacies, New Basics, New Pedagogies. Annual Review of Applied Linguistics, 31, 226–246.
Luff, P. , J. Hindmarsh & C. Heath (2000). Workplace Studies: Recovering Work Practice and Informing
System Design. Cambridge: Cambridge University Press.
Lynch, M. & Lindwall, O. (Eds.). (2023). Instructed and Instructive Actions: The Situated Production,
Reproduction, and Subversion of Social Order. London: Routledge.
Machin, D. (2013). What is Multimodal Critical Discourse Studies? Critical Discourse Studies, 10 (4),
347–355.
Marsh, J. (2019). Researching the Digital Literacy and Multimodal Practices of Young Children. In: O. Erstad
, R. Flewitt , B. Kümmerling-Meibauer & I. Pereira (Eds.), The Routledge Handbook of Digital Literacies in
Early Childhood. London: Routledge.
Martin, J.R. (1992). English Text: System and Structure. Amsterdam: John Benjamins.
Martin, J.R. (2002). Meaning Beyond the Clause: SFL Perspectives. Annual Review of Applied Linguistics,
22, 52–74.
Martin, J. R. (2016). Meaning Matters: A Short History of Systemic Functional Linguistics. WORD, 62 (1),
35–38.
Martin, J.R. & Rose, D. (2007). Working with Discourse: Meaning Beyond the Clause (2nd ed.). London:
Continuum.
Martin, J.R. & White, P.R.R. (2005). The Language of Evaluation: Appraisal in English. London; New York:
Palgrave Macmillan.
Martinec, R. & Salway, A. (2005). A System for Image-Text Relations in New (and Old) Media. Visual
Communication, 4, 337–371.
Matthiessen, C.M.I.M. (2015). Halliday on Language. In: J.J. Webster (Ed.), The Bloomsbury Companion to
M. A. K. Halliday. London: Bloomsbury, pp. 137–202.
Mavers, D. (2009). Student Text-Making as Semiotic Work. Journal of Early Childhood Literacy, 9, 141–155.
McVee, M. , Schucker, K. , Silvestri, K. & Tripp, J. (2023). Using Multiliteracies and Multimodality to Explore
Engineering Literacies and Problem-Scoping for Multilingual Learners: A Perspective on Multiliteracies for
STEM Learning. In: R. Tierney , F. Rizvi & K. Ercikan (Eds.), International Encyclopedia of Education (Fourth
Edition). Amsterdam: Elsevier, pp. 50–63.
Mehan, H. (1980). The Competent Student. Anthropology & Education Quarterly, 11 (3), 131–152.
Mejía-Laguna, J. (2023). Critical Learning Episodes in the EFL Classroom: A Multimodal (Inter)Action
Analytical Perspective. Multimodal Communication, 12 (1), 23–44.
Mills, K. & Unsworth, L. (2017). Multimodal Literacy. Oxford Research Encyclopedia of Education. Retrieved
14 June 2024, from
[Link]
e-232
Mondada, L. (2019). Transcribing Silent Actions: A Multimodal Approach of Sequence Organization. Social
Interaction. Video-Based Studies of Human Sociality, 2 (1).
Mondada, L. (2014). Instructions in the Operating Room: How the Surgeon Directs Their Assistant's Hands.
Discourse Studies, 16 (2), 131–161.
Mondada, L. (2011). The Organisation of Concurrent Courses of Action in Surgical Demonstrations. In:
Jürgen Streeck , Charles Goodwin & Curtis LeBaron (Eds.), Embodied Interaction: Language and Body in
the Material World. Cambridge: Cambridge University Press.
Mondada, L. (2018). Multiple Temporalities of Language and Body in Interaction: Challenges for Transcribing
Multimodality. Research on Language and Social Interaction, 51 (1), 85–106.
Mondada, L. (2019). Contemporary Issues in Conversation Analysis: Embodiment and Materiality,
Multimodality and Multisensoriality in Social Interaction. Journal of Pragmatics, 145, 47–62.
Mondada, L. (2023). Demonstrating and Guiding How to Smell in Tasting Sessions: .nhHHHhh and the
Audible-Visible Production of Sensorial Intersubjectivity. Language & Communication, 88, 111–128.
Moschini, I. & Sindoni, M. (Eds.) (2022). Mediation and Multimodal Meaning Making in Digital Environments.
New York; London: Routledge.
Mora, R. , Tian, Z. & Harman, R. (2022). Translanguaging and Multimodality as Flow, Agency, and a New
Sense of Advocacy in and from the Global South. Pedagogies: An International Journal, 17 (4), 271–281.
Moretti, C. (2021). Sensate Memory: An Introduction to the Special Issue. Multimodality & Society, 1 (3),
245–265.
Moss, G. (2001). To Work or Play? Junior Age Non-Fiction as Objects of Design. Reading: Literacy and
Language, 35, 106–110.
Nevile, M. (2015). The Embodied Turn in Research on Language and Social Interaction. Research on
Language and Social Interaction, 48 (2), 121–151.
Newfield, D. (2014). Transformation, Transduction and the Transmodal Moment. In: C. Jewitt (Ed.), The
Routledge Handbook of Multimodal Analysis (2nd ed.). Milton Park, Abingdon, Oxon: Routledge, pp.
100–115.
Norris, S. (2004). Analyzing Multimodal Interaction: A Methodological Framework. New York: Routledge.
Norris, S. (2013). What is a Mode? Smell, Olfactory Perception, and the Notion of Mode in Multimodal
Mediated Theory. Multimodal Communication, 2 (2), 155–169.
Norris, S. (2020). Multimodal Theory and Methodology: For the Analysis of (Inter)action and Identity. London:
Routledge.
Norris, S. (2021). Multimodal (Inter)action Analysis in a Nutshell: Philosophy, Theory, Method and
Methodology. QUIVIRR, 2, 1–7.
O'Donnell, M. & Bateman, J. (2005). SFL in Computational Contexts: A Contemporary History. In: R. Hasan ,
C.M.I.M. Matthiessen & J. Webster (Eds.), Continuing Discourse on Language: Volume 1 (pp. 343–382).
London: Equinox.
O'Halloran, K.L. (1999). Towards a Systemic Functional Analysis of Multisemiotic Mathematics Texts.
Semiotica, 124 (1/2), 1–29.
O'Halloran, K.L. (2000). Classroom Discourse in Mathematics: A Multisemiotic Analysis. Linguistics and
Education, 10 (3), 359–388.
O'Halloran, K.L. (Ed.). (2004). Multimodal Discourse Analysis. London: Continuum.
O'Halloran, K.L. (2008). Inter-Semiotic Expansion of Experiential Meaning: Hierarchical Scales and Metaphor
in Mathematics Discourse. In: C. Jones & E. Ventola (Eds.), New Developments in the Study of Ideational
Meaning: From Language to Multimodality. London: Equinox, pp. 231–254.
O'Halloran, K.L. (2015a). The Language of Learning Mathematics: A Multimodal Perspective. The Journal of
Mathematical Behaviour, 40 (A), 63–74. [Link]
O'Halloran, K.L. (2015b). Mathematics as Multimodal Semiosis. In: E. Davis & P. J. Davis (Eds.),
Mathematics, Substance, and Surmise. Berlin: Springer.
O'Halloran, K.L. (2015c). Multimodal Digital Humanities. In: P. Trifonas (Ed.), International Handbook of
Semiotics. Dordrecht: Springer, pp. 383–409.
O'Halloran, K.L. (2021). Multimodal Discourse Analysis. In: K. Hyland , B. Paltridge & L. Wong (Eds.),
Bloomsbury Companion to Discourse Analysis (2nd ed.). London: Bloomsbury, pp. 249–266.
O'Halloran, K.L. (2023). Matter, Meaning and Semiotics. Visual Communication, 22 (1), 174–201.
[Link]
O'Halloran, K.L. , Chua, A. & Podlasov, A. (2014). The Role of Images in Social Media Analytics: A
Multimodal Digital Humanities Approach. In: D. Machin (Ed.), Visual Communication. Berlin: de Gruyter, pp.
565–588.
O'Halloran, K.L. , E, M.K.L., Podlasov, A. & Tan, S. (2013). Multimodal Digital Semiotics: The Interaction of
Language with other Resources. Text and Talk: Special Edition for Michael Halliday (edited by Geoff
Thompson), 33 (4–5), 665–690.
O'Halloran, K.L. , E, M.K.L. & Tan, S. (2014). Multimodal Analytics: Software and Visualization Techniques
for Analyzing and Interpreting Multimodal Data. In: C. Jewitt (Ed.), The Routledge Handbook of Multimodal
Analysis (2nd ed.). London: Routledge, pp. 386–396.
O'Halloran, K.L. & Lim, F.V. (2014). Systemic Functional Multimodal Discourse Analysis. In: S. Norris & C.
Maier (Eds.), Texts, Images and Interactions: A Reader in Multimodality. Berlin: Mouton de Gruyter, pp.
137–154.
O'Halloran, K.L. , Tan, S. & E, M.K.L. (2013). ‘Above All’: The Myth of ‘Dreams’ as Advertising Tool. In: B.
Pennock-Speck & M. M. d. S. Rubio (Eds.), The Multimodal Analysis of Television Commercials. Valencia:
Publicacions de la Universitat de València, pp. 113–135.
O'Halloran, K.L. , Tan, S. & E, M.K.L. (2014). A Multimodal Approach to Discourse, Context and Culture. In:
J. Flowerdew (Ed.), Discourse in Context. London: Bloomsbury, pp. 247–272.
O'Halloran, K. L. , Tan, S. and E, M.K.L. (2017). Multimodal Analysis for Critical Thinking. Learning, Media
and Technology, 42 (2), 147–170. [Link]
O'Halloran, K.L. , Tan, S. & Wignell, P. (2016). Intersemiotic Translation as Resemiotization: A Multimodal
Perspective. Signata, Special Issue on Translating Signs, Texts, Practices, edited by J. Fontanille , M.
Sonzogni & R. Troqe , 199–229.
O'Halloran, K.L. , Tan, S. & Wignell, P. (2019). SFL and Multimodal Discourse Analysis. In: G. Thompson ,
W. L. Bowcher , L. Fontaine & D. Schönthal (Eds.), The Cambridge Handbook of Systemic Functional
Linguistics. Cambridge, UK: Cambridge University Press, pp. 433–461.
O'Halloran, K.L. , Tan, S. , Wignell, P. , Lange, R. , Chai, K. & Wiebrands, M. (2022). Big Data and Managing
Multimodal Complexity. In: David Caldwell , John S. Knox & J. R. Martin (Eds). Appliable Linguistics and
Social Semiotics: Developing Theory from Practice. London: Bloomsbury Academic, pp. 157–179.
O'Halloran, K.L. , Wignell, P. & Tan, S. (2022). Big Data and Managing Multimodal Complexity. In: D.
Caldwell , J. S. Knox & J. R. Martin (Eds.), Appliable Linguistics and Social Semiotics: Developing Theory
from Practice. London: Bloomsbury.
O'Toole, M. (2004). Opera Ludentes: the Sydney Opera House at Work and Play. In: K. O'Halloran (Ed.),
Multimodal Discourse Analysis: Systemic Functional Perspectives. London: Continuum.
O'Toole, M. (2011). The Language of Displayed Art (2nd ed.). London; New York: Routledge.
Pahl, K. (1999). Transformations—Meaning Making in Nursery Education. Stoke-on-Trent: Trentham Books.
Pigg, S.L. & Kunwar, S. (2021). On the Roadside: Pangs of Memory, Tastes of Futures. Multimodality &
Society, 1 (3): 350–365.
Pollner, M. (1979). Explicative Transactions: Making and Managing Meaning in Traffic Court. In: Everyday
Language: Studies in Ethnomethodology, edited by G. Psathas . New York: Irvington, pp. 229–255.
Potter, J. (2024). Theory-Building in the Social, Material and Postdigital Worlds of Play: Participatory
Research and Multimodal Discourse Analysis. Multimodality & Society, 4 (2), 175–196.
Potter, J. , Olusoga, Y. & Cowan, K. (2023). The Play Observatory: An Observatory of Children's Play during
COVID-19. Sheffield: University of Sheffield.
Price, S. , Jewitt, C. , Chubinidze, D. , Barker, N. & Yiannoutsou, N. (2021). Taking an Extended Embodied
Perspective of Touch: Connection-Disconnection in iVR. Frontiers of Virtual Reality, 2.
[Link]
Prior, P. (2005). Moving Multimodality Beyond the Binaries: A Response to Gunther Kress’ ‘Gains and
Losses.’ Computers and Composition, 22, 23–30.
Prosser, J. , Clark, A. , Wiles, R. (2008). Visual Research Ethics at the Crossroads.
[Link]/publication/277803303_Visual_Research_Ethics_at_the_Crossroads
Rampton, B. , Bezemer, J. , Jewitt, C. & Lefstein, A. (2007). Illustrations of Linguistic Ethnography in Action:
Indicative Analyses of a Job Interview. Unpublished manuscript. [Link]
Ravelli, L.J. & McMurtrie, R.J. (2016). Multimodality in the Built Environment. London: Routledge.
Reershemius, Gertrud K. (2024). Leaving One's Mark: Self-Authorized Commemorative Practices in a Rural
Semiotic Landscape. Multimodal Communication. [Link]
Ridell, K. & Walldén, R. (2024). Prompting Story Elements in First Grade: An Intermodal Approach for
Exploring Two Teachers’ Orchestrations. Multimodality & Society, 4 (1), 29–57.
Riordan, P. , Revell, L. , Bowie, B. , Hulbert, S. , Woolley, M. & Thomas, C. (2024). Multimodal Classroom
Interaction Analysis Using Video-Based Methods of the Pedagogical Tactic of (Un)Grouping. Pedagogies:
An International Journal. [Link]
Rowsell, J. (2024). Affecting Photos: Photographs as Shared, Affective Ethnographic Spaces. Qualitative
Research, 0 (0).
Rowsell, J. , Kress, G. , Pahl, K. & Street, B. (2018). The Social Practice of Multimodal Reading: A New
Literacy Studies—Multimodal Perspective on Reading. In: Alvermann, D., Unrau, N., Sailors, M. & Ruddell,
R. (Eds). Theoretical Models and Processes of Literacy (7th ed.). London: Routledge.
Sacks, H. , E. Schegloff & G. Jefferson . (1974). A Simplest Systematics for the Organization of Turn-Taking
For Conversation. Language, 50, 696–735.
Sakr, M. , Price, S. & Jewitt, C. (2015). Exploring Whole-body Interaction and Design for Museums,
Interacting with Computers. [Link]
Samuelsson, R. (2022). Touch and Translanguaging in a Multilingual Early Childhood Education Setting.
Multimodality & Society, 2 (3), 300–322.
Saussure, F. de (1983 [1916]). Course in General Linguistics (translated by Roy Harris ). London:
Duckworth.
Sekimoto, S. & Brown, C. (2023). Race and Multimodality: An Introduction to the Special Issue. Multimodality
& Society, 3 (3), 199–209.
Serafini, F. & Reid, S. F. (2023). Multimodal Content Analysis: Expanding Analytical Approaches to Content
Analysis. Visual Communication, 22 (4), 623–649.
Schegloff, E. A. (2007). Sequence Organization in Interaction: A Primer in Conversation Analysis, Vol 1. New
York: Cambridge University Press.
Schegloff, E. & H. Sacks (1973). Opening Up Closings. Semiotica, 8, 289–327.
Scollon, R. & Scollon, S. W. (2003). Discourses in Place: Language in the Material World . London:
Routledge.
Selander, S. & Kress, G. (2021). Design for Learning: A Multimodal Perspective. Lund, Sweden:
Studentlitteratur AB.
Silver, C. & Lewins, A. (2014). Using Software in Qualitative Research: A Step-by-Step Guide (2nd ed.).
London: Sage.
Silverman, D. (1999). Warriors or Collaborators: Reworking Methodological Controversies in the Study of
Institutional Interaction. In: S. Sarangi & C. Roberts (Eds.), Talk, Work and Institutional Order. Berlin: Mouton
de Gruyter, pp. 401–425.
Sindoni, M. & Moschini, I. (Eds.). (2021). Multimodal Literacies Across Digital Learning Contexts. New York:
Routledge.
Staal, L. A. (2000). The Story Face: An Adaptation of Story Mapping that Incorporates Visualization and
Discovery Learning to Enhance Reading and Writing. The Reading Teacher, 54 (1): 26–31.
Stein, P. (2012). Multimodal Pedagogies in Diverse Classrooms: Representation, Rights and Resources.
London: Routledge.
Stöckl, H. (2023). Bold and Impactful: A Reappraisal of Gunther Kress's (Social) Semiotic Legacy in the Light
of Current Multimodality Research. Text & Talk, 44. [Link]
Street, B. V. , Pahl, K. & Rowsell, J. (2014). Multimodality and New Literacy Studies. In: Jewitt, C. (Ed.), The
Routledge Handbook of Multimodal Analysis (2nd ed.). Milton Park, Abingdon, Oxon: Routledge, pp.
227–237.
Stukenbrock, A. & Nhi Dao, A. (2019). Joint Attention in Passing: What Dual Mobile Eye Tracking Reveals
About Gaze in Coordinating Embodied Activities at a Market. In: E. Reber & C. Gerhardt (Eds.), Embodied
Activities in Face-to-Face and Mediated Settings. London: Palgrave Macmillan, pp. 177–213.
Suchman, L. (1987). Plans and Situated Actions: The Problem of Human Machine Communication.
Cambridge: Cambridge University Press.
Tan, L. , Zammit, K. , D'Warte, J. & Gearside, A. (2020). Assessing Multimodal Literacies in Practice: A
Critical Review of its Implementations in Educational Settings. Language and Education, 34 (2), 97–114.
Thompson, G. (2004). Introducing Functional Grammar. London: Arnold.
Thompson, G. , Bowcher, W.L. , Fontaine, L. & Schönthal, D. (Eds.). (2019). The Cambridge Handbook of
Systemic Functional Linguistics. Cambridge, UK: Cambridge University Press.
Townsend, L. & Wallace, C. (2017). The Ethics of Using Social Media Data in Research: A New Framework.
In: K. Woodfield (Ed.), The Ethics of Online Research (Advances in Research Ethics and Integrity, Vol. 2).
Leeds: Emerald Publishing Limited, pp. 189–207.
Tuan, Y.-F. (1991). Language and the Making of Place. Annals of the Association of American Geographers,
81, 684–696.
Unsworth, L. (2007). Image/text Relations and Intersemiosis: Towards Multimodal Text Description for
Multiliteracies Education. In: L. Barbara & T. Sardinha (Eds.), Proceedings of the 33rd International Systemic
Functional Congress. Presented at the 33rd International Systemic Functional Congress. Sao Paulo, Brazil:
PUCSP, pp. 1165–1205.
Unsworth, L. (Ed.) (2008). Multimodal Semiotics: Functional Analysis in Contexts of Education. London:
Continuum.
Unsworth, L. & Cleirigh, C. (2014). Multimodality and Reading: The Construction of Meaning through Image-
Text Interaction. In: C. Jewitt (Ed.), The Routledge Handbook of Multimodal Analysis (2nd ed.). London; New
York: Routledge, pp. 151–163.
van Leeuwen, T. (1991). Conjunctive Structure in Documentary Film and Television. Journal of Media and
Cultural Studies, 5 (1), 76–114.
van Leeuwen, T. (1999). Speech, Music, Sound. London: Macmillan.
van Leeuwen, T. (2005). Introducing Social Semiotics. London: Routledge.
van Leeuwen, T. (2008). Discourse and Practice: New Tools for Critical Discourse Analysis. Oxford: Oxford
Univerity Press.
van Leeuwen, T. (2022). Multimodality and Identity. London: Routledge.
van Leeuwen, T. & Selander, S. (2024). The Semiotics of Toys and Games. London: Bloomsbury.
Varvantakis, C. & Nolas, S. M. (2019). Metaphors We Experiment with in Multimodal Ethnography.
International Journal of Social Research Methodology, 22 (4), 365–378.
Vatanen, A. (2023). Embodied Noticings As Repair Initiations: On Multiactivity in Choir Rehearsals. In: Pentti
Haddington , Tiina Eilittä , Antti Kamunen , Laura Kohonen-Aho , Iira Rautiainen & Anna Vatanen (Eds.),
Complexity of Interaction: Studies in Multimodal Conversation Analysis. London: Palgrave Macmillan, pp.
99–141.
Ventola, E. , Charles, C. & Kaltenbacher, M. (Eds.). (2004). Perspectives on Multimodality. Amsterdam: John
Benjamins.
Ventola, E. & Moya, J. (Eds.). (2009). The World Told and the World Shown: Multisemiotic Issues.
Hampshire: Palgrave Macmillan.
vom Lehn, D., Gibson, W. & Ruiz-Junco, N. (Eds.). (2023). People, Technology, and Social Organization:
Interactionist Studies of Everyday Life. London: Routledge.
Wei, L. (2022). Translanguaging, Multimodality, Southern Theory, and Pedagogical Possibilities.
Pedagogies: An International Journal, 17 (4), 408–412.
Wildfeuer, J. & Lehmann, C. (2024). Editorial: Drawing Multimodality's Bigger Picture: Metalanguages and
Corpora for Multimodal Analyses – in lieu of a Festschrift for John A. Bateman. Frontiers in Communication,
9, 1436821.
Wittenburg, P. , Brugman, H. , Russel, A. , Klassmann, A. & Sloetjes, H. (2006). ELAN: A Professional
Framework for Multimodality Research. In: Proceedings of LREC 2006, Fifth International Conference on
Language Resources and Evaluation, May 22–28. Genoa, Italy, pp. 1556–1559.
Wodak, R. (2006). Critical Linguistics and Critical Discourse Analysis. In: J. Verschueren & J.-O. Östman
(Eds.), Handbook of Pragmatics. Philadelphia; Amsterdam: J. Benjamins, pp. 1–24.
Wohlwend, K. (2012). The Boys Who Would Be Princesses: Playing With Gender Identity Intertexts in
Disney Princess Transmedia. Gender and Education, 24 (6), 593–610.