MULTIMODALITY
What is Multimodality?
Multimodality in Discourse
• Spoken Discourse; • Written Discourse;
• Intonation • Typographic
• Voice quality expression
• Illustration
• Facial expression • Layout
• Gesture • Colour
• Posture
• Dress
• Hairstyle
What do you think about the
multimodality in these covers?
What do their faces say?
History of Multimodality
• Prague School; (see Garvi1n 1964; Matˇejka and Titunik 1976)
• Mathesius; (see Garvin 1964: 159)
• Veltrusky; (Veltrusk´y 1964: 91)
• Bogatyrevs; (1971)
• Paris School;
• Barthes; (e.g., 1967, 1977, 1983)
• Getty Images
What do these pictures mean to you?
• American linguists’ studies;
Multimodal transcription;
Analyzing of Multimodality
• Visual Grammar
• Multimodality and Genre
• Applications
Visual Grammar
• “meanings belong to culture rather than to
specific modes”
• There is a difference between expressing the
“same” meaning visually or verbally.
• ‘modal material’ is historically and culturally
specific and subject to change.
• The affordances of different modes … have
profound effects on that which is to be realized
in the mode. This is the insight gained from
the “linguistic turn” of the 1970s which
showed that language was not a neutral vehicle
for representation. All modes have that effect.
Knowledge changes in shape when it is
realized in the different modal material. (Kress
2003: 50)
Kress and van Leeuven’ grammar of
visual design;
• Two pictures from Autralian primary school
social studies text book.
• First caption: “The settlers gun-stalked the
Aboriginals.” (The British used guns)
• Second caption: “Axe, basket and wooden
sword.”
• The concept was comparing “technology” in
both pictures.
Arrow vectors;
Narrative Person;
• Some semiotic models aren’t easily described
by means of system network like;
– Voice Quality,
• pitch range / loudness / vocal tension / roughness /
breathiness / vibrato
• Experiential Metaphor;
– “No metaphor can ever be comprehended or even adequately
represented independently of an experiential basis.”
Multimodality and Genre
• Genre has played an important role in
multimodal discourse analysis.
• Each genre has different features in it.
– A recipe;
What do this picture and layout tell you?
What could be the
setting/feelings/attitudes?
What could be the aim of this picture?
• Colour?
• Type of car?
• Slogan?
Let’s talk about the design of a webpage
Applications
• Multimodal analysis has been applied in a
number of fields;
– Education
– Media discourse
– Organization studies
– Health communication
Education;
• New London Group’s application of
multimodal analysis to education;
– studies of the development of literacy in young
children,
– studies of multimodal learning resources,
including textbooks,
– studies of multimodal classroom interaction.
The development of literacy in young children;
• Gunther Kress’s three-year-old child investigation
on drawing a car with circles and blood circulation
with high school students.
– As children are drawn into culture, “what is to hand” becomes
more and more that which the culture values and therefore makes
readily available. The child’s active, transformative practice
remains, but it is more and more applied to materials which are
already culturally formed. In this way children become the
agents of their own cultural and social making. (Kress 1997: 13)
Multimodal learning resources, including
textbooks, the Internet;
• Jewitt (2006) studied how children use
computer games to learn science, struggling to
compromise the rules of the game with their
everyday experiences. (bouncing with arrows)
Multimodal classroom interaction;
• Studies of classroom interaction have also
moved from the traditional emphasis on
linguistic exchange structures to strong
contextualization and detailed attention to
nonverbal communication and setting.
Media Discourses
• Conceptual images of this kind are produced to fit into
multimodal designs;
• using restricted color palettes,
• leaving space for words,
• generic rather than specific,
• using a range of decontextualizing devices,
• a restricted vocabulary of attributes to indicate the
identity of people and places,
• a sign that the language of the visual is tightening into
vocabularies of pictograms and logos.
Conclusion
• Multimodality is seen as a key toward better
learning, with different modes enabling the
representation of different aspects of, and
perspectives on, the objects of the learning.
Visual racism
Multimodality in CDA in relation to
nonverbal communication:
• Communicative style is a matter of language in the
broadest sense – certainly verbal language (words),
but also all other aspects of the complex bodily
performance that constitutes political style
(gestures, facial expression, how people hold
themselves and move, dress and hairstyle, and so
forth). A successful leader’s communicative style is
not simply what makes him attractive to voters in a
general way, it conveys certain values which can
powerfully enhance the political message.
For further studies;
• in relation to the kinds of normative discourses that govern
contemporary uses of semiotic resources,
• to the way in which multimodality has allowed the
development of new affective and poetic semiotic resources,
• in relation to the complexities of contemporary data
visualization,
• increasingly important role of digital semiotic technologies
in contemporary communicative practices.
• the interaction between global trends and local traditions
• initiatives will become an increasingly important field of
research
Reference
• Leeuwen, T. V. (2015). Multimodality. In The
handbook of discourse analysis, Second
edition, Deborah Tannen, Heidi E. Hamilton,
and Deborah Schiffrin (Eds.), pp. 447-465.
UK: John Wiley & Sons, Inc.