Introduction to Multimodal Analysis
What is Multimodal Analysis?
Multimodal analysis is a theory of visual communication that connects how society and power
operate with the way visual materials convey ideas, values, and structure our actions and
interactions. It focuses on how different elements of communication are used in increasingly
integrated ways.
Key Concepts in Multimodal Analysis
Social Semiotics
Inspired by Halliday's linguistic theory, social semiotics views communication as a "social
semiotic" system. It differs from traditional semiotics (like Barthes') by:
Systematic Analysis: It aims for a more systematic analysis of the entire composition, not
just individual symbols or connotations.
Holistic Approach: Similar to linguistics, it examines how meaning is created through the
combination and placement of elements, rather than just the isolated meaning of one
element.
Semiotic Choices
Communication involves making deliberate choices from available options. These choices,
regarding elements like fonts, colors, layout, and space, shape the meaning conveyed to a
specific audience within a particular socio-historical context.
Meaning Potential: Every semiotic resource (e.g., font thickness, color saturation) has a range
of potential meanings. The analysis involves understanding these potentials and identifying
the specific choices made.
System of Choices: In language, we choose words based on context and desired outcome.
Similarly, in visual communication, designers make choices from a "system of choices"
available for each semiotic resource.
Inventories: A key method in social semiotics is creating inventories of available choices for
specific semiotic resources (e.g., ways to indicate levels of certainty in language). This helps
analysts identify and interpret the choices made in practice.
Semiotic Materials and Affordances
Semiotic Materials: These are the actual "things" we encounter and interact with, like
websites, advertisements, or physical objects. We experience them as whole entities, not just
a collection of individual choices.
Affordances: Semiotic materials come "loaded" with ideas, assumptions, and potentials that
shape communication and social behavior. For example, packaging influences how we
communicate about food and ethics. The design of a café shapes how people behave and
interact within it.
Discourse
Discourse refers to socially constructed knowledge, ideas, and ways of understanding how the
world works and how people behave.
Transport of Meaning: Semiotic materials and choices can "connote" complex ideas about
the world. For instance, a flag can connote nationalism, patriotism, or even racism, depending
on its context and interpretation.
Socially Constructed Knowledge: Discourses are not inherent but are built through social
interaction and communication. They can be promoted through various semiotic materials
like monuments, advertisements, and buildings.
Power Interests: Discourses often reflect and serve particular power interests within society.
For example, war monuments can promote a discourse of national pride and sacrifice that
masks underlying political or economic motives.
Social Practices
Social practices are the "doings" and organizational forms in society that are influenced by
discourses. Semiotic materials play a role in shaping these practices, influencing how we act,
what is considered acceptable, and the ideas we associate with our actions.
The Shift to Multimodality
Communication is increasingly becoming multimodal, meaning different semiotic modes (like
text, image, sound, gesture) are used in combination.
From Monomodality to Multimodality: Previously, modes were often used in isolation
(monomodality). Now, there's a greater integration of different modes, seen in schoolbooks,
magazines, and bills.
Technologization: Advancements in technology have made it easier to combine and
manipulate various semiotic resources (text, image, graphics). This allows for precise control
and shaping of communication to meet economic, political, or institutional objectives.
Integrated Design: In multimodal communication, different semiotic resources are used
together in systematic ways to convey meaning. Causalities, identities, and connections that
were previously explained in running text are now communicated through combinations of
text, images, graphics, and colors. This "new writing" often leaves the explicit explanation of
how elements connect to the viewer.
New Commodified Communication Setting Moral
Standards
Commodification: Aspects of life, including ideas like health or ethics, are increasingly coded
as commodities that can be packaged and sold. This can shape how we think and act, but the
underlying complexities may be obscured.
Neo-liberalism: The current dominant model of governing society, which runs on market
principles, drives commodification and codification. This leads to measuring and comparing
aspects of life, like education, based on performance metrics.
Moral Dimension: Marketing often incorporates moral values, framing consumption as an
ethical act (e.g., Fairtrade, sustainability). This can colonize social relations, with consumer
culture taking over roles traditionally held by politics.
Communicative Rationality vs. Instrumental Rationality:
Communicative Rationality: Striving for mutual understanding through comprehensible
language, where participants are truthful and aim for agreement.
Instrumental Rationality: Focusing on achieving specific outcomes or goals, often through
efficient means.
Integrated designs can obscure communicative rationality by abstracting and fragmenting
everyday experiences, potentially leading to "violent abstractions" where instrumental
rationality is no longer balanced with communicative understanding.
Coding of Semiotic Resources
The meaningfulness of semiotic resources is established through three principles: distinctions,
contextualizations, and configurations.
Distinctions
Categorization: All codes rely on making distinctions. For example, the continuous spectrum
of color is divided into named colors.
Meaningful Differences: These distinctions allow us to identify and interpret differences in
meaning. A pale yellow might mean something different from a rich, saturated yellow, based
on the distinction between muted and intense experiences.
Contextualizations
Context-Dependent Meanings: The exact meanings of distinctions are determined by their
context. The meaning of a color like red can vary significantly depending on the semiotic
material and social context (e.g., a baby rattle vs. a bank logo).
Domain Relevance: Certain distinctions become relevant in specific social contexts or
domains. Shapes in flow charts have specific meanings that differ from their use in graphic
design or other contexts.
Configurations
Co-articulation: Semiotic resources are used together in configurations, where their meanings
are understood in relation to each other and the overall design.
Established Patterns: These configurations are often conventionalized within specific
domains and relate to established social practices. The meaning of colors in a café is
understood in relation to its furnishings and textures. The meaning of a bar chart is
understood through the configuration of shapes, numbers, and axes.
Identifying Codes
Two methodological principles are used to identify codes: inventories and the commutation test.
Inventories
Systematic Sampling: Collect a systematic sample of the semiotic material or resource being
researched.
Pattern Exhaustion: Aim to create an inventory that feels comprehensive, allowing for the
identification of recurring patterns and distinctions.
Commutation Test
Changing Elements: Systematically change one element within a multimodal configuration
(e.g., change the typeface, color, or shape).
Meaning Shift: Observe if changing an element alters the overall meaning. If it does, it
indicates that the changed element is significant and contributes to the meaning. For
example, if changing a font on healthy food packaging changes its perceived meaning, the
font choice is a meaningful distinction.
Summary of Key Concepts
Semiotic Choice: Communication involves selecting from available options to shape
meaning.
Semiotic Material: The actual objects and things that communicate meaning, shaping social
organization and interactions.
Material Affordances: The inherent communicative aims and evolved potentials of semiotic
materials.
Discourses: Socially constructed ideas about how the world works, reflecting power interests
and embedded in our environments.
Social Practices: Societal actions and organizations based on discourses, shaped by
semiotic materials.
Technologization: The systematic coding and use of communication resources, driven by
marketization, commodification, and technology, for specific objectives.
Integrated Design: The systematic combination of semiotic resources, where meaning is
conveyed through interrelations rather than explicit explanation.
Communicative Rationality: The striving for mutual understanding through clear, truthful
communication.
Distinctions: The basic divisions made within semiotic resources (e.g., colors, shapes) that
enable coding.
Contextualizations: How the specific meanings of distinctions are determined by their social
and semiotic context.
Configurations: The way semiotic resources are combined and used together in established
patterns within specific domains.
Inventories: Method for systematically sampling and documenting semiotic materials to
identify patterns.
Commutation Test: A method of changing elements in a configuration to test their impact on
meaning and identify significant distinctions.