See discussions, stats, and author profiles for this publication at: [Link]

net/publication/370120768

A social semiotic multimodal approach to translation

Chapter · April 2023

DOI: 10.4324/9781003161448-23

CITATIONS READS

6 1,866

1 author:

Elisabetta Adami
University of Leeds
77 PUBLICATIONS 1,632 CITATIONS

SEE PROFILE

All content following this page was uploaded by Elisabetta Adami on 23 May 2023.

The user has requested enhancement of the downloaded file.

Adami, E. (2023) A social semiotic multimodal approach to translation. In K. Marais and R. Meylaerts
(eds.) Routledge Handbook of Concepts and Theories of Translation. London: Routledge. 369-388

A Social Semiotic Multimodal Approach to Translation

Elisabetta Adami, University of Leeds

[Link]

Abstract
The chapter introduces a Social Semiotic perspective on translation. It starts by presenting
key tenets of Social Semiotics, the approach that first theorised the concept of multimodality.
Then it discusses the relation between multimodality and translation, as well as the
challenges faced by translation studies and practice in the contemporary semiotic landscape.
It finally introduces key Social Semiotic concepts and frames them as dimensions of
translation, conceived as process of resemiotisation at the intersections between modes,
media, genres, domains, semiotic spaces and discourses. The Social Semiotic approach
sketched in the chapter provides a perspective on sign- and meaning-making that can be
useful in re-framing translation in its scope, possibilities and applications.

Introduction

The chapter introduces Social Semiotics as a theoretical perspective that has first developed
the concept of multimodality and provided heuristic tools to understand sign-making and
meaning-making, i.e., how we make signs and sign complexes by combining different
semiotic resources and how we make meaning out of anything produced by others and
present in our environments. By considering how we make signs and meanings with more
than language, we can question established assumptions on what translation is. This is
particularly useful today, given the complexities of contemporary communication and the
challenges faced by Translation Studies and translation practice. Social Semiotics provides a
theoretical perspective and conceptualisation of sign- and meaning-making that can be useful
in re-framing possible approaches to translation.

Social Semiotics is a theoretical perspective that has first introduced the concept of
“multimodality” (Kress & van Leeuwen, 2001), and has proposed a take onto
communication, social interaction and semiosis that accounts for the fact that humans make
meaning by combining multiple semiotic resources, or modes. In recent years Translation
Studies – along with much research in linguistics as well as education and other disciplines in
the humanities and social sciences – have been undergoing a “multimodal turn” (Adami &
Ramos Pinto, 2020; Boria & Tomalin, 2020), stemming from the recognition that
communication is multimodal and humans make meaning with more than language. This has
helped the field to draw attention to non-linguistic aspects of texts, such as images and layout
in static texts (e.g. in comics, Borodo, 2015), and moving images in audio-visual texts (e.g.
for subtitling, Taylor, 2004). Yet most studies considering multimodal aspects of translation
focus on interlingual contexts, i.e., from a source named/national language1 or
“languaculture” (Agar, 1995) to another. Thus, even when expanding the attention to
translation resources other than speech and writing, an overall logocentric and interlingual
definition of the contexts and needs for translation still remains (Adami & Ramos Pinto,
2020; Marais, 2019). If instead we shift from a linguistic to a Social Semiotic perspective,
 Page 2 of 22

and consider meaning making and re-making taking place in and across all modes, media,
genres, domains, contexts (or semiotic spaces), and discourses, we can conceive of
translation as resemiotisation (Iedema, 2001), that is how “semiotics are translated from one
into the other as social processes unfold” (Iedema, 2003, p. 29). Translation then becomes
“transposition of meaning” (Kress, 2020) across modes, media, genres, domains, semiotic
spaces and discourses regardless of whether the re-making of meaning involves addressees
that speak a different named/national language. As a consequence, a wealth of questions and
possibilities open as to how we should or could re-conceptualise translation contexts and
practices. This is particularly relevant in the light of the complexities of the contemporary
communication landscape, characterised by increased transnational circulation of people,
multimodal artefacts and semiotic practices as well as an increased multiplication of equally
transnational yet fragmented and connected semiotic spaces, which redefine boundaries of
sharedness and non-sharedness in semiotic knowledge (Adami & Ramos Pinto, 2020).

The present chapter tries to unpack the above by introducing a Social Semiotic perspective on
translation. It first presents key tenets of Social Semiotics. It then introduces key concepts
and frames them as dimensions of translation, conceived as process of resemiotisation at the
intersections between modes, media, genres, domains, semiotic spaces and discourses.

Social Semiotics and Multimodality

Social Semiotics developed in the works of Robert Hodge, Gunther Kress and Theo van
Leeuwen (for key references see Hodge & Kress, 1988; Kress, 2010; van Leeuwen, 2005a). It
stemmed from Michael Halliday’s functional linguistics, chiefly in his work “Language as a
Social Semiotic: The social interpretation of language and meaning” (Halliday, 1978).
Against generativist and structural approaches, which explained language on the basis of
either mind-internal or system-internal rules and processes, Halliday pointed to social
dynamics as the reason for how language and its resources are as they are at any given time
and in any social group. Humans develop their linguistic resources to respond to the social
functions they need to fulfil. Social groups’ communicative functions, including establishing,
maintaining and contesting power relationships, explain levels of variation in language
choices and use. Language represents society and contributes to shape it – for example, how
it categorises power relations among speaker and listener (i.e., through systems of deference
or distance that languages might express through pronouns, morphological traits, and/or
honorifics) both represents and shapes their roles in the communicative event and in their
societies.

Social Semiotics has expanded this approach from language to all semiotic resources. It has
drawn on Halliday’s concept of “mode” (which he used to distinguish between speech and
writing) to label all socially-developed organised sets of resources for meaning making,
including speech, writing, image, gaze, face expression, music, proxemics, colour, dress,
architecture and so on. Starting from Kress and van Leeuwen’s (1996) seminal work on the
analysis of texts combining writing and image, multimodal analysis frameworks have been
developed through different theoretical perspectives (for a review Adami, 2017; for an
introduction Jewitt, Bezemer, & O’Halloran, 2016), to investigate meaning made by the
combined use of different semiotic resources (or modes) both in disembodied representations
and in embodied interaction. Research in multimodality has led to increased attention to
resources other than language and their role in meaning-making, and to questioning a
supposed primacy of language as paradigmatic to understand the whole of communication.
Page 2 of 22
 Page 3 of 22

While other theoretical perspectives in the field of Multimodal Studies (K. O’Halloran &
Smith, 2011) have taken a more systemic approach from Halliday’s Systemic Functional
Grammar (Halliday, 1985; for a systemic functional approach to intersemiotic translation see
K. L. O’Halloran, Tan, & Wignell, 2016), aimed at investigating the “grammatics” (K. L.
O’Halloran et al., 2016, p. 204) of semiotic systems from their use in multimodal texts,
Social Semiotics has instead taken chiefly Halliday’s social stance to language, to focus on
instances of sign- and meaning-making as revelatory of the sign- and meaning-maker’s
interest, and hence their positioning in relation to broader social dynamics.

This focus on the relation between social dynamics and individual agency is both a
theoretical approach and a call to respond to the complexities of contemporary
communication. We live in and make meaning of a world that is characterized by instability,
fragmentation and provisionality, with an increased circulation of people, multimodal texts
and semiotic practices, combined with a multiplication of interconnected and yet fragmented
semiotic spaces both online and offline, and we do this in interaction with others (as well as
with machines). Rather than systems and rules of usage, investigating the processual and
emerging aspects of sign- and meaning-making, and what they reveal of people’s positioning
in relation to social dynamics seems not only more feasible but also more useful to help
navigating such a complex, variable and ever-changing landscape and ecology.

- Mode as a heuristic tool; mode vs code

Although in academic fields outside Multimodal Studies, modes are often thought of chiefly
in terms of image and writing, or speech and gesture, multimodal analysis has developed
frameworks to research also semiotic resources such as sound and music (van Leeuwen,
1999), colour (Kress & van Leeuwen, 2002), architecture and space in the built environment
(Ravelli & McMurtrie, 2018), dance (Maiorani, 2021), touch (Bezemer & Kress, 2014;
Jewitt, Price, Mackley, Yiannoutsou, & Atkinson, 2020), typography (van Leeuwen, 2005b),
and texture (Djonov & van Leeuwen, 2014), for example. Most importantly, beyond the
investigation of specific modes in isolation, multimodal analysis concerns meaning that is
produced through the relation of different modes in texts and communicative events.

Mode is a socially developed organised set of resources to make meaning; it is material that
has been socially developed to make meaning. Debates on what constitutes a mode and how
to categorise different modes abound in Multimodal Studies (see for example Bateman, 2011;
Stöckl, 2009). In Social Semiotics, the concept of mode is a heuristic and is not predefined in
its boundaries. Given the “social” as the primary principle driving semiotic dynamics, what
constitutes a mode is socio-culturally determined in that it varies across social groupings and
their specific meaning-making needs (Bezemer & Kress, 2008). Wine can be considered a
mode among wine makers and tasters, as they have developed its resources to express a full
range of meaning potentials (Adami, 2017); the same is for fonts for typesetters, perfume for
perfume designers and so on. Boundaries between modes are also not clear-cut. For example,
by relying both on the visual channel of perception, writing and image share the potential of
blurring their boundaries in semiotic properties; so, graphic designers and graffiti artists, for
example, often exploit and develop the visuality of writing and blend letters with shapes and
other resources of images. In sum, mode is a flexible label useful if conceived as a heuristic
for analysis, rather than as a set in stone categorisation that reifies and essentialises semiotic
resources and the many uses that social groups make of them.

Page 3 of 22
 Page 4 of 22

“Mode” in social semiotics is preferred to the concept of “code” used in traditional semiotic
approaches (as well as in linguistics, e.g., in “code-switching”), as it allows for a less stable,
fixed and rule-determined approach to communication and sign- and meaning-making. Social
semiotics moves away from models of communication that explain it in terms of coding and
decoding processes, according to which successful communication is a matter of using signs
by applying the rules of a shared code in a specific context. Rather than codes and rules,
Social Semiotics instead refers to modes or semiotic resources and regularities, in terms of
communalities in patterns of form-meaning association emerging and traceable in specific
social spaces, which reflect hegemonic social dynamics – with exceptions to these
regularities being equally relevant for the understanding of meaning making in actual
instances of communication. Rather than sign use/usage, Social Semiotics conceives of
communication in terms of sign-making.

- Sign-making and Meaning-making2

In Social Semiotics, particularly in Gunther Kress’ (2010) conceptualization, signs are not
used. Signs are made every time somebody chooses a semiotic resource (some material form)
to express selected aspects (i.e., aspects that the sign maker considers as “criterial”) of the
meaning which they wish to express. They do so by choosing the most apt resource (or
form/signifier) that is available to them to express that meaning (or signified). Aptness is in
relation both to the materiality and the (sign-maker’s experience of) past uses of that specific
resource. Analogously, in “meaning-making”, somebody makes meaning of a form they
encounter on the basis of criterial aspects of its materiality and (the meaning-maker’s
experience of) its past uses.

In this perspective, “past uses” is a label that, compared to “conventions”, allows for a less
stable, fixed and rule-governed take on regularities and dynamics of sharedness in semiotic
practices. It is preferred over conventions also because, in this approach onto signs and
meanings being newly made in every instance of communication, the association between
form and meaning is never arbitrary but motivated (Kress, 1993). In this sense, Kress’ social
semiotics stresses the relation between the social and the agency of sign- and meaning-
makers, and, in Saussurean terms, shifts its focus of attention from the langue to the parole. It
does so also by reviewing Saussure3 in the light of Peirce’s (1931) tripartite conception of
sign-object-interpretant, thus stressing the interpretant as a key element in all signification.

- Sign as motivated; meaning-making as hypothesis making

Any act of sign-making and meaning-making is the result of a process of making hypotheses
on the most apt association between forms and meanings, motivated on the basis of the
materiality and past uses of semiotic resources. This is so in sign-making, as I choose the
form that is most apt to express (criterial aspects of) the meaning I want to make, in relation
to my interest, aims and (my hypotheses onto) the meaning-making universe of my
addressees. It is also the case in meaning-making, as I make meaning of somebody else’s sign
by making hypotheses on the basis of the materiality of the form used and (my experience of)
its past uses; by doing so, I make hypotheses on the interest (in terms of overall social
positioning and specific positioning in the situation at hand) of the sign-maker who produced
the sign. Hence, in a Social Semiotic perspective, there is no fixed (or established) meaning,
but always a meaning potential.

Page 4 of 22
 Page 5 of 22

Meaning (i.e., what one wants to communicate) is prior in driving forms of expression, in that
forms develop as choice options on the basis of the meanings that need to be made. At the
same time, whenever there is choice among forms, there is meaning potential. As an
example, since when our mobile phones have introduced “heart” emojis in different colours,
difference in colour has become a potentially signifying resource, as I have personally
learned through discussions with my students. Some of them would use a red heart emoji
only with their romantic partner, to signify “love”, while they would use a yellow one with an
acquaintance, a purple one with their (women) friends, and a green one with mixed gender
groups of friends. Other students in my classroom would make a different meaning out of
different coloured heart emojis, instead. What matters is that when options among different
forms exist, the choice we make is potentially meaningful. Choice among possibilities in
semiotic resources is what has meaning potential. And, as emerged in my class discussion on
colour of the heart emoji, we make meaning by making hypothesis on others’ uses of
resources on the basis of their materiality and our (experience of) past uses. In that discussion
those students (and myself) who normally would not intentionally make meaning through
choice in colour in that emoji, then realised that our addressees might well have instead made
meaning out of our own use of colour, always in a motivated way through their experience of
past uses and materiality.

- Three levels of meaning (or meta-functions)

While discussing differences in my class on the use of colour as a resource to make meaning
with heart emojis, we were able to see how each student drew both on the materiality and on
their experience of past uses of these colours (including immediate past uses, as in the colour
used by others in that or other interactions) to associate them to different types of affection
and appreciation for different types of relationships in their networks. Colour in the heart
emoji would be used to make specific ideational and interpersonal meanings. Always
drawing on Halliday, Social Semiotic multimodality uses a three-layered system to analyse
meaning. Modes have resources to (1) express something about the world, i.e., the “ideational
metafunction” (in the heart emoji case, the type of affection expressed by the colour red as
“romantic love” vs yellow as “friendly love”); (2) express something about the producer and
the addressee of the message, and their relation, i.e., the “interpersonal metafunction” (the
type of relation with the addressee expressed by the specific coloured heart emoji, i.e., “my
relation with you is romantic” vs “we are friends”); and (3) express relations within the text,
such as cohesion, information structure etc., i.e., the “textual metafunction” (the relation of
meaning that the coloured heart emoji establishes with its co-text in the interaction as
“approval”, “solidarity”, “sympathy”, or “encouragement” etc.).

When I met a neighbour while climbing the stairway that leads to my flat, I smiled and turned
my back at him. I chose to do so because the materiality of turning the back helps distancing
their face from mine and hence, while we have been undergoing the Covid-19 pandemic, in a
space such as a stairway that does not allow for social distancing, turning the back mitigates
the risks of possible contagion. Hence, by virtue of its materiality, I associated that body
movement, in that specific situation, with a meaning of respect and thoughtfulness towards
my neighbour. Yet I also hypothesised that my neighbour’s experience of past uses of turning
the back could associate it instead with a meaning of avoidance and hence possible rudeness
(again, historically developed with that meaning because of its materiality, in that you hide
your face from the other hence refuse to interact). Ideationally, the turning of the back
signifies “avoidance of contact”; interpersonally it can signify either “I do not want to
interact with you; I disrespect you in not engaging with you” or “I care for you and want to

Page 5 of 22
 Page 6 of 22

avoid any possible contagion risk in case I am positive”. By smiling at the neighbour before
turning, I hypothesised that, at the textual level of meaning, the relation with the smile could
help my neighbour disambiguate which of these meaning potentials my turning of the back
had in my making of the sign.

- Communication as interpretation

Communication happens when there is interpretation (Kress, 2010). Stemming from Roland
Barthes’ Death of the Author (Barthes, 1977), communication is not defined by a successful
decoding of the producer’s intended meaning (whatever that could ever mean), but by
interpretation, any sort of interpretation. Sign-making is a punctuation of the ceaseless
process of semiosis (Kress, 2010), of which it leaves some traces. These traces enable us to
make some hypotheses about it, and hence try to reach some level of understanding.
Following this view, and combining it to recent complexity approaches to language (Larsen-
Freeman, 2015), I conceive of meaning-making as always processual, situated, dynamic,
relational and emergent in actual instances of interaction with others, with texts and the
environment.

In a research project I was involved in, we collected data on communication practices in the
superdiverse context of Kirkgate market, in the city centre of Leeds, UK (Adami, 2018). A
butcher had placed a lucky cat in his shop window. In an interview with him it emerged that
he had done so to attract the attention of potential Chinese customers; he had placed the lucky
cat on the window close to pork belly because in his experience Chinese customers use to
buy that cut of meat. He made a sign out of an object, by associating the signifier “lucky cat”
to meaning along all three metafunctions: 1) ideationally, he associated it to a meaning
“Chinese”, out of its materiality – the object has Chinese scripts on, for example – and his
experience of past uses, e.g., contexts where he had seen the object, including shops that he
would interpret as Chinese as well as media products such as films; 2) interpersonally, to
address Chinese customers – the materiality of the object, with the cat looking in front of
itself, and its arm moving up and down in a form of salute, addresses directly the viewer, thus
calling for attention; and 3) textually, through positioning it in front of the window shop to
achieve salience and through proximity with the belly pork, just behind it on the shelf, to
point deictically their attention to that cut of meat. Thus, he made hypotheses on the lucky cat
“speaking” to Chinese customers and on its proximity to the pork belly as directing their
attention to it. During our fieldwork we did not verify how potential Chinese customers
interpreted that sign (most likely we would have found very different interpretations among
them). I, however, had interpreted it differently before hearing the butcher’s account. I had
recently arrived from Italy to live in the UK; the Brexit referendum had just taken place, and
had made me particularly aware of possible anti-immigration stances among British citizens.
The butcher had also a Union Jack (i.e., the British national flag) on the banner of his shop,
which he used to signify the tradition of his activity and the provenance of the produce he
sold. Because of my own social positioning, I had become alerted whenever encountering a
national flag of a possible nationalist and anti-immigrant meaning association. When I saw
the lucky cat, I interpreted it as a sign of openness towards foreign cultural practices, thus
mitigating that possible association. So my meaning making of the lucky cat as sign was,
ideationally “not-British”, interpersonally “this butcher is open to migrants, hence I as a
migrant am welcome here”, and textually, in relation to the Union Jack “this flag is not meant
as a nationalist and anti-immigrant sign”.

---

Page 6 of 22
 Page 7 of 22

To sum up, in a Social Semiotic perspective, we communicate by combining different

semiotic resources, we use a form/signifier and associate it with a meaning/signified, in
making a new sign. We orchestrate our communication through sign complexes that make
meaning through the intertwined combination of different resources in different modes. We
do this in relation to our interest, both in terms of our social positioning and of our specific
interest at hand at our moment of sign-making, and in relation to our assumptions (or
hypotheses) on the meaning making of our addressees. Meanings can be analysed through
their intertwined three-layered function, i.e., ideationally, for what they represent about the
world, interpersonally, for what they express about the relation between the sign-maker and
their addressee, and textually, for what they express in relation to other signs in the same text
or communicative event. Any interpretation of meaning, both in analysis and in everyday
communication, is always a matter of hypotheses, informed by the materiality and (our
experience of) previous uses of the resources chosen to make signs. This does not mean that
regularities cannot be traced, as the reflection of people’s associations with broader social
forces, in terms of hegemonic ways of doing things, yet these are – also in analysis – always
in terms of hypotheses, and are always dynamic, relational and emergent in the unfolding of
the interaction.

Some social groups have also developed more fixed systems of association between forms
and meanings in specific modes, through a codification process, for example, in specific
fields of technical drawing for image (think of maritime maps, MRI scans, or engineering
circuit diagrams, for example), symbolism in mathematics, notation in music, body
movement in codified dance traditions like ballet (unlike more contemporary dance) and so
on. The mode of writing (more than speech) has been codified in script systems, and in
named/national languages, so has the mode of gesture in sign-languages. This has been done
through literacy education, the production of dictionaries and also the development of canons
and systems of equivalence between named languages in the long history of interlingual
translation. Yet even for languages, form-meaning association is not fixed as mode per se, as
the degree of codification depends on contexts and genres; normally only highly specialised
and technical contexts have developed high levels of codification in form-meaning
associations. In this sense, for all modes, codification is a process developed to restrict the
range of meaning potentials of semiotic resources when used in specific social contexts and
organisations that require fixity for carrying out their social actions (e.g., consider the army,
for example, and how interaction in all resources is highly codified, from dress, to body
posture, to gaze, face expression etc.) – in sum, it is the character and needs of the social
practice, rather than the nature of human semiosis or the type of semiotic resource, that
determines more or less fixed associations of form and meaning in sign- and meaning-
making. If we consider the whole of communication in its many contexts and functions, and,
even more importantly, all resources used in combination whenever we communicate, we
realise that codification phenomena are relatively an exception in sign- and meaning-making.

Besides and beyond individual modes, conceiving of meaning-making as hypotheses on the

motivation between forms and meanings is especially useful at the level of actual instances of
communication. Because we do not communicate through isolated signs, made in one single
semiotic resource, but by orchestrating multiple sign-making acts together into sign
complexes combining different semiotic resources (i.e., a gaze, with a smile, with a gesture,
with body proxemics, handling of object, spoken words, and so on – or an image, with some
writing, with specific fonts, colours and layout), a perspective that looks at sign-making

Page 7 of 22
 Page 8 of 22

rather than sign use is particularly useful when considering the complexities of actual situated
communication.

Translation is an excellent example in this regard. If signs as sign-complexes were simply

used out of a shared code or system of form-meaning association rules, translation would be
reversible. Instead, even in a strictly interlingual conception of translation, given a translated
text from language A to language B, no reverse translation into language A will ever produce
exactly the same “source” text. If, beyond strictly linguistic conceptions of text, we consider
the multimodality of communication, this becomes even more evident as no reverse
translation of a written description of an image will ever produce the same “original” image
(see an illuminating example of this discussed in Kress, 2020); no reverse translation of a
film adaptation will ever produce the same “original” novel; and no reverse translation of the
whole complex of social distancing practices taking place between staff and customers in a
shop will ever produce the same “original” written and spoken regulations given by shop
owners to shop assistants, nor a reverse translation of these regulations will ever produce the
same “original” government directives that led to the enforcing of social distance in the shop.

From the above examples it might become clear how a Social Semiotic perspective onto sign-
and meaning-making (and its re-making) has the potential to reframe conceptualisations of
translation.

Multimodality and Translation in the contemporary semiotic landscape

Although communication has always been multimodal, in recent years the multimodality of
communication and its import on translation have come to the fore in academic debate,
mainly because of two interrelated factors, i.e., globalization and technological innovation in
means of text production and distribution.

The increased transnational flows of goods, people and money of today’s globalized world
involve an increased circulation of representations, media products, communications and
cultural artefacts, as well as an increased transnational composition of our social and semiotic
environments, in terms of people, objects, and texts that characterize the landscape of social
interactions, both offline and online. At the same time, corporations have increasingly
perceived the need to adapt their products, branding, and advertising/marketing strategies to
the cultural specificities of local markets; so, globalization dynamics have intertwined with
localization practices of both verbal and nonverbal resources in corporate and media products
targeting different local audiences/consumers, such as websites, packaging, and
advertisements, as well as audio-visual products, such as games and TV adverts.

Technological innovation, through digital means of text production, has facilitated the
creation of representations combining different semiotic resources, beyond specialized
professional production. Along with facilitated means for multimodal text production,
technological innovation has also enhanced means of text distribution, through online
environments that enable immediate transnational circulation. As a consequence, the semiotic
composition of texts (broadly defined as meaningful wholes) has changed, with a salient
presence of visual resources. This “visual turn” (Jay, 2002) in text-based communication has
questioned the role of writing as the sole or main resource making meaning in a text.
Technological innovation has also increased the number of tools facilitating interlingual
translation, either machine-generated (such as built-in functionalities for automatic
translation, like those available on Web browsers, and automatic translation captioning, like

Page 8 of 22
 Page 9 of 22

those available on YouTube), or user-generated (e.g., in the case of subtitling tools that
facilitate “from below” practices, as in fansubbing), which is de facto challenging the role
and import of professional translation.

Both globalization and technological innovation are complex phenomena characterized by

contradictory dynamics (Appadurai, 1996; Lash & Urry, 1994). Technological innovation has
led to a re-distribution of power from “one-to-many” to “many-to-many” (Rafaeli & LaRose,
1993) forms of text production and distribution, with sign-making and translating practices
originating “from below” (see the concept of “produsage” in Bruns, 2006) and responding to
diversified community-specific needs (e.g., fan-communities in the case of fansubbing). This
new semiotic backdrop has exposed us to visual, auditory and material resources produced in
other areas of the world, while contributing to a multiplication and fragmentation of
communities with their own specific and ever emerging sign-making practices (New London
Group, 1996), leading us to question the idea of homogeneous nationally- and language-
bound communities sharing the same cultural background for meaning interpretation (see
discussion in Adami & Ramos Pinto, 2020). In simple terms, if meanings belong to cultures
rather than to codes (Kress, 2020), who shares the “same” culture (and hence meanings) for
specific uses of, e.g., image+music+writing is now increasingly difficult to determine.

New contradictions and challenges arise for translation when conceived as translating a text
from a source culture and language to a target one. Firstly, with a fragmentation and
multiplication of communities, affinity spaces (Gee, 2005b) and sign-making practices, as
well as with enhanced transnational circulation, it is increasingly difficult to draw on the
elusive concept of “target audience” as defined by a supposed “shared culture”. Secondly,
facilitated tools for user-generated translation enhance issues of professional recognition, in a
world where virtually “anybody can translate” (even machines). This is particularly the case
when artefacts are designed to facilitate transnational circulation while minimizing the need
for interlingual translation, like, e.g., video recipes circulating on social media, which use
writing only for ingredients and units of measures. Thirdly, while texts are increasingly
multimodal, so that meaning can hardly be made by relying on language alone, time- and
cost-pressures in professional translation practice have led to technological innovations that
isolate the verbal component (as in the case of Computer-Assisted-Translation Tools, but also
in remote interpreting), thus de facto constraining professionals to make translation choices
relying on partial renditions of the overall meaning of a multimodal text.

Academic research has started to address some of these issues. An increased

acknowledgement of a changing semiotic landscape, has brought attention to the concept of
multimodality in Translation Studies. An increasing number of works analyze translation and
localization of multimodal products (e.g., Dicerto, 2018; O’Hagan & Mangiron, 2013); yet
the cases examined are still very specific and isolated, making it difficult to compose a wider
picture of generalizable significance for the field as a whole and for day-to-day translation
practice. Useful theoretical discussions have so far made sense of the state of the art on the
issue (Boria & Tomalin, 2020; Dicerto, 2018; Gambier & Ramos Pinto, 2016; O’Sullivan,
2013; O’Sullivan & Jeffcote, 2013; Pérez-González, 2014; Tuominen, Hurtado, & Ketola,
2018). Yet, in the overall majority of cases (with notable exceptions in works on accessibility
and multisensory translation), also when considering multimodal aspects of translations,
research draws on data from interlingual contexts – hence even in works on multimodal
translation, the context is logocentrically- and interlingually-defined and looks at – say –
images as rendered in a target text in a different named/national language. However, the

Page 9 of 22
 Page 10 of 22

scope of translation is potentially much wider than the sole making of meaning (of even
multimodal aspects of texts) from one language to another.

Among those few who push for a move from a linguistic to a semiotic approach to
translation, Marais (2019) makes an extremely convincing case for how such a shift could
help re-conceptualise translation, both in its field of study and for the possible developments
in its professional domain and applications. Still, Marais’ work considers multimodality only
as a communicative phenomenon that drives needs and demands for changes, without
drawing on multimodal theories on meaning- and sign-making to reconceptualise
translational phenomena. Instead, Kress (2020) as well as my own co-authored work (Adami
& Ramos Pinto, 2020; Ramos Pinto & Adami, 2020) have started to outline a Social Semiotic
multimodal approach to translation. In what follows, I build on this approach to suggest how
Social Semiotic concepts could reframe the dimensions of translation, and hence the scope of
translation as a whole.

Social Semiotic concepts for rethinking the dimensions of translation

Social semiotic research has developed various detailed frameworks for the analysis of
specific aspects of texts and communicative events. Here I am proposing only a few key
concepts that are relevant to grasp the broad dynamics of semiotic practices in situated
communication, specifically for how they can help to shed light onto translation.

- Modes, media, affordances

Although “multimodal” can be often confused with “multimedia” and used interchangeably
(see for example Marais, 2019), in Social Semiotics the concept of mode does not coincide
with that of media (for a detailed discussion, Jewitt, 2004). Modes are technologies for
representation, i.e., the socially-shaped material we use to make meaning. Media are
technologies for designing, producing and distributing4 these representations (Kress, 2010;
Kress & van Leeuwen, 2001). A text in the mode of speech can be designed through different
media, e.g., by rehearsing thoughts out loud through our phonatory apparatus or by typing on
a laptop (when writing notes for a speech, or the script of a presentation); it can be produced
through our phonatory apparatus or through a writing-to-speech software app; it can be
distributed through a microphone, a radio, or a Whatsapp audio message. Analogously, an
image can be designed through sketches with a pencil on paper or through various
preparatory attempts by looking at a camera (without considering the media used in staging
the components to be captured on camera for a staged image). The “same” image can be
produced through paint on canvas, through a professional camera or a webcam, through
knitting, or through a drawing software app. It can be distributed printed on paper, or on a
mug, on a knitted jumper, shared on social media, shown on screen or projected onto a wall,
for example. When designed, produced and distributed in different media, the material
actualisation of any mode and their combination as a text will be different, and so will their
meaning potentials differ to some extent (e.g., whether moving image, in the format of a
video, is produced through a mobile phone or through a professional camera), and so will
also our ways of engaging with them and experience them, e.g., whether the same video is
watched on a social media platform on a mobile phone, on a large screen TV or in a cinema.
From design, through production up to distribution, the media carrying meaning and the
modes representing it will change, hence meaning is transformed, i.e., translated into
different materialities.

Page 10 of 22
 Page 11 of 22

Different media also afford different modes and multimodal combinations, e.g., the radio
affords speech, sound and music; the tv affords also writing and moving image and, within it,
gesture, face expression, body movements etc. Media, like all tools, have affordances
(Gibson, 1979), in terms of possibilities and limitations for designing, producing and
distributing multimodal texts, and for engaging with them. These affordances are material
and become also socially meaningful, hence the choice of a medium rather than another can
be interpreted as a sign in itself, e.g., whether somebody breaks up a relationship by meeting
face to face or through a mobile phone message (see in this regard research on “polymedia”
Jansson, 2016; Madianou, 2015; Madianou & Miller, 2013).

Remediation involves translation, as website designers well know when coding a website
with a so-called “adaptable layout”, so that it translates by re-arranging its content and visual
appearance (i.e., its semiotic resources) when accessed on a desktop, tablet or mobile phone
screen (i.e., different dissemination media). Analogously, a train ticket will have a different
materiality and multimodal composition of its semiotic resources if it is printed from a
machine at the station, purchased on an app, or sent as a pdf receipt through an email.
Somebody has designed templates for these different versions to enable the same genre to be
produced and disseminated in different media. These different versions are translations of the
“same” ticket.

Modes too have affordances, rooted in their materiality and in how social groups have
developed their resources for specific uses. Image makes meaning through simultaneous co-
deployment of elements in space while speech does it through sequential unfolding in time;
gesture makes meaning both in space and time. Simultaneity and sequence are two different
logics that impose an “epistemological commitment” (Kress, 2010, pp. 16–17) onto
representing reality, and hence onto meaning that is made. Image shows while writing names,
so while image represents reality analogically, writing labels it discretely. The fact that
modes have different affordances, makes them more suitable for making certain meanings
rather than others, e.g., image can more easily show relations between elements while speech
can more easily tell sequences of events. Humans can work around these material affordances
to stretch the meaning potentials of modes, so comics have been developed to represent
sequences of events through the sequencing of images, while in speech we have developed
resources to describe relations, compare and measure. In sum, the affordances of modes at
any given time and for any social group derive both from their materiality and from their
social histories, i.e., the semiotic work that has been done by that social group onto the
materiality of the mode. Because of the epistemological commitment involved in the
affordances of each mode one can never express exactly the same meaning across two
different modes. Transposing meaning from one mode to another involves translation. Kress
names “transduction” (Kress, 2010, p. 123) this specific type of translation.

Possibly more significantly, the fact that modes have affordances, i.e., they are more readily
available to produce certain meanings rather than others, is the very reason why humans
communicate multimodally; quite simply, to express our meanings, it is more efficient to
combine different resources, each playing a specific role, for what it can do best. This means
that, while meaning re-making from one mode to another and from one media to another are
two different and intertwined translation processes, the translation of texts as products of
meaning re-making, involves more than translation from one mode to another; it involves
translation of multimodal wholes.

Page 11 of 22
 Page 12 of 22

- Genres and domains

Humans communicate by means of multimodal texts, artefacts and events, designed,

produced and distributed through media, to fulfil specific social functions. “Genre” is the
semiotic label used to distinguish between text types and events that fulfil different social
functions; it “addresses the semiotic ‘emergence’ of social organization, practices and
interactions” (Kress, 2010, p. 113). Genre frames social action in terms of recognisable
practices and hence it “mediates between the social and the semiotic” (Kress, 2010, p. 116).
A ticket is a type of multimodal text, or genre that fulfils the function of giving access to a
specific service, like a train ride; the way it is shaped frames the social actions involved in
providing and gaining access to a service. A CV is a genre that fulfils the function of
presenting myself in applying for a social position – the way it is shaped in different domains
and contexts gives indications on the aspects of a life that are significant or not, as well as the
social relations among those involved in a recruitment process in a given social context (in
terms of formality and distance, expressed in the register of its semiotic resources, for
example, from image, font, layout and writing for a traditional CV, to moving image
including speech, gaze, face expression, dress code, location and sound/music for a video
CV). I will pose differently for an online dating profile photograph or for a CV photograph.
They both mean “this is me”, yet, through the semiotic resources I will choose, I will select
different aspects of my identity for each, and will shape differently the relation with my
addressee. Transposing meaning across genres too involves translation.

As abstract categories, modes, media and genres do not coincide but always come in an
interrelated relation instantiated in actual texts and communicative events. Changes in genres
may and often do involve also changes in modes and media, which invariably produce
differences in meaning. The translation process in transposing meaning across genres is
significant also when this involves changes in genres within broadly the same intended
audience and domain. For example, the modes and media I would use to present the
“content” of the present chapter at a conference would be different, combining modes such as
speech, face expression, gesture, and body movements, along with images and writing, for
example, produced through media composed of different parts of my body and through
Powerpoint slides on a laptop projected onto a screen. The different affordances of both
modes and media, combined with the different material constraints of these genres (e.g., word
limit for a chapter vs time limit for a conference presentation) and my experience of past uses
of resources in them (e.g., different levels or degrees of formality, of abstraction, detail and
exemplification in each) require a considerable translation effort in terms of choices in
transposing meaning from chapter to presentation, which of them to transpose, which to leave
out, and in which forms (all of which will inevitably result in new meanings).

Genres cross domains, i.e., subject areas, or fields of social practice, such as, for example,
finance, sport, medicine etc. Like for all the other concepts introduced in this section, I
conceive of domain as a heuristic that can be applied to different levels of generalisation and
intersection (so banking could be a domain, or a sub-domain of finance; academia could be a
domain, intersecting with education and research, while different disciplines could be sub-
domains in it; sport could be further sub-divided in different domains for different sports,
etc.). Translation occurs also when meaning is re-made in a different domain; so, for
example, a presentation of the contents of this chapter will have to be changed if given at a
conference in linguistics, in semiotics or in translation studies.

Page 12 of 22
 Page 13 of 22

Translation (in the sense of meaning re-making) across domains involves also translation of
genre practices. Genres, media and modes travel and are shared across domains (so you can
have an advertisement for banking products, for a university degree and for a sport event),
while texts and events belonging to the “same” genre in these domains have specific
realisations and regularities, as well as borrowing practices from each other (consider for
example the differences and similarities between an academic article in physics and in
linguistics). Those who produce advertisements, for example, need to translate advertising as
a genre to the specificities of each domain, whenever they are commissioned an
advertisement in finance rather than in education. Those who change domains in their
profession need to translate the knowledge they have achieved for carrying out social actions
from the old to a new one (education names this kind of translation as “transfer”, as in
“knowledge transfer” between academia and the industry, and in “transferrable skills”).

- Semiotic spaces and discourses (or the issue of culture)

Actual texts and communicative events (in different modes, media, genres and domains)
occur in contexts, that is, specific semiotic spaces in which participants play different roles,
and carry out specific social practices. Different semiotic spaces develop different semiotic
practices, i.e., different ways of carrying out social practices and interacting with other
participants. Semiotic space too is a heuristic that can be applied at different levels of
abstraction, each sharing some degree of sharedness in semiotic practices. We can conceive
of Instagram as a semiotic space, when trying to define the emerging semiotic practices that
are common in that media platform and distinguish it from other social media spaces. But we
can also consider a specific Facebook group or a football fan club as a semiotic space. The
more I am familiar with a space and its practices, the more I will be able to perceive
differences in it and hence sub-divide it into different semiotic spaces. A semiotic space is a
(metaphorical rather than necessarily physical) space that gathers people who share similar
interests to perform shared activities, and hence develop shared semiotic practices.

In this sense, following Gee (2005b), I choose the label “space” instead of “community” to
shift the focus from people’s identity and membership to the actual semiotic practices that
emerge and develop through people’s participation. This is for a series of reasons. Firstly,
space rather than community avoids essentialising people by forcing them into identity labels
(for a critique of essentialism, see Phillips, 2010). Each of us is in fact a dynamic and
relational conglomerate of multiple emerging identities, which are activated and performed
selectively only in relevant contexts, or spaces; so, for example, my being an academic is
relevant only in some spaces, and can be occasionally activated in others, such as at the pub
with friends if I am asked about aspects related to my work. Secondly, space rather than
community avoids the problematic issue of defining membership, and hence who belongs or
not in a specific community, and by which defining criteria this is so. There are indeed
different levels of participation and also self-identification in different spaces; I cycle, but I
would not define myself as a cyclist, although others who cycle as much as I do might
instead; I participate in a series of Facebook groups but I do not necessarily identify myself
with them. Thirdly, space frames more immediately the character of contemporary
communication and our forms of participation in it; indeed we all participate in multiple
spaces and, although this is acknowledged also for contemporary definitions of the concept of
“community” (e.g., the literature on communities of interest and communities of practice
Barton & Tusting, 2005; Wenger, 1998)), the term lends itself more easily to exclusivity and
exclusionary inferences, in terms of either/or belonging. This is the case, for example, of
national labels; so, being born in Italy and living in the UK, I would find it problematic to

Page 13 of 22
 Page 14 of 22

relate to labels such as “the Italian community” and “the British community”, while I can say
that I have participated in and have some knowledge of semiotic practices taking place both
in Italy and in the UK as national semiotic spaces (whatever that might mean, in terms of
generalisations that I might activate), and I also have some familiarity with and access to
online semiotic spaces such as Facebook groups that are participated by some Italians living
in the UK. Finally, and possibly more importantly both epistemologically and pragmatically,
to research and practise both communication and translation, a focus on semiotic practices is
both more feasible and more useful than a focus on people themselves. Indeed, practices can
be observed, analysed, understood and compared, both in research and in our own practice, to
make informed choices on how to make our meanings and signs in the most apt ways. In the
example of interlingual translation practice and training, observing semiotic practices in
specific semiotic spaces (i.e., researching parallel corpora) can enable more evidence-based
translation choices than any reference to supposed ideas of intended audience.

In sum, texts and communicative events not only combine different modes, produced in
different media and belonging to different genres, in different domains, but also occur in
semiotic spaces that share specific semiotic practices. Transposing meaning across spaces (in
the sense of changes in positioning, which inevitably involve changes in meaning) involves
translation. Social media managers are well aware of this, when re-designing the “same” post
for different social media spaces. I too, would have to reshape and reconfigure the meanings
of this chapter if I had to talk about it with friends in a pub. Again, changes in semiotic
spaces involve changes in modes, media, genres and/or domains.

For what I propose as semiotic spaces, Translation Studies and practice tend to use the label
of (source and target) audience and the even more problematic one of (source and target)
culture, associated to different languages in interlingual translation contexts. Even when
liberated of essentialised associations of nationality, and even when not simply referred to
named/national languages (as instead is often done in translation studies or at least teaching),
culture presupposes sharedness among all members. Besides the thorny issue of defining
membership mentioned earlier, this is problematic for two further reasons. Firstly, if we
conceive of texts as multimodal, the concept of a source and target culture would imply
sharedness in form-meaning associations in all semiotic resources. Yet, as discussed in
Adami and Ramos Pinto (2020), because of their different histories of social development
and type of transnational circulation, we cannot assume that sharedness in a given language
coincides with sharedness in gesture or image, for example. Semiotic resources such as
music, gesture, image, clothing, and architecture travel across linguistic borders and we make
meaning with them and engage with them differently than language – for one, irrespectively
of where they come from, we normally engage with them without anybody translating them
for us, unlike language. This makes our meaning making with them both potentially more
transnationally shared (on the basis of hypotheses on their materiality) and more
individualised, on the basis of the different semiotic spaces that each of us participates in
(hence on our different experience of past uses of these resources) – consider the example of
music or face expressions, of clothing as well as ways of walking, and of entire lifestyles and
associated semiotics (e.g., in gesturing, clothing, face expressions, body movements and
music of hip hop, how they circulate globally and yet in specific spaces, and how the
meanings we make of these semiotic ensembles are both transnationally shared and
individualised at the same time). In banal terms, we cannot speak of an English-native
semiotics vs an Italian-native semiotics, because the use of different semiotic resources will
not map onto meaning associations across the same English vs Italian borders. Secondly, the
contemporary media landscape is characterised by a multiplication and fragmentation of

Page 14 of 22
 Page 15 of 22

semiotic spaces (consider, for example, the multiplicity of spaces offered by social media
platforms today, compared to the very few handful institutional media sources provided by
national newspaper and tv channels of a few decades ago), and each of us participates in
multiple ones, bringing with us knowledge of semiotic practices from the spaces we are
familiar with. Hence an approach like the one sketched in the previous section that conceives
of signs being made every time rather than used, and meaning made through hypotheses on
materiality and past uses is particularly useful. In sum, instead of concepts of source and
target culture, language and/or audience, which trigger conceptions of shared knowledge in
all resources (and coinciding with national languages), it seems more realistic to work with
concepts of source and target semiotic spaces. Semiotic knowledge will always vary to some
extent between producer and audience; doctors around the world could read MRI scans,
engineers could read circuit diagrams, and seafarers could read maritime maps, while they
might not share the same language instead. At the same time, sharing the same language
would not enable somebody without knowledge of these codified uses of image to read them.
Translation across spaces in this sense would involve translating the meaning of an MRI scan
to a patient, for example, that is, translation across non-sharedness in semiotic knowledge –
which might most likely involve also translating meaning across modes, media and genres
(e.g., from the image of the MRI scan to a speech, gaze and gesture multimodal complex,
during a doctor-patient consultation).

One last aspect to consider is the fact that, because people participate in multiple semiotic
spaces, the same space is both characterised by emerging shared semiotic practices and
participated by people having different semiotic knowledge and worldviews, or ideologies.
For example, in a Facebook group, some semiotic practices will be shared (say, for example,
whether to take things posted seriously or as a joke, or how to interpret intertextual references
in relation to the theme of the group), while in the comments of any post participants might
argue showing different values and attitudes, or mindsets towards the same topic. So, even in
the same semiotic space, to find common ground and pursue understanding, one will have to
“translate” their meanings so that they speak to the other’s worldviews. Here too, to avoid
essentialised and reified approaches in labelling worldviews, instead of using the concept of
culture, I prefer the concept of discourse, following Scollon et al. (2012) who propose to
reframe instances of intercultural communication as interdiscursive communication.
Discourse in this case, in the Foucaultian tradition, refers to “big D” Discourse in Gee
(2005a), as socially constructed knowledge on the world (Kress, 2010; Kress & van
Leeuwen, 2001). As a banal example immigration can be seen as a crisis or a need, as an
invasion or as a resource, and so on for any aspects of life (consider for example how also in
the same social media space people clash on different discourses about COVID-19
vaccination, with some seeing it as an act of collective and civic responsibility and others as a
control strategy and an imposition to individual freedom). Any text entextualises multiple
intertwining discourses. Discourses are realised through modes, produced through media, in
texts of different genres, in specific domains and semiotic spaces. Making signs and
meanings in different modes, media, genres, domains and semiotic spaces involves
translation across discourses, to respond to the different worldviews of these spaces and then,
in interaction, also to the possibly different worldviews of the participants within the same
space.

Conclusions: Dimensions of translation processes

Texts combine multiple modes, and take place through media, as instantiations of genres in
specific domains and semiotic spaces, characterised by emerging, dynamic and relational

Page 15 of 22
 Page 16 of 22

sharedness in practices and knowledge, and differently shared discourses. As a consequence,

the variables for what we consider translation processes and needs multiply compared to
when one considers only interlingual translation. Any form of meaning remaking involves
some changes in (aspects of) mode, media, genre, domain, semiotic space and/or discourse. It
all requires translation.

While we are all familiar with Jakobson’s tripartite categorisation of translation as

intersemiotic, interlingual and intralingual (Jakobson, 1959), it becomes apparent from the
above that this taxonomy is not sufficient to consider all the dimensions that come into play
whenever meaning is re-made. Given the aspects that are always involved in any instance of
meaning-making, i.e., Mode, Media, Genre, Domain, Semiotic Space and Discourse, a
translation process is in place whenever resemiotisation (i.e., the remaking of meaning in
social practice) crosses any of them. So we can consider the following dimensions of
translation:

- Intramode and intermode: if meaning is re-made in the same mode or into a different
mode;
- Intramedia and intermedia: if meaning is re-made in the same media or into a
different media;
- Intragenre and intergenre: if meaning is re-made in the same genre or into a different
genre;
- Intradomain and interdomain: if meaning is re-made in the same domain or into a
different domain;
- Intraspace and interspace: if meaning is re-made in the same semiotic space or into a
different semiotic space;
- Intradiscourse and interdiscourse: if meaning is re-made in the same discourse or into
a different discourse.

All these dimensions come into play in communication, if we conceive of a text as the
punctuation of a ceaseless process of semiosis as resemiotisation. Furthermore, intra- and
inter- might not be mutually exclusive. This is the case, for example, for the “modal”
dimension, given the multimodal nature of our text production; so, within translation of or
into the same text, some meanings might be translated in the same mode and others in a
different one. The same is for the discourse dimension, as some worldviews entextualised in
a text might be shared and others might not. In sum, following Marais’ (2019) stress for the
need of conceiving translation as a process, to counterbalance a research tradition too focused
on the products of translation, these dimensions do not define translation as product, but
rather identify dimensions of the translation process, which might (most likely often)
intertwine together in a translation product i.e., a text or communicative event as the
punctuation of a ceaseless process of resemiotisation.

One could ask at this point about the place of interlingual translation in this. The remaking of
meaning into a different language is intramodal and interspace, as it deals with re-making
meaning in the same mode (if it is always in writing) for a space that has different form-
meaning associations (or semiotic knowledge) in that mode. It might be intermodal if
meaning is re-made from writing to speech or sign-language, and might always involve
interdiscourse aspects too (as in the very well-known phenomenon of handling cultural
references in interlingual translation practice), in the same or different media, genres and
domains. Assessing the translation process involved in which of these dimensions is useful
when approaching also translations as products in the current interlingual translation practice.

Page 16 of 22
 Page 17 of 22

For example, when translating an audiovisual text, like an anime, it might be useful to
consider the semiotic knowledge in each mode for the “target” semiotic space as some uses
of resources of image might be shared (given that anime circulate in transnational semiotic
spaces), while language might not.

---

Whenever we make meaning of something, we translate – or better, some translation process

takes place to relate that reality to our personal (and socially-acquired) universe of reference.
The process might be more visible when we face a reality that stands out as “new” for us
(something we have not seen, heard, touched, tasted, smelled before, or more generally
encountered and experienced before) or as “unexpected” in the way it appears in some
context (something that, in its form, we know, but if we apply to it the meaning association
that we are familiar with, the “sign” does not quite make sense in that context, as when, for
example, our neighbour turns their back at us when we encounter them on the stairway
leading to our flat and we ask ourselves what that might mean). Yet, although more
frequently unnoticed, some translation process occurs every time we make meaning, even
when faced with apparently more familiar realities or sign complexes.

Translating is about establishing some new relations – i.e., making new meaning – for us
and/or for others. It is a key process of all communication in that, by definition, every
instance of communication presents something new to its addressee, or – quite simply – there
would be no purpose in communicating at all (see in this regard Lotman, 2004, in his
assumption that human communication presupposes non-identity between “producer” and
“receiver”). Far from a mere application of pre-given coding-decoding rules or pre-fixed
systems of equivalence, making meaning (and translation as part of it) is about making
hypotheses on what a certain form could mean for the other and for us. A Social Semiotic
perspective on communication and translation, as sign- and meaning-making and re-making
as hypotheses making seems more suitable to handle the complexity of the contemporary
semiotic landscape. Conceiving of translation along those dimensions could be useful also in
rethinking the possible social role of translators today, beyond the strict domain of
interlingual practices (increasingly challenged in its scope), as “expert” meaning re-makers.
Translators as mediators of meanings could become key figures for positive change, in
societies that are traversed by a multitude of different semiotic practices and increasingly
divided in worldviews (or discourses), in spite of speaking the same (national) language.

References
Adami, E. (2017). Multimodality. In O. Garcia, N. Flores, & M. Spotti (Eds.), Oxford
Handbook of Language and Society (pp. 451–472). Oxford: Oxford University Press.

Adami, E. (2018). Multimodal sign-making in superdiverse contexts: The case of Leeds

Kirkgate Market. In E. Adami & A. Sherris (Eds.), Making Signs, Translanguaging
Ethnographies: Exploring Urban, Rural, and Educational Spaces. Bristol: Multilingual
Matters.

Adami, E. (2023). Multimodality and the issue of culture: A social semiotic perspective onto
the interculturality of communication. In U. Schröder, E. Adami, & J. Dailey-O’Cain (Eds.),
Multimodal Communication in Intercultural Interaction (pp. 17-40). London: Routledge.

Adami, E., & Ramos Pinto, S. (2020). Meaning-(re)making in a World of Untranslated Signs:

Page 17 of 22
 Page 18 of 22

Towards a Research Agenda on Multimodality, Culture, and Translation. In M. Boria, Á.

Carreres, M. Noriega-Sánchez, & M. Tomalin (Eds.), Translation and Multimodality: Beyond
Words (pp. 71–93). London/New York: Routledge.

Agar, M. (1995). Language Shock: Understanding the Culture of Conversation. New York:
William Morrow.

Appadurai, A. (1996). Modernity at Large. Cultural Dimensions of Globalization. Public

Culture (Vol. 1). Minneapolis: University of Minnesota Press. [Link] APP

Barthes, R. (1977). The Death of the Author. In R. Barthes & Heath (Eds.), Image, Music,
Text (pp. 142–148). New York: Hill and Wang.

Barton, D., & Tusting, K. (2005). Beyond Communities of Practice. Cambridge: Cambridge
University Press.

Bateman, J. (2011). The decomposability of semiotic modes. In K. L. O’Halloran & B. A.

Smith (Eds.), Multimodal Studies: Multiple Approaches and Domains (pp. 17–38). London:
Routledge.

Bezemer, J., & Kress, G. (2008). Writing in Multimodal Texts: A Social Semiotic Account of
Designs for Learning. Written Communication, 25, 166–195.

Bezemer, J., & Kress, G. (2014). Touch: a resource for making meaning. Australian Journal
of Language and Literacy.

Boria, M., & Tomalin, M. (2020). Introduction. In M. Boria, Á. Carreres, M. Noriega-

Sánchez, & M. Tomalin (Eds.), Translation and Multimodality: Beyond Words (pp. 1–23).
London: Routledge.

Borodo, M. (2015). Multimodality , translation and comics. Perspectives: Studies in

Translatology, 23(1), 22–41.

Bruns, A. (2006). Towards Produsage: Futures for User-Led Content Production. In F.

Sudweeks, H. Hrachovec, & C. Ess (Eds.), Cultural attitudes towards technology and
communication (pp. 275–284). Murdoch: Murdoch University.

Dicerto, S. (2018). Multimodal Pragmatics and Translation: A New Model for Source Text
Analysis. London: Palgrave.

Djonov, E., & van Leeuwen, T. (2014). The semiotics of texture: from tactile to visual.
Visual Communication, 10(4), 541–564.

Gambier, Y., & Ramos Pinto, S. (2016). Special issue on Audiovisual Translation:
Theoretical and Methodological Challenges. Target, 28(2).

Gee, J. P. (2005a). Discourse vs. discourse, 1–5.

Gee, J. P. (2005b). Semiotic Social Spaces and Affinity Spaces: From The Age of Mythology
to Today’s Schools. In D. Barton & K. Tusting (Eds.), Beyond Communities of Practice:

Page 18 of 22
 Page 19 of 22

Language, Power and Social Context (pp. 214–232). Cambridge: Cambridge University
Press.

Gibson, J. (1979). The Ecological Approach to Visual Perception. Boston: Houghton Mifflin.

Halliday, M. A. K. (1978). Language as Social Semiotic: The Social Interpretation of

Language and Meaning. London: Arnold.

Halliday, M. A. K. (1985). An Introduction to Functional Grammar. London: Arnold.

Hodge, R., & Kress, G. (1988). Social Semiotics. Cambridge: Polity.

Iedema, R. (2001). Resemiotisation. Semiotica, 37(1/4), 23–40.

Iedema, R. (2003). Multimodality, resemiotization: extending the analysis of discourse as

multi-semiotic practice. Visual Communication, 2(1), 29–57.

Jakobson, R. (1959). On Linguistic Aspects of Translation. In R. Brower (Ed.), On

Translation,edited by, HarvardUP, 1959, pp.232-39 (pp. 232–239). Cambridge, MA:
Harvard University Press.

Jansson, A. (2016). Polymedia Distinctions. Nordicom Review, 36(2), 33–50.

[Link]

Jay, M. (2002). That visual turn. The advent of visual Culture. Journal of Visual Culture,
1(1), 87–92.

Jewitt, C. (2004). Multimodality and new communication technologies. In P. LeVine & R.

Scollon (Eds.), Discourse and Technology: Multimodal Discourse Analysis (pp. 184–195).
Washington, DC: Georgetown University Press.

Jewitt, C., Bezemer, J., & O’Halloran, K. (2016). Introducing Multimodality. London/New
York: Routledge.

Jewitt, C., Price, S., Mackley, K. L., Yiannoutsou, N., & Atkinson, D. (2020).
Interdisciplinary Insights for Digital Touch Communication. Cham: Springer Nature.

Kress, G. (1993). Against Arbitrariness: The social production of the sign as a foundational
issue in critical discourse analysis. Discourse and Society, 4(2), 169–191.

Kress, G. (2010). Multimodality. A social semiotic approach to contemporary

communication. London: Routledge.

Kress, G. (2020). Transposing meaning: Translation in a multimodal semiotic landscape. In

M. Boria, A. Carreres, M. Noriega-Sànchez, & M. Tomalin (Eds.), Translation and
Multimodality: Beyond Words (pp. 24–48). London.

Kress, G., & van Leeuwen, T. (1996). Reading Images. The Grammar of Visual Design
((2006 2nd). London: Routledge.

Page 19 of 22
 Page 20 of 22

Kress, G., & van Leeuwen, T. (2001). Multimodal Discourse: The Modes and Media of
Contemporary Communication. London: Arnold.

Kress, G., & van Leeuwen, T. (2002). Colour as a semiotic mode: notes for a grammar of
colour. Visual Communication, 1(3), 343–369.

Larsen-Freeman, D. (2015). Complexity Theory. In B. VanPatten & J. Williams (Eds.),

Theories of Second Language Acquisition (2nd Editio, pp. 227–244). London/New York:
Routledge.

Lash, S., & Urry, J. (1994). Economies of Signs and Space. London: Sage.

Lotman, J. (2009). Culture and Explosion (1992nd ed.). Berlin/New York: Mouton de
Gruyter.

Madianou, M. (2015). Polymedia and Ethnography: Understanding the Social in Social

Media. Social Media and Society, 1(1).

Madianou, M., & Miller, D. (2013). Polymedia: Towards a new theory of digital media in
interpersonal communication. International Journal of Cultural Studies, 16(2), 169–187.

Maiorani, A. (2021). Kinesemiotics. Modelling How Choreographed Movement Means in

Space. London: Routledge.

Marais, K. (2019). A (Bio)Semiotic Theory of Translation. London: Routledge.

New London Group. (1996). A Pedagogy of Multiliteracies: Designing Social Futures.

Harvard Educational Review, 66, 60–92.

O’Hagan, M., & Mangiron, C. (2013). Game Localization: Translating for the Global Digital
Entertainment Industry. Amsterdam/New York: John Benjamins.

O’Halloran, K. L., Tan, S., & Wignell, P. (2016). Intersemiotic Translation as

Resemiotisation: A Multimodal Perspective. Signata, (7), 199–229.

O’Halloran, K., & Smith, B. A. (2011). Multimodal Studies. In K. O’Halloran & B. A. Smith
(Eds.), Multimodal Studies: Exploring Issues and Domains (pp. 1–13). London/New York:
Routledge.

O’Sullivan, C. (2013). Introduction: Multimodality as challenge and resource for translation.

Journal of Specialized Translation, 20, 2–14.

O’Sullivan, C., & Jeffcote, C. (2013). Special issue on Translating Multimodalities. Journal
of Specialized Translation, 20.

Otheguy, R., García, O., & Reid, W. (2015). Clarifying translanguaging and deconstructing
named languages: A perspective from linguistics. Applied Linguistics Review, 6(3), 281–307.

Peirce, C. S. (1931). Collected Papers. (C. Hartshorne, P. Weiss, & A. W. Burks, Eds.).
Cambridge/MA: Harvard University Press.

Page 20 of 22
 Page 21 of 22

Pérez-González, L. (2014). Multimodality in Translation and Interpreting Studies. In S.

Bermann & C. Porter (Eds.), A Companion to Translation Studies (pp. 119–131). Chichester:
Wiley-Blackwell.

Phillips, A. (2010). What’s wrong with essentialism? Distinktion: Scandinavian Journal of

Social Theory, 11(1), 47–60.

Rafaeli, S., & LaRose, R. J. (1993). Electronic Bulletin Boards and “Public Goods”.
Explanations of Collaborative Mass Media. Communication Research, 20(2), 277–297.

Ramos Pinto, S. (2016). Ya care how me speaks, do ya? The translation of linguistic
varieties and their reception. InTRAlinea, special issue on The Translation of Dialects in
Multimedia.

Ramos Pinto, S., & Adami, E. (2020). Traduire dans un monde de signes non traduits :
l’incidence de la multimodalité en traductologie. Meta: Journal Des Traducteurs, 65(1), 9–
28.

Ravelli, L., & McMurtrie, R. J. (2018). Multimodality in the Built Environment. London:
Routledge.

Saussure, F. de. (1916). Cours de linguistique générale. (C. Bally, A. Séchehaye, & A.
Riedlinger, Eds.). Paris: Payot.

Saussure, F. de. (2005). Scritti Inediti di Linguistica Generale. (T. de Mauro, Ed.). Milano:
Laterza.

Scollon, R., Scollon, S. W., & Jones, R. H. (2012). Intercultural Communication. A

Discourse Approach (3rd ed.). Chichester: Wiley-Blackwell.

Stöckl, H. (2009). The language-image-text. Theoretical and analytical inroads into semiotic
complexity. AAA - Arbeiten Aus Anglistik Und Amerikanistik, 34(2), 203–226.

Taylor, C. (2004). Multimodal Text Analysis and Subtitling. In E. Ventola, C. Cassily, & M.
Kaltenbacher (Eds.), Perspectives on Multimodality (pp. 153–172). Amsterdam: John
Benjamins.

Tuominen, T., Hurtado, C. J., & Ketola, A. (2018). Why methods matter: Approaching
multimodality in translation research. Linguistica Antverpiensia, 17, 1–21.

van Leeuwen, T. (1999). Speech, Music, Sound. London: Macmillan.

van Leeuwen, T. (2005a). Introducing Social Semiotics. London: Routledge.

van Leeuwen, T. (2005b). Typographic meaning. Visual Communication, 4(2), 137–143.

Wenger, E. (1998). Communities of Practice. Cambridge: Cambridge University Press.

Page 21 of 22
 Page 22 of 22

1
I follow the works on translanguaging in using the label “named language” to signal the socially constructed
nature of what we traditionally conceive of “national languages”, whose boundaries are defined politically
rather than by intrinsic language properties or actual speakers’ linguistic practices – see in this regard, e.g.,
Otheguy, García, & Reid (2015).
2
There is no explicit distinctions between “sign-making” and “meaning-making” in works that use a social
semiotic approach and the terms are often used interchangeably. I instead (Adami, 2023) find it useful to
distinguish between “sign-making” and “meaning-making” in what Kress (2010) mentions respectively as
meaning made “outwardly” (when expressing something, by making signs through the use of forms) and as
meaning made “inwardly” (when interpreting something, by giving a meaning to a form produced by others or
present in our environment). In both cases new signs are made, either outwardly, through expression, or
inwardly, through interpretation – so in some sense, we are making meaning and making signs in both cases.
However, the two labels of sign-making and meaning-making, and consequently of sign-maker and meaning-
maker, to distinguish between the two cases and roles is useful as a heuristic, to avoid reductionist binaries such
as “production vs reception” (as there is always production of form-meaning associations, also in interpretation),
and to respond to the need to distinguish between outward and inward processes and roles in communication.
3
Or more precisely, the Saussurean structuralist tradition, given that, from unpublished writings (Saussure,
2005), Saussure seemed much less focused on langue and more concerned with social aspects than his pupils
transcribed in his Cours (Saussure, 1916) and his followers interpreted thereafter.
4
The stages of design, production and distribution can be separate (i.e., taking place at different times, carried
out by different people, and/or different media) or coincide – so for example, speech in a face-to-face
conversation is not pre-designed, while production and distribution coincide both in media and in the speaker.
For a detailed discussion see Kress & van Leeuwen (2001) who devote a chapter for each stage with a specific
discussion of how modes and media intertwine in each and how meanings change upon changes of modes and
media in each stage and across stages.

Page 22 of 22

View publication stats