Relevance: the key to visual & multimodal (and all other) communication

Charles Forceville

Dept. of Media Studies

University of Amsterdam

Introduction

Humanities disciplines have contributed many crucial insights to understanding how communication and interpretation works – think of rhetoric, linguistics, semiotics, metaphor theory, blending theory, narratology, and stylistics, among others. But there is no single framework within which these approaches can be combined … or is there?

In my view the foundations of an inclusive theory were laid in Sperber and Wilson’s (1995 [1986]) relevance theory. Building on Grice’s (1975) “cooperative principle,” the authors propose that, in fact, his four “maxims” (informally phrased: be truthful, be brief, be relevant, and choose the best stylistic form for your message) can be reduced to a single one: be relevant.

As thoroughly intentional beings, humans constantly monitor their environment for whatever they judge to be either potentially wholesome (which of course can comprise highly altruistic considerations) or potentially detrimental to their well-being. This cognitive principle of relevance is what helps us survive physically as well as emotionally in the world – and the excellent use we have made of it has resulted in our species’ position at the top of the evolutionary pyramid.

But nature is indifferent to us. Fellow human beings, by contrast, are not. Indeed, other things being equal, our conspecifics are biologically inclined to empathize with us (as suggested by the discovery of “mirror neurons” – see e.g., Gallese and Guerra 2019), and help us take care of ourselves and enhance our well-being. The awareness of this communicative principle of relevance is as hard-wired in our brains as is its cognitive progenitor. The central claim of relevance theory (RT) is that every act of communication “communicates a presumption of its own optimal relevance” (Sperber and Wilson 1995: 260), to which I add “… to its target audience.” Importantly, while Grice formulated his maxims in the form of exhortations, in RT the “presumption of optimal relevance” is biologically entrenched in people’s cognition, so they can’t help but obey it. Of course, a presumption is not a guarantee: as we all know, very often interlocutors presume they are relevant to us, while in fact they are boring us to death.

RT has the potential to develop into a fully comprehensive theory of communication. However, to fulfil its promise, it needs to be extended, adapted, and refined in various ways. Most importantly, it must be capable of accommodating not just the face-to-face communication between Mary and Peter that governs “classic” RT, but also needs to demonstrate how it can deal with mass-communication. And since much mass-communication is not (only) verbal in nature, the theory should espouse monomodal visual, sonic, musical, and gestural communication as well as its multimodal combinations (see e.g., Bateman et al. 2017).

In Visual and Multimodal Communication: Applying the Relevance Principle I explain how RT can accommodate mass-communicative messages consisting of static visuals, and static-visuals-accompanied-by-short-stretches-of-written-text. The first, theoretical, part of the book discusses in more detail how the communicative principle of relevance is rooted in the cognitive principle of relevance; provides a crash course in classic RT for non-linguists; shows where classic RT needs to be adapted to suit visual and multimodal mass-communication; briefly sketches pertinent aspects of mass-communication; and presents genre as the most important pragmatic factor governing the interpretation of mass-communicative messages. The second half of the book consists mainly of case studies pertaining to the following genres: pictograms, traffic signs, and logos; advertisements; cartoons; and comics.

Some key RT concepts

Here I can only provide a mere glimpse of RT’s key components – and not even all of them. I will do while largely dispensing with RT’s admirably precise (but for non-experts rather daunting) terminology. Please note that unless indicated otherwise I will follow classic RT’s convenient practice to designate the communicator as female (and the addressee as male).

If an agent wishes to communicate something to her target audience, her first duty is to attract the attention of that audience. Only if this attempt is successful can she hope that the audience will process the message – and only then the audience will decide whether to accept it as pertinent to their well-being, and thus judge its relevance.

To increase the chances of such a positive outcome, the communicator has to take into account several things. In the first place, she must make an assessment of the sum total of what the audience knows, believes, and values. After all, if the message does not mesh with the audience’s mindset, it will not “land” very well (think of an advertisement promoting high-quality meat to vegetarians). In the second place, she needs to think of a way to present the message in the best possible way to achieve a relevant effect on the audience. An effect is relevant if the information or emotion in the message is somehow beneficial for that audience, in however small or big way. Clearly, the cognitive effect of: “your age-group will receive the anti-Covid vaccine with top-priority” will for most addressees be higher than: “tomorrow it will be cloudy.” But “effect” is only half of what accounts for relevance; the other half is “effort.” The more mental effort you need to invest to understand a message, the less relevant it becomes. In order to protect an audience from having to spend unnecessary effort on things it already knows from other sources (e.g., general knowledge, previous exchanges in the communication process, the ad hoc situational context), the communicator seeks to produce a message that strikes an optimal balance between effect and effort. To achieve this, many messages are abbreviated: e.g., “yesterday” instead of “we saw the live-stream of the Hans van Manen ballet yesterday” as a response to “When did you see the live-stream of the Hans van Manen ballet?”; and “why?” instead of “why did you just now smash to pieces all the priceless wine bottles my father gave me as a present after I won the Ig Nobel prize for my fantastic joke-telling robot?”

The interpretation of a message requires first of all the processing of any explicit information it may contain. “Willem Alexander of Orange Nassau became king of the Netherlands on 30 April 2013” is an example of explicit information. Whoever knows English can decode this information. It would for instance be adequate as a relevant response to the question “who is the king of the Netherlands?” But fruitfully processing a message usually requires more than decoding its explicit components, as a message often contains implicit information that must be inferred by combining its explicit parts with contextual information. Picture the following (factually correct) scenario: A Dutch audience reads on Saturday 16 October 2020 the newspaper heading “Willem Alexander returns to The Netherlands.” The decoded explicit information, enriched by “… of Orange Nassau,” is clear enough; but this message achieves relevance only when it is combined with the right background knowledge. The pertinent background knowledge here is that the king and his family had decided to fly to their holiday house in Greece on the preceding Friday – a few days after Dutch Prime Minister Rutte had urged the population not to travel abroad, in the service of reducing the risk of spreading the Covid19 virus. The explicit information in the newspaper heading is thus to be complemented by crucial information that remains implicit – namely that the king apparently responded to massive criticism in the Dutch media of his decision by rapidly returning home (and later publicly apologizing on TV). In short, RT postulates a continuum from completely explicit information (which is to be decoded) to highly implicit information (which is to be inferred).

A brilliantly insightful claim in RT is that the more explicit a message is, the more the responsibility for its interpretation resides with the communicator. The more the message shifts to the implicit pole of the continuum, the more the responsibility for its interpretation shifts to the audience. I consider both the continuum from explicit to implicit communication and the corresponding shift of responsibility for the interpretation of a message enormously useful dimensions of the RT model, since it helps resolve – on a theoretical level, that is – the endless bickering over “the” meaning of a text or picture.

A fundamental difference between verbal and visual communication is that pictures have building blocks (e.g., Peircean icons and indices) and structures, but no vocabulary and grammar. Kress and Van Leeuwen’s (1996) influential textbook has many qualities, but unfortunately it also over-stretches the analogy between visuals and language, and in that respect has done a disservice to visual studies and multimodality scholarship (see Forceville 1999 for more discussion). In my view the relevance principle provides a much more fertile base for studying visual and multimodal discourse than the notion of “reading images: the grammar of visual design.” That being said, RT is no less but also no more than a model for communication, and has little to contribute to the analysis of a specific text or discourse. As a consequence, RT will always have to be complemented by insights from other models or approaches (linguistics, semiotics, stylistics …).

A case study

In this section I will show what an RT perspective on a specific picture (not discussed in the book) would accentuate.

Unless you are Dutch and/or are well-informed about recent developments in Dutch politics, I suspect you will be baffled by figure 1. That is precisely why I chose it: it provides me with an opportunity to demonstrate that even though supposedly “a picture tells more than a thousand words” often a lot of background knowledge is required to understand it. A second reason I selected it is that it has virtually no written information, which shows that written language is not necessary for communication.

Attracting attention. This picture appeared in a newspaper. Subscribers and anyone who happened to buy the newspaper on 29 August 2020 would naturally come across it while reading, or even just flipping through, the newspaper.

The target audience. The cartoon appeared in NRC-Handelsblad, a liberal, high-quality newspaper in The Netherlands. The cartoon thus self-selects its target audience, namely NRC-Handelsblad readers. In the recontextualized form in which it appears here, it will attract your (i.e., my envisaged audience’s) attention if you have read as far as here – and hopefully prove relevant to you.

Genre. The picture is a political cartoon. You might already have guessed this, but to the readers of NRC-Handelsblad this would be crystal clear: they know that Ruben Oppenheimer is one of the newspaper’s regular cartoonists, and recognize the place where the cartoon appears, namely as part of a weekly appearing article that critically evaluates the state-of-the-art in Dutch politics. Realizing this is a political cartoon – and thus is supposed to provide a critical and witty/funny comment on a political event – viewers already know in what direction their interpretation should go.

Figure 1. Ruben Oppenheimer, NRC-Handelsblad 29-8-2020

Explicit information. Explicit information in pictures, I propose in my book, pertains to identifiable elements and scenarios. In this cartoon we recognize a number of Peircean icons: a damaged and dirty shoe with a rather flamboyant print, a broken lace, and flies hovering above the shoe. The flies also constitute a Peircean index, metonymically cueing “stench.” There is also a Peircean symbol in the form of the repeated letters “CDA.” A verbalization of the explicit information inhering in the cartoon would be something like: “This is a strikingly chequered but damaged, stinking shoe with the letters ‘CDA’ repeatedly printed on it.”

Cultural background information the target audience is expected to access. The target audience of this cartoon knows that “CDA” is the acronym of “Christen-Democratisch Appèl,” Holland’s Christian Democrat party. This audience is also aware of a number of other crucial facts: that this type of flashy shoe is typically worn by Minister of Health, Well-being and Sports Hugo de Jonge (and therefore serves as an index of him); that de Jonge was, at that moment, one of the candidates for the CDA party leadership; and that, although in the recent internal CDA party elections de Jonge had narrowly won, there were allegations of fraud, which jeopardized his victory. All of this is information that Oppenheimer assumes his target audience to possess, and to be able to recruit. The explicit information and the (implied) contextual information need to be combined by the viewer to make the cartoon relevant and lead to the inference that Hugo de Jonge’s position as future CDA-leader is at risk. Oppenheimer has decided, we can presume, that the present cartoon was the best possible way he could think of to communicate this information at minimal mental effort on the part of his target audience – thereby achieving optimal relevance.

Understanding and accepting the message. RT distinguishes between understanding a message and accepting it. Some NRC readers may believe that the accusations of fraud in the CDA-party leader elections were slanderous, or they may be fans of de Jonge, and thus understand but not accept the suggestion that his victory was in any sense tainted. In chapter 10 of my book I discuss a number of visual and multimodal discourses in which the communicating agent shows herself to be either incompetent or unreliable. RT can accommodate lying and deceiving, since con-men and con-women’s communicative acts come no less with the presumption of optimal relevance than those coming from bona fide communicators.

Concluding remarks

If the claim that RT can form the basis of an inclusive theory of communication is accepted, a lot of work is waiting to be done. My final chapter sketches various steps that must be taken to help develop the theory further. What is needed is in-depth research into other genres, other modes (semiotic resources), and other media. One avenue for follow-up research is to examine how the RT model fares when confronted with a medium such as film, which has moving images rather than the static ones I examined, and moreover features the spoken-language, the sonic, and the musical modes – each with its own affordances and constraints. Another challenge is to study digital communication (Facebook, Twitter, Instagram …) from an RT perspective. In this respect, Francisco Yus has done pioneering work (e.g., Yus 2011; see also Forceville and Sanchez-Querubin in prep.). Finally, taking as a starting point the work by primatologist Frans de Waal (2016), I have pitched the idea that relevance theory can also serve as the basic model for studying chimpanzees’ (and other animals’) communicative behaviour (Forceville 2020).

References

Bateman, John, Janina Wildfeuer, and Tuomo Hiippala, eds (2017). Multimodality: Foundations, Research and Analysis. A Problem-Oriented Introduction. Berlin: de Gruyter Mouton.

De Waal, Frans (2016). Are We Smart Enough to Know How Smart Animals Are? London: Granta.

Forceville, Charles (1999). ”Educating the eye? Kress and Van Leeuwen’s Reading Images: The Grammar of Visual Design (1996).” Language and Literature 8(2): 163-78.

Forceville (2020, 6 October). “Rooting chimp communication in relevance theory” (blogpost). https://blog.oup.com/2020/10/rooting-chimp-communication-in-relevance-theory/

Forceville, Charles, and Natalia Sanchez-Querubin (in prep.). “Relevance theory perspectives on web-based communication.”

Gallese,Vittorio, and Michele Guerra (2019). The Empathic Screen. Oxford: Oxford University Press.

Grice, H.P. (1975). “Logic and conversation.” In: Peter Cole and Jerry L. Morgan (eds), Syntax and Semantics 3: Speech Acts (41-58). New York: Academic Press.

Kress, Gunther, and Theo Van Leeuwen (1996). Reading Images: The Grammar of Visual Design. London: Routledge.

Sperber, Dan, and Deirdre Wilson (1995 [1986]). Relevance: Communication and Cognition (2nd edition). Oxford: Blackwell.

Yus, Francisco (2011). Cyberpragmatics: Internet-mediated Communication in Context. Amsterdam: Benjamins.

Bioline

A cognitivist, Forceville works on visual and multimodal metaphor/ argumentation/ narrative discourse – e.g., in advertising, pictograms, comics & cartoons, documentary, and animation. He wrote Pictorial Metaphor in Advertising (Routledge 1996) and co-edited Multimodal Metaphor (with Eduardo Urios-Aparisi, Mouton de Gruyter 2009); Creativity and the Agile Mind (with Tony Veale and Kurt Feyaerts, Mouton de Gruyter 2013); and Multimodal Argumentation and Rhetoric in Media Genres (with Assimakis Tseronis, Benjamins 2017). A chapter co-authored with Coral Calvo-Maturana is scheduled to appear in Jesús Moya Guijarro and Eija Ventola (eds), Challenging Gender Stereotypes and the Traditional Family Unit in Children’s Picture Books: A Multimodal Analysis (Routledge).