Leeuwen, Theo van

Van Leeuwen’s interest in semiotics was sparked during his undergraduate studies at the Dutch National Film Academy in Amsterdam, from which he graduated in 1972. In 1980, he studied cinema semiotics under Christian Metz, at the Centre for Interdisciplinary Studies in Semiotics, Anthropology and Sociology, Paris. Completed at Macquarie University, his MA honours dissertation, Professional speech: Accentual and junctural style in radio announcing (Van Leeuwen, 1982), showed that the intonation of radio announcers is subject to a socio-culturally determined professional code and associated values and relations between announcers, the institutions they represent and their audiences (e.g. focus on immediacy and impartiality in news reading vs. companionship in easy listening programs). His doctoral research (Van Leeuwen, 1993) – supervised by James R. Martin, a leading figure in systemic functional linguistic discourse analysis, at the University of Sydney – offered a model for studying how written texts represent social practices, based on Halliday’s (1978) theory of language as a social semiotic, systemic functional linguistics (SFL). In addition to social semiotic theory, major influences on Van Leeuwen’s work include: Roland Barthes and Roman Jakobson (and French and Prague School Semiotics generally); Rudolf Arnheim’s psychology of visual perception and studies in film and art theory; Basil Bernstein’s sociology of education; Raymond Murray Schafer’s studies of music and sound; and John Gage’s theory of colour. The most formative influence on his research and academic career, however, remains his long-lasting collaboration with Gunther Kress, which began in the mid 1980s and initially focused on developing a framework for the social semiotic analysis of visual design.      

Van Leeuwen has published widely on multimodality and social semiotics and critical discourse analysis. His books include: Reading Images: The Grammar of Visual Design (2006 [1996]) and Multimodal Discourse: The Modes and Media of Contemporary Communication (2001), both co-authored with Gunther Kress; Speech, Music, Sound (1999); Introducing Social Semiotics (2005); Global Media Discourse (2007, with David Machin); Discourse and Social Practice: New Tools for Critical Discourse Analysis (2008); The Language of New Media Design (2009, with Radan Martinec); and The Language of Colour (2011). In 1991, he co-founded the journal Social Semiotics and is a founding co-editor of the journal Visual Communication established in 2001 (with Carey Jewitt). He is also on the editorial boards of several other international peer-reviewed journals that provide publication platforms for research on multimodality and critical discourse analysis. 

Social Semiotics

Van Leeuwen’s key contributions to social semiotics and main directions in his research overall are captured in Introducing Social Semiotics (2005), which is written in his signature, accessible and engaging, style.

Van Leeuwen views social semiotics as “a form of enquiry” that “comes into its own when it is applied to specific instances and specific problems” and is therefore “not ‘pure’ theory, not a self-contained field” (Van Leeuwen, 2005, p. 1) but ‘appliable’ (Halliday, 1985) and necessarily multidisciplinary. Social semioticians pursue three interconnected aims:

1. “collecting, documenting and systematically cataloguing semiotic resources – including their history”
2. “investigating how these resources are used in specific historical, cultural and institutional contexts, and how people talk about them in these contexts – plan them, teach them, justify them, critique them, etc.”; and
3. “contributing to the discovery and development of new semiotic resources and new uses of existing semiotic resources”.  (Van Leeuwen, 2005, p. 3). 

Semiotic resources are the actions, materials and artefacts we use for communicative purposes, whether produced physiologically – for example, with our vocal apparatus, the muscles we use to make facial expressions and gestures – or technologically – for example, with pen and ink, or computer hardware and software – together with the ways in which these resources can be organized.

Semiotic resources have a meaning potential, based on their past uses, and a set of affordances based on their possible uses, and these will be actualized in concrete social contexts where their use is subject to some form of semiotic regime.

(Van Leeuwen, 2005, p. 285)

Van Leeuwen’s focus on the relationship between the interests/agency of meaning-makers and the ways in which specific institutional and broader social contexts govern people’s use of semiotic resources is inspired by Hodge & Kress’s (1988) Social Semiotics. Building on Halliday’s (1978) idea that language is only “one of the semiotic systems that constitute a culture” (p. 2) and on his model of the dynamic relationship between text (i.e. a social exchange of meaning) and context, this seminal publication charted principles for developing a social semiotic theory that could foster interdisciplinary dialogue on communication in all its forms and across different institutional contexts, a theory for which “texts and contexts, agents and objects of meaning, social structures and forces and their complex interrelationships together constitute the minimal and irreducible object of semiotic analysis” (Hodge & Kress, 1988, p. viii). This goal has motivated multimodal and critical discourse studies in the social semiotic tradition as well as more recent efforts to combine the two and explore the role that non-verbal semiotic resources and their interaction with language and with each other play in establishing and perpetuating or challenging social divisions, norms and stereotypes. These efforts have been spearheaded by Van Leeuwen’s independent and collaborative investigations of visual racism in history and society textbooks, stereotypes materialized in the visual and kinetic design of children’s toys, and global media discourse (Machin & Van Leeuwen, 2007; Van Leeuwen, 2000, 2009b; Van Leeuwen & Caldas-Coulthard, 2004; Van Leeuwen & Kress, 1995), to name just a few.

Van Leeuwen’s advocates a holistic approach to studying semiotic resources, practices and change. This approach involves considering how people use semiotic resources and technologies in specific socio-historical contexts, in relation to the ways in which they talk about and justify (aspects of) these practices. By examining both semiotic practices and discourses about them, Van Leeuwen (2005, pp. 47-68) has developed an inventory of semiotic rules (regimes) that govern people’s meaning-making, which include rules developed by observing and conforming to trends, emulating role models, and drawing on the opinion of experts as well as rules imposed by people in power (personal authority), by writing (the law, religion, etc.), tradition and the design of objects (e.g. toys) and technologies (e.g. PowerPoint) used in communication (impersonal authority). Awareness of such norms is key to understanding and contributing to semiotic change.

Modes, media and multimodality

Van Leeuwen has pursued and contributed to laying the foundations for the two main directions in multimodality:

1. exploring the use and mapping the meaning-making potential of individual semiotic resources, and
2. studying the ways they interact to create meaning in multimodal communication.

Distinctive about Van Leeuwen’s contribution to the first direction is a strong interest in the meaning-making potential of material resources (e.g. colour, texture, sound and (kinetic) typography) and their potential to partake in or transform into modes such as language and visual design, that is, into semiotic resources that are subject to wider use and more established conventions, and therefore easier to teach, debate and model in abstract ways. In the second direction, one of Van Leeuwen’s main goals is to understand the relationship between semiotic technology (i.e. technology for making meaning such as office software, sound-recording tools, pen and paper, etc.) and changes in the (co)deployment of different semiotic resources and in the discourses that govern their use in specific social contexts. In these ways, his work has explored the relationship between modes and media (material resources and technologies), which is a central challenge in multimodality research and semiotics in general. His research has also investigated how both culture and nature contribute to semiosis and problematized other distinctions common in semiotics and functional linguistics, for example, between communicative functions, personal style and aesthetics.

The two main directions in multimodality research were drawn in Kress & Van Leeuwen’s landmark publications Reading Images: The Grammar of Visual Design (2006 [1996])and Multimodal Discourse: The Modes and Media of Contemporary Communication (2001).

The first edition of Reading Images: The Grammar of Visual Design was published in 1996, and considerably extended their work presented in a much shorter (text)book (Kress & Van Leeuwen, 1990). It incorporates insights from iconography, structural semiotics, Gestalt psychology, film and the fine arts and explores a rich variety of Western-culture visual texts (advertising and news images, maps and technical diagrams, pages from magazines, picture books, and textbooks, three-dimensional objects such as sculptures and toys, and web pages) from different historical periods. The book presents an analytical framework based on two central tenets of Halliday’s systemic functional linguistics (SFL). The first is that every text simultaneously realizes three broad types of meaning, or ‘metafunctions’:

• ideational/representational – representing patterns of experience (as configurations of processes, participants and circumstances) and the logico-semantic relations between them (e.g. additional, temporal sequence, casuality)
• interpersonal/orientational – enacting social interactions, relations and values
• textual/organizational/compositional – interweaving ideational and interpersonal meanings into cohesive and coherent texts.

The second tenet is that the meaning potential of semiotic modes can be modeled paradigmatically, as systems of interrelated options, each of which is realized through a distinctive structure. The simple system in Figure 1, for instance, represents the two main types of process that Kress and Van Leeuwen (2006 [1996]) recognize in visual representations, borrowing Arnheim’s (1974) concepts ‘volume’ and ‘vector’. Narrative/dynamic processes include one or more vectorial relations between volumes, or visual entities perceived as distinct (e.g. two people depicted holding hands and/or looking at each other, where the hands and directions of their gazes form vectors). Conceptual ones, by contrast, present such entities (e.g. a passport photo) without vectors. 

Although viewing SFL as “a good source for thinking about all modes of representation” (p.20), Kress and Van Leeuwen caution against blindly adopting categories developed for language for studying other modes; while different semiotic resources may have the ability to make the same general types of meaning, they explain, the formal structures and principles (e.g. temporal vs. spatial organisation) through which they construct meaning are different and so are their affordances and limitations. For example, both language and images can construe versions of reality that carry different truth values for different communities, a meaning potential captured in SFL’s system of ‘modality’ for English grammar (Halliday & Matthiessen, 2004, pp. 143-150). Linguistic resources for realizing modality include modal verbs and modal adjuncts that can be used to construct degrees of certainty between the polarity values of ‘yes’ and ‘no’, for example degrees of probability or obligation. In images, Kress & Van Leeuwen (2006 [1996]) show that modality depends on the complex interaction of several cues such as degrees of colour saturation, colour differentiation, brightness and detail, which together may invite viewers to interpret a visual representation as more or less naturalistic, abstract, sensory or technical. Consistent with their understanding that semiotic resources are incommensurate is also Kress & Van Leeuwen’s (2006 [1996]) very loose use the term ‘grammar’ – not to suggest that the visual semiotic is stratified in the same way as language, nor that it has analogous grammatical structures, but to emphasize that unlike earlier semiotic approaches to visual analysis, which tended to focus on the meaning of individual objects and visual elements (e.g. a particular colour or shape), in other words on what may be seen as analogous to ‘lexis’ in language, their analytical framework focuses on the structures formed by such elements within a visual composition such as a single image or the front page of a newspaper.

While Reading Images focuses on visual design as an individual semiotic resource, especially with reference to still images, Kress and Van Leeuwen’s Multimodal Discourse (2001, p. 2) presents “a view of multimodality in which common semiotic principles operate in and across different modes”.  This view reflects the contemporary semiotic landscape and especially the fact that increasingly sophisticated digital technologies allow non-specialists, often through the same interface, to select from and combine semiotic resources (e.g. typography, sound, layout) that were previously associated with discrete and highly specialized domains. Following Kress & Van Leeuwen (2001), such multimodal communication calls for “a unified and unifying semiotics” (p. 2).

Multimodal Discourse presents two guiding ideas for developing a unified theory of multimodality. The first is that such a theory needs to discover which broad semiotic principles apply across different modes (in accordance with each mode’s unique characteristics) and semiotic practices, and then use these principles to develop tools for analysing multimodal communication. Modality is one such principle, as not only language and images, but sound, too, can represent different degrees and kinds of truth depending on the extent to which it appears authentic or manipulated with sound production and distribution technologies (Van Leeuwen, 1999). Van Leeuwen’s research has demonstrated the value of several such principles – genre, style, framing, salience, rhythm, modality, and conjunctive/logico-semantic relations – for analysing intra- and inter-semiotic relations in a wide variety of texts (e.g. magazines, textbooks, three-dimensional objects, film, architectural space, soundtracks and hypermedia) (e.g. Van Leeuwen, 1991, 2003, 2005, 2011b) and for designing new media texts (Martinec & Van Leeuwen, 2009). 

The second idea Kress and Van Leeuwen (2001) present is that multimodal communication creates meaning through each of four strata: 

1. Discourse, “socially constructed knowledge(s) of (some aspects of) reality” (p. 4);

2. Design, the realization of discourses through various semiotic resources; 

3. Production, the material articulation of a semiotic event or artifact;

4. Distribution, “the technical ‘re-coding’ of semiotic products and events, for purposes of recording […] and/or distribution” (p.21).

The concepts of expression, production and distribution draw attention to the role materiality and technologies for producing/recording and distributing multimodal texts play both in creating meaning in specific semiotic practices as well as in the gradual transformation of these practices through the emergence of new modes and social relations.

A strong focus on materiality in semiosis defines Van Leeuwen’s unified theory of sound presented in Speech, Music, Sound (1999). With roots in his background in film production (where speech, music and sounds can all be part of a soundtrack), jazz music practice, and research on intonation and rhythm (rhythm is a key organization principle for time-based modes and media), the book presents a framework for analysing sound in any form and incorporates principles from phonology, musicology, the psychology of perception and conceptual metaphor theory. Again using social semiotic tools such as the metafunctions as a springboard for thinking about meaning-making in general, Van Leeuwen argues that sound is better equipped for realizing interpersonal and textual rather than ideational meanings. He also sees a ‘bottom-up’ approach, starting from material qualities (e.g. timbre, tempo) rather than larger structures, as more suitable for mapping the meaning potential of material semiotic resources, where “meaning is constructed quite differently, on the basis either of an experiential meaning potential, hence grounded […] in our bodily experience of [its] materiality, and/or provenance, hence grounded in intertextuality” (p.192). Provenance relies on familiarity with the origin of a signifier and related associations. Van Leeuwen (1999) exemplifies this with the use of the sitar as an instrument from India to create associations with mediation and drugs in 1960s pop music. Experiential meaning, on the other hand, is based on our ability to extend prior physical experiences metaphorically into knowledge, as Lakoff and Johnson (1980) have argued for linguistic metaphors (e.g. ‘I’m feeling down today’).

Van Leeuwen (2009a) maps the experiential meaning potential of material resources using what he calls ‘parametric systems’. A parametric system presents those physical qualities, or affordances, of a given resource which people have taken up in communication. These qualities are always gradable and together define the meaning potential of a given signifier such as a certain type of voice, texture or colour. A particular voice for example can be described as a combination of degrees of each of several parameters: tension; roughness; breathiness; loudness; vibrato; and nasality (Van Leeuwen, 1999, 2009a). Van Leeuwen’s notion of ‘parametric system’ is inspired by Jakobson and Halle’s (1956) distinctive features theory. Following that theory, a phoneme can be described using a small number of distinctive features and identifying each feature as either present or absent, which allows one phoneme to be differentiated from another (e.g. the alveolar fricative consonants /z/ and /s/ have the same place and manner of articulation but differ in respectively presence vs. absence of voice). Unlike distinctive features, however, the parameters in Van Leeuwen’s parametric systems are not simply absent or present (not binary choices) but gradable and not only allow one signifier to be differentiated from another but add layers of meaning to it.

In addition to sound, Van Leeuwen has used provenance and experiential meaning potential to map the semiotic potential of resources such as kinetic design (Van Leeuwen & Caldas-Coulthard, 2004), colour (Van Leeuwen, 2011a), typography (Van Leeuwen, 2006) and tactile and visual texture (Djonov & Van Leeuwen, 2011).  In this way, he has drawn attention to semiotic resources that have generally been marginalized in linguistics and semiotics and provided tools for explicitly teaching and discussing them in both semiotic theory and semiotic practice.

Characteristic of Van Leeuwen’s work on multimodality are a strong concern with the role of socio-cultural and political factors in communication and the assumption that analysis and interpretation are themselves meaning-making processes driven by individual and institutional interests. This socio-political orientation, alongside their accessible presentation, has arguably contributed to the influence of the frameworks presented in Reading Images, Multimodal Discourse and Speech, Music, Sound and their take up beyond social semiotics, in areas such as literacy, media and cultural studies. At the same time in the field of multimodality there have also been efforts aimed at avoiding the tendency to directly connect verbal categories for describing physical qualities to the interpretation of the meaning potential of these qualities (McDonald, 2012) and at developing bottom-up methods for systematically and objectively identifying low-level design features (e.g. spatial orientation, variations in font and colour) and relating them to higher-level discourse structures (e.g. genre)(Bateman, 2008). Bateman (2008), for example, offers such a method for page-based documents that can be used to develop valuable explanations of how, say, a document’s layout can best support it in achieving its purpose under various production constraints. Unlike Van Leeuwen’s work, such efforts tend not to concentrate on the ideologies behind meaning-makers’ interests and their differential access to semiotic resources and discourses.

Critical linguistic and multimodal discourse analysis

The socio-political orientation of Van Leeuwen’s research is perhaps most prominent in Discourse and Practice: New Tools for Critical Discourse Analysis (2008a), where he presents a social semiotic framework for critical discourse analysis that is informed by perspectives from anthropology, sociology and philosophy. At its core is the idea that there is a distinction between social practices and their representation in texts, or discourses. Van Leeuwen defines a social practice as a sequence of physical and/or semiotic activities that includes the following elements: social actors, their activities and reactions to these activities or to other elements of the social practice; the location(s)and time(s) of the practice; and grooming, dress, tools and materials required for it.

The framework extends Basil Bernstein’s (1990) notion of recontextualization, which focused on the transfer of knowledge from the contexts in which it is produced to those where it is reproduced and disseminated through pedagogic discourse and the semantic shifts involved in this process, which function to maintain the existing social order. Van Leeuwen (2008a) argues that not only pedagogic discourse but “all discourses recontextualize [or change the meaning of] social practices”(vii), which is why the same social practice may be subject to different representations, or attract “a plurality of discourses” (p. 6). He proposes relating social practices to discourses about them as a method for achieving CDA’s goal of revealing how discourses perpetuate social boundaries, oppression and inequality. The method involves steps: (1) analyse the semiotic practice into its components and then (2) identify how it has been transformed in discourse through the use of verbal and/or non-verbal resources. Such transformations may involve substitution, deletion, and rearrangement of the elements of a social practice and/or addition of evaluations, purposes or legitimations. Van Leeuwen (2008a, pp.17-18) demonstrates, for instance, how in a text about the first day of school a nominalization transforms the action of a teacher separating children from their parents by representing it as a phenomenon through the use of nominalization (‘the separation from families’), deleting the teacher, who is a central actor in this activity, and substituting individual children and parents (e.g. Mary and her mother) with aggregate nouns (‘families’).

Like Norman Fairclough and many other critical discourse analysts, Van Leeuwen employs SFL for analysing the role of language in recontextualizing social practices. A distinguishing feature of his approach to CDA, however, is that it also explores the role non-verbal and multimodal representations play in (re)establishing dominant ideologies. To expose this role, Van Leeuwen (2008a) argues, CDA needs to consider not only what is or is not represented non-verbally or multimodally (e.g. whether ethnic minorities are represented in the media) but also how such representations are constructed. Van Leeuwen (2008a) presents many examples from his earlier research, including the use of oblique horizontal angle to depict a group of people as ‘other’, and create detachment between depicted social actors and image viewers, and ways that the visual and kinetic construction of toys can conform to racial and gender stereotypes. Focusing on women’s magazines and electronic war games, Machin and Van Leeuwen (2007) demonstrate that multimodal genres impose western values and homogenise the formats used to present local content, and Van Leeuwen (2005) reveals how advertising discourses employ combinations of signifiers such as dress, colour, smell and so on to construct and sell lifestyle identities that mask mass consumerism. Such work also highlights popular culture and discourses as a fertile ground for developing tools for productively uniting the agendas of critical and multimodal discourse analysis. Van Leeuwen (2013), for example, argues both that “the discourses that need the scrutiny of a critical eye are now overwhelmingly multimodal and mediated by digital systems that take multimodality entirely for granted” (p. 5) and that “racist stereotypes persist in visual rather than verbal texts, and in comic strips, advertisements and other forms of popular culture rather than in more factual and “highbrow” texts” (p. 2).

Current research: Van Leeuwen’s critical multimodal explorations of semiotic practice and semiotic technology

Van Leeuwen’s current research focuses on developing a theory of and tools for analysing the relationship between semiotic practice and semiotic technology that combine multimodal and critical discourse analysis through his holistic, transdisciplinary social semiotic approach. A central case study in this research is the ubiquitous presentation software PowerPoint but more specialist software such as Adobe After Effects and Photoshop and older technologies (e.g. different types of paint) are also considered.

Van Leeuwen’s approach to semiotic technology (charted in Van Leeuwen & Djonov, 2013) innovates dominant research practices in multimodality and applied linguistics by moving beyond views of software technology as a neutral tool for creating or analysing texts, and beyond a singular focus on either multimodal acts of communication that rely on semiotic technologies (e.g. presentation slides/slideshows and the presentations in which they are used) or on the results yielded by using software to analyse large quantities of linguistic, visual or multimodal data. It considers the semiotic resources (e.g. typography, layout, colour, texture, etc.) available within a given technology in relation to semiotic resources available in the culture in general, especially those encountered in the semiotic practices recontextualized through the software (e.g. Photoshop recontextualizes visual arts, photography and graphic design practices). It also focuses on the ways that the software’s design itself, through the resources it makes available and the ways these are presented through its interface and help menu, privileges some resources other others and certain ways of using different resources. In other words, the approach investigates normativity in the design and use of semiotic technologies. (See further Djonov & Van Leeuwen (2012), who adopt the Prague School Semiotics notions of ‘markedness’ and ‘foregrounding’ in developing a model for exploring normativity in software design and use). 

Van Leeuwen’s approach to studying the relationship between semiotic practice and semiotic technology is also a dynamic one as it explores: how resources provided by the technology interact with other semiotic resources in the unfolding of multimodal events (e.g. gesture and speech in slideshow-supported presentations); how their availability and presentation within the technology as well as their use change over time; and how they vary across and are shaped by and themselves (re)shape diverse social practices. 

To illustrate, Van Leeuwen’s research on PowerPoint has examined not only slideshows designed with the software and slideshow-supported presentations in different socio-cultural contexts (e.g. first-year university lectures in different disciplines versus corporate presentations) but also changes in the design of the software itself over time (e.g. all versions of PowerPoint for Windows from 1992 to 2007). This approach also involves analysing the software both as a meaningful spatio-temporal arrangement (i.e. syntagmatically) and as a system of available semiotic resources (i.e. paradigmatically). A study within that project (Van Leeuwen, Djonov, & O’Halloran, 2013) has investigated David Byrne’s use of PowerPoint to create the art exhibition and album Envisioning Emotional Epistemological Information (2003). With reference to Van Leeuwen (2005), the study argues that Byrne’s work can be read as research on semiotics and semiotic technology congruent with all three goals of social semiotics as it presents inventories of semiotic resources that the software makes available (e.g. the AutoShape menu), reflects an awareness of the ways PowerPoint’s design influences its use and imposes and shapes contemporary corporate culture values, and makes creative use of the software “to express new meanings in new ways” (Van Leeuwen et al., 2013, p. 12).

Van Leeuwen has also emphasized the role semiotic software plays in reshaping broader semiotic practices, thereby highlighting problems in the theorization of these practices. One such practice in which Van Leeuwen (2008b) is interested is what he terms “new writing”. Unlike “old writing”, he argues, new writing follows the logic of space, and in this resembles visual design and consequently blurs the boundary between language and image. New writing presents ideas through words and/or images, but achieves cohesion and coherence in their presentation less through verbal syntax and rhetorical organization and more, and sometimes exclusively, through visual design elements such as layout and consistent colour schemes. New writing is also controlled by and learned not from style manuals and explicit teaching, but through rules built into semiotic technologies such as office software, where one’s spelling can be automatically corrected and bullet lists automatically aligned, have their first word capitalized, and so on. (See further Djonov & Van Leeuwen, 2013, 2014)

Another broad, and relatively unexplored, multimodal semiotic practice that he is currently studying is listening, or the signs that people are listening when someone else is performing or speaking:  

listening is not only a mental activity, an act of interpretation, but also a semiotic activity in which the listener either actively follows and supports the speaker in the way accompanists follow and support singers or instrumental soloists, or silently critiques the speaker, silently articulates a counterpoint discourse of a kind that critical discourse analysis so far has not yet been able to analyze.

(Van Leeuwen, 2014, p. 251)

These more recent directions in Van Leeuwen’s research in semiotics further expand the already considerable breadth of his social semiotics and pave avenues for new transdisciplinary social semiotic projects.