Multimodal Semiosis, Multimodal Semiotics: Digital Technologies and Techniques for Studying Multimodal Communication

*A Report on the Multimodal Analysis Lab, Interactive & Digital Media Institute (IDMI), National University of Singapore

*To read the page with Bulgarian translation – http://www.fatcow.com/edu/multimodal-communication-bl/

Scholars interested in human communication have long recognized that it is necessary to extend the purview of the field of semiotics to include all types of sign-making activity. Barthes (1957/1972: 112), advocating the development of “a semiological science” as earlier suggested by Saussure (1916/1975), drew attention to the diversity and ubiquity of signs: “In a single day, how many really non-signifying fields do we cross? Very few, sometimes none…on the beach, what material for semiology! Flags, slogans, signals, signboards, clothes, suntan even, which are so many messages to me”. It is no surprise then that scholars within the semiotics tradition have attended to the development and proliferation of interactive digital media and software technologies throughout the last century, and the expansion therein of the human capacity for meaningful sign-making activity. This has led to study of multimodality which concerns the often complex interactions of multiple signs, within different semiotic resources such as (spoken and written) language, (static and moving) image, gesture, proxemics, cinematography, sounds, music and displayed art (see Jewitt, 2009; Kress & van Leeuwen, 2006; O’Toole, 1994).

Yet the semiotics of contemporary media and modes of discourse make a range of demands upon analysts. For example, the study of dynamic audiovisual media raises issues of methodology and resources, in terms of the means by which semioticians explore and engage with the texts they study, and how they publish such research. Contemporary interactive media such as games, webpages and the like present further challenges. Dynamism and hyper-textuality are just two aspects of contemporary signification practices that defy representation in the traditional page-based format, and indeed challenge traditional methods and possibly theoretical approaches developed for static page-based media (writing and image).

Another issue of general significance for multimodal semiotics is the immense complexity that arises in the study of multiple signs, both in terms of their integration as components of complex signs and in relation to the holistic perspective on the operation of signs within discourses and within cultures. Lotman’s concept of the ‘semiosphere’ (Lotman, 1984/2005: 208) (cf. also Lotman & Uspensky, 1978) represents a sophisticated articulation of such a holistic perspective:

“All semiotic space may be regarded as a unified mechanism (if not organism).  In this case, primacy does not lie in one or another sign, but in the “greater system”, namely the semiosphere… Just as, by sticking together individual steaks, we don’t obtain a calf, but by cutting up a calf, we may obtain steaks, – in summarizing separate acts, we don’t obtain a semiotic universe.  On the contrary, only the existence of such a universe – the semiosphere – makes the specific signatory act real.”

Work within the social semiotics tradition, derived and adapted from Michael Halliday’s (1978, 2004) systemic functional theory and in particular within the emerging field of multimodal studies (see Lemke, 2009), has likewise presented a holistic perspective on semiosis, seeing signs in terms of their integrations and roles within higher-level social and cultural processes. Such a perspective involves relating the abstract and material planes of semiosis, actual texts to the generalized semiotic potential of the range of semiotic resources on offer to members of cultures, and in identifying the functional roles of choices and structures within texts in terms of higher-level social and cultural processes.

In the (digitally-driven) information age that has characterized human culture in recent decades, such a holistic perspective becomes increasingly relevant, as a means of accounting for the complex flow of signification within contemporary human culture in terms of its manifold aspects, but also increasingly difficult to explore: how does one account, as an analyst, for the immense proliferation and complex interrelations of signs, both within instances of discourse and intertextually (Kristeva, 1986), signs that operate across discourses and across cultures? How does one model the semiosphere and other semiotics concepts as theoretical resources for application to actual analysis tasks? What does a holistic social semiotic perspective entail in terms of a detailed empirical analysis, particularly within the interactive digital environment where annotations and other (e.g. automated, semi-automated computational) techniques of analysis may be applied, stored and later retrieved and interrogated, visualized etc?

The sophistication and levels of abstraction of theories developed within the semiotics tradition, primarily through the medium of page-based discourse, do not easily translate into models for analysis by which scholars can engage interactively with contemporary multimodal documents. What does it mean in an actual analysis to adopt the ‘semiosphere’ perspective on the multiplicity of signs in multimodal discourse and their signifying social functions? These are questions of central relevance to the projects underway in the Multimodal Analysis Lab at the Interactive & Digital Media Institute (IDMI) at the National University of Singapore, which aim to develop software resources and techniques for multimodal semiotic research and teaching.

Interactive Digital Semiotics

In addition to the opportunities and challenges presented to semiotics by the ever-evolving resources for communication in interactive digital media, there are also opportunities in the use of such technologies and techniques for the practice of semiotics itself. Software applications such as Praat for sound, and multimodal annotation tools (see Rohlfing et al., 2006) such as ELAN been used in recent years in a wide variety of studies of human and animal communication, pointing to the immense potential held by software technologies and techniques for the study of communication. However, many software-based resources exist, as well as related (for example mathematical modeling) techniques, which have been taken up in the physical, biological and social sciences but remain relatively underutilized and perhaps unadapted for the study of semiotics.

The Multimodal Analysis Lab is currently the site for a variety of projects which address the fundamental challenge of developing software resources and techniques specifically adapted for the study of semiotics, particularly within holistic perspectives such as those of Lotman and Halliday, and for teaching and learning within the social sciences in general. Under the directorship of semiotician Kay O’Halloran, a large team of computer scientists, scientists, engineers and social scientists at the Lab are collaborating on the development and use of software resources and techniques for semiotics research and teaching, bringing together knowledge and approaches from a variety of disciplinary and theoretical perspectives, primarily computer science and engineering, mathematics, and semiotics, to explore the power of their integration within the software design context.

Here the focus is not only on the study of semiosis using interactive digital media resources, but also on the theoretical challenges and issues digital semiotics raises (O’Halloran, Tan, Smith, & Podlasov, 2010). While there is clearly an immense potential on offer in the use of digital technologies for the development of semiotics theory, models and techniques of analysis, issues arise of epistemology, empiricism, reductionism, complexity, methodology, the relations of the material and the abstract planes – both within semiosis as object of study and within the computer applications we are developing (computational semiosis) – and in the development of models of semiotics theory capable of being applied to analyses of actual data within the interactive digital environment. A key aspect of our approach is that software is semiotic technology (De Souza, 2005; Andersen et al 1993; cf also work within computational semiotics): we approach the design and development of software as a semiotic as much as computational task. Another key aspect of our approach is therefore the close inter-disciplinary nature of our multidisciplinary team and our work which crosses the boundaries between C. P. Snow’s (1961) two cultures of the sciences and the humanities.

A major challenge to semiotics addressed within the Multimodal Analysis Lab is providing access to the signal within multimodal texts (video, sound, interactive webpage, etc), and particular sophisticated annotation resources available within the environment of the source text. While many software resources are available which perform these functions, special attention is being paid in our project to the needs of semiotics scholars, especially in terms of providing a variety of different types and sites of annotation analysis, such as gradient and systemic annotations on dynamic visual and audio media, and a palette of overlay tools for annotating dynamic static and visual media, including webpages and their embedded hyperlinks. These manual techniques may then be complemented by a variety of automated and semi-automated algorithmic analyses which extract mostly low-level information from the media signals. These different types of analysis may then be compared and correlated both within the database and interface environments. The human user is at the centre of the process at all times: the intention is to place a range of sophisticated computational tools into the hands of semiotics scholars.

Thus, low-level and higher-level perspectives may be united within the one location. This is to address an issue of the relations of the material and abstract planes within semiosis, one at the heart of semiotics from the earliest contemporary theories of Peirce and Saussure, through the pioneers of contemporary semiotics such as Hjelmslev, Barthes, Halliday and Lotman, to contemporary concerns centering on semiotics of interactive digital media. Software algorithms have been very successful in detecting low-level information across potentially very large data sets; the advantage here is that such analyses can then be correlated together with and interpreted within the context of higher-manual analyses by semioticians which usually operate at much higher levels of abstraction. The relations of abstract and material phenomena also appears in the development and implementation of models for the software application itself: just as occurs in language and other (inter-)semiotic resources, we have found that increases in the range and complexity of the software features and functions demand more sophisticated and abstract levels of organization within the software programme design, in order to relate different features and functionalities to one another, and lower-level programmes which can handle data and user-input to higher level functions which make the software useful and adapted for semioticians.

Issues involved in relating different analyses to one another and to their source texts within a database environment become apparent when considering multimedial and multimodal data, in particular with respect to the functions of search and retrieval which form the basis for interrogation and visualization of one’s own analyses as well as the source data, with two issues in particular confronting developers of such software resources: how to relate different media and their analyses within the database and interface; how to manage the immense detail and complexity that arises when conducting multiple analyses of multimodal corpora (both by individuals and by teams of scholars). Modelling of such levels of organization and their relations are being developed and implemented which effectively construe computational relations between the materiality of the expression planes within the software database and interface, the media which are studied and the various (manual and automated) analyses which the human analyst applies, as well as the resources for interrogating and visualizing such analyses.

It is one thing to discuss and even conceptualise or model the semiosphere in written or imagic (that is, page-based) discourse, either in general terms or as applied to consideration of actual semiosic activity within texts; it is entirely another to model the semiosphere in computational terms as an entity in relation to its parts: to show how specific analyses (annotations, algorithms) may be related to the holistic perspective of the semiotic universe that ‘makes the specific signatory act real’. The interactive digital environment demands more of the analyst than what Monelle (2000: 3) refers to (with respect to developing theoretical models for the semiotics of music) as “discursiveness and inconsistency…a bundle of essays”. Here, visualization and mathematical techniques to interrogate the data of complex and various analyses are proving potentially very useful in making the semiosphere ‘real’, in the sense of being an organizing concept for relating many different types of data and analyses to one another within a holistic perspective that can identify and/or deduce patterns and higher-level semiotic forms of organization in multimodal discourse. The software environment allows the analyst to collocate within the interface a variety of different analyses (including the results of various computational and mathematical operations on the data and upon semioticians’ analyses) in multiple configurations, such that larger patterns and correlations of patterns emerge. In this way that what may be otherwise accessible only through intuition or discursive exploration is grounded in, and in fact emerges from the empirical analysis of specific features of the text – of signs, and their manifold and complex interplay within texts.

Digital semiotics can of course be readily applied or interpreted as reductionist, particularly when automated (algorithmic) analyses of low-level features, such as shot and pitch detection, event, speaker, face and optical character recognition and detection, are considered. Manual analyses also, within the digital environment, may tend towards empirical descriptivism, as scholars apply annotation techniques in the immediate environment of the texts that defy the levels of abstraction customarily expected within state-of-the-art semiotics discourse. Yet this is the challenge that interactive digital resources offers the semiotician: to turn theories and concepts into some form of ‘material’ reality, or in fact into some symbolic form – into ‘digital signs’ – capable of representation within the software interface and database, capable of implementation within an appliable framework for analysis. The wealth of work within the semiotics tradition demands that, ultimately, digital analysis techniques must answer to the holistic higher-level perspective on a text as a unity, to the integration of signs within the semiosphere which they inhabit and which imbues signs with meaning.

Digital analysis of course must therefore always occur in tandem with more traditional ways of working, and will draw upon those for its materials and methods. The semiologist is always at the centre of the picture: one cannot hope for a digital software that will automatically produce insights such as those of a Hjelmslev, Barthes, Lotman or Halliday. But digital semiotics makes such insights more accessible to testing through text analysis, and provides a much greater capacity for insight into phenomena not readily accessible with the ‘naked eye’. Digital software is semiotic technology: with it, we find new ways to conduct semiotic research and create semiotics discourse. In this sense, we are expanding the potential for semioticians to (borrowing and adapting a phrase from Firth 1957) create multimodal semiosis about semiosis.  There is the potential for creating an analytically recursvive loop whereby our own annotations and other analyses become the site for further study, as we give our analyses signification within the database and interface: we can study our own analyses. This both allows us to detach from particular analytical or theoretical approaches, while discerning higher-level patterning revealed through the correlation of different analyses, through database search and retrieval, mathematical operations and visualisations, or simply through looking at visualisations of the analyses themselves.

In considering the potential of interactive digital software within semiotics and culture in general, we recall the observations of Lotman and Uspensky (1978: 229):

“The possibility of self-reduplication of metalanguage formations on an unlimited number of levels, along with the introduction of ever-new objects into the sphere of communication, forms culture’s reserve in information.”

Multimodal Analysis Lab Team

Kay O’Halloran, Roberto Mariani, Christel Loic Tisse, Alexey Podlasov, Bradley Smith, Arun Nagarajan, Stefano Fasciani, Qin Xiaoling, Richard Roussel, Bertrand Grandgeorge, Naing Thiha Htoon, Muthuveerappan Alagappan, Marissa E Kwan Lin, Melany Legaspi, Alvin Chua Yung Hwa, Ong Kian Peng (Bin), Rohanizah Ali, Sabine Tan, Monica Owyong, Liu Yu, Victor Lim Fei, Zhang Yiqiong, Feng Dezheng, Loh Boon Liang.

References

Multimodal Analysis Lab from JVP McL on Vimeo.