Cognitive Semiotics

Cognitive Semiotics (henceforth CS) is an emerging interdisciplinary matrix of (sub-parts of) disciplines and methods, focused on the multifaceted phenomenon of meaning. It is characterized by the ambition of “…integrating methods and theories developed in the disciplines of cognitive science with methods and theories developed in semiotics and the humanities, with the ultimate aim of providing new insights into the realm of human signification and its manifestation in cultural practices”, as stated on the home site of the journal Cognitive Semiotics: Multidisciplinary Journal on Mind and Meaning (www.cognitivesemiotics.com). This admittedly already broad definition could be further extended to include investigations of “non-human signification”. As shown below, while CS practitioners indeed focus on what is specific about human forms of meaning-making, there is widespread agreement that this can only be properly understood in a comparative and evolutionary framework.

Thus understood, CS cuts through and stretches across existing disciplinary divisions and configurations. For example, it is not to be seen as a branch of the overall field of semiotics, defined either in terms of “domain” (in the manner of e.g. biosemiotics, semiotics of culture or social semiotics), or “modality” (e.g. visual semiotics, text semiotics). Not belonging to a single discipline, it is not a particular “school” of semiotics (e.g. Peircean, Saussurean, Greimasian), and even less a particular theory (e.g. Existential Semiotics). Unfortunately, these are common misinterpretations of the label “cognitive semiotics”, given its instantiation of the modifier-head construction. But labels, while useful for organizing both concepts and fields of knowledge, are not essential. Indeed, many de facto CS practitioners do not attach the label “cognitive semiotics” to their research.

Finally, CS is not just a new and fancier name for (traditional) cognitive science. The relationship between the two interdisciplinary matrixes is complex and deserves more attention than can be given here. There is considerable overlap, and in a number of ways the relation is still open to negotiation. But in a nutshell, cognitive science has from its onset in the 1950s adopted an explicitly physicalist (computational and/or neuroscientific) take on mind, connecting to the humanities quite selectively, and above all to philosophers of mind with a distinctly reductionist bent (e.g. Dennett 1991). CS is considerably more pluralist in its ontological and methodological commitments, and thus arguably, with a firmer foot in the humanities than cognitive science.

The following two sections present a non-exhaustive survey of CS research, in order to give the reader a broad overview of the field. The first section is organized on the basis of groups and academic institutions that have provided a basis for the academic establishment of CS. Then the review turns to a few particular research areas, summarizing the work of key contributors. This overview is by no means fully comprehensive, but hopefully sufficiently so to give the reader an idea of the diverse set of topics under the prevue of CS, and the methods employed in investigating them. On this basis, a number of generalizations on what CS deals with and how it does so. Finally, we will return to the questions of why CS is needed and what its ultimate contributions to knowledge could be.

Researchers

Thomas Daddesio

Given that semiotics is usually defined as the study of signs, or more generally meaning, and the polysemy (and current popularity) of the term “cognitive”, just about any semiotic theory – from Peirce and Saussure to Eco (1999) and Hoffmeyer (1996) – could qualify as a “cognitive semiotics”. However, in the sense outlined in the introduction, CS truly appeared only in the mid-1990s. A pioneer that deserves to be acknowledged is Thomas Daddesio, whose major work bears the characteristic title On Minds and Symbols: The Relevance of Cognitive Science for Semiotics (Daddesio 1995).In it, the author sets out both a conceptual/methodological and an empirical goal for his project: namely, to “…demonstrate both the feasibility and utility of a cognitive approach to semiosis by setting forth a cognitive theory of symbols, which I will then apply to a particularly difficult area of inquiry, the development of symbolic communication in children” (ibid: 2). In a useful historical overview, the author shows how persistent attempts to “de-mentalize” notions such as sign, semiosis, and meaning in the 20th century contributed to a separation between semiotics and cognitive science. While “computation” and “information-processing” were the central concepts of the latter, there was not much to draw on for a “cognitive approach to semiosis”. But in the last two decades of the century, researchers from developmental and cognitive psychology (Bates, Bruner, Tomasello) and linguistics (Langacker, Talmy, Lakoff) turned increasingly to “experiential” notions such as schematization, (joint) attention, metaphor, and narrative. The ground was thus set for a rapprochement. Daddesio was one of the first to state this explicitly, and furthermore, bring in concepts from semiotics to dwell on a “particularly difficult area of inquiry”: children’s semiotic development. This was, of course, addressed insightfully by classics such as Piaget (1962) and Vygotsky (1962, 1978), but new concepts, methods of investigation, and a wealth of data have made it a very fruitful area, as described in the following section.

Centre for Semiotics (CfS), University of Aarhus

Around the same time and apparently independently, CS emerged at the Center for Semiotics in Århus, Denmark (http://www.hum.au.dk/semiotics/). The Center’s long-term research director, Per Aage Brandt, had in a number of publications combined ideas from the “dynamic semiotics” of René Thom and from cognitive linguistics of predominantly West-Coast US pedigree, involving notions such as “construal”, “force dynamics”, “image schemas”, and “conceptual blending”. These have been put to use mostly in the analysis of linguistic semantics, in particular with respect to puzzling phenomena such as subjectivity, iconicity, metaphor, and fictive motion. A major publication is that of P.Å. Brandt (2004) Spaces, Domains and Meanings: Essays in Cognitive Semiotics, whereCS is described as “a new discipline dedicated to the analysis of meaning”. A recent PhD dissertation, developing some of these ideas while drawing on the French tradition of “enunciation”, is that of L. Brandt (2010).

The work of another long-term member of CfS, Svend Østergaard, shows the growing influence of a cognitive – in the sense of psychological – approach to meaning. In The Mathematics of Meaning (1997), Østergaard discusses narration and temporality – as reflected in the classical literary works of Borges and Proust – seeking parallels with fundamental properties of mathematics such as infinity. More recently, Østergaard has turned to ideas from developmental psychology and the study of face-to-face interaction. Currently, Svend Østergaard, Kristian Tylén and Riccardo Fusaroli are collaboratively pursuing a “dynamical account of linguistic meaning making” combining conceptual models from dynamical systems theory and distributed cognition with corpus linguistics and experimental methodologies. Language is investigated as a coordinative activity, where symbolic patterns are aligned and negotiated to facilitate and constrain social coordination (e.g. Tylén et al. 2010; Fusaroli & Tylén 2012). The work of these researchers explicitly combines ideas from linguistics, semiotics, experimental psychology and neuroscience, thereby demonstrating that CS is ongoing practices and not just a programmatic enterprise.

Not all CS research needs to be experimental – or even “empirical” in the narrow sense of observation-based – as shown by the work of Peer Bundgaard in his articles on image schemas, and force dynamics (Routledge Companion to Semiotics, 2009), Husserl’s theory of language (Bundgaard 2010), and aesthetic cognition. The current research director of the CfS, Frederik Stjernfelt, likewise pursues a more purely “qualitative” tradition of conceptual analysis (not in the narrow linguistic sense), including interpretations of Peirce’s ideas on icons and above all diagrams, linking these to Husserl’s phenomenology (Stjernfelt 2007). At the same time, both Bundgaard and Stjernfelt apply their semiotic analyses to empirical phenomena of concern for CS, such as aesthetics, mental imagery, animal communication, and human gestures.

CfS is the only academic institution so far offering an MA program in CS (both in name and content): “Cognitive Semiotics is first and foremost an interdisciplinary program which draws on neuroscience, philosophy, logic, linguistics, anthropology, cognitive science and literary theory” (http://www.hum.au.dk/semiotics/). The program has an impressive number of students and guest lecturers, and contributes to the reputation of CfS as a vanguard of the field. Still, an “emerging paradigm” can hardly be confined to one or two (geographically close) institutions.

Centre for Cognition and Culture (CCC), Case Western Reserve, Cleveland

At the beginning of the millennium, Per Aage Brandt relocated to Case Western Reserve, where the Department of Cognitive Science was headed by Mark Turner, one of the authors of the cognitive semantic “blending theory” (Fauconnier & Turner 2002). Together with Todd Oakley, Brandt established the Centre for Cognition and Culture (CCC), which “…studies art, design, music, language – both as grammar, as text, as literature, and as speech and discourse – sign structures and communicative meaning production in general, differentiated and variable within the unifying potential of the human mind – and applies to this effect a comparative methodology that can be characterized as semiotic in a cognitive perspective: as a cognitive semiotics” (http://www.case.edu/artsci/cogs/CenterforCognitionandCulture.html). Todd Oakley integrated cognitive linguistic concepts with a thorough investigation of the role of attention processes in a recent monograph: From Attention to Meaning: Explorations in Semiotics, Linguistics, and Rhetoric (2008). As suggested by the title, his analyses concern a much wider range of phenomena than the usual “blending” analyses of standard examples such as “my surgeon is a butcher”.

The most notable result from the collaboration between Oakley and Brandt was the establishment of the journal Cognitive Semiotics, which began in 2007. The volumes published so far have been devoted to topics such as agency, consciousness, and cognitive poetics, and have featured prominent authors from both the cognitive sciences and the humanities. However, the journal has not so far received a broad readership, and its current “impact factor” can be much improved.

Centre for Language, Cognition and Mentality (LaCoMe), Copenhagen Business School

An interdisciplinary group – departing from linguistics while expanding to visual communication, gesture, and behavioral studies on consumer preferences – was established in 2007 at the Copenhagen Business School, with Per-Durst Andersen as research director (). The focus multimodal communication has been enhanced by the group’s involvement in the cross-disciplinary FairSpeak project led by Viktor Smith (see below). Søren Brier joined the group, coming from a background in ethology and cybernetics and bringing in an evolutionary and system-theoretic perspective. Brier’s book Cybersemiotics: Why Information is Not Enough (Brier 2008) presents an ambitious attempt to achieve a synthesis of Peircean semiotics and second-order cybernetics, with the aspiration of unifying all domains of human knowing: from those of the physical and biological to the subjective/personal and the intersubjective/cultural. What remains is to spell out how such a framework can change the day-to-day practice of scientists, in particular of linguists and psychologists concerned with notions such as meaning construction and sense making. Brier is pursuing efforts in this direction in collaboration with Ole Nedergaard and other members of the group.

Per Durst-Andersen recently crowned a long period of research in “language, cognition, and mentality” with a theoretical synthesis: the monograph Linguistic Supertypes: A Cognitive-Semiotic Theory of Human Communication (2011). At the center is a linguistic sign concept inspired by the trichotomies of Peirce and Bühler. Durst-Andersen proposes that the grammatical meanings of any particular language tend to orient towards one of the three semiotic poles: Reality, Speaker and Hearer and thus that all languages can be characterized as belonging to one of three “linguistic supertypes”. This controversial proposal is supported by a good deal of linguistic evidence, as well as references to research within cognitive psychology. Empirical studies – e.g., on predicted cognitive differences between speakers of the different language-types along the lines of “linguistic relativity” research – are underway.

The research of Viktor Smith, the third prominent member of the group, insists on combining the “top down” approach characterizing much linguistic theorizing with a “bottom up” understanding of how linguistic communication functions in interaction with other semiotic resources such as pictures and sensory impressions (Smith, Møgelvang-Hansen & Hyldig 2010; Smith, Barratt & Zlatev, under review). Smith’s point of departure is the lexicon in its capacity as a key element in all known manifestations of human language, and a powerful tool for interacting with and shaping the world. His concern for bridging CS matters to “the real world” is manifest e.g. in the FairSpeak project focusing on the communicative potential and fairness of product packaging design. In this project, legal-normative, experiential, and behavioral aspects of food labeling and marketing are being brought together, with the aim of improving producer-consumer communication (Smith et al. 2009, 2011).

Centre for Cognitive Semiotics (CCS), Lund University

The Centre for Cognitive Semiotics at Lund University was established as a six-year program (2009-2014), bringing together researchers from semiotics, linguistics, cognitive science, and related disciplines on a common meta-theoretical platform of concepts, methods, and shared empirical data (http://project.ht.lu.se/en/ccs/). A staff of 10-15 senior and post-doctoral researchers and a larger number of affiliates coordinate their research under five interrelated themes – evolution, ontogeny, history, typology, and experimental semiotics – adopting as much as possible a CS approach to their specific topic. For example, the typology theme deals not only with linguistic typology but also with patterns of correlation in multiple “semiotic resources” such as speech, writing, gestures, pictures, music, and cultural artifacts.

The research director of CCS, Göran Sonesson, states: “I have been involved with phenomenological cognitive semiotics from the very start of my career without knowing it – or rather, without using the term” (Sonesson 2009: 108). Sonesson’s writings since the late 1970s, in particular his comprehensive monograph Pictorial Concepts (1989), can indeed be seen as forerunners of CS in several respects. In particular, he has consistently argued for the primacy of perceptual meaning over other kinds of meaning – including signs – and elaborated a definition of the sign concept on the basis of phenomenological notions such as experienced asymmetry and differentiation. At the same time, Sonesson has maintained that the study of meaning cannot be purely “eidetic” or “autonomous” but must also be based on psychological studies. For the purposes of his analyses of pictorial signs (his specialty), he often refers to Gestalt psychology as well as the ecological psychology of the Gibsonian tradition.

Still, CS cannot be based only on a meta-analysis of the results of the cognitive sciences; for it to come into its own, it should go hand in hand with them to motivate specific empirical studies. In this sense, CS research at Lund University got underway during the first years of the millennium, thanks to collaboration between Sonesson and researchers from linguistics such as Zlatev and cognitive scientists, such as Tomas Persson, a primatologist who applies CS concepts to the study of visual perception and pictorial competence in non-human primates (Persson 2008).

Zlatev’s road towards CS has been guided by the conviction that language – its nature, evolution and development – cannot be understood outside the context of a more general approach, taking both meaning and mind seriously. Influenced by the work of Merlin Donald (see below), Zlatev has tried to elaborate the concept of bodily mimesis, arguing for its central role in both ontogeny and evolution (e.g., Zlatev 2008). Furthermore, Zlatev has struggled with the proverbially “hard problem” of consciousness. In agreement with Sonesson, Zlatev sees phenomenology as providing tools to address the complex interrelations between bodily experience, sociality, and language (Zlatev 2010). Consistent with the work of Thompson (2007, see below), Zlatev has formulated an evolutionary “semiotic hierarchy”: the autopoiesis of living systems is at the basis of all meaning in the universe, followed by the emergence of conscious experience (at least with mammals), which on its side is a precondition for the evolution of sign use (emerging with Homo erectus) and speech (in our own species). Such evolution is essentially bio-cultural, with cultural processes playing a leading role in the evolution of language.

A number of empirical studies on mimetic schemas and children’s gestural development have been carried out (e.g., Zlatev and Andrén 2009). Mats Andrén’s (2010) PhD Thesis Children’s Gestures Between 18 and 30 Months is the group’s most synthetic result so far. In it, Andrén provides detailed descriptions of five Swedish children’s gestural repertoires in the tradition of Adam Kendon (see below), with CS concepts serving to delineate gestures from action and “body language” on the one hand and from signed language on the other. Quantitative analyses show patterns in the developmental trajectories of pointing, iconic, and emblematic gestures with respect to speech and the use of physical objects. The study substantiates claims for an intimate interrelation between and parallel development of speech and gesture.

For reasons of space and fairness, the research of all CCS researchers cannot be summarized here. To give a flavor of the variety of subjects pursued, we can mention the research by Gerd Carling and Arthur Holmer on correlations between linguistic and other semiotic resources in Amazonia; Junichi Toyota on possible interactions between religious beliefs and tense-aspect systems; Anastasia Karlsson on prosody and information structure in East Asian and Southeast Asian languages; Sara Lenninger on the development of children’s use of pictures; Anna Cabak Rédei and Lars Kopp on visual perception and emotion; Gunnar Sandin on the affordances and signs of city architecture; Joel Parthemore on enactive concepts; Elainie Madsen on non-manual forms of pointing; Michael Ranta on visual narratives… If successfully integrated – the major challenge to CCS – such research can serve as the basis for a viable CS tradition at Lund University. As a further step, it will be necessary to secure the program’s “cultural transmission” through an MA program such as that of CfS.

Topics

Gesture

The study of gestures – involving various degrees and kinds of iconicity, indexicality, and conventionality – has from the start called for a more or less explicit semiotic analysis (cf. Kendon 2004). Efron (1941) and later Bouissac (1973) provided some early proposals for how such analyses could be made more systematic, in part through the availability of new technology for recording and analysis. During the 1980s, thanks to the concerted work of Adam Kendon (1980, 2004) and David McNeill (1992, 2005), gesture studies began to emerge as a more or less independent interdisciplinary field, with an International Association for Gestures Studies (ISGS) organizing regular bi-annual conferences since 2002.

McNeill’s approach to gesture is more explicitly cognitive, with references to developmental and neuroscientific evidence and links to cognitive linguistic concepts such as image schemas and conceptual metaphors. His long-time concern is the integration of gesture and speech in a single cognitive system, though with a degree of division of semiotic labor: gesture being in his terms more “imagistic” and speech/language more propositional. In Gesture and Thought (2005), McNeill echoes Vygotsky’s classic Language and Thought and argues for a broader concept of language, combining the more static and systematic aspects of Saussure’s langue with a more dynamic and imagistic side, made visible above all through gesture.

At the same time, it appears that the influence of Kendon’s work runs deeper, both for gesture studies and CS. Originally working in ethology and then in human interaction, Kendon has over the years developed a framework – or perhaps a style – of analyzing live, multimodal interaction that is difficult to match in terms of sensitivity to relevant detail. Combined with what he refers to as a “comparative semiotic” method, his studies of face-to-face interaction, alternate signed languages in Australian aborigines, and gestures of Neapolitaneans are considered classics in the field. Many of these are summarized in his magnum opus Gesture: Visible Action as Utterance (2004). Kendon is also editor of the journal Gesture since its inception.

Cornelia Müller, head of the Berlin Gesture Centre, is perhaps the most prominent inheritor of the different strands in gesture studies – from linguistics and semiotics to neuroscience and primatology – and thus clearly qualifies as a practitioner of CS. This can be seen from an ongoing project, Towards a Grammar of Gesture: Evolution, Brain, and Linguistic Structures, which aims at “the development of fundamentals for a multimodal grammar and its neurological and evolutionary foundation within specific sub-areas” (http://www.togog.org/en/). A forthcoming publication: “Gestural modes of mimesis: Mimetic techniques and cognitive-semiotic processes driving gesture creation” make the connection to CS even more explicit. Her work further concerns what is sometimes called “multimodal metaphor”, a topic on which she has collaborated with Alan Cienki (Cienki & Müller 2008).

Child development

As mentioned in the introduction, the pioneering figures in developmental psychology clearly adopted a cognitive-semiotic approach by investigating interrelations between sensorimotor skills, imitation, imagination, and communicative signs (Piaget); or between thought, “inner speech”, and the semiotic mediation of cognition and development by socio-culturally transmitted sign systems (Vygotsky). This tradition underwent a significant renewal in the 1970s through the work of (among others) Bruner and Trevarthen. Subsequently, however, the child’s mind was “modularized” as it became unfashionable to look for “domain general” capacities, stages, and transitions. Language and cognition were to be kept apart and studied separately. If we fast-forward to the present, the picture looks quite different, with body, affect, and socio-cultural environment all seen as indispensable for growing minds. Colwyn Trevarthen’s long-term research and theorizing on infant and child intersubjectivity (Trevarthen 1979; Bråten and Trevarthen 2007) has been one of the key inspirations for this turn. In collaboration with Stein Bråten and others, Trevarthen has described the first years of development as characterized by increasingly complex layers or levels of intersubjective engagement with others in “trusting relations of companionship” (see Table 1). Inspired by the work of Julia Kristeva, Lüdtke (2012) adds to these a “zero layer” of primordial intersubjectivity preceding birth; she conceptualizes the progression as one of decreasing corporeality and emotional markedness with increasing degrees of abstraction and referentiality. Stern (2000) has likewise emphasized interpersonal relations and emotion, contributing to puncturing (if not tearing down) the wall between therapeutic and cognitive psychology – thereby making it possible to argue that emotional contact and sympathetic interaction serve as “the cradle of thought” (Hobson 1996).

Table 1. Levels of intersubjectivity in the first years of development, adapted from Bråten (2007: 3).

Somewhat less concerned with empathy, and more with sharing cultural meanings, are developmentalists such as Jerome Bruner – whose Acts of Meaning (1990), investigating children’s development of autobiographical memory (and self-concept) through the help of “stories”, marked a turning point for some practitioners of cognitive science.

From the side of semiotics proper, development has been insightfully addressed by Patricia Violi. Inspired by the philosophy of Merleau-Ponty, Violi (in press) argues for an extended sense of “embodiment”, in which the body itself becomes enculturated, as well as “extended” through artifacts.

Bio-cultural evolution

As long recognized, there is an intimate relationship between the development of individuals and the evolution of species. One of the insights of the “new synthesis” of developmental and evolutionary biology (evo-devo) is that “all important changes in evolution are alternations in development” (Thompson 2007: 195). Modern concepts of evolution have moved beyond the (ex-) “modern synthesis” focused on gene selection, to consider that evolution can take place on several different levels (including that of social groups), combing the effects of biological and cultural evolution (Richerson & Boyd 2005).

Several theoreticians with a background in neuropsychology and developmental psychology have addressed the perennial question of the “descent of man” within an extended, bio-cultural perspective on evolution, often explicitly involving concepts from semiotics. An important publication in the area is Merlin Donald’s (1991) Origins of the Modern Mind: Three Stages in the Evolution of Human Culture, presenting an integrated bio-cultural theory of human evolution. A key idea is that a domain-general capacity for skill learning, imitation, and gestural communication lies at the roots of uniquely human cognition and semiosis: “Mimetic skills or mimesis rests on the ability to produce conscious, self-initiated, representational acts that are intentional but not linguistic”(Donald 1991: 168). Speech and language evolved only later, partly through cultural evolution, without relying on innate adaptations. External representations gave rise to writing in relatively recent history, making what Donald calls “theoretical culture” possible. Even from this brief summary, it can be seen that Donald’s approach is clearly cognitive-semiotic: the goal is to understand not only the “origins of the modern mind” but how new semiotic layers have transformed that mind into the unique “hybrid” construction that it is (see also Donald 2001). The role of artifacts, external representations and technology in general for “supersizing the mind” (Clark 2008) has been discussed for some time, and is to some extent generally acknowledged. However, the more precise nature of artifacts and technology in relation to thinking has been the subject of controversies in philosophy (“internalism vs. externalism”) and cognitive science (“extended mind”) and can therefore be pinpointed as a target area for future CS research.

Terry Deacon’s work in evolutionary anthropology relates explicitly to semiotic theory. His widely influential book The Symbolic Species: The Co-Evolution of Language and the Brain (1997) draws on ideas from Peirce to propose that interpretative processes follow a progression of iconism (i.e. recognition), indexicality (space-time contiguity, as in the pairing of stimulus and response in classical conditioning), and most complexly – unique to our species – symbols. What Deacon exactly means by “symbols” has been a matter of much discussion. He has attempted to clarify this recently: “To interpret the wax impression as a symbol of social position, one must also understand these social conventions, because nothing intrinsic to the form or its physical creation supplies this information. The symbolic reference is dependent on already knowing something beyond any features embodied in this sign vehicle” (Deacon 2012). Thus, it is not arbitrariness per se that makes a sign into a symbol but culturally shared knowledge, which Deacon often describes as constituting a “web of symbolic relationships” – at least implicitly drawing on the structuralist tradition emanating from Saussure (cf. Sonesson 2006). Furthermore, he has introduced the intriguing notion of semiotic constraints that are neither innate nor learned but a priori features of symbolic reference. Through such constraints, Deacon proposes to account for language universals such as predication and recursion. A final key concept to his evolutionary theory of human origins is relaxed selection, which implies that rather than becoming more genetically determined, our brains have become less so: thus, more flexible and adaptive to the different cultural niches we live in.

Michael Tomasello must also be mentioned as representative of this research area, with his important contributions directing experimental research in developmental and comparative psychology at the Max Planck Institute for Evolutionary Anthropology in Leipzig since the mid-1990s. His two major publications over this period, The Cultural Origins of Human Cognition (1999) and The Evolution of Human Communication (2008), have likewise proved influential. Tomasello prefers to stay in a more mainstream current of psychology and cognitive science and refrains from using terms such as “signs”, “semiosis”, and “consciousness”. Still, his key concepts include symbols, joint attention and shared intentionality and it is easy to see his theories in a CS context. Being heavily dependent on experimental results, Tomasello’s ideas have changed over the years. Human-specific cognition is no longer characterized by “understanding intentions” but rather by a combination of motivational factors for sharing (from food to attention and knowledge) and a cognitive capacity for maintaining joint commitments. Thus, it can be said that Tomasello’s ideas on infant intersubjectivity have largely converged with those of Trevarthen, outlined earlier. In emphasizing the role of gestures in establishing a basis for language evolution, Tomasello’s evolutionary theory is also similar to that of Donald’s.

Thus, there appears to be an emerging consensus that what is distinct to our species – both cognitively and semiotically – is a unique form of sociality. Still, few have attempted an explanation of the evolutionary conditions that would lead to this. Deacon (1997) has speculated that it could have been a change in reproductive strategy: from polygamy (typical among the great apes) to monogamy. This, however, is unsupported by archeological evidence and at least controversial for the anthropological evidence: (serial) monogamy seems a much more recent, culturally transmitted, non-universal phenomenon. A more persuasive argument for the evolution of a human-specific form of intersubjectivity is presented by Sarah Hrdy in Mothers and Others: The Evolutionary Origins of Mutual Understanding (2009). Reviewing the ethological, anthropological, and developmental psychology literatures, Hrdy builds up a case for the proposal that the crucial reproductive turn that occurred with Homo erectus nearly two million years ago was not to monogamy but to alloparenting or “cooperative breeding”. That would account both for the greater gregariousness of our species towards non-relatives and the willingness of infants to bond and communicate with other than biological parents.

Embodiment

In parallel with – and similar to – the rapprochement between the cognitive sciences on the one hand and “semiotics and the humanities” on the other, as outlined above, there has been a movement of integrating ideas and methods from cybernetics, theoretical biology, and phenomenology, at least since the publication of The Embodied Mind: Cognitive Science and Human Experience (1991) by Francisco Varela, Evan Thompson, and Eleanor Rosch. According to the broad definition involving “integrating methods and theories” offered in the introductory passage, this tradition could even be seen as falling under CS. Unfortunately – at least until recently – there has been little interaction between Embodied Mind scholars and those more overtly involved in CS. Perhaps this is due to the radically anti-representationalist stance in the early stages of the Embodied Mind movement, when the central concept was that of enaction: “a history of structural coupling that brings forth a world… [t]hrough a network consisting of multiple levels of interconnected, sensorimotor subnetworks” (Varela, Thompson & Rosch 1991: 206). Rejecting the excessive (unconscious) representationalism of standard cognitive science (i.e., cognitivism), the enactivists were suspicious of any concept that sounded similar to representation, such as that of sign. Their empirical focus was on the direct experience of perception and action and on resolving the “hard problem” of consciousness – not on sign-mediated meaning. More recently, however, with the addressing of topics such as mental imagery and enculturation (Thompson 2007) as well as gesture (Gallagher 2005), it has become obvious that the classical phenomenological distinction between presentation and representation needs to be respected and theoretically addressed. From the CS side, phenomenologically oriented semioticians such as Sonesson (2011) have made similar arguments, while focusing on the representational (e.g., pictorial) aspects of meaning. Given the mutually consistent, complementary and anti-reductionist orientations of the CS and Embodied Mind approaches, one should expect to see more interaction between them in the near future. Here, I only mention the names and work of a few prominent figures – the first, Francisco Varela, prematurely deceased.

Varela played a key role in establishing the Embodied Mind paradigm. With his background in theoretical biology and in collaboration with Humberto Maturana, Varela co-authored some of the key ideas of autopoiesis theory: “Our proposition is that living beings are characterized in that, literally, they are continually self-producing. We indicate this process when we call the organization that defines them an autopoietic organization” (Maturana & Varela 1987: 43). For reasons that still need to be clarified, there was a rift between the two scholars around that time. Varela proceeded to elaborate the related notion of enaction (Varela et al. 1991) and, importantly, to link his biological theory with a deeper appreciation of phenomenology than present in the 1991 volume. In an oft-quoted paper, Varela (1996) formulated the research program of neurophenomomenology, in which first-person data, obtained by experimental subjects trained to be aware of and reflect on their experiences – i.e., to “perform the phenomenological reduction” – was to be correlated with the third-person data of brain imaging. A number of insightful studies have used, and elaborated on this framework (cf. Lutz and Thompson 2003).

With Varela’s untimely death, Evan Thompson picked up the torch in formulating a new synthesis for mind science, culminating in his impressive Mind in Life: Biology, Phenomenology and the Sciences of Mind (2007). The major theme of the book is “the deep continuity of life and mind”, expanding on the notion of autopoiesis as the minimal condition for both life and meaning and pre-figuring the basic structures of consciousness, such as intentionality (cf. Zlatev 2009). Throughout the book, Thompson skillfully weaves together ideas and findings from “biology, phenomenology and the sciences of mind”, addressing topics such as time consciousness, mental imagery, emotions, and intersubjectivity. On that last point – influenced by the work of the phenomenologist Dan Zahavi (2001, 2003), who has successfully argued that Husserl’s mature work included a rich analysis of “being with others” and the lifeworld – Thompson enriches the methodological pluralism of neurophenomenology, arguing the need for “second-person methods” in the study of consciousness and meaning. As spelled out below, such theoretical and methodological “triangulation” is characteristic of CS research.

Meanwhile, Shaun Gallagher has elaborated upon a central theme of phenomenology, associated most often with Merleau-Ponty (1962), by combining it with empirical and, above all, clinical research: that of the central role of the living body for all forms of experience and meaning. In How the Body Shapes the Mind (2005), Gallagher elaborates experiential distinctions such as those between body schema and body image and between bodily agency and ownership, showing that by “front loading” phenomenology in experimental research – rather than using it to interpret existing findings – one can achieve a productive interaction between first- and third-person methodologies. Gallagher has criticized the traditional “theory of mind” perspective on social cognition – both of the theory-theory and simulation-theory varieties – proposing instead an “enactive” interaction theory in which basic interpersonal understanding is the product of perception and action processes, while more elaborate understanding of others’ motives and goals is due to a shared familiarity with narratives – as also proposed by Daniel Hutto (2008). Together with Zahavi, Gallagher has published The Phenomenological Mind: An Introduction to Philosophy of Mind and Cognitive Science (2008), in which many of the basic ideas of phenomenology – along with empirical applications – are presented to a broader audience. This is something that CS would clearly benefit from emulating.

General characteristics

On the basis of the (non-comprehensive) overview in the previous two sections, one can discern a number of characteristics of CS research. These can serve to narrow down the broad definition of CS as “integrating methods and theories developed in… cognitive science with methods and theories developed in semiotics and the humanities”. At the same time, they are not meant as a “classical” definition in terms of necessary and sufficient conditions but rather as a prototype-based one: i.e., not every CS practitioner should be expected to commit to all of the following five features.

A productive combination of (semiotic) theory and empirical research

In a broad (and trivial) sense, all research is both theoretical and empirical. However, semiotic theory is particularly concerned with explicating higher-order concepts such as meaning, sign use, representation, language, intersubjectivity,etc., along with their interrelations. It is anything but trivial to bring in empirical research that both contributes to such an explication and, at the same time, benefits from it in a way that produces new insights. It is such “mutual enlightenment” – in the words of Evan Thompson – that is the central characteristic of CS. All who have been involved in the study of phenomena such as imagination, gesture, metaphor, etc. will know that it is anything but trivial to combine conceptual and empirical analyses of their nature. There is a natural pull, one could say, to treat these as meaningful phenomena and explicate their features, constituent structures, types, etc. by engaging in systematic conceptual/eidetic analysis. On the other hand, psychologists tend to rush to “operationalize” the concepts, formulate hypotheses, perform experiments, and arrive at theoretical conclusions. But the outcome has often been that behind the same terms (e.g. ‘imagery’, ‘motion’, ‘symbol’) very different, and often diffuse, concepts have been lurking, with resultant cross-talk both across and within disciplines.

How is CS to avoid this? The answer lies in formulating concrete research programs (such as neurophenomenology, mentioned above) that not only state programmatically that the “methods and theories” of the humanities and sciences need to be integrated but actually go ahead and do it. This is important enough to be listed as separate feature.

Methodological triangulation

Essential for CS is the kind of methodological “triangulation” shown in Table 2 (cf. Zlatev 2009). Rather than battling over the proper methods for investigating the object of study, as has been done for over a century in linguistics (e.g., whether or not to use native-speaker intuitions), or define fields on the basis of their respective methods (philosophy as first-person, ethnomethodology as second-person, classical sociology and experimental psychology as third-person, etc.), the goals of methodological triangulation are (a) to acknowledge the validity of all methods within their respective domain of inquiry, (b) to acknowledge the epistemological priority of first- and second-person methods in the study of meaning (since what one wishes causally to explain must first be understood as well as possible, in order to avoid the cross-talk mentioned above), and (c) to integrate the three kinds of methods in the same project (e.g., Andrén 2010).

Table 2. The central task of Cognitive Semiotics: integrating methods, derived from one of the three perspectives, in the study of particular semiotic phenomena, along with their interrelation.

From the perspective of CS, the problem with the “classical” humanities has been a resolute rejection of third-person methods in the study of cultural world as, at best, limited, and at worse as “objectivist” and distorting of the phenomena. While much can be said in favor of such a critique, the steady progress of the sciences, including the study of the “mind/brain”, has given such an attitude a distinctly old-fashioned – if not reactionary – flavor. But on its side, (natural) science has tended to be myopic and dogmatic and has, unsurprisingly, hit a wall in extending the Galilean method to issues of value, meaning, norm and consciousness. It has also performed first-person and second-person methods implicitly, often without knowing it: you will not find sections on the use of intuition and empathy in the “methods” section of experimental psychology textbooks.

The challenges to success in practicing such non-reductive unification of knowledge are many – not the least institutional. CS runs the risk of being caught it the crossfire between the traditionalism of the humanities and the hubris of the sciences. But on the positive side, CS could make a contribution to “mending the gap between science and the humanities”: the subtitle of the last book of the evolutionary scientist Stephen Jay Gould (2003).

Influence of phenomenology

Another common aspect to most CS research is a greater or lesser degree of indebtedness to the philosophical school of phenomenology, as founded by Edmund Husserl at the beginning of the 20th century. There are multiple schools and types of phenomenology, but the basic idea is to depart from experience itself, and to provide descriptions of the phenomena of the world, including ourselves and others, as true to experience as possible – rather than constructing metaphysical doctrines, following formal procedures, or postulating invisible-to-consciousness causal mechanisms that would somehow “produce” experience.

There is continuity between the epistemological challenges of CS outlined above, and those dealt with by Husserl, leading him to develop phenomenology as a possible resolution to what he called the “crisis of European sciences”, caught between the extremes of positivism and relativism. The emphasis on perspective in Table 2 was meant as a reminder that all knowledge is relative to a subject – or an “observer” as Humberto Maturana likes to phrase it (though not as dependent on language as assumed in his theory). This does not entail any form of “monadic” subjectivism for at least three reasons. First, we do not live in separate bubbles made up of “representations”, but in a meaningful lifeworld, co-constituted through our perceptions and actions. This is obvious for cultural meanings, such as those of language, but it applies also to the most basic layers of perception (e.g., of color). Second, even the most subjective experience is communicable – on the type if not token level – “to sympathetic others” (Trevarthen). Third, accepting that the structures of experience as elucidated by phenomenology are “prefigured” in the principles of life itself – as argued by Thompson and others – opens the way towards a naturalization of phenomenology without the reductionism that usually goes with that term.

Apart from an affinity in its epistemological foundations, CS has benefited from phenomenology with respect to specific topic areas: the above-mentioned distinction between presentation and representation, analyses of imagination and “picture consciousness” (Stjernfelt 2007; Sonesson 1989; 2011), of the interrelations between the living body (Körper) and the lived body (Leib) (Gallagher 2005), of intersubjectivity (Zlatev et al. 2008), etc. What would seem to be a natural next step is to take stock of the more dynamic “genetic” (individual) and “generative” (cultural) developments of phenomenology, including analyses of time consciousness (understood as the fundamentally temporal nature of all experience), passive synthesis (opening the door to analyses of the “unconscious”), sedimentation (i.e., of cultural knowledge), etc. That would be consistent with the otherwise strong emphasis on dynamics, prevalent enough to deserve to be listed as a characteristic of CS.

Dynamism of meaning

At the risk of using a notion that has reached almost fetish status during the last decades (“everything changes, nothing is static”), one can make the generalization that CS studies meaning on all levels – from perception to language, along with the various forms of “external”, cultural representations (theatre, music, pictures, film, etc.) – primarily as dynamic processes rather than static products. Though the latter can be a convenient descriptive shorthand (e.g., of the “lexicon” of a language, or the “repertoire” of gestures in a community), nearly all CS scholars have made the point that viewing meaning in purely static, structural terms is insufficient for understanding the essentially relational, subject-relative, and (often) interpretive nature of semiosis. Unsurprisingly, various formulations have been used to capture the dynamic nature of meaning: sense-making (Thompson), meaning construction (Oakley), languaging (Maturana), etc. It may also be reminded that the CfS scholars used the term “dynamic semiotics” prior to adopting “cognitive semiotics”. Thompson (2007) refers to the framework that he is developing as “embodied dynamism”.

There are at least five different time scales to the dynamic semiotic processes under study: (a) microseconds in the study of the emergence of the moment-to-moment experience of meaning(-fullness) as in vision or speech; (b) seconds in the study of the production and understanding of meaningful wholes such as scenes and (oral and gestural) utterances; (c) days, months, years in the study of semiotic development in ontogenesis; (d) decades, centuries in the study of cultural-historic processes, as in language change and sociogenesis; and (e) millennia in the study of biological evolution (i.e., phylogenesis). The levels on which these processes apply are also various, from those of “subpersonal” processes in brains to conscious experience in individuals to co-constructions of meaning in dyads and groups to changes in whole populations and environments.

These are fairly standard scales and levels, not specific to CS. Perhaps what could be seen as criterial for a CS approach to any particular phenomenon (e.g., visual perception, gesture interpretation, or identity formation) is not to focus on a single time scale – and corresponding epistemological approach – but to consider several scales/levels in relation to one another (cf. Andrén 2010). In line with the point about the relational character of meaning, a basic CS tenet is that meaning is not “inside” brains, minds, groups, or communities but is a result of processes of self/other/world interaction.

Transdisciplinarity

In the opening line of this article, CS was preliminarily defined as an “interdisciplinary matrix of (sub-parts of) disciplines and methods”. Keeping to this definition, I discussed the combination of methods and levels of analysis. So: what are the “(sub-parts of) disciplines” involved? Judging from the background of CS practitioners, one can single out (1) semiotics (whether or not it should be seen as a single discipline), (2) linguistics (approaches viewing meaning as the essence of language), (3) psychology (mostly developmental, but also cultural, cognitive, and comparative), (4) anthropology (biological and, hopefully, cultural, despite its deeply ingrained resistance to “biologism”), (5) enactive cognitive science (including neuroscientific and dynamic modeling approaches), and (6) philosophy (above all, in the phenomenological tradition).

These are almost the same list of disciplines that combined forces to define cognitive science in the 1960s. But as stated in the introduction – and hopefully made clear in the article – the new synthesis of CS is quite different. For one thing, the “sub-parts of disciplines” involved in CS are often viewed as antagonistic to those that participated in the synthesis of cognitive science: so one finds cognitive vs. generative linguistics, epigenesis vs. nativism, enactivism vs. cognitivism, phenomenology vs. physicalism. At the same time, such oppositional thinking – and thus opposing CS to cognitive science – is much too schematic. After all, we are participants in ongoing processes of dynamic transformations of society, technology, and attitudes towards knowledge. While cognitive science may seem much more academically established than CS in terms of societies, journals, academic departments, and educational programs, it has not evolved into a self-sufficient discipline and remains in essence an interdisciplinary program with various constellations crystalizing as “paradigms” for a limited period of time: Varela et al. (1991) portray its brief history as passing through the stages of cognitivism, connectionism, and enactivism. With a little good will, CS could even be seen as a fourth stage.

More important for the self-definition of CS is whether it should involve a lower or higher degree of interdisciplinarity. A higher degree is often called transdisciplinarity, especially by those who see “interdisciplinarity” as a temporary coalition between members of different fields when something of considerable complexity is addressed (e.g., the brain as studied by neuroscience or evolution as studied by sociobiology) but without seriously affecting the participant disciplines or the broader field of knowledge. In contrast, transdisciplinarity “concerns that which is at once between the disciplines, across the different disciplines, and beyond each individual discipline. Its goal is the understanding of the present world, of which one of the imperatives is the overarching unity of knowledge” (Transdisciplinarity Wikipedia). From such a perspective, CS can be seen as an emerging transdisciplinary field: meaning does not constitute a specific empirical domain but rather cuts “between and across” disciplines. What has so far lain “beyond” is a coherent approach that “mends the gap between science and the humanities”, in the words of Gould. As stated by Zlatev (2003: 253): “Our conception of meaning has become increasingly fragmented, along with much else in the increasing ‘postmodernization’ of our worldview. The trenches run deep between different kinds of meaning theories: mentalist, behaviorist, (neural) reductionist, (social) constructivist, functionalist, formalist, computationalist, deflationist… And they are so deep that a rational debate between the different camps seems impossible. The concept is treated not only differently but incommensurably within the different disciplines”. To the extent that CS lives up to the challenge of providing a coherent worldview uniting “biology, phenomenology and the sciences of mind” (in the words of Thompson) and even offering a foundation for the systematic study of fields such as visual art and music, it would deserve the label “transdisciplinary field”.

Furthermore, a feature often seen as crucial for transdisciplinary research is “the inclusion of stakeholders in defining research objectives and strategies in order to better incorporate the diffusion of learning produced by the research. Collaboration between stakeholders is deemed essential – not merely at an academic or disciplinary collaboration level, but through active collaboration with people affected by the research and community-based stakeholders” (Transdisciplinarity Wikipedia). It is fair to say that, so far, CS has not achieved this, though there have been encouraging first attempts: Smith’s work with producers, consumer rights advocates, and legal experts in the Fairspeak project; work in Lund with minorities, such as the Roma, on issues of group identity and integration; work in Århus on multiculturalism. Areas of crucial social significance, in which CS – with its participatory approach to knowledge – should be able to involve stakeholders include atypical development (e.g., autism), sex and gender, animal rights, and religion: notably all highly “sensitive” domains characterized by polarized views. Clearly, an approach such as CS, with its promise of mending the gap, could be beneficial.

Conclusions

The fact that similar ideas – and even the term ‘cognitive semiotics’ itself – have emerged in different places over the last decades is hardly a coincidence. At some risk of exaggeration, CS can be seen as called for by historical needs, such as those suggested in this article: the need to unify or at least to “defragment” our world-views, the need to come to terms with increasingly higher levels of dynamism and complexity, the need to understand better – and thus deal with – the dialectical relationship between individual freedom (autonomy) and collective dependence (sociality), etc.

In other words, if Cognitive Semiotics did not exist, we would need to invent it. Its potential as a transdisciplinary field integrating our understanding of life, mind, language and society is considerable. Furthermore, it can help integrate the participating disciplines internally – above all psychology and linguistics, divided as they are in conflicting sub-disciplines that treat their objects of study (i.e., mind and language) in, respectively, biological, mental, and socio-cultural terms. To emphasize again: CS is not a branch, school, or theory of semiotics, the latter understood as a self-contained discipline. It can make equal use of ideas from Peirce, Saussure, Jakobson, Greimas, von Uexküll – or from anywhere else – to the extent that those ideas are productive for empirical research leading to new insights into the nature (and culture) of human beings, as well as other meaning-seeking and meaning-making beings. It could perhaps be better called “semiotic cognitive/mind science”, if the phrase were not so cumbersome and “science” not so often taken to refer solely to natural science.

Acknowledgements

The author wishes to thank, in alphabetical order, Mats Andrén, Johan Blomberg, Paul Bouissac, Riccardo Fusaroli, Joel Parthemore, Chris Sinha, Victor Smith, Göran Sonesson and Michael Ranta for providing helpful feedback on earlier versions. A much shorter version of this article was published as Zlatev, J. (2011). What is cognitive semiotics? Semiotix: A Global Information Bulletin, XN-6. A longer one is to appear as Zlatev, J. (2012). Cognitive semiotics: An emerging field for the transdisciplinary study of meaning, Public Journal of Semiotics IV (1). Special thanks are due to Paul Bouissac, who encouraged these publications, including the present one.