The Melody of the Text – Revisited

Introduction

The text of everyday life (Bouissac) is hard to capture and study. In this paper I discuss some data from a microanalysis of spontaneous conversation which was used as part of a model of the ‘brain in action’. I present some theoretical ideas that account for the data, and them contextualize with discussion of other approaches to the study of the ‘brain in action’

Method – obtaining the ‘score’

I obtained a ‘score’ of spontaneous conversation. Isabella Poggi (Poggi 1996) describes a score as a procedure where ‘signals in two or more modalities are transcribed, analyzed and classified. In a classical score, signals delivered in five different modalities are reported on parallel lines:

v. verbal modality (the words and sentences uttered);

p. prosodic modality (speech rhythm, pauses, intensity, stress, intonation);

g. gestural modality (hand and arm movements);

f. facial modality (head and eye movements, gaze, smile and other facial expression);

b. bodily modality, (trunk and leg movements).

I had most of these modalities in my ‘score’. I used one of the first video tape recorders that had a stable single frame delivery, and constructed a split screen device of angled mirrors. I persuaded two students to chat informally, after painting spots on their noses to aid transcription of nose movement, and I obtained a reading of their nose positions in each successive tenth of a second. I had those plotted out in three dimensions (x,y,t) using an early graphics program (‘Picasso’). I obtained pure Fo (fundamental frequency) traces of their vocal output using a device known as a ‘Laryngograph’ (Fourcin), which measured it from impedance changes across the larynx. I entered a phonetic transcription against the text, and a plain language transcription. In all, at times, there were eight tracks to the score.

Data Presentation

I will show you three of my text fragments to give you a flavour of my material. These will be presented in Mpg format with accompanying hard copy ‘score’, in power point slide format.

The ‘Bomb Scares’ fragment:

Description

This fragment is about a recent event shared by the interactants. There had been a bomb scare in a hall of residence. It was a very animated fragment, with contrapuntal melodic weaving, ‘ complementary melodic and semantic contributions, sound and movement clusters, and onomatopoeic gesturing.

Transcription

Male voice: “You know you’ve had two bomb scares in Canterbury and they never even bothered to tell you let alone get you out. Didn’t even give you the chance to get out if you wanted; still”

Female voice “..(‘salad’ speech) ..yes I know, didn’t really bother with us.. think they’d let us die or something..”

Sound and movement clusters:

Male Voice:

‘You know you’ve had….’ The head moves forward, sideways, and down, and is inclined momentarily to one side

‘…. In Canterbury..’ coincides with a melodic fall and is accompanied by a circular nose movement.

‘…bothered to tell you…’ ‘Tell’ is the melodic apogee, and the head is sliding back to culminate in an eye-brow flash at the precise moment of ‘Tell’.

‘…..get you out….’ The phonetic transcription of ‘out’ is on the record, and the articulatory change points, with the melodic movement and the circular head movement together make one smooth sound and movement complex. The nose movement shape over ‘out..’ was circular, and the speaker initiated it just before the actual word ‘out’ began. It demonstrates onomatopoeically that ‘out’ ends up somewhere else than where it started.

‘…..chance to get out’ There was a small sideways head movement which occurred during the downward trajectory of ‘chance’, Understanding this micro-movement was particularly important. It occurred just after the vocalic nucleus of the word, and together with the ‘…nce’ of ‘chance’ rounded of the sound and movement cluster. The sideways movement was also where he DID get out, metaphorically, of the narrative he was constructing. This illustrates the ‘act on the model’ concept.

‘Still…’ The downgoing intonation, gaze eversion, and sideways head movement concluded the topic for both of them, and stabilized it so that they could move on, which they then did.

Female Voice

‘..salad speech…’ This occurred as the scrambled form of an utterance that had been initiated before the new topic line of ‘You know you had two bomb scares…’. It as if the urgency of response to the new material was overloading the output buffer,

‘..I know…’ Continues the trajectory of the salad speech utterance, but concludes it with a relevant ‘grasp’

‘..haven’t bothered with us…’ There is a rapid head shaking, which tapers exponentially as the utterance concludes.

‘I think they’re… There is a reversal in direction of sideways head drift. It is left incomplete

‘…..I don’t think they really care much about us’. There is a break of eye contact which occurs here, which itself constitutes a ‘giving up’ of engagement synchronous with the rupture in a relationship expressed by the utterance, and she hangs her head at its conclusion

‘…I think they’d let us die or something’ . There is a rapid sideways and downward movement on ‘..die..’, gesturing the concept, as it goes across.

The ‘Swimming Pools’ Fragment

Description

This fragment occurred when the speakers were trying to locate an address in the local town. This fragment demonstrates a supra-individual rhythm running through it, and sound movement synchronies. It also shows the rupture of the model at the synchronously timed event of the laugh.

Transcription

Male voice: “You know where the swimming pool is?” Female Voice: “No”. ‘Plonk’ of wine bottle on table! Both: Laugh.”

Sound and Movement Clusters

Male Voice

…you know where the swimming pool is..

The sound with movement trajectory is precisely coordinated, and the Fo trajectory is only broken by the consonental phonemes. The fragment creates a rhythm by its trajectory, which is carried on throughout the fragment.

Female Voice

..No…’

The delicacy of patterning of this melody demonstrates first a down going trajectory terminating with an up going. This integrates precisely with the sideways head movement, which rounds it off or ‘lets it out’. There was an up going melodic finish to this utterances..

The !Plonk!

The plonk of a wine bottle on the table occurred rhythmically on the beat commenced by the male voice utterance, and which was continued into the female voice utterance

The Laugh

(…Laugh….) This had synchronous onset, and occurred rhythmically on the beat commenced by the male voice utterance.

Jokes away Free..

Description

This fragment with solo female voice occurred after they had been trying to tell each other jokes but…

Transcription

“Let’s get away from telling jokes because I can never remember any and I’m very bad at them”

Sound and Movement Clusters

The fragment is one long cluster, but the Fo trace shows an undulating form in which each down beat defines a ‘point’ which is syntactic semantic, and rhythmical. I also analyzed this according to ‘cognitive change points’ using a catastrophe theory metaphor.

Discussion

In my first papers, on laughter -‘Mirth Measurement, a New Technique’, and ‘Rapport and Timeless Moments. a Micro-dissection’ - I made three observations:

1. Speech Structures Reality

By speech, I mean the actual physical utterance, or Parole, (from Saussure’s Langue/Parole dichotomy). All the patterning on the tape was brain product, and is included in the term ‘speech’. The full transmodal production of the brain is included in the ‘speech with movement’ trajectory.

2. Speech and movement turned out by the same patterning program.

Multi-modality is the theme of this symposium. I used the term Transmodal to emphasize that the integration of motor activity in my subjects transgressed the modalities, and did not respect the analytic distinctions between them. The integration was detailed with a grain finer than my analysis could take it, or that our senses can differentiate (finer than 1/10 second).

3. Immersion in the evolving model

The participants developed a shared history during the encounter (indeed they had a shared history before the encounter), and this was the context for their ongoing contributions. In addition, they were literally immersed in my constructed experimental scenario, which was open plan. If a door slammed, or someone else entered the room and discourse, it was shared. The wider immersion in time and place, language and culture is also a ‘given’ of the scenario.

The Theory of the Melody of the Text

I described the story that was being told on the tape as an ‘evolving model’.

The argument can be summarized:

· The ‘logic in use’ of the evolving argument makes up the text. (C.f. Cicourel for the derivation of this term.)

· The ‘act on the model’ in immediate consciousness is the same for both speech (the virtual auditory domain) and movement (the here and now visual domain). I thought that the paleo-archeology of direct action on the present might be preserved in the trajectory of speech-with-movement.

· The trajectory of the text directly synchronizes or entrains the core brain processes of participants in the evolution of the text.

· The ‘line’ of the argument can end in a number of ways. I recognized a number of relatively stereo typed motor patterns that the text could ‘fall into’ (‘Tropisms’- see below), depending on the configuration of ‘objects’ and their relations in immediate consciousness.

I recognized that in observing the trajectory of speech-with -movement, I was also watching cognitive models in evolution, and that as far as the text generator was concerned, they could be actual here and now (e.g. moving furniture), or virtual, from the sign system of langue.

The common feature was that there was a narrative in the succession of states of affairs. There was logic in the narrative, and this evolving model which was working itself out dialectically in time was the ‘text of everyday life’ (Paul Boussiac’s work introduced me to this notion). I considered the constituents of the immediate text as ‘objects’ and these were concatenated in the text as discrete ‘states of affairs, which could be the symbols of language, or objects in the real world. The object concept in relation to ‘images’ is discussed below. (The germ of the idea is similar to objects and states of affairs in Wittgenstein’s Tractatus).

Model and Topic

Models are more or less complete and self-sufficient entities, I suggested that ‘topics’ in conversation were ‘models’, which were worked through, and that the evolution of topic, which tended to completion, was the immediate context for utterance. Topics are both cognitive and segmental units, but it is their internal logic which determines completeness or ‘well formedness’.

Text Engine

I identified the motor that determined the trajectory of the text as the disjunction in the configurations of objects in the immediate present with projected outcomes from the immediate present. For topics, there is a chain of ‘relevance’ running through the threads formed by topics in evolution. ‘Objects in the immediate present’ were configured into ‘states of affairs’. The trajectory of speech with movement is literally the physical path between stable states of affairs, the ‘melody of the text’.

This is enacted in 4D-space/time, but the model implied at least a 5 dimensional text generator. Intercepting and thus modeling 4-D trajectories in 5 D is commonplace for our pre-human ancestors. I suggested that the ‘dimensions’ of a state of affairs might be considered the same as the ‘variables’ involved in constructing a model for dealing with the world. Expanding these capabilities up to 7 +/-2 may be the cardinal human innovation (c.f. Miller). I located the generator in the core or older structures of the brain, and suggested that the activities of the generator were cyclical (see ‘text bicycle’ diagram above).

I argued that the minimal unit, which organized a text both in terms of segmentation (syntax) meaning (semantics) was the ‘grasp’ or ‘point’.

Projection

The model has implications for the concept of ‘intention’. The ‘text’ of everyday life, as distinct from scripted performances (see Bouissac) is actually manufactured at the advancing edge of time, but with material that come from other times. My observation from the videotape was that the ‘future’ is as much part of the present as the past. The sequence of physical patterning recorded in the tape could be accounted for if I made the assumption that the future was there already in the tape, and that the actual story was a reconstruction. The text was being achieved as if by hooking the present up to the future as well as by the accretion of instances in present time.

I replaced ‘intention’ by ‘projection’. Rather than working with the concept of intention, and all the weighty philosophical baggage that it carries, I suggested that the future is projected from the present, in the way that an artist might project form on a neutral substrate. This is also consistent with early Greek (Platonic) theories of sight, which was thought to be projective from the eye rather than passively received. I offered a theory of the time forms of the melody of utterance, based on this assumption.

I could see the battle, or convergence, to outcomes for topics was faithfully registered in the melody. The aptness of melody to outcome is a conundrum, only resolved by the projection concept. I do not think it is an artifact of the experimental technique.

Topic Control by Melody:

The fundamental rules for topic control involving melody were suggested to be:

1. Following a down-going melody, a further down-going will be harmony if it delivers a ‘state of play’ in accord with the projection of the ‘other’.

2. A similar down-going melody delivering a different ‘state of play’ will be new information, or confrontation (leading to repetition or a ‘tropism)

3. An up going melody delivers a state of play that is intrinsically unstable. Within topic, following a down-going, it back tracks the text, and puts the ‘state of play’ back to where it was before the immediately preceding down-going melody.

4. An up-going melody will be followed by a down going melody if that topic can proceed. (From ‘Steps…’ Mair ibid. )

I also recognized that as well as the ‘evolving argument’ and its melodic evolution, there were behaviour patterns triggered during the text that would ‘grow’ in time from their start point, and which tend to unfold in time in characteristic ways. These I called ‘Tropisms’, after the use of that term in plant physiology, a point from which the text would ‘growing toward’ the conclusion of the pattern, and take over the text while they were doing so. Such were mirth, embarrassment, harmony, tears, bewilder, violence, laughter. These were like ‘fixed action patterns’ (Eibl-Eibesfeldt p 30).

The interaction of the interweaving projections of two or more individuals at the advancing edge of time leaves a melody, which can be recorded, and analyzed. This melody is the actual process by which the states of play in immediate consciousness were being achieved, the ‘thing itself’. Projective segmenting can in interaction deliver an infinite variety of ‘end melodies’.

I achieved a model of the ‘brain in action’ which was a caricature of brain processes, but which tied together the observations on text and some studies on the brain that I had made up to that point. With this microanalysis of physiological quality data and awareness of brain process, I suggested a model for the brain in action that was ‘transparent’. The ‘source’ of the text was not in the brain at all, but in the logic projected from it and through it in the multifarious present moments of its existence.

Interpreting the data fragments

The Bomb Scares fragment:

The trajectory of the Male Voice utterance dominates the fragment, with the female Voice contributions fitting in to complement this at all levels, including rhythm, kinesically, and semantically. The ‘act on the model’ represented by the kinesic flow and the acoustic melody entrains both participants. It becomes comprehensible if you see them following a mutually perceived destination to the fragment, which they follow on through. The form of the gesture is always unique to the moment, and displays the form of the idea going across at that moment, in the context of where it came from, and where it is projected. The gaze avert/equivocal word and melody/ topic change cluster on ‘..still...’ concludes the segment and stabilizes the topic enough to allow it to be left.

The Swimming Pool fragment:

The sinuous sound movement cluster of the Male Voice utterance sets up a rhythm for the fragment, and presents an unstable model for them both, which was the local map without the location of the swimming pool. It was a question, which is always unstable until answered. The ‘..no..’ of the Female Voice response with its up going end curl, further destabilizes the model created between them. This double instability results in the disintegration of the model at the ‘plonk’ of the wine bottle on the table, which was also rhythmically entrained in the fragment. Finally, the external manifestation of this ruptured model was the laugh, and this occurred on the same rhythmic beat.

Let’s get away from telling jokes

The ‘three beat ‘ melody like a sine wave of decreasing periodicity clearly suggests a cycle turning it out. The sound movement synchrony is precise, the movement onomatopoeic of the idea (‘away from…’ is away from – her start position). The velocity trace of the head movement appear to coincide with ‘vocalic nuclei, or the ‘pulse’ of the utterance.

The Theory Updated

With some new knowledge of the infrastructure of consciousness, we can take another look at the ‘text generator’ or storyteller model of the brain in action, and at the text it turns out. I want to contextualize my earlier work with reference to the contribution of some later authors, particularly the neurologist Antonio Damasio, and two contributors to this virtual symposium, McNeill and Poggi. I also reference some work on the neuro-physiology of vision.

My earlier model was inhabited and set in motion by the core brain, and I agree exactly with Damasio when he says in his book: ’The Feeling of What Happens’ (Vintage, 2000 p187):

‘I believe the imaged, nonverbal narrative of core consciousness is swift, that is unexamined details have eluded us for a long time, that the narrative is barely explicit, so half hinted that its expression is almost like the emanation of a belief.’

His theory of the brain reads like a modern and more informed version of the ‘text generator’ model. The basic proposition is that the sophisticated verbal narratives of the human brain are inhabited by ‘core consciousness’, which has non-verbal narrative as its product. Damasio explores the brain structures responsible for this ‘core consciousness’, and they are indeed the older deeper structures of the brain. The ‘self in the act of knowing’ is the old ‘snake brain’, only enhanced in its computational resources, and above all, by the virtual world of the sign systems of language.

In Damasio’s ‘self in the act of knowing’, the old core brain nonverbal storyteller is the one ultimately that is segmenting time. It is the original storyteller. As Damasio puts it:

‘the imagetic representations’ of sequences of brain events, which occurs in brains simpler than ours, is the stuff of which stories are made. (Ibid. p188)….The brain inherently represents the structures and states of the organism, and in the course of regulating the organism as it is mandated to do, the brain naturally weaves wordless stories about what happens to an organism immersed in an environment.’

The dependence of so-called higher functions on the phylogenetically older core brain structures can certainly be substantiated for vision. Our visual control systems are located in the older parts of the brain, turning out a sophisticated instrument for holding an image stable on the retina. This is the beginning of the image, and its reality has been explored with single cell studies, and extensive mapping replicated in great many cortical areas. However these mappings are all projections anatomically from an older part of the brain, the thalamus, which first receives this map from the retina and then re-projects it out to the cortex.

It also projects back to older parts of the brain, to more components of the thalamus and to the superior colliculi in the roof of the primitive forebrain. There are no direct outputs from the cortex to occulo-motor neurons, which actually put out the signals to the eye muscles.(p247, Leigh and Zee). The cortex is only accessed and outputted via the phylogenetically older core brain. (Insert illustration)

Site of the Text Generator

Where is this updating non-verbal model of the present, which produces the story of the states of the organism? Damasio sees it as distributed between a very small number of structures:

In ‘The Feeling of What Happens’ he describes the ‘proto-self’ as a

..’collection of neural patterns which map, moment by moment, the state of the physical structure of the organism in its many dimensions.’

Brain structures for this proto self include (p154)

1. Brain stem reticular, monoamine, and acetylcholine nuclei

2. A representation on the right insular cortex

3. The hypothalamus and basal forebrain.

Further structures do the ‘second order processing’ of this primal input. The superior colliculus, thalamus, and cingulate cortex are responsible for the ‘imaged account’ of the nonverbal story. All the resources of the cortex are ‘run’ by these basal systems. (Damasio; p194)

Insert diagrams

Th cortex is a resource that allows the story to be more complicated, and when it comes to beaming down a projective plan on the succession of instances, the pre frontal cortex plays a crucial role. This is Damasio’s ‘extended consciousness’, and the source of the autobiographical self. From Descartes Error, (p54) there is a combination of impaired decision-making and flat emotion and feeling with pre frontal lobe damage.

People with damage to just these regions “never construct an appropriate theory about their persons, or about their person’s social role in the perspective of the past or the future. (P98)”. Valid action that is ethically appropriate and personally meaningful is never possible for them, and their story ‘unravels’ (to use Damasio’s telling phrase). However, the primer mover and the primal non-verbal text remain turned out by the core brain structures. Damasio describes how the PET (positron emission tomography – a way of studying brain activity in real time) in newborn infants shows activity in the brain stem, hypothalamus, somatosensory cornices, and cingulate. This tends to confirm that these are the core consciousness brain circuits, and that they are active before the cortex is brought on line.

Thoughts as Images: Visual Projective Logic and ‘Growth Points’.

Many writers have emphasized or assumed an image like nature for thoughts. Certainly there is visual dominance of the human brain (about 80% of the brain can be demonstrated to have visual correlates). What must the world be like in order that I should know it? It must be visual, a priori. This may be an example of the anthropic principle (Hawking p130).

Damasio says in ‘The Feeling of What Happens’ (P84):

“The images over which we reason (the images of specific objects, actions, and relational schemas; of words which help translate the latter into language form) not only must be “in focus” – something achieved by attention – but must also be held ‘active in the mind – something achieved by higher order working memory.”

The ‘growth point’ (GP) units, as McNeill describes it is a natural segmentation of the interactional flow. The GP is a unit in which both imagery (from gesture) and language content (present in the form of linguistic categories) is combined.

McNeill references work suggesting ‘so strong is the speech-gesture bond that the complete absence of sight does not interrupt it’ (Iverson & Goldin-Meadow 1998). He takes from Vygotsky that language has a psychological unit, which is the smallest component that retains the property of being a whole, and identifies that with his ‘growth point’ concept.

McNeill describes the stability/instability cycle as the ‘dialectic’, and in his model, a grammatically complete sentence (or its approximation) is a model of stability, a socially constituted state of repose reached after dialectic instability. Further, a cycle will successively deliver states of repose and instability ad infinitum.

Intuitions of ‘well formedness’ define the outcome of the imagery-language dialectic.

The concept is similar to that in my text generator model, except that in my model it is the states of play in a consciousness (made of object/ images) that get rendered stable/unstable. For McNeill, it is from the opposition between imagery and language that the dialectic progresses. They are combined in the ‘growth point, which is a minimal psychological and physical unit.

McNeill sees gesture as actually part of the image; ‘The concept of a

material carrier implies that the gesture, the actual motion of the gesture itself, is a

dimension of thinking. From this viewpoint, a gesture is an image in its most developed’ sense.’

I argued that speech melody actually was the trajectory by which images become immanent in the brain of participants. Despite the existentialist flavour of this assertion, it does appear that the interactants segment the text projectively. Sections of text are demarcated by their own trajectories, a natural minimal unit of language. A trajectory is projective over the time immediately ahead.

This concept is again similar to McNeill’s ‘growth units’ (GPs). I would like to suggest that ‘projection point’ is another helpful label for the beginnings of GPs. It is entirely meaningful to think of the reality immediately ahead as ‘projected’ out through the eye by action on the present.

The entrainment of language and thought in trajectories is may find a parallel in ‘grasping behaviour’ by rhesus monkeys. From single cell studies, mirror neurons have been demonstrated in regions of the pre-frontal cortex which track the trajectories of observed movement. Could melody in speech work like this? Could this be where the infrastructure for calculating and sharing trajectories is located? Mirroring of trajectories is now demonstrated in Broca’s areas in Rhesus monkeys. In Binkofski et al, (2000), we may now have the required phylogenetic link to brain function. ‘Mirror Neurons’ have been demonstrated at a number of sites in the brain, including Broca’s area (the area in the frontal lobes associated with the phonological production of speech). These neurons follow the trajectory of an event in the world, which is simply observed, rather that participated in by the organism. This might give a clue to the grasping shapes of the melody of the text. (also see Skoyles J). It is real time trajectories that are being mirrored. The trajectories of the grasp in rhesus monkeys and of speech in humans appear to be mirrored in a homologous cortical region. The movement is in the virtual world for speech, but the shape of the action and its coordination between participants is preserved in the trajectory of the text, the ‘act on the model’. These are concatenated in a syntactic chain.

Another author suggesting that trajectories are important is J Calvin with ‘the unitary hypothesis’. He writes:

“Natural selection for one of the ballistic movements (hammering, clubbing, and throwing) could evolve a plan-ahead serial buffer for hand-arm commands that would benefit the other ballistic movements as well. This same circuitry may also sequence other muscles (children learning handwriting often screw up their faces and tongues) and so novel oral-facial sequences may also benefit (as might kicking and dancing). An elaborated version of the sequencer may constitute a Darwin Machine that spins scenarios, evolves sentences, and facilitates insight by offline simulation”

Even though the speech trajectories capture virtual world models rather than actual objects on four-dimensional trajectories (like a prey animal moving in the environment), I suggest that the trajectory of speech with movement is non-verbal, the product of the core brain forming the core to the speech act. The ‘point’ is the point. A growth point is defined as the ‘initial form of thinking out of which speech-gesture organization emerges’. (McNeill) It might also be called the ‘projection point’.

The core brain mechanisms underlying human natural story telling can now be glimpsed. Damasio’s core brain text generator in action describes the nonverbal internal structure of gesturing behaviour in speech with movement. It may have functioned projectively on 4D-space time for probably billions of years. Additional control of outcomes is achieved by adding more dimensions or variables to the modeling process, up to our present limit of 7+/-2.

I think that the ‘material’ of speech with movement, via the mechanism of mirror neurons and the shape of the trajectory itself, makes a ‘synctium’ of the communicating minds. They directly collaborate to control the outcomes of the topics in which they are immersed. The core brains are entrained, and the story is shared. However the logic is ‘projective’ on 4D space-time, and I have termed our everyday ‘logic in use’ as ‘visual projective logic’.

Growth point, Projection point, or Simultaneous Expression of Functions?

Isabella Poggi is another author who has identified and thought about the remarkable co-occurrence of synchronies that can be demonstrated in the text of everyday life. In her demonstration of the co-occurrence of movement, speech (in its many parameters) and meaning, she stresses the notion of the simultaneity of occurrence of functions, expressed in ‘the moment’.

She analyzes it in terms of two layers of meaning (Poggi ‘Score….’ P10)

‘From our two-layers score analysis we saw that while first-layer meanings most frequently are content information or information on the speaker's mind, self presentation is present solely but frequently in the second-layer meanings of prosodic, gestural and facial communication.”

The reality of transmodality is hard to encompass in verbal theory. One struggles to contain in theory how semanticity can be enhanced by gesture, and yet be idiosyncratic, the product of the ‘just then, just that’ of immediate exegesis. The rebellion against Cartesian dualism is striking from so many authors, and a review by Keith Devlin ‘Goodbye Descartes’, identifies about a dozen authors who have made a particular point of denying Descartes’ epistemology.

I restate this non-dual thesis from the neurological perspective in the model of the core brain projector and its cortical enhancement. In my analysis, ‘emotion’ is intrinsic to the melody, and as detailed in its patterning as the trajectory of the text itself. I suggest that the ‘growth point’ or projective ‘grasp’ has the detailed participation of the core non-verbal brain, and this is mirrored directly in the ‘other’, participant in parole. (Skoyles. No wonder then that memory also come with an ‘affective tone’, because the dichotomy of affective versus content was itself flawed. The amygdala nuclei in the core brain would appear to participate crucially here. With damage to the amygdala, it has been shown that the ability to interpret manifestations of fear and dread are also lost. The essential point about its activities is that they are non-verbal. The dichotomy of emotion/intellect may itself be a cultural artifact, a victim of Whitehead’s ‘fallacy of misplaced consciousness’.

What are the theoretical options when you find the detailed co-occurrences of speech and gesture that have been demonstrated by micro-dissections of narrative performed by three of the contributors to this symposium? I will tabulate our responses to this conundrum:

Co-occurrences of speech and movement: Models	Poggi	McNeil	Mair
Semantic Import	Goals and Beliefs	Dialectic Symbols-Actions	Partial manifestation of plans in action
Integration of speech and Movement	Co-occurring functions, separate layers	A priori, co-patterned at source	Same trajectory, transmodal, the ‘act on the model’
Dealing with the future	Goals	Intentions	Projection

The Link to the Electronic Health Record

The fact that there might actually BE such natural minimal units in Natural Language has wide implications. The ‘text of everyday life’ is revealed to be a concatenation of instances, organized by plans of various scales, which span past and future, although expressed in the present. These templates exist at the level of ‘growth point’ minimal units, but are syntactically combined and entrained to make the text. The text is ‘projective’ from the ‘ground zero’ of the present, and accumulates behind as shared text.

It has been hard to achieve consensus on a simple design for the process of electronic record keeping. The lack of a shared ‘standard’ or way of doing it that would enable communication between disparate systems has stymied progress with the real time computer interface for medical note keeping. Competition between systems has been immense, but none can take over the world, and yet to work, electronic medicine must communicate globally.

The simple model of text and generator works well for an Electronic Health Records system. It has an assumption that any text consists of the capture of objects in a process that is organized syntactically, i.e. sequentially. Forms or protocols do this, and the design of the Patient Management System that I am involved with (‘Houston Ophthalmology’) allows the creation of many different forms out of objects from a shared vocabulary.

The present status of the evolution of Global Healthcare communication standards is approaching a convergence that could deliver just this base model for the ‘standard’. The standard is about structured data. It is about how systems and components can be exchanged and used, and thus has functional and semantic aspects.

There have been two model types that have approached this. The first, largely through work with a European origin (CEN ENV 13606, The GEHR Object Model) has tried to define the components that an Electronic Health Record should be built from. This kind of ‘top down’ modeling has produced many schemes, most non-implentable. A survivor has been the idea of the clinical object or ‘Archetype’. (see www.openehr.org.)

The second approach began with the concepts of ‘documents’ and ‘mark up’, and has standardized a ‘document header’ with standard components to allow navigation and retrieval of health related documents, the Clinical Document Architecture (CDA) from Health Level Seven (see www.HL7.org ). A second level CDA may contain templates for paragraphs or headings, and third would have ‘clinical objects’ contained in it. These would constitute a searchable database of structured data.

The Structured Documents group of HL7 proposes a standard ‘information asset’ unit with standard structure (information plus markup), similar to the ‘instance’ or ‘grasp unit’ or ‘Growth Point’ that we are identifying for Parole.. It is proposed as a currency for healthcare transactions globally. (Mair 2002).

In my Sydney 2002 paper I use the metaphor of a ‘basket of objects’ for the proposed Healthcare communication standard. The purpose of the proposal is to get healthcare interoperability up and running, getting 80% of the functionality with 20% of the effort. The basket and its objects would constitute a ‘health event summary’ (HES) containing all the harvested data objects from a health event.

A health event has discrete boundaries in time, a beginning and an end. Often synonymous with an ‘encounter’, it is also more generic than that. In any ‘clinical workface’ application, these objects will be structured into ‘clinical templates’ which are ‘good to think’, like objects in a diviner’s basket. (VW Turner). Clinical templates can be as diverse as language itself, but all the objects can be scooped up an exported to a standard basket, from which compliant applications can scoop them out again, and slot them into the templates of their choice.

There is the problem of meaning from its temporal context, that is, from the arrangement of objects on their particular template. The parallel here is again from linguistics, where a word has meaning both from its associative or paradigmatic surround, and also from its place in a syntagm. For a global computer standard this is easily fixed by having companion objects, which describe the relations between classes ‘on screen’ at the moment of capture.

The idea of objects being ‘good to think’, to make meaning out of, comes from Claude Levi Strauss (La Pensee Sauvage) and is a distant echo of his quest:

‘Between and existentialism idealism there is room for a new science and a new philosophy…...”

We can now add ‘…..and a new implementation’.

Mike Mair

References

Binkofski Ferdinand et al : ‘Broca’s Region Subserves Imagery of Motion: A combined Cytoarchitectonic and fMRI Study HumanBrainMapping11:273–285(2000)

Bouissac, P 19 Circus and Culture 1976 Indian University Press

Bouissac, P. 1991 “Semiotics and the Gaia Hypothesis: Toward a Restructuring of Western Thought,” Philosophy and the Future of Humanity 1(2),168-184.

Calvin, W.H. ‘The Unitary Hypothesis’ A Common Neural Circuitry for Planning ahead, and throwing’ In ‘Tools, Language, and Cognition in Human Evolution’, edited by Kathleen R. Gibson and Tim Ingold. Cambridge University Press, pp. 230-250 (1993). Also at http://faculty.washington.edu/~wcalvin/1990s/1993Unitary.htm

Fourcin, A.J. ‘The Layngograph’ in Ventilatory and Phonological Control systems, Ed B

Wyke, London, OUP, 1974

Ciccourel, A.V. ‘Cognitive Sociology’ , Penguin, 1973

W. Condon & W. Ogston. Sound Film Analysis of Normal and Pathological Behaviour Patterns. The Journal of Nervous and Mental Disease. Vol. 143, 4: 338-345.

Damasio, Antonio R, ‘Descartes’ Error’ 1994 Harper Collins

Damasio, Antonio R, ‘The Feeling of What Happens’, 2000, Vintage

Eibl-Eibesfeldt, ‘Love and Hate’, Methuen, 1970, p30

Devlin, Keith ‘Goodbye Descartes’, Wiley, 1977, p276. Devlin lists 17 authors from Husserl to Damasio.

Hawking, S ‘A brief History of Time’ Space Time publications, 1988

Health Level Seven www.HL7.org

Houston Ophthalmology, The New Zealand Software Corporation:

http://www.nzsoft.co.nz/ophthalmology.htm

Iverson, J. & Goldin-Meadow, S.1998. Why people gesture as they speak. Nature 396:228.

ISOTC/215 WG1 The Modeling work group of the International Standards Organization Technical Committee on Electronic Health Records Standards http://www.cihi.ca/Roadmap/IHISD/ISO/program.shtml

‘Access to the Health Record’ draft technical report

Leigh R and Zee D, ‘The Neurology of eye Movements’ OUP 1999

Levi-Strauss, Claude. Structural Anthropology. 1968 The Allen Lane Press

Levi-Strauss, Claude. The Savage Mind 1966 Weidenfield and Nicholas

Mair MW (2002) Electronic Health Record Standards (EHR) –‘Healthcare Information Technology Standards’ Proceedings of International Congress of Ophthalmology, Sydney, April. 2002 http://www.ophthalmology.aust.com/scientific_program.html

Mair M (2001) Report on the Annual Health Level 7 Conference, Salt Lake City, USA, 1-5 October 2001. Available from author

Mair M Report on ISO/TC215 Meeting, Seoul, 26-30 March. 2001 Available from author

Mair MW (1998) “Comprehensive Ophthalmology Management”, in Proceedings of the 1998 New Zealand Health Informatics Conference: Preparing for the New Millennium, Christchurch, NZ.

Mair M.W. (1986) “The Eye in the Control of Attention”, in The Biological Foundations of Gesture: Motor and Semiotic Aspects, Jean-Luc Nespoulous, Paul Perron, Andre Roche ecours (Eds.), Lawerence Erlbaum Associates, London, p.123-146

Mair M.W. ‘A Model of the Text Generator, in ‘the Neurological Basis of Signs in Communication’, Toronto Semiotic Circle, 1981, p 147

Mair M.W. (1980) “The Melody of the Text: Review of ‘Circus and Culture’ by Paul Bouissac”, Semiotica, Vol.32, No.1/2, p.119-138, Indiana University Press, Mouton, The Hague.

Mair M.W. (1978) Steps Toward Principles of Text Regulation, Toronto Semiotic Circle, Victoria University, Toronto.1978 no 2

Mair M.W. (1977) “Mirth measurement: A new technique”, in It’s a Funny Thing, Human, Chapman and Foot (Eds.), Oxford and New York (Pergamon), p.105-111

Mair M.W. (1977) “Rapport and timeless moments: A microdissection”, in Proceedings, Love and Attraction, an International Conference, Mark Cook and Glenn Wilson (Eds.) Oxford and New York (Pergamon).

Mair MW (1975) ‘What do Faces Mean’ Royal Anthropological Institute, 1975, , reprinted in Steps toward Principles of Text Regulation (ibid)

McNeill, D. Gesture Language Dialectic Acta Linguistica Hafniensia green - 1/19/02

McNeill, D. 1992. Hand and Mind: What Gestures Reveal About Thought. Chicago, IL:

University of Chicago Press.

Miller G A ‘The Law of Seven Plus or Minus Two’ (source to be supplied)

Poggi I & Magno Caldognetto E. (1996): “A score for the analysis of gesture in multimodal communication”. InL.Messing (Ed.) Workshop on the Integration of Gesture in Language and Speech, Newark, Delaware and Wilmington, Delaware. October 7-8, 1996.

Poggi, I ‘Toward the Alphabet and Lexicon of Gesture, Gaze, and Touch Paper for Paul Bouissac’s multimodality symposium.

Polanyi, Michael. ‘Personal Knowledge’ London, RKP 1958

Saussure, F Cours de Linguistique Generale Paris, 1916

Skoyles, J ‘Mirror Neurons and the Motor Theory of Speech’ http://www2.psy.uq.edu.au/CogPsych/Noetica/OpenForumIssue9/

Turner, V.W. The Drums of Affliction’ 1967 (source to be supplied)

Wittgenstein L Tractatus Logico-Philosophicus

ILLUSTRATIONS

Fig.1 Tropisms

mirth

violence embarrassment

tears harmony

bewilder

The dialectic of the text continues on the axis vertical to the plane of the paper, but can ‘fall into’ a tropism at any time I enumerated these:

· Harmony: the complete model, the resolved problem

· Violence: the continuation of the text by direct action

· Mirth: the rupture of the model with ‘necker cube’ flip inversion of model relationships, leads to motor ‘uncontrol’

· Tears : the immediate state of affairs that fragments completely. Unlike the ‘cube flip’ of mirth, in which both interpretations of an immediate state of play are co present, there is no immediately ensuing text because there is no ‘state of play’ for that individual

· Embarrassment: a configuration where there is a juxtaposition of objects from normally mutually exclusive categories.

· Bewilder: If a linear argument has been under way, and then the state of play becomes an impossible model (like Escher art work), then this halts the text.

Fig. 2 The Core Brain Text Generator

The Core Brain Text Generator (taken from ‘the Eye in the Control of Attention’ ibid.). Cortical areas are caricatured with three paired plates, occipital, tempero-parietal, and frontal, surrounding and acting as a resource for the core brain cyclical text generator. The cycle is closed by ‘action on the world’, which leads to the generation of the text as a concatenation of percept/plan sequences or ‘points’.