The Melody of the Text – Revisited
Introduction
The text of everyday life (Bouissac) is hard to capture and study. In this paper I discuss some data from a microanalysis of spontaneous conversation which was used as part of a model of the ‘brain in action’. I present some theoretical ideas that account for the data, and them contextualize with discussion of other approaches to the study of the ‘brain in action’
Method – obtaining the ‘score’
I obtained a ‘score’ of spontaneous conversation. Isabella Poggi (Poggi 1996) describes a score as a procedure where ‘signals in two or more modalities are transcribed, analyzed and classified. In a classical score, signals delivered in five different modalities are reported on parallel lines:
v. verbal
modality (the words and sentences uttered);
p. prosodic
modality (speech rhythm, pauses, intensity, stress, intonation);
g. gestural
modality (hand and arm movements);
f. facial
modality (head and eye movements, gaze, smile and other facial expression);
b. bodily
modality, (trunk and leg movements).
I had most of these modalities in my ‘score’. I used one of the first video tape recorders that had a stable single frame delivery, and constructed a split screen device of angled mirrors. I persuaded two students to chat informally, after painting spots on their noses to aid transcription of nose movement, and I obtained a reading of their nose positions in each successive tenth of a second. I had those plotted out in three dimensions (x,y,t) using an early graphics program (‘Picasso’). I obtained pure Fo (fundamental frequency) traces of their vocal output using a device known as a ‘Laryngograph’ (Fourcin), which measured it from impedance changes across the larynx. I entered a phonetic transcription against the text, and a plain language transcription. In all, at times, there were eight tracks to the score.
Data Presentation
I will show you three of my text fragments to give you a flavour of my material. These will be presented in Mpg format with accompanying hard copy ‘score’, in power point slide format.
The ‘Bomb Scares’ fragment:
Description
This fragment is about a recent event shared by the interactants. There had been a bomb scare in a hall of residence. It was a very animated fragment, with contrapuntal melodic weaving, ‘ complementary melodic and semantic contributions, sound and movement clusters, and onomatopoeic gesturing.
Transcription
Male voice: “You know you’ve had two bomb scares in
Female voice “..(‘salad’ speech) ..yes I know, didn’t
really bother with us.. think they’d let us die or something..”
Sound and movement clusters:
Male Voice:
‘You know you’ve had….’ The head moves forward, sideways, and down, and is inclined momentarily to one side
‘…. In
‘…bothered to tell you…’ ‘Tell’ is the melodic apogee, and the head is sliding back to culminate in an eye-brow flash at the precise moment of ‘Tell’.
‘…..get you out….’ The phonetic transcription of ‘out’ is on the record, and the articulatory change points, the with the melodic movement and the circular head movement together make one smooth sound and movement complex. The nose movement shape over ‘out..’ was circular, and the speaker initiated it just before the actual word ‘out’ began. It demonstrates onomatopoeically that ‘out’ ends up somewhere else than where it started.
‘…..chance to get out’ There was a small sideways head movement which occurred during the downward trajectory of ‘chance’, This understanding this micro-movement was particularly important. It occurred just after the vocalic nucleus of the word, and together with the ‘…nce’ of ‘chance’ rounded of the sound and movement cluster. The sideways movement was also where he DID get out, metaphorically, of the narrative he was constructing. This illustrates the ‘act on the model’ concept.
‘Still…’ The concluding intonation, gaze eversion, and sideways head movement concluded the topic for both of them, and stabilized it so that they could move on, which they then did.
Female Voice
‘..salad speech…’ This occurred as the scrambled form of an utterance that had been initiated before the new topic line of ‘You know you had two bomb scares…’. It as if the urgency of response to the new material was overloading the buffer,
‘..I know…’ Continues the trajectory of the salad speech utterance, but concludes it with a relevant ‘grasp’
‘..haven’t bothered with us…’ There is a rapid head shaking, which tapers exponentially as the utterance concludes.
‘I think they’re… There is a reversal in direction of sideways head drift. It is left incomplete
‘…..I don’t think they really care much about us’. There is a break of eye contact which occurs here, which itself constitutes a ‘giving up’ of engagement synchronous with the rupture in a relationship expressed by the utterance, and she hangs her head at its conclusion
‘…I think they’d let us die or something’ . There is a rapid sideways and downward movement on ‘..die..’, gesturing the concept, as it goes across.
The ‘Swimming Pools’ Fragment
Description
This fragment occurred when the speakers were trying to locate an address in the local town. This fragment demonstrates a supra-individual rhythm running through it, and sound movement synchronies. It also shows the rupture of the model at the synchronously timed event of the laugh.
Transcription
Male voice: “You know where the swimming pool is?” Female
Voice: “No”. ‘Plonk’ of wine bottle on table! Both: Laugh.”
Sound and Movement Clusters
Male Voice
…you know where the swimming pool is..
The sound with movement trajectory is precisely coordinated, and the Fo trajectory is only broken by the consonental phonemes. The fragment creates a rhythm by its trajectory, which is carried on throughout the fragment.
Female Voice
..No…’
The delicacy of patterning of this melody demonstrates first a down going trajectory terminating with an up going. This integrates precisely with the sideways head movement, which rounds it off or ‘lets it out’. There was an up going melodic finish to this utterances..
The !Plonk!
The plonk of a wine bottle on the table occurred rhythmically on the beat commenced by the male voice utterance, and which was continued into the female voice utterance
The Laugh
(….Laugh….) This had synchronous onset, and occurred rhythmically on the beat commenced by the male voice utterance.
Discussion
In my first papers, on laughter -‘Mirth Measurement, a New Technique’, and
‘Rapport and Timeless Moments a
Micro-dissection’ - I made three observations:
1. Speech Structures Reality
By speech, I mean the actual physical
utterance, or Parole, (from
Saussure’s Langue/Parole dichotomy).
All the patterning on the tape was brain product, included in the term
‘speech’. The full transmodal production of the brain is included in the ‘speech with movement’ trajectory.
2. Speech and
movement turned out by the same patterning program.
Multi-modality is the theme of this
symposium. I used the term Transmodal
to emphasize that the integration of motor activity in my subjects transgressed
the modalities, and did not respect the analytic distinctions between them. The
integration was detailed with a grain finer than my analysis could take it, or
that our senses can differentiate (finer than 1/10 second).
3. Immersion in the evolving model
The participants developed a shared history during the encounter (indeed they had a shared history before the encounter), and this was the context for their ongoing contributions. In addition, they were literally immersed in my constructed experimental scenario, which was open plan. If a door slammed, or someone else entered the room and discourse, it was shared. The wider immersion in time and place, language and culture is also a ‘given’ of the scenario.
The Theory of the Melody of the Text
I described the story
that was being told on the tape as an ‘evolving model’.
The
argument can be summarized:
·
The ‘logic in use’ of the evolving argument
makes up the text. (C.f. Cicourel for the derivation of this term.)
·
The ‘act on the
model’ in immediate consciousness is the same for both speech (the virtual
domain) and movement (the here and now visual domain). I thought that the
paleo-archeology of direct action on the present might be preserved in the
trajectory of speech-with-movement.
·
The trajectory
of the text directly synchronizes or entrains the core brain processes of
participants in the evolution of the text.
·
The ‘line’ of
the argument can end in a number of ways. I recognized a number of relatively
stereo typed motor patterns that the text could ‘fall into’ (‘Tropisms’- see below), depending on the
configuration of ‘objects’ and their relations in immediate consciousness.
I recognized that in observing the trajectory of speech-with -movement, I was also watching cognitive models in evolution, and that as far as the text generator was concerned, they could be actual here and now (e.g. moving furniture), or virtual, from the sign system of langue. The common feature was that there was a narrative in the succession of states of affairs. There was logic in the narrative, and this evolving model which was working itself out dialectically in time was the ‘text of everyday life’ (Paul Boussiac’s work introduced me to this notion). I considered the constituents of the immediate text as ‘objects’ and these were concatenated in the text as discrete ‘states of affairs, which could be the symbols of language, or objects in the real world. The object concept in relation to ‘images’ is discussed below. (the germ of the idea is similar to objects and states of affairs in Wittgenstein’s Tractatus)
Model and Topic
Models are more or less complete and self-sufficient entities, I suggested that ‘topics’ in conversation were ‘models’, which were worked through, and that the evolution of topic, which tended to completion, was the immediate context for utterance. Topics are both cognitive and segmental units, but it is their internal logic which determines completeness or ‘well formedness
Text Engine;
I identified the motor that determined the trajectory of the text as the disjunction in the configurations of objects in the immediate present with projected outcomes for the immediate present. For topics, there is a chain of ‘relevance’ running through the threads formed by topics in evolution. ‘Objects in the immediate present’ were configured into ‘states of affairs’. The trajectory of speech with movement is literally the physical path between stable states of affairs, the ‘melody of the text’.
This is enacted in 4D-space/time, but the model implied at least a 5 dimensional text generator. Intercepting and thus modeling 4-D trajectories in 5 D is commonplace for our pre-human ancestors. I suggested that the ‘dimensions’ of a state of affairs might be considered the same as the ‘variables’ involved in constructing a model for dealing with the world. Expanding these capabilities up to 7 +/-2 may be the cardinal human innovation. I located the generator in the core or older structures of the brain, and suggested that the activities of the generator were cyclical (see ‘text bicycle’ diagram above). .
I argued that the minimal unit, which organized a text both in terms of segmentation, was syntactically and semantically the ‘grasp’ or ‘point’.
Projection
The model had implications for the concept of ‘intention’. The ‘text’ of everyday life, as distinct from scripted performances (see Bouissac), is actually manufactured at the advancing edge of time, but with material that come from other times. My observation from the videotape was that the ‘future’ is as much part of the present as the past. The sequence of physical patterning recorded in the tape could be accounted for if I made the assumption that the future was there already in the tape, and that the actual story was a reconstruction. The text was being achieved as if by hooking the present up to the future as well as by the accretion of instances in present time.
I replaced ‘intention’ by ‘projection. Rather than working with the concept of Intention, and all the weighty philosophical baggage that it carries, I suggested that the future is projected from the present, in the way that an artist might project form on a neutral substrate. This is also consistent with early Greek (Platonic) theories of sight, which was thought to be projective from the eye rather than passively received. I offered a theory of the time forms of the melody of utterance, based on this assumption.
I could see the battle, or convergence, to outcomes for topics was faithfully registered in the melody. The aptness of melody to outcome was a conundrum, only resolved by the projection concept. I do not think it is an artifact of the experimental technique.
Topic Control by Melody:
The fundamental rules for topic control involving melody were suggested to be:
1. Following a down-going melody, a further down-going will be harmony if it delivers a ‘state of play’ in accord with the projection of the ‘other’
2. 2. A similar down-going melody delivering a different ‘state of play’ will be new information, or confrontation (leading to repetition or a ‘tropism)
3. An up going melody delivers a state of play that is intrinsically unstable. Within topic, following a down-going, it back tracks the text, and puts the ‘state of play’ back to where it was before the immediately preceding down-going melody.
4. An up-going melody will be followed by a down going melody if that topic can proceed. (From ‘Steps…’ Mair ibid. )
I also recognized that as well as the ‘evolving argument’ and its melodic evolution, there were behaviour patterns triggered during the text that would ‘grow’ in time from their start point, and tend to unfold in time in characteristic ways. These I called ‘Tropisms’, after the use of that term in plant physiology, a point from which the text would ‘growing toward’ the conclusion of the pattern, and take over the text while they were doing so. Such were mirth, embarrassment, harmony, tears, bewilder, violence (not on the tape, but inferred!). These were like ‘fixed action patterns’ (Eibl-Eibesfeldt p 30)
The interaction of the interweaving projections of two or more individuals at the advancing edge of time leaves a melody, which can be recorded, and analyzed. This melody is the actual process by which the states of play in immediate consciousness were being achieved, the ‘thing itself’. Projective segmenting can in interaction deliver an infinite variety of ‘end melodies’.
I achieved a model of the ‘brain in action’ which was a caricature of brain processes, but which tied together the observations on text and some studies on the brain that I had made up to that point. With this microanalysis of physiological quality data and awareness of brain process, I suggested a model for the brain in action that was ‘transparent’. The ‘source’ of the text was not in the brain at all, but in the logic projected from it and through it in the multifarious present moments of its existence.
Interpreting the data fragments
The Bomb Scares fragment:
The trajectory of the Male Voice utterance dominates the fragment, with the female Voice contributions fitting in to complement this at all levels, including rhythm, kinesically, and semantically. The ‘act on the model’ represented by the kinesic flow and the acoustic melody entrains both participants. It becomes comprehensible if you see them following on a mutually perceived destination to the fragment, which they follow on through. The form of the gesture is always unique to the moment, and displays the form of the idea going across at that moment, in the context of where it came from, and where it is projected. The gaze avert/equivocal word and melody/ topic change cluster on ‘..still.’ concludes the segment and stabilizes the topic enough to allow it to be left
The Swimming Pool fragment:
The sinuous sound movement cluster of the Male Voice utterance sets up a rhythm for the fragment, an presents an unstable model for them both, which was the local map without the location of the swimming pool. It was a question, which is always unstable until answered. The ‘..no..’ of the Female Voice response destabilizes the model created between them (which was the local map minus a swimming pool, itself unstable). This double instability resolves into the disintegration of the model at the ‘plonk’ of the wine bottle on the table, which was also rhythmically entrained in the fragment. Finally, the external manifestation of a ruptured model was the laugh, and this occurred on the same rhythmic beat.
The Theory Updated
With some new knowledge of the infrastructure of consciousness, we can take another look at the ‘text generator’ or storyteller model of the brain in action and at the text it turns out. I want to contextualize my earlier work with reference to the contribution of some later authors, particularly the neurologist Antonio Damasio, and two contributors to this symposium, (McNeill and Poggi). I also reference some work on the neurophysiology of vision.
My earlier model was inhabited and set in
motion by the core brain, and I agree exactly with Damasio when he says in his
book: ’The Feeling of What Happens’ (Vintage, 2000 p187):
‘I believe the imaged, nonverbal
narrative of core consciousness is swift, that is unexamined details have
eluded us for a long time, that the narrative is barely explicit, so half
hinted that its expression is almost like the emanation of a belief.’
His
theory of the brain reads like a modern and more informed version of the text
generator model. The basic proposition is that the sophisticated verbal
narratives of the human brain are inhabited by the product of ‘core
consciousness’ which is non-verbal narrative. He explores the brain structures
responsible for this ‘core consciousness’, and they are indeed the older deeper
structures of the brain. The ‘self in the act of knowing’ is the old snake
brain as before, only enhanced in its computational resources, and above all,
by the virtual world of the sign systems of language.
In
Damasio’s ‘self in the act of knowing’, the old core brain nonverbal
storyteller, is the one ultimately that is segmenting time in the melody of
speech. It is the original storyteller. As Damasio puts it: ‘the imagetic
representations’ of sequences of brain events, which occurs in brains simpler
than ours, is the stuff of which stories are made. (Ibid. p188)’.. He says:
‘The brain inherently represents the structures and states of the organism, and
in the course of regulating the organism as it is mandated to do, the brain
naturally weaves wordless stories about what happens to an organism immersed in
an environment.
The
dependence of so-called higher functions on the phylogenetically older core
brain structures can certainly be substantiated for vision. Our visual control
systems are located in the older parts of the brain, turning out a
sophisticated instrument for holding an image stable on the retina. This is the
beginning of the image, and its
reality has been explored with single cell studies, and extensive mapping
replicated in great many cortical areas. However these mappings are all
projections anatomically from an older part of the brain, the thalamus, which
first receives this map from the retina and then re-projects it out to the cortex.
It
also projects back to older parts of the brain, more components of the thalamus
and to the superior colliculi in the roof of the primitive forebrain. There are
no direct outputs from the cortex to occulo-motor neurons, which actually put
out the signals to the eye muscles.(p247, Leigh and Zee). The cortex is only
accessed and outputted via the (phylogenetically older) core brain. (Insert
illustration)
Site of the Text Generator
Where is this updating non-verbal model of the present, which produces the story of the states of the organism? Damasio sees it as distributed between a very small number of structures:
In
‘The Feeling of What Happens’ describes the ‘proto-self’ as a ..’collection of
neural patterns which map, moment by moment, the state of the physical
structure of the organism in its many dimensions.’
Brain
structures for this proto self include (p154)
1. Brain stem nuclei,
reticular, monoamine, and acetylcholine nuclei
2. A representation on the
right insular cortex
3. The hypothalamus and basal
forebrain.
Further structures do the ‘second order
processing’ of this primal input. The superior colliculus, thalamus, and
cingulate cortex are responsible for the ‘imaged account’ of the nonverbal
story. All the resources of the cortex are ‘run’ by these basal systems.
(Damasio; p194)
Insert diagrams
Th cortex is a resource which allows the
story to be more complicated, and when it comes to beaming down a projective
plan on the succession of instances., the pre frontal cortex plays a crucial
role. This is Damasio’s ‘extended consciousness’, and the source of the
autobiographical self. From Descartes Error, (p54) there is a combination of
impaired decision-making and flat emotion and feeling with pre frontal lobe
damage.
People with damage to just these regions “never construct an appropriate theory about their persons, or about their person’s social role in the perspective of the past or the future.( P98)”. Valid action (ethically appropriate and personally meaningful) is impossible for them, and their story ‘unravels’ (to use Damasio’s telling phrase). However, the primer mover and the primal non-verbal text remain turned out by the core brain structures. Damasio describes how the PET (positron emission tomography – a way of studying brain activity in real time) in newborn infants shows activity in the brain stem, hypothalamus, somatosensory cornices, and cingulate. This tends to confirm that these are the core consciousness brain circuits, and that they are active before the cortex is brought on line.
Thoughts as Images: Visual Projective Logic and ‘Growth Points’
Many writers have emphasized or assumed an
image like nature for thoughts. Certainly there is visual dominance of the
human brain (about 80% of the brain can be demonstrated to have visual
correlates). What must the world be like in order that I should know it? It
must be visual, a priori. This may be an example of the anthropic principle
(Hawking p130).
Damasio says in ‘The Feeling of What Happens’
(P84):
“The images over which we reason (the images
of specific objects, actions, and relational schemas; of words which help
translate the latter into language form) not only must be “in focus” –
something achieved by attention – but must also be held ‘active in the mind –
something achieved by higher order working memory.”
The
‘growth point’ (GP) units, as McNeill describes it is a natural segmentation of
the interactional flow. The GP is a unit in
which both imagery (from gesture) and language content (present in the form of
linguistic categories) is combined.
McNeill references work suggesting ‘so
strong is the speech-gesture bond that the complete absence of sight does not
interrupt it’ (Iverson &
Goldin-Meadow 1998). He takes from Vygotsky that language has a psychological
unit, which is the smallest component that retains
the property of being a whole, and identifies that with his ‘growth point’
concept.
McNeill describes the stability/instability cycle as the ‘dialectic’, and
in his model, a grammatically complete sentence (or its approximation) is a
model of stability, a socially constituted state of repose reached after
dialectic instability. Further, a cycle will successively deliver states of
repose and instability ad infinitum.
Intuitions of ‘well formedness’ define the outcome of the
imagery-language dialectic.
The concept is similar to that in my text generator model, except that in
my model. It is the states of play in a consciousness (made of object/ images)
that gets rendered stable/unstable. For
McNeill, it is from the opposition between imagery and language that the
dialectic progresses. They are combined in the ‘growth point, which is a
minimal psychological and physical unit.
McNeill sees gesture as
actually part of the image; ‘The concept of a
material carrier implies
that the gesture, the actual motion of
the gesture itself, is a
dimension of thinking. From this viewpoint, a gesture is
an image in its most developed’ sense.’
I argued that speech melody actually was the trajectory by which images
become immanent in the brain of participants. Despite the existentialist
flavour of this assertion, it does appear that the interactants segment the
text projectively. Sections of text are
demarcated by their own trajectories, a natural minimal unit of language. A
trajectory IS projective over the time immediately ahead.
This
concept is again similar to McNeill’s ‘growth units’ (GPs). I would like to
suggest that ‘projection point’ is another helpful label for the beginnings of
GPs. It is entirely meaningful to think of the reality immediately ahead as
‘projected’ out through the eye by action on the present.
The
entrainment of language and thought in trajectories is further demonstrated for
‘grasping behaviour’ in recent monkeys. By single cell studies, mirror neurons
have been demonstrated in regions of the pre-frontal cortex which track the
trajectories of observed movement. Could melody in speech work like this? Could
this be where the infrastructure for calculating and sharing trajectories is
located? The movement is in the virtual world, but the shape of the action and
its coordination between participants is preserved in the melody of the text,
the ‘act on the model’ and these are concatenated in a syntactic chain.
The
mirroring of trajectories now demonstrated in Broca’s areas in Rhesus monkeys.
. In Binkofski et al, (2000), we now
have the required phylogenetic link to brain function.. ‘Mirror Neurons’ have been
demonstrated at a number of sites in the brain, including Broca’s area (the
area in the frontal lobes associated with the phonological production of
speech). These neurons follow the trajectory of an event in the world, which is
simply observed, rather that participated in by the organism. This might
give a clue to the grasping shapes of the melody of the text. (also see
Skoyles J). It is real time trajectories that are being mirrored. The
trajectories of the grasp in rhesus monkeys and of speech in humans appear to
be mirrored in a homologous cortical region.
Another
author suggesting that trajectories are important is J Calvin with ‘the unitary
hypothesis’. He writes:
“Natural selection for one of the ballistic movements (hammering, clubbing, and throwing)
could evolve a plan-ahead serial buffer for hand-arm commands that would
benefit the other ballistic movements as well. This same circuitry may also
sequence other muscles (children learning handwriting often screw up their
faces and tongues) and so novel oral-facial
sequences may also benefit (as might kicking and dancing). An elaborated
version of the sequencer may constitute a Darwin Machine that spins scenarios,
evolves sentences, and facilitates insight by offline simulation”
Even
though the speech trajectories capture virtual world models rather than actual
objects on four-dimensional trajectories (like a prey animal moving in the
environment), I suggest that the trajectory of speech with movement is
non-verbal. The ‘point’ is the point. A growth point is defined as the ‘initial form of thinking out of which
speech-gesture organization emerges’. (McNeill) It might also be called the
‘projection point’.
The core brain mechanisms
underlying human natural story telling can now be glimpsed. Damasio’s core
brain text generator in action describes the nonverbal internal structure of
gesturing behaviour in speech with movement. It may have functioned
projectively on 4D-space time for probably billions of years. Additional
control of outcomes is achieved by adding more dimensions or variables to the
modeling process, up to our present limit of 7+/-2.
I think that the ‘material’ of
speech with movement, via the mechanism of mirror neurons and the shape of the
trajectory itself, makes a ‘synctium’ of the communicating minds. They directly
collaborate to control the outcomes of the topics in which they are immersed.
The core brains are entrained, and the story is shared. However the logic is
‘projective’ on 4D space-time, and I have termed our everyday ‘logic in use’ as
‘visual projective logic’.
Growth point,
Projection point, or Simultaneous Expression of Functions?
Isabella Poggi is another author who has identified and thought about
the remarkable co-occurrence of synchronies that can be demonstrated in the
text of everyday life. In her demonstration of
the co-occurrence of movement, speech (in its many parameters) and meaning, she
stresses the notion of the simultaneity of occurrence of functions, expressed
in ‘the moment’.
She analyzes it in terms of two layers of meaning
(Poggi ‘Score….’ P10)
‘From
our two-layers score analysis we saw that while first-layer meanings most
frequently are content information or information on the speaker's mind, self
presentation is present solely but frequently in the second-layer meanings of
prosodic, gestural and facial communication.”
The reality of transmodality is hard to encompass in verbal theory. One struggles to contain in theory how semanticity can be enhanced by gesture, and yet be idiosyncratic, the product of the ‘just then, just that’ of immediate exegesis. The rebellion against Cartesian dualism is striking from so many authors, and the review by Keith Devlin ‘Goodbye Descartes’, identifies about a dozen authors who have made a particular point of denying Descartes’ epistemology.
I restate this non dual thesis from the neurological perspective in the model of the core brain projector and its cortical enhancement. In my analysis, ‘emotion’ is intrinsic to the melody, and as detailed in its patterning as the trajectory of the text itself. I suggest that the ‘growth point’ or projective ‘grasp’ has the detailed participation of the core non-verbal brain, and this is mirrored directly in the ‘other’, participant in parole. (Skoyles. No wonder then that memory also come with an ‘affective tone’, because the dichotomy of affective versus content was itself flawed. The amygdala nuclei in the core brain would appear to participate crucially here. With damage to the amygdala, it has been shown that the ability to interpret manifestations of fear and dread are also lost. The essential point about its activities is that they are non-verbal. The dichotomy of emotion/intellect may itself be a cultural artifact, a victim of Whitehead’s ‘fallacy of misplaced consciousness’.
What
are the theoretical options when you find the detailed co-occurrences of speech
and gesture that have been demonstrated by micro-dissections of narrative
performed by three of the contributors to this symposium? I will tabulate our
responses to this conundrum:
|
Co-occurrences of speech
and movement: Models |
Poggi |
McNeil |
Mair |
|
Semantic
Import |
Goals
and Beliefs |
Dialectic
Symbols-Actions |
Partial
manifestation of plans in action |
|
Integration
of speech and Movement |
Co-occurring
functions, separate layers |
A
priori, co-patterned at source |
Same
trajectory, transmodal, the ‘act on the model’ |
|
Dealing
with the future |
Goals |
Intentions |
Projection |
The Link to the Electronic Health Record
The fact that there might actually BE such natural minimal units in Natural Language has wide implications. The ‘text of everyday life’ is revealed to be a concatenation of instances, organized by plans of various scales, which span past and future, although expressed in the present. These templates exist at the level of ‘growth point’ minimal units, but are syntactically combined and entrained to make the text. The text is ‘projective’ from the ‘ground zero’ of the present, and accumulates behind as shared text.
It has been hard to achieve consensus on a simple design for the process of electronic record keeping. The lack of a shared standard or way of doing it that would enable communication between disparate systems has stymied progress with the real time computer interface for medical note keeping. Competition between systems has been immense, but none can take over the world, and yet to work, electronic medicine must communicate globally.
The simple model of text and generator works well for an Electronic Health Records system. It has an assumption that any text consists of the capture of objects in a process that is organized syntactically, i.e. sequentially. Forms or protocols do this, and the design of the program allows the creation of many different forms from objects from a shared vocabulary.
The present status of the evolution of Global Healthcare communication standards is approaching a convergence that could deliver just this base model for the ‘standard’. The standard is about structured data. It is about how systems and components can be exchanged and used, and thus has functional and semantic aspects.
There
have been two model types that have approached this. The first, largely through
work with a European (CEN ENV 13606, The GEHR Object Model) has tried to define
the components that an Electronic Health Record should be built from. This kind
of ‘top down’ modeling has produced many schemes, most non-implentable in any
way. A survivor has been the idea of the clinical object or ‘Archetype’ (www.openehr.org.)
The second approach began with the concepts of documents and mark up, and has standardized a ‘document header’ with standard components to allow navigation and retrieval of health related documents. A second level may contain templates for paragraphs or headings, and third would have ‘clinical objects’ contained in it. These would constitute a searchable database of structured data.
One such candidate for the clinical ‘instance’ is the Clinical Document Architecture (CDA www.HL7.org) This proposes a standard ‘information asset’ unit with standard structure (information plus markup), similar to the ‘instance’ or ‘grasp unit’ or ‘Growth Point’ that we are identifying for Parole.. It is proposed as a currency for healthcare transactions globally. (Mair 2002).
In my Sydney 2000 paper I use the metaphor of a ‘basket of objects’ for the proposed Healthcare communication standard. The purpose of the concept is to get healthcare interoperability up and running, getting 80% of the functionality with 20% of the effort. The basket and its objects is a ‘health event summary’ containing all harvested data objects from a health event. A health event has discrete boundaries in time, a beginning and an end. Often synonymous with an ‘encounter’, it is also more generic than that. In any ‘clinical workface’ application, these objects will be structured into ‘clinical templates’ which are ‘good to think’, like objects in a ‘diviner’s basket. (VW Turner). Clinical templates can be as diverse as language itself, but all the objects can be scooped up an exported to a standard basket, from which compliant applications can scoop them out again, and slot them into the templates of their choice.
There is the problem of meaning from immediate context, that is, from the arrangement of objects on their particular template. Then parallel here is again from linguistics, where a word has meaning both from its associative or paradigmatic surround, but also from its place in a syntagm. Again, Mcneill explores this argument. For a global computer standard this is easily fixed by having companion objects, which describe the relations between classes ‘on screen’ at the moment of capture.
The idea of objects being ‘good to think’, to make meaning out of, comes from Claud Levi Strauss and is a distant echo of his quest:
‘Between and existentialism idealism there is room for a new science and a new philosophy…...”
We can add ‘…..and a new implementation’. Such a minimalist healthcare communication standard does broadly follow the model of text generation outlined in the emergent theory of brain and behaviour outlined above. As we get to know more, it will provide more and more insight on the ergonomics of the real time interface design for healthcare workers.
Mike Mair
Timaru
NZ
References
Binkofski Ferdinand et al : ‘Broca’s Region Subserves Imagery of Motion: A combined Cytoarchitectonic and fMRI Study HumanBrainMapping11:273–285(2000)
Bouissac, P 19 Circus and Culture
1976 Indian University
Bouissac, P. 1991 “Semiotics and the Gaia Hypothesis: Toward a Restructuring of Western Thought,” Philosophy and the Future of Humanity 1(2),168-184.
Calvin,
W.H. ‘The Unitary Hypothesis’ A Common
Neural Circuitry for Planning ahead, and throwing’ In ‘Tools, Language, and Cognition in Human Evolution’, edited by Kathleen
R. Gibson and Tim Ingold.
Fourcin, A.J. ‘The Layngograph’ in Ventilatory and Phonological Control systems, Ed B
Wyke, London
Ciccourel, A.V. ‘Cognitive Sociology’ ,
Penguin, 1973
W. Condon & W. Ogston. Sound Film Analysis of Normal and Pathological Behaviour Patterns. The Journal of Nervous and Mental Disease. Vol. 143, 4: 338-345.
Damasio, Antonio R,
‘Descartes’ Error’ 1994 Harper Collins
Damasio, Antonio R,
‘The Feeling of What Happens’, 2000, Vintage
Eibl-Eibesfeldt, ‘Love
and Hate’,
Devlin, Keith ‘Goodbye
Descartes’, Wiley, 1977, p276. Devlin lists 17 authors from Husserl to Damasio.
Hawking, S ‘A brief History of Time’
Space Time publications, 1988
Health Level
Seven www.HL7.org
http://www.nzsoft.co.nz/ophthalmology.htm
Iverson, J. &
Goldin-Meadow, S.1998. Why people gesture as they speak. Nature 396:228.
ISOTC/215 WG1 The Modeling work group of the
International Standards Organization Technical Committee on Electronic Health
Records Standards http://www.cihi.ca/Roadmap/IHISD/ISO/program.shtml
‘Access to the Health Record’ draft technical
report
Leigh R and Zee D, ‘The Neurology of eye Movements’ OUP
1999
Levi-Strauss, Claude.
Structural Anthropology. 1968 The Allen Lane Press
Levi-Strauss, Claude.
The Savage Mind 1966 Weidenfield and Nicholas
Mair MW (2002) Electronic Health Record Standards (EHR)
–‘Healthcare Information Technology Standards’ Proceedings of International
Congress of Ophthalmology,
Mair M
(2001) Report on the Annual Health Level 7 Conference,
Mair M
(2001), Report on the ISO/TC215 – Health Informatics Standards Development
Mair M
Report on ISO/TC215 Meeting,
Mair MW (1998) “Comprehensive Ophthalmology Management”,
in Proceedings of the 1998
Mair M.W. (1986) “The Eye in the Control of Attention”, in The Biological Foundations of Gesture: Motor and Semiotic Aspects, Jean-Luc Nespoulous, Paul Perron, Andre Roche ecours (Eds.), Lawerence Erlbaum Associates, London, p.123-146
Mair M.W. ‘A Model of the Text Generator, in ‘the
Neurological Basis of Signs in Communication’,
Mair M.W. (1980) “The melody of the text: Review of ‘Circus
and Culture’ by Paul Bouissac”, Semiotica,
Vol.32, No.1/2, p.119-138, Indiana University Press, Mouton,
Mair M.W. (1978) Steps Toward Principles of Text Regulation, Toronto Semiotic Circle, Victoria University, Toronto.1978 no 2
Mair M.W. (1977) “Mirth measurement: A new technique”, in It’s a Funny Thing, Human, Chapman and Foot (Eds.), Oxford and New York (Pergamon), p.105-111
Mair M.W. (1977) “Rapport and timeless moments: A microdissection”, in Proceedings, Love and Attraction, an International Conference, Mark Cook and Glenn Wilson (Eds.) Oxford and New York (Pergamon).
Mair MW (1975) ‘What do Faces Mean’ Royal Anthropological Institute, 1975, , reprinted in Steps toward Principles of Text Regulation (above)
McNeill, D. Gesture Language Dialectic Acta Linguistica Hafniensia green - 1/19/02
McNeill, D. 1992. Hand
and Mind: What Gestures Reveal About Thought. Chicago, IL:
University of Chicago
Press.
Miller
G A ‘The Law of Seven Plus or Minus Two’
Poggi I & Magno
Caldognetto E. (1996): “A score for the analysis of gesture in multimodal
communication”. InL.Messing (Ed.) Workshop on the
Integration of Gesture in Language and Speech, Newark, Delaware and Wilmington, Delaware. October 7-8, 1996.
Poggi, I ‘Toward the
Alphabet and Lexicon of Gesture, Gaze, and
Touch Paper for Paul Bouissac’s multimodality symposium.
Polanyi, Michael. ‘Personal Knowledge’ London, RKP 1958
Saussure, F Cours de Linguistique
Generale Paris, 1916
Skoyles, J ‘Mirror
Neurons and the Motor Theory of Speech’ http://www2.psy.uq.edu.au/CogPsych/Noetica/OpenForumIssue9/
Turner, V.W. The Drums of Affliction’ 1967
Wittgenstein L Tractatus Logico-Philosophicus
ILLUSTRATIONS
Fig.1 Tropisms
mirth
violence
embarrassment
tears
harmony
bewilder
The
dialectic of the text continues on the axis vertical to the plane of the paper,
but can ‘fall into’ a tropism at any time I enumerated these:
·
Harmony: the complete model, the resolved problem
·
Violence: the continuation of the text by direct
action
· Mirth: the rupture of the model with ‘necker cube’ flip inversion of model relationships, leads to motor ‘uncontrol’
·
Tears : the immediate state of affairs that fragments
completely. Unlike the ‘cube flip’ of mirth, in which both interpretations of
an immediate state of play are co present, there is no immediately ensuing text
because there is no ‘state of play’ for that individual
·
Embarrassment: a configuration where there is a
juxtaposition of objects from normally mutually exclusive categories.
·
Bewilder: If a linear argument has been under way, and
then the state of play becomes an impossible model (like Escher art work), then
this halts the text.
Fig. 2 The Core Brain Text Generator
The Core Brain Text Generator (taken from ‘the Eye in the Control of
Attention’ ibid.). Cortical areas are caricatured with three paired plates,
occipital, tempero-parietal, and frontal, surrounding and acting as a resource
for the core brain cyclical text generator. The cycle is closed by ‘action on
the world’, which leads to the generation of the text as a concatenation of
percept/plan sequences or ‘points’.