Speech-gesture: grounded imagery in acoustic parameters

Arika Okrent, Linguistics & Psychology, University of Chicago

When people speak, they often produce iconic or metaphorically iconic gestures with their arms, hands, and body. Because the speech and the gesture come out in two different channels, the vocal and the manual, the term 'gesture' is generally exclusively associated with motions of the hands alone. This paper treats 'gesture' not as a physical notion, but as a semiotic notion. When 'gesture' is seen independently of manual actions, one finds a rich area of 'gesture' which emerges as people speak not in their limbs, but in the acoustic parameters of their speech. This gesture is produced simultaneously with their speech.

1.The little bird flew uuuup [with rising pitch] and doooown [with falling pitch].

2. Baseball shouldbemorelikethis [said quickly] and less...like...this. [slowly].

3. If you want me to be able to read your order you shouldn't write 'over easy' [said with sloppy, run-together articulation]. You should write 'o-ver ea-sy' [with precise articulation].

In each of these examples, acoustic parameters articulated with the speech iconically ground (cf. Taub 1996, Liddell 1998) the imagery in the meaning of the utterances. In (1) directional movement is grounded in pitch; in (2) relative pace is grounded in relative speed; in (3) relative neatness is grounded in relative carefulness of articulation.

This paper first argues that the manipulation of the acoustic parameters in the examples like those above is an expression of the same cognitive processes which produce manual gestures during speech as described by McNeill (1992). Both manual and spoken gestures occur simultaneously with speech. Both are one side of the unpacking of a linguistic thought--the side which directly grounds the imagery in that thought. The other side of the unpacking is the speech, which does not directly ground imagery, but rather references meaning mediated through a conventional lexicon.

The second part of the paper discusses some important differences between manual gesture and speech gesture. One difference is that while manual gesture is not produced by the same articulators as those which produce the speech, spoken gesture IS produced by the same articulators as the speech. This can lead to conflict when the acoustic parameter to be used by the gesture is involved in a lexical contrast important for the speech; example (1) would have to be uttered differently in a tone language because pitch would be in use for speech, and not as free to participate in gesture. Yet, even where such conflicts exist, gesture can still come through. Another difference is that acoustic parameters are less capable of iconic representation than are the parameters used in manual gestures. More concepts in the world can be iconically represented visually than acoustically. Still, a range of iconicity is possible through pervasive metaphors which map acoustic parameters onto visual parameters. For example, pitch is mapped onto vertical space. A series of experiments are discussed which show that this mapping is active both perceptually and in production.

Liddell, S. K 1998. Grounded Blends, Gestures, and Conceptual

Shifts. Cognitive Linguistics, 9, 3, 283-314.

McNeill, D., 1992. Hand and Mind: What gestures reveal about

thought. Chicago: University of Chicago Press.

Taub, S. 1996. Analogue-building: A model of linguistic iconicity.

Ms., University of California, Berkeley.