Embodied Language, Cognition, and Communication ( E LCC): Spontaneous gesture with speech

1. Embodied Language, Cognition, and Communication (E LCC):Spontaneous gesture with speech Alan Cienki and Irene Mittelberg Engelse taalkundeAfdeling Taal en Communicatie Faculteit der Letteren LCC presentation, 6 November 2008

2. Some background: gesture studies • Spoken language studied in fields such as anthropology, sociology (e.g., CA), ethnography of communication • Gesture sometimes studied in these contexts • The study of “non-verbal communication” in clinical psychology

3. Gesture studies as its own field • E.g., Adam Kendon’s work since 1970s • Gesture: Visible action as utterance (2004) • Landmark – Hand and mind: What gestures reveal about thought (David McNeill 1992) • International Society for Gesture Studies (ISGS) • ISGS conference series • Founding of the journal Gesture in 2001

4. What are we calling a gesture? • A visible, distinct, effortful movementof part of the body (Kendon) • Special status of manual gestures

5. Background on gesture • Range of conventionality of form-meaning associations – from highly idiosyncratic to highly conventionalized “Kendon’s continuum” (Kendon 1982; McNeill 1992)gesticulation - pantomime - emblems - sign language

6. Gesturing is not usually just handwaving • Not random, structureless or unsystematic • Basic structuring principles of gesture at work: • schematic patterns • abstractions from physical actions, objects, and relations • roles of metonymy and metaphor

7. On the coding of gestural forms • Four parameters are often used (based on research on sign languages): • hand shape • palm orientation • location in gesture space • movement

8. Notation of form features in gestures • status of fingers differentiated (Bressem 2006, in prep.)

9. Notation of form features in gestures • form class “flat hand” (Bressem 2006, in prep.)

10. Models of gesture space etc. -1 0 1 2 3 extreme periphery upper periphery center cc lower right left (Fricke 2005/ in prep.) (McNeill 1992)

11. Some functions of gestures • Discourse structuring function • Pragmatic/interactional function • Referential function • These can overlap • Gestures can have multiple functions

12. Some functions of gestures • Referential function • Concrete reference • Abstract reference

13. Some functions of gestures Concrete reference Oh, an essay. You have to `write fast! {video clip} (video clips have not been included in this PowerPoint since we do not have permission to publish them)

14. Some functions of gestures • Abstract reference: Representation of the abstract (non-physical) in terms of the concrete e.g., an idea as a space, form, or motion • Metaphor

15. Example: metaphors in gestures that, there’s never a situation, that is, ideally r- -- where there’s, an ideal, <@well@>, <at least there isn’t in my life>, where there’s something right, and something absolutely wrong. What you have to do is draw your line, and figure out on which side of it you fall.

16. Example: metaphors in gestures {video clip}

17. Example: metaphors in gestures Possible analysis might include: • SITUATIONS ARE LOCATIONS • MAKING A DECISION IS DRAWING A LINE • EVALUATING YOUR BEHAVIOR IS POSITIONING YOURSELF • Note timing of onset of gestures with respect to speech

18. Metaphor • Metaphor as not just a matter of language, but of mapping one conceptual domain (with imagery!) onto another (Lakoff & Johnson 1980, 1999; and many other works…) • Therefore, we should see evidence of conceptual metaphors in different forms of human expression • Potential for multimodal expression of metaphor in language use (Cienki & Müller 2008; Forceville & Urios-Aparisi, forthcoming; Müller 2008)

19. Side note: perspective in gesture studies • If studying gesture can help in meaning-making, whose meaning-making are we talking about? • the speaker’s? • the addressee’s? • another observer’s? (the researcher’s?)

20. Interaction of metaphor and metonymy in co-verbal gestures Irene Mittelberg

21. points of departure: How do speech and gesture share the semiotic work of rendering abstract knowledge domains more graspable? Previous research has shown that spontaneous co-speech gestures may reveal: embodied and situated aspects of abstract reasoning materializations of spatial metaphor interaction of iconicity and metaphor (also in signed languages) Bouvet 2001; Calbris 2003; Cienki 1998, 2005; Cienki & Müller 2008; McNeill 1992, 2005; Müller 1998, 2003, 2008; Nuñez 2004; Nuñez & Sweetser 2006; Sweetser 1998, 2007; Taub 2001; P. Wilcox 2000, 2004; S. Wilcox 2004; Williams 2008; inter alia

22. While metaphor is central to accessing abstract domains and the gestures in the data of this study are essentially metaphorical in nature - metaphorical modes do not suffice to account for the semantic / pragmatic processes in the multimodal data. => not necessarily a direct iconic link between • the form of the gesture and • the form of its referent. example: palm-up open hand gesture on mention of “noun” as sitting on palm-up open hand (Müller 2004) • contiguity relation between hand and imaginary object

23. Grammatical categories (noun, verb)contiguity relations betweenhands& imaginaryobjects {video clip}

24. cognitive-semiotic approach to co-speech gesture Charles Sanders Peirce’s (1931, 1955) pragmaticist theory of signs, focus on interpretation perspective for analysis: addressee -------------- Roman Jakobson’s (1956, 1960 1961,1963, 1966) theory of metaphor & metonymy -------------- Cognitive linguistics: metaphor, metonymy, image schemas (Dirven 2002; Gibbs 1994, 2006; Johnson 1987; Lakoff 1987, 1993; Lakoff & Johnson 1980, 1999; Radden 2000; Panther & Thornburg 2003, 2004; Sweetser 1990; Taub 2001; Wilcox 2004) Gesture research (Calbris 1991; Cienki 1998; Kendon 2000, 2004; McNeill 1992, 2005; Müller 1998, 2004, 2007; Webb 1996; Streeck 2003; Sweetser 1998; reseach project “Towards a grammar of gesture,“ Berlin / Viadrina FFO) --------- Data: genre: academic discourse / introductory linguistics courses subject matter: linguistic form, grammar, & linguistic theory gesture types: “referential gestures” of abstract concepts and structures (Müller 1998) meta-linguistic & meta-grammatical co-speech gestures

25. inventory of gestural sign carriers in meta-linguistic & meta-grammatical gestures data-driven approach resulted in a set of prominent hand configurations & motion patterns recurrent forms and recurrent form-meaning mappings • manipulation of imaginary objectsof different sizes and dimensions (solid objects and containers with an inside) • geometrical shapes basic shapes: squares, triangles, regtangles, (semi-) circles, etc. lines: straight, curved, wavy along horizontal, vertical und diagonal axes • other patterns evoke some of image-schematic patterns suggested in the cognitive linguistics literature => ‘material’ basis for metaphorical and metonymic projections Mittelberg (2006, 2008, fc)

26. Johnson‘s original definition of “image schemas”: “recurring, dynamic patterns of our perceptual interactions and motor programs that give coherence and structure to our experience.”(Johnson 1987, XIV) Johnson (2005, 31): “But let us not forget that the truly significant work done by image schemas is tied to the fact that they are not merely skeletons or abstractions. They are recurring patterns of organism-environment interactions that exist in the felt qualities of our experience, understanding and thought.”

27. image schemata instantiated in the gesture data SUPPORT (“puoh-tray,” “puoh-cup”) CONTAINMENT (“puoh-cup,” “fist,” “pcoh-box”) OBJECT (“puoh-tray,” “puoh-cup,” “pcoh-box,”, “fist”) SOURCE-PATH-GOAL (“hori-trace,” “vert-trace,” “diag-trace; ) BALANCE (“puoh-tray-bh,” “puoh-cup-bh,” “fist-bh,” “sym-offshoot”) SCALE (“scale”) CYCLE (“circle-bh,” “wrist-rotation,” “rotation lateral”) ITERATION (“wrist-rotation,” “rotation lateral”) FRONT-BACK (“push,” “pull”) FORCE (“push,” “pull”, “hori-join,” “sym-offshoot”) ADJACENCY CONTACT PART-WHOLE (Johnson 1987; Gibbs 2005; Hampe 2005; Lakoff & Johnson 1999; Mandler 1996; Talmy 1988) Gesture research: iconic and metaphoric gestures (Bouvet 2001; Calbris 2003; Cienki 1998, 2005; Cienki & Müller 2008; Mittelberg 2008, fc.; Müller 1998, 2007, 2008, fc.; Sweetser 1998; Williams 2004, 2008; Zlatev 2005; inter alia)

28. metonymic pathways as emphasized by relevance theorists (e.g., Sperber & Wilson 2002) and cognitive linguists (Gibbs 1994; Lakoff 1987; Fauconnier & Turner 2002), “metonymic pathways” are part of the cognitive competence of speakers and listeners => provide natural inference schemata (Panther & Thornburg 2003, 2004) One definition of metonymy: “Metonymy is a cognitive process in which one conceptual entity, the vehicle, provides mental access to another conceptual entity, the target, within the samedomain or ICM.”(Kövecses and Radden 1998: 39) “Proust is on the top shelf.”

29. Jakobson (1956):“Two types of language and two types of aphasic disturbances” inspired by Peirce’s (1955) concepts of similarity &contiguity theory of metaphor &metonymy as two major modes of association & signification Jakobson devotes equal attention to both tropes and those who use his work posit a continuum between the two. (cf. Croft 1993; Dirven 2002; Lodge 1977; Radden 2000; Waugh 1998) no absolute categories: they interact in any process of meaning-making to various degrees hierarchy: the function of a given sign depends on the preponderance of one mode over the others. The aim here is to show how metonymic principles lead into the interpretation of metaphoric gestures: metonymy first, metaphor second

30. Jakobson further distinguished between internal metonymy: contiguity relationships between sign & object synecdoche (pars pro toto) “all hands on deck” (hands stand for entire bodies) external metonymy: contiguity relationships between signs or between entities in extra-linguistic, discursive, or conceptual space “metonymy proper”(adjacency, contact) “The White House remains silent.” “The White House” stands for U.S. president / spokesperson (place for person)

31. I. internal metonymy (within metaphoric gestures)the formation of the gestural sign gestures exhibit the basic principle of semiotic representation => that is, that all semiotic representations are partial these abstraction processes are driven by synecdoche: by picking out the locally pragmatically salient aspects of an object / action e.g., shape, dimensions, texture, manner of movement for example, a gesture representing a picture frame (Bouvet 2001; Cienki & Müller 2006; Gibbs 1994; Müller 1998, fc.; Sweetser 2001; Taub 2001; Wilcox 2004; Wilcox & Morford 2007) most work on metonymy in co-speech gesture has focused on this kind of metonymic motivation

32. internal metonymy (within metaphoric gestures) a horizontal line stands for a sentence “a sentence is a string of words”

33. external metonymy (within metaphoric gestures) space extending between fingers is assigned meaning (adjacency/contact) here: the word “Diana” in a sentence

34. external metonymy / metonymy of place “sub- category” {video clip}

35. Conotiguity relations between body and objects “there is” pointing gesture (“shifters”; Jakobson) “the main verb” CONTAINMENT (cup) metaphor [object manipulation & external metonymy] “there is what‘s called the main verb”

36. Contiguity relations holding between hands & object “sub-category” imaginary OBJECTseemingly being held between two hands [external metonymy] comparably low position in gesture space & in relation to body [metonymy of place]

37. metonymy first, metaphor second (Mittelberg & Waugh fc.) 1. Metonymy (external): source: hands (physical) target: object (imaginary) 2. Metaphor: source: OBJECT (imaginary) target: IDEA (conceptual) co-speech: “sub-category“ (technical) metonymic target = metaphoric source (imaginary object) => PLACE FOR FUNCTION “sub-category”

38. some conclusions The assumption that some metaphors are grounded in metonymy seems to hold in the gestures discussed. (Barcelona 2000; Dirven 2002; Fauconnier & Turner 2002; Geeraerts 2002; Goossens 1995; Jakobson 1956, 1960; Lakoff 1987; Lodge 1977; Radden 2000; inter alia) figures of thought may manifest themselves in gesture even if the concurrent speech is non-figurative (“noun”, “Diane”, “teach-”, “infinitive”, “sentence”, etc.) (cf. Cienki 1998, 2008; Cienki & Müller 2008,fc.; Müller 2008)

39. Gesture and cognitive linguistics Alan Cienki

40. Given: • the goal in cognitive linguistics to characterize language in a way that is coherent with what is known about cognitive processing in general • the cog.ling. focus on actual linguistic behavior • the universality of gesture with speech • Gesture can provide additional insight into conceptualizations underlying semantic and grammatical structures

41. Gesture in relation to the study of several topics in cognitive linguistics • metaphor • cognitive models • mental spaces • metonymy and cognitive reference points • image schemas

42. Metaphor in words and gestures (Cienki 1998, 2008a) • Gestures can provide evidence of activation (on some level) of an image being used to characterize a given topic • (a mapping from one domain to another, i.e., [conceptual] metaphor, on some level) • The inclusion of gesture data provides a way out of the criticism of circularity of conceptual metaphor theory • (that is: language provides evidence of conceptual metaphors, and we find evidence of conceptual metaphors in language)

43. Metaphor in words and gestures (Cienki 1998, 2008a) • Metaphor can appear in words without accompanying gestures • Metaphor can appear in gestures without accompanying metaphorically used words • The ‘same’ metaphor can appear in words and gestures in an utterance • Different metaphors can be expressed simultaneously in words and gestures

44. Cognitive models • A cognitive model as the way an individual characterizes knowledge for a given domain(compare schema, script, frame, etc.) • Cultural models as shared cognitive models • These models can be constituted by metaphors to varying degrees

45. Russian model of chestnost’ Dlia menia chestnost’ eto nekaia abso`liutnaia kategoriia For me chestnost’ is a kind of absolute category. Kogda vot iest’ situatsiia, When there’s this situation, seichas postupit’ chestno `tak. then [you need] to act honestly like this. {video clip}

46. Cognitive models and gesture (Cienki 1999) • Metaphors can highlight aspects of cognitive/cultural models • Gesture data can provide evidence of the speaker’s online use of a cognitive model • This can be valuable for the study of (aspects of) cognitive models which are not expressed in words

47. Mental spaces • Developed from work by Gilles Fauconnier (1985) • Refers to “small conceptual ‘packets’ constructed as we think and talk” • Linguistic forms • reveal the speaker’s use of various mental spaces, • and they provide cues to the listener for how to construct the mental spaces used by the speaker • Examples: hypothetical constructions (if…, then…)

48. Mental spaces: gestural example (Cienki 2008b) It depends on thestudent, but it also dependson the teacher. {video clip}

49. Mental spaces, metaphor, and gesture • IDEAS AS SPACES(a kind of ontological metaphor, i.e., ABSTRACT AS CONCRETE)

50. Metonymy and reference points • Langacker (1993) notes how we regularly make use of one (known) entity to invoke mental contact with another (less well-known) entity: • > cognitive reference point • Metonymy as one means of doing this