Effects of the Teacher’s multimodal communication over the Students’ memory

1. Effects of the Teacher’s multimodal communication over the Students’ memory Giorgio Merola Dip.di Psicologia Università “La Sapienza”, Roma - Lyon, July 15-18, 2005

2. The Teacher’s Multimodal Communication • A Teacher communicates not only through words, but with voice, gesture, gaze, head movement and facial expression, body posture, movements in space, and physical contact • With all of these means the Teacher fulfills his/her communicative tasks

3. The Teacher’s tasks • Affective-relational tasks: to manage emotions and social relationships in classroom interaction • Conversational tasks: turn-taking, back-channel • Cognitive tasks: to provide knowledge in different formats: motor, perceptual, propositional (conceptual, abstract, textual, metatextual, metacognitive)

4. Cognitive tasks • A Teacher’s central task is to provide knowledge and totransmit skills • not only concrete (perceptual or motor information) but also abstract, conceptual, propositional information can be conveyed through gesture (Kendon, 1980; McNeill, 1992). • even gaze, facial expression and posture are not devoted solely to convey the relational and affective aspects of classroom interaction (De Landsheere & DelChambre, 1996). • Gaze can carry information also on propositional contents both concrete and abstract, and has important functions in the delivery of a discourse (Poggi, 2002)

5. Studies about Gesture and Cognition • Goodwin (1986); Kendon (1988); Alibali et al (2000); McNeill (1992) ; Beattie & Shovelton (1999) : relationship between gesture, language and mind (cognitive processes); cognitive categorization of gesture • Kita (2000): role of representational gestures • Goldin-Meadow (2003): effects of gesture on the students’ cognitive processes

6. Studies about direct effects of gestures on the students’ cognitive processes • Goldin-Meadow (2003): teachers’ gestures influence students’ strategies in math tasks • Valenzeno, Alibali & Kletzky (2003): gestures influence students’ learning (kindergarten students) • Roth (2000): gesture and comprehension of scientific disciplines

7. Goldin-Meadow • Studies the effects of the teacher’s gestures on the students’ processes focusing on the mismatch between gesture and words • Suggests to study the effects of gestures on the cognitive processes implied in different disciplines

8. “We know that gestures can help listeners secure a message conveyed in speech when it too conveys that message, but that doesn’t always help(…) At the moment we don’t have any idea why gesture that is redundant with speech makes it easier to grasp the speaker message in some cases but not in others” (Goldin- Meadow, 2003)

9. Signal complexity • In her studies about the effects of gestures, Goldin-Meadow consider only deictic signals • But other types of hand movements, like iconic gestures, seem to be more complex and their effects can be more variable because these signals convey more informations (not necessarly redundant)

10. McNeill’s Typology • Iconic gestures • Metaphoric gestures • Beat gestures • Cohesive gestures • Deictic gestures

11. Iconic gestures taking the Character’s or the Observer’s viewpoint • Gesture reveals the speaker’s viewpoint i.e. “Sylvester climbs up the drainpipe” • Character Viewpoint (C-VPT): move one’s arms up and down, as if climbing a ladder • Observer Viepoint (O-VPT): the hand represents Sylvester as a whole and rises upward

12. Viewpoint • When the narrator’s gestures express a C-VPT, we feel that he is inside the story. The speaker is seeing the event as if the person were performing the act, rather than taking the viewpoint of an observer of the event. • An O-VPT gesture excludes the speaker’s body from the gesture space and his hands play the part of the character as a whole (McNeill, 1992)

13. Relationship between speaker and referent • C-VPT: the speaker uses his body to represent the referent as a whole, or a character who acts with or on it. • O-VPT: speaker’s arms or hands represent the character or the object (referent) as a whole The same arm can either represent a tree or the arm of the character who fells a tree

14. Single words FLOWER • C-VPT: the speaker pretends to smell a flower • O-VPT: the speaker draws petals in the air (pictographic gesture) CAT • C-VPT: the speaker’s arm represents the cat’s paw • O-VPT: difficult to represent

15. Children’s viewpoint • McNeill (1992) suggests that children produce much more C-VPT gestures than O-VPT ones because C-VPT expresses the lesser narrative distance they assume

16. Studies about viewpoint effects on listeners • Beattie & Shovelton (2002) suggested that the different gestural viewpoints might pragmatically be very important because they might differ in the amount of semantic information they communicate

17. Previous works • Merola & Poggi 2004 investigated the effects of the teacher’s multimodal communication on the students’ comprehension and memory of a short story. • Our interest was to single out which elements of multimodal communication influence the students’ memory of the story

18. Viewpoint • The Teachers’ facial expression resulted to be the most important signals for children’s story retrieval • Facial signals often expressed a character viewpoint

19. A new study.Research goals • To single out wich elements of multimodal communication specifically influence the Students’ memory of words • To investigate the differences in the Students’ performance when they see the Teacher expressing either Character Viewpoint or Observer Viewpoint • To obtain quantitative and qualitative data about the influence of the Teachers’ gestural communication on the Students’ cognitive processes

20. Hypotheses • The students of teachers that communicate only by speech (without gesture and signals of other modalities) will obtain lower scores in memory tests • As C-VPT gestures are more accessible to children (McNeill,1992), the observation of C-VPT gestures during the listening of words or short story enhances the students’ performance in memory tasks: students more frequently remember words accompained by C-VPT signals • The effects should be clearer for younger students

21. Participants • 5 Primary Schools in Rome: • 359 students of 20 classes (3 first grade classes; 4 second grade; 4 third; 2 forth and 7 fifth) • 20 teachers: 26 to 58 years old (mean age=44,5)

22. Procedure • Pre-test: Raven Progressive Matrices (A-Ab-B series and I-II series, 1947); memory of 3 lists of 8 words pronounced by the experimenter without gestures or signals in other modalities; • Experiment: memory of words pronounced by the teacher with her personal style (C-VPT;O-VPT; no multimodal signals);

23. Conditions • Words pronounced without gestures (N teachers) • Words accompanied by C-VPT gestures (C-VPT teachers) • Words accompanied by O-VPT gestures (O-VPT teachers)

24. Features of the words 24 words (3 lists of 8): • 8 can be represented by C-VPT gestures • 8 more suitable for an O-VPT representation • 8 can be represented by either C-VPT or O-VPT • Same lenght: 5 letters (2 syllables) • Same frequency • Balanced imagery value

25. Instructions to teachers • On the basis of previous observations of her multimodal communication, each teacher was instructed to use her most typical modality: respctively: N modality: To pronounce words without gestures C-VPT modality: To pronounce words by accompanying them with C-VPT gestures O-VPT modality: To pronounce words with O-VPT gestures The instructions were explained and some examples were provided to Teachers

26. Multimodal Analysis • For each teacher the signals of all modalities were analysed by considering their viewpoint and other parameters, like bi- or tri- dimensionality, vividness and movements

27. Data analysis • We match the frequences of word recall in the 3 conditions and in each class level • for each condition, we consider only the words that teachers pronounced expressing required viewpoint

28. Data analysis • A covariance analysis 3X4 was carried on (modality X class level, factors for indipendent measures), by using the pre-test performance as covariate • Analysis results reveal a main effects for MODALITY and CLASS LEVEL factors (F(2,267)=103,02, p<.001 and F(3,267)=7,02, p<.001) • There is an interaction between these two factors (F(6,267)=5,98, p<.001)

29. Note: the 4 th classes were excluded from the analysis because there were few words that could be matched to the 3-conditions

30. results • In N condition the children’s improvement is linear from first grade to fifth grade students, • In O-VPT modality there is a low and non-linear progress • C-VPT condition is optimal for first graduate students: they perform as good as fifth grade students

31. Multimodality • Multimodality conditions, both C-VPT gestures and O-VPT gestures, always represent an advantage in all the class levels, BUT • this advantage is much stronger for younger children, while it becomes smaller in the following years • In the multimodality conditions first grade students perform like older students

32. First grade students • Duncan Test showed that words recall frequency in C-VPT condition is statistically different from words recall frequency in O-VPT condition (p<.05)

33. Recall frequency: first grade vs. fifth grade students

34. First grade vs fifth grade • As we can see in the previous graphic, the difference between first grade students and fifth grade students decreases in multimodality condition. With C-VPT gestures, first grade children performed little better than fifth grade students.

35. Recall frequency: first grade vs third grade students

36. First grade vs third grade • Previous graph shows that the “viewpont preference” changes during children development. • Third grade students seem to perform better with O-VPT gesture. • The importance of multimodal signals start to decrease

37. Modality vs pretest scorefirst grade and fifth grade students

38. Some examples of frequency recall (%) by 1 st grade students

39. First grade

40. Some examples of frequency recall (%) by 5 th grade students

41. Conclusions (1) • We wanted to single out wich elements of teachers’ multimodal communication influence pupils’ memory • We studied the effects of one element that is potentially crucial: the viewpoint expressed by gestures • Results show that C-VPT gestures seem to be of higher communicative power than O-VPT gestures for younger children, and that multimodality is less necessary in older students

42. Conclusions (2) • It is important to study the effects of viewpoint and of other elements of multimodality, in different tasks like story recall • Data analysis in this study about viewpoint effects in story recall seem to confirm the decrease, during cognitive developement, of C-VPT positive effects. • Qualitative analysis suggests us to consider other elements of multimodal signals that can produce different effects on childrens’ memory