3.2 Experimental procedure

1. 2. Classification of Modern Greek Verbs • The classification of Modern Greek Motion Verbs (Antonopoulou, 1987): • exhibits an overall semantic structure found with motion verbs of several Indo-European languages • present some certain aspectual idiosyncrasies. • Antonopoulou’s taxonomic sets were defined with the use of two groups of criteria: • First group’s criteria: transitivity, causativity, agentivity, intentionality and aspect (non measurable with sensorimotor methods at the moment). • Second group’s criteria: change-of-location, directionality, path, dependent motion, change-of-orientation, manner, medium and instrumentality. • 3. Sensorimotor experiment • 3.1 Verb collection criteria • 25 predicates (Table 1) from Antonopoulou’s long list • Both intransitive and transitive verbs were included (assumed zero knowledge of verbal valence) • Requirements imposed by the nature of sensorimotor experiments and lab limitations: • They can be performed by humans: i.e. kalpazo (gallop) applies to horses only • They represent events with one human participant only • 1. Goals • Combination of linguistic knowledge from corpus data with sensorimotor data • Specification of the minimum conceptual representation of a motion • Perceptual grounding of abstract notions, such as directionality and transitivity • Grounding (monolingual) semantic classifications of verbs in sensorimotor data • Crosslingual study of motion verbs (Modern Greek and American English) Modeling the semantics of motion verbs based on sensorimotor measurementsSionti Marietta1, Claudino Leonardo2, Markantonatou Stella3, Penstein - Rose Carolyn4, Aloimonos Yiannis21University of Athens, 2University of Maryland, 3Institute for Speech and Language/ ‘R.C.Athena’, 2Carnegie Mellon University • 3.2 Experimental procedure • Equipment: A full body Moven system (16 inertial motion trackers, fig.1) • Subjects: • The sexual distribution is 5 men and 3 women • All subjects were native Greek speakers. • They were encouraged to enact each meaning • Action performance: • A step-wise, standardized procedure • The verb or the sentence was uttered by the experimenter • Action limited to a floor area restricted by a quadrangle. • As few objects as possible • Analysis: The detailed sensor data are analyzed to identify latent • factors that represent stable patterns across the many dimensions • of low level data. • These factors appear as : • discrete sets (synergies) of joint angles • and orientations associated. • the base motor data matrix • Extraction of the average action (for the most motor representation) • Normalization in length of these average actions • PCA-processed matrix (Santello, 1998), 2-D visualization of the action scatter (fig.3). • Table1. Verb classes fig. 2a: X-Moven Suit's body planes (manual) Sagittal planeBlue Frontal / coronal plane Green Transverse plane Red fig. 2b: X-Moven Suit's axes (manual) x Blue Y Green Z Red fig.1: Subject performing the action • 3.3 Results • Plot for all verbs • Concerning the two following types of plots: • fig.3 : matrix allowing a 2-D visualization of the action scatter and reflecting 3-D and time information. • fig.4 –10: • combination of vectors of positions that are analyzed in 3 rectangular components (x, y, z) for each joint angle (fig.2), in each principal component (PC1 and PC2). • If x is whiter (more energy) than y,z  displacement back and forth on the sagittal plane; • If y is whiter (more energy) than x,z displacement right and left on the frontal plane; • If z is whiter (more energy) than x,y displacement upward or downward according to the transverse plane. • The two main visual groups of actions are • the red rectangular: includes walking like actions, (leg-related) • the blue ellipsis: includes manipulation of an object (arm-related) (fig.3). • projected on both PC1 (distinct use of hips) and PC2 (emphasis on knees and shoulders) (fig. 4). • The main disparities from the leg-related group (red) in the scatter plot: • actions 6 (mpousoulo-crawl) and 7 (treho-run): - when projected on PC2 (both knees and shoulder), because of the shape of posture (e.g. frequent movement of the arms and shoulders) and speed -but when projected on PC1 - hips x movements highly concerned, (displacement on the sagittal plane) (fig.4)- then closer to the walking-like actions, which is consistent with their linguistically labeled groups (red). • The visualization presented here supports the results of brain imaging studies • the schematic of the distributed semantic representation in the brain of action verb processing is based on the body parts performing them (Wermter et al., 2005), e.g. arm-related and leg-related. • Similarities of this visualization with Antonopoulou’s classification (1987): • Antonopoulou’s classes are more fine-grained than the two big categories in fig.3 –rectangular or ellipsis • the actions of the green ellipsis (leg-related group) correspond to pido-group and kateveno-group • the actions of blue ellipsis correspond to girizo2-group, katevazo and anevazo group • the actions 13, 14 and 15 (pido-jump) that share the same morphological representation pido but not the same performance, eventually form one tight ‘group’. • The actions 20, 21 (girizo1) and 22, 23, 24, 25 (girizo2) also share the same morphological representation girizo but they are • closer to each other on the PC1 projection • separated on the PC2 projection. • It is consistent with Antonopoulou’s classification because the girizo1-group is mainly about leg motion and the girizo2-group about object manipulation that necessarily involves arm displacement as is clearly indicated on PC2. • The distance of actions 7 (treho- run) and 6 (mpousoulo-crawl) from the rectangular group is not actually an indication of deviation of these actions from the class. • In Modern Greek (Antonopoulou, 1987) treho can form its own class on the grounds of velocity. Here, it is projected on the same principal component, which highlights the right and left movement of the hips, especially at the ¾ of the action (fig.4). • In English, Levin (1993) enlists run and crawl both in the run and the meander class, probably based on intentionality, which is non measurable with sensorimotor methods. fig. 3 Average Actions Greek Verbs fig.4 Weights of each joint-direction-timefeature according to the 10-first principal components fig. 6 Weights of each joint-direction-time feature according to the 10-first principal components for the 'anevazo'-group • 3.4 Results • Plots per class • -Grouping the actions as in linguistic classes, • in order to observe the common characteristics • that these actions might have, in terms of joint-angles • -Highlighted areas are of importance in PCA fig. 6 Weights of each joint-direction-time feature according to the 10-first principal components for the 'girizo2'-group • Future work: • Comparison of the plots-per class based on the average action with new plots-per class generated from each subjects data. • Generation of plots per action, in order to see their characteristics and compare them to the plots-per class. • Expansion of the study with American English sensorimotor data. fig.5 Weights of each joint-direction-time feature according to the 10-first principal components for the 'perpato'-group References: Antonopoulou, E. (1987). Prototype theory and the meaning of verbs, with special reference to Modern Greek verbs of motion. Unpublished PhD thesis. University of London Arbib, M. A. (2008) From Grasp to Language: Embodied Concepts and the Challenge of Abstraction. Journal of Physiology Paris 102: pp. 4-20 Fadiga, L., et al. (2006). Language in shadow. Social Neuroscience, 1(2): pp.77-89. Kemmerer, D. (2006). Action verbs, argument structure constructions, and the mirror neuron system. In M. Arbib (Ed.), Action to language via the mirror neuron system. Cambridge University Press. Lakoff, G.& Johnson, M. (1999). Philosophy in the Flesh: The Embodied Mind and Its Challenge to Western Thought. Basic Books, New York Levin, B. (1993). English Verb Classes and Alternations: A Preliminary Investigation, University of Chicago Press, Chicago, IL. Santello, M., Flanders, M., & Soechting, J.F. (1998). Postural hand synergies for tool use. Journal of Neuroscince, 18. Wermter, S., Weber, C., Elshaw, M., Gallese, V. & Pulvermüller, F. (2005). Grounding neural robot language in action. Biomimetic Neural Learning for Intelligent Robots. Springer