1 / 41

Modeling Expressivity in ECAs

Modeling Expressivity in ECAs. Catherine Pelachaud, Maurizio Mancini LINC - University of Paris 8. Behavior. Behavior is related to the (Wallbott, 1998): quality of the mental state (e.g. emotion) it refers to quantity (somehow linked to the intensity factor of the mental state)

kaiser
Download Presentation

Modeling Expressivity in ECAs

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Modeling Expressivity in ECAs Catherine Pelachaud, Maurizio Mancini LINC - University of Paris 8

  2. Behavior • Behavior is related to the (Wallbott, 1998): • quality of the mental state (e.g. emotion) it refers to • quantity (somehow linked to the intensity factor of the mental state) • Behaviors encode: • content information (the ‘What is communicating’) • expressive information (the ‘How it is communicating’) • Behavior expressivity refers to the manner of execution of the behavior

  3. Behavior Representation • Behavior= signal shape, movement, expressivity • Gesticon (Gesture Lexicon): Dictionary of behavior description (B. Krenn, H. Pirker OFAI) • Modalities: • face • hand and arm gesture • body movement and posture • gaze

  4. Behavior Representation • Face: • facial expression • duration: onset, apex, offset • Gesture: • phases: preparation, pre-hold-stroke, stroke, post-hold-stroke, retraction • gesture shape and movement for each phase • Head: • head direction • head movement • Gaze • eye direction

  5. Expressivity Dimensions • Expressivity dimensions: • Spatial: amplitude of movement • Temporal: duration of movement • Power: dynamic property of movement • Fluidity: smoothness and continuity of movement • Repetitiveness: tendency to rhythmic repeats • OverallActivation: quantity of movement across modalities • Implemented for gesture and facial expression

  6. Overall Activitation • Threshold filter on atomic behaviors during APML tag matching • Determines the number of nonverbal signals to be executed.

  7. Spatial Parameter • Amplitude of movement controlled through asymmetric scaling of the reach • Space that is used to find IK goal positions • Expand or condense the entire space in front of agent

  8. Temporal parameter • Determine the speed of the arm movement of a gesture's meaning-carrying stroke phase • Modify speed of stroke Stroke shift / velocity control of a beat gesture Y position of wrist w.r.t. shoulder [cm] Frame #

  9. Fluidity • Continuity control of TCB interpolation splines and gesture-to- gesture • Continuity of arms’ trajectory paths • Control the velocity profiles of an action coarticulation X position of wrist w.r.t. shoulder [cm] Frame #

  10. Power • Tension and Bias control of TCB splines; • Overshoot reduction • Acceleration and deceleration of limbs Hand shape control for gestures that do not need hand configuration to convey their meaning (beats).

  11. Repetitivity • Technique of stroke expansion: Consecutive emphases are realized gesturally by repeating the stroke of the first gesture.

  12. Expressivity • Expressivity values may act: • over the whole animation: EmoTV, analysis-synthesis • every instant of the movement: Greta Music • on every gesture: GEMEP corpus • on a particular phase of the gesture: Attract attention study • Exploratory studies based on various data types: • acted data • real data • 2D cartoon • literature

  13. Research Issue • Behavior representation • what to encode • at which levels of representation • dynamism • Implementation refinement

  14. Expressivity over the WHOLE animation • One set of values are set • extracted manually from annotation of real data video corpus • extracted automatically from video corpus of acted data using image analysis technique

  15. Expressivity over the WHOLE animation From annotations to animation Jean-Claude Martin, Laurence Devillers, LIMSI-CNRS; Maurizio Mancini, Paris8 • Consider what is visible: • annotate signals, how there are displayed, how they are perceived • Model what is visible: • represent signals, animate them with expressivity • Two-steps approach • Elaborate rules by analysis (video corpus) • Animate by “copy synthesis” • No model of the processes underlying the display of the signals • Annotation  extraction  animation

  16. Annotation Extraction Generation Animation EmoTV clip GRETA animation Steps Expressivity over the WHOLE animation • EmoTV: 51 clips French TV Interviews • Annotation • Emotion labels: single and blend of emotions • Multimodal behavior • Expressivity dimensions

  17. Expressivity over the WHOLE animation Expressivity of gestures in mixed emotion values obtained from literature values obtained from annotation

  18. Video

  19. Real World Sensing Virtual Sensing Personality Scene Ontology Interpretation Sensory Storage Perception Goals Attention Planning Generation Expressivity over the WHOLE animation Real and Virtual World SensingA. Raouzaiou, G. Caridakis, K. Karpouzis ICCS; C. Peters, E. Bevacqua, M. Mancini, Paris 8

  20. Expressivity over the WHOLE animation ApplicationScenario

  21. Expressivity over the WHOLE animation • Interpretation: • gesture specified by symbolic name • facial expression: • emotion label: if the facial expression corresponds to one of the prototypical facial expression of emotions • otherwise, FAPs values • Planning: • modulate expressivity parameters • module emotional expressions

  22. Expressivity over the WHOLE animation Generation • Input to ECA system: • a symbolic description of a gesture • emotion label or set of FAPs value • expressivity parameters value • Output: facial and gesture animation

  23. Video

  24. Expressivity on Every Frames • Greta Music Roberto Bresin, KTH - Maurizio Mancini, Paris8 • One set of expressivity paramters values are extracted automatically from acoustic data in real-time and fet to the ECA system.

  25. Expressivity on Every Frames Design a tool for real-time visual feedback to expressive performance

  26. Expressivity on Every Frames From music expression to facial expression • From acoustic cues to emotion : extraction of acoustic cues: • Tempo, Sound Level, Articulation (staccato/legato), Attack Velocity, Spectrum, Vibrato rate, Vibrato Extent, Pitch • From acoustic cues to facial expression: mapping of acoustic cues: • music emotion  facial expression • music volume spatial and power • music tempo temporal and overall activation • music articulation fluidity

  27. Expressivity on Every Frames Music version of Greta This version of Greta allowsonly the following actions: • head moving • eyes blinking • emotional expression • skin colouring Input: Expressivity ParametersOutput: FAP values (animated head)

  28. Expressivity on Every Frames Video

  29. Expressivity on Gesture Phases Attraction of attention • Corpus:videos from traditional animation that illustrate different types of conversational interaction • Analysis: the modulations of gesture expressivity over time play a role in managing communication, thus serving as a pragmatic tool France Telecom

  30. Expressivity on Gesture Phases Attraction of attention • Irregularities • the principle of anticipation: • it enhances the visibility of a gesture • it enhances our propensity to gaze at this gesture. • Discontinuities • create a contrast between successive gestures • function to isolate a particular gesture from a sequence of gestures France Telecom

  31. Expressivity on Gesture Phases Irregularity

  32. Expressivity on Gesture Phases Irregularity – slow motion

  33. Expressivity on Gesture Phases Discontinuity – slow motion

  34. Expressivity on Gesture Phases Application: ECA as web presenter Discontinuity – spatial parameter

  35. Expressivity on Each Gesture • Annotation of multimodal behavior • signals on 3 modalities • arm gesture • head movement • body movement • phases of each signal • each phase: physical shape + timing • expressivity of each signal

  36. Expressivity on Each Gesture Annotation of multimodal behavior

  37. Expressivity on Each Gesture Animation format • we have defined a file format for the specification of behavior for our animation engine (Greta) • we translate from the XML annotation file (ANVIL) to the engine animation file ANVIL annotation animation file

  38. Expressivity on Each Gesture Animation format

  39. Demo • demo!

More Related