Voice & body language
content
HC1: general introduction..................................................................................................................................... 3
Introduction of basic concepts .................................................................................................................................. 3
How important are voice and body language? .......................................................................................................... 4
How do voice and body language interact?............................................................................................................... 4
How do voice and body language develop with age? ............................................................................................... 5
How universal are aspects of voice and body language ............................................................................................ 6
Example of a study (surprise experiment) ................................................................................................................. 6
HC2: what to measure .......................................................................................................................................... 7
Introduction ............................................................................................................................................................... 7
Terminology ............................................................................................................................................................... 7
Measurements ........................................................................................................................................................... 7
HC3:accents and prominence ................................................................................................................................ 8
Accents and prominence ............................................................................................................................................... 8
Experiment 1: How important are visual cues for prominence perception? ............................................................ 9
Experiment 2: Where are the visual cues? ..............................................................................................................10
Crosslinguistic comparisons .........................................................................................................................................11
HC4: manipulation ............................................................................................................................................. 12
Prominence types ........................................................................................................................................................13
Intent manipulation .....................................................................................................................................................13
HC5: chunking .................................................................................................................................................... 14
Boundary marking ........................................................................................................................................................14
Experiment 1: reaction time experiment.................................................................................................................15
Experiment 2: classification experiment ..................................................................................................................15
Turn-taking ...................................................................................................................................................................16
HC6: synthesis .................................................................................................................................................... 17
Introduction: Speech synthesis ....................................................................................................................................17
Affect in synthetic voices .............................................................................................................................................18
James et al. (2020) ...................................................................................................................................................18
Scheutz, Schermerhorn and Kramer (2006) ............................................................................................................18
State-of-the-Art: Vall-E .................................................................................................................................................19
HC7: confidence marketing ................................................................................................................................. 19
Series of studies ...........................................................................................................................................................20
Experiment 1: Production of Uncertainty ................................................................................................................20
Experiment 2: Perception of uncertainty.................................................................................................................22
Experiment 3: Perception of uncertainty.................................................................................................................22
Experiment 4: manipulated data .............................................................................................................................23
Experiment 5: Cultural differences ..........................................................................................................................23
Experiment 5: Perception of uncertainty (based on Brennan & Williams, 1995) ...................................................24
HC8: error handling and feedback....................................................................................................................... 24
Study 1: negations in Dutch .....................................................................................................................................25
, Study 2: echoic reponses in Japanese......................................................................................................................27
HC9 emotions .................................................................................................................................................... 30
Perception study ......................................................................................................................................................30
Presence effects .......................................................................................................................................................30
Studies on emotion ..................................................................................................................................................32
HC10 Stance ....................................................................................................................................................... 35
Stance.......................................................................................................................................................................35
Irony .........................................................................................................................................................................37
HC11 mimicry..................................................................................................................................................... 38
Mimicry of words and sounds......................................................................................................................................39
Experiment 1: lexical adaptation .............................................................................................................................40
Experiment 2: phonological adaptation ..................................................................................................................40
Mimicry of gestures .....................................................................................................................................................41
HC12 Metaphors ................................................................................................................................................ 42
Chinese speakers of Mandarin .................................................................................................................................43
How about eye movements? ...................................................................................................................................44
Experiment 1: processing sentences .......................................................................................................................45
Experiment 2: mental time traveling .......................................................................................................................46
Exam questions .................................................................................................................................................. 48
,HC1: general introduction
This course goal:
• students learn about the way variations in voice and body language (VBL) can steer the interaction between
conversation partners or may signal socially relevant information.
When you follow this course, you will acquire
• basic knowledge about variations in VL
• techniques to record, measure and analyse aspects of VBL
• skills to critically read and reflect on scientific publications on VBL
Today’s programme
• Introduction of basic concepts
• Regarding Voice and Body Language:
o How important are voice and body language?
o How do voice and body language interact?
o How do voice and body language develop with age?
o How universal are aspects of voice and body language?
• Example of a study (surprise experiment)
• Practicalities
Introduction of basic concepts
Take the following sentence: “My girlfriend is pregnant.”
This sentence has one unique meaning: “My female partner with whom I am not (yet) married is expecting a child”
However, the same sentence can have different extra connotations. For example:
1. [Yesss] My girlfriend is pregnant (Joy)
2. [Oh no!!!] My girlfriend is pregnant (Fear)
3. [What??] My girlfriend is pregnant (Surprise)
4. [Arghhhh] My girlfriend is pregnant (Anger)
In other words: a sentence can be used in different ways so that extra meanings are generated (not expressed
through words or syntax)
Extra information can be derived from context (situational, discourse) or from the way a sentence is uttered
Non-verbal communication (NVC)
Informal definition: “The ensemble of features that do not determine what you say, but how you say it”
Two forms:
• Variation in Voice
• Variation in Body Language
Voice
Features that you can ‘hear” (encoded in the speech signal itself)
• Intonation (speech melody)
• Loudness, energy
• Tempo
• Rhythm
• Voice quality
• Pauses
Body Language
Features that you can “see” (not always encoded in the speech signal)
• Facial expressions
• Gaze patterns
• Hand gestures
• Pointing
• Posture
, • Distance
VOICE AND BODY LANGUAGE
There has especially been an interest in use of facial expressions; often without studying auditory prosodic features.
In this series of lectures, focus on the combined use of those 2 sets of features.
What can be signalled in one modality is also often cued in the other modality, but there may be strength differences.
How important are voice and body language?
Long time ago....
• Quintilianus (first century PC): Institutio oratoria
• Rethorics as the art to persuade an audience (e.g, stylistic tricks, ordering of information, ...)
Rethorics
• Rethorics was mainly dealing with oral language (e.g. Cicero)
• One important aspect of rethorics is called “pronunciatio”, which refers to various forms of nonverbal
communication: intonation, but also body language, facial expressions, gestures, ....
• Ideally, those features should match the content of the spoken utterances (e.g. happy message with a happy
voice and face)
Presidential debates: Current-day politicians are very much aware of the potential importance of nonverbal
communication
Importance of nonverbal features
• It is a popular statement to say that nonverbal features account for more than 90% of the communication.
• This can probably be traced back to early experimental work of Mehrabian and colleagues on the recognition of
emotions in stimuli with conflicting cues...
• but it is not at all clear whether it generalizes to all aspects of non-verbal communication
Study of nonverbal features: A relatively new field!
Interesting paradox:
• Strong intuition that nonverbal features matter a lot
• But we are only beginning to learn how inmportant they really are
The study of nonverbal features has long been hampered by a lack of tools to record, measure or analyse specific
features
How do voice and body language interact?
Multimodality
Our perception of reality is multimodal; our perceptual system integrates/fuses information coming from different
sensory modalities (vision, hearing, touch, taste, ...)
Imagine a dinner, served in:
• a nice restaurant with candle light and romantic music
• A dirty stable with cobwebs and the noise of a pneumatic boring machine
Visual information is very important, specifically signals coming from a speaker’s face
Visual expressions
• Multimodal communication has been the most natural form of human communication for ages
• It is a normal situation that a speaker observes his/her addressee, and that the addresee sees the speaker;
spoken communication without visual contact is a relatively new phenomenon and still occurs relatively
infrequently
• Starting assumption: human beings do not only interact through auditory information, but also through visual
information (in addition to other sensory information)
Relation between faces and speech
• Faces are often observed in combination with speech (talking heads)
• When people speak, we often see them speaking.
• The visual information in the face can have an effect on the way we perceive speech
content
HC1: general introduction..................................................................................................................................... 3
Introduction of basic concepts .................................................................................................................................. 3
How important are voice and body language? .......................................................................................................... 4
How do voice and body language interact?............................................................................................................... 4
How do voice and body language develop with age? ............................................................................................... 5
How universal are aspects of voice and body language ............................................................................................ 6
Example of a study (surprise experiment) ................................................................................................................. 6
HC2: what to measure .......................................................................................................................................... 7
Introduction ............................................................................................................................................................... 7
Terminology ............................................................................................................................................................... 7
Measurements ........................................................................................................................................................... 7
HC3:accents and prominence ................................................................................................................................ 8
Accents and prominence ............................................................................................................................................... 8
Experiment 1: How important are visual cues for prominence perception? ............................................................ 9
Experiment 2: Where are the visual cues? ..............................................................................................................10
Crosslinguistic comparisons .........................................................................................................................................11
HC4: manipulation ............................................................................................................................................. 12
Prominence types ........................................................................................................................................................13
Intent manipulation .....................................................................................................................................................13
HC5: chunking .................................................................................................................................................... 14
Boundary marking ........................................................................................................................................................14
Experiment 1: reaction time experiment.................................................................................................................15
Experiment 2: classification experiment ..................................................................................................................15
Turn-taking ...................................................................................................................................................................16
HC6: synthesis .................................................................................................................................................... 17
Introduction: Speech synthesis ....................................................................................................................................17
Affect in synthetic voices .............................................................................................................................................18
James et al. (2020) ...................................................................................................................................................18
Scheutz, Schermerhorn and Kramer (2006) ............................................................................................................18
State-of-the-Art: Vall-E .................................................................................................................................................19
HC7: confidence marketing ................................................................................................................................. 19
Series of studies ...........................................................................................................................................................20
Experiment 1: Production of Uncertainty ................................................................................................................20
Experiment 2: Perception of uncertainty.................................................................................................................22
Experiment 3: Perception of uncertainty.................................................................................................................22
Experiment 4: manipulated data .............................................................................................................................23
Experiment 5: Cultural differences ..........................................................................................................................23
Experiment 5: Perception of uncertainty (based on Brennan & Williams, 1995) ...................................................24
HC8: error handling and feedback....................................................................................................................... 24
Study 1: negations in Dutch .....................................................................................................................................25
, Study 2: echoic reponses in Japanese......................................................................................................................27
HC9 emotions .................................................................................................................................................... 30
Perception study ......................................................................................................................................................30
Presence effects .......................................................................................................................................................30
Studies on emotion ..................................................................................................................................................32
HC10 Stance ....................................................................................................................................................... 35
Stance.......................................................................................................................................................................35
Irony .........................................................................................................................................................................37
HC11 mimicry..................................................................................................................................................... 38
Mimicry of words and sounds......................................................................................................................................39
Experiment 1: lexical adaptation .............................................................................................................................40
Experiment 2: phonological adaptation ..................................................................................................................40
Mimicry of gestures .....................................................................................................................................................41
HC12 Metaphors ................................................................................................................................................ 42
Chinese speakers of Mandarin .................................................................................................................................43
How about eye movements? ...................................................................................................................................44
Experiment 1: processing sentences .......................................................................................................................45
Experiment 2: mental time traveling .......................................................................................................................46
Exam questions .................................................................................................................................................. 48
,HC1: general introduction
This course goal:
• students learn about the way variations in voice and body language (VBL) can steer the interaction between
conversation partners or may signal socially relevant information.
When you follow this course, you will acquire
• basic knowledge about variations in VL
• techniques to record, measure and analyse aspects of VBL
• skills to critically read and reflect on scientific publications on VBL
Today’s programme
• Introduction of basic concepts
• Regarding Voice and Body Language:
o How important are voice and body language?
o How do voice and body language interact?
o How do voice and body language develop with age?
o How universal are aspects of voice and body language?
• Example of a study (surprise experiment)
• Practicalities
Introduction of basic concepts
Take the following sentence: “My girlfriend is pregnant.”
This sentence has one unique meaning: “My female partner with whom I am not (yet) married is expecting a child”
However, the same sentence can have different extra connotations. For example:
1. [Yesss] My girlfriend is pregnant (Joy)
2. [Oh no!!!] My girlfriend is pregnant (Fear)
3. [What??] My girlfriend is pregnant (Surprise)
4. [Arghhhh] My girlfriend is pregnant (Anger)
In other words: a sentence can be used in different ways so that extra meanings are generated (not expressed
through words or syntax)
Extra information can be derived from context (situational, discourse) or from the way a sentence is uttered
Non-verbal communication (NVC)
Informal definition: “The ensemble of features that do not determine what you say, but how you say it”
Two forms:
• Variation in Voice
• Variation in Body Language
Voice
Features that you can ‘hear” (encoded in the speech signal itself)
• Intonation (speech melody)
• Loudness, energy
• Tempo
• Rhythm
• Voice quality
• Pauses
Body Language
Features that you can “see” (not always encoded in the speech signal)
• Facial expressions
• Gaze patterns
• Hand gestures
• Pointing
• Posture
, • Distance
VOICE AND BODY LANGUAGE
There has especially been an interest in use of facial expressions; often without studying auditory prosodic features.
In this series of lectures, focus on the combined use of those 2 sets of features.
What can be signalled in one modality is also often cued in the other modality, but there may be strength differences.
How important are voice and body language?
Long time ago....
• Quintilianus (first century PC): Institutio oratoria
• Rethorics as the art to persuade an audience (e.g, stylistic tricks, ordering of information, ...)
Rethorics
• Rethorics was mainly dealing with oral language (e.g. Cicero)
• One important aspect of rethorics is called “pronunciatio”, which refers to various forms of nonverbal
communication: intonation, but also body language, facial expressions, gestures, ....
• Ideally, those features should match the content of the spoken utterances (e.g. happy message with a happy
voice and face)
Presidential debates: Current-day politicians are very much aware of the potential importance of nonverbal
communication
Importance of nonverbal features
• It is a popular statement to say that nonverbal features account for more than 90% of the communication.
• This can probably be traced back to early experimental work of Mehrabian and colleagues on the recognition of
emotions in stimuli with conflicting cues...
• but it is not at all clear whether it generalizes to all aspects of non-verbal communication
Study of nonverbal features: A relatively new field!
Interesting paradox:
• Strong intuition that nonverbal features matter a lot
• But we are only beginning to learn how inmportant they really are
The study of nonverbal features has long been hampered by a lack of tools to record, measure or analyse specific
features
How do voice and body language interact?
Multimodality
Our perception of reality is multimodal; our perceptual system integrates/fuses information coming from different
sensory modalities (vision, hearing, touch, taste, ...)
Imagine a dinner, served in:
• a nice restaurant with candle light and romantic music
• A dirty stable with cobwebs and the noise of a pneumatic boring machine
Visual information is very important, specifically signals coming from a speaker’s face
Visual expressions
• Multimodal communication has been the most natural form of human communication for ages
• It is a normal situation that a speaker observes his/her addressee, and that the addresee sees the speaker;
spoken communication without visual contact is a relatively new phenomenon and still occurs relatively
infrequently
• Starting assumption: human beings do not only interact through auditory information, but also through visual
information (in addition to other sensory information)
Relation between faces and speech
• Faces are often observed in combination with speech (talking heads)
• When people speak, we often see them speaking.
• The visual information in the face can have an effect on the way we perceive speech
