Summary Multimodal Communication
By Joy Mennings
Lecture 1 (23/10/2023) ........................................................................................................................... 4
The Multimodal Parallel Architecture ................................................................................................. 4
Modalities ............................................................................................................................................ 6
Why modality? .................................................................................................................................... 6
Lecture 2 (30/10/2023) ........................................................................................................................... 7
Modality .............................................................................................................................................. 7
Definition of a modality ....................................................................................................................... 7
Sensory signals ................................................................................................................................ 8
Sensory stimulus.......................................................................................................................... 8
Perceptual system ....................................................................................................................... 8
Production ................................................................................................................................... 8
Sensory primitives ....................................................................................................................... 8
Duration ....................................................................................................................................... 9
Cognitive representations ............................................................................................................... 9
Cognitive primitives ..................................................................................................................... 9
Combinatorial principles ............................................................................................................. 9
Class ............................................................................................................................................... 10
Modality interactions and interfaces................................................................................................. 10
Co-durative interactions ................................................................................................................ 10
Mixed-durative interactions .......................................................................................................... 10
Non-durative interactions.............................................................................................................. 11
Emergent interactions ............................................................................................................... 12
Lecture 3 (6/11/2023) ........................................................................................................................... 12
Grammar types .................................................................................................................................. 12
Simple grammars ........................................................................................................................... 12
One-unit grammar ..................................................................................................................... 12
Two-unit grammar ..................................................................................................................... 13
Linear grammar ......................................................................................................................... 13
Complex grammars ........................................................................................................................ 13
Simple phrase grammar ............................................................................................................ 13
Categorical grammar ................................................................................................................. 13
Recursive grammar .................................................................................................................... 14
Overview........................................................................................................................................ 14
1
, Grammar interactions ....................................................................................................................... 15
How do grammars interact? .......................................................................................................... 15
How are the grammars distributed relative to each other? .......................................................... 15
Case study: substitutive allocation ............................................................................................ 16
Lecture 4 (13/11/2023) ......................................................................................................................... 16
Conceptual semantics ....................................................................................................................... 17
Multimodal interactions .................................................................................................................... 17
Single multimodal interactions ...................................................................................................... 17
Conceptual Domain Model for single images............................................................................ 17
Types of auxiliary domains ........................................................................................................ 18
Overview.................................................................................................................................... 19
Sequential multimodal interactions .............................................................................................. 19
Conceptual Domain Model for sequential images .................................................................... 20
Cues for changes in domain constructions ................................................................................ 20
Overview.................................................................................................................................... 21
Summary ........................................................................................................................................... 21
Lecture 5 (20/11/2023) ......................................................................................................................... 21
The Multimodal Parallel Architecture ............................................................................................... 21
Modality ........................................................................................................................................ 22
Grammar........................................................................................................................................ 22
Meaning......................................................................................................................................... 22
Examples............................................................................................................................................ 23
Unimodal expressions ................................................................................................................... 23
Multimodal expressions ................................................................................................................ 23
Silence ............................................................................................................................................... 24
Lecture 6 (27/11/2023) ......................................................................................................................... 25
Case studies ....................................................................................................................................... 25
2-Image Expressions .......................................................................................................................... 25
Generalized emergent state for 2-I expressions ............................................................................ 25
Before-After messages as 2-I expressions ..................................................................................... 25
Example 1 .................................................................................................................................. 26
Example 2 .................................................................................................................................. 26
How BA-expressions make meaning.............................................................................................. 26
Example 1 .................................................................................................................................. 26
Example 2 .................................................................................................................................. 27
The generalized BA-construction ................................................................................................... 27
2
, The full-formed F-function for BA-expressions ............................................................................. 28
The nature of change..................................................................................................................... 29
Continuity and activity constraints ................................................................................................ 30
Distinct entities .......................................................................................................................... 30
Identical entities ........................................................................................................................ 30
Examples of violations of the continuity constraint .................................................................. 30
Examples of trouble with activity .............................................................................................. 31
Examples of trouble with inherited values ................................................................................ 32
Playing with modality .................................................................................................................... 32
Not all 2-I expressions are BA-expressions .................................................................................... 32
BA-features ................................................................................................................................ 33
Summary ....................................................................................................................................... 33
Internet memes as multimodal expressions ..................................................................................... 34
Comparison of the model .............................................................................................................. 34
Meme evolution ............................................................................................................................ 35
Lecture 7 (4/12/23) ............................................................................................................................... 36
Why do we have the MPA? ................................................................................................................ 36
Types of reference ......................................................................................................................... 36
How can we talk about and point at what we see? ...................................................................... 37
Spatial structure ........................................................................................................................ 37
Structures and modalities ............................................................................................................. 38
Cross-modal interactions ................................................................................................................... 38
Graphicw........................................................................................................................................ 38
Synesthesia .................................................................................................................................... 38
3
, Lecture 1 (23/10/2023)
Multimodality is what happens when an expression uses more than one modality to communicate a
message. The context of the message doesn't matter that much. Examples of multimodal messages are
posters, memes, advertisements, comic strips, diagrams, infographics, traffic signs, speech-gesture
alignment, text-emoji, personal documents, maps and rebuses.
Multimodal messages can be characterized in terms of:
• Modalities used: for example, verbal speech and bodily movements or written speech and
visual images.
• Ordering:
o Ordering in time: they happen simultaneously and are durative (they disappear).
o Ordering in space: they are non-durative (permanent).
• Senses/perceptions: for example, do you need to use your eyes only or your eyes and ears.
Example
1. Does this message use more than 1 modality?
Yes, it uses visuals (drawings) and written
speech (VROOOAAOOOOOAW, Good Year and
Jean Gration).
2. What does this expression mean? Sometimes
the meaning is a simple description of what is
seen on the image. So, what scene do we see?
a. The scene is that of a racetrack.
b. Two men are looking at some
document.
c. Race cars driving by.
d. One of the cars is sponsored by ‘good
year’.
e. They make lots of noise.
3. What part of this meaning is carried by what modality?
a. Visual: (a) the scene is that of a racetrack; (b) Two men are looking at some document;
and (c) Race cars driving by.
b. Verbal: (d) One of the cars is sponsored by ‘good year’; and (e) They make lots of noise.
4. Is the job done jointly, or mostly by one modality? Most meaning is carried by the visuals. The
written parts add to the whole, but not essentially.
5. Look closely at these 3 written parts:
a. “Good Year”: are part of what is shown here. If you were there, you would see them.
b. “VROOOAAOOOOAW”: is a written impression of what can be heard here. If you were
there, you wouldn’t see it.
c. “Jean Graton: is superimposed on the image. It’s part of neither the scene, nor the
action.
The Multimodal Parallel Architecture
The multimodal parallel architecture characterizes communicative forms as interactions of one or
several modalities, grammar types and conceptual structure. Multimodality follows from combinations
of different emergent forms, resulting in multiple activation patterns across the parallel architecture.
4
By Joy Mennings
Lecture 1 (23/10/2023) ........................................................................................................................... 4
The Multimodal Parallel Architecture ................................................................................................. 4
Modalities ............................................................................................................................................ 6
Why modality? .................................................................................................................................... 6
Lecture 2 (30/10/2023) ........................................................................................................................... 7
Modality .............................................................................................................................................. 7
Definition of a modality ....................................................................................................................... 7
Sensory signals ................................................................................................................................ 8
Sensory stimulus.......................................................................................................................... 8
Perceptual system ....................................................................................................................... 8
Production ................................................................................................................................... 8
Sensory primitives ....................................................................................................................... 8
Duration ....................................................................................................................................... 9
Cognitive representations ............................................................................................................... 9
Cognitive primitives ..................................................................................................................... 9
Combinatorial principles ............................................................................................................. 9
Class ............................................................................................................................................... 10
Modality interactions and interfaces................................................................................................. 10
Co-durative interactions ................................................................................................................ 10
Mixed-durative interactions .......................................................................................................... 10
Non-durative interactions.............................................................................................................. 11
Emergent interactions ............................................................................................................... 12
Lecture 3 (6/11/2023) ........................................................................................................................... 12
Grammar types .................................................................................................................................. 12
Simple grammars ........................................................................................................................... 12
One-unit grammar ..................................................................................................................... 12
Two-unit grammar ..................................................................................................................... 13
Linear grammar ......................................................................................................................... 13
Complex grammars ........................................................................................................................ 13
Simple phrase grammar ............................................................................................................ 13
Categorical grammar ................................................................................................................. 13
Recursive grammar .................................................................................................................... 14
Overview........................................................................................................................................ 14
1
, Grammar interactions ....................................................................................................................... 15
How do grammars interact? .......................................................................................................... 15
How are the grammars distributed relative to each other? .......................................................... 15
Case study: substitutive allocation ............................................................................................ 16
Lecture 4 (13/11/2023) ......................................................................................................................... 16
Conceptual semantics ....................................................................................................................... 17
Multimodal interactions .................................................................................................................... 17
Single multimodal interactions ...................................................................................................... 17
Conceptual Domain Model for single images............................................................................ 17
Types of auxiliary domains ........................................................................................................ 18
Overview.................................................................................................................................... 19
Sequential multimodal interactions .............................................................................................. 19
Conceptual Domain Model for sequential images .................................................................... 20
Cues for changes in domain constructions ................................................................................ 20
Overview.................................................................................................................................... 21
Summary ........................................................................................................................................... 21
Lecture 5 (20/11/2023) ......................................................................................................................... 21
The Multimodal Parallel Architecture ............................................................................................... 21
Modality ........................................................................................................................................ 22
Grammar........................................................................................................................................ 22
Meaning......................................................................................................................................... 22
Examples............................................................................................................................................ 23
Unimodal expressions ................................................................................................................... 23
Multimodal expressions ................................................................................................................ 23
Silence ............................................................................................................................................... 24
Lecture 6 (27/11/2023) ......................................................................................................................... 25
Case studies ....................................................................................................................................... 25
2-Image Expressions .......................................................................................................................... 25
Generalized emergent state for 2-I expressions ............................................................................ 25
Before-After messages as 2-I expressions ..................................................................................... 25
Example 1 .................................................................................................................................. 26
Example 2 .................................................................................................................................. 26
How BA-expressions make meaning.............................................................................................. 26
Example 1 .................................................................................................................................. 26
Example 2 .................................................................................................................................. 27
The generalized BA-construction ................................................................................................... 27
2
, The full-formed F-function for BA-expressions ............................................................................. 28
The nature of change..................................................................................................................... 29
Continuity and activity constraints ................................................................................................ 30
Distinct entities .......................................................................................................................... 30
Identical entities ........................................................................................................................ 30
Examples of violations of the continuity constraint .................................................................. 30
Examples of trouble with activity .............................................................................................. 31
Examples of trouble with inherited values ................................................................................ 32
Playing with modality .................................................................................................................... 32
Not all 2-I expressions are BA-expressions .................................................................................... 32
BA-features ................................................................................................................................ 33
Summary ....................................................................................................................................... 33
Internet memes as multimodal expressions ..................................................................................... 34
Comparison of the model .............................................................................................................. 34
Meme evolution ............................................................................................................................ 35
Lecture 7 (4/12/23) ............................................................................................................................... 36
Why do we have the MPA? ................................................................................................................ 36
Types of reference ......................................................................................................................... 36
How can we talk about and point at what we see? ...................................................................... 37
Spatial structure ........................................................................................................................ 37
Structures and modalities ............................................................................................................. 38
Cross-modal interactions ................................................................................................................... 38
Graphicw........................................................................................................................................ 38
Synesthesia .................................................................................................................................... 38
3
, Lecture 1 (23/10/2023)
Multimodality is what happens when an expression uses more than one modality to communicate a
message. The context of the message doesn't matter that much. Examples of multimodal messages are
posters, memes, advertisements, comic strips, diagrams, infographics, traffic signs, speech-gesture
alignment, text-emoji, personal documents, maps and rebuses.
Multimodal messages can be characterized in terms of:
• Modalities used: for example, verbal speech and bodily movements or written speech and
visual images.
• Ordering:
o Ordering in time: they happen simultaneously and are durative (they disappear).
o Ordering in space: they are non-durative (permanent).
• Senses/perceptions: for example, do you need to use your eyes only or your eyes and ears.
Example
1. Does this message use more than 1 modality?
Yes, it uses visuals (drawings) and written
speech (VROOOAAOOOOOAW, Good Year and
Jean Gration).
2. What does this expression mean? Sometimes
the meaning is a simple description of what is
seen on the image. So, what scene do we see?
a. The scene is that of a racetrack.
b. Two men are looking at some
document.
c. Race cars driving by.
d. One of the cars is sponsored by ‘good
year’.
e. They make lots of noise.
3. What part of this meaning is carried by what modality?
a. Visual: (a) the scene is that of a racetrack; (b) Two men are looking at some document;
and (c) Race cars driving by.
b. Verbal: (d) One of the cars is sponsored by ‘good year’; and (e) They make lots of noise.
4. Is the job done jointly, or mostly by one modality? Most meaning is carried by the visuals. The
written parts add to the whole, but not essentially.
5. Look closely at these 3 written parts:
a. “Good Year”: are part of what is shown here. If you were there, you would see them.
b. “VROOOAAOOOOAW”: is a written impression of what can be heard here. If you were
there, you wouldn’t see it.
c. “Jean Graton: is superimposed on the image. It’s part of neither the scene, nor the
action.
The Multimodal Parallel Architecture
The multimodal parallel architecture characterizes communicative forms as interactions of one or
several modalities, grammar types and conceptual structure. Multimodality follows from combinations
of different emergent forms, resulting in multiple activation patterns across the parallel architecture.
4
