When Gestures and Words Synchronize: Exploring A Human Lecturer's Multimodal Interaction for the Design of Embodied Pedagogical Agents

Lai Wei, Kenny K.N. Chow*

*Corresponding author for this work

Research output: Chapter in book/report/conference proceedingConference proceedingpeer-review

Abstract

Embodied Pedagogical Agents (EPAs) possess significant potential for using multimodal interactions to effectively convey social and educational information. While previous studies have examined the impact of EPAs on learning performance, the incorporation of meaningful gestures into EPAs remains relatively unexplored. This study employs a case study approach and systemic functional-multimodal discourse analysis to investigate the multimodal interactions of a human lecturer during teaching and explores the relationship between in-class meaningful gestures and corresponding words. We analyze the alignment between gestures, words, and contextual meanings, with a specific focus on personal pronouns, adjectives, adverbs, and verbs. The findings contribute to the understanding of lecturers' multimodal orchestration in classroom settings. Additionally, we provide a preliminary design guideline for EPAs that integrates meaningful word-synchronized gestures, using multimodal information to enhance students' embodied learning experiences. This research contributes to the advancement of embodied pedagogical agent design and the broader field of multimodal interaction.

Original languageEnglish
Title of host publicationCSCW 2023 Companion
Subtitle of host publicationConference Companion Publication of the 2023 Computer Supported Cooperative Work and Social Computing
EditorsMorgan Ames, Susan Fussell, Eric Gilbert, Vera Liao, Xiaojuan Ma, Xinru Page, Mark Rouncefield, Vivek Singh, Pamela Wisniewski
Place of PublicationNew York
PublisherAssociation for Computing Machinery (ACM)
Pages39-44
Number of pages6
ISBN (Print)9798400701290
DOIs
Publication statusPublished - 14 Oct 2023
Event26th ACM Conference on Computer-Supported Cooperative Work and Social Computing, CSCW 2023 - Minneapolis, United States
Duration: 14 Oct 202318 Oct 2023
https://dl.acm.org/doi/proceedings/10.1145/3584931 (Conference proceedings)

Publication series

NameProceedings of the ACM Conference on Computer Supported Cooperative Work, CSCW
PublisherAssociation for Computing Machinery (ACM)

Conference

Conference26th ACM Conference on Computer-Supported Cooperative Work and Social Computing, CSCW 2023
Country/TerritoryUnited States
CityMinneapolis
Period14/10/2318/10/23
Internet address

Scopus Subject Areas

  • Software
  • Computer Networks and Communications
  • Human-Computer Interaction

User-Defined Keywords

  • Gesture
  • Human behavior
  • Multimodal interaction
  • Pedagogical agent
  • Word synchronized

Fingerprint

Dive into the research topics of 'When Gestures and Words Synchronize: Exploring A Human Lecturer's Multimodal Interaction for the Design of Embodied Pedagogical Agents'. Together they form a unique fingerprint.

Cite this