Enhanced Human-Machine Interactive Learning for Multimodal Emotion Recognition in Dialogue System

Clement H.C. Leung; James J. Deng; Yuanxi Li

doi:10.1145/3579654.3579764

Enhanced Human-Machine Interactive Learning for Multimodal Emotion Recognition in Dialogue System

Clement H.C. Leung
, James J. Deng^*
, Yuanxi Li

^*Corresponding author for this work

Department of Computer Science

Research output: Chapter in book/report/conference proceeding › Conference proceeding › peer-review

3 Citations (Scopus)

Abstract

Emotion recognition has been well researched in mono-modality in the past decade. However, people express their emotion or feelings naturally via more than one modalities like voice, facial expressions, text, and behaviors. In this paper, we propose a new method to model deep interactive learning and dual modalities (e.g., speech and text) to conduct multimodal emotion recognition. An unsupervised triplet-loss objective function is constructed to learn representation of emotional information from speech audio. We extract text emotional feature representation by transfer learning of text-To-Text embedding from T5 pre-Trained model. Human-machine interaction like user feedback plays a vital role in improve multimodal emotion recognition in dialogue system. Deep interactive learning model is constructed by explicit and implicit feedback. Human-machine interactive learning enhanced transformer model can achieve higher levels of accuracy and precision than their non-interactive counterparts.

Original language	English
Title of host publication	ACAI 2022 - Conference Proceedings
Subtitle of host publication	2022 5th International Conference on Algorithms, Computing and Artificial Intelligence
Place of Publication	New York
Publisher	Association for Computing Machinery (ACM)
Number of pages	7
ISBN (Electronic)	9781450398343
ISBN (Print)	9781450398336
DOIs	https://doi.org/10.1145/3579654.3579764
Publication status	Published - 23 Dec 2022
Event	5th International Conference on Algorithms, Computing and Artificial Intelligence, ACAI 2022 - Sanya, China Duration: 23 Dec 2022 → 25 Dec 2022

Publication series

Name	ACM International Conference Proceeding Series

Conference

Conference	5th International Conference on Algorithms, Computing and Artificial Intelligence, ACAI 2022
Country/Territory	China
City	Sanya
Period	23/12/22 → 25/12/22

UN SDGs

This output contributes to the following UN Sustainable Development Goals (SDGs)

SDG 9 Industry, Innovation, and Infrastructure

User-Defined Keywords

human-machine interaction
interactive learning
Multimodal emotion recognition
transformer model

Access to Document

10.1145/3579654.3579764

Cite this

Leung, C. H. C., Deng, J. J., & Li, Y. (2022). Enhanced Human-Machine Interactive Learning for Multimodal Emotion Recognition in Dialogue System. In ACAI 2022 - Conference Proceedings: 2022 5th International Conference on Algorithms, Computing and Artificial Intelligence Article 104 (ACM International Conference Proceeding Series). Association for Computing Machinery (ACM). https://doi.org/10.1145/3579654.3579764

Leung, Clement H.C. ; Deng, James J. ; Li, Yuanxi. / Enhanced Human-Machine Interactive Learning for Multimodal Emotion Recognition in Dialogue System. ACAI 2022 - Conference Proceedings: 2022 5th International Conference on Algorithms, Computing and Artificial Intelligence. New York : Association for Computing Machinery (ACM), 2022. (ACM International Conference Proceeding Series).

@inproceedings{884a4924788744109f103eca0bb47449,

title = "Enhanced Human-Machine Interactive Learning for Multimodal Emotion Recognition in Dialogue System",

abstract = "Emotion recognition has been well researched in mono-modality in the past decade. However, people express their emotion or feelings naturally via more than one modalities like voice, facial expressions, text, and behaviors. In this paper, we propose a new method to model deep interactive learning and dual modalities (e.g., speech and text) to conduct multimodal emotion recognition. An unsupervised triplet-loss objective function is constructed to learn representation of emotional information from speech audio. We extract text emotional feature representation by transfer learning of text-To-Text embedding from T5 pre-Trained model. Human-machine interaction like user feedback plays a vital role in improve multimodal emotion recognition in dialogue system. Deep interactive learning model is constructed by explicit and implicit feedback. Human-machine interactive learning enhanced transformer model can achieve higher levels of accuracy and precision than their non-interactive counterparts.",

keywords = "human-machine interaction, interactive learning, Multimodal emotion recognition, transformer model",

author = "Leung, \{Clement H.C.\} and Deng, \{James J.\} and Yuanxi Li",

note = "Publisher Copyright: {\textcopyright} 2022 Copyright held by the owner/author(s). Publication rights licensed to ACM.; 5th International Conference on Algorithms, Computing and Artificial Intelligence, ACAI 2022 ; Conference date: 23-12-2022 Through 25-12-2022",

year = "2022",

month = dec,

day = "23",

doi = "10.1145/3579654.3579764",

language = "English",

isbn = "9781450398336",

series = "ACM International Conference Proceeding Series",

publisher = "Association for Computing Machinery (ACM)",

booktitle = "ACAI 2022 - Conference Proceedings",

address = "United States",

}

Leung, CHC, Deng, JJ & Li, Y 2022, Enhanced Human-Machine Interactive Learning for Multimodal Emotion Recognition in Dialogue System. in ACAI 2022 - Conference Proceedings: 2022 5th International Conference on Algorithms, Computing and Artificial Intelligence., 104, ACM International Conference Proceeding Series, Association for Computing Machinery (ACM), New York, 5th International Conference on Algorithms, Computing and Artificial Intelligence, ACAI 2022, Sanya, China, 23/12/22. https://doi.org/10.1145/3579654.3579764

Enhanced Human-Machine Interactive Learning for Multimodal Emotion Recognition in Dialogue System. / Leung, Clement H.C.; Deng, James J.; Li, Yuanxi.
ACAI 2022 - Conference Proceedings: 2022 5th International Conference on Algorithms, Computing and Artificial Intelligence. New York: Association for Computing Machinery (ACM), 2022. 104 (ACM International Conference Proceeding Series).

Research output: Chapter in book/report/conference proceeding › Conference proceeding › peer-review

TY - GEN

T1 - Enhanced Human-Machine Interactive Learning for Multimodal Emotion Recognition in Dialogue System

AU - Leung, Clement H.C.

AU - Deng, James J.

AU - Li, Yuanxi

PY - 2022/12/23

Y1 - 2022/12/23

N2 - Emotion recognition has been well researched in mono-modality in the past decade. However, people express their emotion or feelings naturally via more than one modalities like voice, facial expressions, text, and behaviors. In this paper, we propose a new method to model deep interactive learning and dual modalities (e.g., speech and text) to conduct multimodal emotion recognition. An unsupervised triplet-loss objective function is constructed to learn representation of emotional information from speech audio. We extract text emotional feature representation by transfer learning of text-To-Text embedding from T5 pre-Trained model. Human-machine interaction like user feedback plays a vital role in improve multimodal emotion recognition in dialogue system. Deep interactive learning model is constructed by explicit and implicit feedback. Human-machine interactive learning enhanced transformer model can achieve higher levels of accuracy and precision than their non-interactive counterparts.

AB - Emotion recognition has been well researched in mono-modality in the past decade. However, people express their emotion or feelings naturally via more than one modalities like voice, facial expressions, text, and behaviors. In this paper, we propose a new method to model deep interactive learning and dual modalities (e.g., speech and text) to conduct multimodal emotion recognition. An unsupervised triplet-loss objective function is constructed to learn representation of emotional information from speech audio. We extract text emotional feature representation by transfer learning of text-To-Text embedding from T5 pre-Trained model. Human-machine interaction like user feedback plays a vital role in improve multimodal emotion recognition in dialogue system. Deep interactive learning model is constructed by explicit and implicit feedback. Human-machine interactive learning enhanced transformer model can achieve higher levels of accuracy and precision than their non-interactive counterparts.

KW - human-machine interaction

KW - interactive learning

KW - Multimodal emotion recognition

KW - transformer model

UR - https://www.scopus.com/pages/publications/85150369792

U2 - 10.1145/3579654.3579764

DO - 10.1145/3579654.3579764

M3 - Conference proceeding

AN - SCOPUS:85150369792

SN - 9781450398336

T3 - ACM International Conference Proceeding Series

BT - ACAI 2022 - Conference Proceedings

PB - Association for Computing Machinery (ACM)

CY - New York

T2 - 5th International Conference on Algorithms, Computing and Artificial Intelligence, ACAI 2022

Y2 - 23 December 2022 through 25 December 2022

ER -

Leung CHC, Deng JJ, Li Y. Enhanced Human-Machine Interactive Learning for Multimodal Emotion Recognition in Dialogue System. In ACAI 2022 - Conference Proceedings: 2022 5th International Conference on Algorithms, Computing and Artificial Intelligence. New York: Association for Computing Machinery (ACM). 2022. 104. (ACM International Conference Proceeding Series). doi: 10.1145/3579654.3579764

Enhanced Human-Machine Interactive Learning for Multimodal Emotion Recognition in Dialogue System

Abstract

Publication series

Conference

UN SDGs

User-Defined Keywords

Access to Document

Other files and links

Fingerprint

Cite this