In a Silent Way: Communication Between AI and Improvising Musicians Beyond Sound

Jon McCormack; Toby Gifford; Patrick Hutchings; Maria Teresa Llano Rodriguez; Matthew Yee-King; Mark D’Inverno

doi:10.1145/3290605.3300268

In a Silent Way: Communication Between AI and Improvising Musicians Beyond Sound

Jon McCormack, Toby Gifford, Patrick Hutchings, Maria Teresa Llano Rodriguez, Matthew Yee-King, Mark D’Inverno

Academy of Music

Research output: Chapter in book/report/conference proceeding › Conference proceeding › peer-review

33 Citations (Scopus)

Abstract

Collaboration is built on trust, and establishing trust with a creative Artificial Intelligence is difficult when the decision process or internal state driving its behaviour isn't exposed. When human musicians improvise together, a number of extra-musical cues are used to augment musical communication and expose mental or emotional states which affect musical decisions and the effectiveness of the collaboration. We developed a collaborative improvising AI drummer that communicates its confidence through an emoticon-based visualisation. The AI was trained on musical performance data, as well as real-time skin conductance, of musicians improvising with professional drummers, exposing both musical and extra-musical cues to inform its generative process. Uni- and bi-directional extra-musical communication with real and false values were tested by experienced improvising musicians. Each condition was evaluated using the FSS-2 questionnaire, as a proxy for musical engagement. The results show a positive correlation between extra-musical communication of machine internal state and human musical engagement.

Original language	English
Title of host publication	CHI 2019 - Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems
Place of Publication	New York
Publisher	Association for Computing Machinery (ACM)
Pages	1-11
Number of pages	11
ISBN (Electronic)	9781450359702
DOIs	https://doi.org/10.1145/3290605.3300268
Publication status	Published - 2 May 2019
Event	2019 CHI Conference on Human Factors in Computing Systems, CHI 2019 - Glasgow, United Kingdom Duration: 4 May 2019 → 9 May 2019 https://chi2019.acm.org/ (Conference Website) https://dl.acm.org/doi/proceedings/10.1145/3290605 (Conference Proceedings)

Publication series

Name	CHI - Proceedings of the CHI Conference on Human Factors in Computing Systems

Conference

Conference	2019 CHI Conference on Human Factors in Computing Systems, CHI 2019
Country/Territory	United Kingdom
City	Glasgow
Period	4/05/19 → 9/05/19
Internet address	https://chi2019.acm.org/ (Conference Website) https://dl.acm.org/doi/proceedings/10.1145/3290605 (Conference Proceedings)

User-Defined Keywords

AI systems
Improvisation
Extra-musical communication

Access to Document

10.1145/3290605.3300268

Cite this

McCormack, J., Gifford, T., Hutchings, P., Teresa Llano Rodriguez, M., Yee-King, M., & D’Inverno, M. (2019). In a Silent Way: Communication Between AI and Improvising Musicians Beyond Sound. In CHI 2019 - Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (pp. 1-11). Article 38 (CHI - Proceedings of the CHI Conference on Human Factors in Computing Systems). Association for Computing Machinery (ACM). https://doi.org/10.1145/3290605.3300268

McCormack, Jon ; Gifford, Toby ; Hutchings, Patrick et al. / In a Silent Way : Communication Between AI and Improvising Musicians Beyond Sound. CHI 2019 - Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. New York : Association for Computing Machinery (ACM), 2019. pp. 1-11 (CHI - Proceedings of the CHI Conference on Human Factors in Computing Systems).

@inproceedings{2859ede705da4f399f39173ff5263ed7,

title = "In a Silent Way: Communication Between AI and Improvising Musicians Beyond Sound",

abstract = "Collaboration is built on trust, and establishing trust with a creative Artificial Intelligence is difficult when the decision process or internal state driving its behaviour isn't exposed. When human musicians improvise together, a number of extra-musical cues are used to augment musical communication and expose mental or emotional states which affect musical decisions and the effectiveness of the collaboration. We developed a collaborative improvising AI drummer that communicates its confidence through an emoticon-based visualisation. The AI was trained on musical performance data, as well as real-time skin conductance, of musicians improvising with professional drummers, exposing both musical and extra-musical cues to inform its generative process. Uni- and bi-directional extra-musical communication with real and false values were tested by experienced improvising musicians. Each condition was evaluated using the FSS-2 questionnaire, as a proxy for musical engagement. The results show a positive correlation between extra-musical communication of machine internal state and human musical engagement.",

keywords = "AI systems, Improvisation, Extra-musical communication",

author = "Jon McCormack and Toby Gifford and Patrick Hutchings and {Teresa Llano Rodriguez}, Maria and Matthew Yee-King and Mark D{\textquoteright}Inverno",

note = "Publisher Copyright: {\textcopyright} 2019 Copyright held by the owner/author(s).; 2019 CHI Conference on Human Factors in Computing Systems, CHI 2019 ; Conference date: 04-05-2019 Through 09-05-2019",

year = "2019",

month = may,

day = "2",

doi = "10.1145/3290605.3300268",

language = "English",

series = "CHI - Proceedings of the CHI Conference on Human Factors in Computing Systems",

publisher = "Association for Computing Machinery (ACM)",

pages = "1--11",

booktitle = "CHI 2019 - Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems",

address = "United States",

url = "https://chi2019.acm.org/, https://dl.acm.org/doi/proceedings/10.1145/3290605",

}

McCormack, J, Gifford, T, Hutchings, P, Teresa Llano Rodriguez, M, Yee-King, M & D’Inverno, M 2019, In a Silent Way: Communication Between AI and Improvising Musicians Beyond Sound. in CHI 2019 - Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems., 38, CHI - Proceedings of the CHI Conference on Human Factors in Computing Systems, Association for Computing Machinery (ACM), New York, pp. 1-11, 2019 CHI Conference on Human Factors in Computing Systems, CHI 2019, Glasgow, United Kingdom, 4/05/19. https://doi.org/10.1145/3290605.3300268

In a Silent Way: Communication Between AI and Improvising Musicians Beyond Sound. / McCormack, Jon; Gifford, Toby; Hutchings, Patrick et al.
CHI 2019 - Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. New York: Association for Computing Machinery (ACM), 2019. p. 1-11 38 (CHI - Proceedings of the CHI Conference on Human Factors in Computing Systems).

Research output: Chapter in book/report/conference proceeding › Conference proceeding › peer-review

TY - GEN

T1 - In a Silent Way

T2 - 2019 CHI Conference on Human Factors in Computing Systems, CHI 2019

AU - McCormack, Jon

AU - Gifford, Toby

AU - Hutchings, Patrick

AU - Teresa Llano Rodriguez, Maria

AU - Yee-King, Matthew

AU - D’Inverno, Mark

PY - 2019/5/2

Y1 - 2019/5/2

N2 - Collaboration is built on trust, and establishing trust with a creative Artificial Intelligence is difficult when the decision process or internal state driving its behaviour isn't exposed. When human musicians improvise together, a number of extra-musical cues are used to augment musical communication and expose mental or emotional states which affect musical decisions and the effectiveness of the collaboration. We developed a collaborative improvising AI drummer that communicates its confidence through an emoticon-based visualisation. The AI was trained on musical performance data, as well as real-time skin conductance, of musicians improvising with professional drummers, exposing both musical and extra-musical cues to inform its generative process. Uni- and bi-directional extra-musical communication with real and false values were tested by experienced improvising musicians. Each condition was evaluated using the FSS-2 questionnaire, as a proxy for musical engagement. The results show a positive correlation between extra-musical communication of machine internal state and human musical engagement.

AB - Collaboration is built on trust, and establishing trust with a creative Artificial Intelligence is difficult when the decision process or internal state driving its behaviour isn't exposed. When human musicians improvise together, a number of extra-musical cues are used to augment musical communication and expose mental or emotional states which affect musical decisions and the effectiveness of the collaboration. We developed a collaborative improvising AI drummer that communicates its confidence through an emoticon-based visualisation. The AI was trained on musical performance data, as well as real-time skin conductance, of musicians improvising with professional drummers, exposing both musical and extra-musical cues to inform its generative process. Uni- and bi-directional extra-musical communication with real and false values were tested by experienced improvising musicians. Each condition was evaluated using the FSS-2 questionnaire, as a proxy for musical engagement. The results show a positive correlation between extra-musical communication of machine internal state and human musical engagement.

KW - AI systems

KW - Improvisation

KW - Extra-musical communication

UR - http://www.scopus.com/inward/record.url?scp=85067600532&partnerID=8YFLogxK

U2 - 10.1145/3290605.3300268

DO - 10.1145/3290605.3300268

M3 - Conference proceeding

AN - SCOPUS:85067600532

T3 - CHI - Proceedings of the CHI Conference on Human Factors in Computing Systems

SP - 1

EP - 11

BT - CHI 2019 - Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems

PB - Association for Computing Machinery (ACM)

CY - New York

Y2 - 4 May 2019 through 9 May 2019

ER -

McCormack J, Gifford T, Hutchings P, Teresa Llano Rodriguez M, Yee-King M, D’Inverno M. In a Silent Way: Communication Between AI and Improvising Musicians Beyond Sound. In CHI 2019 - Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. New York: Association for Computing Machinery (ACM). 2019. p. 1-11. 38. (CHI - Proceedings of the CHI Conference on Human Factors in Computing Systems). doi: 10.1145/3290605.3300268

In a Silent Way: Communication Between AI and Improvising Musicians Beyond Sound

Abstract

Publication series

Conference

User-Defined Keywords

Access to Document

Other files and links

Fingerprint

Cite this