Synesthesizer: Physical Modelling and Machine Learning for a Color-Based Synthesizer in Virtual Reality

Giovanni Santini

doi:10.1007/978-3-030-21392-3_18

Synesthesizer: Physical Modelling and Machine Learning for a Color-Based Synthesizer in Virtual Reality

Giovanni Santini^*

^*Corresponding author for this work

Academy of Music

Research output: Chapter in book/report/conference proceeding › Conference proceeding › peer-review

7 Citations (Scopus)

Abstract

The Synesthesizer is a software synthesizer inspired to chromesthesia, that kind of synesthesia that connects sounds and colors. While chromesthesia usually produces color perception in response to sound stimulation, this synthesizer does the opposite: sound is generated according to color detection. More precisely, RGB (Red Green Blue) values are detected (one pixel at a time) and used to determine the behaviour of five physical models for virtual instruments. The motivation for creating such a synthesizer arose from the will to generate a timbral continuum out of the color continuum, allowing to explore the relation between color spectrum and sound spectrum. The Synesthesizer has two additional possible applications:

A picture can become a sort of score; graphic scores can have a different source of interpretation;

Given its intuitiveness, it might allow even non-experts to explore the possibilities of sound synthesis.

The current version has been developed in a Virtual Reality (VR) environment.

Original language	English
Title of host publication	Mathematics and Computation in Music
Subtitle of host publication	7th International Conference, MCM 2019, Madrid, Spain, June 18–21, 2019, Proceedings
Editors	Mariana Montiel, Francisco Gomez-Martin, Octavio A. Agustín-Aquino
Publisher	Springer Cham
Pages	229-235
Number of pages	7
Edition	1st
ISBN (Electronic)	9783030213923
ISBN (Print)	9783030213916
DOIs	https://doi.org/10.1007/978-3-030-21392-3_18
Publication status	Published - 31 May 2019
Event	7th International Conference on Mathematics and Computation in Music, MCM 2019 - Madrid, Spain Duration: 18 Jun 2019 → 21 Jun 2019

Publication series

Name	Lecture Notes in Computer Science
Volume	11502
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Name	Lecture Notes in Artificial Intelligence
ISSN (Print)	2945-9133
ISSN (Electronic)	2945-9141

Name	MCM: International Conference on Mathematics and Computation in Music

Conference

Conference	7th International Conference on Mathematics and Computation in Music, MCM 2019
Country/Territory	Spain
City	Madrid
Period	18/06/19 → 21/06/19

User-Defined Keywords

Machine Learning
Physical modeling
Synesthesia
Synthesizer
Virtual reality

UN SDGs

This output contributes to the following UN Sustainable Development Goals (SDGs)

Access to Document

10.1007/978-3-030-21392-3_18

Cite this

Santini, G. (2019). Synesthesizer: Physical Modelling and Machine Learning for a Color-Based Synthesizer in Virtual Reality. In M. Montiel, F. Gomez-Martin, & O. A. Agustín-Aquino (Eds.), Mathematics and Computation in Music: 7th International Conference, MCM 2019, Madrid, Spain, June 18–21, 2019, Proceedings (1st ed., pp. 229-235). (Lecture Notes in Computer Science; Vol. 11502), (Lecture Notes in Artificial Intelligence), (MCM: International Conference on Mathematics and Computation in Music). Springer Cham. https://doi.org/10.1007/978-3-030-21392-3_18

Santini, Giovanni. / Synesthesizer: Physical Modelling and Machine Learning for a Color-Based Synthesizer in Virtual Reality. Mathematics and Computation in Music: 7th International Conference, MCM 2019, Madrid, Spain, June 18–21, 2019, Proceedings. editor / Mariana Montiel ; Francisco Gomez-Martin ; Octavio A. Agustín-Aquino. 1st. ed. Springer Cham, 2019. pp. 229-235 (Lecture Notes in Computer Science). (Lecture Notes in Artificial Intelligence). (MCM: International Conference on Mathematics and Computation in Music).

@inproceedings{e70ca7628434488a99fcb2d9bd87c64d,

title = "Synesthesizer: Physical Modelling and Machine Learning for a Color-Based Synthesizer in Virtual Reality",

abstract = "The Synesthesizer is a software synthesizer inspired to chromesthesia, that kind of synesthesia that connects sounds and colors. While chromesthesia usually produces color perception in response to sound stimulation, this synthesizer does the opposite: sound is generated according to color detection. More precisely, RGB (Red Green Blue) values are detected (one pixel at a time) and used to determine the behaviour of five physical models for virtual instruments. The motivation for creating such a synthesizer arose from the will to generate a timbral continuum out of the color continuum, allowing to explore the relation between color spectrum and sound spectrum. The Synesthesizer has two additional possible applications: A picture can become a sort of score; graphic scores can have a different source of interpretation;Given its intuitiveness, it might allow even non-experts to explore the possibilities of sound synthesis. The current version has been developed in a Virtual Reality (VR) environment.",

keywords = "Machine Learning, Physical modeling, Synesthesia, Synthesizer, Virtual reality",

author = "Giovanni Santini",

note = "Publisher Copyright: {\textcopyright} 2019 Springer Nature Switzerland AG; 7th International Conference on Mathematics and Computation in Music, MCM 2019 ; Conference date: 18-06-2019 Through 21-06-2019",

year = "2019",

month = may,

day = "31",

doi = "10.1007/978-3-030-21392-3_18",

language = "English",

isbn = "9783030213916",

series = "Lecture Notes in Computer Science",

publisher = "Springer Cham",

pages = "229--235",

editor = "Mariana Montiel and Francisco Gomez-Martin and Agust{\'i}n-Aquino, {Octavio A.}",

booktitle = "Mathematics and Computation in Music",

edition = "1st",

}

Santini, G 2019, Synesthesizer: Physical Modelling and Machine Learning for a Color-Based Synthesizer in Virtual Reality. in M Montiel, F Gomez-Martin & OA Agustín-Aquino (eds), Mathematics and Computation in Music: 7th International Conference, MCM 2019, Madrid, Spain, June 18–21, 2019, Proceedings. 1st edn, Lecture Notes in Computer Science, vol. 11502, Lecture Notes in Artificial Intelligence, MCM: International Conference on Mathematics and Computation in Music, Springer Cham, pp. 229-235, 7th International Conference on Mathematics and Computation in Music, MCM 2019, Madrid, Spain, 18/06/19. https://doi.org/10.1007/978-3-030-21392-3_18

Synesthesizer: Physical Modelling and Machine Learning for a Color-Based Synthesizer in Virtual Reality. / Santini, Giovanni.
Mathematics and Computation in Music: 7th International Conference, MCM 2019, Madrid, Spain, June 18–21, 2019, Proceedings. ed. / Mariana Montiel; Francisco Gomez-Martin; Octavio A. Agustín-Aquino. 1st. ed. Springer Cham, 2019. p. 229-235 (Lecture Notes in Computer Science; Vol. 11502), (Lecture Notes in Artificial Intelligence), (MCM: International Conference on Mathematics and Computation in Music).

Research output: Chapter in book/report/conference proceeding › Conference proceeding › peer-review

TY - GEN

T1 - Synesthesizer: Physical Modelling and Machine Learning for a Color-Based Synthesizer in Virtual Reality

AU - Santini, Giovanni

PY - 2019/5/31

Y1 - 2019/5/31

N2 - The Synesthesizer is a software synthesizer inspired to chromesthesia, that kind of synesthesia that connects sounds and colors. While chromesthesia usually produces color perception in response to sound stimulation, this synthesizer does the opposite: sound is generated according to color detection. More precisely, RGB (Red Green Blue) values are detected (one pixel at a time) and used to determine the behaviour of five physical models for virtual instruments. The motivation for creating such a synthesizer arose from the will to generate a timbral continuum out of the color continuum, allowing to explore the relation between color spectrum and sound spectrum. The Synesthesizer has two additional possible applications: A picture can become a sort of score; graphic scores can have a different source of interpretation;Given its intuitiveness, it might allow even non-experts to explore the possibilities of sound synthesis. The current version has been developed in a Virtual Reality (VR) environment.

AB - The Synesthesizer is a software synthesizer inspired to chromesthesia, that kind of synesthesia that connects sounds and colors. While chromesthesia usually produces color perception in response to sound stimulation, this synthesizer does the opposite: sound is generated according to color detection. More precisely, RGB (Red Green Blue) values are detected (one pixel at a time) and used to determine the behaviour of five physical models for virtual instruments. The motivation for creating such a synthesizer arose from the will to generate a timbral continuum out of the color continuum, allowing to explore the relation between color spectrum and sound spectrum. The Synesthesizer has two additional possible applications: A picture can become a sort of score; graphic scores can have a different source of interpretation;Given its intuitiveness, it might allow even non-experts to explore the possibilities of sound synthesis. The current version has been developed in a Virtual Reality (VR) environment.

KW - Machine Learning

KW - Physical modeling

KW - Synesthesia

KW - Synthesizer

KW - Virtual reality

UR - http://www.scopus.com/inward/record.url?scp=85067865172&partnerID=8YFLogxK

U2 - 10.1007/978-3-030-21392-3_18

DO - 10.1007/978-3-030-21392-3_18

M3 - Conference proceeding

AN - SCOPUS:85067865172

SN - 9783030213916

T3 - Lecture Notes in Computer Science

SP - 229

EP - 235

BT - Mathematics and Computation in Music

A2 - Montiel, Mariana

A2 - Gomez-Martin, Francisco

A2 - Agustín-Aquino, Octavio A.

PB - Springer Cham

T2 - 7th International Conference on Mathematics and Computation in Music, MCM 2019

Y2 - 18 June 2019 through 21 June 2019

ER -

Santini G. Synesthesizer: Physical Modelling and Machine Learning for a Color-Based Synthesizer in Virtual Reality. In Montiel M, Gomez-Martin F, Agustín-Aquino OA, editors, Mathematics and Computation in Music: 7th International Conference, MCM 2019, Madrid, Spain, June 18–21, 2019, Proceedings. 1st ed. Springer Cham. 2019. p. 229-235. (Lecture Notes in Computer Science). (Lecture Notes in Artificial Intelligence). (MCM: International Conference on Mathematics and Computation in Music). doi: 10.1007/978-3-030-21392-3_18

Synesthesizer: Physical Modelling and Machine Learning for a Color-Based Synthesizer in Virtual Reality

Abstract

Publication series

Conference

User-Defined Keywords

UN SDGs

Access to Document

Other files and links

Fingerprint

Cite this