Meta.Morf X – Digital Wild / Dokkhuset, March 6 – 7, 2020 / Curator: Zane Cerpina
Øyvind Brandtsegg and Axel Tidemann [no]
Shape: an adaptive musical instrument using feature extraction and machine learning to map gestural qualities to sonic transformations
We present the prototype of a new musical instrument, where the instrument learns to know the performer rather than the other way around. The method is interface-agnostic and can be adapted to various input devices (camera, sensors, keyboards, etc.). Performative gestures are automatically mapped to sonic output retaining some of the gestural qualities of the input. This is done via a process based on feature extraction from the gestural data and from the synthesized sound. A machine learning mapping process is used to make a mapping from gestural data to synthesis parameters. The goal of the artificial intelligence (AI) is to optimize the correlation between gestural and sonic qualities. Quality deviations of known gestures result in modulations of the sonic output. Significant deviation from known gestures establish new gestural repertoire, allowing the instrument to develop according to the performative traits of the player. Such new gestures are automatically trained to yield gesturally equivalent sonic outputs.
The work is based on previous productions exploring AI, e.g. the robotic entity [self.] (2014) https://youtu.be/HErOfnqREBQ, where the entity started with a blank memory, learning everything from interaction with a gallery audience. The current production attempts to explore a more nuanced and intimate interaction with the intentional expressive gestures of a performer. The presentation will include a working prototype as well as reflections and questions on how this mode of adaptive automation can be used as extensions to AI within other domains of application.
Øyvind Brandtsegg
Øyvind Brandtsegg is a composer and performer working in the fields of algorithmic improvisation and sound installations. He is also a professor of music technology at the NTNU.
Axel Tidemann
Axel Tidemann is a drummer, and a senior research scientist at Telenor Research, with an interest in using artificial intelligence to model human cognition and find structure in complex data.
