In the music landscape, there have been many advancements in live performance that allow for just a single artist to control all components of musical production. Through gesture control technology, there have been different models created known as computer systems for expressive music performance (CSEMP). Let’s explore these systems, and how it answers the question of if humans and AI can collaborate creatively in musical spaces.
Below, is an example of a generic model for most CSEMP. Let’s evaluate some of these components, and how a human and AI are collaborating in this specific instance. The first stage involves having examples of live performances, as well as context for the performance, and music for the AI to analyze. The performance example and music analysis are then combined for the adaption process. Then, the music analysis and performance context are combined for the performance knowledge. While this is all being generated, it almost works like a loop. The sound is used for the adaption process, which is then used for the performance knowledge, which then computes the instrument model that allows the sound to occur. Basically, all of these components are essential in most CSEMP models, even if they don’t all rely on each other.

An example of a generic model for most current CSEMP in the buisness
Now, let's evaluate a video of these performances in action, and how the gesture recognition technology actually works. Corpus Nil is one of the leading examples of humans and AI collaborating in musical spaces. The system was created by Baptiste Caramiaux and Marco Donnarumma, where bioelectrical and bioacoustic sensors are attached to a human performer. The body motions of the performer directly impact the audio that the signal generates, as it aims to examine what a body “defiled by algorithms” might look and move like.
Through the company's website, it showcases a video of Donnarumma using the technology in a performance from 2016. He used eight loudspeakers and a subwoofer to really enhance the audio generated from the bioelectrical and bioacoustic sensors. Although the music performed in this instance is definitely specific in taste, there is a lot to be encouraged from this. As similar technology becomes advanced, there is the potential for rappers to use this in live performances. Here’s a hypothetical scenario, imagine a rapper waving their hands in the air as they are performing a song. If they had similar technology and sensors connected to their body, it could potentially help them create the beats as they are performing. They will become more immersed in the song, and it has the potential to make for a better product for the paying consumers.

Marco Donnarumma in 2015
Diving deeper into the topic, let’s take a closer look at machine learning systems that are being used creatively in the music industry. The four components of this Assisted IML workflow system include sound design, agent exploration, play, and human feedback. Most of the processes in this workflow rely on each other, but unlike the generic model that we evaluated earlier, it does not work like a loop. It is very technical and requires more in-depth knowledge, and that why machine learning is crucial for making this system useable.
One aspect that is used in this system which has been discussed in the RTA 950 course, is artificial neural networks. They are inspired by real networks, such as a brain for example, but they are not a simulation of them. Understanding that is essential when watching these performances, as the neural networks are used for mapping the sound that comes through the sensors. As discussed in the course, there is an input, some magic is used to create the data, and then it outputs it to the next source. There can only be a neural network used in this model, as there is training data created by the sound design. The network is trained to give the correct answer for training data, and that allows for the rest of the process to be mapped out.

An example of an Assisted IML workflow system
Overall, there are different ways for humans and AI to collaborate creatively in musical spaces, as technology is advancing. Let’s take a look at video games as an example, and how they were able to allow the user to enjoy a more creative experience. In the NHL video games created by EA Sports, they originally started off as 2D animation where you had limited control over the AI. Flash forward over twenty years, and there is much more creative control in how the AI moves and what you can do in the game. Although there are many differences between music and video game technology, the methodology is still the same. It will only continue to improve as researchers continues to further study this topic.
References
1. Corpus nil. Marco Donnarumma. (2022, August 27). Retrieved October 24, 2022, from https://marcodonnarumma.com/works/corpus-nil/
Comments