Creating and Collecting Meaningful Musical Materials with Machine Learning

This dissertation explores how machine learning and artificial intelligence can be applied within music composition and production. My approach in this research stems from an underlying perspective that these technologies are deeply intertwined with the people who use them or are affected by them: we can’t hope to understand one side of the picture without looking at the other. From this vantage point, I explore the following questions: How do we design algorithms, datasets, and models to support the processes of composers, producers, and other creators? How can we design meaningful interactions with these algorithms? And finally, how do music creators and listeners experience interacting with algorithms for creating music in situations when they have reasons to be emotionally invested in the music?

First, I explore new approaches to music creation technology with machine learning, focusing on two musical settings: beat-making and orchestration. I find that creative tools can benefit from incorporating machine learning if we introduce models in specific contexts motivated by well-defined musical goals. For beat-making, I use machine learning to expand the possibilities for groove in drum machines by modeling the timing and dynamics of professional drummers, and for orchestration, I use machine learning to predict the timbral characteristics that result when mixing many different instruments together. In both cases, I find that careful data collection and management are key components.

Next, I investigate some of the main technical choices that need to be made when using machine learning in creative musical contexts: data representations and controls for guiding models. Here, I find that allowing users to provide more than one demonstration at a time can allow for more diverse and more precisely controlled model outputs. I also find that using data representations designed to capture musical gestures can provide benefits in settings with limited musical data.

Finally, in the last part of the dissertation, I conduct a series of qualitative studies to investigate the individual experiences of listeners and musicians over the course of interactions with algorithms that generate personalized music samples using machine learning. I find the musical “quality” of algorithmically generated music to be comparatively unimportant to participants; instead, the degree to which music fits with the participants’ existing narratives or creative intentions matters more to them. These findings highlight the need to understand the contexts in which algorithms are deployed as well as the artistic choices that listeners may or may not feel comfortable turning over to an automated process when working with emotionally sensitive materials.