Classical Music & AI
Artificial intelligence (AI) is shaping up to be the next great technological frontier, prompting excitement and trepidation about its applications in all fields, from academia to medicine to the military. Music is no exception.
This past month, the big story was how producers used AI to create The Beatles’ “last” song, “Now and Then,” released on November 2. In the late 1970s, John Lennon had recorded a few demo tapes, which his widow, Yoko Ono, bequeathed to his three surviving bandmates in the mid-1990s. They remixed John’s old vocals and recorded some new music to create two “new” songs, “Free as a Bird” and “Real Love.” However, the audio of the third song, “Now and Then,” was so poor that they ultimately couldn’t do anything with it. John’s voice was too muffled by the piano and background noise, and there was no way to isolate and enhance these elements.
Cut to 25 years later when director Peter Jackson was working on his monumental docuseries for Disney Plus, The Beatles: Get Back. Jackson’s team developed a technology that uses machine learning to split all the components of an analog recording into separate tracks to make them editable. (Machine learning is a branch of AI that uses data and algorithms to imitate how humans learn, gradually improving over time with more experience.) This technology allowed them to take old footage of the band in the studio that was previously unintelligible and enhance just the Fab Four’s voices so we could hear their conversations.
This breakthrough gave the remaining Beatles an idea: they could use this technology to isolate John’s vocals and piano from the lo-fi demo tape, clean them up, and rebalance them. Paul and Ringo finished the song by recording new drum and bass parts, and producer Giles Martin added a sweeping string arrangement. Paired with guitar parts George had recorded in 1995, the resulting track is a true Beatles song with all four members represented.
So, how can AI be applied to classical music, a field generally tethered to tradition and characterized by raw human ability? AI has already created a furor among visual artists whose work has been used without permission to train AI systems and create art for free—even sometimes winning awards over human artists. AI music still lags behind AI visual art, but that doesn’t mean musicians aren’t anxious about what this technology portends.
Replacement anxiety is nothing new in the music world. John Philip Sousa, the March King, said this in reaction to the invention of the Edison phonograph: “The time is coming when no one will be ready to submit himself to the ennobling discipline of learning music. Everyone will have their ready-made or ready-pirated music in their cupboards.” While the phonograph did not kill music tuition entirely, sales of sheet music did drop now that one no longer needed to play the music oneself to enjoy it at home. Then in the 1990s with the advent of midi technology, session musicians feared they would be replaced by computers, yet that has not entirely happened either.
Instead, technological developments give us new tools that help us do our jobs better and open up new worlds of possibility. With these developments come new sounds and even new notions of what music is. This is nothing new. As classical music columnist Anne Midgette writes, in the late 18th century, “the harpsichord was beginning to yield to the pianoforte; and the makers of keyboard instruments were exploring all kinds of avenues--from adding bells and cymbals, literally, with the popular ‘Janissary stops,’ to simply improving and developing the basic components of their instruments—as they tried to come up with the definitive piano of the future.” That said, most of the instruments that make up the modern orchestra have changed little since the late nineteenth century, and our notions of what classical music is often seem stuck in this time period.
So, what can classical musicians take away from advancements in AI, and will implementing these tools compromise the inherent humanity of music composition and performance? To try and answer these questions, let’s take a look at some of the ways AI has been implemented in classical music so far.
DeepBach
In 2016, a team of computer scientists developed a “steerable model” for generating Bach chorales. The model was fed 2,503 pieces by the famed baroque composer, off of which it learned to produce its own chorales following the same principles of harmony, counterpoint, and style.
Check out this video from the Orchestra of the Age of Enlightenment to see if you can tell which chorale was computer-generated and which came from the mind of the master. Before revealing the answer, mathematics professor Marcus du Sautoy offers his two cents on the issue. Instead of thinking of AI as a threat to humanity, Du Sautoy sees it as a useful collaborator that could help us get out of our creative ruts and see new possibilities.
Schubert’s "Finished" Unfinished Symphony
In 2019, to demonstrate the capabilities of its smartphones, Chinese company Huawei set out to finish one of the most famously incomplete works in the classical canon, Franz Schubert’s “Unfinished” Symphony No. 8. The question of why Schubert never finished the symphony has eaten at scholars ever since its first performance 37 years after the composer’s death. Why would Schubert abandon a work that so obviously outshone any of his previous symphonic attempts? One theory is that he felt he could not sustain the quality and innovativeness of the first two movements. Or perhaps he got distracted by other projects. Illness might have also been a factor, as it was around this time that he became infected with syphilis, the disease that would ultimately claim him at the age of 31.
To envision what it might have sounded like had Schubert lived long enough to finish Symphony No. 8, Huawei engineers fed the smartphone’s dual neural processing unit volumes of data on Schubert’s musical style so it could identify patterns and try to recreate it. The phone created melodies from this information, from which (human) composer Lucas Cantor chose his favorites. He orchestrated and further finessed the AI-generated musical content to create two final movements to complete the symphony. Cantor said that working with AI was “like having a collaborating partner who never gets tired, never runs out of ideas.”
While the resulting music does undoubtedly sound like Schubert, even mirroring the opening structure of the first movement with melodies that sound like the original's inverses, there are still elements that do not sound quite right or natural (much like the featured image at the top of this article). Watch the guys from TwoSet Violin react to the final product and philosophize on what this means for classical music and humanity at large in this video.
Beethoven Symphony No...10??
When Ludwig van Beethoven died in 1827, he left behind sketches of what would have been his tenth symphony. To celebrate the composer’s 250th birthday in 2020, a group of musicologists, composers, and computer scientists released a version of “Beethoven’s” Symphony No. 10 created by AI. Developers fed the AI Beethoven’s complete oeuvre and notes on those works, plus sketches of the Tenth Symphony, which a computational music expert had transcribed. A human composer then blended the AI-generated musical material with Beethoven's sketches to create the final product. The two-year effort culminated in a premiere performance in Bonn, Germany, Beethoven’s birthplace. Even in light of this achievement, Ahmed Elgammal, the Rutgers University professor in charge of the computer science aspect of this project, conceded that algorithms will never replace the human creative process. Instead, he believes AI will be a tool that facilitates artistic expression.
These are just three examples of how AI has been used to “copy” or build upon the outputs of famous composers of the past. A common theme that has emerged from these projects is that human creativity and expertise are still needed to make something worth listening to. Though AI will, and already has, improved with startling speed, we want to know what other possible applications of AI there might be in classical music.
One person who has thought a lot about this is composer, inventor, and teacher Tod Machover, who runs the Opera of the Future group at the Massachusetts Institute of Technology’s Media Lab. There, he helps students test their ideas to advance the future of musical performance and composition. Machover believes AI is a useful tool for expanding access to musical expression. Opera of the Future grew out of Machover’s “hyperinstruments” project begun in the mid-1980s. These hyperinstruments use sensors, signal processing, and software to give virtuosic performers extra power and finesse. In 1992, Machover refocused these efforts to expand access to composition and performance to nonprofessional musicians, students, music lovers, children, and the public. For instance, he developed tools where users can shape musical pieces through gestures or word descriptions in real time.
This past March, Machover premiered a 30-minute chamber opera based on Richard Powers’s Pulitzer Prize-winning novel, The Overstory. In Overstory Overture, Machover used AI to create the secret musical language of trees, feeding a detuned cello and various electronic sounds into the AI system. Machover then blended these electronically produced sounds with acoustic sonorities of the string orchestra, marimba, bass drum, and mezzo-soprano soloist Joyce DiDonato, who sings the role of the novel’s protagonist, dendrologist Patricia Westerford.
These are just a few examples of AI complementing rather than replacing the human creative process. As the technology continues to develop at breakneck speed, more applications we can’t even conceive of today will undoubtedly surface. Whether we allow them to infiltrate our sometimes ossified notion of classical music is up to us. But we should not be so afraid that robots will totally replace us in creative endeavors. As music theory YouTuber Adam Neely puts it, “we kept playing chess, even though computers reliably beat us every time.” So too will we continue to compose and perform music—because it’s an innately human act.
