In “The surprising thing about musical surprise,” philosopher and musician Jenny Judge refers to the experience of listening to the first thirty seconds of Led Zeppelin’s1‘Black Dog’: “whatever else you might say about the experience, an adequate discussion of its phenomenal character would have to mention musical surprise.”2
The surprise in ‘Black Dog’ is rhythmic; the guitar riff disrupts the regular sense of meter3 established by the vocal line. That this surprise was deliberately designed becomes clear when considering the effort that went into making ‘Black Dog’, its relation4 to Fleetwood Mac’s5‘Oh Well’, and something not mentioned in these two articles. When you turn the volume up6, you can hear John Bonham—probably with his drumsticks—giving metric indications at 0:11 (two clicks), 0:23 (one click), 0:30 (one click), and 0:34 (one click). Was the musical surprise so essential that it had to be executed perfectly to happen, with the risk of being “spoiled” by revealing the mechanics or trick? J. Brackett7 hints at the answer by quoting the musicians during a rehearsal of the song ‘In My Time of Dying’:
Bonham: “Well you can’t count from where you stop ’cause your vocals might be a different – I mean – your voice just might go off a beat and we’re gonna be f…”
Plant: “Ah, but if you do that, it’ll be like ‘Black Dog’ then it gives me room to move and solo…”
Bonham: “Yes, but the reason we did ‘Black Dog’ is because we counted it and you did it afterwards … That’s the only way we can do this.”
The vocals were overdubbed8 later to give the impression of spontaneity and freedom. On live versions, one can hear how greater metrical regularity is established, which allows the band to perform in sync but restricts its freedom. Musical surprises come in many forms and can involve rhythm, harmony, melody, timbre, dynamics, color, and so on.
There is a debate about the nature9 of these musical surprises, which fall under the category of perceptual surprises and are characterized as frustrating because they thwart prior expectations.
If musical surprises are such an interesting element for composers, performers, and the audience, it is worth investigating whether they can be generated and repeated. Is there a formula or method, a process to make musical surprises arise? And what happens to the surprise if you listen to a piece over and over again? I turned to a field of music I had some prior experience in: music and audio for video games. I’ll build on previous personal projects that all use the same or related audio technologies but each focus on one specific topic. Instead of rehashing the techniques, tools, and workflows I already had experience with, I wanted to use the potential of this research to further explore the specific chance and random elements that are central to adaptive audio but are also making their way into music creation apps, generative music software, and devices. These random features intrigued me the most and are also the ones that were only present as side functions or sketches in my other projects.
How can randomness be made to work in the creator’s favor and capture the audience’s interest?
Serendip Revisited
The word serendipity only exists in the English language. It was coined by Horace Walpole in 1754 in a letter referring to an episode from the Persian tale “The Three Princes of Serendip”. The three princes from today’s Sri Lanka traveled the world “making discoveries, by accidents and sagacity, of things they were not in quest of…”.
The Merriam-Webster dictionary defines serendipity as the “faculty or phenomenon of finding valuable or agreeable things not sought for”, the Oxford Learner’s Dictionary as “the fact of something interesting or pleasant happening by chance”, and the Oxford English Dictionary as “the faculty of making happy and unexpected discoveries by accident”.
Today, serendipity is overused as a buzzword in the humanities and natural sciences. In 2009, journalist Richard Boyle10 lamented that the commonly accepted definitions did not do justice to Walpole’s “more complex and metaphorical original meaning”. This holds especially true if one considers its use in natural sciences.
It is in the Oxford Reference, not mentioned by Boyle, that one can find a definition that is closer to the original meaning envisioned by Walpole:
“Discovery of new information by a happy accident when actually seeking something altogether different. A famous example was the discovery by the British medical scientist Alexander Fleming (1881–1955) of the antibiotic property of penicillin mold.”
For neuroscientist David R. Colman:
“Serendipity plays an important part in research of all kinds, but it operates only in a special environment; as Pasteur famously stated, ‘Chance favors the prepared mind.’ In research, what serendipity really means in practical terms is that scientists discover things in the course of their investigations that they were not looking for. And these new findings are often not the products of cold logic.”11
I’ll make use of the above definition in my research on randomness and chance to create surprise, expectation, resolution, and alternatives, but the primary focus will be on the more precise definition of serendipity with the idea of adding and mixing elements from a composition that will appear on an album and in an app.
Artistic Research Focused on Creating a Changing Music Video Clip
Artistic research12 projects will display most of the following features:
“The art work is the focal point. The art work tops the list of the priorities, from places 1 to 22, and still continuing”.
“Artistic experientiality [_sic] is the very core of the research._”
“Artistic research must be self-reflective, self-critical and an outwardly-directed communication.”
“The placement of artistic research in the historical and disciplinary context.”
“A diversity of research methods, presentation methods and communication tools and their commitment to the needs and demands of each particular case.”
“Emphasizing the fruitfulness and necessity of the dynamic research group situation, which in a collective effort provides the closest critical environment, the protective realm for experimentation and the ability to share thoughts and emotions.”
I’ll consider points 1, 2, and 5 as most relevant to my undertaking.
One of the main compositions used for the app will serve as the base for my research on creating an interactive video clip that offers musical surprises but keeps the composer’s intentions intact. Can serendipity happen for those who are prepared and put in some effort, planting the seeds?
While adaptive audio in itself, as used in multimedia projects, mostly video games, is an interesting subject, I’d like to focus on one particular aspect: randomness as a surprise for the composer, performer, and listener. Randomness as a feature has become ubiquitous in music software and hardware (cf. methodology). “Which tools do you use to make music?” might not be one of the most interesting questions, albeit one of the most asked by musicians in popular music circles, but tools do have an impact on music making, on the creative process, and can expand or restrict the horizons of the possible and desirable.13 The reasons for using a well-established technology versus more advanced applications, from algorithmic and AI music research or even well-documented tools such as Max, Pure Data, or SuperCollider (cf. glossary), will also be explained in the methodology chapter. The main reason is that I was looking into technologies that would make it possible to create and publish an end product (video clip with changing music and restricted moments of interaction) in different media accessible to the largest possible audience: apps (iOS, Android), computer platforms (Mac, Linux, PC), web browsers, and even game consoles (Xbox, Switch, etc.).
Update: the end product is out. After this thesis was submitted, the project reached its audience. The free The Aquatic Museum app has been available on the App Store since April 2023. The album The Aquatic Museum followed at the end of May 2023 on CD and streaming platforms; its track “Souvenir Shop (Download the App)” is the fixed counterpart of the ever-changing version in the app. Album, app, singles, and band are presented on the project website, theaquaticmuseum.com. More details can be found in the process chapter.
Furthermore, examples in the inspiration and process chapters will draw on projects I’m working on, on my studies with my teachers at the Maastricht Conservatorium, and on the Karnatic Rhythm to Western Music Program on South Indian rhythmic concepts that I’m following at the Conservatorium van Amsterdam.
The reflection report published in this website format seems best adapted to combine advanced teaching research and MA research because it focuses on surprise and chance, offers wiki-style cross-references and links to external references, and also uses the same technologies that allow the reader to experience and grasp the different examples.
The literature review (II.) succinctly summarizes the research on algorithmic music and provides further links and examples. The methodology section (III.) gives an overview of the tools I used with hands-on examples to interact with the music. Interactive examples are also available in the ATS research: randomness and surprise in teaching. In the process documentation part (IV.), I shed light on the Aquatic Museum album and app creation process but also extend the research into more experimental territory. This intro section (I.) is rounded off with an overview of personal projects that gave me the idea and impulse for this research.
Black Dog - Led Zeppelin IV (2 December 1971) - Led Zeppelin - John Paul Jones, Jimmy Page, Robert Plant ↩︎
Judge, J. (2018). The surprising thing about musical surprise. Analysis (Oxford), 78, 225–234. ↩︎
Brackett, J. (2008). Examining rhythmic and metric practices in Led Zeppelin’s musical style. Popular Music, 27, 53–76. ↩︎
Oh Well, Part 1 - single (26 September 1969) - Fleetwood Mac - Peter Green ↩︎
High volume levels suit the song well; furthermore, by raising the volume on just those drumstick clicks, a student can play along and not miss the beginning of the riff. ↩︎
Huron, D. B. (2006). Sweet Anticipation: Music and the Psychology of Expectation. MIT Press. Judge, J. (2018, p. 226) argues against this generally accepted assumption: “Many musical surprises can be explained by the falsification of assessments of the present, rendering the appeal to expectation unnecessary.” ↩︎
Colman, D. R. (2006). The three princes of Serendip: Notes on a mysterious phenomenon. McGill Journal of Medicine: MJM, 9, 161. ↩︎
Hannula, M., Suoranta, J., & Vadén, T. (2005). Artistic Research. Theories, Methods and Practices. ↩︎
cf. e.g. Stadnicki, D. A. (2017). Play like Jay: Pedagogies of drum kit performance after J Dilla. Journal of Popular Music Education, 1, 253–280. ↩︎
Inspirations
For the Aquatic Museum app, I’ll build on previous personal projects that all use the same or related audio technologies but each focus on one specific topic. Instead of rehashing the techniques, tools, and workflows I already had experience with, I wanted to use the potential of this research to further explore the specific chance and random elements that are central to adaptive audio but are also making their way into music creation apps and generative music. These random features intrigued me the most and are also the ones that were only present as side functions or sketches in my other projects.
Subsections of Inspirations
Bata Project Inspiration
Interplay in Batá percussion music
I’ll extensively use original material from two projects I co-founded which are continuously being developed in a team of four. The first one is percussiontutor.com, a library and a practice app with rhythms from Cuba, Puerto Rico, Dominican Republic, Peru, Brazil and West Africa and expanding to other cultures (e.g., Flamenco was added in 2019). A large section is dedicated to the highly complex Cuban Batá drum culture. The concept is similar to other play-along software relying on separate tracks, with the notable difference that PercussionTutor focuses only on rhythm and doesn’t rely on MIDI takes but solely on recordings by skilled musicians specialized in specific genres.
Mother & father dialogue
Batá drums in Cuba consist of three cylinder-shaped drums of different sizes: iyá, itótele, and okónkolo, also referred to as “mother,” “father,” and “child.” While performing, the iyá player uses specific patterns to call the itótele player. The expected answer also consists of a highly coded rhythmical pattern. The calls can appear at random moments during the main groove pattern, although their position within the cycle is fixed. Here is an example of the Yacota1 base rhythm.
Yacota is a 6/8 rhythm played at a slow to medium tempo. In the conversation section, the iyá and itótele converse with one another while the okónkolo keeps time. The iyá calls the itótele in two similar ways. The itótele offers one response by adding a note in front of the open tone.
This is a good example of the use of random elements in a musical situation. It is reductionist to give too much weight to this feature considering the richness and cultural value of Batá music, but it is an instance of a musical language with call-and-response codes that are taught in this way and that performers respect during play. The following example will play the calls randomly; notation serves as reference only. If you can’t hear any audio, please refer to the Why This Format section.
The Flamenco Tutor app is a collaborative project with Flamenco dancer Niño de Los Reyes and percussionist Sergio Martinez that helps dancers configure their own music pieces for their choreographies and helps musicians learn Flamenco comping and composition. The app offers recordings of multiple alternate takes in different Flamenco music styles. Features include customizable sections that can be looped, instruments that can be selected and muted, and adjustable tempo. Users will also find musical and textual notation explaining the content, as well as instructional videos and practice exercises.
Here is a looped example of a verse in the Soleá por bulería style and a screenshot of the user interface. Please note that the channel count, Channels Playing, goes from 0 to 4 because what is actually played is not only one WAV or MP3 file but a stream of 4 channels of synchronized audio tracks that can be soloed or muted in the app and on this website. The right side of the user interface shows the structure of the composition with sections that can be repeated or reshuffled by dragging. A random button shuffles the sections according to musically meaningful rules (e.g., transitions are respected):
Interakt Inspiration
Interakt: a multiplayer collaborative rhythm game
Interakt is a multiplayer immersive arcade game designed by Alex Greenwood and me. Up to 4 players have to cooperate and compete at the same time to master rhythm challenges with auditory and visual cues.
Interakt can be used as a fixed arcade installation or as a stand-alone app for iOS. It relies on the Unity game engine coupled with Ableton Link for wireless music sync and FMOD for generative music. A touch screen or any standard MIDI controller can be used as a game pad. A wireless version with iPhone pads was also designed. All input methods can be mixed and matched.
Four players have to cooperate in order to keep a central sphere from expanding and finally outgrowing the play field. The game ends when the end of a level is reached or when the sphere overtakes the players.
Random rhythmic patterns
Besides the collaborative aspect, the originality of the game lies in the generation of random rhythmic patterns. The linearity is broken during gameplay because all players have to solve their individual random rhythmic puzzles and still manage together to prevent the sphere from expanding. This demo video shows a fast-moving game session with only one player actively engaged:
Music games can broadly be categorized1 as either rhythm action games or electronic instrument games. Frequency (2001) or Rhythm Tengoku (2006) are good examples of the first category of games, the most famous being Guitar Hero. Many clones appeared along the way, with less mainstream2 artist choices:
I decided to publish my reflection report under the following web-based format so that the reader can also listen to, explore, and interact with the different examples. All audio samples are loaded quickly and will work with sufficient speed in different browsers. Furthermore, different apps and software are referenced and are part of my literature review and bibliography. For the sake of demonstration and for the benefit of the reader, it is more practical to use video clips and interactive sequences.
The game audio middleware runs on many platforms, ranging from PC and Mac to web browsers, mobile devices, and gaming consoles. The regular audio examples use the following player, here with the opening song of The Aquatic Museum album:
The random examples use the following icon to indicate random features during playback: , or for looping. I also added play, stop, and mute buttons for instruments on multi-track files. Here is an example from the process section about rhythmic transformations. Three different loops of different lengths and meters (4/4, 13/8, and 21/8) are randomly shuffled, and the bass or drum track can be muted. Another example on swing ratios in the Advanced Teaching Skills section also uses different loops that are randomly shuffled and assigns different probabilities to four loops (3×30% and 1×10% chance). This is also a good test of the reader’s playback system, in case the buttons are grayed out or no audio is playing:
Publishing this content would have been difficult on a standard blog engine (e.g., WordPress), so I had to rely on different technologies. This website also offers wiki-style cross-links between the reflection report research and the pedagogical research.
The main focus is not on technological aspects, but the ones that are relevant to the main subject will be mentioned and linked in the following callout boxes:
Note
Links to the different tools used on this website:
Netlify, a platform that helps developers to build, test and deploy websites
The Chrome browser is recommended, but this site has also been tested with Safari and Firefox. It should function on most recent mobile devices, Android as well as iOS. In some rare cases on mobile devices, it may be necessary to request1 the desktop version of a website. Should you encounter any bugs or technical problems, please email me, and I can send you all the media files in more traditional formats.
De Zee-Monsters - publisher widow Gijsbert de Groot, 1692 - 1717 - Rijksmuseum - CC0 1.0
Subsections of Literature Review
Research Fields
Different sources were helpful for this report, and the research can usefully be divided into academic, theory-based research (generative and aleatory music grouped under the meta-category of algorithmic music) and research focusing on media and publications that fit the form of the release version of The Aquatic Museum (TAM) app. The final form of the app will be a hand-drawn three-minute video clip with an evolving, changing title song, The Souvenir Shop.
Focusing solely on adaptive audio and music as used in the TAM video app could have filled this research (e.g., for an application of a game audio engine1 in theater), but I was more interested in exploring one of the trademarks of music found in video games: random and chance features also found in music-making software and apps—a characteristic I also explored in my teaching research in Part II on this website.
The TAM app is based on Astrid Rothaug’s original short animated film and follows a linear timeline with timed interactions that influence the music. It can also be tied to experiments in film (e.g., interactive Netflix shows). The TAM app is in fact only one element of the TAM album release—a three-minute teaser that provides insight into the creative process and offers different perspectives2 on the same song.
Swift, S. (2018). FMOD, an Audio Engine for Video Games, Adapted for Theater. The 59th Annual USITT Conference. ↩︎
For an original approach, cf. Beck’s 2012 album, released only as sheet music and left to fans to perform. ↩︎
Algorithmic Music
Algorithmic Music
Generative Music
“Though I may have the pleasure of discovering musical processes and composing the musical material to run through them, once the process is set up and loaded it runs by itself.” in Schwarz, K. R. (1981). Steve Reich: Music as a Gradual Process Part II. Perspectives of New Music, 225–286.1
Steve Reich’s quote summarizes the essence of generative music as discovering musical processes and then letting the system run by itself. Randomness is one of the process elements that enters into the equation.
Brian Eno and Steve Reich
“I truly believe that our grandchildren will one day say to us: do you mean you really listened to the same piece over and over again?” Brian Eno (1996)2
“I’ve always been lazy I guess. So I’ve always wanted to set things in motion that would produce far more than I had predicted” (Brian Eno 1996)
Brian Eno’s career3 spans a long musical arc, from the seventies (as co-founder of Roxy Music) to today’s innovative multimedia projects. I’m going to focus on his contributions to algorithmic music and his links with Steve Reich. He has been at the forefront of creating and releasing music in different formats that allow the listener to experience the nonlinearity of generative music.
It’s worth quoting Brian Eno, the musician who owns the domain name generativemusic.com and coined4 the term, from a 2010 interview5 in The Guardian:
“I came out of this funny place where I was interested in the experimental ideas of Cornelius Cardew, John Cage and Gavin Bryars, but also in pop music. Pop was all about the results and the feedback. The experimental side was interested in process more than the actual result – the results just happened and there was often very little control over them, and very little feedback. Take Steve Reich. He was an important composer for me with his early tape pieces and his way of having musicians play a piece each at different speeds so that they slipped out of synch. But then when he comes to record a piece of his like, say, Drumming, he uses orchestral drums stiffly played and badly recorded. He’s learnt nothing from the history of recorded music. Why not look at what the pop world is doing with recording, which is making incredible sounds with great musicians who really feel what they play. It’s because in Reich’s world there was no real feedback. What was interesting to them in that world was merely the diagram of the piece, the music merely existed as an indicator of a type of process. I can see the point of it in one way, that you just want to show the skeleton, you don’t want a lot of fluff around it, you just want to show how you did what you did. As a listener who grew up listening to pop music I am interested in results. Pop is totally results oriented and there is a very strong feedback loop. Did it work? No. We’ll do it differently then. Did it sell? No. We’ll do it differently then. So I wanted to bring the two sides together. I liked the processes and systems in the experimental world and the attitude to effect that there was in the pop, I wanted the ideas to be seductive but also the results.”
Steve Reich’s tape works directly influenced6 Brian Eno and led him to explore and establish the principles of ambient music. He “became interested in creating musical systems that produced music of infinite length that never repeated itself rather than linear works that had a fixed structure and time frame”.7 The company SSEYO was founded in 1992 and developed Koan, a generative music engine. Brian Eno received a copy in 1995, which led him to coin the term “generative music”.
Algorithmic Music
Generative music falls under the larger category of algorithmic music. An algorithm can be understood as a well-defined set of operations or rules, but “algorithmic music” extends to:
“a rich field of activity, defined by the urge to explore and/or extend musical thinking through formalized abstractions. In the process of making music as … algorithms, we express music through formal systems of notation, taking a view of music as the higher order interplay of ideas.” 8
Minimal music from the 1960s by Glass and Reich is algorithmic in nature, although manually composed and with a defined start and end point (e.g., Clapping Music, Piano Phase). Ligeti’s Continuum for harpsichord (1968) uses “rigorous algorithmic procedures” on pitch and rhythmic structures “permitting complex rhythmic juxtapositions and transformations”.
Examples of Brian Eno’s Generative Works
Ambient 1: Music for Airports (1978)
“I worked on things like [algorithmic music] for a while: Music for Airports and Discreet Music were examples, but what they represented were recordings of these processes in action. What I really wanted to do was to be able to sell the process to somebody, not just my output of it.” (Brian Eno, Dredge 2012)
The following apps released for iOS (and some for Android) seem like the logical answer to the difficulties of distributing algorithmic, generative, ever-changing music on a physical medium. Fixing them on vinyl or CD was always a dead end. In “Algorithmic Music for Mass Consumption and Universal Production,”9 Yuli Levtov states that distribution is the main challenge for algorithmic music in reaching a larger public, or in some instances any public at all. Brian Eno, in collaboration with Peter Chilvers, released several generative apps allowing users to influence different generative parameters.
In the methodology section, the reader can find examples of generative and semi-generative music I created and can further explore the distinction and links between adaptive audio and generative music.
Aleatoric Music
“Aleatoric music, or aleatory music or chance music (from the Latin word alea, meaning dice) is music in which some element of the composition is left to chance.”10
“the pool of possible outcomes is so large that any waltz you generate with the dice and actually play is almost certainly a waltz never heard before. If you fail to preserve it, it will be a waltz that will probably never be heard again”.10
Music of Changes by John Cage (1951)
Uses an ancient Chinese text (the I Ching) to generate random numbers for tempo, dynamics, and note duration.
Klavierstück XI by Karlheinz Stockhausen (1956)
Nineteen musical fragments that can be played in any order; the musician can start with any fragment.
Pithoprakta by Iannis Xenakis (1956)
It uses physical and mathematical principles and is an example of what Xenakis called stochastic music.
In C by Terry Riley (1964)
Riley suggests “a group of about 35 is desired if possible but smaller or larger groups will work”. Consisting of a series of short melodic fragments, In C is often cited as the first minimalist composition.
Schwarz, K. R. (1981). Steve Reich: Music as a Gradual Process Part II. Perspectives of New Music, 225–286. ↩︎
Gardner, Martin. The Colossal Book of Mathematics: Classic Puzzles, Paradoxes, and Problems: Number Theory, Algebra, Geometry, Probability, Topology, Game Theory, Infinity, and Other Topics of Recreational Mathematics. New York: W.W. Norton and Company, Inc., 2001. Print. ↩︎↩︎
Artist-Specific Music Apps
Besides music games, there is a specific category of music-centric apps, software, and installations that target either the music consumer or the music producer. Those projects serve as vehicles for band promotions, CD releases, or artistic creations by artists. These are the ones most relevant to my research, although some of their features focus more on generative music and soundscape design.
Björk Projects
Among the music apps released by musicians, Björk’s projects stand out—one is an application linked to an album but also designed as a creative tool for making music with the sounds from the album.
Biophilia was released in 2011 for Android and iOS and is described as follows:
“Biophilia is an extraordinary and innovative multimedia exploration of music, nature and technology by the musician Björk. Comprising a suite of original music and interactive, educational artworks and musical artifacts, Biophilia is released as ten in-app experiences that are accessed as you fly through a three-dimensional galaxy that accompanies the album’s theme song Cosmogony. All of the album’s songs are available inside Biophilia as interactive experiences: Crystalline, Virus, Moon, Thunderbolt, Sacrifice, Mutual Core, Hollow, Solstice, and Dark Matter.”
Features:
Three-dimensional galactic interface with the song Cosmogony
Nine song apps available as in-app purchases
Music scores with karaoke playback
Abstract song animation
Lyrics
Essays
MIDI out to drive instruments
Björk: Solstice
Björk: Solstice is a standalone app from the Biophilia suite:
“At the center of the app is a sun from which the user pulls rays of light to form a circular harp of strings plucked by colorful orbiting planets; tilt the device and the orbital planes turn into the branches of a three-dimensional Christmas tree, the stars becoming snowflakes. In Solstice, Björk plays the “Solstice” song with a specially commissioned pendulum-harp which embodies the idea of gravity that was central to the song’s creation.”
There was no follow-up to these projects, and the last one, a VR recording, dates back to 2015:
Björk: Stonemilker VR
First released as an app, this clip can now be seen on YouTube:
Ninja Jamm - Remix App for iOS from Ninja Tune and Seeper (2013)
This app by Ninja Tune made headlines in 2013 because it offered an easy and playful way to interact with the electronic compositions of different artists:
Zeemonster gevangen tussen Scheveningen en Katwijk - Anonymous 1661 - Rijksmuseum - CC0 1.0
Subsections of Methodology
Adaptive Audio
Subsections of Adaptive Audio
Adaptive Audio Overview
Adaptive Audio
To illustrate the creation of the TAM app with its two distinct sections—a semi-generative opening sequence and a traditional song structure whose elements are reshaped and transformed—the methodology section gives an overview of personal examples of adaptive audio tools and creations in video game engines with a focus on their random features. The bibliography section references recent books on composing and creating music and sound for video games.
Different techniques exist to compose adaptive music. A distinction is generally made between:
Horizontal Resequencing:
“Horizontal resequencing is a method of interactive composition where the music is dynamically pieced together based on the actions of the player. For example, when the music is playing underneath the gameplay, it may reach a decision point where it could either go to a new section of music or repeat the previous section depending on the player’s actions.”1
Vertical Remixing:
“Vertical remixing is an interactive composition technique in which layers of music are added or taken away to create levels of intensity and emotion. This method of scoring is useful when the composer needs multiple quick changes to intensify the score, where harmonic changes based on gameplay are not as important.”2
These adaptive audio tools can be analyzed and placed into a wider context of algorithmic music (cf. next pages), into which generative and semi-generative music falls, with randomness being a key element.
Sweet, M. (2015). Writing Interactive Music for Video Games: A Composer’s Guide. Pearson Education, p. 143. ↩︎
For The Aquatic Museum app, I use the FMOD audio middleware combined with the Unity game engine. While this research is not about the technical aspects, it is important for the methodology section to give an overview of the tools used in my creative process. I use the same tools in my advanced teaching skills research by taking advantage of software resembling traditional DAWs but designed for video games. However, differences from traditional audio workstations do appear in the media creation process. This research also focuses on one specific element or feature used in adaptive audio and other music creation tools: randomness or aleatory features.
Composing music for games requires a different approach from the more linear thinking used in film music.
“an end-to-end solution for adding sound and music to any game. Build adaptive audio using the FMOD Studio and play it in-game using the FMOD Engine.”1
Three market segments are listed: sound designers, programmers, and producers. The idea is to facilitate the video game audio and music creation process and the division of tasks between audio designers and coders.
Adaptive Ambient FX Audio Example
Excellent tutorials can be found on YouTube2, and for book recommendations, please refer to the
Game Music and Audio Book References section. The simplest entry into the technical aspects comes from a personal example from the 2021 Global Game Jam in Luxembourg. The setting is a spaceship—yes, exactly as you’d imagine one: dark, damp, menacing, and claustrophobia-inducing. Furthermore, dear reader, note how stressful the absence of a stop button is. The ship’s soundscape was created with different sound samples, some of them hand-recorded with voice, paper, and tube accessories.
Drag and release the slider to morph from a lighter to a darker ambiance.
Here is a video of the actual project in FMOD Studio, the main design tool of the suite. Two important things to note: first, I’m not using the timeline in this example.3 Second, the slider triggers different sound objects (called instruments in FMOD) when moved by the user or triggered by code, and some audio samples are triggered asynchronously. Automation lanes, shown at the end of the clip, control volume as well as the parameters of the different instruments.
The audio designer can now collaborate with the coding team to implement this single atmospheric ambiance. This is a simple example with a single parameter, so the amount of coding is limited, but even for simple audio tasks, the work required can grow exponentially despite the general tendency of software tools to rely more on visual design and less on coding.
Notice the visual similarities to DAWs (e.g., Cubase, Ableton), but with a fundamentally different adaptive audio logic. ↩︎
Music in Video Games
Adaptive audio tools provide advanced music creation possibilities that differ from the features found in more traditional DAWs. Here is the theme for one of the monster encounters (cf. sound in video games). It is a collage I created from a Donizetti opera1 recorded in 1911, Omnisphere synth sounds, and the preceding page’s soundscape. Some sonic elements as well as the volume balance are adapted dynamically during gameplay.
Adaptive Audio for a Platform Game
For the next example, I used only Tempest, a drum machine designed by Roger Linn, to create a groove with different drum and synth layers. The main groove uses the Batá rhythm Olokun as a template.
Drag and release the slider to start the percussion playback on 1 and increase the volume on 2 and 3. Slide back to 0 to mute the percussion instruments. Note that the percussion loop will only start every 2 bars due to its quantization settings. The only random element is a 3-bar section of the main loop that has 2 alternatives. Channel count increases to 2 when the Batá drums are playing.
I simplified this example and created 3 videos to illustrate the different adaptive layers and random features. An easy way to imagine the use of FMOD is to think of having a scaled-down version of Ableton Live that can be hidden in a game, an app, or (like here) on a website:
Complex Example with Changing Song Structure and Randomly Triggered Percussive Instruments
Similarities and Differences with DAWs
These examples only scratch the surface of what is possible in an audio game engine. To understand the full potential of FMOD, imagine a compact DAW hidden in a game or multimedia installation. Real-time effects and interactive mixing further extend the possibilities for The Aquatic Museum app.
Una furtiva lagrima (L’elisir d’amore, Act 2) - Gaetano Donizetti - performed by Enrico Caruso (public domain). ↩︎
Types of Sounds
As the preceding examples show, the design and creation process of audio and music for games is based on traditional composing, recording, and creation practices, but the assembly and delivery of the media assets require other tools and an adapted kind of thinking and planning.
A common classification of sounds in games, adapted from film, is as follows:
Diegetic Sounds
Diegetic sound is heard by both the characters and audience. Also called ’literal sound’ or ‘actual sound’.
Sound effects: car engine, collisions, explosions
Music: car radio, band playing on screen
Vocals: dialogue, voices, crowds
Non-Diegetic Sounds
Non-diegetic sound is represented as coming from a source outside the story space, i.e., its source is neither visible on the screen nor has been implied to be present in the action. Also called ’non-literal sound’ or ‘commentary sound’. wiki
Readers who prefer to jump right into the process documentation can skim the “Tools of the Trade” section, but game engines and adaptive audio middleware go hand in hand to create an immersive user experience. The Aquatic Museum app is created with both the Unity game engine and FMOD. Although you can use FMOD for other software projects (as it is used here on this website), its main purpose lies in the flexibility it offers when working with game engines.
Game Engine
Today, most games (from indie to AAA) are created using software called “game engines”. They provide many essential features found in most games out of the box and require less coding:
Graphics
Physics
Input
Sound
Networking
Artificial Intelligence
Good game engines nevertheless offer complex coding options for maximum flexibility, and most non-trivial games still rely on them.
#include"MyOpenGLWindow.h"classVec3 {
public:union {
struct{
float x, y, z;
};
float data[3];
};
Vec3() {}
Vec3(float x_, float y_, float z_) {
x = x_;
y = y_;
z = z_;
}
voidset(float x_, float y_, float z_) {
x = x_;
y = y_;
z = z_;
}
Vec3 operator*(const Vec3& r) const { returnVec3(x * r.x, y * r.y, z * r.z); }
Vec3 operator+(float s) const { returnVec3(x + s, y + s, z + s); }
Vec3 operator-(float s) const { returnVec3(x - s, y - s, z - s); }
Vec3 operator*(float s) const { returnVec3(x * s, y * s, z * s); }
Vec3 operator/(float s) const { returnVec3(x / s, y / s, z / s); }
};
// etc. + 300 LINES
Supported Platforms
All projects developed in Unity can potentially run on all the following platforms, ranging from PC, Mac, and Linux to game consoles (PlayStation, Xbox, etc.) and mobile devices. I say “potentially” because most of the time, platform-specific development is required, which can require substantial additional development effort.
Types of Game Engines
There are many, but only the three most famous are linked here. They are also used extensively for VR projects, installations, and animated films.
Godot is completely free and open source under the MIT license. “No strings attached, no royalties, nothing. Your game is yours, down to the last line of engine code.”
Godot doesn’t have any AAA games to show (yet), but it is becoming increasingly popular:
Game Jams Are the Best Way to Learn
One of the most famous game jams is globalgamejam.org, which takes place every year at different locations around the world. Participants are given a main theme (e.g., “Waves” or “Repair me!”), and multidisciplinary teams have 48 hours to build a complete game and upload it. In January 2020, there were 934 locations in 118 countries, and teams created 9,601 games in one weekend.
Musicians are always in high demand, and those with minimal coding skills and knowledge about game audio engines even more so.
Global Game Jams take place every January, and registration for the Netherlands can be found on this website: globalgamejam.nl.
48 Hours to Finish a Game
Here is one of the sessions for which I did the audio.
Use a microphone-enabled device to sing the same note that the enemies are producing to destroy them with a beam of sound. Uses FFT to calculate the pitch of the singer’s voice. We integrated the Google Cardboard API to allow the player to play the game without using any buttons. It uses pitch detection to destroy specific enemies. Works on windows and mac. We tested it on iOS and it works pretty well but the enemy sound are not working. The music works fine though. (Global Game Jam Luxembourg 2017)
Game Audio Engines
There are also quite a few choices of audio engines and authoring tools for video games. The two most widely used are listed here.
For Wwise, there is an older step-by-step tutorial built around Limbo:
Process Documentation
Subsections of Process Documentation
The Aquatic Museum (TAM): Collective & Album & App
Interactive Ever-Changing Music Video Clip?
The following chapter details the iterative creation process of an app specially designed for a new project, The Aquatic Museum (TAM)—the name given to a band project, an album, and an app. The sections below lay out the project and the preproduction steps of the album and app. I’ll then take an excursion into other musical creations using chance and surprise elements, some of which have found their way into the TAM app, while others were deemed, in a democratic artistic collective process, too experimental. I’ll conclude with finalized examples of audio interactions in the TAM app. Album and app were released in 2023; the last section of this page presents the release.
The Aquatic Museum Project Presentation
Release: Album, App, and Website
After this thesis was submitted, The Aquatic Museum moved from prototype to public release in spring 2023. The three outlets complement each other: the album fixes one version of each song, the app lets the title track keep changing, and the website ties everything together.
Album
The album The Aquatic Museum (11 tracks, about 39 minutes) was released at the end of May 2023 on CD and on streaming platforms (Spotify, Apple Music) with composiitons by Claire Parsons and myself. The collective, which I co-founded in 2021 with Claire Parsons and Nicole Miller, recorded the album with Claire Parsons (vocals), Maia Frankowski (violin), Nicole Miller (viola), Annemie Osborne (cello), Eran Har Even (guitar), Jérôme Klein (drums), and me (electric and double bass), joined by the Belgian Q-Some Big Band for the big band arrangements prepared during the score layout and orchestration phase. Charles Stoltz recorded, produced, and mixed the album at Holtz Sound, Markus Schneider mastered it at Skywalk, and Astrid Rothaug created the artwork.
Four singles accompanied the release: “Entrance,” “Large Pleasure Watercraft,” “Trash Tub” (a live recording with Q-Some), and “Trapped Air Bubbles,” the latter featuring underwater images by technical diver and photographer Audrey Cudel. Visual artist Jeanne Held contributed music videos. “Entrance” opens the album and is also the song used to demonstrate the audio player in Why This Format?:
The free The Aquatic Museum app has been available on the App Store since 7 April 2023. It runs on iPhone and iPad (iOS/iPadOS 15 or later), on Macs with Apple silicon, and on Apple Vision Pro, while this website hosts web-based examples of the same audio technology. The store description summarizes the concept developed in this chapter:
“The Aquatic Museum App features the multilayered album track ‘Souvenir Shop (Download the App)’ that will transport you into an intricate and imaginative aquatic world hand-drawn by Astrid Rothaug. Take a journey through various museum chambers as you’re guided by visual and musical cues, courtesy of the original Aquatic Museum Band, the Mechelen-based Q Some Big Band, and Valencia-based flamenco percussionist Sergio Martínez (ES).”
The subtitle “(Download the App)” turns the album track into a pointer: on the album, Souvenir Shop is one fixed recording, while in the app the same song becomes the ever-changing music video described on the following pages. The credits also connect the app to my earlier projects: flamenco percussionist Sergio Martínez co-created the Flamenco Tutor app. As during the research, development remained iterative after the launch: version 1.2, released two weeks later, added further interactions, and later updates reduced the download size and refined the user interface.
Website
The project website, theaquaticmuseum.com, gathers the album with streaming links, the app with a video preview, the singles and their music videos, and the band with all collaborators. Like this reflection report, it is built with Hugo.
Evolving Music Video
Film Rhythm After Sound
TAM singer Claire Parsons had worked with Vienna-based Astrid Rothaug on a previous music clip for her In Geometry album. For TAM, we wanted to give Astrid as much freedom as possible in creating a three-minute short film with hand-drawn animated frames and a story loosely spun around an Aquatic Museum theme. For inspiration during our first meetings, we provided Astrid with demo and pre-production material and settled on a simple pop music tune (with some twists), co-written1 by Claire and me, The Souvenir Shop—a constant in every museum visit.
The process2 of creating a clip for music or music for a short video was never clearly laid out. The final song choice came later during the album writing and creation process. Because we had the unique opportunity to work with an artist who spent weeks creating frame-by-frame hand-drawn animations in high resolution, we discarded several other music app ideas based more on traditional video game mechanics. Hand-drawn graphics are rare in video games (cf. literature review chapter) because the translation from hand-drawn artwork into computer-animated sprites can lead to aesthetic clashes. Hand-drawn paper-style flat graphics don’t lend themselves easily to an interactive graphic style. Also, to respect Astrid’s vision and to stay true to the original, we maintained the clip’s linearity.
Create a three-minute repeatable experience that imitates the music video format but adds a new musical surprise twist. The animated short with few user interaction opportunities and a linear storyline is the backdrop to The Souvenir Shop tune.
Take away the power from the user who expects to be spoiled with features (that was the most daring choice). Of course, this was also done to simplify our work, but above all to bring the main focus onto music without compromising audio quality, number of loops, or sonic experiments. Offering no interactive options is still on the table, and the app would then produce a randomly shuffled soundtrack. Music and audio in video games is a central atmospheric mood element, but too often in indie games musical considerations come very late in the development process. Furthermore, sound performance and quality have to make room for graphic performance and user interaction priorities.
Finally, it is key to be able to release it in many different formats (iOS, Android, web—as shown here—desktop computer, YouTube clip, and others; cf. Brian Eno’s generative apps).
Graphical user interface design is a challenging domain of human–computer interaction. How do you guide the user in a game or software? For TAM, I chose to use color to indicate possible interactions: Astrid’s footage is kept in black and white3, and color is used sparingly.
In the TAM app, the different possible interactions that will influence the music are indicated by color spots and surfaces. The doorbell from clip 1 and the transforming whale from clip 2 are examples among other larger and smaller color indications:
The tree example shows multiple user interactions where different trees trigger random voice tracks. Later in the clip, a chessboard appears when the Souvenir Shop tune reaches the solo section. A press by the user will trigger another instrument playing a solo, and a press on the same spot while a certain solo instrument is playing will slowly fade into another solo version. As an Easter egg, some instrumentalists agreed to play a funny or even poorly executed version of their solo that will only rarely be triggered (1 out of 1000 times). Interaction times are limited to a few seconds, the solo section is short, and the user has no built-in recording option.
Basic touchscreen single-finger interactions are supported, which also translate to mouse navigation: touch (single, double), drag, slide. Some user actions will have visual feedback cues, but audio is always front and center. The app is semi-generative4 and not generative in the sense of the original definition because it only uses the loop and soundscape material that we recorded and that fits the song.
The majority of pieces for the TAM album were written by Claire Parsons with song and lyric contributions by me on several tunes. ↩︎
Jacobs, L. (2015). Film Rhythm after Sound: Technology, Music, and Performance. University of California Press. ↩︎
Music and audio in video games is certainly a central atmospheric mood element, but too often in indie games musical considerations come very late in the development process. Furthermore, sound performance1 and quality have to make room for graphic performance and user interaction priorities. As a new take on the music video clip, the TAM app takes the opposite perspective and puts music first. The animated short with few user interactions and a linear storyline is the backdrop to The Souvenir Shop tune.
The first step consisted of laying out a lead sheet for the rhythm section (piano, guitar, bass, drums) and adding a base orchestration for big band and string ensemble; the TAM album features several compositions with big band2 and string trio3.
To come up with a list of musical parameters worth changing, I drew on workshops and classes1 at the Conservatorium Maastricht and on research in the fields of music education2 and perception.3 The overall musical design process was less planned and more iterative than the following table would suggest. Intuition, feasibility regarding the software used (here mostly FMOD), time and budget constraints, testing, and personal live concert experience all played a role in the final decisions.
Musical Parameter
Audience Change Perception
Melody
high
Harmony
medium
Rhythm/Groove
high
Orchestration
medium
Form
low
Dynamics
medium
Color
low
Register
low
The impact on the perception of the audience is a key element, but one of the objectives was also to analyze whether the tools, methods, and research into randomness could lead to interesting results from the perspective of the composer or performer, for whom all parameters are essential.
Recording November to March 2022
The recordings were done during several sessions in different studios. Here are the different takes for all instruments and variations, recorded into Pro Tools:
Chapter on “The Parameters of Music Education”. Swanwick, K. (2015). A Developing Discourse in Music Education (The selected works of Keith Swanwick). Routledge. ↩︎
Friberg, A., Schoonderwaldt, E., & Hedblad, A. (2011). Perceptual ratings of musical parameters. Gemessene Interpretation-Computergestützte Aufführungsanalyse Im Kreuzverhör Der Disziplinen, Mainz: Schott, 237–253. ↩︎
Random Opening Voicings and Instruments
The opening scene from the TAM app gives the user a window of a few seconds to press a button that turns red. This triggers a random C minor chord leading into verse 1 of the Souvenir Shop tune. Not only is the voicing randomly picked, but so are the instruments playing the chord (piano, big band, strings).
The following was an experiment for the in praise of folly string trio1 (violin, viola, and cello). Some voices were written out, and the final bar only gives minimal indication of which notes to play: a pentatonic-derived scale with a 9th, no 7th and no 6th:
Here is an unedited extract of the trio playing random picks of bar 68. I left their spontaneous reactions at the end.
inpraiseoffolly.co a string quartet, in their regular formation, based in Brussels. ↩︎
Changing Speeds
This example might not find its way into the final version of the TAM app but will be used during live performances. The main bass groove is transformed by two metric modulations: the first uses the triplet (bar 6) as the new reference speed, and the second the quintuplet (bar 11). Pianist and composer Vijay Iyer used these rhythmic transformations in his adaptations of Mystic Brew and Human Nature (cf. my original post here peckels.com/blog/mystic).
The Limits of Adaptive Music
These rhythmical transformations are the hallmark of South Indian music. There are different ways to indicate these rhythmical devices in Western music notation. I chose to use the metric modulation indication with new tempo markings. A tuplet notation with group bracketing1 would also be possible.
Transforming this example into an adaptive, randomized version in FMOD reveals the limits of using audio game engines as well as most DAWs for writing complex rhythmical music with an added layer of structure options. Rhythm changes are easy to manage in Pro Tools, Logic, etc., but this is not the case with tuplet writing. While Logic has built-in functions up to quintuplets, only Cubase handles everything above gracefully. Over the last year on iOS, some interesting non-linear DAWs have been designed with different plugins; for example, Victor Porof’s Atom Piano Roll 2 offers a piano roll-like editor with divisions up to 13 and can be combined with AUM, an audio mixer, plugin host, and recorder.
Two problems occur when trying to implement these rhythmical transformations in one of the randomized versions of The Souvenir Shop tune. First, the metrical grid is extended from 8 to 13 to 21 eighth notes, so all the other instruments would need to be re-recorded to make them fit, and different adjustments would be necessary to give a convincing rendering of the tune.
The second difficulty lies in the expanding number of combination possibilities with only these three different metric grids:
every four bars (1, 9, and 11) a version could repeat itself
every version could go to one of the two other versions (a total of six possible combinations)
each time the right metric modulation would have to be played (one could imagine ignoring this transition and doing a “hard cut” transition).
(N.B. but key: real musicians have none of those limitations, given enough practice)
By leaving out the tuplet metric modulations, the randomized version is easier to integrate with FMOD. The metric changes now happen randomly, with the bass no longer anticipating the new tempo with a triplet or quintuplet but simply playing a sustained note:
Thanks to these additive and subtractive rhythmic devices, the transitions are still interesting and don’t sound like simple tempo changes.
In generative music, these rhythmic transformations with added random elements are promising. I will now take a detour into further Karnatic techniques and link them to live coding and modular synthesis, where random features are widespread.
Reina, R. (2017). Applying Karnatic Rhythmical Techniques to Western Music. Routledge, p. 45. ↩︎
Karnatic Rhythmical Devices
Karnatic Practice
Part 1 of the detour uses as an example the following étude, which I wrote for the first-year assignment1 of the Applications of Karnatic Rhythm to Western Music program, using only devices taught during the first year. The following composition is entirely based on one eight-note melodic cell (F D A Bb G D Eb C) in the violin lead voice and the rhythmic cells of the drone voice (cf. bar 1 for both), with melodic and rhythmic transformations applied throughout.
1. Karnatic Étude
2. Cell-Based Melodic and Rhythmic Patterns
One striking characteristic of South Indian music is the range of development techniques that can be applied to a basic pattern, cell, phrase, or melodic idea—“basic” in the sense of forming an essential foundation or a starting point. Here is my example of a rhythmic development2 based on three sextuplet cells (a b c).
This cell-based pattern approach links the thesis to the software shown in the next video…
For an authentic example, cf. p. 67 in Reina, R. (2017). Applying Karnatic Rhythmical Techniques to Western Music. Routledge, with audio example 38 on the author’s website. ↩︎
The TAM app was released on the App Store in April 2023, followed by the album at the end of May 2023; both are presented on the project website, theaquaticmuseum.com (cf. release details). A complete critical assessment of the app will benefit from real-world feedback now that it is publicly available. Online polls and small focus groups can provide further insights into usability questions. In the meantime, I’ll reflect on the creation process and the exchanges with the musicians, and give some personal reflections on the potential audience of music apps. Some of the following points are discussed in more detail in the process section.
Creator Perspective
Adaptive audio is here to stay and offers challenges and opportunities for the composer and performer, moving away from the linear constraints of composing music. The aleatory, chance-related functions and features will become powerful assistants in composing and performing.
Research led me to explore Karnatic development; although it will take months and years (I’ll add a second year of lessons in 2022/2023) to gain a solid grasp of the theory and possibilities, the groundwork is there to apply this knowledge to generative electronic and more exploratory music.
Other tools would have led to more experimental results; for example, Max by Cycling ‘74 or Pure Data are designed for advanced audio manipulation, but the FMOD audio engine combined with the Unity game engine guaranteed my ability to release a finished product. Max packages and apps can be created and distributed, but they don’t match the user experience of native apps and, most importantly, they cannot be ported to mobile devices or the internet at this time. The Unreal game engine, with its sequencing possibilities and growing iOS support, is interesting because it offers a set of synthesis and audio manipulation tools.
To put it very bluntly: “Is all this effort for a 3-minute pop song worth it?” Considering the way pop music is produced1 today on an industrial scale with industrial processes, this question is worth asking.
The potential of generative music is also genre-dependent. Brian Eno’s ethereal music is probably more suited to the generative music process than groove-based music.
It was difficult, and in some cases even impossible, to break out of the loop framework. Advanced rhythmic techniques require re-recording every instrument unless one opts for electronic sounds only. Despite these inherent limitations of the FMOD tool, the TAM app is a unique opportunity to explore all available options for one simple reason: in this project, I am not limited by heavy audio compression or by having to worry about performance issues with sound and music. In standard video games (i.e., 99.9% of them), game performance comes first. Audio and sound are ranked second, which means using heavy compression to reduce file size. This is not the case here because the visual media content consists of a three-minute hand-drawn video with minimal color information. The interactive parts are all timed, and the user has only a short window of opportunity to interact with the material.
Copyright is also a domain that hasn’t been tackled in this research. The question of registering the Souvenir Shop song is still an open one.
Pertinence for the TAM Project
Audience Perspective
One open question concerning the audience and the app as a marketing tool is: Will the app motivate the audience to listen repeatedly to the same 3-minute Souvenir Shop tune? Will they notice the differences?
The originality of the TAM app also reveals its weak point. Today’s audiences have high expectations when playing video games or interacting with apps. This is one reason to limit interactions to a few seconds by removing options.
Summary
To conclude, I’ll add the following points:
Now that the app has been released, a complete critical assessment from the audience perspective can draw on real-world feedback.
Other tools would have led to more experimental results; for example, Max by Cycling ’74 or Pure Data. However, the publishing format is key.
Despite these limitations, this research was a unique opportunity to explore all available options: in this project, I am not limited by heavy audio compression or by having to worry about performance issues with sound and music.
Research led me to explore Karnatic development and experiment with an electronic music approach; although it will take more time to gain a solid grasp of the theory and possibilities, the groundwork is there to apply this knowledge to generative electronic and more exploratory music. Live coding is next.
Postscript: The Rise of AI-Generated Music
Since this research was completed, the landscape of generative music has shifted dramatically with the emergence of large-scale AI music models. Platforms such as Suno and Udio can generate full arrangements — vocals, lyrics, and instrumentation — from text prompts alone, while Google DeepMind’s MusicLM (Agostinelli et al., 2023) and Meta’s MusicGen (Copet et al., 2023) have demonstrated high-fidelity generation conditioned on text, melody, or both. These tools represent a fundamentally different paradigm from the rule-based and stochastic approaches explored in this thesis: where TAM relies on carefully authored musical material recombined through adaptive audio middleware, neural music generation produces material ex nihilo from learned statistical distributions.
This distinction matters. The compositional decisions embedded in TAM — voicing choices, Karnatic rhythmic structures, metric modulations — reflect deliberate musical craft. AI-generated music, by contrast, raises unresolved questions about authorship, originality, and the role of the composer (Sturm et al., 20192). The copyright concerns noted earlier in this chapter have only intensified: at the time of writing, lawsuits from major record labels against AI music platforms are ongoing, and the legal status of AI models trained on copyrighted material remains unsettled.
Nevertheless, the convergence is worth noting. Future iterations of projects like TAM could potentially combine hand-crafted adaptive structures with AI-generated material, using models not as a replacement for compositional intent but as another source of controlled surprise — yet another king or queen of Serendip.
Seabrook, J. (2015). The Song Machine: Inside the Hit Factory. W. W. Norton & Company. ↩︎
Sturm, B. L., Iglesias, M., Ben-Tal, O., Miron, M., & Gómez, E. (2019). Artificial intelligence and music: Open questions of copyright law and engineering praxis. Arts, 8(3), 115. ↩︎
Bibliography
Sources
Austin, J. H. (2003), Chase, chance, and creativity: The lucky art of novelty, MIT Press.
Bernardo, F., Pestana, P. D. and Martins, L. G. (2015), The Smart Stage: Designing 3D interaction metaphors for immersive and ubiquitous music systems.
Boyle, R. (2009, March). Serendipity and Zemblanity. Himal Southasian Magazine.
Brackett, J. (2008). Examining rhythmic and metric practices in Led Zeppelin’s musical style. Popular Music, 27, 53–76.
Brown, P. (2005), Is the Future of Music Generative?, Music Therapy
Today (online) (2), 215-274.
Brown, A. R., Gifford, T. and Wooller, R. (2010), Generative Music
Systems for Live Performance, First International Conference on Computational Intelligence pp. 290.
Cheung, V. K., Harrison, P. M., Meyer, L., Pearce, M. T., Haynes, J.-D.
and Koelsch, S. (2019), Uncertainty and surprise jointly predict
musical pleasure and amygdala, hippocampus, and auditory cortex
activity, Current Biology 29(23), 4084--4092.
Collins, N. (2008), The analysis of generative music programs,
Organised Sound 13(3), 237--248.
Colman, D. R. (2006), The three princes of Serendip: Notes on a
mysterious phenomenon, McGill Journal of Medicine: MJM9(2), 161.
Craig, J. (2017), Adaptive audio engine for EEG-based horror game, in Audio Engineering Society Convention 142.
Daniels, D., Naumann, S. and Thoben, J. (eds.) (2015), See This Sound: Audiovisuology: A Reader, Verlag der Buchhandlung Walther König.
Dean, R. T. (2018), The Oxford Handbook of Algorithmic Music, Oxford
University Press.
Dias, R. and Guedes, C. (2013), A contour-based jazz walking bass generator, Sound and Music Computing Conference 2013.
Dillon, S. and Jones, A. (2009), Exploring new kinds of relationships
using generative music-making software, Australasian Psychiatry
17(1‑suppl), S163--S166.
Dorin, A., McCabe, J., McCormack, J., Monro, G. and Whitelaw, M. (2012),
A framework for understanding generative art, Digital Creativity
23(3-4), 239--259.
Friberg, A., Schoonderwaldt, E. and Hedblad, A. (2011), Perceptual
ratings of musical parameters, Gemessene Interpretation-Computergestützte Aufführungsanalyse im Kreuzverhör der Disziplinen, Mainz: Schott, 237--253.
Ghisi, D. (2017), Music across music: towards a corpus-based, interactive computer-aided composition, PhD thesis, Paris 6.
Hannula, M., Suoranta, J. and Vadén, T. (2005), Artistic Research.
Theories, Methods and Practices.
Huron, D. B. (2006). Sweet Anticipation: Music and the Psychology of Expectation. MIT Press.
Jackendoff, R. and Lerdahl, F. (1981), Generative music theory and its
relation to psychology, Journal of Music Theory 25(1), 45--90.
Jacobs, L. (2015), Film Rhythm after Sound: Technology, Music, and Performance, University of California Press.
Jedwillat, L. and Nowack, N. (2020), A game with music or music with a game? About the video game karmaflow / Ein Spiel mit Musik oder Musik mit einem Spiel? Zum Computerspiel Karmaflow Наука телевидения
16(4), 85--108.
Jehan, T. (2000), Creating Music by Listening, PhD thesis, Massachusetts Institute of Technology.
Judge, J. (2018), The surprising thing about musical surprise,
Analysis (Oxford) 78(2), 225-234.
Léo, S. (2020), The Effect of Dynamic Music in Video Games: An Overview of Current Research. Uppsala University.
Levitin, D. J. (2006), This is your brain on music: The science of a human obsession, Penguin.
Loughran, R. and O’Neill, M. (2016), Generative music evaluation: why do we limit to ‘human’. Proceedings of the first Conference on Computer Simulation of Musical Creativity (CSMC 2016), Huddersfield,
UK.
Mäkelä, M., Nimkulrat, N., Dash, D. and Nsenga, F.-X. (2011), On
reflecting and making in artistic research, Journal of Research Practice 7(1).
Magnusson, T. and McLean, A. (2018), Performing with Patterns of Time. The Oxford Handbook of Algorithmic Music.
Margulis, E. H. (2007), Surprise and listening ahead: Analytic
engagements with musical tendencies, Music Theory Spectrum
29(2), 197-217.
Miles, S. A., Rosen, D. S. and Grzywacz, N. M. (2017), A statistical
analysis of the relationship between harmonic surprise and preference in
popular music, Frontiers in Human Neuroscience 11, 263.
Morley, P. (2010), On gospel, Abba and the death of the record: an
audience with Brian Eno, The Guardian.
Mudd, T., Holland, S. and Mulholland, P. (2019), Nonlinear dynamical
processes in musical interactions: Investigating the role of nonlinear dynamics in supporting surprise and exploration in interactions with digital musical instruments, International Journal of Human-Computer Studies128, 27-40.
Peerdeman, P. (2010), Sound and music in games, VU Amsterdam.
Plut, C. and Pasquier, P. (2020), Generative music in video games:
State of the art, challenges, and prospects, Entertainment Computing
33, 100337.
Reina, R. (2017), Applying Karnatic Rhythmical Techniques to Western
Music, Routledge.
Rumsey, F. (2015), Game audio: Generative music, emotions, and
realism, Journal of the Audio Engineering Society 63(4),
293-297.
Stefan, J. and Brown, A. R. (2015), Generative music video
composition: using automation to extend creative practice, Digital Creativity 26(2), 110--120.
Sterne, J. (2012), The Sound Studies Reader, Routledge.
Sweet, M. (2015), Writing Interactive Music for Video Games: A Composer’s Guide, Pearson Education.
Swift, S. (2019), FMOD, an Audio Engine for Video Games, Adapted for Theater.
audio and music in video games that reacts to changes in gameplay.
audio middleware
also called game audio engines; third-party software that sits between a game engine and the audio hardware to provide adaptive audio tools (e.g., FMOD, Wwise).
music that creates itself from an initial set of rules. Semi-generative music is a term not generally accepted that refers to a more limited, preset version of a generative system.
live coding (music)
live improvisation with software code to create sounds and music (e.g., reference projects: Sonic Pi, TidalCycles, SuperCollider).
state machine
a behavior model consisting of a finite number of states (finite-state machine, finite automaton); cf. automata theory. In video games, a way to create simulacra of autonomous characters or soundscapes.
system
A set of principles or procedures according to which something is done; an organized scheme or method. (Oxford English Dictionary)
The Aquatic Museum (TAM)
band, album, and app project my research is linked to.
Unity, Unreal
game engines, i.e., software frameworks that provide tools to create video games for different platforms. Unity and Unreal are the most widely used platforms, e.g., “over 50% of new mobile games are created in Unity”.
Copyright
Copyright and License Information
The video game creation software tools I listed (FMOD, Wwise, Unity, Unreal) can be downloaded and used for free up to a certain generated annual revenue. For games created in Unity, for example, there is a $100,000 threshold for the most recent twelve-month period.
For all other media content (e.g., Unity’s Platformer Microgame), read the license conditions carefully if you want to base a project on these assets.
Creative Commons License for This Report
For this report, some of the content I created is under an Attribution-NonCommercial-ShareAlike 4.0 license. This includes the written text on this website and the files from the download section, but not the videos, songs, and other media files related to The Aquatic Museum band and app project, including Astrid Rothaug’s beautiful hand-drawn illustrations. Also excluded—and this should go without saying—is all content that is simply linked or embedded on this website, such as a YouTube video. Just drop me an email if you are unsure.
You are free to:
Share — copy and redistribute the material in any medium or format
Adapt — remix, transform, and build upon the material
The licensor cannot revoke these freedoms as long as you follow the license terms.
Under the following terms:
Attribution — You must give appropriate credit, provide a link to the license, and indicate if changes were made. You may do so in any reasonable manner, but not in any way that suggests the licensor endorses you or your use.
NonCommercial — You may not use the material for commercial purposes.
ShareAlike — If you remix, transform, or build upon the material, you must distribute your contributions under the same license as the original.
No additional restrictions — You may not apply legal terms or technological measures that legally restrict others from doing anything the license permits.
Downloads
Please email me for a Dropbox link if you are interested in the FMOD files or other media examples. I’m not offering a download on these pages in order to save on bandwidth costs. Please visit the copyright section for information on which content falls under the Attribution-NonCommercial-ShareAlike 4.0 Creative Commons license.
Elements of Randomness and Surprise in Teaching Improvisation
Maastricht Academy of Music, Zuyd University of Applied Sciences Degree Programme: Master of Music
Advanced Teaching Skills Research Coach: Jo Didderen
Master Research Coach: Matthias Akeo Nowak, John Goldsby
2022
Laurent Peckels
Studieblad met tekenvoorbeelden: ogen, koppen en dieren, Michael Snijders, after Adriaen Collaert, 1610 - 1672 - Rijksmuseum - CC0 1.0
Subsections of Randomness and Surprise in Teaching Improvisation
Introduction
Practice Time
No app, software, or method alone will make someone a seasoned improviser. Opening the next pages with such a bold statement might seem unscientific—which it is—but there is consensus1 among jazz players and teachers that there is no way around lifelong experiences of interacting with fellow musicians. There is also agreement on the value of, in jazz lingo, woodshedding or shedding—i.e., time spent secluded and focused on improving and developing new improvisation skills. Methods and tools are created to help make these sessions more interesting, stimulating, and “productive”.2
Interaction is the key word, because what most play-alongs and software tools3 still lack today is even minimal interactivity. Until artificial intelligence reaches a convincing level of real-time interaction with improvising human musicians, we are stuck with currently existing4 technologies.
The purpose of this research is to critically review one element that is becoming increasingly prominent in music creation apps but hasn’t yet found its way into music learning apps and software on a large scale: randomness. As I will try to demonstrate, it is one small step on the path toward real interaction. How to use and add chance, aleatoric, serendipitous, and surprise features in jazz and pop exercises is the focus of this research, linking it to my Master’s reflection report Kings and Queens of Serendip in Part I on this website. I will illustrate the technical side using adaptive audio technologies from video games to create exercises with random elements.
Electronic tools and helpers have taken a prominent role in music education. I have been involved in and have also created different software projects, listed on this website in the reflection report section (PercussionTutor, FlamencoTutor, Interakt). These apps and software projects all use random features, but, with the exception of Interakt, they only scratch the surface of their potential.
My sources include research on the element of surprise5 in a pedagogical setting as well as on tools (digital and otherwise), e.g., the use of flash cards. Furthermore, I used different jazz bass and improvisation methods to select the exercises that seemed best suited to integrating elements of surprise (cf. research section), with feedback from my bass teachers Matthias Akeo Nowak, John Goldsby, and Jo Didderen.
The 10 lessons in the research part all refer to specific exercises related to improvisation in a jazz and pop music context. When necessary, the different learning objectives and pedagogical and didactic actions are listed for beginner, intermediate, and more advanced students.
Berliner, P. F. (1994). Thinking in Jazz: The Infinite Art of Improvisation. Chicago Studies in Ethnomusicology. ↩︎
How exactly one might define productivity in improvising remains an open question. ↩︎
As a reference to the ironic “actually existing capitalism”, as e.g. in Fisher, M. (2009). Capitalist Realism: Is There No Alternative? Zero Books. ↩︎
E.g., Fortin, C., Gonzalez, E., & Carle, S. (2017). When the element of surprise fosters learning. Pédagogie Collégiale Vol. 30, N° 3, Spring 2017. ↩︎
Vision on Teaching
“Why did I want to become a music teacher and what is important in my relationship with students?”
I’ll answer this question from the perspective of someone who has been teaching for 12 years now after a career change. I divide my time between teaching, performing, and studying.
First, two closely related factors have driven me to start making music and to explore it. The first motivation could be considered somewhat selfish, but it is counterbalanced by my second driver. I only started playing and practicing late in my teens. Until then, I had always viewed music from the listener’s perspective, and although I felt very passionate about it, I thought its practice aspects were too similar to sports, for which I didn’t have a particular passion. I was completely wrong, of course; music and sports couldn’t be more different.
I have also always had a knack for languages, so the first key moment came when my bass teacher, during our very first lessons at age sixteen, encouraged me to listen to as many different genres as possible without being judgmental and without having a particular goal. So, the first motivation was to make connections, to learn music, and to play music that was presented to me as a language—more mysterious than spoken languages and remaining elusive at its core even today.
This passion for music as a language and cultural heritage is linked to my motivation to share that knowledge, and it ties in with my teaching activities. Of course, one of the main objectives is to transmit craftsmanship, but this doesn’t happen in a vacuum separate from cultural practices (listening to and performing music, the music industry, musicians’ roles, etc.). I also think that because I started late in my teens—which can be an impediment to reaching embodiment and mastery—I can relate more to students struggling with certain aspects of music-making or adults picking up an instrument later in life. These exchanges remain a source of inspiration and pleasure for me and, I hope, for my students.
“Why the choice of this subject for pedagogical research?”
Over the last 10 years, I have co-created several educational music apps with Jérôme Goldschmidt, Jonathan Levi, and Alex Greenwood. While they are all centered around rhythm and use different web and game audio technologies, the random element was always more of a side note. This research is an opportunity for me, on the one hand, to explore a teaching approach incorporating surprise elements and, on the other hand, to apply my experience with multimedia projects to create a proof of concept, as the following pages intend to demonstrate.
Research Question
Random elements in the context of teaching improvisation
In a sea of music methods, tools, software, apps, journals, and books on how to teach and practice improvisation, this research will focus on a single element and one main question:
Can an element of chance or randomness, and by extension the surprise it creates, be an interesting and productive factor in teaching improvisation?
Sub-questions
To go beyond a purely theoretical or literature-based approach, I created a series of interactive exercises intended as proof of concept. I hope this will allow readers, students, and other teachers to explore the different concepts. Each exercise will be presented first, and then specific pedagogical questions will be introduced.
The main sub-question could be phrased as:
Can typical jazz improvisation exercises benefit from elements of randomness?
As with any software project, specific technical questions, not directly relevant to research on teaching, will still need to be considered. I will list the most important ones: time necessary to create the exercises, time investment versus expected results, required technical expertise, and interactive features of a web-based app.
Methodology
Sources
The title of this research refers to randomness and surprise. My MA research opens by referencing an article1 on the essence of musical surprises. Musical surprises take on many forms: rhythmic, harmonic, melodic, timbral, and so on. Musical surprises are generally considered to fall under the category of perceptual surprises, characterized2 as frustrating prior expectations.
In the field of cognitive and behavioral science, research has been undertaken into the potential of surprise to stimulate learning:
“Music ranks among the greatest human pleasures. It consistently engages the reward system, and converging evidence implies it exploits predictions to do so. Both prediction confirmations and errors are essential for understanding one’s environment, and music offers many of each as it manipulates interacting patterns across multiple timescales. Learning models suggest that a balance of these outcomes … optimizes the reduction of uncertainty to rewarding and pleasurable effect.”3
The focus will be on these random elements and their potential to create surprises when introduced into typical jazz improvisation exercises and their impact on learning. One might view these music exercises as an elaborate form of flashcards (for an example of research on the potential of flashcards4 in increasing science vocabulary, see the reference). I also note the use of games in training musical skills. For example, rhythm exercises seem promising, as studies5 point to the importance of using games for rhythmic6 training. It is also worth noting that randomized tests are widely used in research on musical parameters such as pitch and rhythm recognition, as in pitch memory research.7
Every exercise also concludes with a short overview of:
the main learning objectives
teaching material used (mostly jazz bass and improvisation methods)
comments
Technical Aspects
Guidelines used for the exercise creation:
easy to use
good-sounding despite mostly using samples
multitrack8 enabled (mutes for different instruments)
short loading times
compatible with mobile and desktop devices
multiple browser support
concise in their scope
Here is an example of a flamenco multitrack playback (note: you should see “Channels Playing” and “CPU,” and use the playback and mute buttons):
Technical Details
For a detailed overview of the technical aspects of creating the exercises, please refer to the methodology section of my MA research. All the exercises were created with free tools from the field of music and audio design in computer games (cf. sections on adaptive audio and music in video games).
This methodology continues on the next pages with further sources and methods I used to compile the improvisation exercises, reference a final assignment9 on “How to Teach Improvisation,” and list different music apps.
Judge, J. (2018). The surprising thing about musical surprise. Analysis (Oxford), 78, 225–234. ↩︎
Huron, D. B. (2006). Sweet Anticipation: Music and the Psychology of Expectation. MIT Press. Judge, J. (2018, p. 226) ↩︎
Gold, B. P., Pearce, M. T., Mas-Herrero, E., Dagher, A., & Zatorre, R. J. (2019). Predictability and uncertainty in the pleasure of music: a reward for learning? Journal of Neuroscience, 39, 9397–9409. ↩︎
Aronin, S., & Haynes-Smith, H. (2013). Increasing Science Vocabulary Using PowerPoint Flash Cards. Science scope (Washington, D.C.), 37, 33–36. ↩︎
Bégel, V., Loreto, I. D., Seilles, A., & Bella, S. D. (2017). Music Games: Potential Application and Considerations for Rhythmic Training. Frontiers in Human Neuroscience, 11. https://doi.org/10.3389/fnhum.2017.00273↩︎
Duffy, S., & Pearce, M. (2018). What makes rhythms hard to perform? An investigation using Steve Reich’s Clapping Music. PLOS ONE, 13, 1–33. https://doi.org/10.1371/journal.pone.0205847↩︎
Schellenberg, E. G., & Trehub, S. E. (2003). Good pitch memory is widespread. Psychological Science, 14, 262–266. ↩︎
The multitrack feature in a web browser is interesting in itself, but the focus here is on randomness. ↩︎
From a music pedagogy course at the Luxembourg Conservatory in 2008 (Teacher: Martin Bertemes). ↩︎
Subsections of Methodology
Improvisation Scope
Teaching improvisation is too large a subject and not the focus of this research, so I’ll only refer to my collection of documents and methods on how to teach improvisation and post the synthesis documents here.
Methods I’ll reference include:
Sikora, F. (2017). Neue Jazz-Harmonielehre: verstehen, hören, spielen; von der Theorie zur Improvisation. Schott Music.
Crook, H. (2001). How to Improvise. Advance Music.
Siron, J. (2004). La partition intérieure : jazz, musiques improvisées. Outre Mesure.
Levine, M. (1995). The Jazz Theory Book. Sher Music.
Goldsby, J. (2002). The Jazz Bass Book: Technique and Tradition. Backbeat Books.
Goldsby, J. (2009). Bowing Techniques for the Improvising Bassist. Jamey Aebersold Jazz.
van de Geyn, H. (2007). Comprehensive Bass Method For Jazz Players (Book 1 & 2).
Brown, R., & others. (1999). Ray Brown’s Bass Method. Hal Leonard Corporation.
Sabin, R. (2020). Progressive Jazz Double Bass Repertoire.
Balse, H. (2005). L’évolution du rôle de la contrebasse dans le jazz jusqu’en 1960. De Jimmy Blanton à Scott LaFaro. Université Paris 4 - La Sorbonne.
Sher, C. (1979). The improviser’s bass method: For electric and acoustic bass. Sher Music.
Sher, C., & Johnson, M. (1993). Concepts For Bass Soloing. Sher Music Company.
Galper, H. (2005). Forward Motion. Sher Music.
Ligon, B. (1996). Connecting chords with linear harmony. Hal Leonard Corporation.
Levine, M. (1989). The Jazz Piano Book. Sher Music.
Pratt, G., Henson, M., & Cargill, S. (1998). Aural awareness: Principles and practice. Oxford University Press. (relevant topics for improvisation)
Overview
The following is a plan for teaching improvisation that I designed for a final assignment in Martine Bertemes’ pedagogy course at the Luxembourg Conservatory:
Parameters Mindmap
I use this mindmap, which summarizes several jazz methods, as a study guide for students:
Compositional and Improvisational Techniques in Jazz
In this selection of music apps that I use regularly, only a few offer random functions. Those are most often found in creative rhythm apps and not in the ones designed for practicing. Some melody and harmony training music apps rely on randomly generated exercises to train pitch or chord hearing (e.g., EarMaster).
Ear Training Apps
EarMaster
It has random flashcard-like functions for training intervals, scales, chords, etc.
The author, Avi Bortnick, calls it a “self-muting compound time metronome.” It is one of the few apps with a random setting slider to mute certain notes.
“Time Guru is the only metronome with the ability to mute its sound at random, in patterns, or both, so that you can assess whether you tend to rush or drag. Time Guru periodically leaves you on your own so that you strengthen your own internal sense of time, rather than relying on the constant, rigid, external time keeping of a metronome. Sometimes the training wheels should come off! It is the ultimate tool for becoming a rock-steady time guru.” avibortnick.com/time-guru/
Tempo Advance
Tempo Advance by Frozen Ape is a classic with many options that don’t get in the way:
Elastic Drums is a drum-machine-like app with extensive sound-tweaking capabilities. Some random play functions have been added:
PercussionTutor
www.percussiontutor.com, my own creation: a library and practice app with rhythms from Cuba, Puerto Rico, the Dominican Republic, Peru, Brazil, and West Africa, expanding to other cultures (e.g., Flamenco). The Batá section contains a random dialogue between the iyá and itótele drums.
Apple
GarageBand, etc., but these tend to be distracting if a timekeeper is all that’s needed.
Play Along
These apps are useful for creating automated backing tracks based on chords. Band-in-a-Box is one of the oldest in that genre, but it’s expensive and only works on PC and Mac. iReal Pro and especially SessionBand are cheaper and often more useful for designing short exercises on scales, etc.
Band-in-a-Box
Automatic chord and melody generation was introduced in 2014:
Impro-Visor (short for “Improvisation Advisor”) is a music notation program designed to help jazz musicians compose and hear solos similar to ones that might be improvised.
A free academic software project with auto-generated rhythm-section accompaniment.
SessionBand offers a great choice of styles recorded by professionals. It sounds more authentic than iReal, and the only negative point would be a limited choice of chord voicings.
Soundslice
The soundslice.com technology is used on many websites and offers state-of-the-art tools for combining audio, video, and sheet music:
NomadPlay
NomadPlay is a French company specializing in play-along recordings of classical pieces. The website and app likely use the soundslice technology.
Music Creation
BandLab
BandLab is a full-fledged DAW that can be used on iOS, Android, and PC/Mac in a browser:
TextMusic
TextMusic is an app based on Jianpu numbered musical notation. All musical elements are written in plain text. Compositions or scale exercises can be quickly designed and exported to standard MIDI.
Sight Singing on iOS and Android records and evaluates your singing.
Research
Subsections of Research
Scales and Arpeggios
This part is dedicated to aspects of scale practice and usage in improvisation.
Subsections of Melodic Material
Major and Minor Scales
Major and Minor Scale Practice and Its Shortcomings
The adapted exercise with random chord changes (cf. end of section) is an exercise for beginners that will help them to:
learn minor and major scales
internalize the weight and direction of each scale
understand and hear the relationship between relative major and minor
experience the weight of the lowest bass note, which changes the perceived harmony (minor or major)
The major scale and its relative natural minor scale are a minor third apart and share the same notes. A typical scale practice exercise uses a G major or an E natural minor scale in one position up to the octave, ascending and descending.
In this situation, although the repeated G in bar 3 is quite common, it is not a very musical way to practice scales over a backing track because chord tones in the second half will not fall on strong beats. In the last bar, the F# is played before going back to G, which smooths out the line. For the E minor scale, the final E is doubled, which breaks the flow.
Furthermore, although the same notes are to be played in the two scales, no connection is revealed to the student. The following exercise is an adapted version where the flow is maintained throughout and the two shortcomings of the base exercise are addressed:
chord tones now fall on strong beats in the descending scale
the two scales are linked visually and in terms of practice
the exercise becomes musically circular
Here is a seven-bar circular version of the same exercise. The harmony starts to blur, with the lowest note influencing the perception of minor/major. This exercise could also start on E:
The accompaniment track with guitar and piano either plays a G chord at a slow tempo (80 bpm):
or an Em7(add 11) voicing:
Learning Objectives with Randomized Chords
This is the randomized exercise. Every two bars, either an Em7(add 11) or a G will be played on guitar and piano. The main objective for a beginner will be to use the correct fingering and aim for legato playing at a slow tempo. A more advanced student can focus attention on the sound and emotional effect produced by randomly changing chords.
Continuous Scale Exercise
Altered and Octatonic Scale over iiø7-V7-i and ii-V7-I
Learning Objectives for the Continuous Scale Exercise
Several standard jazz improvisation methods1 list the continuous scale exercise as one of the main exercises to master. It can be adapted for beginners and more advanced students. The example above only uses two chord progressions with an altered chord resolution going to a minor or major chord. The preceding ii will give away the continuation of the exercise and could be left out.
Randomized Chord Progressions
When the two progressions are randomized, the student (intermediate to advanced in this case, especially if played in thumb position) has to make minor adjustments in bar three when resolving. This helps students rely on their ears to anticipate the resolution and trains them to differentiate altered chords and their respective resolutions.
This exercise can easily be extended, and the evaluation and learning points can be gradually integrated and checked by adding more chord progressions or more difficult ones. The student can use the same exercise to train at home and during class.
E.g., Levine, M. (1995). The Jazz Theory Book. Sher Music. ↩︎
Pentatonic Exercise
Strength of the Pentatonic Scale
Pentatonic scales are used extensively in many different musical styles and have been widely adopted in jazz. Pentatonic scales have a strong independent sound and “it is very easy to perceive them as bitonal elements on top of the harmony” (van de Geyn, 2011, p. 8).
In an instructional video from the 1980s, John Scofield1 discusses his approach to pentatonic scales. The following table summarizes the available inside-sounding options2 according to chord type:
Chord Type
HVG
HVG
HVG
Scofield
CMaj7
A-
E-
B-
C, A-, D, Ddom penta, B-
C-7
G-
C-
C-, G-, D-
C-Maj7
D-
C7(9 13)
A-
C, Cdom, Ddom, G-
C7alt
Eb-
Eb-
C7ø7
F-
Eb-
Ab dom
Single Pentatonic Major Scale Over Changing Chords
The following is an exercise I designed for intermediate-level students that helps them perform pentatonic major (and their relative minor) scales all over the neck on all four strings in different positions. It’s built around the chords of “Sunny”3, a tune with a melody based solely on minor pentatonic notes but with changing chords.
In the following version in E minor, adhering strictly to the E minor/G major pentatonic scale will create tension on certain chords, but overall strong melodic motion will result. In “The Serious Jazz Book II,” B. Finnerty4 lists similar exercises with arpeggios or different cells.
Here is the exercise with piano and bass:
Here is the exercise as I uploaded it to the MuseScore website, with advanced MIDI playback (but no chord playback):
Scofield, J. (1983). John Scofield on improvisation. ↩︎
Scofield also refers to a “dominant pentatonic scale”: C D E G Bb. ↩︎
Finnerty, B. (2008). The Serious Jazz Book II. Sher Music. Cf. “Part 2 - An Integral Part Of Seven Diatonic Scales And Their Modes”. ↩︎
Groove
Groove, Rhythm, Time Feel
This part is dedicated to aspects of rhythmic accuracy, groove, and performing complex rhythms. Time is most likely the key element to master for every jazz and pop musician, but it is also one of the most elusive parameters. Pianist Fred Hersch, referring to a performance, puts it this way:
“There should be ten, fifteen different kinds of time. There’s a kind of time that has an edge on it for a while and then lays back for a while. Sometimes it rolls over the bar, and sometimes it sits more on the beats. That’s what makes it interesting. you can set a metronome here and, by playing with an edge of playing behind it or right at the center, you can get all kinds of different feelings. That’s what makes it come alive. People are human, and rhythmic energy has an ebb and flow.”1
Most instrument methods or books on jazz improvisation contain specific chapters on rhythm, but very few are entirely dedicated to purely rhythmic aspects, with the exceptions of, for example, a book2 by drummer Bob Moses or a method3 by drummer Billy Martin.
A few words of caution on working with metronomes and machines appear in the following video4 by Richie Beirach on developing a better sense of time:
“Listen to great examples … check Out Speak no Evil, listen to Elvin Jones, Herbie, Freddie, Wayne, Ron Carter”
“Forget about the metronome”
“Metronomes are suicide for the Jazz musicians”
“Metronomes are good for kids because when you are practicing you are lonely and scared and it helps to feel not so lonely”
“Perfect time doesn’t help you with your jazz rhythm.”
“Is a click track great for pop music? Yeah, it keeps everybody together. Pop music can be creative, but it is not improvised; it is a whole different genre. I believe in the click track too.”
Berliner, P.F. (1994). Thinking in Jazz: The Infinite Art of Improvisation. Chicago Studies in Ethnomusicology. ↩︎
Moses, B. (1984). Drum Wisdom. Modern Drummer Publications (republished in 2022 as a Kindle edition). ↩︎
Martin, B. (2006). Riddim: Claves of African Origin (D. Thress, Ed.). Alfred Music. ↩︎
Beirach, R. (2021). Richie Beirach on developing a better sense of time. https://vimeo.com/539936515/5c86756b12. (These wise words might serve as a natural antidote, but it is still advisable to get your shots.) ↩︎
Subsections of Groove
Time Feel
Breaks and Drum Fills
In pop music, drum fills are often played at specific points in the cycle—for example, every 4 or 8 bars—and are often improvised. The following exercise will help you keep time during the fill. The first example uses simple in-time fills with no rests. The learning objectives in this case can be loosely defined—for example, use an easy, well-mastered exercise to play along and keep time.
Exercise 01: Fills
Note that the 3-bar groove consists of 3 different drum groove alternatives, and the 5 fills are played randomly every 4 bars. The groove is inspired by “Kissing My Love”1 by Bill Withers.
Exercise 02: 2-Beat Rests
Exercise 2 is more challenging, leaving two beats of rest in bar 4.
Exercise 03: Start With Rest During Break
Exercise 3 reverses the concept of ex. 2 and starts random breaks with two beats of rest.
Exercise 04: Full Bar of Rest
Exercise 4 stops right before beat 4 of bar 3 and leaves a full bar of rest. This exercise can now be extended to two or even three bars of rest.
Random Mix of Rests and Fills With Count-In
The next iteration would be very difficult to recreate in some DAWs, even Ableton (especially if non-metered audio files were used). It leverages the adaptive audio features by creating a mix of all preceding exercises. Now, for every cycle of four bars, the following happens at random:
one of 4 drum patterns is picked
one of 5 fills is selected
one of 5 exercises is selected
Behind the Scenes
On the technical side, the same short drum loops are used in all the preceding examples and then recombined, which multiplies the number of possibilities from simple base material. Please refer to the chapter from the main thesis for further technical details and the problems with creating perfect loops and testing all possibilities.
The Time Guru metronome (see methodology section) uses a random slider with a percentage setting to select the probability of drum hits being played. A gradual fade option is added to this “random mute slider”:
Many DAWs and plugins offer functions to set playback probabilities, although as a standard feature in some DAWs, this is a recent1 development.
Drum Kit Mutes
The next exercise applies this idea to a drum kit by randomly fading out certain parts of the drum kit, limited for the sake of this example to bass drum, snare, and hi-hat. Although fades happen randomly, the exercise was constructed so that there are no long blanks with every element of the kit being muted. This silence setting is one key element in finding the right difficulty level for this exercise.
The objectives can be varied—for example, as in the previous exercise, use a simple exercise and play along without losing time, or for more advanced players, increase the fade times and play more advanced improvisations or exercises.
A chance setting for notes was only introduced in Ableton Live 11 in February 2021. ↩︎
Rhythmical Displacements
Melodic Displacements
The following example uses 4 rhythmic displacements (eighth note, quarter note, and sixteenth note). These rhythmic devices are used extensively in composition1 and improvisation. This is a basic example, though only the eighth-note and quarter-note displacements might be appropriate for beginners.
Randomized Displacement With Different Exercise Options
Options a, b, c, and d are played randomly, with a 15% chance setting for option d, which can be considered the most difficult performance-wise.
Also note that:
the drum groove is 4 bars long and randomly plays 3 different variations
there are 2 possible drum fills, one with a 1-bar drum silence
the guitar part is 8 bars long with different rhythmic variations on E7#9.
This is a good example of the potential for spontaneous variation by using short, similar loops of different lengths.
Based on this simple example, many variations can be created. By muting the drums, the guitar, or both, the playback becomes a time-keeping exercise. The quantize setting can be randomized, and more advanced subdivisions become imaginable, such as transformations2 into other beat subdivisions (triplets, quintuplets, septuplets) and their respective displacements.
Cf. rhythmical sangatis, p. 61, in Reina, R. (2017). Applying Karnatic Rhythmical Techniques to Western Music. Routledge. ↩︎
Swing Ratios
Swing ratios are commonly used in music notation programs to emulate swing feel. Expressed as a percentage, the ratio refers1 to consecutive eighth (or sixteenth) notes performed as long-short patterns. Microtiming2 studies have shown that these ratios vary widely depending on style, tempo, musicians’ affinities, and cultural background.
In a fascinating article3 on the impact hip-hop producer J Dilla had on live drum kit performance and pedagogy, Daniel Stadnicki analyzes “how the so-called ‘Dilla-feel’ is emulated by drummers and rhythm section players through a range of informal learning strategies and extended techniques, which include practices of online teaching and learning.”
J Dilla used4 the Akai MPC music workstation, which offers sampling and sequencing capabilities. Swing ratios can be set as a percentage. In DAWs, different names are used—for example, grooves in Ableton, groove quantize presets in Cubase, or groove templates in Logic. Another way to express this swing ratio is by using tuplet notation. This approach certainly doesn’t do justice to the different shades of microtiming, but it has the merit of offering a clear approach to these types of grooves.
The cowbell from the Afro-Peruvian Festejo rhythm is an example of an original swing feel between binary and ternary time:
Festejo with cajita, quijada, congas, bongo, and cajon:
Feel the Swing
I took inspiration for the next exercise from a 16-year-old beginner student who, while working on shuffle songs, asked me why the groove of some hip-hop tunes felt5 so different from, for example, blues shuffle songs. In the next example, the drum groove over the first 2 bars is straight, which corresponds to a 50% swing ratio or no swing at all; the two notes on the hi-hat have the same length. The swing feel on the hi-hat in bars 4 and 5 equals a 66.6% swing ratio, or a perfect triplet division, with the first note taking up 2/3 of the beat division and the second 1/3. A 60% swing ratio would correspond to 3 and 2 quintuplets (bars 5 and 6). A septuplet division (5 and 3 septuplets) yields approximately a 57% swing ratio.
Guess the Ratio
When randomizing this example with two-bar fragments, different exercises are conceivable for beginner to intermediate students:
try to identify whether a change happens (a two-bar sequence might be played twice or more; 25% chance setting)
identify the ratio
play along with only quarter notes
try to emulate the feeling by adapting your bass line to the drum groove
Friberg, A., & Sundström, A. (2002). Swing Ratios and Ensemble Timing in Jazz Performance: Evidence for a Common Rhythmic Pattern. Music Perception, 19, 333–349. ↩︎
Collier, G. L., & Collier, J. L. (1996). Microrhythms in jazz: A Review of Papers. Annual Review of Jazz Studies, 8, 463–483. ↩︎
Stadnicki, D. A. (2017). Play like Jay: Pedagogies of drum kit performance after J Dilla. Journal of Popular Music Education, 1, 253–280. ↩︎
There is a wide range of techniques and approaches to intonation exercises on the double bass. Some rely on play-along software, reference tones, or backing tracks.
Relying on a drone-like instrument or synthesizer sound can help you feel the distance between notes. In Karnatic and Hindustani music, a drone—tambura (or tanpura in North India)—is used to lay the groundwork1 for the melodic instruments:
Many different tunings exist, and the one using the root note, the fifth, and the octave is widely used.
Here is an example of a drone on D (with 440 Hz reference tuning). The fifth A is tuned to a pure 3:2 interval ratio.
Randomized change of root note and fifth with a drone
The following exercise randomly changes the root note, either D or Eb, with added perfect fifths and octaves, and also randomizes when this change occurs.
Beginner students can make use of this exercise by:
focusing only on the root change from D to Eb
playing D and Eb in all ranges across the neck
using bowing or pizzicato
adding a fifth or an octave
experimenting with different fingerings or same-finger shifts
Typical Turnarounds on Rhythm Changes or Blues Form
Here are four patterns (a, b, c, d) used in bars 9–12 of a typical F jazz blues:
Randomized Turnaround Options
The following playback will select one of the four options by chance. The 8-bar piano part is underlaid with a 6-bar drum groove grouped in sections of 2 bars each, with loops randomly picked from 4 options. This creates additional layers of variation.
Objectives
Here again, the basic principle is an audio flashcard. Students of different levels can focus on specific things:
recognize the changes as quickly as possible
sing along
learn one option at a time
master all the changes and force yourself to pick one in advance and play it even when recognizing another option. This is an interesting exercise and will help students think less rigidly about harmonic forms—for example, a student plays option d while hearing c, which functions as a subV.
Rhythm Changes
Chord Change Variations Over a Rhythm Changes Form in the B Section
Many jazz tunes use the rhythm changes harmonic progression as a base template. The B section is a playground for different1 reharmonizations.
Here is an audio example of the three options, preceded by four bars over Bb6:
Randomized Versions
This is a variation of the blues turnaround memorization exercise. Each option (a, b, or c) is played twice. This allows the student to hear the progression once and then repeat it, or at a later stage, play each option two times.
Thanks to WDR bass player and teacher John Goldsby for option c. ↩︎
Conclusion
The preceding pages offer only a glimpse into the potential of using random elements in teaching improvisation and bridging the gap between human interaction and playing with a machine. Each of the topics (rhythm, harmony, etc.) could be investigated separately and would require focus groups and extensive testing.
The rhythm exercises, with random elements very similar to those found in video games (which use random events at the core of gameplay), seem the most promising with regard to recent research.1 Other studies point to the potential of using games for rhythmic training.2
I created some of these exercises for my own practice during the two years of MA study and devised other exercises for beginning and intermediate students. While the novelty effect was real and the ease of use and learning objectives were well understood, the main weak point is the amount of time required to create each exercise. The creation steps are as follows, not counting the initial development required to arrive at a working system:
design of an exercise
chord and melody creation in music editing software (Dorico)
DAW editing (Pro Tools and Ableton in this case)
DAW mixing
export to FMOD (game audio engine)
web page design and coding
This amount of work would increase when recording real musicians. The input/output ratio needs to be further critically investigated. The system is also well suited to a collaborative and remote working approach.
Inspiration for Teachers
A positive and very rewarding aspect of this research was the challenge and enjoyment of creating exercises that use these random elements. Reviewing and inventing typical jazz improvisation exercises—from simple scale and arpeggio practice to more complex upper structure concepts—was rewarding and shed new light on exercises that might often seem repetitive or even stale. Play-along exercises could be made more engaging even through minimal random variations.
Bégel, V., Loreto, I. D., Seilles, A., & Bella, S. D. (2017). Music Games: Potential Application and Considerations for Rhythmic Training. Frontiers in Human Neuroscience, 11. https://doi.org/10.3389/fnhum.2017.00273↩︎
Duffy, S., & Pearce, M. (2018). What makes rhythms hard to perform? An investigation using Steve Reich’s Clapping Music. PLOS ONE, 13, 1–33. https://doi.org/10.1371/journal.pone.0205847↩︎
Bibliography
Aronin, S. and Haynes-Smith, H. (2013), Increasing Science Vocabulary Using PowerPoint Flash Cards, Science scope (Washington, D.C.) 37(3), 33–36.
Bégel, V., Loreto, I. D., Seilles, A. and Bella, S. D. (2017), Music Games: Potential Application and Considerations for Rhythmic Training, Frontiers in Human Neuroscience 11.
Beirach, R. (2021), Richie Beirach on developing a better sense of time.
Berliner, P. F. (1994), Thinking in Jazz: The Infinite Art of Improvisation, Chicago Studies in Ethnomusicology.
Brown, R. and others (1999), Ray Brown’s Bass Method, Hal Leonard Corporation.
Cheung, V. K., Harrison, P. M., Meyer, L., Pearce, M. T., Haynes, J.-D. and Koelsch, S. (2019), Uncertainty and surprise jointly predict musical pleasure and amygdala, hippocampus, and auditory cortex activity, Current Biology 29(23), 4084–4092.
Crook, H. (2001), How to Improvise, Advance Music.
Finnerty, B. (2008), The Serious Jazz Book II, Sher Music.
Fisher, M. (2009), Capitalist Realism: Is There No Alternative?, Zero Books.
Fortin, C., Gonzalez, E. and Carle, S. (2017), When the element of surprise fosters learning, Pédagogie Collégiale vol. 30, n° 3, Spring 2017.
Friberg, A. and Sundström, A. (2002), Swing Ratios and Ensemble Timing in Jazz Performance: Evidence for a Common Rhythmic Pattern, Music Perception 19, 333-349.
Galper, H. (2005), Forward Motion, Sher Music.
van de Geyn, H. (2007), Comprehensive Bass Method For Jazz Players (Book 1).
van de Geyn, H. (2007), Comprehensive Bass Method For Jazz Players (Book 2).
van de Geyn, H. (2011), 12 Jazz Etudes.
Gold, B. P., Pearce, M. T., Mas-Herrero, E., Dagher, A. and Zatorre, R. J. (2019), Predictability and uncertainty in the pleasure of music: a reward for learning?, Journal of Neuroscience 39(47), 9397–9409.
Goldsby, J. (2002), The Jazz Bass Book: Technique and Tradition, Backbeat Books.
de Graaf, D. (2017), Beyond Borders. Broadening the Artistic Palette of (Composing) Improvisers in Jazz.
Huron, D. B. (2006), Sweet Anticipation: Music and the Psychology of Expectation, MIT Press, Cambridge, Mass.
Judge, J. (2018), The surprising thing about musical surprise, Analysis (Oxford) 78(2), 225–234.
Levine, M. (1989), The Jazz Piano Book, Sher Music.
Levine, M. (1995), The Jazz Theory Book, Sher Music.
Ligon, B. (1996), Connecting chords with linear harmony, Hal Leonard Corporation.
Margulis, E. H. (2007), Surprise and listening ahead: Analytic engagements with musical tendencies, Music Theory Spectrum 29(2), 197–217.
Martin, B. (2006), Riddim: Claves of African Origin, D. Thress (ed.), Alfred Music.
Moses, B. (1984), Drum Wisdom, Modern Drummer Publications.
Sabin, R. (2020), Progressive Jazz Double Bass Repertoire.
