Adaptive Audio

Adaptive Audio Overview

Adaptive Audio

To illustrate the creation of the TAM app with its two distinct sections—a semi-generative opening sequence and a traditional song structure whose elements are reshaped and transformed—the methodology section gives an overview of personal examples of adaptive audio tools and creations in video game engines with a focus on their random features. The bibliography section references recent books on composing and creating music and sound for video games.

Different techniques exist to compose adaptive music. A distinction is generally made between:

Horizontal Resequencing:

“Horizontal resequencing is a method of interactive composition where the music is dynamically pieced together based on the actions of the player. For example, when the music is playing underneath the gameplay, it may reach a decision point where it could either go to a new section of music or repeat the previous section depending on the player’s actions.”¹

Vertical Remixing:

“Vertical remixing is an interactive composition technique in which layers of music are added or taken away to create levels of intensity and emotion. This method of scoring is useful when the composer needs multiple quick changes to intensify the score, where harmonic changes based on gameplay are not as important.”²

As FMOD is one of the main tools I used for my research and the TAM app, the methodology section as well as the advanced teaching skills examples provide a detailed overview of these tools.

These adaptive audio tools can be analyzed and placed into a wider context of algorithmic music (cf. next pages), into which generative and semi-generative music falls, with randomness being a key element.

Sweet, M. (2015). Writing Interactive Music for Video Games: A Composer’s Guide. Pearson Education, p. 143. ↩︎
ibid. p. 155 ↩︎

Sound and Effect Design in Video Games

For The Aquatic Museum app, I use the FMOD audio middleware combined with the Unity game engine. While this research is not about the technical aspects, it is important for the methodology section to give an overview of the tools used in my creative process. I use the same tools in my advanced teaching skills research by taking advantage of software resembling traditional DAWs but designed for video games. However, differences from traditional audio workstations do appear in the media creation process. This research also focuses on one specific element or feature used in adaptive audio and other music creation tools: randomness or aleatory features.

Composing music for games requires a different approach from the more linear thinking used in film music.

For the FMOD creators, their studio package is

“an end-to-end solution for adding sound and music to any game. Build adaptive audio using the FMOD Studio and play it in-game using the FMOD Engine.”¹

Three market segments are listed: sound designers, programmers, and producers. The idea is to facilitate the video game audio and music creation process and the division of tasks between audio designers and coders.

Adaptive Ambient FX Audio Example

Excellent tutorials can be found on YouTube², and for book recommendations, please refer to the Game Music and Audio Book References section. The simplest entry into the technical aspects comes from a personal example from the 2021 Global Game Jam in Luxembourg. The setting is a spaceship—yes, exactly as you’d imagine one: dark, damp, menacing, and claustrophobia-inducing. Furthermore, dear reader, note how stressful the absence of a stop button is. The ship’s soundscape was created with different sound samples, some of them hand-recorded with voice, paper, and tube accessories.

Drag and release the slider to morph from a lighter to a darker ambiance.

Please wait... Loading...

Here is a video of the actual project in FMOD Studio, the main design tool of the suite. Two important things to note: first, I’m not using the timeline in this example.³ Second, the slider triggers different sound objects (called instruments in FMOD) when moved by the user or triggered by code, and some audio samples are triggered asynchronously. Automation lanes, shown at the end of the clip, control volume as well as the parameters of the different instruments.

The audio designer can now collaborate with the coding team to implement this single atmospheric ambiance. This is a simple example with a single parameter, so the amount of coding is limited, but even for simple audio tasks, the work required can grow exponentially despite the general tendency of software tools to rely more on visual design and less on coding.

▶ Video on YouTube: https://youtu.be/l4_2NV4a_8o

www.fmod.com/studio ↩︎
on YouTube: official FMOD channel and Scott Game Sounds ↩︎
Notice the visual similarities to DAWs (e.g., Cubase, Ableton), but with a fundamentally different adaptive audio logic. ↩︎

Music in Video Games

Adaptive audio tools provide advanced music creation possibilities that differ from the features found in more traditional DAWs. Here is the theme for one of the monster encounters (cf. sound in video games). It is a collage I created from a Donizetti opera¹ recorded in 1911, Omnisphere synth sounds, and the preceding page’s soundscape. Some sonic elements as well as the volume balance are adapted dynamically during gameplay.

Adaptive Audio for a Platform Game

For the next example, I used only Tempest, a drum machine designed by Roger Linn, to create a groove with different drum and synth layers. The main groove uses the Batá rhythm Olokun as a template.

Drag and release the slider to start the percussion playback on 1 and increase the volume on 2 and 3. Slide back to 0 to mute the percussion instruments. Note that the percussion loop will only start every 2 bars due to its quantization settings. The only random element is a 3-bar section of the main loop that has 2 alternatives. Channel count increases to 2 when the Batá drums are playing.

Please wait... Loading...

I simplified this example and created 3 videos to illustrate the different adaptive layers and random features. An easy way to imagine the use of FMOD is to think of having a scaled-down version of Ableton Live that can be hidden in a game, an app, or (like here) on a website:

Complex Example with Changing Song Structure and Randomly Triggered Percussive Instruments

Similarities and Differences with DAWs

These examples only scratch the surface of what is possible in an audio game engine. To understand the full potential of FMOD, imagine a compact DAW hidden in a game or multimedia installation. Real-time effects and interactive mixing further extend the possibilities for The Aquatic Museum app.

Una furtiva lagrima (L’elisir d’amore, Act 2) - Gaetano Donizetti - performed by Enrico Caruso (public domain). ↩︎

Types of Sounds

As the preceding examples show, the design and creation process of audio and music for games is based on traditional composing, recording, and creation practices, but the assembly and delivery of the media assets require other tools and an adapted kind of thinking and planning.

A common classification of sounds in games, adapted from film, is as follows:

Diegetic Sounds

Diegetic sound is heard by both the characters and audience. Also called ’literal sound’ or ‘actual sound’.

Sound effects: car engine, collisions, explosions

Music: car radio, band playing on screen

Vocals: dialogue, voices, crowds

Non-Diegetic Sounds

Non-diegetic sound is represented as coming from a source outside the story space, i.e., its source is neither visible on the screen nor has been implied to be present in the action. Also called ’non-literal sound’ or ‘commentary sound’. wiki

Sound effects: off-screen sound effects, e.g., footsteps

Music: background music, tunes, themes (boss theme etc.)

Vocals: voice-over, narration