The Aquatic Museum (TAM): Collective & Album & App

Interactive Ever-Changing Music Video Clip?

The following chapter details the iterative creation process of an app specially designed for a new project, The Aquatic Museum (TAM)—the name given to a band project, an album, and an app. The sections below lay out the project and the preproduction steps of the album and app. I’ll then take an excursion into other musical creations using chance and surprise elements, some of which have found their way into the TAM app, while others were deemed, in a democratic artistic collective process, too experimental. I’ll conclude with finalized examples of audio interactions in the TAM app. Album and app were released in 2023; the last section of this page presents the release.

The Aquatic Museum Project Presentation

Release: Album, App, and Website

After this thesis was submitted, The Aquatic Museum moved from prototype to public release in spring 2023. The three outlets complement each other: the album fixes one version of each song, the app lets the title track keep changing, and the website ties everything together.

Album

The album The Aquatic Museum (11 tracks, about 39 minutes) was released at the end of May 2023 on CD and on streaming platforms (Spotify, Apple Music) with composiitons by Claire Parsons and myself. The collective, which I co-founded in 2021 with Claire Parsons and Nicole Miller, recorded the album with Claire Parsons (vocals), Maia Frankowski (violin), Nicole Miller (viola), Annemie Osborne (cello), Eran Har Even (guitar), Jérôme Klein (drums), and me (electric and double bass), joined by the Belgian Q-Some Big Band for the big band arrangements prepared during the score layout and orchestration phase. Charles Stoltz recorded, produced, and mixed the album at Holtz Sound, Markus Schneider mastered it at Skywalk, and Astrid Rothaug created the artwork.

Four singles accompanied the release: “Entrance,” “Large Pleasure Watercraft,” “Trash Tub” (a live recording with Q-Some), and “Trapped Air Bubbles,” the latter featuring underwater images by technical diver and photographer Audrey Cudel. Visual artist Jeanne Held contributed music videos. “Entrance” opens the album and is also the song used to demonstrate the audio player in Why This Format?:

▶ Video on YouTube: https://youtu.be/NTksgpozl6A

App

The free The Aquatic Museum app has been available on the App Store since 7 April 2023. It runs on iPhone and iPad (iOS/iPadOS 15 or later), on Macs with Apple silicon, and on Apple Vision Pro, while this website hosts web-based examples of the same audio technology. The store description summarizes the concept developed in this chapter:

“The Aquatic Museum App features the multilayered album track ‘Souvenir Shop (Download the App)’ that will transport you into an intricate and imaginative aquatic world hand-drawn by Astrid Rothaug. Take a journey through various museum chambers as you’re guided by visual and musical cues, courtesy of the original Aquatic Museum Band, the Mechelen-based Q Some Big Band, and Valencia-based flamenco percussionist Sergio Martínez (ES).”

The subtitle “(Download the App)” turns the album track into a pointer: on the album, Souvenir Shop is one fixed recording, while in the app the same song becomes the ever-changing music video described on the following pages. The credits also connect the app to my earlier projects: flamenco percussionist Sergio Martínez co-created the Flamenco Tutor app. As during the research, development remained iterative after the launch: version 1.2, released two weeks later, added further interactions, and later updates reduced the download size and refined the user interface.

Website

The project website, theaquaticmuseum.com, gathers the album with streaming links, the app with a video preview, the singles and their music videos, and the band with all collaborators. Like this reflection report, it is built with Hugo.

Evolving Music Video

Film Rhythm After Sound

TAM singer Claire Parsons had worked with Vienna-based Astrid Rothaug on a previous music clip for her In Geometry album. For TAM, we wanted to give Astrid as much freedom as possible in creating a three-minute short film with hand-drawn animated frames and a story loosely spun around an Aquatic Museum theme. For inspiration during our first meetings, we provided Astrid with demo and pre-production material and settled on a simple pop music tune (with some twists), co-written¹ by Claire and me, The Souvenir Shop—a constant in every museum visit.

The process² of creating a clip for music or music for a short video was never clearly laid out. The final song choice came later during the album writing and creation process. Because we had the unique opportunity to work with an artist who spent weeks creating frame-by-frame hand-drawn animations in high resolution, we discarded several other music app ideas based more on traditional video game mechanics. Hand-drawn graphics are rare in video games (cf. literature review chapter) because the translation from hand-drawn artwork into computer-animated sprites can lead to aesthetic clashes. Hand-drawn paper-style flat graphics don’t lend themselves easily to an interactive graphic style. Also, to respect Astrid’s vision and to stay true to the original, we maintained the clip’s linearity.

1. Create a three-minute repeatable experience that imitates the music video format but adds a new musical surprise twist. The animated short with few user interaction opportunities and a linear storyline is the backdrop to The Souvenir Shop tune.
2. Take away the power from the user who expects to be spoiled with features (that was the most daring choice). Of course, this was also done to simplify our work, but above all to bring the main focus onto music without compromising audio quality, number of loops, or sonic experiments. Offering no interactive options is still on the table, and the app would then produce a randomly shuffled soundtrack. Music and audio in video games is a central atmospheric mood element, but too often in indie games musical considerations come very late in the development process. Furthermore, sound performance and quality have to make room for graphic performance and user interaction priorities.
3. Finally, it is key to be able to release it in many different formats (iOS, Android, web—as shown here—desktop computer, YouTube clip, and others; cf. Brian Eno’s generative apps).

Opening Sequence

▶ Video on YouTube: https://youtu.be/xlkv5853w9w

Metamorphoses

▶ Video on YouTube: https://youtu.be/Bqiw5xK4oBQ

Color Codes

Graphical user interface design is a challenging domain of human–computer interaction. How do you guide the user in a game or software? For TAM, I chose to use color to indicate possible interactions: Astrid’s footage is kept in black and white³, and color is used sparingly.

In the TAM app, the different possible interactions that will influence the music are indicated by color spots and surfaces. The doorbell from clip 1 and the transforming whale from clip 2 are examples among other larger and smaller color indications:

The tree example shows multiple user interactions where different trees trigger random voice tracks. Later in the clip, a chessboard appears when the Souvenir Shop tune reaches the solo section. A press by the user will trigger another instrument playing a solo, and a press on the same spot while a certain solo instrument is playing will slowly fade into another solo version. As an Easter egg, some instrumentalists agreed to play a funny or even poorly executed version of their solo that will only rarely be triggered (1 out of 1000 times). Interaction times are limited to a few seconds, the solo section is short, and the user has no built-in recording option.

Basic touchscreen single-finger interactions are supported, which also translate to mouse navigation: touch (single, double), drag, slide. Some user actions will have visual feedback cues, but audio is always front and center. The app is semi-generative⁴ and not generative in the sense of the original definition because it only uses the loop and soundscape material that we recorded and that fits the song.

Score Layout and Orchestration

Simple Structure

Music and audio in video games is certainly a central atmospheric mood element, but too often in indie games musical considerations come very late in the development process. Furthermore, sound performance¹ and quality have to make room for graphic performance and user interaction priorities. As a new take on the music video clip, the TAM app takes the opposite perspective and puts music first. The animated short with few user interactions and a linear storyline is the backdrop to The Souvenir Shop tune.

The first step consisted of laying out a lead sheet for the rhythm section (piano, guitar, bass, drums) and adding a base orchestration for big band and string ensemble; the TAM album features several compositions with big band² and string trio³.

Lead Sheet

Musical Parameters

Perception

To come up with a list of musical parameters worth changing, I drew on workshops and classes¹ at the Conservatorium Maastricht and on research in the fields of music education² and perception.³ The overall musical design process was less planned and more iterative than the following table would suggest. Intuition, feasibility regarding the software used (here mostly FMOD), time and budget constraints, testing, and personal live concert experience all played a role in the final decisions.

The impact on the perception of the audience is a key element, but one of the objectives was also to analyze whether the tools, methods, and research into randomness could lead to interesting results from the perspective of the composer or performer, for whom all parameters are essential.

Recording November to March 2022

The recordings were done during several sessions in different studios. Here are the different takes for all instruments and variations, recorded into Pro Tools:

▶ Video on YouTube: https://youtu.be/7TLKZAJKCoY

Random Opening Voicings and Instruments

The opening scene from the TAM app gives the user a window of a few seconds to press a button that turns red. This triggers a random C minor chord leading into verse 1 of the Souvenir Shop tune. Not only is the voicing randomly picked, but so are the instruments playing the chord (piano, big band, strings).

The following was an experiment for the in praise of folly string trio¹ (violin, viola, and cello). Some voices were written out, and the final bar only gives minimal indication of which notes to play: a pentatonic-derived scale with a 9th, no 7th and no 6th:

Here is an unedited extract of the trio playing random picks of bar 68. I left their spontaneous reactions at the end.

Changing Speeds

This example might not find its way into the final version of the TAM app but will be used during live performances. The main bass groove is transformed by two metric modulations: the first uses the triplet (bar 6) as the new reference speed, and the second the quintuplet (bar 11). Pianist and composer Vijay Iyer used these rhythmic transformations in his adaptations of Mystic Brew and Human Nature (cf. my original post here peckels.com/blog/mystic).

The Limits of Adaptive Music

These rhythmical transformations are the hallmark of South Indian music. There are different ways to indicate these rhythmical devices in Western music notation. I chose to use the metric modulation indication with new tempo markings. A tuplet notation with group bracketing¹ would also be possible.

Transforming this example into an adaptive, randomized version in FMOD reveals the limits of using audio game engines as well as most DAWs for writing complex rhythmical music with an added layer of structure options. Rhythm changes are easy to manage in Pro Tools, Logic, etc., but this is not the case with tuplet writing. While Logic has built-in functions up to quintuplets, only Cubase handles everything above gracefully. Over the last year on iOS, some interesting non-linear DAWs have been designed with different plugins; for example, Victor Porof’s Atom Piano Roll 2 offers a piano roll-like editor with divisions up to 13 and can be combined with AUM, an audio mixer, plugin host, and recorder.

Two problems occur when trying to implement these rhythmical transformations in one of the randomized versions of The Souvenir Shop tune. First, the metrical grid is extended from 8 to 13 to 21 eighth notes, so all the other instruments would need to be re-recorded to make them fit, and different adjustments would be necessary to give a convincing rendering of the tune.

The second difficulty lies in the expanding number of combination possibilities with only these three different metric grids:

• every four bars (1, 9, and 11) a version could repeat itself
• every version could go to one of the two other versions (a total of six possible combinations)
• each time the right metric modulation would have to be played (one could imagine ignoring this transition and doing a “hard cut” transition).
• (N.B. but key: real musicians have none of those limitations, given enough practice)

By leaving out the tuplet metric modulations, the randomized version is easier to integrate with FMOD. The metric changes now happen randomly, with the bass no longer anticipating the new tempo with a triplet or quintuplet but simply playing a sustained note:

Thanks to these additive and subtractive rhythmic devices, the transitions are still interesting and don’t sound like simple tempo changes.

In generative music, these rhythmic transformations with added random elements are promising. I will now take a detour into further Karnatic techniques and link them to live coding and modular synthesis, where random features are widespread.

Karnatic Rhythmical Devices

Karnatic Practice

Part 1 of the detour uses as an example the following étude, which I wrote for the first-year assignment¹ of the Applications of Karnatic Rhythm to Western Music program, using only devices taught during the first year. The following composition is entirely based on one eight-note melodic cell (F D A Bb G D Eb C) in the violin lead voice and the rhythmic cells of the drone voice (cf. bar 1 for both), with melodic and rhythmic transformations applied throughout.

1. Karnatic Étude

2. Cell-Based Melodic and Rhythmic Patterns

One striking characteristic of South Indian music is the range of development techniques that can be applied to a basic pattern, cell, phrase, or melodic idea—“basic” in the sense of forming an essential foundation or a starting point. Here is my example of a rhythmic development² based on three sextuplet cells (a b c).

This cell-based pattern approach links the thesis to the software shown in the next video…

Pattern Seeds

▶ Video on YouTube: https://youtu.be/DXXxXCPg7Hg