Evolving Music Video

Film Rhythm After Sound

TAM singer Claire Parsons had worked with Vienna Astrid Rothaug on a previous music clip for her In Geometry album. For TAM, we wanted to give Astrid as much freedom as possible for creating a three minutes short movie with hand drawn animated frames and a story loosely spun around an Aquatic Museum theme. For inspiration during our first meetings we provided Astrid with demo and pre-production material and settled on a simple pop music tune (with some twists), co-written1 by Claire and me, The Souvenir Shop a constant in every museum visit.

The process2 of creating a clip for music or music for a short video was never clearly laid out. The final song pick came later during the album writing and creation process. Because we had the unique opportunity to work with an artist who spend weeks in creating frame by frame hand drawn animations in high resolution we discarded several other music app ideas more based on traditional video game mechanics. Hand drawn games are rare in video games (cf. literature review chapter) because the translation from hand drawn artwork into computer animated sprites can lead to aesthetic clashes. Hand drawn paper style flat graphics don’t lend themselves easily to an interactive graphic style. Also, to respect Astrid’s vision and to keep true to the original we kept the clip’s linearity.

  1. create three minute repeatable experience that imitates the music video format but adds a new musical surprise twist. The animated short with few user interactions opportunities and a linear story line, is the backdrop to The Souvenir Shop tune.
  2. take away the power from the user who expects to get spoiled with features (that was the most daring choice.) Of course this was also done in order to simplify our work but above all to bring the main focus on music without regards to limit audio quality, number of loop and sonic experiments. Offering no interactive options is still on the table and the app would then produce an a randomly shuffled sound track. Formats Music and audio in video games is a central atmospheric mood element but too often in indie games musical considerations come very late in the development process. Furthermore, sound performance and quality has to make room for graphic performance and user interaction priorities.
  3. finally it is key to be able to release it under many different (out AI etc. MAX): iOS, Android, web (as shown here), Desktop Computer, (simple youtube clip). Cf. Brian Eno

Opening Sequence

Metamorphoses

Color Codes

Graphical User design is a challenging domain of human and computer interaction. How to guide the user in a game or software? For TAM I chose to use color to indicate possible interactions. Astrid’s shot is kept in black and white3 and color is used sparingly to indicate possible interaction.

In the TAM app the different possible interactions that will influence the music are indicated by color spots and surfaces. The doorbell from clip 1 and the transforming whale form clip 2 are examples among other larger and smaller color indications:

smaller

The tree example shows an example of multiple user interaction where different trees start random voice tracks. Later in the clip a chessboard is reached with when the Souvenir Shop tune reaches the solo section. A press of the user will trigger another instrument playing a solo and a press on the same spot, while a certain solo instrument is playing, will slowly fade into another solo version. As an Easter egg some instrumentalists agreed to play a funny or even a bad, failed version of their solo that will only rarely be triggered (1 out of 1000 times). Interaction times are limited to a few seconds, the solo section is short, and the user has no built-in recording option.

Basic touchscreen one finger interactions are supported that will also translate to mouse navigation: touch (single, double), drag, slide. Some user actions will have visual feedback cues, but audio is always upfront. The app is semi-generative4 and not generative in the sense of the original definition because it only uses the loop and soundscapes material used that we recorded and that fit the song.


  1. The majority of pieces for the TAM album were written by Claire Parsons with song and lyric contributions by me on several tunes. ↩︎

  2. Jacobs, L. (2015). Film rhythm after sound: technology, music, and performance. University of California Press; First edition (December 18, 2014). ↩︎

  3. For another example of digital bi-color cf. e.g. 1-bit pixel games, a category dating back to the 80ties: Pedro Medeiros’s tutorial on 1-bit Pixel Art ↩︎

  4. An inspiring project by Joonas Turner. ↩︎