TAM singer Claire Parsons had worked with Vienna Astrid Rothaug on a previous music clip for her In Geometry album. For TAM, we wanted to give Astrid as much freedom as possible for creating a three minutes short movie with hand drawn animated frames and a story loosely spun around an Aquatic Museum theme. For inspiration during our first meetings we provided Astrid with demo and pre-production material and settled on a simple pop music tune (with some twists), co-written1 by Claire and me, The Souvenir Shop a constant in every museum visit.
The process2 of creating a clip for music or music for a short video was never clearly laid out. The final song pick came later during the album writing and creation process. Because we had the unique opportunity to work with an artist who spend weeks in creating frame by frame hand drawn animations in high resolution we discarded several other music app ideas more based on traditional video game mechanics. Hand drawn games are rare in video games (cf. literature review chapter) because the translation from hand drawn artwork into computer animated sprites can lead to aesthetic clashes. Hand drawn paper style flat graphics don’t lend themselves easily to an interactive graphic style. Also, to respect Astrid’s vision and to keep true to the original we kept the clip’s linearity.
Graphical User design is a challenging domain of human and computer interaction. How to guide the user in a game or software? For TAM I chose to use color to indicate possible interactions. Astrid’s shot is kept in black and white3 and color is used sparingly to indicate possible interaction.
In the TAM app the different possible interactions that will influence the music are indicated by color spots and surfaces. The doorbell from clip 1 and the transforming whale form clip 2 are examples among other larger and smaller color indications:
The tree example shows an example of multiple user interaction where different trees start random voice tracks. Later in the clip a chessboard is reached with when the Souvenir Shop tune reaches the solo section. A press of the user will trigger another instrument playing a solo and a press on the same spot, while a certain solo instrument is playing, will slowly fade into another solo version. As an Easter egg some instrumentalists agreed to play a funny or even a bad, failed version of their solo that will only rarely be triggered (1 out of 1000 times). Interaction times are limited to a few seconds, the solo section is short, and the user has no built-in recording option.
Basic touchscreen one finger interactions are supported that will also translate to mouse navigation: touch (single, double), drag, slide. Some user actions will have visual feedback cues, but audio is always upfront. The app is semi-generative4 and not generative in the sense of the original definition because it only uses the loop and soundscapes material used that we recorded and that fit the song.
Jacobs, L. (2015). Film rhythm after sound: technology, music, and performance. University of California Press; First edition (December 18, 2014). ↩︎