Interactive live music
Focus
Live interaction
Tools
TouchDesigner
Duration
4 months
Overview
Overview
Live music experiences present a unique way to enjoy our favourite songs with other fans and even the artists themselves–but what if there was a way to take it one step further?
Together with with Belinda Chen, Mikhael Ledger, and Zainab Ilyas, we developed live reactive visuals using input from MIDI instruments and Kinect sensors fed into TouchDesigner to enhance these experiences and foster further immersion in the show. These visuals act as a dialogue between the performer and the audience, allowing them to directly interact with one another. I took the role as the lead interaction and systems designer, overseeing the technical work in TouchDesigner and with the hardware.
Live music experiences present a unique way to enjoy our favourite songs with other fans and even the artists themselves–but what if there was a way to take it one step further?
Together with with Belinda Chen, Mikhael Ledger, and Zainab Ilyas, we developed live reactive visuals using input from MIDI instruments and Kinect sensors fed into TouchDesigner to enhance these experiences and foster further immersion in the show. These visuals act as a dialogue between the performer and the audience, allowing them to directly interact with one another. I took the role as the lead interaction and systems designer, overseeing the technical work in TouchDesigner and with the hardware.
Research
Research
Through up-to-date insights and statistics on WGSN, we analyzed relevant trends that aligned with our concept before diving into any development. It was important for us to understand whether or not our concept was viable in the current landscape of live performances and how we can fit it into the mainstream.
Through up-to-date insights and statistics on WGSN, we analyzed relevant trends that aligned with our concept before diving into any development. It was important for us to understand whether or not our concept was viable in the current landscape of live performances and how we can fit it into the mainstream.
We formulated these key How Might We statements to steer our direction:
We formulated these key How Might We statements to steer our direction:
...create experiences that feel authentic and emotionally grounded across multiple adult generations?
...create experiences that feel authentic and emotionally grounded across multiple adult generations?
...integrate the use of technology that fosters connections with others without feeling overwhelming?
...integrate the use of technology that fosters connections with others without feeling overwhelming?
We started by exploring songs that we were all familiar with to provide a base around which we could develop a visual identity. For this project, we looked for one that Mikhael and I could play on instruments that we knew, so we could later demo the final visuals.
We started by exploring songs that we were all familiar with to provide a base around which we could develop a visual identity. For this project, we looked for one that Mikhael and I could play on instruments that we knew, so we could later demo the final visuals.
Exploration
Exploration
We conducted visual research to find inspiration for abstract visuals that we could try in TouchDesigner. Zainab and Belinda took it further with a lyrical analysis to look into the themes of the song, as well as finding ways to integrate MIDI and body input from the instruments and Kinect, respectively.
We conducted visual research to find inspiration for abstract visuals that we could try in TouchDesigner. Zainab and Belinda took it further with a lyrical analysis to look into the themes of the song, as well as finding ways to integrate MIDI and body input from the instruments and Kinect, respectively.
I conducted some early experiments with a Focusrite Scarlet Solo and a MIDI keyboard controller to understand how input from each device works in TouchDesigner, the different types of data I receive, and how I can utilize it to manipulate the corresponding visual output.
I conducted some early experiments with a Focusrite Scarlet Solo and a MIDI keyboard controller to understand how input from each device works in TouchDesigner, the different types of data I receive, and how I can utilize it to manipulate the corresponding visual output.
As part of our larger vision, he also investigated the interior of Marquee and explored potential locations for installing trusses to support projectors and Kinects, which could be mounted and displayed in a live performance setting.
As part of our larger vision, he also investigated the interior of Marquee and explored potential locations for installing trusses to support projectors and Kinects, which could be mounted and displayed in a live performance setting.
Development
Development
For our first visual, I worked together with Belinda to develop an interactive hidden message for audiences to play with. I did some initial text effect explorations while she created the particle map textures inspired by the official music video before diving into the particle system.
For our first visual, I worked together with Belinda to develop an interactive hidden message for audiences to play with. I did some initial text effect explorations while she created the particle map textures inspired by the official music video before diving into the particle system.
Once she finalized the particle textures, I integrated them into ParticlesGPU and used the Optical Flow tool to use the Kinect to interact with the shapes. We also decided to forgo the text effects, knowing that it may make the design overwhelming for audiences to interact with.
Once she finalized the particle textures, I integrated them into ParticlesGPU and used the Optical Flow tool to use the Kinect to interact with the shapes. We also decided to forgo the text effects, knowing that it may make the design overwhelming for audiences to interact with.
Mikhael further refined this by integrating the use of his MIDI drumkit and creating “pools” of colour that change according to the drums and have a variety of different lyrics for audiences to uncover.
Mikhael further refined this by integrating the use of his MIDI drumkit and creating “pools” of colour that change according to the drums and have a variety of different lyrics for audiences to uncover.
Our second visual, I worked with Zainab to develop an immersive environment that envelops the audience in the music being played. Using my MIDI keyboard, we devised a system that takes in key input and transforms it into this “ink fall” feedback. The Kinect takes in body index to show the audience’s silhouettes and create a frosted glass disruption in the key feedback.
Our second visual, I worked with Zainab to develop an immersive environment that envelops the audience in the music being played. Using my MIDI keyboard, we devised a system that takes in key input and transforms it into this “ink fall” feedback. The Kinect takes in body index to show the audience’s silhouettes and create a frosted glass disruption in the key feedback.
Finally, Belinda worked on creating a refined scale model of the Marquee to simulate the installation of this experience.
Finally, Belinda worked on creating a refined scale model of the Marquee to simulate the installation of this experience.
Takeaways
1.
Design with and for your audience
Design with and for your audience
Developing for “play” goes beyond thinking about just aesthetics, focus on clarity, responsiveness, and accessibility to deliver a great experience.
Developing for “play” goes beyond thinking about just aesthetics, focus on clarity, responsiveness, and accessibility to deliver a great experience.
2.
Balance between creativity and stability
Balance between creativity and stability
Visually exciting effects must still be optimized to avoid any performance drops and be intuitive enough for users to easily pick up.
Visually exciting effects must still be optimized to avoid any performance drops and be intuitive enough for users to easily pick up.
3.
Iterate, iterate, iterate
Iterate, iterate, iterate
The smallest changes can drastically affect how intuitive or confusing the interactions feel for both the performers and the audiences.
The smallest changes can drastically affect how intuitive or confusing the interactions feel for both the performers and the audiences.
Interactive live music
Focus
Live interaction
Tools
TouchDesigner
Duration
4 months
Interactive live music
Focus
Live interaction
Duration
4 months
Tools
TouchDesigner
Overview
Live music experiences present a unique way to enjoy our favourite songs with other fans and even the artists themselves–but what if there was a way to take it one step further?
Together with with Belinda Chen, Mikhael Ledger, and Zainab Ilyas, we developed live reactive visuals using input from MIDI instruments and Kinect sensors fed into TouchDesigner to enhance these experiences and foster further immersion in the show. These visuals act as a dialogue between the performer and the audience, allowing them to directly interact with one another. I took the role as the lead interaction and systems designer, overseeing the technical work in TouchDesigner and with the hardware.
Research
Through up-to-date insights and statistics on WGSN, we analyzed relevant trends that aligned with our concept before diving into any development. It was important for us to understand whether or not our concept was viable in the current landscape of live performances and how we can fit it into the mainstream.
We formulated these key How Might We statements to steer our direction:
We started by exploring songs that we were all familiar with to provide a base around which we could develop a visual identity. For this project, we looked for one that Mikhael and I could play on instruments that we knew, so we could later demo the final visuals.
...create experiences that feel authentic and emotionally grounded across multiple adult generations?
...integrate the use of technology that fosters connections with others without feeling overwhelming?
Exploration
We conducted visual research to find inspiration for abstract visuals that we could try in TouchDesigner. Zainab and Belinda took it further with a lyrical analysis to look into the themes of the song, as well as finding ways to integrate MIDI and body input from the instruments and Kinect, respectively.
I conducted some early experiments with a Focusrite Scarlet Solo and a MIDI keyboard controller to understand how input from each device works in TouchDesigner, the different types of data I receive, and how I can utilize it to manipulate the corresponding visual output.
As part of our larger vision, he also investigated the interior of Marquee and explored potential locations for installing trusses to support projectors and Kinects, which could be mounted and displayed in a live performance setting.
Once she finalized the particle textures, I integrated them into ParticlesGPU and used the Optical Flow tool to use the Kinect to interact with the shapes. We also decided to forgo the text effects, knowing that it may make the design overwhelming for audiences to interact with.
Development
For our first visual, I worked together with Belinda to develop an interactive hidden message for audiences to play with. I did some initial text effect explorations while she created the particle map textures inspired by the official music video before diving into the particle system.
Mikhael further refined this by integrating the use of his MIDI drumkit and creating “pools” of colour that change according to the drums and have a variety of different lyrics for audiences to uncover.
Our second visual, I worked with Zainab to develop an immersive environment that envelops the audience in the music being played. Using my MIDI keyboard, we devised a system that takes in key input and transforms it into this “ink fall” feedback. The Kinect takes in body index to show the audience’s silhouettes and create a frosted glass disruption in the key feedback.
Finally, Belinda worked on creating a refined scale model of the Marquee to simulate the installation of this experience.
Takeaways
Design with and for your audience
Developing for "play" goes beyond thinking about just aesthetics, focus on clarity, responsiveness, and accessibility to deliver a great experience.
Balance between clarity and stability
Visually exciting effects must still be optimized to avoid any performance drops and be intuitive enough for users to easily pick up.
3.
Iterate, iterate, iterate
The smallest changes can drastically affects how intuitive or confusing the interactions feel for both the performers and the audiences.
