A Creative Coding Interactive Sound Installation Using Hand Gesture Technology

Weave is an immersive interactive installation that turns hand gestures into music. Built with Creative Coding (p5.js and ml5.js), the project lets participants mix and conduct sounds in real time, becoming music conductors with nothing but their hands.

Exhibited at the SVA Interaction Design Open Studio in New York City, Weave invited people of all ages to explore gesture-based interaction and creative technology.

The vision behind Weave was to create an interactive and playful installation that reminds people of the power and artistry in their hands. Music is a universal connector, and by combining it with gesture-based interaction, the project encouraged participants to rediscover creativity and expression through movement.

One of the main challenges was coding the sound blending. I needed the music to fade in and out smoothly as people moved their hands, so that the experience felt natural rather than abrupt.

Another challenge was fine-tuning the hand-detection area to make gestures responsive across different positions. I overcame both by experimenting with coding libraries and refining the logic until the transitions were seamless.

Weave was displayed in a cozy, semi-private space with a glass wall and a projector that showcased participants’ movements. Visitors received playful cards with mystical symbols and thoughtful quotes to take home, adding to the immersive experience. People of all ages loved experimenting with sounds and seeing how their hands could "conduct" music.

Future versions of Weave will include more hand gestures, like fist-clenching to stop sounds or pinching to adjust volume. I’m also considering allowing users to upload their own sounds to make the experience even more personal.