MuSyC is a project which aims at building a music to colour synaesthesia visualizer.
Synaesthesia is a neurological phenomenon in which perception of a certain stimulus, such as the musical note ‘A’, involuntarily elicits another seemingly unrelated sensation, such as seeing the colour red. There are various types of synaesthesia, one of the most common of which is music to colour associations. However, explaining the qualia or feeling this cross-sensory activation elicits is hard to do – thankfully, music visualizers have made it easier to showcase how sound can influence and shape images. Much in the way the father of modern abstract art, Wassily Kandinsky, used his music to colour synaesthesia as an inspiration for his artwork, we hope that this visualizer can both raise awareness about synaesthesia but also be a source of artistic or musical inspiration to composers, artists, disc jockeys, conductors and performance artists. MuSyC is a device which is sensitive to amplitude and frequency (or pitch) of a sound and flashes certain colours in response to particular defined notes – such as the universal A, B, C, D, E and F.
This project integrates and transcends digital signal processing, art, music and neuroscience – the kind of interdisciplinary work that the Mount Holyoke MakerSpace is encouraging . Not only will this device allow non-synaesthetes to experience the world and music from the eyes of a music-colour synaesthete, in collaboration with musical orchestras, choirs and rock bands across the Five Colleges, this project also provides an excellent opportunity to blur the lines between music and performance art with digital technology. Looking into the future, we hope to make MuSyC a wearable device which can easily be clipped onto an instrument or the body of the musician and, using LEDs or Neopixels, be able to create accessories such as headbands or bowties which light up to the corresponding frequency, pitch or volume of the musical stimulus. Alternatively, we hope to develop a digital interface or app so that MuSyC can be used on people’s phones, taking advantage of the integrated microphones within the phone.
Moreover, we hope that this device can also be used as an educational tool for those beginning their training in music and those who need an additional visual cue to help distinguishing between closely related notes. For the untrained ear, it is very difficult to differentiate between the flat, whole and sharp note (F vs F#). MuSyC will be a tool to help them gain proficiency at picking out those differences. We hope that this device could be an additional resource for not only Mount Holyoke’s music department but music teachers everywhere. Additionally, MuSyC can be used as an accessability tool for those who are deaf or hard-of-hearing as a way to experience music visually. We also hope to make MuSyC a customizable tool, giving individuals the ability to alter the corresponding colours for their LEDs to reflect their own synaesthetic experiences.
This project was inspired by a class taught by Professor Susan Barry and Professor Linda Laderach at Mount Holyoke called ‘Art, Music and the Brain’ which discussed the neurobiological basis of our sensations and perceptions. Though this tool was inspired by a neurological phenomenon, we project this as an idea to be used in artistic and musical practice using digital technology and circuitry. This project was started in January 2016 and we were able to build our first prototype at the end of February 2016. That being said, we will not stop there. We hope to nurture MuSyC into becoming a more robust synaesthesia simulator and we’re super excited to see where it will take us!
I’d like to acknowledge Cassiel Moroney (MHC ‘19) and Kyoko Sano (Hampshire ‘18) – the HampHack MuSyC team, who were crucial to the development of the first prototype of MuSyC. I’d also like to thank Luke Jaeger, my project advisor for MuSyC, and the Department of Computer Science at Mount Holyoke without whom MuSyC would still just be an idea. I’d also like to thank Shani Mensing and the Mount Holyoke MakeSpace which has provided all the necessary tools and skills to be able to even dream of being able to build this device, as well as the Five College Digital Humanities for the microgrant which enabled me to purchase the components to build our first prototype.
Photos of HampHack 2016