As part of the project “Embodied Gestures” we present here a workshop and user study where participants produced physical mock-ups of musical interfaces directly after miming control of short electroacoustic music pieces. Our goal was understanding how people envision and materialize their own sound-producing gestures from spontaneous cognitive mappings.
This online document resumes the contents of the paper presented at NIME conference 2019 “Material Embodiments of Electroacoustic Music: An Experimental Workshop-Study“. Please access this paper for further and detailed information.
The goal of our workshop study was understanding how people envision and materialize their own sound-producing gestures into physical characteristics when designing musical interfaces. This information will inform the next steps of an artistic research project called ‘Embodied Gestures’.
Our workshop study departs from many previous studies describing how people mime musical control. Musical cognition is always situated and sonic memories allude to certain objects to explain interaction. In sum, during the spontaneous rendering of movement people also envision artifacts.
During the workshop, 50 participants from different creative backgrounds modeled more than 180 physical artifacts. Participants were filmed and interviewed for the later analysis of their different multi-modal associations about music.
The audio contents used to conduct the workshop study are five electroacoustic studies (named C1, C2, C3, C4 and C5) by Thomas Gorbach:
Therefore, at this workshop we proposed the production of mock-ups that, participants imagine, can perform the music they hear. Our intention was also obtaining a repertoire of artifacts and movements which could inform us about the most important physical features and actions we may include into our future designs. We are not arguing here that the resulting mock-ups will be able to universally represent certain human movements. Each person will develop different cognitive mappings for envisioning form from sonic gesture. Instead, our intention was studying the whole series of mock-ups produced during the workshop for identifying possible design patterns and trajectories.
The process is summarized at the poster we presented at the International Conference on Interdisciplinary Musicology (CIM) celebrated in September 2019 in Graz, Austria:
To illustrate the process of embodiment, envision and ideation, some results can be observed in the following two videos. Both videos are related to composition 4 (C4) and for the participant on the right (white t-shirt). First, the spontaneous cognitive mappings rendered while mimicry can be observed and compared among participants. In the second video, the participant on the right explains us the instrument he envisioned and its material features.
The workshop study was attended by 50 volunteers. They all were bachelor or master students of three different universities: 20 students of Informatics from Vienna University of Technology (TU Wien); 5 dance students and 7 computer music students from the Anton Bruckner University of Linz; 20 students from the Interface Culture media art department of the Art University of Linz. Their age ranged from
20 to 37 years except two persons aged 60 and 64. Participants from different creative backgrounds modeled more than 180 physical artifacts. Some of these mock-ups are illustrated in the following photos (galleries corresponding to compositions 1 to 5).
REPERTOIRE OF SOUND-PRODUCING ACTIONS
During the phase of analysis we observed patterns of mock-ups with similar sound-producing actions. As we have explained, the complete repertoire of actions is described at Table 1. For instance, participants envisioned mostly two types of actions for composition 1. First, participants imagined interfaces which have to be touched in a linear or circular way with their fingers. Second, a relevant part of participants imagined they play interfaces which have to be scratched. The existence of these design patterns reflect that our participants shared similar ideas about the main sound-producing action producing the sonic gesture.
This repertoire of actions, obtained in this case from only five sonic gestures among the many existing, will be ex tended with others along the duration of our project. In our opinion, this information can help other designers in our community about the particular actions people mostly identify with certain sonic gestures.
How do people design musical interfaces
with our method?
Our analysis shows that 87.5% of the mock-ups afford similar or highly compatible sound-producing gestures to the existing actions in the composition. The other 12.5% of participants created interfaces exploring alternative types of actions, non-related to the ones they had identified in the compositions. It is then clear that following our design method, the vast majority of participants prioritized sound-producing gestures. This is not surprising as each phase of the workshop emphasized this aspect.
From the analysis of the videos we observed that participants firstly envisioned excitatory gestures. The designof instrumental features enabling sound transformation was resolved at a second stage. Therefore, transformation was understood as an action modulating an already existing excitatory gesture. Certainly, this logic would allude to the causal schemes found in traditional musical instruments. We have also observed in which ways sound transformation was engineered by participants. Usually, they added an additional complementary affordance to its initial form or configuration (i.e., a new degree of freedom to the object) like knobs, sliders, buttons, additional sensors, acoustic effects, etc.
The role of materiality in this workshop
A pending aspect is discussing the material aspects of participants’ designs. All mock-ups were produced with clay. However, both at the introductory verbal descriptions and during the interview participants were asked to describe the materiality of each interface they imagined.
Verbal descriptions showed that a relevant number of participants associated what they listened with some specific material (Composition 1– 90%; Composition 2– 95%; Composition 3– 88%; Composition 4– 76%; Composition 5– 72%). The actual materials existing in the compositions were identified in different degrees (Composition 1– 69%; Composition 2– 46%; Composition 3– 72%; Composition 4– 85%; Composition 5–94%).
From the interviews, we observed that the 69% of participants produced instruments envisioning only one material while the rest explained their artifacts could be made with various sorts of them. These materials were combined with others to afford acoustic or digital transformations (e.g., reverb or pitch shifting). Our perception is that although materials would be intimately connected to sound-producing gestures, participants understood it would not be practical to implement interfaces with certain materials (i.e. water or stone). Therefore, a metaphorical approach to these ma terials, in which a material is suggested but it is substituted with another -more practical or reliable-, would be a possibility for solving this design issue.
We have presented a method to study how people design musical tools inspired by their own sound-producing gestures. The method, based on a workshop, allowed us the rapid production of physical mockups representing musical interfaces. The results indicate that participants envisioned specific design patterns connected to sound-producing gesture. Therefore, from this workshop study we obtained a repertoire of actions that participants understood as interesting to perform particular sound gestures.
Excitatory gestures and the physical materials of the interface were the central aspects defining design. In order to enable morphological transformations of sound (i.e.changes in pitch, dynamic and timbre), participants incorporated complementary affordances during the design process. In this paper we have also presented the repertoire of sound-producing gestures that participants have envisioned with our setup, inspired by five short electroacoustic compositions. In our opinion, the actions which are enacted following our method could inform musical interface designers, especially those prioritizing sound-producing gestures.