iPhone app. WIP
Embodied Interaction and Dynamic Composition
For three years now I am working on systems which use full body movement as input and react with musical feedback. The latest project in this journey is called Affine Tuning. In contrast to my earlier projects, which were presented as installations, this is an iPhone app which players can try in their personal space.
Body Capture Data
Players interact with the system completely through body movement and posture, captured by the phone camera and analysed through „pose estimation“ software. The interactions are based on the position of the body limbs (both individually and in relation to each other), the expansion in space, the rotation and others. The feedback is audio only – there is no screen. Nevertheless I used some data visualisation for inspiration of how this data could be used to influence dynamic music.
Interaction With Music
Sound is a very immediate medium which hits us directly and from all sides and easily triggers movement. Particularly dominant is the reaction to rhythm, which actually not only means to react, but to anticipate future impulses. Certain other organizations of sound and compositional elements, such as a long string, can inspire our movement patterns in space. Think of sliding from bar to bar in a waltz, or headbanging in a metal concert.
One core idea of the project is to present a system which is quite open, using a combination of rudimentary principles. Those are relating the intensity of rhythm to the energy of the movement, and the pitch of the sound to the raise of certain limbs. There are more, specialized, interaction methods, but these are the basic methods.
The reasoning for this rather loose model is that both the player and composer should have freedom for artistic expression without coupling every instance of it in a tight feedback loop. My intention is not to create an instrument that can be controlled. Rather, I am developing a system that enables a cycle of interpretations, of questions and answers, in which an ephemeral experience with a lasting impression can take place.
The music in Affine Tuning are mostly original pieces. The first composition is called "Nefrin", which means "Curse" in Farsi. A few more pieces will follow before the work is published.
All these descriptions are not really satisfying, and the obvious makeshift, the videos on this page, are also only a shadow of experience. My focus is explicitly not on this kind of presentation, but on the individual experience. I think that if you perceive it only from "outside", it almost becomes a different work.
This is why I am looking forward to release Affine Tuning in early 2021 for the iPhone. Right now it is planned to have at least three songs in the app, and to update the software with more compositions after that.
If you want to go on a deep dive about the project, you can read my thesis documentation.