Beginning in 2010, while in residence as a Fulbright Research Fellow at the Logos Foundation in Ghent, Belgium, I began working on a set of algorithms to facilitate live improvisations between robotic musical instruments and human performers. Some of the first of these human and robot improvisation performances, and a bulk of those that followed, were collaborations with Dana Jessen (bassoon) and Michael Straus (saxophone), aka the EAR Duo. A number of performers on various instruments have subsequently improvised with the robots using this system (a few recordings are included below). In terms of the robotic instruments utilized, the first performance with Jessen and Straus made use of six monophonic aerophones of the Man and Machine Robot Orchestra created by Godfried-Willem Raes. This first collaboration led directly to the commissioning of EMMI (Expressive Machines Musical Instruments) by the EAR Duo to create MARIE (Monochord-Aerophone Robotic Instrument Ensemble), with which many improvisatory performances have been created. Subsequently, new robotic instruments have been added to the mix, including my vocal robot Stemmetje.

In these works, the software controlling the musical robot(s) “listens” to the live audio stream(s) of the human performer(s), analyzing features of the audio stream in real-time such as frequency, amplitude, timbral characteristics, note-level segmentation and classification, tempo, etc. Patterns of continuous and segmented features are recorded, and variations are generated using a variety of techniques including various pseudorandom mutations, Markov chains, and genetic algorithms. Higher level control of features such as texture, density, sectional transitions, etc. can be managed by a human operator and/or the system itself (in practice this duty is usually shared, with ultimate control given to the the human operator). The system allows for direct, intuitive interactions between freely improvising human performers and the robots, which also leads to often surprising and novel navigation through the parameter spaces of the robotic instruments.