Piano, Sensors and Live Electronics

The Performer’s Perspective

by Joan Riera Robusté

Introduction

This article describes and discusses different characteristics and problems involving the use of sensors while preparing the piece Deformacios n.2 (2009 / ca 17:00), for piano, sensors and real-time electronics. 1[1. Deformacios n.2 was commissioned by the Fundació Phonos in Barcelona and premiered in 2009 at the SGAE — Sociedad General de Autores y Editores, Fundació Phonos by Jordi Sánchez Puig, with Sound Engineer José Miguel Fernandez. Other performances took place at Music Biennale 2008 in Koper (Slovenia), also performed by Fernandez, and at Sant Joan de Vilatorada in Barcelona, performed by this issue’s Guest Editor, Sebastian Berweck.] Although the piece is originally written for piano and sensors, only the part involving sensors will be here discussed.

Technical Description

This piece uses a Pressure Sensor and a Position Sensor, both produced by Doepfler (more information here). The Pressure Sensor has two functions: to activate pre-recorded sounds and to manipulate the final output volume by pressing the sensor harder or lighter. The Position Sensor is used to change the values of the different sound filters by moving a finger on its surface. The filters used are: Frequency Shifter, Harmonizer, Chorus, Muger (granular) and Delay. Both sensors are connected to Arduino BT (Bluetooth), which takes the information from the sensors and sends it to the computer. All the data is finally received and used by the programme Max/MSP. The distribution and movement of the sound is diffused over four speakers using the Ambisonics programme.

What is the Purpose of using Sensors?

The most important thing is that the sensors allow the user to manipulate sound files in real time. That means that the piece is actually being created during the performance, as the pressure and movement actions on the sensors modify the volume and sound parameters. Another important fact is that the sensor itself could be considered an instrument, as the actions are done by a performer. That means that the performer is actually performing the piece by creating the musical objects and influencing its timbre.

In this piece, the action on the sensors is pre-determined, and the performer must follow the instructions of a written score, although there are some open sections where the performer can, so to speak, improvise. An example of what the score looks like is presented in Figure 1. Note that the sensor is divided in six parts that correspond to each line in the score. The numbers correspond to sounds, which are chronologically ordered through the piece. The different signs or drawings correspond to the “gestures” that the performer must do on the sensors, in order to build the correct sound objects.

From the Musical Idea to its Realization

In the creation of Deformacios n.2, there were two basic musical ideas that involved the use of the sensors. Departing from a static musical texture consisting of different sound layers, the first idea was to create a dynamic situation where the sensors could increase the volume of one particular layer and transform its sound qualities. Then, to bring the layer back to the same level of the others and repeat the action with another layer, and so on. The second idea was to create short musical gestures or figures. For this purpose, it was necessary to activate several sounds in a rapid succession of actions, and the gestures were to be performed very precisely following a rhythmic pattern indicated in the score. During this process, some timbral modification of the sounds also occurs.

Working on the Max/MSP Patch

Once the idea was clear, we started to develop the patch to control the sensors. In order to have several sounds playing at the same time we divided each of the sensors in six parts, or sections (both sensors are quite identical in shape: 45 x 15 cm and very thin), and placed the position sensor on top of the pressure sensor. Dividing the sensor in six parts allows each portion to be large enough to offer the performer sufficient space to effectively control different parameters as the finger moves around the surface of each part.

With one finger the performer controls the sound distortion (whatever it might be at a given moment) and volume. In order to build a sound object, the degree of distortion depends on the volume and vice versa. As the two actions act together to form the sound object, placing the sensors one on top of the other allows the performer to control two separate parameters simultaneously via the same gesture (movement and pressure of the finger), allowing for a high degree of control of the result.

In the patch, the sounds follow the order assigned in a preset list. So the Pressure Sensor activates a bang with a number, this number goes to a list of sounds that are ordered chronologically (so bang 1 corresponds to sound 1, which in turn corresponds to the first event in the piece, etc.). Once the sound is chosen, it goes through a set of sound filters, whose values will be manipulated by the position sensor. At the same time, the values of the output volume will be controlled by the pressure sensor.

The important issue here is that in each of the six sections, the performer must activate an individual sound and manipulate its qualities independently, without affecting the others. That is why we need to add each filter six times, one for each layer. That was a big problem at the beginning, because the processor of our computer was not fast enough. It takes a great deal of processing power to have 36 filters functioning simultaneously, and being manipulated in real time! This was only possible after we acquired the Mac OSX with the Quad-Core Intel Xeon processor.

Sensor Limitations

One important limitation we came about just at the beginning was that you can only produce one input action at one time, so only one finger can be placed on the sensor. If you place two fingers, the sensor will read only the one which is closer to the right, where the output of the sensor is. It is like having a string instrument and pressing the same string with two fingers; only the note that is closest to the bridge will sound. However, having two sensors one on top of the other, it is possible to have one action involving two output signals. In this specific piece, by pressing and moving the finger, the final output volume and the sound parameters can be modified simultaneously.

Problems with the Sensitivity of the Sensors

The difficulty was to assign the correspondent level of sensitivity to the filters so that their numerical response is adjusted to the gestures and energy of the performer. This is very important because the two sensors are placed one on top of the other, and any action that is done on one sensor affects the other one as well. Sometimes, while the performer moved the finger on the position sensor in order to modify its sound parameters, another sound was suddenly activated by mistake by the pressure sensor. In order to prevent the wrong bangs from occurring while the performer moves the finger to change the sound qualities, the pressure sensor must be in some way controlled.

The Use of Filters

To solve this problem it was necessary to use numerical filters with Max that recognize a range of numbers above a given one and ignore the ones below. We decided to set the pass values for the pressure sensor very high, so in order to get a bang, one needs to press it hard. In this way we prevented most of the wrong bangs. Although, we solved one problem but created another one…

The action on the pressure sensor to activate a sound was too hard, so the starting volume of the sound was too high and too abrupt. We solved the problem by setting a filter with a maximum output level coming from any hard short attack (normally used to activate the sound). For any bang that was too hard, its intensity was ignored and set to the same maximum level. Only the higher volumes resulting from gradual pressure circumstances (normally happening while manipulating the sound qualities with the position sensor) are recognized. To prevent us from also causing any abrupt change with the position sensor, the volume action will also go through a filter with a pre-established fade-in fade-out envelope. In this way the performer can prevent the levels from going too high.

Although those were good decisions, the sensors were still sending wrong bangs…

Minimizing the Wrong Bangs

Although much effort was made to resolve these problems, the sensors’ sensitivity still presented problems. After applying the bang filters and during the performance, some sounds were still not activated, and others wrongly activated. It is very difficult to know exactly where the problem is; it could be the values assigned to the filters or the actions of the performer… or the design of the sensors. For both of the sensors the portion of the surface that is sensitive is very narrow, so any gesture that is not exactly in the right place can lead to failure. This is important when the performer must activate different sections rapidly, as it is easy to wrongly position the finger due to the narrow area of sensitivity. So the performer has to be very precise in striking the right spot of the sensor with the correct energy to be sure that the sensor receives the correct information to activate the sound.

We were able to minimize these accidents by programming the patch in such a way that the order of the events and the location of the events were fixed. As I mentioned before, the number of the bang is related to a specific sound. In addition, every number or sound is assigned to a specific division of the sensors out of the six possible. All this is done with the intention that a bang will be ignored if it doesn’t come from the right division. For instance, bangs with numbers 1, 2, 3, 4, 5 and 6 will be assigned to divisions 1, 2, 3, 4, 5 and 6 of the sensors. Sound number 7 will be activated only through striking the first division again. That means — and this is very important — that the performer can play and manipulate the sounds in divisions 2 to 6, with no possible wrong bangs being activated. Once he wants to keep going with other sounds, he must hit again division 1 to activate sound 7 and so on. Eureka! that was the solution for most of the problems with the wrong bangs!

This was very useful for many other reasons: as every sound corresponds to a fixed division, the performer knows exactly what sound he is manipulating and prevents him to be lost. In this sense, a computer should be placed near the performer, so he can follow in the screen the numbers corresponding to the bangs.

Conclusion

The delicate part using sensors is the Max patch. It has to be programmed carefully so all the possible mistakes coming from the sensors or the performer can be minimized. On the other hand the Arduino BT functioned perfectly in all the performances.