Tanaka2000 Sensor Instruments
Tanaka2000 Sensor Instruments
Atau Tanaka
Faculty of Media Arts and Sciences
Chukyo University, Toyota-shi, Japan
atau@[Link]
Introduction
Performance has traditionally been a principal outlet for musical expression – it is the moment when music is
communicated from musician to listener. This tradition has been challenged and augmented in this century –
recorded and broadcast media technologies have brought new modes of listening to music, while
conceptual developments have given rise to extremes such as music not to be heard. Computer music makes
up part of these developments in new methods of creation and reception of music. It is at its origin a studio
based art, one that gave composers possibilities to create a music not possible with traditional performative
means. Realizing music in the electronic and computer music studios freed the composer from traditional
channels of composition-interpretation-performance, allowing him to focus in a new way on sonic materials.
Advancements in computer processing speeds have brought with it the capabilities to realize this music in
real time, in effect giving us the possibility to take it out of the studio and put it on stage. This has
introduced the problematic of how to articulate computer generated music in a concert setting.
Concurrently, developments in the fields of computer-human interface (CHI) and virtual reality (VR) have
introduced tools to investigate new interaction modalities between user and computer (Laurel 1990). The
intersection of these fields with computer music has resulted in the field of gestural computer music
instrument design. The creation of gestural and sensor based musical instruments makes possible the
articulation of computer generated sound live through performer intervention. While this may be a recent
area of research, the musical issues of engaging performance are something that have been addressed
before the arrival of the computer. So while from a technical standpoint research in this field may represent
354
advances in musical and perceptual bases of gestural human-machine interaction, musically these efforts in
computer music have precedence in instrumental music. It can be said that musically this brings us full circle,
back to the concert as forum for the communication of music.
If concert performance is the medium of communication then the instrument becomes the conduit between
performer and listener. The listener’s perception of the music is contingent on the instrument’s efficiency at
transmitting the performer’s musical expression, and the performer's ability to channel his creativity through
his instrument. We must integrate what we know about human machine interaction with our musical
sensibility. What follows is a personal reflection on a performance practice developed on sensor based
instruments, drawn from several years of concert performance with the BioMuse (Tanaka 1993), ensemble
work with Sensorband (Bongers 1998) in the creation of the Soundnet, and the application of instrumental
thinking to an installation and network based project: Global String. Observations have been made from
personal reflections during practice and development, discussion with other artists, audience reactions, and
interaction with students.
Tools
A musical instrument, as part of an engaging performance, becomes an expressive device in the hands of the
performer. This gives an instrument a distinguishing characteristic when compared to a simple tool. The term
tool implies that an apparatus takes on a specific task, utilitarian in nature, carried out in an efficient
manner. A tool can be improved to be more efficient, can take on new features to help in realizing its task,
and can even take on other, new tasks not part of the original design specification. In the ideal case, a tool
expands the limits of what it can do. It should be easy to use, and be accessible to wide range of naive users.
Limitations or defaults are seen as aspects that can be improved upon.
A musical instrument’s raison-d’être, on the other hand, is not at all utilitarian. It is not meant to carry out a
single defined task as a tool is. Instead, a musical instrument often changes context, withstanding changes
of musical style played on it while maintaining its identity. A tool gets better as it attains perfection in
realizing its tasks. The evolution of an instrument is less driven by practical concerns, and is motivated
instead by the quality of sound the instrument produces. In this regard, it is not so necessary for an
instrument to be perfect as much as it is important for it to display distinguishing characteristics, or
"personality". What might be considered imperfections or limitations from the perspective of tool design
often contribute to a "personality" of a musical instrument.
355
Computers are generalist machines with which tools are programmed. By itself, a computer is a tabula rasa,
full of potential, but without specific inherent orientation. Software applications endow the computer with
specific capabilities. It is with such a machine that we seek to create instruments with which we can establish
a profound musical rapport.
The input device is the gateway through which the user accesses the computer software’s functionality. As a
generalist device, generalized input devices like the keyboard or mouse allow the manipulation of a variety
of different software tools. Music software can be written to give musically specific capabilities to the
computer. Input devices can be built to exploit the specific capabilities of this software. On this general
platform, then, we begin to build a specialized system, each component becoming part of the total
instrument description.
Instrument definition
With a traditional acoustic instrument, defining what comprises the instrument is straightforward - it is the
object that the performer holds or plays on. It and its acoustical properties are the medium between
performer action and sound. Through the instrument, the performer is able to affect all aspects of the music:
from the micro-level of timbre to the event level of note articulation, to the macro level that reflects
compositional structure. In a wind instrument, the breath of the performer affects both the micro-timbral
qualities as well as articulate the note-events from the instrument – articulating the event, defining the
harmonic being articulated. The fingerings work in tandem to further define the note-level event. An
attachment like a mute can be seen to work on the micro level, affecting the timbre of the instrument at the
same time it can be considered to work at a structural level – causing a section of a piece to take on a certain
sound (Wessel 1987).
In the case of a computer based instrument, the medium between performer input and sound output is
comprised of several distinct stages, spanning input device, software, and hardware. It is a reprogrammable
system, differing from the dedicated nature of a traditional instrument. The input device is often a sensor
system capturing the physical gesture of the performer. This input is detected and digitized, producing
computer data representing the actions of the musician. This data is passed on to the software component
of the instrument that maps gestural data to musical and sonic parameters. This software most commonly
runs on the host CPU but could reside in firmware on the input device or in the sound synthesis unit. The
sensors define the physical modes of interaction between musician and instrument while the software
defines the responsiveness and articulative possibilities of the instrument.
356
Much as an acoustic instrument operates on the multiple micro, event, and macro levels, a computer
instrument can be viewed in this way. Choices made by the instrument designer and composer are
implemented in software and define the performative qualities of the instrument. Certain gestures can
manipulate sound synthesis directly (Wanderley 1998), while others articulate notelike events, while other
actions direct the structural progress of the piece. Applying these concepts to computer realization gives rise
to new possibilities that blur the boundaries separating event articulation and structural modification. A
single articulation can elicit not just a note, but an entire structural block. Variations in the articulation
could very well modify the timbre with which that structure is played, or could introduce slight variations on
the structure itself or could modify which structure is elicited. The potential combinatorial space of
complexity is quite large. It is also possible to focus entirely on one level, in effect ignoring others: with
computer generated sound, a musical event in the classical sense does not have to be articulated for the
sound to be modified. One could imagine a system that is comprised simply of an ongoing stream of sound
that is sculpted by the performer’s gestures.
Instrument types
Given this broad, encompassing definition of an instrument, we can define several categories to help
delineate the different classes of gesture instruments (Pressing 1990), (Cadoz 1994). Instruments can be
physical or non-physical, based on their mode of interaction. They can independently be mechanical or non-
mechanical. Turning a potentiometer is a physical mechanical action translated into electrical potential,
whereas creating gestures to modulate light being picked up by a photocell, as much as it is a physical
gesture, is not physical or mechanical from the point of view of the sensor. A biosensor is not mechanical,
but is physical as it detects corporeal activity.
A computer instrument may or may not have an associative connection with a traditional instrument. This
association with an existing instrument can be direct, metaphorical, or an extension of the instrument. With
direct association, the natural sound of the instrument can be used as a control source (Bailey 1993). Or,
sensors can be placed on an existing instrument in such a way to pick up traditional performance practice on
the instrument – a MIDI guitar can be viewed in this way. Extensions to the instrument can include sensors to
pick up new articulations on the instrument particular to the sensors (Katayose 1994). Or the traditional
instrument can become the metaphorical model for a new instrument, providing a point of familiarity and
departure. The Synthophone (Hurney) is an example that is extremely close in keeping to the original
instrument - to the extent that it modifies an actual saxophone - but where the end result is a pure
electronic instrument that no longer makes acoustical sound. Suguru Goto’s SuperPolm is an example of
fantasies on the metaphor of the violin [Editors’ note: see the article by Goto in this volume].
357
In the area of totally new instruments not based on a traditional instrument as referent, new gestural
instruments can be created. Despite the independence from existing instruments, they can nonetheless be
designed based on the model of a musical instrument as an articulatory musically expressive object in the
hands of a musician. Alternatively, new paradigms can be investigated: the definition of a musical
instrument can be expanded to embrace environments such as public installations experienced by non-
musicians. Even the concert spaces can conceptually be considered instruments: as resonant acoustic bodies
that respond to performer’s gestures, with loudspeakers as transducer/actuators (Bongers 1998). In the case
of network music, the network can be considered an instrument that responds with certain characteristics to
user gestures represented as network data (Tanaka 1999).
BioMuse
The BioMuse is a biosignal musical interface developed by Hugh Lusted and Ben Knapp of BioControl
Systems (Knapp 1990, Lusted 1996). It was developed at Stanford University's Center for Computer Research
in Music and Acoustics (CCRMA), Medical School, and Electrical Engineering Departments. It takes
bioelectrical signals in the form of electroencephalogram (EEG), electromyogram (EMG) and
electrooculogram (EOG) and translates them into serial digital data and MIDI. Surface gel electrodes on the
skin establish electrical contact, conducting neuronal impulses from the muscles and brain. Each of the eight
(8) input channels receives a differential signal from an electrode triplet. Upon reception by the BioMuse
main unit, the signals are amplified, filtered, digitized, and processed according to a number of user
configurable algorithms including envelope following and spectral analysis.
I began work with the BioMuse in 1992 and was commissioned by BioControl to create its concert premier. I
continued concertizing with it in Europe from 1992 to 1997. Upon moving to Japan in 1997, I have been in
contact with various teams seeking to make miniaturized EMG devices (Nagashima 1998). My work with the
BioMuse has been exclusively with the EMG (muscle) channels. Elastic bands are used to place the electrodes
in locations to detect arm gestures. The BioMuse runs a program implementing envelope following on the
muscle tension signal. This is sent to the host computer as MIDI or 12 bit serial data. This data enters the
MAX software environment (Puckette 1990), is further processed, then drives the MSP real time audio
engine.
The placement of electrodes on the two arms is asymmetrical - on my left arm, I have electrodes on the inner
and outer forearm. On my right art, the electrodes are on a general lower arm location and triceps. While
the differential nature of the signal averages muscle contraction impulses across the area covered by the
electrode triplet, placement of the triplets allows differentiation of large muscle groups. The triceps band
therefore detects outward and upward movement of the upper arm. The general lower arm band detects
358
average muscle tension in the lower arm, while the outer and inner lower arm bands can differentiate two
muscle groups. The outer band detects wrist movement that lifts the hand whereas the inner band detects
gesture that curls the hand inward (Putnam 1993).
Fig. 1. The author in performance at the festival Manca in 1995. The Biomuse controls the
sound and image using MidiKaleido by Eric Wenger.
It is interesting to apply observations of the way the body works in selecting electrode placement. The
movement of any member is often created by muscle action on the member above it. Thus hand and wrist
motion and rotation implicates the lower arm muscles. Even finger movement can be detected in the tension
trajectories of the lower arm muscles. This creates the interesting possibility to detect small level gesture
without having to put sensors on every finger, but to observe these gestures through data from a larger
member. The data from the lower arm, then, is rich in information. Data processing can begin to extract this
information from the signal (Warner 1995).
359
The signal entering the host computer is a continuous data flow that is the amplitude envelope of the
muscle group under each electrode set. This data can be treated in several simple ways to render it musical.
We can program level sensitive triggers to transform the continuous input to event-type data, useful for the
macro, event, or structural interventions mentioned in the previous section. The trigger is implemented in
MAX, armed to detect the input signal crossing a defined threshold. Hysteresis is added such that a separate
downward threshold is defined that re-arms the upward trigger, often referred to as a Schmitt trigger
(Strange 1983). This has the immediate application of suppressing multiple false triggers. A hysteresis
combined with a dynamic upward trigger level could be used in tandem to depict a kind of "effort" of
articulation. This kind of use of hysteresis then is a special case of memory of the trigger. A software based
trigger can hold data on the number of times and frequency with which it has been articulated, with this
data affecting the nature of the trigger itself.
The continuous nature of the muscle envelope signal is well suited for timbral control of the sound synthesis.
It can also shape or affect the nature of an event trajectory. Simple processing on the continuous input
include defining the active range, scaling linearly or non-linearly, and using in conjunction with triggers
listening to the same data. Using the continuous data from more than one of the arm bands together can
create the possibility of dependencies and variations. The inner and outer lower arm signals can be
combined and averaged to give a general lower arm band tension signal, or can be combined in
counterphase to provide one signal that represents left-right or inner-outer "steering" action of the wrist.
Other possibilities include applying integrators (for accumulation, with the possibility of a "leaky"
parameter) and differentiators (used in the calculation of velocity and acceleration).
Based on the concepts outlined in the previous section on instrument definition, we can define the musical
instrument built around the BioMuse system. The instrument for me, then, is made up of the specific
placement of the electrode armbands, the BioMuse hardware, the firmware program running inside the
BioMuse, the MAX software on the host computer post-processing the biosignal data, and the sound
synthesis implemented in software in MSP and realized in hardware on the Macintosh. This instrument has
remained more or less a constant. With each piece, the middle area between the MAX processing of
BioMuse input and the sound output change - the composition then is the choice and combination of
different biosignal post-processing techniques as connected to various synthesis articulations, place in
context of a musical structure over time. The "shell" of the MAX patch - input from BioMuse, output to
synthesis - is part of the instrument, while the inner contents of the MAX patch - the specific mappings of
gesture to sound and their development in time - form the score of the composition.
360
The BioMuse is not a mechanical instrument, however it is quite a physical one as it depends directly on
corporeal gesture. Though not based on any existing instrument, it is possible to program the software to
emulate instruments. For example, the BioMuse/MAX system can be programmed to create an "air guitar".
It is a kind of arbitrary imitation, since the biosignals used to drive the air-guitar patch are in no way
scientifically linked to a true guitarists gestures. My work with the BioMuse has not been concerned with
instrumental emulation. However, my use of the BioMuse is an abstraction of instrumental gesture in the
absence of a physical object to articulate music through corporeal gesture.
Soundnet
Soundnet is a large scale multiuser sensor instrument created by Sensorband (Atau Tanaka, Zbigniew
Karkowski, Edwin van der Heide). It is a musical instrument that takes on an architectural dimension and
monumental proportions (Sensorband). It is a giant web measuring 11meters x 11 meters, strung with 400m
of 16mm thick shipping rope. At the end of the ropes are eleven sensors that detect stretching and
movement. The three musicians of Sensorband perform on the instrument by climbing it. It is an instrument
commissioned by the festivals Visas and Exit in France, and has been performed at the DEAF festival
organized by V2 in Rotterdam.
Soundnet is inspired by The Web, a 1 meter diameter spider's web created by Michel Waisvisz, at STEIM
(Krefeld 1990). The idea with Soundnet was to create a life sized version of The Web. The sensors were
designed by Bert Bongers and fabricated by Theo Borsboom. Each sensor consists of a cylinder and piston
that can sustain 500kg of force. Inside is a large spring connected to a linear potentiometer. As the
performers climb the ropes, the sensors stretch and move in response to their movements. This displaces the
faders inside, sending a variable resistance control signal to an I-Cube interface which in turn digitizes the
data, sending the result as MIDI continuous controllers to the host computer.
The output from the sensors are continuous signals. As the piston is spring loaded, the sensor movement
tends to be in non-regular oscillation about a static tension point. Interesting qualities to extract from the
signal thus include velocity and acceleration, and change of direction. These data has been used to control
filter parameters as well as granular synthesis.
Soundnet is a musical instrument that is physical and mechanical. Its scale and proportion is such that the
physical part overwhelms the technological part. This creates an inversion of the typical concerns
surrounding interactive technology, de-emphasizing issues of sensing and processing, and focusing more on
the pure organic nature of a human confronted with a dynamic architectural system.
Global string
Global String is a multi-site network music installation, a collaboration between myself and Kasper Toeplitz
(Tanaka and Toeplitz). It is a musical instrument where the network is the resonating body of the
instrument, by use of a real time sound synthesis server. The concept is to create a musical string (like the
string of a violin) that spans the world. Its resonance circles the globe, allowing musical communication and
collaboration among the people at each site.
The installation consists of a real physical string connected to a virtual string on the network. The real string
(12 mm diameter, 15 m length) stretches from the floor diagonally up to the ceiling of the installation space.
On the floor is one end point - the earth. Up above is the connection to the network, connecting it to
another end point at a remote location. Sensors translate the vibrations into data to be transmitted to the
other end via network protocol. The server is the "bridge" of the instrument - the reflecting point. It runs
software that is a physical model of a string. Data is streamed back to each site as sound and as data –
causing the string to become audible, and to be actuated to vibrate, on the other end. A real time video
conferencing connection, gives a visual connection between the users. Through this connection, visitors at
each of the remote sites can play together. Visual and aural cues allow the development of ensemble
practice over distance. Their tandem interventions on the single conceptual string create interference
patterns and harmonies.
Global String extends the concept of instrument definition in several ways. It is one string that is comprised
of distinct components – physical endpoints and virtual midpoints. The installation space, as well as the
auxiliary communications channel (video) become part of the instrument. Most importantly, the medium of
transmission, the network, becomes part of the instrument. In implementation and practice, Global String is
a scalable multi-mode installation. The number of sites can range from two and up – creating the possibility
of multiple end points not easily realized in the physical world. The installation can be used in concert mode
for soloists at each site to perform together or as an installation for gallery visitors to share a musical
experience.
364
Limits
Musical instrument characteristics are defined as much by their limitations as they are by their capabilities.
These limits, such as note range and articulative speed, must be considered and respected to compose
playable, idiomatic music for that instrument. The topic of idiomatic writing will be discussed below. The
limits are seen not as hindrances to the composer, but as constraints that can be challenged and extended.
A computer is often thought of as a tool with no limits. When designing a sensorbased instrument, the
standard response to queries about an instrument’s capabilities is that anything is possible. In reality,
computers and interfaces to computers are in fact quite limited machines. Typical constraints include
processor speed and degrees of freedom of an input device. As these are regarded as technical limitations,
composers do not have the reflex to use the limitations as creative material, and instead wish for an
365
amelioration in an ensuing version of the hard or software (Zicarelli 1992). Though development includes
the pursuit of fewer such limitations, it can also be said that the lack of limitations creates an absence of
boundaries against which a composer can play.
Discussions of computer based instruments often tend to focus on the power or capability of the instrument.
With sensor instruments the question typically raised is how many synthesis parameters it allows the
musician to control. The temptation is to follow this with the possibility that sensor technology opens up
possibilities to control not just synthesis and musical parameters but other media, including lighting and
image. The danger is in ending up not with a gesamtkunstwerk, but with a kind of theme park "one-man
band" (Puckette 1993). While this may show off the power of computers to simultaneously control multiple
parameters across multiple media, it does little to convey the expression of the performer. Instead, viewing
the situations from the standpoint of creative applications of limitations may yield more musical results.
With the BioMuse, limitations can arise from the human body’s own capabilities. This supports the argument
to include the body as part of the instrument definition. A very basic limitation of the body is fatigue. The
performer has a certain capacity for continued muscle exertion over time. Another limitation, quite
characteristic of biosignals is the difficulty of maintaining an absolutely stable tension level. The EMG signal
is a truly living signal, and will invariably waver. This creates an output that is quite distinct from other types
of continuous input devices.
Limitations of the Soundnet are related to its physical scale and interconnected nature. Moving the sensors
takes enormous effort on the part of the performer. At the same time, there is considerable inadvertent
movement of the sensors in response to displacement of the performers. Isolating individual sensor
movement is nearly impossible. These effects combine to give the instrument its qualities of structural
resonance and creates a dynamic where pure control in the strict sense is put in question (Broeckmann 1997).
With Global String, data transmission latency can be considered defining qualities of the instrument. The
transmission delay is significant, and is non-constant. Typically in network music applications, these effects
are defaults to be eliminated or improved upon. In the case of Global String, the instrument has been
conceived to include these network characteristics as part of the instrument definition. As a vibrational
feedback is created, the transmission delay creates a resonance. The variable transmission delay creates a
breathing resonance that prevents self oscillation. If delay were not present, the feedback would
degenerate instantaneously.
366
Idiomatic composition
By virtue of the relationship established between performer and instrument, we create a gestural language
for the instrument. This is the basis of development of a performance practice. Issues of idiomatic writing
invariably enter discussions of composing for specific instruments. Idiomatic writing is the art of writing
music that is suited, adapted, and optimized for an instrument. In this way, a standard vocabulary of
performance practice for a particular instrument is established, giving the instrument a musical identity. The
ultimate result is the creation of a body of work, a repertoire for that instrument. We can retrace the
development of repertory for instruments such as the violin and piano in this way – each evolving as
instrument based on a predecessor (the de gamba family in the case of the violin, the harpsichord and
clavichord with the piano), culminating in virtuoso showpieces of the romantic era, furthered in experiments
of extended technique in the 20th century.
The tenets of idiomatic writing take into consideration features of the instrument such as basic articulation,
and note-range. Articulation technique and resulting timbre are considered together to consider what kinds
of phrases and passages are most appropriate for the instrument. In this way the sonic timbral qualities are
tied to the event structure in defining a characteristic for the instrument. A wind instrument may be capable
of smooth legato, but the fact that its articulation is based on the human breath, for example, puts a
practical limit on the length of a phrase. This is the standard material of books on orchestration.
An instrument may be able to play in a wide pitch range, but more easily in its principle register than in an
extended register. This has consequences on the resulting performance that must be anticipated by the
composer. Mastering thus the variations in facility and effort help the composer create a dramatic trajectory
for a composition that is in fact based on the nature of the instrument itself. In this way, he is composing
with an eye towards the energy of performance. Knowing the idioms of performance practice on an
instrument help the composer imagine and thus compose in absence of the instrument and in anticipation
of a performance.
These basic idioms for each instrument can be challenged through extended technique. This practice
explores alternate articulations and methods for extending the pitch and timbral range of an instrument. As
much as they may challenge what is considered idiomatic for that instrument, extended technique in fact
builds on, adds to, and can even negate the existing vocabulary of that instrument. All such considerations
are taken in respect of an instrument’s capabilities and limitations. Recognizing the limits of an instrument
(be they physical, acoustic, or practical), delineate a reference point that help to distinguish standard
practice vs. extended technique.
367
How can this notion of idiomatic writing be applied to new instruments? The lack of history that new
instruments enjoy means that what is idiomatic has not yet been defined. There is a priori no codified
performance practice for a new instrument – this must be created for each new sensor instrument. The
performance vocabulary can be particular to a piece, it can also be specific to a composer’s own oeuvre. This
gives enormous freedom for a composer to create a personal voice for an instrument. Sometimes a new
performance vocabulary is created for each new piece. From the listener’s point of view, however, there is
no point of reference as key to comprehension. This places the burden of introduction, elucidation, and
development of a performance practice on the composer.
I have had the unique opportunity with the BioMuse to be at once composer for the instrument and
performer on it, an instrument that otherwise is not widely used in this context. This allowed for me
development of an extremely personal vocabulary on the instrument. Whether this establishes a more
general performance practice for the BioMuse is unsure. With Global String, its multiple uses gives us
immediate indications on the potential or a performative idiom on the instrument. It is a collaborative
project, so as the two artists perform on it, not only will our personal tendencies emerge, but a common
tendency that is identified with the instrument. As different visitors play on the installation version, the
instrument’s orientation and potential will become clear. First reactions of users are often quite useful in
identifying these potentials of an instrument.
Improvisation
In improvisation, many of the concepts of instrumental vocabulary and idiom apply, they are broached on
the spot in performance placing focus on spontaneity and response of the instrument. Despite the
limitations and well defined range of articulations associated with a traditional instrument, a traditional
instrument has a remarkable capacity of responding to spontaneous gesture on the part of the performer
with a large variation in timbre. A violinist for example can very quickly move from arco to pizzicato to sul
ponticello to col legno. Relative to the timbral range of a violin, the computer can be said to have fewer
limits, but here we encounter the problem of the ease and organic sense of moving from one timbral
variation to another and the relationship and coherence that tie them together.
Synthesizers and computer generated sounds are typically preprogrammed, the different timbres being
accessed by selecting and switching among presets. Input from a sensor, on the other hand, is often a
continuous signal. The question becomes then how to find a way to move from one timbre to another that
is consistent with gestural input. Solutions to this lie both in the sound synthesis programming on the
368
computer side and some thought given to articulative range and variation on the sensor interface. Given a
certain sensor system, the software determines the suitability for improvisational performance. Software can
be created for compositional performance, improvisation, or somewhere in between.
the piece as it is part of the instrument definition. In an improvisational context, the patches are
programmed in such a way to give the widest possible range of expression in each sound without changing
settings.
As a concert instrument, Global String is an instrument that can be played to realize a composition between
remote performers. As an installation, this same instrument must take on the capability to respond to an
improvisatory situation. The types of visitors at the installation, as well as the ways in which they will play
the instrument are unknowns. The instrument is designed to be played with a certain articulatory technique,
but the installation should not force the visitor to follow these techniques. We can only evoke a musical
image - we cannot presuppose the response of the visitors. The software taking physical input at the string
and articulating sound synthesis, then, must be general enough to respond musically to a wide range of
possible input.
Relationship
In traditional instrument performance the relationship between the musician and their instrument is highly
developed. This includes the years of time necessary to learn the instrument, establish a proficiency on it,
then finally master the instrument. This results in a dynamic relationship that goes beyond the athletics of
technique, and highlights the musical qualities of a personal interaction created between the musician and
his instrument. In this regard the relationship of a musician and his instrument are deeper than what we
expect in typical human machine interaction. A relationship with an instrument is based on the time we
spend living with it. In the accelerated pace of hi-technology development, we rarely have time to spend
with any one given hardware or software configuration before it is replaced by a new version. This is quite
contrary to the time needed to develop a deep relationship with an instrument.
If an instrument is to become a communicative vehicle for the musician, he must attain a certain fluency. In
musical instrument performance, we often talk of the state where the musician "speaks" through their
instrument. Ultimately, this produces virtuosi who combine unparalleled technical proficiency and expressive
charisma on their instrument. Creating a performer/instrument dynamic this intimate is a worthy goal in
computer instrument design. It means creating a situation where the performer can attain a level of
intuition with regard to his instrument where he is not preoccupied with every detail of its manipulation.
How to achieve this intuitive musical fluency with a technology based instrument becomes an artistic
challenge.
370
More important than the purely athletic qualities of technique, clarity and subtlety of musical delivery is an
important test of the relationship between a musician and his instrument. Compared with a sensor
instrument, subtlety is probably greater in a traditional instrument due to its history and expressive range
developed within instrumental limitations. Acoustical consistency and preexisting perceived image of the
instrument lend to clarity in performance. These qualities of an instrument not only help the audience
comprehend a performance, but help the performer to feel at ease with his instrument. The clarity of
interaction then goes beyond the simple question of parameter and event mapping of "what controls what"
to a more high-level analysis of how different gestural inputs are used to derive the different types of
articulative signals (Mulder 1996), and how different variations on these types of input affect and express
the sonic and structural space of the music.
This issue of performer-instrument relationship is a central issue in each of the projects introduced here.
With the BioMuse, this relationship is deepest – I have been performing on the instrument since 1992. While
the host computer and MAX software has been updated several times in this period, my basic approach to
the instrument has remained constant. The electrode placement was established over the course of three
years and has since stayed the same. The basic gestural vocabulary has developed in an evolutive way over
the course of different pieces and projects.
Soundnet, on the other hand, poses problems in this regard. As the instrument requires significant space and
means to set up, we have had little time to live and work with the instrument. The instrument was
developed in the abstract, then prototyped and presented in small scale. The sensor mapping and synthesis
programs were written apart from the instrument. The only opportunity to have the complete instrument
installed has been at the concert sites. This experience has been akin to composing for orchestra – one
typically does not have the means to have the full orchestra onhand for experiments during the composition
process. The composer relies on his knowledge of the components of the ensemble to create for the whole.
The performance brings the ultimate test, a realization of the abstract act.
In concert, the Soundnet exhibited a behavior and response more or less was consistent with what we had
imagined in the abstract during its development. As performers, however, we felt like beginners, trying to
make articulations on an enormously powerful instrument with little practice time. These elements, plus the
larger than life size of the instrument introduce an enormous challenge to the notion of instrumental
virtuosity. With Soundnet, the concept of control is put completely in question - the instrument is too large
and complex for humans to thoroughly master. The conflict of control vs. discontrol becomes a central
conceptual focus of Soundnet.
371
Global String is, as it has been mentioned, a multi-mode instrument. It can be used in concert or as an
installation. In concert, it will be performed upon by musicians who know the instrument. As an installation,
it will be played by gallery visitors who are encountering the instrument for the first time. These visitors may
or may not be musicians, and may or may not be sensitive to interactive or network technologies. In its
design, then, Global String must accommodate these different situations. In the hands of a trained
performer, it must be an expressive and sensitive instrument. When played by an uninitiated visitor, it must
be clear, and responsive. The instrument does not so much adapt to these different situations inasmuch as it
needs to continue to be musical across this range of use.
Feedback
The intuitive sense the performer has with his instrument gives him the "verve" that communicates to the
audience. It is a confidence in his instrument that helps the musician to create a flowing musical dynamic
that is conveyed to the audience. Of the factors that contribute to helping the musician achieve this,
feedback is an essential component. Feedback can take on several forms, but in essense is information that
gives the musician an indication or confirmation of what he has just articulated. It can be physical, tactile, or
sonic and musical. The successful integration of more than one of these modes (e.g. tactile, sonic)
contributes to creating a clear understandable coherence. While feedback is first apparent to the musician,
giving him a point of reference for what he is doing, the action/reaction feedback dynamic should also be
apparent to the audience. This gives the listeners the key to following the expressive space that the
performer is creating.
Tactile feedback in acoustical instruments is an inherent characteristic. Traditional instruments are
mechanical, and display mechano-acoustical coupling. This coupling can fold back to provide an acoustically
based form of tactile feedback to the performer. This is seen in the harmonics beating in the lips of a brass
player (Chafe 1993). With computer instruments on the other hand, mechanical action is acquired, digitized,
and mediated through logic and data processing to produce synthesized sound. Tactile feedback in the
traditional sense then must also be synthesized (Gillespie 1992). As much as it may help recreate a sought
after dynamic, it is in its very nature artificial. For it to be effective, coherence in the generated feedback
with respect to the generated sound are paramount considerations.
Other forms of physical feedback can be achieved through sonic means. A ’cello body is resonant, and
vibrates in close contact with the performer’s body, providing a valuable secondary feedback channel.
Computer instruments in general do not possess such acoustical properties. That is, they lack resonant
bodies. The acoustical space of sound diffusion, then becomes the closest thing to an "instrument body".
Coupling then takes place through the speakers of the sound system through the concert space to the body.
372
Modern sound systems are powerful enough and of sufficient quality, they can produce air pressure modes
that can set the concert space in resonance and the perceiver’s body into vibration. This is perhaps one of
the purest sense of tactile feedback we have in technology based systems.
In cases where synthesized tactile feedback or sonic pressure feedback are not possible, we resort to the
essential case of musical feedback. The physical gesture of the performer must be apparent in some resulting
musical gesture. This is important both to the musician to have a sense of what he is doing, and for the
audience to have an understanding of what is taking place. This replaces the direct physical effect of
acoustic coupling with a kind of intellectual language parsing. Independent of the channel of feedback,
representation creates the essential response from the instrument that helps the performer create his
relationship with the instrument as described above. These considerations become important in sensor
instrument design: Creating a successful performance dynamic rests not on a unidirectional control
paradigm, but on an organic set of interactions between performer and instrument.
The physical nature of the three instruments discussed here gives each of them a dimension feedback.
Although none of them have implemented explicit user feedback mechanisms, a tactile sense is apparent.
With both the BioMuse and Soundnet, it is the performer’s own body which confront the instrument, each
in its distinct way. With the BioMuse, there is a near contradiction, since there is no object of action that
provides feedback. Instead it is pure exertion fatigue in absence of an object. With Soundnet, the object of
action is completely overwhelming in scale relative to the performer.
With Global String, the hybrid physical/virtual nature of the instrument leads to a mixed feedback strategy.
The remote distance between performers creates a separation. Actuating the string at an end point with the
action of the performer on the opposite end gives a sense of telepresence. This supplements the
videoconferencing connection that provides the visual link. In view of the transmission delay for action and
sound from one side to reach the other, the performer must feel his own actions manifested immediately –
helping him distinguish his contribution to the total sound. These actions are both sonified by local synthesis
and visualized onscreen as vibrational data is displayed.
Effort
While the performer has feedback cues from this instrument, and the perception of cause/effect from
creative intention to musical result, the audience arrives at an understanding of the music through an
abstract reconstruction of this chain. One key to the audience is the manifestation of feedback as exertion of
effort on the part of the performer.
373
A problem arises when considering effort and execution on the computer. The nature of typical computer
input devices is that they take little physical force to manipulate. This results in interfaces that do not
differentiate power of articulation. Musical instruments are quite different - most instruments have a
characteristic and particular response to applied force in articulation. The response differs among different
instruments, and varies within an instrument according to range or dynamic.
Regulating exerted effort can be used in an articulatory fashion to shape expressivity in a phrase. It can also
be employed compositionally to shape the evolution of a piece. The realization of an energy trajectory can
be the basis for choices of instrumental register and timbre through a piece. The misuse of energy exertion
in gratuitous display or for theatrical effect, however, have the potential of rendering the music equally
ineffective (Ryan 1992). In fact, rules of inversion can apply. A virtuoso is often appreciated not for his
display of playing technically difficult passages, but more because he is able to play such difficult passages
with ease. A theatrical performer who plays an easy passage with an air of difficulty is bound to be less
appreciated.
Sensor instruments often lack the sophisticated inherent feedback characteristic that causes them to require
real physical effort in execution. With the absence of varying effort to achieve different articulations on an
instrument, compositional development begins to lose its dramatic sense. Artificially adding physical
obstacles to require effort would only create the undesired gratuitous effect described above. It becomes
thus a challenge for the instrument builder to conceive of coherent model for requirement or
representation of physical effort on a technology based instrument.
The BioMuse and Soundnet share a characteristic in that they are both extremely physical instruments. The
effort of the performer is apparent in performance, and is conveyed intuitively to the audience. Physical
limits of the performer's strength create interesting boundaries as the performer strains to reach a certain
level or point. These asymptotic trajectories are heard in the music, and have been successful in building a
dynamic tension in performance.
Nonessential gesture
Similar to the issue of the fine distinction between genuine physical exertion and gratuitous display, the
topic of essential vs. nonessential gesture on an instrument treads the thin line between musical artistry and
vain theatre. In any instrument, there are gestures essential to sonic production on the instrument as well as
those that are independent of the acoustic of the instrument (Delalande 1988). To understand the notion of
non-essential gesture, consider the piano as example. As a percussive instrument, the essential gesture tied
to the physics of the instrument is the striking of a key. According to the mechanics of the piano action, any
374
gesture following the strike theoretically has no acoustical effect on the sound produced by the instrument.
However, musically we cannot invalidate the practice seen among numerous pianists to dig into the keys
after the strike. Visually more evident is the raising of the hand and wrist following the attack, arching to
the next note. Here we meet the fine line between theatre and musicality. Some pianists admittedly over-
theatricize this aspect of their performance, but this superficial nature is quickly uncovered. In a musically
satisfying case, however, this moderated gesture makes a positive contribution to the musicality of the
performance. The claim is that gesture non-essential to the physics of the piano itself is not all theatre, but
musically vital element that directs musical flow, phrasing and articulation.
The first step in investigating nonessential gesture is to define the gesture that is tied to the instrument’s
mode of action - the essential gesture. From there, we can define the ancillary gestures not essential to the
direct functioning of the sensor, but that are important to the musician’s articulation on that instrument.
This issue applies most directly to the BioMuse. Although it is an instrument that sense corporeal gesture, it
does not sense gross movement per se. The actions detected are muscle tension, and not limb displacement.
This means that the instrument could be played while standing or sitting absolutely still, simply by flexing
the muscles where the electrodes are placed. However in practice, this is not so musical – much in the way a
violinist who saws mechanically at his instrument is not as musical as one who sways gracefully into certain
phrases, or a pianist who flowingly lifts his hand in anticipation of the next passage. These effects are not
purely visual, but affect the musical phrasing and articulation. In the same way, although each muscle
tension trajectory could probably be realized standing exactly still, I have found that certain trajectories are
more naturally realized with the aid of certain arm movement and gesture.
Memory
Instruments can be designed and created with a specific performance in mind (Aperghis). Instruments have
also been designed for and custom built for particular musicians. In both cases, the instrument can retain it’s
singular nature to the situation or performer, or can expand its acceptance and use beyond the scope of the
original situation. In the latter case, it is the range of articulative expression possible with an instrument and
its adaptability to new musical situations that gains a wider acceptance for a new instrument.
For the listener, comprehension of instrumental performance practice is based on association and familiarity.
The ease with which the listener parses different musical parts in a group performance is based on their
prior knowledge of the instruments making up the ensemble. New ensemble combinations or musically new
375
compositions can be understood based on the listener’s existing knowledge of the sound of each
instrument. This association and memory also provide keys to comprehension as the listener listens to a
recording in the absence of visual cues.
The question arises then of how to establish a base of association and knowledge when the instrument is
completely new to the listener. What are the keys to understanding unfamiliar instruments? The most
straightforward answer is history, consistency, and repetition. Repeated performances or different pieces
making consistent use of an instrument establish in the listener a cognitive memory that can be applied to
appreciating a new piece or the use on the same instrument or to distinguish that instrument from others.
Solutions can be compositional – an instrument can be presented in an informative way in the piece itself,
though composers should not be reduced to creating educational demo pieces.
The field of computer music is already one where the propos is that we are creating a new universe of sound
for each piece. The audience has little associative sonic memory between pieces at their disposition as an aid
to understanding the music. One thing that sensor instruments potentially afford is an associative element
to unfamiliar computer generated sounds, a kind of visual key. To fully serve this potential, the composer’s
responsibility becomes to develop a performance vocabulary on the new instrument. The success of such a
performative language particular to an instrument is contingent on the coherence and clarity with which
performance gesture is tied to the musical result that is heard (Choi 1998). Part of Sensorband's approach as
a sensor instrument ensemble has been to work with this compositionally - at moments creating clarity
where the three instruments are differentiated, and at moments confusing this differentiation - to construct
a musical dramaturgy.
Communication
a musical instrument. The focus then should perhaps not be on how much a computer allows us to do, but
on how we can begin to approach the kind of fluid communicative way we have with acoustic instruments.
This is not to say that we should necessarily create computer emulations of existing instruments. Rather, my
goal has been to seek an organic interaction with the machine, to establish a conversational exchange with
the instrument. Seen in this light, emphasis is placed not on power and control, but on clarity and fluidity.
This places an emphasis of the musician/instrument dynamic on expression, articulation, and spontaneity.
These are qualities we seek to achieve with any musical instrument – be it technological or not. We can draw
from traditional instrument performance practice to offer valuable guidance in developing a musical
language with technology based musical instruments.
Evaluation of an instrument is often made from the action standpoint, that is, judging the articulatory fit
and satisfaction for the performer. The evaluation can however also take place from the reception
standpoint, viewing the situation from the regard of the perception of the listener (Iazzetta 2000, in this
volume). The success of an instrument in both regards indicates a high potential for the creation of a
musically communicative experience. A traditional instrument benefits from its consistent nature of sound
production for subtlety, and benefits from established image for clarity. The subtlety and clarity perceived is
dependent on how gesture types are associated with resulting musical output. Different types of
articulation can be associated with timbral, event, and structural modes of expression. These types of
articulation can be derived from the nature of the sensor itself or be defined in software.
The issues discussed in the previous sections - idiomatic composition, feedback, memory, all contribute to
establishing a communication channel between performer and listener. In this new field, these become
important considerations in instrument design, composition, and performance. Although the BioMuse is
capable of sensing brainwaves and detecting eye movement, I have chosen only to use muscle signals.
Generating music from the brain is a fascinating topic (Rosenboom 1990), but one that in its very nature
better adapted for personal experience than for performance. Transition between brain states provide no
visual cue for the audience to connect with. Although eye movement has a great potential for gesture-music
mapping, the ocular channel is also difficult to exploit in performance. While eye movement is quite evident,
it is a rather fine gesture that may be difficult to perceive at a distance. More importantly, there are many
involuntary movements of the eyes that risk to cause artifacts.
Conclusion
Gestural, or sensor-based instruments is an inclusive term that encompasses a wide range of related musical
instruments. Here in this text, we have created a broad definition of a musical instrument as a basis for more
specific investigation. Traditional acoustical instruments are used as a model, not in any specific sense, but as
377
a guide to understanding the composer performers' relationship with an instrument. The definition of
musical instrument as proposed here applies to all musical instruments, whether acoustic or computer-based.
This is used as a point of departure to introduce three sensor-instrument projects I have worked with over
the past several years.
Instruments are regarded as systems that respond musically to performer action. They take some form of
physical intervention by the musician as the articulative component to producing music. Specific details
include the locus of the performer’s action, the manner in which it is transduced, and the destination where
this is musically manifested. This process of transformation becomes the defining characteristic of an
instrument.
Beyond the purely technical description of an instrument, important human factors arise. In this way,
instruments are distinguished from tools. These may be subjective from the standpoint of the performer or
audience, and deal with not just performer-instrument interaction but the role of the instrument in
performer-audience communication. Seen in this light, the successful design of a sensor-based musical
instrument is the result of a fusion of computer-human interface design and acoustic instrument lutherie.
If the limitations of an acoustical instrument contribute to characteristics giving it a personality or identity,
the open-ended nature of computer based instruments at first preclude this kind of identity. The separation
of gestural input from sound production means that here is no hardwired gestural vocabulary connecting
action to sound. This connection is created in software — bringing programming into the realm of creative
musical instrument building.
The relationship that the performer creates with his instrument becomes the source of musical
communication. The relationship is nurtured by responsive qualities of the instrument such as feedback.
Fluency, coherence and clarity are seen as essential criteria. The classical model of a performer's relationship
with an acoustic instrument is seen as a valuable model, however this is not intended to establish a
traditionalist or reactionary stance. The simple emulation or imitation of existing instruments is not what is
advocated, but rather the application of these rich traditions to new technology.
The use of new instruments open possibilities of performance integrating other media. The concept of
presentation can be expanded beyond the classical concert to embrace new spaces and modes of interaction.
Applying the instrumental concept to these situations, they can be treated in a musical way. It is argued,
however, that the focus should remain on the efficiency of gestural communication and expression over
techno-wizardry.
378
We are faced with a dilemma – computer sound synthesis gives us an enormous range of parametrized
control. Sensor devices give us a freedom from mechanical and acoustical constraints. Despite this, an
acoustic instrument is capable of responding to a player’s expressive subtlety in a way we can only hope to
approach with new instruments. And as much as the computer allows an unlimited sonic palette, an
acoustical instrument is extremely rich in the range of related and articulated timbres it can produce.
With each new instrument, a balance must be found between the expansive possibilities and the
unfamiliarity of newness. While the lack of preconceived association give the composer enormous freedom,
it also places the burden of educating and informing the audience on the composer. A performative
vocabulary and memory must be developed for each new instrument, sometimes even within the course of
one piece. By doing so, the composer helps the performer to establish the clarity and coherence of gesture,
and provides the audience with keys to musical comprehension.
References
Bailey, N. J., A. Purvis, I. W. Bowler and P. D. Manning. 1993. "Applications of the Phase Vocoder in the Control
of Real-time Electronic Musical Instruments." Interface 22:259-275.
Bongers, B. 1998. "An Interview with Sensorband." Computer Music Journal 22(1): 13-24.
Broeckmann, A. 1997. "Remove the Controls." The Beauty and the East (nettime, ed.) ZKP4 Ljubljana. http://
[Link]/nettime/zkp4/[Link].
Chafe, C. 1993. "Tactile Audio Feedback." Proceedings of the International Computer Music Conference, San
Francisco: International Computer Music Association, pp. 76-79.
Choi, I. 1998. "From Motion to Emotion: Synthesis of Interactivity with Gestural Primitives." Emotional and
Intelligent: The Tangled Knot of Cognition, AAAI Fall Symposium, Orlando FL. (October):22-25.
Delalande, F. 1988. "La gestique de Gould: éléments pour une sémiologie du geste musical." Glenn Gould
Pluriel (ed. G. Guertin). Verdun Quebec: Louise Couteau.
379
Gillespie, B. 1992. "The Touchback Keyboard." In Proceedings of the International Computer Music
Conference, San Francisco: International Computer Music Association, pp. 447-448.
Goldstein, M. 1998. "Gestural Coherence and Musical Interaction Design." In Proc. IEEE Systems, Man and
Cybernetics Conf., San Diego, Calif.
Katayose, H., T. Kanamori, S. Simura and S. Inokuchi. 1994. "Demonstration of Gesture Sensors for the
Shakuhachi." In Proceedings of the International Computer Music Conference, San Francisco:
International Computer Music Association, pp. 196-199.
Knapp, R. B., and H.S. Lusted. 1990. "A Bioelectric Controller for Computer Music Applications." Computer
Music Journal 14(1): 42-47.
Krefeld, V. 1990. "The Hand in the Web: An Interview with Michel Waisvisz." Computer Music Journal 14(2):
28-33.
Laurel, B. (ed.). 1990. The Art of Human-Computer Interface Design. New York: Addison-Wesley.
Lucier, A. [Link]
Lusted, H. S. and R. B. Knapp. 1996. "Controlling Computers with Neural Signals." Scientific American
(October):58-63.
Mulder, A. 1996. "Hand Gestures for HCI." Technical Report 96-1. Simon Fraser University.
Nagashima, Y. 1998. "Biosensorfusion: New Interfaces For Interactive Multimedia Art." In Proceedings of the
International Computer Music Conference, San Francisco: International Computer Music Association,
pp. 129-132.
Pressing, J. 1990. "Cybernetic Issues in Interactive Performance Systems", Computer Music Journal. 14(1): 12-25.
Puckette, M., and D. Zicarelli. 1990. MAX – An Interactive Graphic Programming Environment. Menlo Park,
Calif.: Opcode Systems.
380
———, and Z. Settel. 1993. "Nonobvious Roles for Electronics in Performance Enhancement." In Proceedings of
the International Computer Music Conference 1993. International Computer Music Association, pp. 134-
137.
Putnam, W. 1993. "The Use of the Electromyogram for the Control of Musical Performance." Masters Thesis,
San Jose State University, Calif.
Rosenboom, D. 1990. Extended Musical Interface with the Human Nervous System. Berkeley, Calif.:
International Society for the Arts, Sciences, and Technology.
Ryan, J. 1992. "Effort and Expresssion." In Proceedings of the 1992 International Computer Music Conference.
San Francisco, International Computer Music Association, pp. 414-416.
Strange, A. 1983. Electronic Music: Systems, Techniques, and Controls. Dubuque, Iowa: Wm. C. Brown.
Tanaka, A. 1993. "Musical Technical Issues in Using Interactive Instrument Technology." In Proceedings of the
International Computer Music Conference, San Francisco: International Computer Music Association,
pp. 124-126.
———. 1999. "Network Music: A Perspective." Festival du Web. Paris: Web Art.
Tanaka, A., and K. Toeplitz. "Global String: A Network Musical Instrument Installation and Performance
Environment." Web site: [Link]
Vertegaal, R., T. Ungvary and M. Kieslinger. 1996. "Towards a Musician’s Cockpit: Transducers, Feedback and
Musical Function." In Proceedings of the International Computer Music Conference, San Francisco:
International Computer Music Association, pp. 308-311.
Wanderley, M., N. Schnell and J. Rovan. 1998. "ESCHER – Modeling and Performing Composed Instruments in
real-time.", Proceedings of the IEEE Systems, Man and Cybernetics Conference, San Diego, Calif.
Warner, D., J. Sale, T. Anderson and J. Johanson. 1995. "BIO-CYBERNETICS: A Biologically Responsive Interface."
In Adventures In The Next Paradigm Of Human Computer Interaction.
381
Wessel, D., D. Bristow and Z. Settel. 1987. "Control of Phrasing and Articulation in Synthesis." In Proceedings
of the International Computer Music Conference, San Francisco: International Computer Music
Association, pp. 108-116.
Zannos, I., P. Modler, and K. Naoi. 1997. “Gesture Controlled Music Performance in a Real-Time Network.” In
Proceedings of KANSEI - The Technology of Emotion Workshop, pp. 60-63.
Zicarelli, D. 1992. "Music Technology as a Form of Parasite." In Proceedings of the International Computer
Music Conference, San Francisco: International Computer Music Association, pp. 69-72.