"Words are very much like
lizards, they change colour according to position"
(Lafcadio Hern: Talks to
This paper describes the inspiration and implementation of SONOMATERIA,
a multi-user sound sculpture, installation, tangible sound interface
and intersensory composition. The work aims to explore the mutual
reinforcing effect that the manipulation of tactile and auditory
perceptions can have on each other in the context of art and design.
An artistic system reflecting and representing the sonic properties
of complex physical materials is described. Relevant works from
disciplines including Sound Design, Psychoacoustics, Sonification,
Interaction Design, and Tangible User Interface Design are considered.
Sound and touch have a strong relationship that
dates back to the very origins of primitive music. Throughout
the history of music, musicians and performers have had intimate
physical relationships with their instruments. In the intimate
interaction loop between performer and instrument, feedback had
always a fundamental role. Next to visual and aural feedback,
performers rely on haptic/tactile feedback, which is a vital element
for playing or controlling most musical instruments. The Ability
to interpret cues received in the form of tactile feedback while
playing a musical instrument suggests that there is a pre-existing
link in our minds between (musical) sounds and tactile sensations.
In his book "The Psychology of Music"
, James Mursell remarks that sound timbre is found to have
very definite and consistent tactile attributes, and indeed, we
tend to speak of sound in a similar way we describe tactile sensations:
e.g. hard and soft, rough and smooth. We tend to describe high-pitched
sounds as "Sharp", low frequency sounds as "Heavy",
"Fat" or "Full". Furthermore, extensive psychophysical
research looking into the links between sound and touch perception
indicated that sonic information influences touch perception in
highly systematic ways. Studies have shown that certain types
of sounds can affect some aspects of touch perception and that
touch alone can also have great affect on sound perception .
For example, the sensation of "chills" often experienced
in response to an emotionally charged piece of music, and the
"sharpness" we may feel from the sound of fingernails
scratching the wall or the sound of a breaking glass.
These strong associations between sound
and touch and the mysterious and rather unexplored psychological
rules governing the interrelationships between these two
different sensory modalities, were the main motivating force
for creating an artistic medium which could be used as a
vehicle for conveying art in a sono-tactile form. Namely,
communicating art to the spectator via a hybrid sensory
channel which operates during the interaction between the
sense of touch and hearing.
In order to exploit the potential of these
ideas, an artistic experiment was designed based around
a system comprised of a tangible interface and a sono-tactile
2. PROJECT DESCRIPTION
SONOMATERIA takes the form of a tangible
interface that allows spectators to manipulate digital sound
via tactile interaction with a selection of physical objects
made of distinct materials. It is a tactile composition
interconnected with a dynamic sound palette, inviting multiple
spectators to navigate through, re-arrange, deform and manipulate
a pre-existing sound composition. The interface responds
with tactile and sound feedback, attempting to position
the spectator in a loop of intersensory synesthesia. The
work draws upon and extends surrealist cut-up techniques
popularized in the late 1950s by writer Willam S. Burroughs.
The cut-up technique is an aleatory literary technique in
which a text (or multiple texts) is cut up into smaller
portion at random, and rearranged to create a new text.
In SONOMATERIA, this technique is extended from the literary
domain to the musical domain, and instead of words, big
portions of sound are cut down to small fragments. The audience
is invited to interact with the fragments via a tangible
interface that invokes tactile sensations, and thereby access
an intersensory composition coded into a hybrid of touch
2.1 TANGIBLE INTERFACE
SONOMATERIA’s tangible interface is comprised
of a black cubic table (85cm height, 97x97cm width) with inner
illumination and a tabletop surface made of translucent acrylic.
Placed on the edges of the tabletop are 16 cubes of identical
size (6x6x6cm) and color (black), each made of a distinct type
of physical material: Polystyrene, Coal, Cork, Wood, Industrial
Rubber, Clay, Sponge, Stone (Granite), Concrete, Silicone, Wax,
Synthetic Leather, Vinyl, Gaffer Tape, Iron, and Plasticine. Surrounding
the cubic table-top are four loudspeakers arranged in quad configuration.
The audience is invited to place cubes in the illuminated part
of the tabletop surface. Using object tracking techniques (discussed
further in section 3), when a cube gets in contact with the surface,
it triggers a random sound taken from a sound library assigned
to the particular material the cube is made of. For example, a
cube made of wood will trigger sounds from the wood sound library,
Quality of sounds in these libraries are related
to and reflect the properties of each material, considering its
texture, weight and unique tactile quality (discussed further
in section 2.2). The position of a cube and its rotational angle
are linked to parameters that manipulate the attributes of sounds
and their spatial distribution in the installation space. Multiple
users can place cubes on the surface simultaneously, limited only
by the number of cubes available, causing sounds to mix with each
other. The project is intended to be presented in a darkened space,
and the consistent use of black color and uniform cubic shape
is meant to subordinate the visual field as much as possible,
and encourage focus of attention on auditory and tactile perception.
(above) 3D model of SONOMATERIA
installation setup. (right) Position of cubes on the surface
is proportional to the surround spatial distribution of
sounds in the installation space.
2.2. SONIC COMPOSITION & BEHAVIOR
The audible part of SONOMATERIA is aimed at exploring
the musical manifestation of touch. A library of sounds was designed
for each material, containing a selection of different sounds
reflecting and responding to the properties and relative tactile
qualities of each particular material. The challenge here was
to find a sonic language that could represent the various types
of materials and simultaneously take a position towards the spectator
via communicating a story in the form of tactile and sonic cues.
Contemporary findings in the fields of psychoacoustics and auditory
perception [3, 4] were extensively used as guidelines in the compositional
process. All sound elements were carefully tuned on the basis
of theories concerned with measurements of emotional and sensational
responses to sound and the relationship between psycho-acoustical
sound descriptions and sensations.
2.3. INTERACTION DESIGN
Considering the unique properties of each material,
each cube reveals a different sound behavior when in contact with
surface. Acting as an individual instrument, each cube has a sound
behavior that aims to make contact with the tactile sensation
felt by the spectator in the very same moment – the feeling
of a particular material in his hand. For example, while in contact
with the surface, a cube made of industrial rubber having a rough
texture, would trigger a hectic sound reaction when touched, displaying
sounds with "rough" qualities yet with "flexible"
and "elastic" attributes that correlate with the relative
tactile quality of the material. The interaction experience is
aimed at manipulating the border (or hierarchy) between two sensory
channels for artistic expressions: is the sound accompanying the
tactile sensation or the other way around? In its ultimate effect,
the interaction experience will position the spectator as a conductor
of intersensory dialogue between sound and touch. The idea of
this interaction model was inspired by the study of sensory substitution
systems – an umbrella name for emerging technologies that
gather environmental energy which would normally be processed
by one sensory system and translate this information into stimuli
for another sensory system.
3D model - SONOMATERIA cube
tagged with a fiducial ID marker
The system is comprised of two computers
connected via network, some fluorescent lamps for diffused
inner illumination, a Sony PS3eye USB camera for object
tracking and a Firewire audio interface with four outputs.
Each cube has a unique pattern attached to its bottom, making
it trackable by a special software for pattern recognition
The tracking software is based on reacTIVision
, an open-source framework for the recognition of objects tagged
with so-called "fiducial markers" (visual ID markers).
A camera mounted inside the table captures these fiducial markers
from underneath, tracking the position and rotation of each cube.
This information is encoded using the TUIO protocol  and sent
over using OSC protocol to SONOMATERIA sound generation software.
The software is written in Max/MSP, a visual programming language
for audio and multimedia, and was designed to translate object
tracking data to control parameters in a custom set of sound processing
tools. The software incorporates several kinds of digital audio
processing techniques including pitch-shifting, sample rate reduction,
filtering, tempo control, time-stretching and more. Parameters
controlling these manipulation techniques are accessed directly
via moving and rotating cubes on the tabletop surface.
screenshot of SONOMATERIA sound
generation software in action
5. STATUS AND FUTURE WORK
The current setup presents a composition combining
static, "non- interactive" tangible interface next to
dynamic, moving sound. While SONOMATERIA effectively embodies
an intersensory composition for sound and touch, yet much is to
be improved and further explored in terms of tactile feedback.
A main feature the current form of the work lacks is direct haptic/tactile
reaction and stimulation from the materials themselves (e.g. controlled
vibrations, electronic tactile stimulators), which could potentially
be synchronized with sound. Sound spatialization was introduced
into SONOMATERIA as an attempt to somewhat compensate on the lack
of tangible feedback, giving the spectator the option to distribute
sounds in the installation space and thereby around his body.
Evidences from psychophysics research suggests
that our skin can comprehend an expressive language of communication
comparable to the sonic language of sound and music. One of the
next steps in this project is to experiment with configurations
that incorporate synchronized or simultaneous sound and tactile
feedbacks. The reader is asked to imagine musical interfaces that
stimulate a tactile feeling corresponding to the sound they produce.
Consider a futuristic musical instrument consists of a smart transformable
material which could shift from one physical state to another
via tangible user control, and vary the quality of sound it produces
according to its physical state in any given moment.
 Mursell, James L. The Psychology of Music. W.W. Norton and
Company, Inc., New York, 1937
 McAdams, S. 1984. "The Auditory Image: A Metaphor for
Musical and Psychological Research on Auditory Organization",
in Cognitive Processes in the Perception of Art,edited by W. R.
Crozier, A. J. Chapman (Elsevier, Amsterdam), pp. 63-87
 McAdams, S. and Saariaho, K. 1985. "Qualities and functions
of musical timbre", Proc. 1985 Int. Computer Music Conf.,
edited by B. Truax, Computer Music Association, San Francisco.
 McAdams, S. 1993. "Recognition of sound sources and events".
in Thinking in Sound: The Cognitive Psychology of Human Audition,
edited by S. McAdams & E. Bigand (Oxford University Press:
Oxford), pp. 146-198
 M. Kaltenbrunner and R. Bencina. reacTIVision: a computer-vision
framework for table-based tangible interaction. In Proceedings
of the 1st international conference on Tangible and embedded interaction,
pages 69–74. ACM New York, NY, USA, 2007
 M. Kaltenbrunner, T. Bovermann, R. Bencina, and E. Costanza.
TUIO: A protocol for table-top tangible user interfaces. Proc.
of the The 6th Int’l Workshop on Gesture in Human-Computer
Interaction and Simulation, 2005.