SONOMATERIA
Audio-tactile Composition

Irad Lee

ABSTRACT

This paper describes the inspiration and implementation of SONOMATERIA, a multi-user sound sculpture, installation, tangible sound interface and intersensory composition. The work aims to explore the mutual reinforcing effect that the manipulation of tactile and auditory perceptions can have on each other in the context of art and design. An artistic system reflecting and representing the sonic properties of complex physical materials is described. Relevant works from disciplines including Sound Design, Psychoacoustics, Sonification, Interaction Design, and Tangible User Interface Design are considered.

1. BACKGROUND

Sound and touch have a strong relationship that dates back to the very origins of primitive music. Throughout the history of music, musicians and performers have had intimate physical relationships with their instruments. In the intimate interaction loop between performer and instrument, feedback had always a fundamental role. Next to visual and aural feedback, performers rely on haptic/tactile feedback, which is a vital element for playing or controlling most musical instruments. The Ability to interpret cues received in the form of tactile feedback while playing a musical instrument suggests that there is a pre-existing link in our minds between (musical) sounds and tactile sensations.

In his book "The Psychology of Music" [1], James Mursell remarks that sound timbre is found to have very definite and consistent tactile attributes, and indeed, we tend to speak of sound in a similar way we describe tactile sensations: e.g. hard and soft, rough and smooth. We tend to describe high-pitched sounds as "Sharp", low frequency sounds as "Heavy", "Fat" or "Full". Furthermore, extensive psychophysical research looking into the links between sound and touch perception indicated that sonic information influences touch perception in highly systematic ways. Studies have shown that certain types of sounds can affect some aspects of touch perception and that touch alone can also have great affect on sound perception [2]. For example, the sensation of "chills" often experienced in response to an emotionally charged piece of music, and the "sharpness" we may feel from the sound of fingernails scratching the wall or the sound of a breaking glass.

These strong associations between sound and touch and the mysterious and rather unexplored psychological rules governing the interrelationships between these two different sensory modalities, were the main motivating force for creating an artistic medium which could be used as a vehicle for conveying art in a sono-tactile form. Namely, communicating art to the spectator via a hybrid sensory channel which operates during the interaction between the sense of touch and hearing. In order to exploit the potential of these ideas, an artistic experiment was designed based around a system comprised of a tangible interface and a sono-tactile composition.

2. PROJECT DESCRIPTION SONOMATERIA takes the form of a tangible interface that allows spectators to manipulate digital sound via tactile interaction with a selection of physical objects made of distinct materials. It is a tactile composition interconnected with a dynamic sound palette, inviting multiple spectators to navigate through, re-arrange, deform and manipulate a pre-existing sound composition. The interface responds with tactile and sound feedback, attempting to position the spectator in a loop of intersensory synesthesia. The work draws upon and extends surrealist cut-up techniques popularized in the late 1950s by writer Willam S. Burroughs. The cut-up technique is an aleatory literary technique in which a text (or multiple texts) is cut up into smaller portion at random, and rearranged to create a new text. In SONOMATERIA, this technique is extended from the literary domain to the musical domain, and instead of words, big portions of sound are cut down to small fragments. The audience is invited to interact with the fragments via a tangible interface that invokes tactile sensations, and thereby access an intersensory composition coded into a hybrid of touch and sound.

2.1 TANGIBLE INTERFACE

SONOMATERIA’s tangible interface is comprised of a black cubic table (85cm height, 97x97cm width) with inner illumination and a tabletop surface made of translucent acrylic. Placed on the edges of the tabletop are 16 cubes of identical size (6x6x6cm) and color (black), each made of a distinct type of physical material: Polystyrene, Coal, Cork, Wood, Industrial Rubber, Clay, Sponge, Stone (Granite), Concrete, Silicone, Wax, Synthetic Leather, Vinyl, Gaffer Tape, Iron, and Plasticine. Surrounding the cubic table-top are four loudspeakers arranged in quad configuration. The audience is invited to place cubes in the illuminated part of the tabletop surface. Using object tracking techniques (discussed further in section 3), when a cube gets in contact with the surface, it triggers a random sound taken from a sound library assigned to the particular material the cube is made of. For example, a cube made of wood will trigger sounds from the wood sound library, etc.

Quality of sounds in these libraries are related to and reflect the properties of each material, considering its texture, weight and unique tactile quality (discussed further in section 2.2). The position of a cube and its rotational angle are linked to parameters that manipulate the attributes of sounds and their spatial distribution in the installation space. Multiple users can place cubes on the surface simultaneously, limited only by the number of cubes available, causing sounds to mix with each other. The project is intended to be presented in a darkened space, and the consistent use of black color and uniform cubic shape is meant to subordinate the visual field as much as possible, and encourage focus of attention on auditory and tactile perception.

(above) 3D model of SONOMATERIA installation setup. (right) Position of cubes on the surface is proportional to the surround spatial distribution of sounds in the installation space.

2.2. SONIC COMPOSITION & BEHAVIOR

The audible part of SONOMATERIA is aimed at exploring the musical manifestation of touch. A library of sounds was designed for each material, containing a selection of different sounds reflecting and responding to the properties and relative tactile qualities of each particular material. The challenge here was to find a sonic language that could represent the various types of materials and simultaneously take a position towards the spectator via communicating a story in the form of tactile and sonic cues. Contemporary findings in the fields of psychoacoustics and auditory perception [3, 4] were extensively used as guidelines in the compositional process. All sound elements were carefully tuned on the basis of theories concerned with measurements of emotional and sensational responses to sound and the relationship between psycho-acoustical sound descriptions and sensations.

2.3. INTERACTION DESIGN

Considering the unique properties of each material, each cube reveals a different sound behavior when in contact with surface. Acting as an individual instrument, each cube has a sound behavior that aims to make contact with the tactile sensation felt by the spectator in the very same moment – the feeling of a particular material in his hand. For example, while in contact with the surface, a cube made of industrial rubber having a rough texture, would trigger a hectic sound reaction when touched, displaying sounds with "rough" qualities yet with "flexible" and "elastic" attributes that correlate with the relative tactile quality of the material. The interaction experience is aimed at manipulating the border (or hierarchy) between two sensory channels for artistic expressions: is the sound accompanying the tactile sensation or the other way around? In its ultimate effect, the interaction experience will position the spectator as a conductor of intersensory dialogue between sound and touch. The idea of this interaction model was inspired by the study of sensory substitution systems – an umbrella name for emerging technologies that gather environmental energy which would normally be processed by one sensory system and translate this information into stimuli for another sensory system.

3D model - SONOMATERIA cube tagged with a fiducial ID marker

3. HARDWARE The system is comprised of two computers connected via network, some fluorescent lamps for diffused inner illumination, a Sony PS3eye USB camera for object tracking and a Firewire audio interface with four outputs. Each cube has a unique pattern attached to its bottom, making it trackable by a special software for pattern recognition and tracking.

4. SOFTWARE

The tracking software is based on reacTIVision [5], an open-source framework for the recognition of objects tagged with so-called "fiducial markers" (visual ID markers). A camera mounted inside the table captures these fiducial markers from underneath, tracking the position and rotation of each cube. This information is encoded using the TUIO protocol [6] and sent over using OSC protocol to SONOMATERIA sound generation software. The software is written in Max/MSP, a visual programming language for audio and multimedia, and was designed to translate object tracking data to control parameters in a custom set of sound processing tools. The software incorporates several kinds of digital audio processing techniques including pitch-shifting, sample rate reduction, filtering, tempo control, time-stretching and more. Parameters controlling these manipulation techniques are accessed directly via moving and rotating cubes on the tabletop surface.

screenshot of SONOMATERIA sound generation software in action

5. STATUS AND FUTURE WORK

The current setup presents a composition combining static, "non- interactive" tangible interface next to dynamic, moving sound. While SONOMATERIA effectively embodies an intersensory composition for sound and touch, yet much is to be improved and further explored in terms of tactile feedback. A main feature the current form of the work lacks is direct haptic/tactile reaction and stimulation from the materials themselves (e.g. controlled vibrations, electronic tactile stimulators), which could potentially be synchronized with sound. Sound spatialization was introduced into SONOMATERIA as an attempt to somewhat compensate on the lack of tangible feedback, giving the spectator the option to distribute sounds in the installation space and thereby around his body.

Evidences from psychophysics research suggests that our skin can comprehend an expressive language of communication comparable to the sonic language of sound and music. One of the next steps in this project is to experiment with configurations that incorporate synchronized or simultaneous sound and tactile feedbacks. The reader is asked to imagine musical interfaces that stimulate a tactile feeling corresponding to the sound they produce. Consider a futuristic musical instrument consists of a smart transformable material which could shift from one physical state to another via tangible user control, and vary the quality of sound it produces according to its physical state in any given moment.