Visual Rhythms is a collaborative project between Simon Longo and Max Schleser. In this article the two creative practitioners intend to explore the transversal synergy between sound and video, placing the Bergsonian concept of intuition at the basis of their creative discovery. The section Transduction and Transmateriality in contemporary live AV practice will outline the latter creative concepts by means of exploring the work of three international live performers. The Visual Process section further elaborates on Henri Bergson through Dziga Vertov's interval theory and the Audio Process section outlines sound as an oscillatory mode. Visual Rhythms utilises digital technology and electronic media creating a temporal artistic action. The live performance can be described as a constantly changing transversal process. The audio-visual interaction creates synergies, which materialise to a new experience during the performance.
This article will explore the processes that lead to convergence within this experience. In numerous performances over the last five years (see www.visual-rhythms.net) similar patterns were realised. Sound in these performances is explored as a system of interactive energetic fields. Here these will be discussed in relation to holistic theories of perception (Lehar, 2003) and theories of information (Eco, 1962). Within the context of the performance, sound and video are understood as an oscillatory field of energy and information Within this field, the relationship between audio and visual is a non-linear, non-hierarchical microcosm, in an audio-visual context full of transversal lines (Rauning, 2010). The two modalities of aural and visual experience are understood as distinct and separate entities, which formulate transversal relationships in the synthesis of a performance "space across" cutting across the various fields involved. Drawing upon the philosophical approach of Henri Bergson, intuition is taken to be the creative currency (Bergson, 1907: 332) in the performance.
The performances' dynamic interactions of sound and visuals create rhythms of their own. The article will explore the collaborative effort involved in creating and engaging with these rhythms. It will also explore how the environment influences the work. The visual composition is based on the rhythm integral to the video itself. Noise becomes information and energy, and this is also echoed in the visual layer. The result is the construction of a kind of stroboscopic movement (Arnheim, 2004: 372). This synergy of the visual and aural channels becomes the catalyst that triggers an experience that arises within the experimentation with patterns of sensory information, digital technology and the transversal perception produced by the simultaneous fluctuation of the visual and aural rhythms.
In the Visual Rhythms performances all audio-visual material is collected by the creatives themselves. The performance is not planned or scripted.
Transduction and Transmateriality in contemporary live AV practice
Writing in Transduction, Transmateriality and Expanded Computing, Mitchell Whitelaw describes a transducer as 'a device that converts one kind of energy to another' (Whitelaw, 2009). Furthermore, he refers to computers in a transmaterial perspective as machines for shifting patterns through time and space. Referring to Youngblood's expanded cinema, the term expanded computing is proposed. Computers and media 'transduce anything to anything else' and in doing so create new media experiences. In parallel to this argument, Warwick Mules refers to 'contact aesthetics' (Mules, 2006: online), which bring new things into life by undoing the material of already constituted objects and reconfiguring formal arrangements. Live AV performances have these qualities of reconfiguration, and of bringing new life into things via the mediation process.
Live AV performance can provide a particularly useful framework in which to analyse the operations of transduction and the transversal. The live AV performances mentioned in this article are understood as experiments that explore the boundaries of audio-visual media. Live AV performance also opens up a space for a debate that can be explored through the critical lens of transversal theoretical frameworks. This article will engage with the work of three prominent practitioners; Karen Curley, Camille Baker and Ana Carvalho, whose projects will be seen to amplify our own investigations. The VJ Theory project is crucial to such a discussion. It has facilitated intensive discussion of philosophical and theoretical approaches to Vjing and real-time interaction. Writing about the VJ Theory project in Vague Terrain, Ana Carvalho reflects on the hybridities and crossovers involved, stating that 'every performer or collective of performers is unique and should be taken forwards into new areas, as for example in the construction of theory that informs and is informed by practice' (Carvalho, 2010).
As well as her theorising of VJ practice, Ana Carvalho performs her work internationally under the artist name Magenta Interior. Both her theoretical and practical work are collaborative and both explore technologies from a transversal perspective. One example of the collaborative work is the book VJing, created by 375 Wikipedians at the Mapping Festival in 2010. An example of her practical work as a live performer can be found from 2005/6, when Carvalho explored Photoshop as a performance instrument, constructing images in real time as a kind of transversal event (Tyzlik-Carver, 2006). Magda Tyzlik-Carver reviewed Carvalho's work in the article 'The "new primitive" methods in the age of digital technologies', writing
Her most recent work in the "VJ" arena uses even more primitive techniques. As large corporations produce more and more "off the shelf" real time products, Carvalho has adopted elements of early video art approaches, rejecting the use of video images in favour of mixing colour and form live. The effect is very different to the usual fast-cutting, filtered graphic images which often can be seen in clubs. (Tyzlik-Carver, 2006: online).
Carvalho herself sees her work as a dialogue between the audio and the visual, as much as between the producer and the audience. In these interactive performances the process shifts into the foreground and the tools and technologies become part of the work. In the 2008/2009 performances titled Bikinikill, turntables are used to create visuals, with the movements involved transduced into a projection.
Writing in '"Liveness" and "presence" in bio-networked mobile media performance practices: emerging perspectives', Camille Baker outlines the MindTouch mobile VJ project. This project uses bodily rhythms and data (breath flow, heart rate, skin response) from biofeedback sensors, to create a live AV performance. The body becomes a kind of interface. The data is then transmitted via mobile devices. Consequently, MindTouch is said to create a collective consciousness through a participatory engagement that is simultaneously an experience of liveness and presence. The live AV performance becomes an exchange or sharing of 'dream imagery, feelings and sensations'. As part of this project, videos are collected using camera phones in workshops, during which participants are asked to video 'dream states and relaxation; non-verbal emotional and affective senses and states; internal sensations and embodiment' (Baker 2010: 120). These videos are assembled in a database and are streamed back to mobile devices responding to the body data.
The phones in this project act as controllers, or essentially as the computer processor, as well as a receiver and visual performance mixer taking the biofeedback data from the body via the biofeedback sensors. (Baker 2010: 120)
The video is generated, and flows through a number of local and remote devices, in a transversal process. Baker states that,
… physical presences and sensations…are intended to be transformed or transduced into a digital form and used to 'touch' and 'play' with others, remotely or non-locally, using their mobile media devices to represent their presence in a live networked context. (Baker 2010: 122)
It can be seen from the briefly described workflow that the project also brings 'meta-concepts' into a transversal perspective, via technical operations. These are concepts such as 'affect' (Massumi, 2002), 'embodiment' (Rettie, 2005) or 'presence and liveness' (Auslander, 1999).
We can now turn to the work of Karen Curley and Servando Barreiro. In their live AV performance entitled [i/o], both visuals and sound are created through an organic process, using gestural and tactile interfaces. Water, ink and oil are mixed in a glass bowl to create abstract fluid images, which are projected on a large scale (using direct camera output). These are then manipulated in real-time by means of a direct light source and then scanned via a bespoke optical sensor system to trigger responsive soundscapes. Utilising open source hardware and software technology called 'Minitronics', which was developed by Barreiro, 'visuals become sonified and sound is visualized'. The technical development of this visual sonification experiment using optical sensors is documented online (http://www.vimeo.com/1660290). The collaborators describe the performance that results as a performative conversation, during which the interfaces transduce various forms of materiality, generating the AV material through gestural, optical and modular control techniques. The tangible interfaces aesthetically expand the capacities of digital video, and through their fluid projection, new means to produce a responsive live performance are created. In this context transduction is processed through a number of digital and analogue input/output devices. Karen Curley and Servando Barreiro refer to their live AV performance as
… a connected organic environment, where sound is generated by filmic transformations and visual projections are in turn influenced by audio events …Overlaid upon this analogue world is a series of computational motion design elements, constructed digitally as generative graphical pieces, which are formed as a direct response to audio signals in real-time. Thus a mutual system of feedback is induced and live content drives its own production and transformation dynamically. (Curley, 2009: 55)
This live performance project [i/o] was selected for international premiere in Bergen, Norway, at the 7th annual 'Piksel '09' festival. Additional related project development documentation is available online (Curley, 2009).
Having given a series of important contextual examples and described the processes involved within recent live AV performance, we will now move to a more detailed description of our own work.
In Simon's sound performance sound is treated as paint on an invisible temporal canvas. The audio process in all the performances is improvised in real time and the sound playback is specified for the particular performance space. Simon takes advantage of the full range of frequencies available, and this enhances the physical impact and tactile perception via both the low and high frequency aspects of the sound spectrum. Simon also creates specific multi-channel effects through the arrangement of loudspeakers. He normally works with arrangement of 3, 6, or 12 channels of sound diffusion, making extensive use of DSP processing to alter the characteristics of the sound in real time. Primarily by enhancing the dynamic energy aspect of the spectrum, the audio performance creates rhythms and patterns out of that which normally remain imperceptible. Thus Simon works with specific details and patterns that are usually hidden within the source material. He reveals them through dynamic processing, equalisation and filtering. Simon's approach to rhythm is experimental. He tries not to program or introduce artificial rhythmical figures into the performance unless this is necessary for a specific purpose. Instead he unveils the intrinsic rhythmical structures in the sound material, especially the micro-structures hidden within field recordings. Depending on the performances and the set-up in the performance location, Simon uses different types of commercial software, such as Ableton Live, Steinberg Cubase and Pro Tools, sometimes running on multiple computers. Granular synthesis and other sample-based processes are brought into the performances, using NI Reaktor, Max/Msp and PD. Simon's composition method includes specific wave-shaping techniques and a signature style which he has been developing from his early work in live composition. This early work consisted in live MIDI sequencing of synthesizers and the real-time modulation and filtering of the sound generated from them so as to create a coherent flow or sound mass of atmospheric sound and rhythms. It was primarily intended for underground techno parties. His work in these early days of electronic music already contained the seeds of his current ideas of flow fields and waveforms in relation to the crowd, of interconnected consciousness and field-waveform interaction.
He currently manipulates sounds in real time as a single flow of textures and rhythmical patterns that are blended together in multilayered soundscapes. In many performances, a live microphone feed from the performance space is combined with the wave-shaping technique. This allows an exploration of the acoustic space along with a level of interaction that includes the aleatory element of feedback as part of the composition. Simon uses both pre-recorded source material as basic building blocks meant to be re-assembled during the live performance and also sound created live by either performing with granular synthesis or modulating a microphone feedback loop (using the Larseng effect). The pre-recorded audio files generally originate from processed field recordings. Simon mainly uses sounds which have a particular physical and tactile presence when directed through a subwoofer. Through all this, Simon seeks to explore and exploit subjective synaesthetic associations as tools.
Simon's compositional method is, in sum, based on improvisation and intuition. His approach to sound composition is conceptual and holistic, in opposition to a fragmented approach of note phrases and discrete durations. Simon sees physical space as a container of energy fields, resonating modes, and also as the material substrate for fields of ideas, (Sheldrake: 2009) information and culture (Eco: 1962). The work links Gestalt theories of perception with nested fields of acoustic waveforms within the performance space (Lehar, 2010; Gibson, 1979). Moreover, Simon considers the synesthetic percept as the direct percept-knowledge of a continuum, the direct awareness of reality as it appears before cognition and the formation of constructs and meaning. For Simon, "kinematic" movement in the visual field could be seen as a temporal wave pattern within or before or after the processes of cognition. He sees neuronal firing patterns in the brain encoding external stimuli (Varela, 2001) as possible Chaldni patterns of the brain and draws parallels between the diverse concepts and events found in acoustic sound pressure gradients, standing-waves, the Gestalt theory of perception described by Steven Lehar's concept of Harmonic Resonance (2010), the morphogenetic field of Sheldrake and the polychromatic organism of Mae-Wha Ho (Mae-Wan Ho, 1996). For Simon, then, the transversal is the creative gesture that intuitively crosses all these fields and brings them together in the sound performance.
Simon's field of work is also introspective. He deliberately explores his own synesthetic perception and investigates subjective impressions in the understanding that synesthesia is an expressive projection of inner cognitive processes. At the same time Simon recognises the neo-futurist tradition and the gap between science and art as the space were his practice finds fertile terrain and space, as in the futurist Luigi Russolo's 'Art of noises' (Russolo, 1913). Simon's use of sound is meant to produce a temporal painting, and fabrics which he use to compose abstract surfaces. At lower density, textures turns into rhythms and patterns, and at higher density into smooth surfaces.
Depending on the performance space and technical set-up available, Max uses multiple screens and a live video input. The ideal set up consists of three video screens, two video-output channels and one live video as a separate video input channel. Max remixes or rather live edits the video focusing on the quality of movement. Based on the cinematic tradition of the Kino-eye interval theory, Max aims at producing a continuum between the visible, the audible and the performance space, creating a kinetic flow of visual rhythms. All videos are filmed by Max, most of them on mobile devices producing low resolution images, thus creatively exploring the limitations of digital video (for further discussion of this aspect, see Schleser, 2009). Other files come from his experimental documentary projects, and some from B roll footage or cityscapes filmed for the performances on location around the world. The footage ranges from close-ups to wide shoot time-lapses. During the performance Max focuses on creating a visual experience through the rhythm of the videos, in tandem with Simon's sound performance. At times incorporating a live video input of the performance space, the audience and the AV duo themselves, Max thereby incorporates the location and its specific atmosphere into the performance. The VJ applications allow video files to be analysed based on their movement and this allows visual rhythms to become key for the taxonomy by which they are organised. Abstract elements are explored to create a structure, such as the dominant movement in the frame or dominant color of the clip. Only a very few effects of the VJ applications are utilized. Once a visual rhythm is set into motion, it is the guiding principle for the video performance. The visual rhythm in the video files is created not only through the movement within the frame, but also the movement in-between the frames. The process as a whole is reminiscent of the kinkos' interval filmmaking method.
The School of kino-eye calls for construction of the film-objects upon 'intervals', that is upon the movement between shots, upon the visual correlation of shots with one another, upon transitions from one visual stimulus to another. Movement between shots, the visual 'interval', the visual correlation of shots, is according to kino-eye, a complex quantity… (Vertov in Michelson 1984: p.90).
As Max Schleser has pointed out in a VJ theory article, the 1920s avant-garde filmmakers foreshadowed the practice of editing clips on the fly. The experimental filmmakers of the 1920s can therefore be defined as pioneering VJ technique (see Schleser, 2008). The film practises before the introduction of sound film in 1928 also treated the visual and audio as two separate entities with their own aesthetics as in Visual Rhythms. Movement becomes key to the way these come together in performance.
Bergson's phenomenological approach and Vertov's interval theory theoretically frame concepts related to movement. In Bergson's Creative Evolution time is defined as 'heterogeneous flux and movement is reality itself' (Bergson, 1907: 278). Bergson famously describes time in terms of duration, which 'means invention, the creation of forms, the continual elaboration of the absolutely new' (Bergson 1907: 14). In contrast to a scientific approach to the capture of movement, for Bergson movement is related to, and best understood via, intuition. Movement is an act of becoming and becoming is qualitative and evolutionary. Bergson conceives of a union of the theory of knowledge and theory of life, which ' should join each other, and, by a circular process, push each other on unceasingly' (Bergson, 1907: xxiv). He works towards a method beyond the limitations of scientific discourse. Where knowledge is accumulated experience, his project works towards reality as life in movement and movement as an act or process of creation. Experience is the currency of creativity. In short, creative activity emerges from intuition. For Bergson, then, intuition is positioned as a better form of knowledge than formulas and statistics.
Bergson was also famously suspicious of the cinema. In Bergson's understanding of the cinematic apparatus, the cinema operates as if any single movement is a movement between two stops and therefore an absurdly motionless trajectory. In the cinema, 'Movement is made of immobilities' (Bergson, 1907: 334) or 'movement coincides with immobility' (Bergson, 1907: 336). For Bergson, it is crucial to overcome the absurd cinematographic habits of our intellect. This can be done by taking up reality as becoming. Becoming is an experience through which one can 'escape the cinematographic mechanism of thought' (Bergson, 1907: 341). Bergson proposes that one should place oneself with the transition.
The reason is that there is more in the transition than the series of states, that is to say, the possible cuts—more in the movement than the series of positions, that is to say, the possible stop. (Bergson 1907: 341)
Being in the movement, one can analyse transition while actively engaging with(in) it. One is not merely passively reacting to a "given" reality, but engaging with its taking shape. In all this, 'Experience confronts us with becoming: that is sensible reality' (Bergson, 1907: 341). Bergson defines this as a more, not less, intelligible reality. The essence is positioned in the idea, but events should not be reduced to ideas, as these are in turn located in the movement process, 'the quality … is a moment of becoming; the form … is a moment of evolution' (Bergson, 1907: 315).
How can we reconcile an AV practice, especially AV practices as discussed in this article, with their emphasis on movement, with what Bergson suggests should be the overcoming of the 'cinematic habits of thought'? Bergson was of course working in the early days of cinema and this was a problem that the cinema itself was only just beginning to confront. As Deleuze, an admirer of Bergson, puts it, the 'evolution of cinema, the conquest of its own essence or novelty, was to take place through montage…' (Deleuze, 1986: 3). It is to these early days of montage, and other forms of AV experimentalism that we will now turn. These have many obvious parallels to the contemporary AV work we have discussed.
Montage by its very nature is based upon movement. The well-known early Russian film experimentalist Dziga Vertov's interval theory is based on movement in-between the connected shots, and these intervals can be related to the Bergsonian notion of becoming. Yet this experimentation with movement did not just concern images. In 1916 Denis Abramovich Kaufman (a.k.a. Dziga Vertov) commenced experimenting with sounds. He called these experiments "a montage of stenographic notes and sound recordings" (Vertov in Artforum, 1972: 82). Vertov, influenced by the poetry of Mayakovsky, composed 'communication structures' in a 'new vision of external reality' (Petric 1987: 29). Vertov used filmed fragments (shots) with the intention of disrupting the film's linear development and thwarting the reader's narrative expectations (Petric, 1987: 29). He introduced a conceptual approach based on an interval technique to filmmaking, which he termed "Kinochestvo" (Vertov in Michelson, 1984: 5). The interval theory involved is defined according to the 'art of organisation' (Vertov in Michelson 1984: p.5). Here we find a clear precursor to our own work with Visual Rhythms. Vertov utilizes
the transition from one movement to another… the necessary movements of objects in space as a rhythmical artistic whole, in harmony with the properties of the material and the internal rhythm of each object… providing a kinetic resolution. (Vertov in Michelson, 1984: 5)
Vertov was primarily interested in the space between the frames. Weibel quotes Vertov from his manifesto We, written in 1922;
The material – the artistic elements of motion – are the intervals (the transition from one movement to another), but not motion itself. They (the intervals) also direct the action towards a kinetic resolution. The organisation of motion is the organisation of its elements, that is to say of the intervals within a phrase…but [here] the interval between two frames, which is the important element of the articulation of meaning. The grouping of such units of meaning (of such units of articulation) forms a phrase. (Vertov in Weibel, 1979: 47)
Roberts points out that Vertov has difficulties expressing his cinematic method in written language (Roberts, 2000: 35). This is perhaps because Vertov is trying to describe something that is essentially cinematigraphic, yet not in the mode to which Bergson objects. First, for Vertov, film has its own dynamism. Vertov writes that montage 'means organising film fragments (shots) into a film-object. It means writing something cinematic with the recorded shots. It does not mean selecting the fragments for "scenes" (the theatrical bias) or for titles (the literary bias)' (Vertov in Michelson, 1984: p. 88). Second, as mentioned, the key to kino-eye productions is the movement in-between the frames, not the "stoppage" of the frames to which Bergson objects. It is this movement in the interval that also allows the linkage of video fragments to construct one coherent work within the AV performance. Indeed, the temporal construct of the AV performance takes the notion and reality of becoming as a leitmotif. The video's live montage is driven by the movement/intuition of the performers, along with the other aspects of the assembled performance. Vertov's interval theory is suggestive not only of the linking of diverse video elements, but of a transversal cutting across various fields. The next section will further outline the performance space as an experimental framework for transduction.
"Trans" – Visual Rhythms
The Visual Rhythms performance can be described as 'reconfiguring the material of already constituted objects and formal arrangements' (Mule, 2006).
The interaction is based in the movement of field of abstract information, or more correctly the intervals between them, such as perceptual cues, light, texture, colour, and ideas. Transversality is understood as, in Glen Fuller's terms, a tool (Fuller, 2007). The audio-video transductions themselves transform the decisions within the audio video process. More than this, in its movement the materials are to an extent both self-organizing and self-transforming. As Mitchell Whitelaw puts it, 'from this transmaterial perspective a computer' becomes 'a machine for shifting patterns through time and space' (Whitelaw, 2009). The transversal processes themselves incorporate audio-visual fragments and temporal patterns, while we move/flow with them, placing them in a new context. As Shaviro puts it, 'Transduction … is the continual transfer of patterns both within a given medium, and from one medium to another" (Shaviro, 2006). If, according to Murphie a 'transversal is a line that cuts across other lines…bringing the fields together in a new way … recreating fields as something else' (Murphie, 2006), then video and audio performances such as Visual Rhythms recreate fields at fundamental junctures of perception and technology, arguably returning elements of the "cinematic" to movement and intution (and therefore creativity) as they do so.
Since sound film appeared in 1927 the audio and the visual often seem to have become unified in one sphere in mainstream media, despite their differing aesthetic capacities and different modes of movement. Rather than forcing them into one entity as on a film strip, Visual Rhythms utilizes their own distinctive qualities, but also as 'new lines cross[ing] between older disciplines, older fields and older cultural practices" (Murphie, 2006).
Russolo, Luigi, L'arte dei rumori (Futurist Manifesto 1913)
Lehar, Steven, Gestalt Isomorphism and the Primacy of the Subjective Conscious Experience: A Gestalt Bubble Model, (2003) http://sharp.bu.edu/~slehar/ and Cambridge Journal, Behavioral and Brain Sciences.
Simon Longo, aka Dithernoise, is a London based sound and audiovisual artist, his work includes also site-specific installations. Simon's work is inspired through electronica, digital and organic aesthetics to create interdisciplinary art with reference to neurosciences, synestesia and primarily exploring the perceptual interaction between sound and vision. Simon started with producing 12" vinyl and CDs under several artist names and alias in early 90's released primarily under the Italian Label Muzak, Tax Disk and its own UK Label Modern Innovation. Simon currently work as independent artist and researcher experimenting through the use of time based media. www.simonlongo.com - www.dithernoise.net
Max Schleser is a filmmaker who explores mobile devices as creative and educational tools. His portfolio (www.schleser.co.uk) includes various experimental and collaborative documentary projects, which are screened at film and new media festivals in the UK and internationally. Max is the co-founder of FILMOBILE (www.filmobile.net) and MINA [www.mina.pro]. His publications include articles in Journal of Media Practice, VJ theory, and Culture Visuelle, amongst others. Currently Max is the Subject Director of Digital Media at Massey University (New Zealand).