Dynamic Performance of Nature: Augmenting Environmental Perception Through Social Media and Architectural Informatics

Brian W. Brush, Yong Ju Lee & Noa Younse

Abstract:

Architecture has always functioned as a mediating structure between humans and the environments in which they live; a static assemblage of semi-inert materials orchestrated, amongst other things, to temper environmental forces for human habitation. With advances in material and communications technology, architectural assemblies no longer perform as impassive boundaries separating discrete conditions of occupation between environments. They are becoming supple matrices for inter-environmental information exchange and perception.

Dynamic Performance of Nature is our permanent architectural media installation in the Leonardo Museum located in Salt Lake City, Utah. DPoN intends to augment environmental perception in museum visitors by communicating global environmental information through a dynamic and interactive interface, facilitated by social media, and embedded in the material of a high-tech media wall. It's conceived upon the notion that sustainability for the 21st century should be crafted to evolve beyond conventional application of green techniques and biomimetic pastiche into something alive and integrated with the environment.

Keywords:

Architecture, social media, Twitter, sustainability, Processing, data visualization, interaction design, materiality, mapping, light, biomimicry, intelligent environments.

DPoN utilizes Processing, the open-source visualization software, to create a data visualization which connects to the Google Weather, USGS, and Twitter APIs, manifesting the data in a dynamic LED color spectrum seen flowing through the wall. Visitors interact with DPoN using Twitter to send messages to @LeoArtwall, the wall's unique Twitter handle, that change the global weather feed. Visitors can also "paint with social media" by sending tweets with series of color in the message body. With DPoN we've injected static materials with live information to create a flowing picture of the world. DPoN invites curious inquisition as well as detached contemplation of the synthesis between light, material, space, and global environmental information.

This paper will present our theoretical, aesthetic, and technical principles utilized in conceiving and realizing DPoN interspersed with a discussion of resulting implications in the realms of architecture, computer mediated communication, interactive art, mapping, sustainable design, and social media. As our project is a rare example of information visualization manifesting in rather large scale built form, significant attention will also be given to the architectural aspects of the project alongside discussion of the information aspects.

Background

Dynamic Performance of Nature began as a response to an open, two-stage competition for the design of a major permanent art installation in the Leonardo Museum which we ultimately won and built. The RFP called for a work of art which, according to the Leonardo, should show an understanding of the blend of science, technology and art operating within the programmatic framework of the museum which seeks to ask, research and answer the most compelling and life-changing questions about how we live, work, learn, and relate to each other. There were very specific constraints given which defined the physical parameters of the piece: it should occupy a roughly 1,300 square foot area approximately 90 feet wide by 14 feet tall; function in a way loosely defined as a wall that creatively separates the paid and unpaid spaces of the museum; be self-supportive or structurally anchored to the building; permit circulation through and/or underneath the wall; be durable, low maintenance, permanent, and safe for children and adults; and emphasize sustainable and green materials.

As architects, we saw the project with its ambitious goals and remarkable scale as a unique opportunity to interrogate the most compelling issues in contemporary architecture and design related to technology, space (both physical and social), environment, and material. The project presented fertile ground to explore subsequent critical responses within an architectural assembly of significant gravity and physical presence. Our initial responses dealt largely with reconciling our own ambitions for the technical and visual virtuosity of the piece with its required functional constraints. We quickly acknowledged the design problem as one of projecting something as innovative, high-tech, and futuristic as the call demanded while maintaining a rather staid role as a massive wall operating in a working building.

Interrogating Nature and Sustainability through Environmental Information

There are many conceptual, theoretical, and methodological avenues one can take in formulating a response to design problems such as the one we faced which requires solvency in the discourses of the high-tech, dynamic, interactive, scientific, and sustainable. One such paradigm which is frequently pursued in design today is that of living architecture and intelligent design. Projects engaging this paradigm often tread a biophilic path emphasizing the simulation and re-creation of "life" or "nature" in design through biomimetic look, physical movement, or haptic play. These projects, articulating a biomimetic futuristic sustainability, are often defined by a nature-like visual appearance achieved through formal resemblance to obscure and exotic biological species such as those found in the pioneering artwork by the artist and philosopher Ernst Haeckel (1) or the more familiar biological structures analyzed by D'Arcy Thompson in On Growth and Form (2).

Drawings of radiolaria by Ernst Haeckel (left) and morphological analysis of pelvic bones by D'Arcy Thompson (right)

The result frequently is a co-opting of highly-evolved and highly-specific bio-morphological structures applied to altogether unrelated spatial contexts in order to achieve a phylogenetic bridge between the natural and artificial which is inevitably syncopic. This approach has become a panacea for the projection of nature-inspired contemporary architecture and often results in a thin pastiche of natural artifice. This myopic focus on what constitutes "living" vis-à-vis movement or "look" alone limits the ability of architecture and other design enterprises to expand their operational territories and uncover new paradigms for creating high-tech, sustainable environments.

In Dynamic Performance of Nature we deliberately sought a way out of this problematic in favor of an approach which was more critical and inquisitive of what can constitute "living" and "sustainable" in architectural assemblies. Our claim is that sustainability for the 21st century should be crafted to evolve beyond conventional application of green techniques and biomimetic pastiche into something intelligent and integrated with the environment, taking cues from the logic of extant structures in nature and architecture alike. Building on this argument, we developed a conceptual approach which placed information and the ability of architectural material assemblies to communicate live data through computer-mediated social-material interaction at the front of our intervening strategy. This approach had implications in the following aspects of the design which will be discussed in the following sections: the formal aesthetics and physical identity of the wall; the type of information communicated and the techniques for translating it through the wall; the material and geometric effects integrated with the information; the social aspects of the interactive mechanism; and the emergent effects as a composite experience derived from these individual parameters.

Figuration of Flow

Current approaches to using and visualizing live information privilege the ability of it to be harnessed in a continuous manner. Feeds, updates, and streaming of data across the internet dominate the lexicon of data handling procedures across platforms in front-end interfaces and back-end databases both mobile and desktop-oriented. To be sure, the emphasis on information continuity and our abilities to "tap" into it will only increase with technological development providing the opportunity. Subsequently, the spatial and temporal gap between the event and context of information acquisition and our own physical spaces, let alone bodies, is diminishing towards the instantaneous. These converging trajectories indicate an impending simultaneity of space, matter, and information held together by synthetic communication between them all. Articulating an architecture which could facilitate such simultaneity became our objective. Returning to the idea of continuity and then considered relative to numerous domains such as the cosmological, geometric, social, and aesthetic we found an appropriate paradigm to drive the figuration of our wall in the notion of "flow."

Playing out the notion of flow, DPoN takes on the form of a series of nested sine waves, captured and frozen in the material thickness of a wall as if they were a continual energy flowing beyond the visual, material, and perceptual boundaries of the Leonardo space. The sine waves not only give the wall a dynamic and kinetic order, but also signify flows of energy that define the physical behavior of natural phenomena the installation will sense. They also signify the mechanism with which that sensing will be transformed into tangible, interactive expression: light. So manifested, these captured waves are a tribute to the Leonardo as a place of learning, exploration, and experimentation where the most innovative of creative forces from around the world will be captured and sustained for the discovery and delight of all its visitors.

In the same way the wall captures environmental dynamism external to the Leonardo, it also conceals and reveals, in its functional role, information which is internal to the space: the exhibit. In this respect, the wall acts very much like a curtain, dividing the space of performance, the exhibit, from the space of the audience, the lobby. The presence of the wall creates a permeable boundary, maintaining a mystery about what is inside while simultaneously providing glimpses of what's behind to entice a visitor forward. Where one crosses the threshold between these two spaces, the "curtain" appears to lift, providing a physical opening to the exhibition.

Physically, the wall is a sophisticated performance in terms of technology, design, energy, and material. The interactive system takes advantage of LED technology, currently developing as one of the world's most energy efficient and intelligent lighting devices. Each of the 1,888 interactive LEDs within the system is powered by a mere .5watts, is individually IP-addressable through on-board micro-processors and can reproduce millions of colors through its full spectrum RGB capability. This interactive LED system and the components that drive it are embedded in an assemblage of 176 vertical fins made from 100% recycled HDPE plastic. Each fin is fabricated using a cutting‐edge procedure of digital parametric modeling and computer‐numerically‐controlled machining which allows for performative design versioning and material conservation.

DpoN Elevation View showing "sine waves," flowing geometric effect, and the 176 fins embedded with 1,888 LEDs

In developing the formal and figurative approach we were ever-cognizant of the discourse on spaces of flows by Manuel Castells which addresses the necessity for urbanism to reconcile simultaneous individuation and communalization as a result of communication and space-time repositionings in the information-age networked society. We attempted to address quite literally his call to architecture "…to recreate symbolic meaning in the metropolitan region marking places in the spaces of flows" (3). An intuitive response to manifesting the space of flows might lead one to emphasize light-ness, thin-ness, and weightlessness given the notion of information flow is rather immaterial or gossamer. Instead we found it important to give weight, mass, and volume to our marking of flow in order to pronounce its gravity as a meaningful figuration of information communication in contemporary culture. For although information may be ephemeral or transitory, distributed and consumed rapidly, its presence and influence in urban and social spaces is perceptively palpable, thick, and persistent.

Augmenting Environmental Perception

Cognizance not only defined our self-reflexive approach to galvanizing social and urban theory through the project but also played a major part in the type and nature of information the wall would communicate. As stated before, the ability to communicate information was a driving force in the formulation of our position on sustainability. We wanted the wall to "sense" as much as it would be sensed so that the sensation mechanism could augment environmental perception in the museum visitors. In order to accomplish this, the wall must look outward and inward simultaneously. As Michael Fox and Miles Kemp have observed, "Daily changes in the environment rarely make much of a long-term impact on the way that we organize space or build structures…The main point of adaptable structures that are environmentally aware is that they have the ability to educate the people that use them on their current condition and the changing state of the environment. As this environmental information becomes increasingly available and apparent, we have the opportunity to become more connected to our buildings as they take on lifelike forms" (4).

DPoN's Sensation-Interaction Diagram indicating its position as a translating body between global environmental sensing, computer processing, social media interaction, and human perception.

Parallel to this spirit we crafted a framework for inter-environmental interaction structuring DPoN to communicate live local and global environmental phenomena gathered from ubiquitous sensing apparatuses and databases, processed through information visualization software and managed through social media. We limited the default palette of environmental data to temperature, wind speed, wind direction, sky condition, and seismic activity. These are tactile environmental factors that most people are familiar with and interact with every day, forming personal relationships to environmental comfort in terms of these very factors. Although they could be construed as mundane candidates for such a rich visualization environment, these factors were chosen to maximize the potential for inter-environmental perception by the museum visitors. The point wasn't to wax on the novelty of exotic data streams. The goal was to create the potential for a perceptual connection between what someone knows and feels on the outside and a reciprocal translation of that very same information through the geometric, material, and formal language of our projective architecture. This mental leap, we contend, closing the gap between a "natural" perception of nature and a "mediated" or "artificial" perception of nature, is fundamental to a large-scale shift in cultural understanding of the integration of human and biological environments necessary for the domain of sustainability to expand and evolve.

Part of the reason we believe DPoN is successfully facilitating this mental leap is in how we've crafted the actual visualization of the information to be accessible. We utilize the Google Weather database to retrieve weather conditions from any city in the world feeding Google. Temperature is indicated by color, but it isn't direct; it's on a loose spectrum. If the temperature is in the mid-60's Fahrenheit, the color will show as a flowing spectrum between a medium green and a light yellow. If it's cooler such as in the 40's Fahrenheit, it will flow green-blue. The direction the color flows along the wall indicates the direction the wind is blowing. The wall is cardinally oriented so that up, towards the ceiling is north, down towards the floor is south, left is west, and right is east. So a wind out of the NW flows in a direction upper left to lower right. How fast the color flows indicates how fast the wind is moving directly. Finally, the weather or sky condition such as rainy, cloudy, foggy or sunny, etc. is given by a filter placed over these colors. So if the sky condition is rainy and the temperature is 60 degrees, the yellow-green spectrum will be very desaturated as if a filter were placed over the color resulting in an almost white-ish appearance. But if the sky is clear and 90 degrees, the wall will exhibit a vibrant and fully-saturated red-orange. These are aesthetic gestures that can be explained to the general audience that make sense almost instantly.

DPoN showing temperatures in the low-60's, clear skies, and winds out of the south (indicated by the vertically-oriented yellow streak to the right)

Most people can also relate to maps on some intuitive level. So when an earthquake magnitude 1.0 or higher on the Richter Scale occurs anywhere on the globe, a distorted map of the world projects through the entire wall indicating its location and intensity. North America is located on the left side of the wall with South America, Europe and Africa squeezing in over the opening archway and Asia and Australia displayed along the right side. This geographic orientation of the map was done primarily to increase the resolution of areas marked by frequent seismic activity, notably the "ring of fire" region of the Pacific Ocean. Where the earthquake occurred is marked by a flashing light, the color of which indicates the magnitude of the earthquake: low magnitude such as 1.0=blue, high magnitude such as 8.0=red and continuous interpolated color in between for other values. The flashing frequency also relates to this: lower magnitude=slower flashing, higher magnitude=faster flashing. The value on the Richter scale of the earthquake determines the number of minutes the earthquake visualization displays: 6.0 magnitude=six minutes displayed. After that time the map fades and the weather feed returns.

DPoN showing an earthquake in Northern California approximately 2.0-3.0 magnitude, indicated by yellow color. Distorted geographic projection of the world map is visible in white lights. Alaska can be seen stretched in the upper left with South America in the lower right.

Although reading information communicated through the wall in flowing color is meant to be an accessible and almost intuitive act, the formation of where the color flows is rather determinate. The colors don't simply appear on the wall as if a giant TV display was blanketed across it in a rigid matrix of pixels. We decidedly located the LED points along what's referred to, in both the mathematical and 3D digital modeling world, as iso-parametric curves, or iso-curves of the surface. Iso-curves function as a descriptive linear geometry allowing for more accurate rendering and visual clarification of parametric surfaces in virtual environments. They are an essential meta-data of the geometry both deriving from and characterizing the geometric form of the wall, acting in built form as a replication of the visual and mathematic substructure of the defining virtual surfaces. By flowing light along this subcutaneous infrastructure of the geometry, information and light merge with the geometric order of the wall itself, resulting in the emergence of a synthetic information-body manifold where one is inseparable from the other.

Virtual model of DPoN showing geometric iso-curves

The material of the wall, a semi-translucent, recycled white HDPE plastic within which the LEDs are embedded, extends this manifold to incorporate materiality. When information is communicated presently through the LED colored light, the material glows and reflects off of itself, amplifying the communicative effect of the light. When information and color is absent from any particular LED, the material appears opaque with no visual indication of the LEDs inside; an entirely different visual state than when lit. Thus rendered, the dissociation in the wall both performatively and visually, as in natural organisms, between its morphological structure, communication apparatus, and its integumentary medium is non-existent.

Lighting and material detail

Architecture, Information, and Social Media

As its base condition the wall displays the current weather feed for Salt Lake City updated continuously as conditions change. This is the default state primarily so that visitors can make the visceral connection between what they are seeing and what they just experienced walking from the outside to the inside of the museum. But to create the possibility of interaction between the wall and museum-goers, we added another level of intelligence to the wall: a Twitter handle. Visitors are able to tweet the wall, whose handle is @LeoArtWall, to change the weather feed to any city they want that is in the Google Weather database, which is abundant. They can tweet a zip code such as "10003"or a city, state such as "Nome, AK" or an international city name such as "Timbuktu."

A museum visitor tweeting the wall

Sending a tweet to @LeoArtWall

Along with the information feed interactivity, DPoN also leverages the capabilities of Twitter to do what we call "painting with social media." Using the 140 character limit, visitors can tweet series of colors in strings such as "blue blue red green pink" to initiate a randomized color chase across the wall. A color flash seeds in a random cluster of LED nodes along the wall and proceeds flowing back and forth along the length of the wall in response to the other chasing color flashes. After a few minutes the "painted" colors fade back into the weather visualization. Although there is no direct user control over the exact trajectory and motion of the color flashes, each tweet will result in a totally different color chase, encouraging continued play through different combination of colors in further tweets. In terms of functionality, visitors can combine their color tweets with their designation of a change in the weather feed such as sending "red purple green Los Angeles, CA." As far as information visualization priority is concerned, the base visualization is always the weather feed. When an earthquake occurs, the map supersedes the weather feed and is displayed on top with the weather feed still visible but very faint underneath. Tweeting colors to the wall at any time displays them on the very top, prioritizing the reaction to user input. Alongside the wall a monitor displays a textual live-stream of the back-end program operations including the tweet roll from all visitors and the API queries from the data sources. This element was important for reinforcing the relationship between the software operations driving the visualization and the "hardware" of the wall and providing an opportunity for visitors to explore the raw structure of the programming environment behind it.

A red flash from a tweet flowing through the wall atop a weather feed showing rainy conditions

Beyond managing the data visualization within DPoN, the use of Twitter as an interactive component in the project is our attempt to address a burgeoning direction in the notion of intelligent design. As pervasive computing continues to grow the number of actors connecting and exchanging information on the web is tipping away from a human majority towards a non-human majority. In fact recent estimates by Cisco placed that tipping point back in 2008 (5). All of these sensors, management devices, surveillance tools, and mobile communications devices are no longer relegated to perfunctory roles buttressing human activity. They are acquiring social agency. As Actor Network Theory, described by Stephen Graham, demonstrates "Agency is a purely relational process. Technologies only have contingent, and diverse, effects through the ways in which they become linked into specific social and cultural contexts by linked human and technology agency. The boundaries between humans and machines become ever-more blurred, permeable and cyborgian" (6). Extending a social media identity to an element of architecture, which occupies a scale and level of public influence well beyond any mobile device, compounds this effect many fold. It's a powerful leap in the pervasion of social networking to every facet of our lives, including the buildings we inhabit and the architectural spaces that bound them. And it begs the question, what is the future of the built environment when people, devices, and buildings are all communicating to one another, socializing with one another, and exchanging information towards an ever-growing collective social intelligence? Furthermore, what will result when the social domain inculcates the natural environment as well?

A companion visualization to the physical installation illustrates the "back-end" operations that drive DPoN. A live feed of the database queries, API connections, Tweets, and data feed changes is displayed in text on a monitor near the wall for museum visitors to see.

For now, only the museum visitors can take advantage of the interactive and social responses of DPoN since they are the only ones who can currently pay witness. But effectively anyone using Twitter can interact with the project. This means people from all over the world can influence the environment of the Leonardo, tagging the project with their diverse spatial, temporal and cultural signatures. There are plans in effect for hosting a live video feed of the wall on the internet so that the interactive mechanism can truly connect the Leonardo to a worldwide contingency. Furthermore, the program driving the installation is not by any means closed-ended. Our earliest intentions in formulating the "soft" approach to the project imagined an evolving platform which could accommodate myriad other additions, plug-ins, enhancements, and entire alternate versionings of information visualization. As information visualization trajectories evolve, the wall can be programmed as necessary to continue its role as an innovative and intelligent actor in the convergence of information and our environments. And as social media evolves towards greater sentient capacity, DPoN will be sure to flex in response, perhaps one day acquiring its own sentience. In this sense the project truly has the ability to grow with time. In its greatest capacity DPoN will exist as a site, a place for development upon which a future paradigm of sustainable architecture, prioritizing social, spatial, and material communication, can be interrogated and constructed for future generations.

Technical Description

The initial focus of the program was to use the physical form of the wall as a medium for visualizing data. The challenge was to determine which data should be gathered and how it could interact with the piece.

Early discussions about the source of the data led us to consider Pachube, an online hub for collecting and dispensing real-time information, as a possible source of raw metrics. However, while the available data can be very detailed, ranging from personal Arduino sensor feedback in Argentina to train arrivals in the Netherlands, it lacks a sense of global cohesiveness or consistency. This led us to seek out a more widely available data source that could be dependably accessed - the Google weather API.

The API allows us to retrieve the current weather conditions for a location by querying a zip code or city name. Both domestic and international, it relays up-to-date information on the temperature, wind, sky conditions, and more.

The nature of the wall is such that precise images and representations are not prioritized. With the ‘resolution' of the installation constrained to less than that of the average cell phone, in order to emphasize pixel integration with the physical aspects of the piece (rather than a higher resolution overlay of a screen on the wall), a more abstract display had to be employed. Initial studies focused mainly on creating an interesting visual display using the small image workspace. Geometric branching algorithms as well as progressive explosions were explored and formed the foundation for the current program. To make the digital performance relate back to the physical piece the softness of the changing lights had to reflect the subtle nature of the progressive curves. Similar to how the edges smoothly transition from one fin to the next, the ebb and flow of the lights would have to pass by subtly and effortlessly.

Processing, the program developed by Ben Fry and Casey Reas, was chosen as the tool by which the external information would be gathered, analyzed, and visualized. By using a few relatively simple commands, the Google, Twitter, and United States Geological Survey [USGS] APIs can be accessed and their information parsed to provide a rich mix of data.

The primary data used to control the program is collected from Twitter. Twitter4j, a Java library, is used to parse Twitter information and also tweet the status of the program as it functions. The system is set to query Twitter at a set increment for tweets directed at its screen name. The system first examines each word to determine if the user has input a simple command - a color, then moves on to search for a location within the message. If a color keyword is discovered, a new spot with that color is created and displayed on the led array. The spot is given a ‘lifespan' attribute upon its inception to determine how long it will be in the system. The spot randomly chases around the display until it surpasses its lifespan and slowly fades away. This low-level interaction provides the user with a rapid visual response to their command.

After any colors are dealt with, the program searches for any domestic cities listed within the tweet. Each word of the input string is treated as a possible city name and is analyzed against an internal database. The computational requirements of the function warrant a separate thread to prevent the main display from lagging. If a domestic city is not found the program runs a similar algorithm to search for international cities within a different database.

If a city is found, its information is pulled from the appropriate database and added to the weather thread's queue. The thread connects to Google's weather API with a concatenated URL containing the city information. The API returns an XML file that is parsed using Processing's built-in XML tools. After the desired information is scraped from the file the main system is notified that a new weather focus should be implemented. This information commands the program to generate a limited number of ‘weather objects' to be displayed across the installation. Similar to the color spots, the weather objects are instances of color with individual attributes to control their appearance and behavior. The current weather status for a city reflects in the weather objects. The temperature correlates to the color, the wind direction and speed controls the flow and intensity, and the sky condition affects the color saturation of the objects.

As the main program de-queues city weather instructions, a separate thread searches a CSV file, located on the USGS website, for listed earthquakes with a magnitude of 1.0 or greater within the past hour. If a new earthquake is listed, the thread will notify the main program to overlay a low-resolution map of the world on top of the existing weather display and indicate where the quake occurred with a flashing pixel.

In terms of the information management framework responsible for designing, detailing, and fabricating the material of the wall, a parametric design software known as Grasshopper was utilized. Grasshopper functions as a plugin for the 3D modeling software Rhinoceros produced by Robert McNeel Associates. As a graphical algorithm editor, Grasshopper allows the design process to shift from manual manipulation of geometry, to associative design of the mathematical relationships between geometric entities. These relationships are controlled in a separate design environment where graphical representations of geometric assemblies are connected to each other and manipulated separately from the parallel 3D environment. Geometry must still be created, but it is done in such a way that numeric and graphic input is the main procedure for manipulation.

3D-modeled LEDs and fin in both the Rhinoceros 3D environment (left) and Grasshopper graphical environment (right). The geometry on the left is manipulated by changing parameters on the right.

The great advantage of such a design environment is in the pervasive interrelationship of all the geometric entities. When a change is implemented in one element, corresponding updates occur in the rest. Due to its relational structure, Grasshopper also allows for rapid distribution of repeated geometric elements and geometric operations across the entire population of geometric entities. For example, in the distribution of LEDs along the iso-curves of the wall, a single LED model was described and deployed to every LED node point without the need for designing each individual LED. In this way geometric information is allocated and conserved for more efficient modeling practice. Similarly, when outputting shop drawings and fabrication files, all of the drawing elements are parametrically derived from the 3D associative geometry such that a very few number of actions are need to proceed from the finished 3D model to the corresponding drawings needed to manufacture the actual physical parts. This process allows for increased control over the transformation of complex virtual geometry to low-latency physical output and demonstrates the level of responsibility information commands across experiential as well as managerial dimensions of DPoN.