Dynamic Modulations (proposal)
Data visualization based on motion tracking



The new Clinical Research Facility at the Tulsa Graduate School will feature 3 aquariums stacked in a single column one per floor, in the front entrance and waiting areas of the clinic. The proposed artwork Dynamic Modulations consists in motion tracking the movement of the fish and using that collected data to map the fishes' movements and behaviors.

Dynamic Modulations is designed to feature electronically processed visualizations on 6 large LCD screens of the continuous activities taking place inside the three main fish tanks. The visualizations were designed to feature each tanks' activities on 2 screens, one showing a slightly altered recognizable video representation, the other an abstracted animation to visualize the aggregate behavior over time. The 6 screens are to be located in one location, offering viewers a global overview of activities of the three floors perceived in one location, and the ability to compare activities of the moment with behavior over time, creating contrast and context through perceptual enhancement not possible otherwise.

The visualizations aae to be generated in real-time and therefore convincingly convey what is actually taking place at the moment. Contrast of change over time becomes interesting when behavior of the moment is shown to be different from behavior from another time, for instance, comparison of night activity in relation to daytime behavior.




Dynamic Modulations is a visual artwork of continuously changing electronic images. Video cameras inside and above each tank record video showing the activities inside the tank at the rate of up to 30 frames per second. These video images are digitized into data which are then mathematically processed, filtered, and stored, and on the basis of the calculated information, animations are generated and shown on the large LCD screens. The setup for each of the tank consists of a video camera which sends data to a processing computer, which then sends its data to a visualization computer which outputs its information to 2 screens.

Information Visualization

There are many possibilities of how to visualize the recorded information and this will be one of the studies generated through the preparation of this artwork. The field of Information Visualization bridging disciplines such as Library Science, Information Science, Scientific Visualization, Geography, Design and Art, with the goal to feature information in 2D space, and over time. The scientific research focuses on data collection, its analysis and mapping physical phenomena. The artistic brings in aesthetic experience, complexity, poetics, and culturally meaningful results.

Kinds of Animations

The source of the action will be the activity of the fish registered by video cameras. The fish may be active, or not, and these behaviors will activate the animations. Behaviors of speed, direction, spatial area of greatest activity, change in behavior based on circadian periodicity will be the source information to be collected, and then statistically processed to activate animations of visual compositions.

The animations will function as a form of changing visual composition, the abstract shapes designed according to formal, aesthetic qualities, and simulating organic movement through computational means. The function of the visualizations will be to provide an aesthetic experience to the viewer and at the same time, signal to the spectator that the animations they are seeing correlates in some way with real fish behavior in the source fish tanks.

Visualizations may therefore result any combination of the following:
1) Tracing the paths the fish take with density building up in areas that are covered extensively
2) Mapping areas of activity and highlighting them with circular boundary definitions
3) Fish movement activates any number of actions such as automated building block constructions that fade with no action; or rubber band type area highlighting, or small explosions, or bubbles at location of movement
4) Highlighting and fading of nodes in a network determined by areas of most activities
5) Fish motion can be represented by the simulated movement of algae animation where one senses the presence of fish by the movement of the plants.

Project Production

Wesley Smith, computer vision research

"The OU Fish Visualization Project consists of two main systems: a computer vision system and a visualzation system. The computer visualization system’s task is to identify and track fish for extended periods of time. It processes video data from cameras trained on fish tanks and outputs tracking data to the visualization system in real-time. The visualzation system then interprets the data, producing imagery across a series of monitors."

Flow Diagram of CV System
A schematic view of the fish tracking computer vision system. Shows the interconnection of the various video processing subsytems.

Schematic of the Testing System
A diagram of the remote video testing system between the MSI and Legrady Studio. Video is captured and streamed from the MSI but controlled and processed in the studio.

Image Processing System
A visual walkthrough of the current image processing algorithm.

Status Report pdf
Report and technical description, 6.16.2006.

Test Video
Video footage showing improved tracking algorithm.

Other Fish-Based Artwork Project

Infiltrate, 2003, Ken Goldberg, Pietro Perona, John Bender, ILan Lobel, Karl Chen for the Neuro exhibition at the Pasadena Art Center, Pasadena, California. Ken Goldberg, professor of Industrial Robotics at UC Berkeley in conjunction with Pietro Perona, Director of the NeuroMorph Institute at the California Institute Technology teamed up to create Infiltrate an art installation focused on fish motion-sensing research.

Fishbowl 2002-2003, Tiffany Holmes, In Fishbowl, the goldfish controls the feed from the four submersible spy cameras. A fifth camera hidden beneath the tank tracks the fish’s precise position. The computer then locates the camera nearest to the fish’s position and turns on the feed for that camera only.

Mediated Encounters, 2001, Ken Rinaldo, Prof of Robotics, Dept of Art, Ohio State University Mediated Encounters is an interactive robotic installation of four fish tanks designed to allow Siamese fighting fish to determine the movement of two grapevine sculptures. The fish determine the direction and speed of the robotic structures by crossing any of six light break-beams connected to the computer which activate motors that move the tanks in the direction the fish look to the outside world.

The Edison Effect, Paul deMarinis, Professor of Art, Stanford University1989-1993
A red laser beam crosses a fishbowl. Music is played in the background. Whenever the fish block the laser beam, the music stops.