The following summarizes my research and production work conducted while in Professors Jerry Gibson and George Legrady's MAT course "Visual Design Through Algorithms".  The professors were interested in issues such as video quality to determine human perceptual models and the aesthetic and technical issues in the production of an interactive visual environment.  This correlates nicely with my interests, namely the relationship between information and meaning and interaction, interfaces and communication in digital media (these are discussed below in more detail).  To that end, we collectively as a class decided to learn and use the Max/MSP/Jitter software package (Jitter in particular as it designed for real-time video processing) to realize a group project.  The follwing details some of my experiments and contributions to that project.

Introduction

How are information and meaning related?  The answer to this question varies depending on who you ask.  Artists and critical theorists may refer to the cultural "information" (i.e. meaning, or lack of meaning) in a given painting phototgraph or other image.  However, an engineer, refering to Claude Shannon's Information Theory (which gives a specific narrow definition of information) would say that there is no information there, as there is nothing new being generated. In fact to an engineer, full spectrum white noise (like the "snow" on television) contains the most amount of noise possible. So this leaves me with the question:  Does adding or removing information (however you define it) from an image through resampling, compressing noise, blurring, etc, alter its meaning?  This question forms the basis of my investigations.

Exploring the relationship between information and meaning

Starting with the idea of adding and removing information from real-time video signals (i.e. noise, resampling and otherwise

"deconstructing" or "reconstructing" the video image) and using motion tracking/sensing to accomplish this, I began to explore the relationship between information and meaning.  With digital data there is - in a sense - "no there there".  With it's quality of being infinitely alterable, digital data is by nature subjective, arbitrary and ephemeral.

Interaction

With a mouse or any other control device, all the system knows about you is what it attains through that device and whatever you as a user do with it.  I have always been interested in exploring better modes of interaction and communication both between humans and technology and between humans themselves, mediated by technology.  The way we interface with our technology shapes the way the content provided by the technology is experienced and perceived. Interface and content are inextricably linked.  In short, interface = content.  Interfaces can distort our perceptions but can also, if the systems are well-designed, promote or enhance new modes of communication.  This is why for our group project I proposed an interactive installation to explore these issues.  My idea was an installation that would consist of two spaces, each containing a dark, fuzzy or otherwise indiscernible image, which is in fact a live video feed from the other person's space.  If their images happen to intersect, a "portal" opens up and each participant can see the other, allowing for interaction and communication via motion and gesture (which is required to keep the portal open).  The portal then becomes the interface.  It is the boundary zone of interaction and communication and it shapes how one interacts and how one experiences the piece.  Interface = content.  More details on this project are available at the MAT 256 course website.

Production and Experiments in Jitter

A. Motion tracking and resampling

One of my first experiments was to using motion tracking to affect the amount of resampling in an image.  Basically depending on where one stood

relative to the camera, a different portion of the image would be resampled based on the amount of motion that was detected.  If no motion was detected the image would downsample all the way down to 2 x 2 pixels.  The more motion that was detected, the higher the sampling rate of the image. Thus, to make the image recognizable as something, one would have to generate sufficient motion in enough parts of the image. Figure 1 shows the results of this experiment.

Figure 1. Motion tracking affects image resampling (Click the image to see the full Jitter patch).

B. Simple background subtraction

The next experiment involved simple frame differencing, edge detection and background subtraction. In this experiment everything in the image is taken out except the area where the current frame of the video signal is different (in luminance values) form the previous frame (i.e. - motion). Edge detection is then applied to further outline the motion.

Figure 2. Simple motion detection via frame differencing, with edge detection added (Click the image to see the full Jitter patch).

C. More resampling

I also tried some spatial resampling using the jit.resamp object. I generated random values and

sent them to an an oscillator bank which was then used to affect the horizontal and vertical shifting and scaling of the video image.

Figure 3. Spatial resampling using an oscillator bank (Click the image to see the full Jitter patch).

D. Noise and Filtering

My final experiment consisted of using noise and spatial and temporal filtering to spatially repostion (warp, distort, etc.) a video feed. A matrix of random pixles (noise) is generated and sent to the aformetioned filters. This generates abstract shapes that are then mixed with the video image, thus warping and distorting it.

Figure 4. Noise eand filtering used to warp a video feed (Click the image to see the full Jitter patch).

Conclusion

I am not sure how well these experiments relate to my original ideas or the origianl goals of the course, but I do believe that I now have a basis for further investigations. I hope to continue my research and production in this area as I progess

through my studies in MAT.

References

Jitter
Weaver, Shannon, Information Theory Baudrillard, Jean, "The Implosion of Meaning in the Media"