Teja Ream and Michael Quinn

MAT594E - Winter 2004

DIFFERENCE

Project Description

This system employs real time video processing to detect and measure motion. It accomplishes this by a delayed frame that is subtracted from the present which shows what has changed.

[ t h e  p a s t  i s  s u b t r a c t e d  f r o m  t h e  p r e s e n t ]

The feedback created from movement is representative of a brief history of motion occuring in the past. While the linear 1pixel wide line than pans left to right will record motion in increments of grayscale. White to record the most movement and varying shades of grey for time with varying levels of motion. Audio is similarly made to respond to motion in the varying levels of its pitch. A representation of what is before the camera only takes place with motion. A stillness before the camera results in a black, seemingly empty, response. Motion incites the system to respond with representation. However, there exists a tension with that 1 pixel line representation of the past erasing the real time representation.

pdf version of trifold distributed at presentation


Technical Details

Time Difference

The first process through which we put the video was a simple time difference. Figure 1 shows the schematic of this process. The equation for this process is

D(t) = X(t) - a1*X(t-d0)

where d is the delay, specified either in time or frames. In our project, a1 was always set to value 1. What D(t) gives us is a measure of what has changed in the past d1 span of time and of what magnitude the change was. Each frame of D is averaged and this mean value, Dmean(t), controls both the sound frequency and the video overlay magnitude.


Figure 1 - Simple Time Difference

Moving Average Process

The second process we used on the video was a moving average (MA) process. Figure 2 shows the schematic of the MA process for three delays. The equation for the MA process is:

Y(t) = D(t) + a1*D(t-d1) + a2*D(t-d1-d2) + a3*D(t-d1-d2-d3)

where a1, a2, and a3 are the weighting coefficients and d1, d2, and d3 are time delays. The values coefficients a1, a2, and a3 typically depend on the application as well as the desired output. One experiment set these values with a random number generator which yielded an effect which varied over time.

The moving average, Y(t), provides information about times further into the past than D(t), as input values as far back as X(t-d0-d1-d2-d3) are present.


Figure 2 - Moving Average Process

Autoregressive Process

Another process which was explored was the autoregressive (AR) process. This process, pictured in Figure 3, has the equation

Y(t) = D(t) + a1*Y(t-d1) + a2*Y(t-d1-d2) + a3*Y(t-d1-d2-d3)

The coefficients, a1, a2, and a3 are typically chosen to have a magnitude less than one to provide stability. The AR process actually feeds back (or regresses) onto itself. As such, the output is affected by both past values of D(t) as well as past values of Y(t).


Figure 3 - Autoregressive Process

Examples

Example 1 (Quicktime .mov - 8.25MB)

 Here, we take a simple time difference, D(t) = X(t) - X(t-d1), where d1 = 15 frames (= 1/2 second). The mean of each frame is then calculated and overlaid onto the output display. Before display, the mean values are themselves averaged to give a smoother transition. M(t) = (Ymean(t) + Ymean(t-d2) + Ymean(t-d3))/3.0. (d2 = 5 frames, d3 = 10 frames). M(t) is also used as input to an oscillator. More movement increases the frequency of the sound. Grayscale.

Example 2 (Quicktime .mov - 4.72MB)

 Here, we perform the same basic operation, D(t) = X(t) - X(t - d1). Here, d1 is a random number between 0 and 15 frames. For sound, the image is divided into 4 quadrants. The mean of each quadrant is calculated independently and used to drive 4 separate oscillators - each at different frequencies. Color.

Example 3 (Quicktime .mov - 3.5MB)

 Here, we first calculate the time difference D(t) = X(t) - X(t-d0). We then calculate a "moving average" of D(t). Y(t) = SUM(a(i)*D(t-d(i))) where i - 1,2,3,4, d(i) = {5,10,15,45} and a(i) are random numbers in the range [-1,1]. Grayscale.

Example 4 (Quicktime .mov - 5.4MB)

 Here, we used D(t) to drive an autoregressive process. The equation for this is Y(t) = D(t) + SUM(a(i)*Y(t-d(i))) where i = 1,2,3,4, d(i) = {5,10,15,20}, a(i) = {0.05, 0.1, 0.2, 0.5}. Grayscale.

Example 5 (Quicktime .mov - 4.4MB)

 Here, we explored crossfading between the live feed and another movie, depending on motion activity in live feed. This hasn't been refined and so the fading is pretty rough. More work needs to be spent on scaling the amount of fade to get smoother, more gradual fades. The motivation here was that when you cease activity and begin to disappear, other events take over and overshadow you. Grayscale.

Example 6 (Quicktime .mov - 7.87MB)

 This is just a video showing the simple time difference. The delay was set at about 15 frames. Grayscale.