Modular Structure and Image/Text Sequences: Comics and Interactive Media
Published in "Comics Culture: Analytical and Theoretical Approaches to Comics". A. Magnussen and H-C Christiansen, (eds.), Tusculanum Press, Copenhagen 2000, pp. 79-90


As an artist working in interactive digital media, with a background in photography and an interest in a conceptual approach to the structuring of cultural narratives, comics always seemed to be a kind of distant cousin, something to consider as a reference model, a resource to examine and to "borrow" from. I was drawn to the medium because of its formal properties - its emphasis on visuality, the staging of narrative deliminated in frames, its structuring through the weaving of image sequences, usually woven through textual flow, etc. Studying comics and photo novels allowed me to further consider the constructing of meaning as they occur in sequenced images, especially when contrasted to the still photograph and cinematic narrative unfolding in time.

My research has not been guided by any systematic approach but rather through chance encounters and browsing. It has primarily consisted of spending blocks of time in the extensive "bande dessinées" section at FNAC, the large Parisian media/books department store, joining the large crowd of readers deeply immersed in the fantasy narratives of their material. Or else scavenging at newstands or first and second hand bookshops in search of material that would exemplify conventions, explorations, innovation or creative authorship related to the medium. As a child raised in the French Canadian educational system, I would naturally spend much time with such standards as Tintin and Asterix but this was a form of consumption rather than analysis. I had been searching for some time for theoretical material that would discuss comics from perspectives such as narrative construction analysis, syntax and reception as I had been interested in the second level, connotative aspects related to the experience of readings comics and what it might give the viewer beyond the telling of the story. Watching readers in the comic book stores mentioned earlier made me realize that comics seem to provide some kind of a quick entry into another space, one that is private, fantasy driven, and lodged within a psycho- emotional and subliminal state.

My work in interactive media is nonetheless more closely related to creative production rather then interpretation. As a producer of image/text narratives, my commitment is to the effectiveness of narrative development and the exploration of its sequential organization. My contribution to this conference is a discussion that reflects on extending the linearity of the comic format into a hybridized form influenced by multimedia conventions. Since the comic format consists of a sequence of frames organized on the page either in a linear and modular fashion, the potential exists to orchestrate relationships and plot development in such a way that the viewer has a choice in the unfolding of the narrative, similar to the multi-directional reading options in crossword puzzles. Another approach might include the flexibility to sequence image/text frames according to one's interests and see what possible narrative might evolve. In both cases, the outcome pushes the comic form into a hybridized variant beyond its conventions. Such formats may well develop in the near future as digitization becomes implemented in all forms of communication. It is worth noting that I first came across this conference as an announcement on the internet. I begin my paper with this reference to the internet as an information source to underscore certain strategies of information search and access prevalent today, modular and multi-linear in structure, which will serve here as a background model for my presentation today.

Modular informational structures have become a standard organizing principle in our everyday life, in particular since the mid-eighties' introduction of Apple computers' multiple window desktop interface metaphor. In this metaphor environment familiar to all of us working with computers, frames called windows, can be opened and closed in any order, positioned in any sequence, linear or layered. They can be moved around virtually on the computer desktop through the use of the mouse, another metaphoric device that came into common usage at the time of the Mac desktop interface. Each window functions as an open-ended container, a form of organizing structure in which one can group any set of information textual or visual, facilitating information access and retrieval at a later date. This desktop environment is a space for orchestrating information that functions purely on the level of the metaphoric.

Digitized information clusters are by nature fragmented, discreet and can be ordered in any sequential structure. Fragmentation, sampling, quick reading, frame-by-frame communication, serial offerings, are standard modes by which we interact with information resources, obvious examples being, stock exchange listings, television news, newspapers such as "USA Today" with its graphics dominated info boxes. The design of information technologies has also steadily moved in the direction of modular, non-linear interaction. One of the main properties of digitized information storage systems like CD's, laserdiscs and DVD's is that the user can jump around in a non-linear way from one song or video frame to another, bypassing the author's and distributor's predetermined sequencing. In the music industry, software that allows this kind of re-editing of the material where the new version is constructed from samplings of the original left intact, is called "non-destructive" editing. Through this ability of storing the re-sequencing of the material, the listener/ viewer/ reader engages in a form of authorship, not dissimilar to the assemblage artist. One juxtaposes pre-existing data blocks to create new meaning through their relationships. Narrative construction ofthis nature, are the premise of interactive, non-linear multimedia.

From an info-cultural perspective, the modular format of comics fits the information processing paradigm of contemporary culture. In the processof reading a comic, where one proceeds from one frame directly to the one next to it, the story unfolds linearly and evolves through the sequence not unlike reading a text or experiencing time-based visual and aural media, notably cinema and music. The viewer does not intervene in the sequencing of the material. Nonetheless the discreet nature of the comic frame, where each frame is a self-enclosed container, can also potentially function as a branching or connecting node for additional narrative layers that might run parallel to the main sequence in which the frames are ordered. I am here referring tostandard comics devices where texts, objects, etc. are drawn in such a way that they break out of their frame boundaries, crossing into other frames and thereby shattering the illusion of the comic frame as the window through which the story occurs.1 In fact the narrative occurs on many levels, for instance the frames exist in relation to each other in a vertical and horizontal matrix, ordered in sets bound by the limits of the page. These groupings provide diegetic meaning as well. Looking at the page, we can then observe that reading the frames does not necessarily have to follow in a linear direction but can possibly occur in all directions if the author designs it in such a way.

Whereas the comic format is structured in a two dimensional matrix, interactive multimedia can be considered as the non-linear combination of images with texts and sounds where the user selects and sequences the elements in time. The viewer's role becomes one of active participation, as the assembler of one variation of the story (construction of a meta-narrative), according to the evolving sequence based on selections of elements chosen according to chance and interests. The hierarchical relationships between the elements are generally predetermined by the author who defines them in a multidimensional network of connections according to a system based on a concept or metaphor by which to give the relationships meaning. In such a network environment, the elements are conceived as a set of connecting points or nodes, each representing some component of the document. Hodges and Sasnett point out that the "linkages among these nodes define the relationships among the components.."2. The links are not bound by any sequential connections except those determined by the author of the document and can be set with greater and lesser degrees of order or randomness following some form of organizing principle. For instance, "they can be organized as trees, lists, or interconnected webs. They can be used to represent physical or logical relationships...Cross-references from one to another are encoded as linkages. The essential point is that a simple system of nodes and links makes a powerful conceptual tool for describing many different structures of information."3 The aim is to allow for variations in the sequencing of data relationships thereby reducing predictability and increasing diversity in the viewing experience.

As an example, "Slippery Traces"4, an interactive CD-ROM produced in 1995, functions in such a way through the exploration of fragmentation and juxtaposition in the process of narrative development. In brief, "Slippery Traces" is a visual narrative in which the viewer assembles a "story" by going from one postcard image to another. A number of conditions and filtering processes are encoded mathematically into the computer program that selects the next image in response to the viewer's actions. The postcards have first been grouped in categories. Every image in the work is related to other images according to properties of similarities and differences defined in a database. Each postcard contains approximately five "hot spots", [illus. "Slippery Traces 1] each of which links to about two other categories and images. The user constructs a viewing sequence by clicking the mouse on a hotspot over an area of interest in the current image on the screen. The program searches and brings up the next image through a random generating function that responds to the dynamically pre-selected images according to the criteria defined in the database. The resultant sequence of viewed images can be reviewed to examine the evolving "meta-narrative" twelve images at a time [illus. "Slippery Traces 2]. Here one can see the particular connections that relate one image to the next. The essence of the viewing experience in the work can be understood as being located in the play and contrast of expectations between knowing that the program will bring forth an image that is somehow related to the clicked hotspot and the resultant degree of closeness or distance between the potential suggestion of the clicked hotspot's content and one's expectation of what the new image might be. The emphasis is also on the dynamic potential of information sorting and sequencing possible through digital data processing.

A key component of non-linear interactivity is the capability of the viewer to access arbitrarily elements in the networked structure. Whereas the cinematic form produces meaning through the predetermined sequencing of time based scenes, comics rely on sequence as well, but instead of timed visual sequences and sound, groupings of multiple image/text frames on the page function as structural guides to create meaning. Compared to these two forms, non-linear interactive media can be understood as an environment in which the viewer actively assembles from the various sets of given elements and pre-defined relationships. Meaning in the interactive work results through the sequential selection of components the viewer assembles in the viewing process. The viewer can then be considered as someone who actively constructs the narrative through the assembling of fragmented, or modular information elements. The sequential sum of viewed selections becomes the narrative.

The term montage derived from cinema can be utilized in a discussion concerning both comics and multimedia. Umberto Eco defines adifference in the way the term functions in comics: "The relationship between one frame to the next is governed by a series of montage rules. I have used the term 'montage', though the reference to the cinema should not make us forget that the montage in a comic strip is different from a film, which merges a series of stills into a continuous flux. The comic strip, on the other hand, breaks up the story's continuum into a few essential components. Obviously the reader welds these parts together in his imagination and then perceives them as a continuous flow."5 In the case of multimedia, Hodges and Sasnett expand the term's meaning to not only refer to transition from one context to another but also to mean the on-screen spatial organization of multiple information segments (i.e., windows) when they say: "Montage treats the combination of scenes - which are chosen, how they are sequenced, the transition from one to the next."....The combination of contexts-how they share the screen or how the transitions are made from one to another." 6

Once information is digitized, it is by nature fragmented, discreet and can be ordered in any sequential structure. Scott McCloud, the author of a book that analyses the structure and narrative strategies in comics by discussing it through the comic book format proposes the following definition to underscore the medium's fragmented form: "comics panels fracture both time and space offering a jagged, staccato rhythm of unconnected moments, but closure allows us to connect these moments and mentally construct a continuous, unified reality."7 In its descriptive approach of a particular kind of grammatical structure, this definition emphasizes a reading where the viewer is engaged in filling in between gaps and the assembly of relatively distinct elements to achieve narrative meaning. Christian Metz describes this operative act in his discussion on cinematic form as a process where "going from one image to two images, is to go from image to language. 8 He is here referring to the complex set of syntactic and grammatical rules that become activate when discreet units are brought into play to generate linguistic ordering. In comics, cinematic and multimedia environments, texts are used in multiple ways, primarily to describe something or as a linking device between frames. In an early article on the meaning of the photographic9, Roland Barthes proposes that the function of text positioned next to an image is ideological, first to anchor the meaning and thereby directing the viewer's reading of the image "towards a meaning chosen in advance". He argues that in the case of comics, the text has a complementary relationship where "the word, in the same way as the images, are fragments of a more general syntagm and the unity of the message is realized at a higher level, that of the story". He then differentiates this relation between image and text in film as being more critical as it "advances the action by setting out, in the sequence of messages, meanings that are not to be found in the image itself."10

If the individual frame can be considered as the smallest closed syntagmatic unit in the comic format, Phillipe Marion referred in his paper in this conference to the comics frame as being secondary to the larger unit of the page11. As an example, we can look at any of the double page spreads in the first chapter of Alex et Daniel Varenne's "Ardeur 3. La Grande Fugue"12 [Illus. Grande Fugue] (there are any number of other examples that could function here) can be looked at from this perspective - the comics double page as a complete narrative unit in itself. Each of these two page spreads are designed in such a way that they stand on their own visually and in terms of narrative content. These two-page units are then assembled sequentially into a chapter. The organization of frames on the page allow for a conventional reading, as one moves from left to right, top to bottom. Given that the page functions as a closed structure in itself, a different kind of reading could be introduced in addition to the expected linear narrative unfolding from frame to frame. This reading would involve bypassing the conventional sequential flow in favor of letting the page become an open space in which the frames can be potentially interconnected from all angles, resulting in greater narrative complexity. If transported into the multimedia environment, one can imagine such pages in terms of menu functions in the interactive mode, where each frame could function as anode - a start, junction or end point for second level sub-stories.

In the interactive, CD-ROM version of "Froid Equateur", the author, Enki Bilal remarks in a 1996 interview that in terms of narrative, it is better to go from the cinematic to the comics format but the reverse does not work13. In the non-linear multimedia environment, there are many possible methods by which a story can be visualized and designed to evolve. The decision by the CD-ROM design team to present each comics frame of Bilal's CD-ROM version of "Froid Equateur" as a single full screen image unfortunately proves him right. Even though his drawings are rich in texture and meaning, I was struck by the loss in the translation from print to multimedia. The expansion to the cinematic, full screen format with text at the bottom and linear forward/backward sequencing capabilities through clicking buttons erased any sense of context for each image's relation to the larger narrative. The resultant fullscreen linearity introduces a rigidity into the narrative flow revealing (at least for this work) the importance of the page as an organizing narrative structure where the frames function as units of a syntax, receiving much of their meaning by being seen in next to each other.

When the sequence of events in a story can be reshuffled from the original sequence to enhance the narrative plot development, the viewer is put into a greater creative role - interpretation enhanced by plot construction results in increased layering of meaning. In an article that discusses strategies of narrative Seymour Chatman relates that the French "la narratologie", the study of narrative, developed during a time when cinema and semiotics blossomed. One of the observations to come out was that "narrative itself is a deep structure quite independent of its medium"14 meaning that there is a separation between how one tells a story and the story itself. Chatman refers to this relationship between a story and its retelling as "double time structuring." He points out that "all narratives, in whatever medium, combine the time sequence of plot events, with the time of the presentation of those events in the text, which we call "discourse-time". What is fundamental to narrative, regardless of the medium, is that these two time orders are independent. In realistic narratives, the time of the story is fixed, following the ordinary course of life,...but the discourse-time order may be completely different"15, for instance a story can be told starting at the end, moving to the beginning then onwards to the middle, etc. Following this argument, the key enhancement of non-linear multimedia narrative structuring is that the narrative event may not need to have a prescribed beginning, or an end anymore. Since the viewer constructs the story through the selection of subject matter which eventually develops into a sequence, the narrative event in the interactive mode begins when the viewer's selection of the first image or event, and ends when the viewer walks away form the computer or installation.

In Slippery Traces, the viewer moves from one information source selected out of a range of possible choices to another, also selected out of other possible choices. As mentioned earlier, these linkages are defined through keywords (hidden from the viewer) in the database according to common literal or metaphoric properties. The work's title "Slippery Traces"16, makes reference to Jacques Lacan's particular use of the term "slip" to describe the unstable relationship between a sign and its meaning. In his remark that "meaning emerges only through discourse, as a consequence of displacements along a signifying chain17" he is referring to the notion that the meaning of things are defined not in themselves but through their relation to other signs. Lacan argues against the Saussurian notion that there is a stable relation between a signifier and what it refers to. Another example to consider is Jacques Derrida's observation that in the construction of meaning, a signifier always signifies another signifier: no word is free from metaphoricity. The example of the dictionary is offered. When we search for the meaning of a word, our recourse is to look in a dictionary where instead of finding meaning we are given other words against which to compare our word. From this we can gather that meaning, otherwise expressed as the term "signified", emerges through discourse, as a consequence of displacements along signifying chains. Both of these references consider meaning as taking place through the interaction of information modules sequenced in relation to each other

Slippery Traces had its roots in a two-projector slide show created to explore the ways that the meanings of images change when juxtaposed with other images. Images are normally seen in relation to each other, and like words positioned together in a sentence, they oscillate each other, slightly expanding, re-adjusting, imperceptibly transforming their meaning through contrast, association, extension, difference, etc. Transferred to the non-linear dynamic environment of the computer, the shifts in meaning are exponentially increased as the images are freed from their slide-tray linear positions, to be constantly re- situated in relationship to each other as determined by criteria defined in the computer code. The 240 postcards in this project were selected from my collection of over 2000 postcards. These were gathered over a period of twenty years mostly through visits to second-hand stores and flea markets. In addition to exotic or eccentric approaches to the convention of the postcard, I was looking for images that revealed the photographic image's relation to cultural beliefs and also images that expressed that which could only be a result of the photographic. The first step in the production of "Slippery Traces" involved selection and classification of the material. This process, subjective in approach as I followed my "common sense", was nonetheless systematic in the way that Claude Lévi-Strauss described ordering as a first step towards a rational approach to making sense of the world.18

Following the selection of images, categories were created through the simple act of stacking them according to common themes. The 25 categories that emerged based on what was "at hand" consisted of such topics as nature/culture, colonialism, the future, military, industry, the exoticization of the Other, scenic views, morality tales, etc. Images that did not have a category of their own were grouped into their closest thematic area extending the categories' function from simple classification to that of narrative. Another factor in the selection process was to search for cultural and ideological expressions through postcards of 20th century Western world views on global development, tourism and cultural exchange. Additional criteria for selection also included culturally significant or relevant subject matter or visually interesting compositions that expressed perceptions based on the photographic paradigm.

These relationships were encoded into a database where each category, each image and each grouping of hotspots had to be painstakingly systematically cross-referenced to maintain an equilibrium in the flow between the sections in order to avert bottle necks and dead-end repetitions. The outcome can be envisioned as an imaginary three-dimensional, nerve-cell-like membrane network in which all 240 images are interlinked with over 2000 connections criss- crossing to form a unified whole. Connections, or hot spots have something thematically in common with the image they call-up. Each time the viewer clicks on a hot spot to move to another image, he or she weaves a path in this dense maze of connections, a path that is recorded for the duration of the viewing event and there to be looked up to see what sequence on has traced through one's choices.

Viewers follow their own desires within an environment predefined by my perceptual filters encoded into the database. By perceptual filters, I mean not only the way I have categorized the postcards but essentially the way they function within the program's structure. The database then is intended both as an artistic expression - a condensation of a particular way of looking at imagery, and also as an authoring tool - a device for narrative deployment. The conditions of this looking have been encoded through computer programming, specifically by usage of dynamic database structures. The navigation and sequencing flow of Slippery Traces incorporates the form and function of database structures as a creative device, and underscores a philosophical approach to computer programming as aesthetic practice. Whereas in the cinematic model the narrative experience is deeply lodged in the temporal unfolding projection on the screen, the interactive media model shares with comics its investment in the frame, the sequencing and juxtaposition of frames, discreet fragmentary segments that can potentially result in multi-directional narrative structuring. I have come to this conference as a media artist in search of new perspectives. In exchange, I hope to have generated some thoughts on new ways of conceptualizing how comics might evolve when integrated into a non-linear narrative structure environment.