Diagrams are pictorial, yet abstract, representations of information. Maps, line graphs, bar charts, engineering blueprints, and architects' sketches are all examples of diagrams, whereas photographs and video are not. Diagrammatic representation and reasoning is a recently rejuvenated area of research that is concerned with how humans or machines can represent information using diagrams, and then reason (solve problems with, answer questions from, etc.) using those diagrams. What makes diagrams interesting as a subject of study is that while they do not have the complexity introduced by the richness of detail (which is usually, but not always, unnecessary) of photographic depictions, they are still spatio-visual representations that utilize spatial and visual properties of constituent components to capture and convey information.
More broadly, we are interested in visual representation and reasoning. What are visual representations? Besides graphs, maps, pie-charts, schematic diagrams and such, paintings, photographs, video clips, computer generated 2D and 3D graphics, animations etc. are also visual representations. Clearly, this rubric applies to a broad category of representations. But not all representations are visual. For instance, we do not include propositions and written or spoken language in the class of visual representations.
Contributors to these pages are motivated by an interest in how information can be represented, communicated, and used in ways that take advantage of the spatial properties (e.g., relative locations in space) and visual properties (e.g., color) of the units of representation, in addition to their inherent or defined semantics. This differentiates such representations from sentential representations (I desist from saying symbolic representations because visual representations can be constructed out of symbols too). Sentential representations capture and convey information by virtue of the semantics of their constituent symbols (e.g., words) and how those symbols compose to form more complex structures (e.g., sentences). The meaning of such a representation is dependent upon the meanings and context of constituent symbols, but not on the spatio-visual properties (such as location, font or point size) of these symbols. One can certainly construct visual representations out of meaningful and context-sensitive units, but in addition can also utilize spatial and visual properties of the constituent units and their arrangements to capture and convey information. Thus, by moving from sentential to visual representations, we can exploit additional dimensions for representing information.
This is the root of representational leverage that visual representations provide; and the source of fascination that such representations hold. It also opens up an entirely new realm of challenging issues of comprehension and reasoning: how can these kinds of representations be constructed, used for communication (e.g., in human-computer interaction), comprehended by both humans and machines, and reasoned with?
One approach to answering these questions is to begin by investigating how we, humans, create and use such representations. For example one phenomenon that is quite commonplace in occurrence, but has proved somewhat elusive to researchers, is mental imagery. It is not difficult for most of us to think of an occasion when we recalled an image in our mind in order to make an inference - imagining the living room while visiting a furniture store to see if the color and patterns of furniture will match is a typical example. What kinds of representations and reasoning processes are the mind using for this? Is the experienced immediacy of imagery a mere figment of one's imagination, or is there an underlying reality to it - a reality rooted in spatio-visual mental representations? There is a rich and colorful history of research by philosophers, psychologists and computer scientists on this.More recently, with the advent and spread of multimedia, there has also been research on how people understand and reason with schematic diagrams, animations, and combinations of these with textual representations. Another, very different, approach is to build theoretical foundations, algorithmic processes and computational realizations of spatio-visual processes that can operate on visual representations. There is a rich body of literature on this too.
Here you will find links to a great deal of research and published literature on all these approaches. Nevertheless, you may end up with more questions than answers as you peruse these pages and traverse the links in them. But that is precisely what spurs further inquiry, isn't it?
Bon voyage!!
Hari Narayanan
Last updated February 22, 1997