Virtual Reality: Tomorrow's Infor- mation System, or Just Another Pretty Interface? Gregory B. Newby University of Illinois at Urbana- Champaign gbnewby@uiuc.edu ABSTRACT Virtual reality is the most promis- ing new area for human computer interaction since the Macintosh computer. It appears to have more power to effect changes in the integration and convergence of technology than any other technol- ogy in recent history. In fact, though, the roots of virtual real- ity may be traced to the early 1960s in such diverse areas as flight simulation, art, and computer graphics. As a set of interface opportunities for information sys- tems, virtual reality (VR) offers potential for systems with higher levels of interactivity and more sophisticated models for search and retrieval of information than are currently widely available. This paper examines the promise that VR offers for information systems by describing the state-of-the-art in such systems which employ VR tech- niques, and by projecting possible developments for the future. It will consider types of information problems which might be served by a focus on VR, and which problems will not be helped. The purpose of this paper is to identify for information scientists those areas of our field which may be served by VR techniques, and to consider the most fruitful directions for future work, in the context of current information system applications. WHAT IS VR? Virtual reality, or VR, has to do with highly interactive computer environments. Virtual realities create feelings of being elsewhere for those who experience them. They respond to input in ways which are intended to give the user a feeling of control over the virtual envi- ronment. Users are provided with a point of view -- a particular out- look on the virtual world. In con- trast with a typical computer application, virtual worlds are much more highly interactive, they have more flexibility in the range of acceptable input, a greater range of possible output, and they might provide very different expe- riences to different people who use them. Currently, VR is most easily defined by the types of input and output devices with which it is associated, rather than by a set of common concepts. Gestures made by the hand or body are commonly used as input, rather than a keyboard. Virtual realities often focus on visual forms of output, via a dis- play system which is mounted on the head rather than a regular monitor. These Head Mounted Displays (HMDs) allow the scene to change according to where in the virtual space the user is looking. Sounds might be localized anywhere in the virtual world. Most virtual worlds are gen- erated and maintained entirely by computer, although not all are. The computer is used to process data about where the user is in the vir- tual world and act accordingly. A simple example of a virtual envi- ronment is a virtual reading room. The user dons a HMD and a special glove input device. Both the HMD and the glove have a position tracker mounted on the back, so that the computer knows where the user is looking and where his or her hand is. Sensors on the glove mea- sure finger position. The user steps on a treadmill which enables him or her to "walk" in the virtual world, but stay in the same place in the "real" world. Our experimental VR system would enable the user to see a shelf of books which contains the subset of books in the entire library which most closely matches a profile of the user or a particular informa- tion need expressed earlier. The user could look at the titles of the books (when his or her gaze is directed at a set of books, their titles would be highlighted for easiest reading), and touch any book of interest. The book touched would be opened for the user to peruse. One of the difficulties in an exam- ple such as this is that it leaves the reader uncertain as to what is possible and what is not -- there are components which are within the reach of current VR technologies, and those which are not currently within reach. In Newby (1) I devote considerable effort to distinguish- ing between VR as it exists in the more popular media and VR as it exists in various laboratories and applications around the world. In the reading room example, HMDs and glove input devices are cur- rently available. Unidirectional treadmills are not widely used, but might be built. The computer-gener- ated visual world in which the user is immersed might appear more as a cartoon image than the real world - - this is typical of "real time" computer graphics -- but it would be a reasonable approximation of the real world. HMD technologies which are available do not allow for high enough resolution to read any but the largest text in a vir- tual world, which means that there would not be too much reading going on in this virtual realm. Another component of the technology in the example which is not readily available is the capability to ren- der the contents of an entire library or electronic database such that only those few which are most relevant to a particular user and her information need are presented. It should not seem strange at all to information scientists that the new set of input and output devices, and the capability to nav- igate physically and visually through an information space, does not immediately solve any of the traditional problems of information retrieval. This remainder of this paper will briefly examine some of the promise that VR offers for information access and interface design in gen- eral. It will be posited that there are aspects of VR which increase the power of currently available methods for human-computer interac- tion -- more interactivity, less rigidity, etc. These increases add up to what appears to be a revolu- tion in computer interface design, a revolution which did not need to wait for VR techniques to start, but which is helped greatly and partially enabled by these new devices. VR AS A PRETTY INTERFACE Virtual reality follows from a log- ical progression of advances in methods by which humans interact with computers (2). Aside from some computers which were programmed with switches, punched cards and punched tape were the first wide- spread method for interacting with computers, with output going on paper or perhaps more punched cards. Magnetic tapes were later used for the same purpose. These methods for human-computer interac- tion (HCI) were not highly interac- tive: the user would need to wait for hours or days until her job was completed, and in the event of an error would need to re-submit the entire job. In the 1960s, various forms of interaction via keyboard and VDT (video display terminal, or CRT, cathode ray terminal: a com- puter monitor) became available. Along with time-sharing, these devices enabled the user to get immediate feedback from the com- mands which she typed. Various methods for displaying out- put in color became available (starting with the pen plotter and ending with high-resolution CRTs). These could display pictures, as well as text. In 1984, the Apple Macintosh computer became the first widely available computer which employed a gesture input device (a mouse) and a visual scheme for dis- playing output (the Macintosh "desktop"). In fact, Ivan Suther- land had developed a visual method for interacting with computers in the early 1960s called "Sketchpad." Sketchpad (3) would take input from a pointing device such as a light pen, and display output on a CRT. Output was a series of geometric shapes which could be manipulated in a way similar to but simpler than MacDraw(tm). Today, the mouse is a common compo- nent for desktop computer systems. The mouse was invented in the mid- 1960s by Douglas Englebart (4). Around the same time, Sutherland was working on a pre-cursor to the HMD (5). He used computer graphics to display geometric shapes to the user which would move as the head position moved. They were displayed to each eye, and appeared to over- lay the "real" world. From the late 1960s to the present, Myron Krueger has designed interac- tive experiences which were par- tially computer generated and partially human generated (6). Peo- ple would enter a special room and see a rear-projected shadow of themselves on the wall, or would notice that the floor panels would light up in a certain pattern, and would be guided in their interac- tion with this world. In one such experience called "CRITTER," the user would see a computer-generated creature appear on his shadow. The critter would move about the shadow, responding to the user's movements. With practice, the user could engage in various tricks with the critter. VR of today includes the HMD, pro- jection screen displays of virtual worlds, highly interactive environ- ments, and various gesture input devices. Computerized graphical worlds constitute the visual out- put. The only "standard" tool for VR which was not invented in the 1960s was the glove input device - - this was made available in the mid-1980s by a company called VPL (they also produced the first com- mercially available HMD). Three- dimensional sound is also a fairly recent advance, but the principles have been known since at least the 1970s. Touch is incorporated in very few virtual worlds, and con- sists mostly of small bladders which might be inflated to produce feeling in the fingers. Taste and smell are not currently found at all in virtual environments. Strangely, it is necessary to return to the late 1950s and early 1960s to find a virtual reality which incorporates sound, smell, vision, and tactile feedback (7). Heilig's device used a combination of (non-computerized) visual output and real vibrations and smells to produce a feeling of immersion in a virtual world. Today, most virtual worlds rely more on the computer to produce all the desired feelings of immersion. In the future, virtual worlds may rely more on the physi- cal world for a convincing experi- ence. The logical progression of sophis- tication in computer input and out- put devices thus includes a return to the past. Today, computer graph- ics are available on desktop machines. The mouse and various other gesture-oriented input devices are widely available (including touch screens or light pens). From punched cards, computer input has progressed to the key- board to the gesture. From print- outs, computer output has progressed to color displays. Vir- tual reality is the next step in input and output devices: input via gesture using the hands or other parts of the body (or voice). Out- put which is linked to the user's point of view, and which provides a feeling of "being" in the applica- tion, rather than just observing. This logical progression includes some backwards movement. The HMD technology does not approach the quality of output on a VGA computer monitor. Glove input devices are not sensitive enough to type on a "virtual keyboard," at least not without undue attention to individ- ual variation in hand movements during typing. Position trackers are only accurate within about 1/4 inch, as opposed to a mouse which measures 90,000 positions per square inch (at 1/4 inch resolu- tion, a common brand of VR position tracker can reliably discern 48 different positions in a cubic inch area, or 16 in a square inch). Com- puter graphics which are produced on a high-powered desktop machine lose their effectiveness in a HMD device, which only produces 260 x 340 pixels (about the same as the CGA standard, but spread across 100 degrees of the field of view). Higher performance devices are pos- sible, but not yet widely distrib- uted due to the high cost of such a setup (commonly up to $100,000 U.S., not including the graphics computer). VR is a pretty interface, but vir- tual worlds in the early 1990s appear more like cartoon worlds than the real world, and interac- tion with them is usually through a limited set of gestures made with the hand or hands. Instead of dis- playing data on a flat screen, VR enables the user to change his point of view and move through an (often) three dimensional visual representation. DIRECT MANIPULATION AND THE NEXT GENERATION INTERFACE Modern interfaces to traditional computer applications are becoming more highly customizable. Your word processor, for example, can main- tain data on which font, printer, footnote style, and so forth you prefer. Even information retrieval systems, such as CD-ROM systems, might come up with a thesaurus of related terms if your query terms are not found, and enable you to go back to previous parts of your search (rather than typing the whole thing over again). Virtual realities can have these same qualities. In a visual world, for instance (8), you might be able to move objects where you want them. A virtual "office" might con- tain all the items of your regular office, but exactly where you'd like them (e.g, you could hang a printer from the ceiling, and have bookshelves which allow a single book to be in multiple locations). VR is not just a pretty interface, though. Aside from the populariza- tion of a "new" set of input and output devices, consider again the qualities which separate virtual reality environments from tradi- tional computer systems: Virtual realities are highly interactive and allow for a point of view. These qualities enable users to directly manipulate their environment. Direct manipulation (9) often is associated with graphical user interfaces, but they are not lim- ited to manipulation of visual objects. The benefit of direct manipulation is that the user can know immedi- ately the effect of his or her input on a computer system. Feedback may be via a variety of channels: sight, sound, or touch. Contrast this with the punched card days when any error in input was not known until hours or days later. Virtual reality is a driving force to make graphical computation faster and of higher quality. Vir- tual reality researchers work to provide interfaces which enable people to use their pre-existing skills for spatial navigation and perception to be applied to inter- action with computers. Pointing- and-clicking seems to be a natural way of interacting with computers to many people. Even so, it needed to be learned: a method for commu- nication which has little applica- bility outside of a computerized domain. VR offers reaching-and- grabbing, and general navigating as alternatives. VR FOR INFORMATION SYSTEMS Application areas for VR are cur- rently not yet highly developed. Medical applications are forthcom- ing (10), as are various forms of entertainment (video games, role- playing adventure games, etc.). In some cases, there is a clear path from a real-world application to a virtual reality application: sur- gery training, for example, might involve a representation of a phys- ical body part with which the sur- geon may interact. There are other areas for which there is not a direct physical model from which to draw the virtual representation -- perhaps there will be art exhibits which fool the senses by contrast- ing the "real" with the impossible. An application area for which vir- tual reality techniques might apply, especially the visual and navigation aspects of virtual real- ity, is information retrieval. VR FOR INFORMATION SYSTEMS There are as yet only a small number of direct references to virtual reality applications for informa- tion retrieval. In an imaginative view of what could be, Halbert writes of an information system of the future in which a physical landscape represents concepts and relations among them, and the user navigates through the landscape looking for formations which meet his or her information need (11). Information scientists recognize the important problem with such a notion: representation methods for information have long been problem- atic, and there is little agreement about how to best present informa- tion so that someone might find what they need. This problem is added to by the difficulty in know- ing which situational factors or aspects of a given information need might play a role. In 25 years since George Miller (12) called for more focus on visu- alization for information systems - - charging system designers to allow people to use their existing skills for spatial navigation and perception -- there has not been a great focus of effort for visual- ization in our field. Virtual reality offers an impetus for us to reconsider the applicability of direct manipulation, navigation cues, and spatial representation for information retrieval. In the 1992 proceedings (13) I reported on an information system which attempted to facilitate nav- igation through information space. The system employed a visual inter- face. Since that time, I have re- implemented the system using a large projection-screen monitor and stereo glasses for 3D effects. Interaction is via DataGlove(tm) instead of mouse. This system is closer to "The Matrix" as envi- sioned in popular fiction (14) than other virtual realities, the major- ity of which are fly-through car- toon worlds. In this system, relations among data are presented visually, and the user's point of view can move to evaluate in detail any aspect of the information space chosen. The user reaches out and "grabs" documents which appear use- ful. For virtual worlds which do not have a physical analog -- espe- cially those based on information - - information scientists can lead the way towards better understand- ing of the appropriate representa- tion schemes and access methods. Information science can offer a conceptual basis for information space as that set of concepts and relations among them held by an information system. Through inves- tigations of cognitive space -- the dynamic aspects of human cognition which are involved in the origin and satisfaction of an information need -- and the relation of cogni- tive space to information space through interaction with a retrieval system, we can offer a foundation for the building of vir- tual information spaces. CONCLUSION This paper has given only brief treatment to the technologies and current efforts in VR. The 1993 ARIST volume offers a comprehensive review of the area. The paper has instead focused on pointing out the possible roles which VR and infor- mation science might play together. The history of VR shows that the devices for interaction with com- puters are not particularly new -- there was nothing to prevent the methods for human-computer interac- tion which are the focus of today's VR researchers from being developed in ernest 20 years ago. Today's efforts in VR are part of a shift in the nature of the computer interface towards something more personal -- computer systems which are capable of engaging in highly interactive sharing of experiences. Yet the field of VR has not yet realized its own identity. Schol- arly conferences are held on the VR, and a number of new methods are available for interacting with com- puters by gesture, vision, touch, and so forth. VR researchers have difficulty when thinking about how to get from where they are now -- a set of input and output devices -- to where they want to be: interact- ing with a virtual world which is highly responsive to personal point-of-view and (information) needs. Perhaps it is not so strange that one of the central goals of virtual reality research is comparable to that envisioned by Bush in 1945 (15). A central goal of VR is to produce an interface through which the user might navigate as though it were an extension of his or her own memory -- "travelling" in huge dataspaces with few physical ana- logs, and manipulating both real and virtual devices as an extension of his or her own consciousness. Although information science has not produced a "Memex machine," there is a history which informa- tion scientists may draw on and apply to the problems of today's research on advanced methods for human-computer interaction. There is hope that the many VR research- ers might pool together with infor- mation scientists to achieve mutual goals which were previously unob- tainable by each alone. Notes 1. Newby, Gregory B. 1993. Virtual Reality. In: Williams, Martha E., ed. Annual Review of Information Science and Technology. Medford, NJ: Learned Information. 2. See the ARIST chapters from 1960s on human-computer interaction to see that some of the types of things which we now have, and some of those which we still dream about, were being envisioned then. e.g., Licklider, J.R.C. 1968. Man- Computer Communication. In: Cuadra, Carlos A., ed. Annual Review of Information Science and Technology: Volume 3. Chicago: Encyclopedia Britannica. pp. 201-239. 3. Sutherland, Ivan E. 1980. Sketchpad: A Man-Machine Graphical Communication System. New York: Garland Publishers. 176p. (Origi- nally published as the author's thesis, Massachusetts Institute of Technology, 1963). 4. English, William K.; Englebart, Douglas C.; Berman, Melvyn L. 1967. Display Selection Techniques for Text Manipulation. IEEE Transac- tions on Human Factors in Electron- ics HFE-8(1): 5-15. 5. Sutherland, Ivan E. 1968. A Head-Mounted Three Dimensional Dis- play. AFIPS Conference Proceedings Fall Joint Computer Conference. New York: Thompson Book Co. pp 757-764. 6. Krueger, Myron W. 1991. Artifi- cial Reality II. Reading, MA: Add- ison-Wesley Publishing Company. 7. Heilig, Morton. 1992. El Cine del Futuro: The Cinema of the Future. Presence 1(3): 279-294. (Originally appeared in: Espacios. 1955 January). 8. This work focuses on the visual aspects of virtual realities because they are the most highly developed. However, auditory compo- nents are also highly developed in some virtual worlds. A few provide more tactile feedback. 9. Shneiderman, Ben. 1983. Direct Manipulation: A Step Beyond Pro- gramming Languages. Computer 16(8): 57-69. 10. Murphy, Harry J., ed. 1992. Persons with Disabilities: Proceed- ings of the Seventh Annual Confer- ence. March 18-20; Los Angeles Airport Marriott Hotel. Northridge, CA: California State University. Available from: Office of Disabled Student Services, California State University, Northridge, 18111 Nor- dhoff Street - DVSS, Northridge, CA 91330. 11. Halbert, Martin. 1992. Knowbot Explorations in Similarity Space. In: Miller, R. Bruce & Wolf, Milton T., eds. Thinking Robots, an Aware Internet, and Cyberpunk Librarians: The 1992 LITA (Library and Informa- tion Technology Association) Pres- ident's Program. Chicago, IL: American Library Association. pp 143-156. 12. Miller, George. 1968. Psy- chology and Information. American Documentation 19(3): 286-289. 13. Newby, Gregory B. 1992. An Investigation of Navigation for Information Retrieval. In: Shaw, Debora, ed. Proceedings of the American Society for Information Science Annual Meeting 28: 20-25. Medford, NJ: Learned Information. 14. The most widely cited work of fiction on virtual reality is Gib- son, William. 1984. Neuromancer. New York: Ace Books. Gibson's book is widely cited in scholarly works on VR, and is at least partially responsible for the strong interest in VR today. Unfortunately, the blending of fiction and reality which results in the crossover from sci fi to scholarly publications leaves confusion as to what appli- cations really exist (or will soon exist), and which are simply spec- ulation. 15. Bush, Vannevar. 1945. As We May Think. Atlantic Monthly 176(1): 101-108.