Computer Science 474 Spring 2010 Introduction

Transcription

1 INTRODUCTION Interactive computer graphics is a means of visually communicating and interfacing with the computer. It consists of both hardware and software that allows the user to display, manipulate, and interact with data for a wide variety of applications. This reading will look briefly at some of the history of interactive computer graphics, common application areas, and the hardware and software that comprise an interactive computer graphics system. It will also introduce the general concept of the graphics pipeline for generating images. Finally, the Processing language and environment is introduced. A Little Bit of History Visual display of computer data has been around since the early days of computing. Cathode Ray Tube (CRT) technology was developed in 1897 and used in oscilloscopes long before computers existed. Plotter output devices were available in the 1950 s and used with early computers. One of the earliest examples of using CRT s to display graphical information was the Air Force s SAGE program (Semi Automatic Ground Environment) that used radar data to display aircraft tracks as shown in Figure 1. The SAGE program was begun at MIT in 1954 and fully deployed in Beginning in the early 1960 s, vector terminals, terminals that only displayed lines, were used to display monochrome drawings of computer data. Because of the high cost of memory, raster displays, or frame buffers, capable of color and shading did not appear until the 1970 s. Raster displays differ from vector displays in that the screen is composed of a 2D array of pixel values that are constantly refreshed. Evans and Sutherland, a well known computer graphics company, released the first commercial frame buffer in 1974 with a resolution of 512 by 512 pixels and 256 shades of gray; at a cost of $15,000! Raster displays began to appear in personal computers in the 1980 s with increasing resolutions and color depths, and decreasing costs. Modern computer display devices are typically around 1280 by 1024 pixel resolution with full 24 bit color. High end displays are currently capable of 4096 by 2160 resolution. Figure 1. The SAGE graphics system. Ivan Sutherland is often credited with the first application that interacted with a visual display in a novel program called SketchPad, created in 1961 as part of his PhD thesis at MIT. The program allowed the user to interact with an XY point plotter display and a lightpen to create and modify 2D line drawings. The program was revolutionary and was the forerunner of modern Computer Aided Drafting (CAD) 1

2 programs and Graphical User Interfaces (GUI s). Ivan Sutherland later went on to start up Evans and Sutherland, and was awarded the prestigious Turing Award in 1988 for his early work with SketchPad shown in Figure 2. Figure 2. Sketchpad. Computer graphics is often associated with gaming. In 1962, another MIT student, Steve Russell, created the first video game known as SpaceWar shown in Figure 3. The program was written for the PDP 1 and was a huge success with DEC customers. In 1971, SpaceWar became the basis for the first video arcade game, Computer Space. A year later, the famous Pong video game was created and the company Atari was formed. A number of arcade style game and consoles followed. Increases in processor speeds and graphical display capability led to a series of high profile releases such as Quake, the Sony Playstation, Sega, Everquest, Half Life, World of Warcraft, the Sims, Xbox, and Halo 2. In addition to standalone gaming platforms, increases in internet and World Wide Web development led to a new home for gaming. Wolfenstein 3 D was released in 1992 as one of the initial first person shooter games followed quickly by the infamous Doom which was released in Today, online gaming consoles such as Xbox and Massively Multiplayer Online Role Playing Games (MMORPG) dominate the gaming market. Specialty display hardware on high end graphics cards allow realistic 3D scenes to be generated in real time for a highly interactive and seemingly real gaming experience. It is estimated that gaming is a $12 billion industry and the 68% of American households play computer or video games. 2

3 Computer Science 474 Spring 2010 Introduction Figure 3. Screenshot from SpaceWar. In terms of GUI s, Apple unveiled Lisa, the first personal computer sold with a graphical interface in Microsoft quickly respondedd with its own graphical interface, Windows 1.0, in Apple sued Microsoft in 1988 for copyright infringement. The case lasted four years. Apple lost the lawsuit with the exception that the court found the trash can and folder icons were infringing. The decision was affirmed on appeal and Apple s appeal to the U.S. Supreme Court was denied. While minor modifications and improvements have been made, and screen sizes and resolutions have increased, the basic windowing user interface remains the same today. Some of the greatest advances in computer graphics and animation have been made in the film industry. Hand drawn animation has been used in films since the early days of film making. Some of the significant film achievements for the use of Computer Generated Imagery (CGI) are described in the timeline below: 1973: WestWorld is one of the earliest examples that used CGI in short segments for animated faces 1977: While Star Wars relies primarily on the use of models and traditional film techniques, it uses CGI for sequences such as the Death Star 1982: Tron is the first film to make significant use of CGI as the characters enter and interact in a video game world 1983: The Genesis Effect in 1983 s Star Trek II uses a particle system approach (a modeling/rendering technique) to create a realistic special effect 1984: The Last Starfighterr made significant use of CGI rather than using traditional models for spacecraft 1988: Willow introduced the effect of morphing in which one image transforms into another 1989: CGI effects are recognized with an Academy Award for the water creature in The Abyss 3

4 1991: Terminator uses significant CGI effects including advanced morphing techniques 1993: Jurassic Park sets new standards for realistic computer generated creatures that interact with live actors 1994: CGI and advanced digital compositing are used to put Forrest in several historical settings in Forrest Gump 1995: Toy Story becomes the first fully computer generated feature film, and Pixar begins its long run of highly popular animated movies or, as the phrase goes: to infinity and beyond! 2001: Final Fantasy: The Spirits Within is the first attempt to make a photorealistic rendered 3D feature film Applications Interactive computer graphics has become pervasive across many disciplines and applications. Brief descriptions of some of the common application areas are listed. Data Visualization In a general sense, all of computer graphics is based on some underlying data to display. The term data visualization refers to the use of visualization techniques to represent, manipulate, and analyze content from some data source. Two general types of data visualization are often referred to. Scientific visualization refers to physical data that is obtained through measurement or simulation and is used to better understand underlying structure and relationships. Examples are visualizing airflow over a surface to understand the effect of different airfoil designs, displaying the temperature across a mechanical part as color, or modeling and animating air particles to visualize how a tornado is formed. Such visualization is useful for dealing with the vast amounts of physical data collected or generated through experiments, sensors, or computer models. Information visualization is the other general category that refers to more abstract or derived data such as stock prices, network traffic, or gene sequences. Visualization of this type of data generally requires a more abstract technique, such as data graphing, versus trying to represent it with a physical interpretation. CAD Computer Aided Design was one of the early applications of interactive computer graphics to assist engineers and designers in the modeling and analysis of automobiles, mechanical parts, and building structures. The auto industry is a good example of the benefits derived from using CAD. New car designs used to be created by hand with a full sized clay model built to see what it looked like in 3 D. The model could take days or weeks to produce. Today, with the assistance of interactive graphics, all models are created in the computer. Designers can quickly change the model to see the results of different styles, the image can be interactively manipulated to see it from all angles, and engineering computations can be 4

5 performed on the model to determine its performance characteristics. In some cases, the computer model can be sent directly to a manufacturing device for construction. Gaming From the earliest days of Pong, users have been entertained by interacting with computers in some type of gaming environment. As computer processing speeds increased, and dedicated graphic display hardware allowed for faster drawing of scenes, more and more realistic games were developed that allowed the player to be immersed in a visual environment that appeared life like and moved smoothly with the actions of the player. The advent of the internet and high speed connectivity has led to games in which large numbers of players can interact or compete with each other across the globe. The term MMORPG, or massively multiplayer online role playing games, was coined to refer to these types of interactive games. The popularity of interactive computer games has created a huge industry in game development and services; estimates are the online gaming industry will reach $4.4 billion dollars by Film Some of the historical achievements of computer graphics in film were previously listed. Today, almost all movies include some type of computer graphics whether it is computer generated characters, digitally enhanced special effects, or simple digital cleanup to remove mistakes. Computer generated characters are becoming more realistic, both in appearance and in their movement. Some futurists go so far as to claim they will eventually replace live actors! Using CG to enhance special effects is both cost effective and allows for effects that could not be created through other means. One of the challenges of combining live action with digital special effects is to make the interaction between live actors and digital effects appear natural and seamless. Art Digital artists have explored the use of computers and graphics to produce creative and aesthetically pleasing images. Conferences and exhibits have been created to provide forums for artists to discuss and display their work. Sometimes, the art is created using digital tools that mimic traditional media, such as paint systems and sketching software. The computer also offers the ability to create artistic images from mathematical formulations, such as fractals and procedural modeling. Education Interactive computer graphics is becoming widely used as a teaching aide, especially for online courses. Graphics can be used to represent either physical or abstract concepts and allow the student to interact with them to understand cause and effect relationships. One of the strengths of using interactive teaching tools is that a student can proceed at their own pace, and if a concept is unclear, they can replay a scenario, or repeat an action to better understand it. Training Similar to education, interactive computer graphics have become commonplace in many training environments. For some applications, using CG is not only cost effective, but provides a safe environment for students to learn. Flight simulation is a good example. Students can be put into unsafe conditions to learn how to react in times of emergency without risking lives or equipment. 5

6 Communication All of computer graphics is a form of communication. Sometimes the primary focus of the graphics is to communicate information to another person or group. Presentation graphics is a term sometimes used to refer to the creation of data graphs and visualizations to communicate a concept. It has been around for much longer than computers, as illustrated in the 1800 s by the famous flow graph showing Napoleon s disastrous Russian campaign (see Figure 4. Animation is also used to communicate time based events such as simulating the movement of cars in an accident for a jury. Figure 4. Flow graph showing Napoleon s march. Image Processing Image processing refers to the manipulation and analysis of digital imagery from sources such as digital cameras. This may be something as simple as using filters to improve personal photographs, to something as complex as identifying specific objects in a satellite photograph. A field that has relied heavily on image processing is robotics. For a robotic device to interact successfully with its environment, such as picking up an object or moving in a room, it needs to be aware of where objects and obstacles are located and perform sufficient analysis on them to accomplish its tasks. The advent of inexpensive digital cameras has greatly increased the number of software tools available to perform simple image processing techniques. User Interfaces Graphical User Interfaces (GUI s) are ways of using the human visual system to allow users to interact with a program. Effective interfaces can be the key to the usability and perceived value of a system. Computer human interface is an entire discipline that has grown up around how to make computer based systems easier to use and understand. The increased processing speeds and display capability of modern computers have made graphical interfaces the norm for how users interact with software and hardware. These are but some of the common applications of interactive computer graphics. In many ways, the field is ubiquitous as almost all modern systems rely on user interaction and the visual display of information for their operation. 6

7 Hardware /Software Hardware to support interactive computer graphics can be broken into three categories: input devices, output devices (primarily displays), and processing devices. Each of these will be briefly described. Input Devices Input devices can be classified by the functional operation they perform. Any single function may have several different physical devices that support that function, and any single physical device can serve many different functions depending on how it is used. The Graphical Kernel System (GKS) was an early ISO standard for low level computer graphics that was implemented on many machines in the early days of computer graphics. While the standard is now defunct, the logical classification of input devices into six categories is useful for describing common input functions: Locator returns a 2D or 3D coordinate in the world space (the coordinate system in which objects are defined) Stroke returns a sequence of coordinates for an application such as drawing Valuator a scalar value, such as turning a knob or sliding a graphical slider Choice selects one of a discrete number of options such as buttons on a screen Pick selects an item displayed on the screen, such as a part of a mechanical drawing String returns a sequence of characters Common physical devices, such as a mouse, can be used for several of these logical functions. For example, clicking on a point on the screen could be a locator, sliding the mouse with a button pressed could be a valuator, selecting a part of a drawn image could be a pick. Several physical devices are common to us such as a mouse, joystick, trackball, touchpad, touch screen, digitizing table, keyboard, webcams, image scanners, and voice systems. It is easy to imagine how these can be mapped to the logical functions above. Some of the less common input devices are: Light pen Designed to work with vector based drawing systems, the light pen consisted of a pen like device with a light sensitive diode in the end that reported a signal when exposed to light. The user would hold the pen on the screen at the object of interest, and when the CRT drew the line across the pen, it would activate the signal identifying the object being drawn as the object being pointed at. 3D Scanner / Digitizing System For inputting the coordinates of a 3D object, the user would touch the points on the object with a pointing device. The device would use sonic or electromagnetic signals to interpret the 3D position of the pointing device. A more modern approach is to use lasers with ranger finders to automatically scan an object and determine its distance from the laser. By rotating the laser around the object, or using mirrors, a complete 3D model can be created. 7

8 Output Devices Force Feedback Device This is typically both an input and output device. As the user interacts with the system, such as using a joystick to move a cursor, the amount of resistance felt is controlled by the software to provide the user feedback. Data Glove Popularized by the film industry, the data glove is a wired device worn on the hand to interact with the computer in a virtual reality environment. The glove reports movements and hand gestures to the system to cause actions to occur. Similar approaches make use of a camera and image processing to interpret gestures without requiring a glove. Others More exotic input devices are things like eye motion detection which can determine in what direction an individual is looking. Motion tracking devices such as used in the Wii game console, can tell the position and orientation of the sensor. The most common output device for interactive computer graphics is the display. Displays consist of different technologies with different characteristics such as size, resolution, color depth, refresh rate, and brightness. For many years, the CRT was the most common display with an electron gun shooting an electron beam onto a phosphor coated screen to excite the phosphor and make a point of light. The phosphor had to be re excited before it would start to dim, so the beam had to be re swept over the same points again and again. In vector displays, the user specified the endpoints that the beam would draw to create a line drawing on the screen. To get color line drawings, there were three different electron guns, one each for red, green, and blue. To produce shaded displays, it was necessary to excite every point on the screen, so the electron guns continually painted every point on the screen. To avoid dimming, and perceived flickering of the image, the guns had to refresh the screen at a consistent rate, usually around 24 frames per second. Today s more common LCD, LED, and Plasma technologies do not suffer from flicker and offer much thinner and lighter alternatives to CRT s. A raster display is composed of a rectangular array of picture elements, or pixels. The resolution of the display determines how many pixels wide and high the display is. Each pixel is associated with a memory location that specifies the color to put at that point. In modern displays, each pixel contains 24 or 32 bits, referring to 8 bit values for Red, Green, and Blue, and 8 bits of other information contained at that pixel. Lower end displays, such as mobile devices, may only contain 8 bits per pixel which means only 256 distinct colors can be displayed at any one time. To allow greater flexibility in what is displayed, a color look up table (CLUT), or palette, is sometimes used. A CLUT is a mechanism that transforms the value at each pixel to a new value based on a simple lookup table. The concept of palettes is also used in the Graphics Interchange File (GIF) file format. Images are stored as 8 bit images and the associated 256 entry palette is stored in the file with the image. This allows for much smaller file sizes, but limits the visual complexity of the image. Some of the interesting newer display technologies include e paper, or e ink, displays. An e ink display does not require refreshing and is not backlit so it can be viewed in full lighting conditions without 8

9 degradation. Although currently only commercially available in monochrome, different companies are experimenting with full color and higher resolution versions. Common applications are e readers, digital newspapers, and electronic billboards. Textbooks are currently being made available in an e reader format, so that students will only have to carry a single reading device for all of their courses! Another interesting new technology is virtual retinal display (VRD) which uses low level lasers to draw a raster display directly onto the retina of the eye. The user sees an image that appears to float in front of them. It has the added advantage of having the image laid over our view of real objects. Thus, for applications such as training and maintenance, the user can interact with the real world while visually accessing computer stored information. Various exotic technologies have been proposed or experimented with for 3D displays. Stereoscopic displays use different means to present different views to different eyes mimicking stereoscopic vision. Another approach has been to experiment with different types of volumetric displays such as the vibrating mirror concept in which different slices of an image were displayed in sync with the displays physical position in space. One 3D technology that has captured the imaginations of film makers and science fiction writers is the holographic display. By creating interference patterns in the light waves, a true 3 dimensional object can be displayed for viewing. Although displays are the most common output media for computer graphics, there are also a number of hardcopy devices that have been produced or proposed. Pen plotters were an early form of such devices. More modern approaches include various technologies for creating 3D physical objects from computer models. Milling machines begin with a block of material and remove matter to create the final object. Rapid prototyping machines build up an object by layering successive cross sections using materials such as plastic, resins, and wax. Processing Devices To speed up the process of drawing images, most modern computers take advantage of Graphical Processing Units (GPUs) to offload the task of drawing 2D and 3D images to the screen. GPU s are specialized hardware devices with parallel processors, large internal memory, and highly customized logic to perform the drawing and shading calculations necessary for real time rendering. High end video cards from vendors such as NVidia can have 128 parallel processors making real time gaming systems much more realistic. A currently active research area is how to use the parallel processors for general purpose computing and speed up computationally intensive tasks. Software Most modern languages have libraries available for common graphic drawing and interactivity functions. Various attempts have been made over the years to try and standardize the conventions, functions, and parameters associated with graphic calls. Currently, the most common standard is OpenGL which was first developed by Silicon Graphics Inc. (SGI) in OpenGL specifies a library of functions and the exact way in which they should perform. Different hardware vendors can implement the functions in whatever means is most efficient for the underlying technology. The details of implementation are 9

10 transparent to the user who, in theory, can write graphic applications using the OpenGL specification and seamlessly transport the program to different machines and graphics hardware devices. In addition to libraries, there are a great number of software tools available to perform different tasks associated with interactive computer graphics. Both open source and proprietary tools are available to perform: 3D modeling Rendering Animation Image manipulation Video creation and editing Data visualization Digital art creation Computer vision etc. The Processing Language Processing is an open source programming language and environment based on Java ( It was developed in 2001 at the MIT Media Lab for artists, designers, animators, and hobbyists who did not have extensive programming experience but wanted to quickly learn how to write programs that could generate images and animations. It is free for download and runs on GNU/Linux, Mac OS X, and Windows. Although the language is Java based (it actually converts programs to native Java before execution), it has a simple user interface and does not require knowledge of object oriented concepts, classes, or interfaces. For example, the traditional Hello World program can be written in a single line: println( Hello World ); Because it is Java based, programs can include existing Java classes and libraries. There is an extensive users group and many examples of how to accomplish different tasks in the language online. Programs are saved in sketchbooks which are folders containing the program and associated data files. The original idea of the language was to provide a way to rapidly prototype ideas, or sketches, in a simple to use environment. As a result of its popularity, it has grown to be used in much more complex applications and several supporting tools exist. The Processing Development Environment (PDE) is a simple editor for creating and running sketches. Programs can be written as a simple sequence of statements and executed (static mode). For example, the following sketch opens a window and draws a line, circle, and rectangle to the screen: 10

11 size(400,400); line(100,100,300,150); ellipse(200,200,100,100); rect(100,300,200,50); //open a window to draw in //draw a line (0,0) in upper left //a circle is an ellipse //a wide short rectangle To provide animation and interaction, programs are written in interactive mode which uses the predefined functions setup and draw. Setup is called once when the program begins to perform any initialization. The draw routine is continually called so that any changes will appear animated. Here is an example of a simple bouncing ball: int y = 100; int dy = 5; //global variable for ball s y position //global variable for ball s y velocity void setup(){ size(400,400); } //open a window void draw(){ background(0); //clear screen to black ellipse(200,y,30,30); //draw circle at current y position y = y + dy; //update y position to new position if((y < 100) (y > 350)) //if ball at top or bottom of bounce, dy = dy; // reverse direction } A well documented reference to the available functions in processing is provided under the help menu in the PDE. When desired, a sketch can be easily exported as a standalone Java application or as a Java applet for embedding in web pages. The Rest of the Readings Interactive computer graphics is a huge area and cannot be covered exhaustively in a single course or textbook. This set of readings attempts to cover the fundamental concepts and encourage the reader to further explore ideas on their own. Code samples will be provided in Processing as examples and for experimentation. As a companion to these readings, there are a number of interactive demonstrations available, all written in Processing. These tools are designed to describe concepts and allow the user to interact with them for greater understanding. The source code for all of the demonstrations is available to view. 11

12 LAGNIAPPE 1 The Making of Jurassic Park The movie Jurassic Park is often hailed as one of the landmark achievements in computer graphics. Steven Spielberg created a visually realistic depiction of extinct dinosaurs that far exceeded previous attempts at computer generated live action. However, it did not start out that way. When Spielberg first obtained the rights to the film in 1990, the use of computer generated scenes in the movies was limited and not terribly realistic. The initial decision for Jurassic Park was to use animatronics and stop action figures. Stan Winston, a famous visual effects supervisor and make up artist, best known for movies such as The Terminator and Aliens, was hired to create the dinosaurs. Phil Tippett, an award winning visual effect supervisor and producer, known for movies such as Star Wars and Dragonslayer, was brought on to use stop motion animation to move the dinosaurs. As production moved forward, Winston s studio created fully detailed models of dinosaurs, molded latex skins, and fitted them over complex robots. Tippett created stop motion animatics of major scenes. However, as the early results were completed, Spielberg was not happy with the dinosaur effects using these traditional techniques. His goal was to portray the dinosaurs as animals rather than monsters. Despite the best attempts at motion blurs, Spielberg still found the end results unsatisfactory in terms of working in a live action feature film. At the same time that these efforts were going on, two animators, Mark Dippe and Steve Williams, were experimenting in creating a computer generated walk cycle for the T. Rex skeleton. Based on their success, Spielberg approved them to do more. Given these successes, when Speilberg and Tippett saw an animatic of the T. Rex chasing a herd of Gallimimus, Speilberg told Tippett You re out of a job, to which Tippett replied Don t you mean extinct? Spielberg later wrote this dialog into the script. 1 According to Mark Twain, the word lagniappe is an excellent word a word worth travelling to New Orleans to get; a nice, limber, expressive, handy word. It is the equivalent of the thirteenth roll in a baker s dozen. It is something thrown in, gratis, for good measure. Each reading includes just a little something extra for the reader. 12