A static representation for ToonTalk programs Mikael Kindborg mikki@ida.liu.se www.ida.liu.se/~mikki Department of Computer and Information Science Linköping University Sweden Abstract Animated and static program representations have different properties, resulting in different user experiences. These properties and differences are analysed in the context of ToonTalk, a programming tool for children. ToonTalk has an animated representation of programs inspired by cartoons and video games. Based on a usability analysis of the animated representation, a static representation that uses comic strips is suggested. The visual language of comics has properties that combine the dynamics of an animated representation with the overview provided by a static representation. The study shows that the representation based on comic strips has the potential to reduce some of the usability problems related to understanding and editing programs, and that the animated representation has advantages when creating and debugging programs. 1. Introduction This paper outlines a static representation for ToonTalk programs that is based on comic strips. ToonTalk is a programming tool that uses an animated programming language to make programming accessible to children (Kahn 1995). Learning to program is traditionally considered as a difficult task. One reason for this is that it is difficult to create a mental model of the behaviour of the program and the underlying computation model. Limitations of the human working memory make it hard to maintain an internal representation of a program. An external representation could help to understand the program s structure and its behaviour. (Internal and external representations are also known as knowledge in the head and knowledge in the world, Norman 1990, pp. 79-80.) Research in the area of visual programming (Burnett 2000) has suggested the use of visual program representations to help the user create and maintain a mental model of the program, thereby bridging the gap between the programmer and the program. However, the research efforts that have been made have not resulted in a widespread adoption of visual programming (Kahn 1996, pp. 2-3). One possible 1
reason is that the graphical representations that have been used, such as flow charts and data flow diagrams, are too abstract and do not visualise the dynamic aspects of a program. The shortcomings of static representations have led to the suggestion that animated program representations could be better suited to serve as an external representation of a program s dynamic behaviour (ibid.). Another reason why programming is difficult is related to the general level of many programming languages (Nardi 1993, pp. 37-40). End users find it easier to learn specific languages than to learn general purpose languages, and visual programming have failed due to lack of task specific programming constructs (ibid., pp.61-63). A related observation is that even though ToonTalk is based on a general purpose programming language, it is domain specific in its use of video game graphics and in its support for game programming. ToonTalk has built-in support for pictures, moveable sprites, collision detection, and other game and graphics related constructs that appeal to children. The purpose of this study is to address some of the usability issues of animated and static program representations. This is done by analysing and comparing the properties of animated and static program representations in the context of ToonTalk. The motivation for choosing ToonTalk is its unique use of a fully animated programming language. This makes for an interesting contrast between animated and static program representations. Further, the cartoon-like language of ToonTalk (ToonTalk means Talking in Cartoons) shares many properties with comics, making a comparison of the two representations meaningful. 2. The animated language of ToonTalk ToonTalk has an animated programming language that is inspired by video games. The motivation for using animation is that programs describe dynamic processes. Thus, animation should be a better form for representing a program than static pictures, such as diagrams or re-write rules. The appeal of video games and video game graphics is another strong motivation, especially in a system that is meant for use by children. ToonTalk has been described by a child as a video game for creating video games. ToonTalk is based on the computation model of concurrent constraint programming (see Janson & Haridi 1993 for an introduction to concurrent constraint programming). In ToonTalk each programming abstraction is represented by a concrete object. For example guarded clauses are represented by robots, the guard is represented by a thought bubble, the body by actions carried by the robot, and variables are represented by boxes. Programming in ToonTalk is done by demonstration (Kahn 2000). The programmer trains a robot to perform actions on a concrete example. When the training is done the robot can be generalised to work with generic data by relaxing the guard in the robot s thought bubble. Figures 1-4 show how a very simple robot is trained to add the number 1 to any number given to it. It goes without saying that paper is 2
a poor medium for recreating the experience of an animated medium. To see ToonTalk live, a demo can be downloaded from http://www.toontalk.com/. Figure 1. A new robot is given a box with the number 1 in it. Note that the thought bubble of the robot is empty, indicating that this is a new robot that has not been trained before. Dropping a box on an untrained robot will open the robot s thought bubble and the training of the robot can start. The hand holding the box represents the programmer. To the right is a toolbox which contains the various building blocks used to create programs. Figure 2. Inside the robot s thought bubble. Here a number pad with the value 1 is dropped on the number inside the box given to the robot. Dropping a number on another number results in adding the two numbers. When training a robot the hand representing the programmer is replaced by the rocket-powered robot seen in this picture. The robot will remember the programmer s actions. When the training is done, the programmer hits the Escape key to exit the thought bubble. 3
Figure 3. The box in the thought bubble represents the guard for the robot s actions. The robot will only work if the content of the box given to it matches the guard. By erasing the digit with the vacuum (left) the guard is generalised. The robot will then accept any number, not just the number that was used to train it. When a box is given to the trained robot it will add the number one to the number in the box (right). Figure 4. The mouse with the hammer is an animation that represents the add operation. When a number is dropped on another number the mouse comes running to hammer them. The robot will continue to work as long as the guard matches the box. In this example the robot will continue until it is stopped. In ToonTalk there exists no traditional iterative control structures, such as while-loops. Iteration is implicit in a robot s action. A robot will continue to work on a box until the guard does not match, or the programmer stops it. Note that when a robot is working, it looks the same as when the programmer was training it in the thought bubble (see figure 4). It is common in ToonTalk to first test the robot by watching it work, then place it to work someplace hidden, like in a separate house or on the back of a picture. The Add 1 robot could for instance work on the x-coordinate sensor of a picture, which would result in the picture moving to the right. 4
3. Usability analysis In the area of visual programming various graphical representations have been used to bridge the gulf between the programmer and the program, thereby making it easier to program for end-users. The program consists of objects and operations and an underlying computation model that determines the behaviour of the program when it is executed. The program representation (the source code) is a human (and machine) readable representation of the program that should help the programmer to understand the structure and the behaviour of the program. The present analysis will be structured around the following usability issues: Understanding. How easy is it to understand what a program will do? Creating. How easy is it to create a program that will perform the intended actions? Debugging. How easy is it to examine and to understand the behaviour of a running program? Editing. How easy is it to make changes to a program? Below we analyse these tasks in the context of the animated program representation of ToonTalk. We use an approach similar to a heuristic evaluation (Nielsen, 2000), where the author is the only evaluator. For each issue, the properties of the program representation are analysed from the perspective of the user. Understanding. In ToonTalk there is no static representation of the program so the only way to find out what a program does is to run it and watch. Since the animated representation and the animated program behaviour are essentially the same thing, understanding what a program does can be done by simply watching it. Note that an understanding of the computation model is required to understand the program. In ToonTalk it is for example necessary to understand the role of the guard in the thought bubble and the implicit iteration. This understanding can be formed in the programmer s mind in many different ways. A child could for instance understand these concepts at the level of the concrete ToonTalk objects, and the animated representation can help to gain this understanding. Creating. To create a program in ToonTalk is done by demonstration. The effects of actions are seen immediately as a robot is trained. Again, the program representation is virtually identical to the program behaviour. To correct mistakes during training is not possible, which is problematic for long program sequences. Debugging. To examine the behaviour of a running program, the programmer gives a box with the test data to a robot and then she watches it work. This is the same procedure as is required to understand what a program does. In ToonTalk debugging can be thought of as execution in slow motion. Editing. ToonTalk has no equivalent of a source code editor. If a robot turns out to not work, or if a mistake is done during the training of a robot, the robot has to be re-trained. It is possible to go inside a robot s thought bubble and watch it work 5
on the box it was originally trained to work on, and then take control at a certain point in the action sequence and continue the training from that point. One conclusion from this analysis is that an animated representation does not support editing very well. This requires a static representation that is real time independent. It is can also be difficult to get a picture of what a program does and to get an overview of a program. Animation maps well to dynamic behaviour, but is less suited to give an overall view. While it can be clarifying to watch the program work, it is also the case that this requires the programmer to remember the behaviour of the program. In a sense, an animated representation vanishes as it is observed, it is consumed by the passing time. There is no persistent view available to the programmer in the form of an external representation that helps her to maintain a mental model of the program. The paradox of immediateness and ungraspability appears to be generic for representations that are experienced in real time, like film and music (note that a music score is represented in a static form, and that the musician transforms the static score to dynamic tones). 4. A static representation for ToonTalk A robot editor based on storyboards has been suggested for ToonTalk (Kahn 1996b, p. 15). A storyboard consists of a sequence of static pictures that shows the key frames of a movie. Storyboards are commonly used in both movies and animated films to plan the structure of a scene prior to shooting. The purpose of a storyboard is not only to be a diagram of events, it should, more importantly, communicate the experience and reflect the feeling of a sequence (Thomas & Johnston 1981, p. 197). Comics add several dimensions to storyboards. The visual language of comics is intended for a static medium, not an animated one, and has therefore developed a rich set of visual elements that express dynamics and time. One of the most basic devices for expressing time is shared with the storyboard, a sequence of panels or frames (Eisner 1985, p. 28). When we learn to read comics we learn to perceive time and space spatially, time and space become one and the same (McCloud 1993, p. 100). Thus, comics can be thought of as spatial time. In addition to the frame, comics employ a rich set of graphical markers for indicating dynamics. A primary device for capturing speech and sound is the voice balloon. Other examples are markers for motion and emotions. Speed lines are a common type of motion markers (McCloud 1993, p. 110-113). Interestingly, elements such as speed lines are now also found in animated cartoons, for instance in Dennis the Menace. Comics also employ many animation techniques such as squash and stretch, anticipation and exaggeration (Thomas & Johnston 1981, pp. 47-69). Thus, there is a strong relationship between comics and animation, and there is a two-way influence between the two art forms. 6
The similarities between animations and comics suggest that comic strips are a candidate for a static representation of ToonTalk programs. In figures 5-7 examples of comic strip representations for the Add 1 robot are shown. Only the most basic element of comics has been used in the above examples, the framing of time. The graphics for frames are intended to resemble that of boxes. Additional graphical language elements like motion markers could also be used, but this is not shown in these examples. Figure 5. This strip corresponds to the training sequence of the Add 1 robot shown in figure 2. Frame three contains an intermediate step that shows the mouse adding the numbers with his hammer. Figure 6. This is a shorter version of the previous strip. Here the frame showing the intermediate step of the mouse adding the numbers has been removed. Figure 7. The shortest possible strip for the Add 1 robot. In this strip only the start and end states are shown. This could be useful as a compact representation of the result of a robot s actions. The following is a usability analysis of the comic strip representation, using the same usability issues as for the analysis of the animated representation. 7
Understanding. To find out what a program does, the programmer has to examine the strip that describes the sequence of actions. The static representation and the actual behaviour are not identical, and therefore the programmer must visualise the program behaviour inside her mind. The strip provides good support for doing this, but it is less immediate than the animated representation. The static representation does not help the programmer to gain an understanding the computation model in the same direct way as the animated representation does. However, if the programmer is familiar with the computation model, she might be able understand the program at a glance, which can be significantly less time than it takes to watch the corresponding animation. It is possible for the programmer to examine the strip independent of any real time constraints, as she can go back and forth in the sequence at her own pace. The static representation also supports an external overall view of the program that in the case of the animated representation must be created as an internal representation in the mind. Creating. To create a program using the static representation the programmer would have to edit the strip and place objects such as boxes and numbers in the frames. Since the program is not created by demonstration, the programmer would not be able to see the effects of the actions in the same direct way as when programming with the animated representation. However, programming by demonstration could be used also with the comic strip representation. Each step (frame) in the computation could for instance be executed locally when the frame is composed, and the system could also generate a template for the next frame that contains the result of the current step in the computation. There is also the possibility of recording the action by demonstration entirely, and view and edit it in the comic strip form. This can be compared to algorithm visualisation, however, the relation between the representations is reversed, as the static representation is used to give an overview of the animated representation. Debugging. The comic strip representation could be used to visualise an executing program by showing how the content of a box is changed from frame to frame. The frame currently executed could be highlighted. While this is not strictly a static representation since it changes over time, the programmer could be in control of the speed of execution and she could also go back in time. This control could of course also be used for the animated representation, and recent versions of ToonTalk have support for time travel. Editing. Editing would be done in the same way as programs are created, by arranging objects in frames. To edit the frames and the contents of the frames, the usual box editing operations of ToonTalk could be used, for instance the vacuum for copy and paste, the magic wand for copying, the mouse for composition, and so on. Note that we have not detailed the use of these in this paper. To conclude the above analysis, it seems as if the major functions provided by the comic strip representation are overview and edit. Since many advantages of the animated representation is lost, such as programming by demonstration and debugging by slow execution, it seems reasonable to view the static representation as a complement in the case of ToonTalk. For instance, programs could be created and debugged using the animated representation, and the static representation 8
could then be generated by the system to provide overview and additional editing capabilities. A technical issue is how to slice an animation into strips frames. What should be included in each frame? The examples above show that the number of intermediate steps between atomic actions can vary. An often discussed issue in visual programming is the use of screen space. Visual representations can require a large display to be comprehensible. An advantage with animated representations is that they are consistent in the use of screen space. A long animation does not automatically use more screen space than a short one. The opposite holds for static representations. A long comic strip uses more screen space than a short one. There are, however, interface techniques for managing space. In ToonTalk objective zoom is frequently used. (Objective zoom means changing the size of objects, rather than the position of the viewer, as in subjective zoom.) This zooming technique could also be used with comic strips. Only the strips currently of interest need to be picked up and enlarged. 5. Discussion Comics are interesting with respect to programming, because a static program representation is needed to edit a program in a straightforward way. A comic strip is like frozen time laid out on a format that provides an overview and helps creating a mental model of the sequence of events. Comics share many properties with the animated representation of ToonTalk. The visual language is well suited to express dynamics, it is concrete, it is well known by the user group (children), and it is also fairly universal and standardised. This can be contrasted to formalism such as flow charts and data flow diagrams that are rather abstract and lack elements such as concrete objects and characters. Graphical re-write rules are similar to comic strips in that they also use frames that contain concrete objects to show the computation state. The StageCast system is an example of how programming by demonstration can be combined with a static program representation (Smith et al. 2000). However, the authors make no reference to the visual language of comics. As has been show, comic strips that use ToonTalk computation objects can also express programs in the concurrent constraint computation model, a more powerful and expressive computation model than graphical re-write rules. To learn more about the properties of static and dynamic program representation, empirical studies must be made. Here are a number of things that could be tested together with children: Show programs in the suggested comic strip representation to children and ask them what the program will do. Show strips of varying length to children and ask what the program will do. How much can be eliminated from the strip while retaining understandability? Show a program that consists of many strips and ask what the program will do. (Multiple strips could be evaluated for instance by trying the guard frames sequentially, or non-deterministically.) 9
Show an animated program to children and ask them to draw strips that describe the program they have seen. Ask children to create simulated programs by composing strips using objects cut out from paper. References Burnett, Margaret. (2000) Visual Programming Language Bibliography. Available at: http://www.cs.orst.edu/~burnett/vpl.html. August, 2000. Eisner, Will. (1985) Comics & Sequential Art. Poorhouse Press, Florida. Janson, Sverker and Haridi, Seif. (1993) An Introduction to AKL. A Multi- Paradigm Programming Language. Swedish Institute of Computer Science, December, 1993. Available at: http://www.sics.se/. Kahn, Ken. (1995) ToonTalk An Animated Programming Environment for Children. Available at: http://www.toontalk.com/. Kahn, Ken. (1996). Drawings on Napkins, Video Game Animation, and other ways of Programming Computers. Communications of the ACM. August, 1996. Available at: http://www.toontalk.com/. Kahn, Ken. (2000). Generalizing by Removing Detail. Communications of the ACM. Vol. 43, No. 3, pp. 104-106. March, 2000. McCloud, Scott. (1993). Understanding Comics, The Invisible Art. HarperCollins Publishers. New York. Nardi, Bonnie A. (1993). A small Matter of Programming. The MIT Press. Cambridge, Massachusetts. Nielsen, Jacob. (2000). Heuristic Evaluation. Available at: http://www.useit.com/papers/heuristic/. September, 2000. Smith, D. C., Cypher, A., Tesler, L. (2000). Novice Programming Comes of Age. Communications of the ACM. Vol. 43, No. 3, pp. 75-81. March, 2000. Thomas, Frank and Johnston, Ollie. (1981). Disney Animation, The Illusion of Life. Abbeville Press. New York. 10