Supporting the Evolution of a Software Visualization Tool Through Usability Studies

Transcription

1 Supporting the Evolution of a Software Visualization Tool Through Usability Studies Andrian Marcus, Denise Comorski, Andrey Sergeyev Department of Computer Science Wayne State University Detroit MI {amarcus, dcomors, Abstract The paper presents a usability study conducted with graduate and undergraduate computer science students, designed to evaluate the effectiveness of a software visualization tool named sv3d, and to provide necessary user data for the evolution of the system. Sv3D is a software visualization tool for comprehension of large software, capable of displaying source code and associated metrics in three dimensions. The participants in the study answered two types of questions: one set provided objective measurements to support the formulated hypotheses with respect to the accuracy and speed of the users answering questions using sv3d; the second set of questions provided subjective measurements that were used to support the evolution of sv3d. We formulated two null hypotheses with respect to accuracy and time respectively. The collected data supported one hypothesis and rejected the other. 1. Introduction Software visualization is a maturing area of research. In addition to taxonomies [18, 20, 23, 24], a variety of techniques and tools have been developed, which address a range of tasks from algorithm animation to support learning to visualization of software structure, data, and metrics to facilitate comprehension. All of these tools and technologies share a common promise that they help the user better understand aspects of the software and, in turn, help better perform specific software engineering tasks. Software visualization is situated at the intersection of information visualization, software engineering, human computer interaction, graphics, and cognitive psychology. Thus, researchers in this area borrow a number of methodologies from these fields, in particular, those involved with the evaluation of software visualization and comprehension techniques and tools. There are several ways in which researchers and practitioners choose to evaluate their tools. The ultimate proof for the quality of a tool is, of course, its wide adoption and use in the research community or in industry. To achieve this goal a mature tool is needed. While the software visualization field itself is maturing, many of the existing tools are still prototypes, more or less advanced. Some of them are generally used for proof-of-concept purposes, while others have a restricted user base outside their originator research group. There is no single evaluation procedure that fits all software visualization or comprehension systems. The issue of what type of evaluation studies is best suited for a particular type of applications has been studied by information visualization and cognitive psychology researchers [16]. The problem is still far from being completely solved. The consensus in the field is that the type of studies (e.g., usability, user, or case studies) one needs to perform depends on the technology and comprehension task at hand. With all this in mind, researchers in software visualization seem to prefer the following types of evaluations: a) Interviews with users of software visualization systems; b) Case studies performed by a very small group of users (usually the builders of the tool) on one or more subject software systems; and c) Usability studies involving several human subjects that are required to solve particular tasks using some software visualization tool (or tools). The subjects level of expertise ranges from students to seasoned software professionals. The last two categories (b and c) can be further divided into studies that are aimed at the evaluation of a single tool or comparative studies of multiple software visualization systems. An inherent problem faced by researchers in the field is access to appropriate subjects to conduct evaluation experiments. Students make up most of the participants in such experiments. This is well suited for tools that are

2 aimed at supporting learning activities (e.g., algorithm animations), but it is less adequate for tools that are intended to support software comprehension during the activities of software professionals in an industrial setting. This paper presents a usability study conducted with computer science students, for the evaluation of a software visualization tool, sourceviewer3d (sv3d) [19]. The goal of the study is two-fold: to assess the affects of using sv3d as a new technology to support program comprehension and to gather feedback information from users, which will support further development of sv3d. 2. Related Work There is a certain body of work available that focuses on evaluating 3D visualization systems in comparison to 2D ones. Hubona et al. [11] suggest that users' understanding of a 3D structure improves when they can manipulate the structure. Ware and Franck [29] indicate that displaying data in three dimensions instead of two can make it easier for users to understand the data. In addition, the error rate in identifying routes in 3D graphs is much smaller than in 2D graphs [30]. 3D representations have also been shown to better support spatial memory tasks than 2D [28]. Koike [15] examined the benefit of using a third dimension in visualizing information from parallel/concurrent computer systems. Cockburn et al. [4] evaluated the affects of 2D, 2½D and 3D interfaces with 69 computer science students conducting spatial memory tasks. Outside the 2D vs. 3D debate, several researchers conducted usability and/or case studies for the evaluation of software visualization tools. Several industrial case studies were done with SeeSoft [5], one of the most popular software visualization tools that introduced file maps, also used in sv3d. G SEE [6] was also used extensively in an industrial setting and a related analysis of the adoption issues of a software visualization tool is presented in [7]. Based on the file map metaphor, TARANTULA [13] and GAMMATELLA [22] were also subject to studies for fault localization, like the Aspect Browser [8] and the Aspect Mining Tool [9]. Other major studies with software visualization tools are presented in [2] on using visualization to help identify classes that are likely to change. Several usability studies on software visualization tools were performed by researchers in an academic setting. Macdonald and Miller [17] provide an evaluation of text-based inspection of software versus inspection using a tool called ASSIST (Asynchronous/Synchronous Software Inspection Support Tool). The participants, 43 software engineering students, inspected a single document using a tool or paper-based approach. Their research showed that users performed approximately equally well with the text or with using the tool. Storey et al. [27] used three software comprehension tools in a study to determine their effectiveness in high-level program understanding tasks. Their study involved 30 computer science students using one of three tools: Rigi, SHriMP, or SNiFF+. The students used the tools to complete program understanding tasks against a Monopoly game program. They were encouraged to think aloud as they performed the tasks. They were then given a questionnaire to gauge the usability of the tool and an exit interview. They found that the tools did aid comprehension, even though they appeared to hinder the users in some cases. Ruthruff et al. [25] studied the effects of three visualization techniques for fault location in end-user programming. Through an empirical formative study, they found varying degrees of effectiveness in locating faults, depending on the specific technique. The study was named as such because the end users created test suites, which in turn helped identify areas of effectiveness in the design, which was still in development. Several other studies on software visualization tools, which involved software engineering students have been published: Bladh et al. [3] extended the Treemap metaphor from two dimensions to three; Hendrix et al. [10] studied the effectiveness of a graphical representation called the control structure diagram (CSD); Hundhausen [12] conducted two experiments on algorithm animation; Kehoe et al. [14] performed an experiment in which they evaluated the use of algorithm animations in a homework-style setting. Figure 1. A container in sv3d. It represents the Java application used in the usability study. Each poly cylinder represents a class. The height of the poly cylinder is mapped to the maximum complexity of the class, while the color of the poly cylinder shows the average lines per method in that class.

3 Figure 2. The control panel in sv3d. It allows the user to define the mappings of data to the height or color of the poly cylinders. It also allows setting the transparency for a selected color, and displays the data associated with a selected poly cylinder (bottom part of the figure). 3. An Overview of sv3d Sv3D [19] is a visualization front-end for multiattribute data sets, designed to work specifically with visualization of source code and associated data (i.e., metrics, execution traces, versioning data, etc.). It uses a 3D metaphor for rendering; it can also be used for 2D visualizations. The sv3d metaphor has two main components: poly cylinders and containers (see Figure 1 and Figure 3). Data attributes can be mapped to the height and color of each poly cylinder using the control panel (see Figure 2). Users have available object and space-based manipulations. An individual container can be rotated (see Figure 1 and Figure 3), panned or zoomed. The entire visual space can also be rotated, panned, and zoomed. Individual poly cylinders can be selected by clicking on them. Upon selection, all the associated data is shown in the control panel display window. Colors associated with selected poly cylinders can be set at any transparency to alleviate occlusion and help with selection. Users can save the data mappings and the current position of the light and camera. They can load data with a different representation of the source code (e.g., classes, files, methods, lines of code, etc.) at any time. Images can also be rendered as.bmp files at any time. The current implementation of sv3d is in C++ and sv3d is decoupled from any analysis tool or development environment. The data to be visualized is provided as a set of XML and source code or data files. We are currently working on extending the functionalities of sv3d and are working towards integrating it with a development environment such as Visual Studio.NET. We are also planning to port it to Java as a plug-in to Eclipse. With these major changes in mind, we designed and conducted a usability study to help us evaluate the current prototype and determine what features need to be added and/or modified. 4. Usability Study Design The usability study was aimed at both verifying a set of hypotheses about the efficiency and effectiveness of the sv3d prototype and also to provide support information for the further development of the tool. It has been documented [1, 21, 26] that the introduction of new technology (such as a visualization tool) into a software development environment has several side effects especially in the transition period, before the new technology is fully adopted and personnel properly trained. We expect that the initial usage phase of sv3d will suffer from the same problems. Still we are hoping that sv3d will help improve the accuracy and speed in the analysis of software related data. With this in mind, we formulated two hypotheses and their respective alternatives. The first hypothesis refers to the accuracy of answers. The first null and alternative hypotheses are: H 01 : participants using sv3d will answer questions as accurately as the participants answering questions using text. This means that the average number of correct answers per user will not differ significantly between the two groups. H a1 : participants using sv3d will answer questions with different accuracy than the participants answering questions using text. This means that the average

4 Figure 3. The Java application viewed in sv3d. Each container represents a file. Each poly cylinder represents a line of text from the source code. Nesting level is mapped to height; control structure is mapped to color. The file container representing the AddAnnotations.java file is rotated to show how the third dimension can be utilized. number of correct answers per user will differ significantly between the two groups. The second hypothesis refers to the time required to answer questions. The second null and alternative hypotheses are: H 02 : participants using sv3d will require less or an equal amount time on average to answer a question than the participants answering questions using text. H a2 : participants using sv3d will require more time to answer questions on average than the participants answering questions using text Data Sets The data used in the study consisted of the source code from a documentation software application (HMS) implemented in Java. The software has approximately 4200 lines of code in 27 classes implemented in 18 files. In addition to the source code, two sets of data were available to the users. The first set consisted of information about the source code for every file in the system. The following data was available for every line of text: line number, nesting level, control structure, contains comment or not. The second set of data consisted of a set of metrics computed for each class in the system. Table 1 shows the data for a typical class (i.e., AddAnnotations). Although this type of information may not be considered useful for skilled software engineers, this particular set of metrics was chosen considering the background and experience of the participants. The set includes metrics that would be easily understood by the participants and would not influence the results and major objectives of the study. Table 1. Available data for classes in the HMS system. This particular data is for the class AddAnnotations. Metric Value Lines 181 Lines blank 13 Lines code 123 Lines comment 46 Average lines per method 20 Average lines of comment per method 2 Average method complexity 1 Maximum class complexity 5 Ration comment/code Participants We had 36 participants in the usability study, 12 for the first, pilot phase and 24 for the second phase. The

5 participants were students from the software engineering classes in our department. The department has a sequence of 3 software engineering courses: one undergraduate and two graduate courses (i.e., CSC4110, CSC6110, and CSC7110 respectively). The participants were divided randomly into two groups: one group answered the questions using the sv3d tool, while the other group of students answered the questions using the tabular data with metrics and the source code, utilizing search features in Visual Studio.NET. We called the group of students that used sv3d the sv3d group and the other group the text group. In the first, pilot phase, the more advanced students (i.e., from the CSC7110 course) participated in the usability study. Students who participated in the second phase were less experienced (i.e., from the CSC6110 and CSC4110 courses). The two phases were carried out during the subsequent semesters. Table 2 shows how the students were distributed among the two groups. The results of one of the students were discarded because the computer crashed during his test. We felt that if he were to repeat the test, prior knowledge would affect his performance. So, there were a total of 35 participant students with valid answers: 18 in the sv3d group and 17 in the text group. The students received extra credit in the project component of their respective course. Their performance in the usability study was not a grading factor. None of the students had prior experience with using sv3d. The students in the text group were familiar with the IDE used in the study. Table 2. The distribution of the students from the different classes in the two groups. Students CSC4110 CSC6110 CSC7110 sv3d group Text group Questions The students answered two sets of questions. The first set was used to generate data points to test our hypotheses and the second set of questions was aimed at evaluating their opinion of the tool and the study in general. Considering the background of the users and the goal of the usability study, the questions were relatively easy (at least from the point of view of an experienced software engineer). By formulating easy questions we tried to reduce the bias due to the fact that some of the participants would simply not understand them. The following questions were asked for all participants and used to verify our hypotheses. To avoid confusion, we call these test questions: 1. How many classes does this project contain? 2. Which is the largest class (in terms of lines of text)? 3. Which class or set of classes has the fewest lines of comments? 4. What is the maximum number of lines of code per class? 5. What is the maximum complexity of the class with the most lines of code? 6. Which class or set of classes has the highest comment-to-code ratio? 7. Which three files (not classes) have the fewest lines of text? 8. List the control structures contained within the file CreateHTMLFile.java. Control structures are: if, else, else if, for, switch, and while. 9. How many if/else structures are in the CheckHTMLFile.java file? In order to correctly answer questions 1 through 6, the users needed to access the class metric visualization or table data. In order to answer questions 7 through 9, the users needed to access the file data visualization or the source code. The following questions were given to the participants in the sv3d group. A similar set of questions was given to the students in the text group. The goal of these questions was to gather feedback from first time users about some of the sv3d features. This data will be used for the further evolution of the tool. In addition, some of the questions were aimed at the general evaluation of perceived difficulty of the test questions. To avoid confusion, we call these evaluation questions : 10. How would you rank the overall ease of finding data using sv3d? Multiple choice question with the following possible answers: very difficult; somewhat difficult; neutral; somewhat easy; very easy. 11. Do you feel you had enough information to answer all the questions accurately in a reasonable amount of time? Multiple choice question with the following possible answers: not at all; had some of the information I needed; neutral; had most of the information I needed; had all or nearly all of the information I needed. 12. What information would you have liked to have to help answer the questions more easily? 13. What is the benefit of using sv3d over looking at raw data? Name none, one, or several. 14. What is the drawback of using sv3d over looking at raw data? Name none, one, or several. 15. Would you use sv3d on a work or school project? Multiple choice question with the following possible answers: not likely; maybe; neutral; probably; definitely. 16. What is the best feature of sv3d? 17. What is the worst feature of sv3d?

6 18. What could be added to sv3d to make it a better application? Name nothing, one thing, or several things. 19. Which was the easiest question? Why? 20. Which was the most difficult question? Why? 21. Use the space below to list any question(s) you did not understand. 22. Please provide any additional comments Visualizations and Instructions The users in the sv3d group were required to use the sv3d tool to answer the questions. Two visualizations were available to them. One was at a class level (see Figure 1), where each class is represented by a poly cylinder. The user could map, in this context, all of the metrics from Table 1 to either the height or the color of the poly cylinders. For example, Figure 1 shows the HMS system with the height of the poly cylinder mapped to the maximum complexity of the class and the color of the poly cylinder mapped to the average lines per method in that class. The users were able to rotate, move, zoom, use transparency, and to change the mappings at run time. By clicking on any of the poly cylinders, all the metrics for the corresponding class were displayed in the control panel window (see Figure 2). The other available visualization represented each file in the system as a container and each line of text as a poly cylinder (see Figure 3). Once again, the users could map the available data to both the height and the color of the poly cylinders. The same standard interactions were available to the users as above. The users from the sv3d group also had access to the source code through the sv3d project panel (not shown here), but none of them used the feature during the study. The users in the text group had access to table data with all the metrics from Table 1 for each class, in a single text file. They also had access to the source code through the Visual Studio.NET environment. They were allowed to use any features of the IDE Procedure Each participant in the study had a short training session before the actual test. In the pilot study (with the CSC7110 students) the students in the sv3d group participated in a 45 minute training session just prior to the administration of the test. One of the co-authors made an approximately 30 minute presentation of the tool and answered questions. The users were allowed to use the tool during the presentation to experiment with the presented features and they had additional 15 minutes to practice. The users in the text group participated in a 30 minute training session where the data was presented, along with sample questions similar to those in the questionnaire. Each group was provided with a set of printed instructions and shortcuts (for sv3d or Visual Studio) that they could consult during the test. Following the pilot, we decided to have the training session a few days prior to the test. In addition, we made preparatory training material for the sv3d group available over the web. This included a copy of the program that they could install on their home computer, a 7 minute downloadable video demo, and a set of sample data and questions. The demo sv3d application had the same features as the one used in the study, but visualized different data than what was used in the study. The users were instructed to complete the web training prior to the training session. Thus, the major difference in the second usability study was the fact that the sv3d participants completed preparatory work prior to the test. # of correct answers Users sv3d Figure 4. Number of questions correctly answered (Y axis) by each participant (X axis) in each group. Once the preparatory work was complete, the participants were brought in to practice with sv3d. They were free to ask any questions during this time. Once the participants stated that they were comfortable with the tool they were permitted to leave. The training in the second phase for the users in the text group did not change from training in the pilot study. The training time was the only change in the procedure between the two phases. After the training, in each phase, the participants proceeded to answer the test questions in the questionnaire. Additional software was developed to capture answers in the questionnaire submitted by the participants. This software collected such statistics as how long it took the participant to answer each question, how many times the participant returned to the question, how many times the participant changed the answer, and a history of their answers. The students were allowed to change their answers as many times as they liked, and take as long as necessary to complete the test. If users did not understand a particular Text

7 question, they were free to ask the test coordinators (2nd and 3rd authors of the paper). The test coordinators gave assistance on technical use of the tool or question explanations only. Table 3. Statistics for accuracy data (no. of correct answers) for users in the text and sv3dgroups: maximum, minimum, mean, standard deviation. Max Min Mean SD sv3d group Text group Results In order to see if the collected data supported our null hypotheses we examined the objective measurements. We analyzed how many questions were answered correctly by each participant and how long it took each participant to complete each question in the questionnaire. The analysis of the data yielded that sv3d users on average answered 6.17 questions correctly, whereas the students from the text group on average answered 5.94 questions correctly (see Table 3). Figure 4 shows a comparison of the number of correctly answered questions between the each participant from the two groups. The analysis of time it took each participant to complete the questionnaire provided some unexpected results. We expected that the participants using sv3d would retrieve needed data faster than those participants using text (tables and IDE with loaded solution). The usability study showed that it took on average sv3d participant seconds to answer each individual question, whereas it took participants from the text group seconds on average per question (see Table 4). Therefore, participants from the text group on average needed less time per question. Figure 5 shows a comparison between each participant from the two groups of the average time (in seconds) they needed to answer each question. Table 4. Statistics for time data (seconds needed to answer a question on average): maximum, minimum, mean, standard deviation. Max Min Mean SD sv3d group Text group Accuracy For the participants using the visualization tool we observed slightly higher accuracy than that produced by the participants from the text group, but it was not enough to be statistically significant. The Kolmogorov-Smirnov Normality test was performed on the data and there was no significant evidence (p=0.0775) that the data distribution does not satisfy normality (see Figure 6). Time (seconds) Users sv3d Figure 5. Average time per question (in seconds Y axis) needed for each user (X axis) to answer. Subsequently, we performed the F-test on the data to test whether the variances are different in statistical terms. Our variance ratio (F=1.52) is less than (F_Critical=2.28), so our variances do not differ significantly. After the Kolmogorov-Smirnov Normality test and the F-test, we can validly perform two-sample t- test on the data, assuming equal variances. Figure 6. Histogram showing the distribution of accuracy data. Number of questions answered correctly X axis. Number of participants Y axis. The t-test showed that there is no significant evidence to conclude the means for two the independent samples differ (p=0.72). Since the means are very close and there is no significant evidence to conclude the means for the two independent samples differ our first null hypothesis H 01 stands - participants using sv3d will answer questions as accurately as the participants answering questions using text Time As a result of time analysis, our null hypothesis H 02 : participants using sv3d will require less or an equal Text

8 amount of time on average to answer a question than the participants answering questions using text was rejected. In fact, the opposite was true - participants using text (text tables and source code loaded into IDE) required less time to answer questions than the participants using sv3d answering questions. 6. Discussion The overall results of the usability study were somewhat surprising. This section discusses our thoughts and explanations of the results, beyond their statistical significance Objective Measurements While we expected the sv3d users to perform better both in terms of accuracy of answers and time, we were happy to observe that the sv3d users did not answer worse than the text users. It took them, however, longer on average to answer the questions. We believe, also based on the evaluation questions that this fact is due to the lack of experience with the tool. More than that, it seems that a novice software engineer needs several (or more) hours of tutoring and practice before a gain in efficiency is observed. We observed how the students used sv3d and several usage patterns were inferred. Very few of them used the tool in a similar manner as the students experienced with sv3d, like the authors. Frequently, some participants expressed frustration over technical aspects of sv3d, such as how to disable the track ball feature or how to drill down to the underlying text. This was a clear result of their lack of training. The answers we received on the evaluation questions reaffirmed our suspicions that some participants did not feel fully comfortable using sv3d. We obtained comments such as This tool is cool really. The idea and the way to present the software source code. However, it should be much easier to use and Nice tool, but it takes some time to get used to it. Another factor that may explain the time results is related to the size of the data used in the experiment. Software visualization tools such as sv3d are known to be more effective when large amounts of data need to be explored and are visualized. Our data supports this theory and it shows that the benefits of using software visualization tools for small systems are not very clear. As part of our future work, we plan to conduct additional experiments against larger software. Another surprise element was the quality of the responses. We expected most students to answer in average at least 75% of the questions correctly. We believe the difference comes from two sources: some students misunderstood questions and were reluctant to ask for clarification, and since the rewarding factor was an extra credit for the course project and participation was voluntary, many of the weaker students in the class participated. Once again, the evaluation questions partially confirmed this theory Subjective Measurements The discussion so far focused on the analysis of the results from questions 1 through 9, used to validate our hypotheses. Questions 10 through 22 were extremely important from the evolution of the tool point of view. They also helped explain some of the results observed in the test questions. Table 5 summarizes the answers to these questions. These answers provided us with valuable information that supports the implementation of new features in sv3d. While these features were planned, not all were high on our priority list. For example, the participants suggested a feature allowing the user to collapse containers that are irrelevant to the task at hand, history (Undo/Redo), and user defined mapping. Only the user defined mappings feature was high on our list Conclusions Validity Wohlin et al. [31] describe four types of validity threats in empirical studies: conclusion validity, internal validity, external validity, and construct validity. These are all relevant for our study. Several factors potentially affect the validity of our observations. First of all, one has to consider the background and experience of the participants. In each phase of the study we distributed the participants randomly in the two groups. Most of the graduate students were international students, some having considerable language problems that might have influenced how they interpreted the questions. Some of the undergraduate students had a poor knowledge of object-oriented programming. For example we had two subjects answering question #4 What is the maximum number of lines of code per class? with the answer no limit instead of a numerical answer. One of them was from the sv3d group and one of them from the text group. In fact, as mentioned before, we expected that the students will respond better on average. In addition, students were assured of the anonymity of their answer, and all of them received the extra credit simply for participation. This fact possibly influenced the seriousness of some of them in answering the questions. Second, the change in the procedure between the two phases could be perceived as a threat to the validity. One of our conclusions was that the long time required by the users in the sv3d group to answer was due to their lack of experience with the tool. It was expected that in the second phase the average response time per question

9 would decrease. In reality the opposite happened; in the pilot phase the average response time per question for the sv3d users was 187 seconds, while in the second phase was 227 seconds. Table 5. Summary of the answers for the evaluation questions 10 through 22 (see section 4.3) Q# Answer Summary 10 Most participants reported answers tending towards easy. A majority chose Somewhat easy and Very easy. Five reported Somewhat difficult. 11 A vast majority of participants responded that they had most or nearly all the information they needed. 12 Many participants responded that they wished they had more examples or an explanation of how to approach specific problems. One participant expressed a desire to redo the test. 13 Participants responded that sv3d was beneficial for finding information quickly, as opposed to looking at every source code file. It made patterns in the data quickly obvious. A number of respondents cited that sv3d aided in making some relationships more apparent. 14 Several participants responded that there was no drawback to using sv3d. Several others cited that occlusion was problematic in the visualization, but rotation and transparency helped in alleviating this problem. 15 Participants were about equally divided on this question. An equal number of participants responded Maybe as did Probably. Other responses were approximately equally distributed on both sides of the median. 16 A number of participants mentioned color as the best feature of sv3d. Participants also mentioned the concept of visualized data versus text as being a strong point. 17 Participants mentioned various technical aspects of the GUI, such as zooming, panning, and track ball. Several participants said the features they mentioned simply required time to get accustomed to the tool. 18 Half the participants reported that no features needed to be added to make sv3d a better application. Of those who did mention items, several reported the desire to work on a container individually, without having to deal with the entire visualization. 19 Participants reported that questions in which a simple mapping could be applied were the easiest. For example, finding the class with the greatest number of lines of text was reported to be easy. 20 Participants reported that the questions which required them to look at every container or every poly cylinder were the most difficult. Often it was difficult to distinguish between two similar colors in the visualization. 21 Several participants did not understand the questions that referred to control structures. 22 Several participants responded that the tool looked promising and would be good for large systems. A few responded that the interface could be improved. The decrease was insignificant. We also assumed that more training time will allow for better answers in the second phase. On the other hand, the experience level of the participants was much lower. Once again, the opposite happened; in the pilot phase, the sv3d users answered 72.22% of the questions correctly on average, while in the second phase they answered on average 67.36% of the questions correctly. 7. Conclusions and Future Work While surprising, the results of the study are not disappointing. They underline the inherent difficulty of users to quickly adapt to new technology, in this case the visualization tool, despite their overall positive impressions about it. We have shown that even with first time users, with little training time, the accuracy of the answers of the participants from the sv3d group was not worse than that of those in the text group. It took on average longer for the users in the sv3d group to answer. This is however, to be expected with the introduction of new technology. We are planning subsequent studies that will focus on more complex questions on software comprehension, with more data, and possibly with more experienced users. We will also investigate which type of visualization (i.e., mappings supported by sv3d) is best to answer certain classes of questions related to specific software engineering tasks. 8. Acknowledgements Several people were involved in the design and implementation of the initial sv3d prototype: Louis Feng and Jonathan Maletic. We thank Sorin Draghici for his help with the statistical interpretation of the collected data. We also would like to thank the students for participating in the user study. 9. References [1] Anderson, J. A. and Ward, E. S., "Technology transfer: experiences in introducing object-oriented methods to government projects", in Proceedings of 8th Annual Washington Ada Symposium & Summer SIGAda Meeting on Ada, McLean, VA, 1991, pp [2] Bieman, J. M., Andrews, A. A., Yang, H. J., "Understanding change-proneness in OO software through visualization", in Proceedings of the 11 th IEEE International Workshop on Program Comprehension, May 10-11, 2003, pp [3] Bladh, T., Carr, D., Scholl, J., "Extending Tree-Maps to Three Dimensions: A Comparative Study", in Proceedings of the 6th Asia-Pacific Conference on Computer-Human Interaction (APCHI 2004), Rotorua, New Zealand, June 29 - July 2, 2004, pp

10 [4] Cockburn, A., McKenzie, B., "Evaluating the effectiveness of spatial memory in 2D and 3D physical and virtual environments", in Proceedings of the SIGCHI conference on Human factors in computing systems, Minneapolis, MN, USA, 2002, pp [5] Eick, S. G., Steffen, Joseph L., Sumner, Eric E., "Seesoft - A Tool For Visualizing Line Oriented Software Statistics", IEEE Transactions on Software Engineering, vol. 18, no. 11, November, 1992, pp [6] Favre, J.-M., "A Flexible Approach to Visualize Large Software Products", in Proceedings of ICSE'01 Workshop on Software Visualization, Toronto, Ontario, May [7] Favre, J.-M., Estublier, J., and Sanlaville, A., "Tool Adoption Issues in a Very Large Software Company", in Proceedings of 3rd International Workshop on Adoption- Centric Software Engineering (ACSE'03), Portland, Oregon, USA, May 9, 2003, pp [8] Griswold, W. G., Yuan, J. J., and Kato, Y., "Exploiting the Map Metaphor in a Tool for Software Evolution", in Proceedings of 23rd International Conference on Software Engineering, Toronto, ON, May , pp [9] Hannemann, J. and Kiczales, G., "Overcoming the Prevalent Decomposition in Legacy Code", in Proceedings of ICSE 2001 Advanced Separation of Concerns Workshop, Toronto, Canada, 15 May [10] Hendrix, T. D., Cross II, J. H., Maghsoodloo, S., McKinney, M. L., "Do visualizations improve program comprehensibility? experiments with control structure diagrams for Java", ACM SIGCSE Bulletin, vol. 32, no. 1, 2000, pp [11] Hubona, G. S., Shirah, G. W., and Fout, D. G., "3D Object Recognition with Motion", in Proceedings of Computer Human Interaction, 1997, pp [12] Hundhausen, C., "Integrating algorithm visualization technology into an undergraduate algorithms course: ethnographic studies of a social constructivist approach", Computers & Education, vol. 39 (3), Nov. 2002, pp [13] Jones, J. A., Harrold, M. J., and Stasko, J., "Visualization of Test Information to Assist Fault Localization", in Proceedings of 24th International Conference on Software Engineering (ICSE02), Orlando, FL, 2002, pp [14] Kehoe, C., Stasko, J. Taylor, A., "Rethinking the Evaluation of Algorithm Animations as Learning Aids: An Observational Study", International Journal of Human- Computer Studies, vol. 54, no. 2, Feb. 2001, pp [15] Koike, H., "The role of another spatial dimension in software visualization", ACM Transactions on Information Systems (TOIS), vol. 11, no. 3, July 1993, pp [16] Kosara, R., Healey, C. G., Interrante, V., Laidlaw, D. H., and Ware, C., "Visualization viewpoints", Computer Graphics and Applications, vol. 23, no. 4, July-August 2003, pp [17] Macdonald, F., Miller, J., "A Comparison of Tool-Based and Paper-Based Software Inspection", Empirical Software Engineering: An International Journal, vol. 3, no. 3, September 1998, pp [18] Maletic, J. I., Marcus, A., and Collard, M. L., "A Task Oriented View of Software Visualization", in Proceedings of 1st IEEE Workshop of Visualizing Software for Understanding and Analysis, Paris, France, June , pp [19] Marcus, A., Feng, L., and Maletic, J. I., "3D Representations for Software Visualization", in Proceedings of 1st ACM Symposium on Software Visualization (SoftVis'03), San Diego, CA, June , pp [20] Myers, B. A., "Taxonomies of Visual Programming and Program Visualization", Journal of Visual Languages and Computing, vol. 1, no. 1, March 1990, pp [21] Nishiyama, T., Ikeda, K., and Niwa, T., "Technology Transfer Macro-Process A Practical Guide for the Effective Introduction of Technology", in Proceedings of 22nd International Conference on Software Engineering (ICSE00), Limerick, Ireland, 2000, pp [22] Orso, A., Jones, J., and Harrold, M. J., "Visualization of Program-Execution Data for Deployed Software", in Proceedings of 1st ACM Symposium on Software Visualization (SoftVis'03), San Diego, CA, June , pp [23] Price, B. A., Baecker, R. M., and Small, I. S., "A Principled Taxonomy of Software Visualization", Journal of Visual Languages and Computing, vol. 4, no. 2, pp [24] Roman, G.-C. and Cox, K. C., "A Taxonomy of Program Visualization Systems", IEEE Computer, vol. 26, no. 12, December 1993, pp [25] Ruthruff, J., Creswick, E., Burnett, M., Cook, C., Prabhakararao, S., Fisher II. M., Main, M., "End-User Software Visualizations for Fault Localization", in Proceedings of ACM Symposium on Software Visualization, San Diego, CA, June 2003, pp [26] Smith, A. L., Jr. and Douglas, A., Jr., "Introducing new technology to the air traffic controller: Implications for skill acquisition and training", in Proceedings of 17th Digital Avionics Systems Conference, Oct Nov , pp [27] Storey, M.-A. D., Wong, K., Muller, H.A., "How Do Program Understanding Tools Affect How Programmers Understand Programs", Proceedings of the 4 th Working Conference on Reverse Engineering (WCRE'97), Amsterdam, The Netherlands, October 6-8, 1997, pp [28] Tavanti, M. and Lind, M., "2D vs 3D, Implications on Spatial Memory", in Proceedings of IEEE Symposium on Information Visualization (INFOVIS'01), San Diego, CA, October , pp [29] Ware, C. and Franck, G., "Viewing a Graph in a Virtual Reality Display is Three Times as Good as a 2D Diagram", in Proceedings of IEEE Visual Languages, 1994, pp [30] Ware, C., Hui, D., and Franck, G., "Visualizing Object Oriented Software in Three Dimensions", in Proceedings of the 1993 Conference of the Centre for Advanced Sudies on Collaborative Research (CASCON'93), Toronto, Ontario, October 1993, pp [31] Wohlin, C., Runeson, P., Host, M., Ohlsson, M. C., Regnel, B., and Wesslen, A., Experimentation in Software Engineering: An Introduction, Amsterdam, Kluwer Academic Press, 1999.