Empirical Software Engineering Research - The Good, The Bad, The Ugly

Transcription

1 Empirical Software Engineering Research - The Good, The Bad, The Ugly Elaine J. Weyuker AT&T Labs - Research 180 Park Avenue Florham Park, NJ weyuker@research.att.com Abstract The Software Engineering Research community has slowly recognized that empirical studies are an important way of validating ideas and increasingly our community has stopped accepting the sufficiency of arguing that a smart person has come up with the idea and therefore it must be good. This has led to a flood of Software Engineering papers that contain at least some form of empirical study. However, not all empirical studies are created equal, and many may not even provide any useful information or value. We survey the gradual shift from essentially no empirical studies, to a small number of ones of questionable value, and look at what we need to do to insure that our empirical studies really contribute to the state of knowledge in the field. Thus we have the good, the bad, and the ugly. What are we as a community doing correctly? What are we doing less well than we should be because we either don t have the necessary artifacts or because the time and resources required to do the good is perceived to be too great? And where are we missing the boat entirely in terms of not addressing critical questions and often not even recognizing that these questions are central even if we don t know the answers. We look to see whether we can find some commonality in the projects that have really made the transition from research to widespread practice to see whether we can identify some common themes. I. INTRODUCTION A long, long time ago, I can still remember how the papers used to make me smile. (Sorry for the musical allusion). A. The Early Days (The Bad Old Days) Papers in the early days of Software Engineering were typically about some new technique to do something related to software - specify, design, edit, write, test, or maintain software, for example. We had great new ideas and wanted to tell everyone about them. We gave plausible explanations about why our new techniques were better than, maybe even far better than, earlier techniques. We all read those papers and we often learned from them, and they spawned new great ideas and we wrote about them, and those new ideas spawned even newer and greater ideas and we wrote about them... and so on for years. But at some point, some researchers recognized that much of the research literature was not having a big impact on industrial software production and our community wondered why not, although it was easy to blame the ignorant practitioner who either didn t read the literature we were producing, or didn t perceive the value. During this early time period, few, if any, Software Engineering research papers came with any sort of evidence that they worked in the large. Instead they came with arguments about why they were improvements over what researchers believed were the norm in practice. Certainly testing papers, including my own widely cited initial Dataflow Testing papers [14], [15], provided no concrete empirical evidence that they were improvements over such techniques as statement or branch testing which we believed everyone did. Finally some in our community realized that we really weren t providing practitioners with the type of evidence necessary to get them to change the way they were doing things, and maybe that was the problem. But it took us a long time to come to this realization. Another factor may have been that most of the people writing Software Engineering research papers had little or no real experience specifying, developing, testing, or maintaining large software systems that were in regular use for any period of time. The reality was that most of the authors were not system builders and never had been, and so while the research papers may have contained some gems of ideas, they may not have resonated with people who did have experience regularly developing large software systems because the ideas didn t connect with the process or reality that many practitioners had. That is why I italicized the word believed in the paragraph above. Many in our community will never make the contribution that they could be making because they have no real understanding of how software is built and what the stumbling blocks or impediments are when building large software systems. In a sense, this is our ugly. I will be returning to this point later to discuss how we can beautify our field. Of course, even in the early days, some researchers did try to provide at least very simple examples of how their technique would work, typically using toy code. For example, in our dataflow testing papers, we did provide some small program fragments showing cases in which dataflow testing would find a problem while branch testing would not. In retrospect, I cannot imagine that that convinced anyone

2 other than other researchers, who went on to write other papers that we researchers read and practitioners ignored. The sad part is that I should have known better. After all, I had spent two stretches working in industry writing code. The first time was between my Bachelor s and Master s degrees when I worked as a programmer for a large oil company, and the second was between my Master s degree and Ph.D. degrees when I worked as a systems engineer for a large computer manufacturer. One thing that we did provide in [14], [15], which at the time was highly unusual, were mathematical proofs that dataflow testing was better in some concrete sense than statement or branch testing. We also looked at various ways of formally measuring better [19], [7], [8]. Therefore, while we did not provide much in the way of empirical evidence, we at least did provide some analytical evidence of the usefulness of our ideas. But that was extremely unusual and most people did not provide any real evidence of either an empirical or analytical nature to assess the usefulness or effectiveness of their proposals. The idea that one could write and publish research papers in the best conferences and journals, while providing neither analytical nor empirical evidence that the proposed technique was worthwhile, is testimony to the lack of maturity of our field through much of the 1980 s. B. The Not So Bad Not So Old Days Once our community recognized that we really needed to provide some concrete evidence that our new ideas were worthwhile, we started seeing empirical studies included in papers. These typically were more proofs of concepts than empirical studies likely to convince a practitioner to invest time and effort in trying to use the newly proposed technique in the large. Since a substantial percentage of the research literature is written by academics, their empirical studies often used students as subjects and the artifacts being implemented (or specified, designed or tested) were frequently toy programs. Typically they were being written for the study, during a portion of a semester, and therefore were of necessity small, and would never be used in the field. In fact they would probably never be used by anyone other than the student and then, only for this assignment, or by the researcher to collect data of some sort. To justify why this was a reasonable way to assess the practicality of their proposed techniques, these researchers often argued that students were proto-programmers. After all, the students would graduate and become programmers soon, so there was not much difference between them and professional programmers or at least programmer trainees. Additionally, it was sometimes argued that some percentage of the students already had some professional work experience. While there may be some truth to those assertions, by and large these arguments were unconvincing. However, what makes these sorts of studies, particularly unhelpful is the nature of the artifacts being produced - small and throwaway as compared to large and long-lived. In my opinion, there is little reason to believe that we can draw conclusions about the behavior of large long-lived software systems and the behavior of professional software developers, by studying the behavior of student programmers producing small, throw-away software. Of course, underlying these student studies was the fact that many of the researchers had no access to professional system specifiers, designers, programmers or testers and so they used what they had, namely students. And the other issue was that most researchers had no access to the code of large software systems, since industrial software code and everything related to it, is typically proprietary. If you didn t work in industry, you generally had no possibility of access to the artifacts you needed to do substantial empirical studies or the personnel needed if you wanted to study them. As I discovered while I was a professor hired to give advice about software testing for a number of large companies, being an academic consulting about some Software Engineering topic probably did not qualify you to have contact with actual software artifacts such as code or to know about things like the true nature of quality issues. The reason is that it is generally too risky to allow an outsider to see these things. Additionally, it is far more cost effective to have a high-priced consultant give advice on how to do something rather than the time-consuming, and therefore expensive, process of actually doing it. There is another issue that distinguishes the programs in these early empirical studies from real programs. Real software systems contain bugs. Of course student programs also have bugs in them, perhaps even more than those produced by professional programmers, but it is quistionable whether the defects that professional programmers make when developing, modifying, or maintaining a large system as part of a team are closely related to the types of defects seen in student-written programs, nor is the rate of bugginess likely to be similar. To deal with that situation, some researchers deliberately inserted or seeded bugs into the subject programs they were studying, or controlled the types and distribution of software defects. It was argued that these seeded bugs somehow represented typical defects that would be seen in the field. But since there have been few if any studies detailing the distribution of bug types in fielded software systems, or even good, useful taxonomies of faults types, and since they would presumably vary substantially depending on such things as the nature of the application, language, programmer skill and experience, it is hard to even know what typical means in this context. Another common practice in software testing research

3 papers with early empirical studies was to consider programs containing a single (seeded) bug at a given time so that it was possible to determine whether or not it was identified by the testing technique being studied. The problem with this approach is related to the problems mentioned above that there is no standard defect rate that is appropriate for all software, and the research community knows little about the types and distribution of bugs in large production systems. Additionally, the limitation to single faults neglects the affects of bug interactions which are frequently a very serious issue for real software systems. Note that I have deliberately chosen not to cite papers that had studies using students and/or toy programs. There were many papers in all areas of Software Engineering that fall into this category, and since I am arguing that while many researchers had reasons for doing these sorts of studies, they were not really valuable if what they were trying to show was that the ideas proposed in the paper were usable in large long-lived systems written by teams of professional developers. Also, we are talking here about the dark ages (20 to 30 years ago). II. THE START OF EMPIRICAL SOFTWARE ENGINEERING DURING THE BAD OLD DAYS In the last section, we talked about the dark ages when there were essentially no empirical studies, through the 1980 s when empirical studies began to be included in research papers, but only sporadically. Even then, they typically involved student subjects and small toy programs. One prominent exception to this was research pioneered by Victor Basili and Marv Zelkowitz, professors at the University of Maryland, along with their colleagues and students. It was performed in collaboration with practitioners at the Software Engineering Laboratory (SEL) at NASA Goddard Space Flight Center. Since this facility was nearby, it allowed for regular interaction between the researchers and the practitioners. Frank McGarry, a NASA employee, established and headed the Lab. It was his foresight that allowed the research done by the University of Maryland researchers to flourish. The SEL was formed in the late 1970 s to analyze the software development process and the software produced in order to understand the development process, the software product itself, the effect of various improvements on the process with respect to the methodology, and to develop quantitative measures that correlate well with intuitive notions of good software [3]. What the founders of the SEL understood was that too much Software Engineering research was based on anecdote or folklore rather than experience and observation. Their goal was to systematically observe how the application of research improvements actually affected the products produced by practitioners. This was done by observing software development projects in progress and measuring the effects of process changes. While they recognized that replicated controlled scientific experiments would be highly desirable, they also recognized that that was unrealistic because building multiple versions of complex systems was prohibitively expensive, and therefore they relied on case studies. Often Basili and Zelkowitz began studying a topic with students, but unlike most other academic researchers of that era whose assessment began and ended with student studies, when this team found something that looked promising in the university setting, they then tried to identify small NASA projects to try the process on with the eventual goal to transitioning the process into the standard NASA development process. Their research was highly influential but only slowly led others to recognize the value of these sorts of empirical studies. As an aside, the first industrial empirical study I was involved with [10] in the early 1980 s, was an extension of an empirical study performed by Basili and Weiss [4] in the SEL. Our study was done using data collected from a production industrial project at Univac. We had access to this system because my coauthor, Tom Ostrand, was then an employee of Univac, while I was a consultant there. An obvious question is why the work at the SEL didn t cause many other researchers to begin including large empirical studies in their papers? Why wasn t the Software Engineering literature truly revolutionized by their work, at least in the short term? As mentioned above, while Basili and Zelkowitz were discussing the need for systematic empirical studies done on real systems, and performing such studies, most other researchers were not emulating them and doing similar large-scale assessments of their proposed new Software Engineering techniques. Why not? The retrospective article, [3], provides insights into why their work did not set the standard for all Software Engineering papers during the bad old days, since as mentioned above, few papers during this time period contained any empirical evidence, and hardly any contained the sorts of large, systematic, careful studies advocated and performed in conjunction with the SEL. First, as they point out in this very interesting article, there were few publishable results during the first few years while they learned how to collect data, did the actual collection of initial data, refined the process, and learned what sorts of analysis was feasible and meaningful. This is exactly what we learned in our study described in [10] and many others we ve done during the past decade - doing large empirical studies are very time-consuming and often offer few publications during the early years until a sufficient corpus of data can be extracted and assessed. Many researchers who have not performed large industrial empirical studies often fail to recognize just how timeconsuming and difficult this very necessary part of the

4 process is, and resent that they are unable to freely access the data that other researchers have worked so hard to collect. Often there are issues of proprietary software which prevent the distribution of artifacts, and in some cases, as in telecommunications or medical data, there are legal and/or regulatory prohibitions against the distribution and access to such data. But the reality is that the owner of the data has generally spent significant time and effort doing the data collection, during which time they were unable to publish anything. Additionally, understanding what data to collect and how to do it, is in itself a contribution which is rarely recognized as valuable and therefore generally not publishable. Another important, and often overlooked factor is that data is often not meaningful in a vacuum. Without knowing the process or being able to speak with the people who created the artifacts from which the data is culled, it is often very difficult to understand the meaning of the data. For example, we found that even for systems using the same change management system within AT&T, different projects sometimes use certain fields in non-standard ways. Without being able to speak with project personnel to learn this, data could be easily misinterpreted, yielding misleading or incorrect results and conclusions. In my opinion, the biggest impediment to academic researchers doing large empirical studies has always been this lack of access to large production software systems to study and often an unfamiliarity with software development processes in industry. As Basili and Zelkowitz emphasized, there needs to be a partnership between researchers and practitioners for empirical Software Engineering research to succeed. Because of the availability of open source software, this is somewhat less of an issue today, but in the bad old days, this was an absolute requisite. This meant that during this period, in order to perform empirical studies using production software, researchers needed to find projects that were willing and able to participate in the studies. This was true whether the researchers worked in academia or in industrial or government research labs. Effectively this meant that researchers had to connect with development managers and convince them that is was worthwhile. But this was and still is generally a tall order. How are these contacts going to be made, and how can we convince these managers that they should run the risk of disrupting their schedules and potentially miss deadlines in order to use an unproven process, tool, or approach. Essentially it means that the researcher must convince the managers that the time and resources spent will eventually be repaid with an improved process, leading to higher quality software or software built using fewer resources. Without a lot of evidence, that is very difficult to do. Related to this is the fact that data collection is often a very expensive process, and frequently needs to be done by practitioners rather than researchers, so unless the development management can be convinced that it is a worthwhile expense, it is very unlikely that they will be willing to cooperate. But without their cooperation, doing an empirical study in a production environment is likely impossible. A second critical necessity is access to developers and close proximity to developers so that it is possible to have a symbiotic relationship and get regular feedback from developers, allowing the research process to be refined as needed. As with getting access to systems, getting access to personnel generally requires buy-in from high-level management. Again, this is ofter the show-stopper for academics as they are not known or trusted by management who may well view them as an impediment to successful maintenance of the schedule. In most cases, deadlines have to take precedence over all else, and therefore, even when there is an agreement from management to cooperate or partner with researchers, there will be times that a study must be put on hold in order for the system builders to focus entirely on their project without interruption. We have encountered that on a number of occasions when involved in research studies at AT&T. III. SUCCESS STORIES DURING THE BAD OLD DAYS While a small number of Software Engineering researchers were slowly trying to convince the rest of the community of the need to validate the usefulness of their ideas by doing serious empirical studies, there were a few research groups who were having a significant impact on practitioners who build, test and maintain large software systems. Although they were not formally doing empirical studies, I would argue a major reason why their work had such an impact on the world is the environment in which they worked which gave them access to groups which functioned as empirical studies. Additionally, I will also argue that these researchers were fundamentally different from most academic Software Engineering researchers. A. UNIX The first success story from that period that I will discuss, may not typically be thought of as a Software Engineering research success story, but I believe it should be. In any case, I think it sheds light on some of the issues our community has had to deal with, and continues to deal with, and ultimately helps us to understand how other research projects can have similar impact. I am thinking of the introduction of the UNIX operating system which came from the venerable AT&T Bell Labs Research Division. UNIX revolutionized operating system design in many ways and can reasonably be thought of as representing a new way of engineering operating system software, and as a result, had important implications for how software was engineered in general.

5 The over-arching goal of UNIX was to design an operating system that would allow many programmers to use a computer at the same time. A computer was an expensive and scarce resource at that time and UNIX allowed the sharing of this resource with each user feeling like they alone were using it. To help readers understand just how revolutionary this idea was at the time, I will mention my own experience of working as a professional industrial programmer which took place during the same timeframe that UNIX was being designed, built, and was first being introduced. I was working for a large multi-national corporation as a programmer. Although this was a huge corporation, using one of the most popular large mainfrome computers of the day, (personal computers were not yet invented) it was still only possible for a single job to be run at a time. Therefore, it was common for the turn-around time on a single run of a program to be multiple days. The programmer submitted a deck of punched cards which needed to be transmitted to the computer center at a different location, and waited up to three business days to get your output returned. This meant that the consequence of errors were very significant since it would be days till you got the next output back, and therefore very careful hand simulation of the code (so-called desk checking) was common in that era. My personal experience was that it was not unusual to get both clear compilations and bug-free runs on the first submission because so much effort was put into catching mistakes before the code was submitted to run. Presubmission efforts were intense and the norm. Another important characteristic was that UNIX was designed to be portable in the sense that it could run on many different machines or platforms. Most operating systems before UNIX, ran on one single type of platform. This meant that when a project needed to be ported to a new platform, it often required substantial effort to do so. In fact it was very common for companies to have to run emulators on their new machines because it was just too expensive to port all of their mission-critical software to the new platform. An emulator caused the new machine to behave as if it was the old machine. This was done because a company might have a huge amount of resources invested in mission-critical software for the old computer, and it was prohibitively expensive to rewrite it all in code that would run on the new machine. The solution was that for those programs written in the older machine s formats, it was necessary to be able to run them on the new machine and so the only alternative was to forgo many new features of the new machine and have it behave like the old machine. Anecdotal history tells us that twenty years after the last IBM 1401/1410 machine was available for purchase, and virtually none were still running, the most widely run programs were written in 1401/1410 Autocoder, being run on the IBM 360/370 family of machines in emulation mode. Autocoder was the 1401/1410 machines specific very low level (assembly) language. In fact, according to [1], in the year 2000, more than forty years after these machines were introduced by IBM, there were still businesses running IBM 1401 Autocoder programs. Since these machines had very small memories, there was the fear that there would be significant problems due to the so-called Year 2000 problem, and the fact that years were almost always stored as two digits on programs written for these machines in order to save space. Another novelty of UNIX was the recognition by its designers that many users needed certain standard types of applications and tools. A typical example of an application is a sort program. While sorts may have been well-understood, frequently implemented programs, it nonetheless took resources to design, implement and test them. Therefore, the UNIX designers reasoned, instead of every project writing their own sort routines, UNIX would provide a library of commonly-used routines like sorts, that any user could freely access. For most users, the UNIX libraries were a new, and very welcomed, feature of UNIX. UNIX also offered a suite of tools such as which was a moderately new idea, and much more sophisticated typesetting options than were ever available prior to UNIX. Many of the original UNIX tools and variants developed at Bell Labs, are still in widespread use. It may be true that ideas similar to those embodied in UNIX had been discussed in research papers earlier, but it was UNIX that took root and had incredible impact on the entire world of computing. What made UNIX different and successful where earlier similar projects never gained widespread traction? Basically, UNIX was designed and implemented by system builders for the use of system builders to solve concrete problems that were observed in the field. That is, it was designed to solve a problem that practitioners observed rather than solving a problem that practitioners did not realize they had, although it is possible that this latter situation might in fact be very helpful at times. Why did UNIX get so widely accepted and adopted? First and foremost, I believe, is because it was good and solved many real problems that developers had been complaining about for years. In addition, by the time UNIX was readily available outside of Bell Labs, there had been lots of real projects inside of Bell Labs that had successfully used it. As the old saying goes: the proof of the pudding is in the eating. Alternatively viewed, these were the empirical studies that convinced potential users that they should try it. Without the implementation by these experienced systems builders, and the validation of the usefulness by having real industrial projects using it, it is doubtful, in my opinion, that the underlying ideas would have gotten much traction.

6 One might reasonably argue that the simple fact that it is still in widespread use with many of the original underlying principles intact more than 30 years after it was introduced, means that these principles were, in some sense fundamental to its success. But the fact that there were many empirical studies done, i.e. production projects within Bell Labs that were built using UNIX, helped convince other projects to take the leap. And again the proximity of the researchers and the developers was critical to UNIX s eventual success, in my opinion. B. Programming Languages Another set of successes that impacted how software was engineered that came from research environments during that time period, were a series of new programming languages. Again these successes are not generally associated with the Software Engineering research community, but they do shed light on research projects that successfully transitioned from the research lab to the practitioner community, and they did impact the way that software was engineered. During the same timeframe that UNIX was being developed, (the early 1970 s), the C programming language was also being developed in the same laboratory at Bell Labs. Roughly a decade later, in the early 1980 s, C++ was introduced, again at Bell Labs, which extended C by adding object-oriented programming concepts. Typically these languages were of a higher level than those in common use prior to UNIX, meaning that each statement had a relatively great expressive power and so the programmer could do in one statement what it took many statements to do in a lower level language. Granted that ideas related to object orientation had been discussed in research papers earlier, and incorporated in languages such as Simula67, Smalltalk, and several that were developed within the Artificial Intelligence Research community at academic institutions such as MIT. While these papers and languages may well have had important influences on the development of C++, it wasn t until C++ was introduced and used in development projects at Bell Labs that object oriented programming became a central programming paradigm. More recent widely used object oriented languages include C# developed at Microsoft, and Java developed at Sun Microsystems. We again consider why C++ and these successors have had such an impact of the way software is engineered, while the languages developed in their research paper predecessors did not. The bottom line in my opinion are the same two factors that led UNIX to succeed where its predecessors did not have similar impact. Notice that all of these languages that gained widespread acceptance were designed and implemented by people working in industry, albeit often in industrial research labs. These were people who were involved with building large systems and had the opportunity to interact with practitioners and get feedback on the usefulness of their innovative ideas. Additionally, there was the possibility of doing large empirical studies by having production projects using their ideas. By partnering with these projects, they were able to gain assurance as to what would work and what would not. The sad reality is that one of the critical problems is that many Software Engineering researchers have never themselves actually engineered any real software systems. Certainly they may have written software as students, and perhaps built tools for their own and other researchers use, but my experience is that few Software Engineering researchers have ever worked as a software developer, tester, specifier, or maintainer for any sustained period of time and this has a very important impact on the usefulness and acceptance of their research. We will discuss the implications of this further in Section VII. IV. THE MIDDLE AGES Fast forward to the 1990 s. In 1994, Hutchins et al. wrote a paper [9] comparing dataflow-based and controlflow-based techniques for selecting test cases. To make the comparison, they did an empirical study that turned out to be very influential because of the study subjects. While the authors all worked at the Siemens Corporate Research Lab, and routinely were involved with production software projects, they nonetheless did not have access to these types of systems for experimentation. Therefore, they wrote a set of seven small C programs to experiment with, ranging in size from fewer than two hundred lines of code up to approximately seven hundred lines of code. None of these programs were ever used at Siemens for anything but experimentation, and were clearly very small. The authors created a series of versions of each program by seeding faults into each of the base programs. They had what they called a pool of test cases from which they sampled to create a test suite for the study. They defined a way of deciding whether too many or too few test cases detected the fault. If too many test cases detected the defect, then it was considered too easy and so was discarded from consideration. If too few test cases would detect the defect, it was too unlikely that one of the relevant test cases would be selected and so additional test cases were added to the pool so that it was more likely that one of them would be chosen. Perhaps because these programs were written by people working in industry, albeit in an industrial research lab, rather than being written by students or academic researchers, or perhaps because the authors generously distributed these programs freely, they became sort of a de facto benchmark set of programs for software testing experimentation.

7 While there is certainly something to be said for being able to compare the effectiveness of techniques by using a standard set of programs, since these were still a set of toy programs, written for experimentation rather than for actual usage, and the bugs in the different versions were seeded, they suffer from many of the same issues that earlier studies suffered from as discussed in Section I-B. As late as 2004, it was reported [6] that a majority of published software testing papers in the leading conferences and journals during the prior decade contained no empirical studies at all. Of those that did, the majority used this socalled Siemens Suite. Is it surprising that practitioners have not embraced the Software Engineering community s research when this is the best evidence of success we were able to produce? Even today, more than fifteen years after the suite was used in the study described in [9], it is still common to see this suite of programs used as the only empirical evidence to evaluate a new technique. In the next section we will consider more satisfactory recent empirical study solutions. As an aside, it is interesting to note that Ostrand, the only co-author of the study describe above that developed the Siemens suite to currently regularly publish Software Engineering research papers, has never again used that suite but has instead relied solely on large industrial software systems for experimentation. V. ARE THESE THE GOOD NEW DAYS? We now have a journal dedicated to Empirical Software Engineering, and at least one conference (this one) also dedicated to empiricism, and lots of other Software Engineering conferences and journals at which papers including empirical studies are welcome and published. That must mean that we re all converted and understand that it is no longer good enough to describe a new Software Engineering technique and assure everyone that because you think it is a great idea, and can even provide a plausible explanation about why it must be better than what others have been doing, it must be so. Right? Well not exactly. Certainly there are now many papers that contain first-rate empirical studies to help provide evidence that a technique works. By first-rate I mean ones performed carefully, with detailed information about the process used, and data collected or mined, and done using one or more production software system. Recall that in the bad old days, academics who represent, afterall, the majority of Software Engineering researchers, would complain they have no software to use for experiments or empirical studies, other than toy programs, and that is why they relied on either the Siemens or similar programs and students for subjects. However, now there is a world of open source software. Two of these systems that are among the most widely used for empirical studies include the Apache HTTP Server and the Eclipse software development platform. For example, Zimmermann et al. studied pre- and postrelease defects in the Eclipse bug database in [20] to be used in empirical studies on software fault prediction. Denaro and Pezze [5], collected data from one Apache version and used logistic regression to make predictions of fault-proneness for a later Apache version. These are only two of many Software Engineering research papers that have used these large open-source systems to do empirical studies to validate their ideas. Our research group, in contrast, has done a number of empirical studies involving large industrial software systems in which we considered a variety of questions and models that we used to predict which files of such a system were most likely to contain the largest number of bugs in the next release. Some of our papers include [12], [16], [17]. Similar research has been done by researchers at Microsoft Research, for example [11], [13], [21], and interesting studies have been done by researchers at Simula including [2]. Of course other groups working in other subfields of Software Engineering are also doing first-rate empirical studies to validate the effectiveness and usefulness of their research. Since both open source systems and proprietary industrial systems involve software that is widely used, large, longlived, and built by teams, it is both interesting and important to study both types of systems and perhaps eventually to decide whether there are characteristics that are common to both classes of systems, and what characteristics may hold for one class of systems but not the other. VI. HARD SELLS So what makes much of our research a hard sell? First and foremost, much of our research results are, rightly or wrongly, perceived as solutions in search of a problem. Why? I m, of course, not certain, but what I think is that we generally don t provide enough evidence or at least not enough compelling evidence. And, I think a big problem is that not enough researchers have any or substantial experience specifying, designing, building, testing, or maintaining large software systems and it shows. When I say real systems, I mean ones that are going to be in the field for multiple years, with many users. This typically means they are built and maintained by teams of people, often large teams that change with time and which may be both developed and deployed around the world. VII. CONCLUSIONS AND HOW TO MAKE THE FUTURE BETTER At a recent workshop on the future of Software Engineering research, Tom Ostrand and I discussed what we thought needed to change in order for our community s research to have a greater impact. [18].

8 In particular we pointed out that production software systems are getting bigger, and we increasingly need to address issues related to embedded systems, cloud computing, and systems of systems which are more and more the realities that practioners are dealing with. In the software testing community in particular, we need to focus more on nonfunctional qualities of software such as performance, risk, safety, ease of use, clarity, ease of modification as well as reliability and correctness. Often these areas are not considered central to Software Engineering, but rather are some other community s problems. Perhaps the biggest issue in my mind is the state of Software Engineering education. In particular, who teaches our college students about Software Engineering, and what are they being taught? When a student graduates from college as a newly educated Software Engineer, is there a standard body of knowledge we can expect that all will know? Will they all have some common practical experience with engineering software systems? In our paper at this workshop, we suggested that it would be very helpful for undergraduate Software Engineering students to have internships which are essentially apprenticeships, in which they learn by working with experienced professional engineers and get real hands-on training. This is common in many other engineering disciplines, although such programs frequently extend an undergraduate engineering degree from four to five years. In many fields, engineering graduates cannot legally call themselves an engineer without passing a licensing exam, and that often has a work experience requirement. It is true that a university is not a vocational school, but isn t there some standard body of knowledge and experience that all or our students should have? Shouldn t there be some lower bar for admission to the field as a professional? Can we imagine a student graduating with a degree in mathematics who hasn t studied calculus or algebra, for exampble, or a physics major who hasn t learned about electricity and magnetism? Another set of issues we raised in [18] involves the sorts of training a Software Engineering educator needs. As mantioned above when discussing Software Engineering researchers, we pointed out that perhaps one reason that our research is often not deemed relevant is that our researchers often have no real experience engineering software systems. The same is often true for Software Engineering educators, who are, of course, often also researchers. Software Engineering courses are frequently taught by professors who have a Ph.D. in Computer Science, but lack real Software Engineering experience. Therefore, students are being taught about how professors imagine people engineer software, and what they believe the significant problems are, or what they have learned by reading papers written by other researchers who have no first-hand experience. How can we rectify this situation so that all Software Engineering research is first rate and relevant? First and foremost, I think, we have to improve the education of our Software Engineering students and faculty. I ve mentioned a way to improve it for students by making internships an integral part of their eduction. But it also requires that their instructors be able to teach more relevant material. A rather radical solution is for funding agencies to offer summer or even year-long fellowships for Software Engineering faculty to work at industrial software development organizations. In this instance I am not including industrial research organizations - that seldom involves really learning how practitioners specify, design, build or test software, since in many industry labs, researchers are almost as far removed from practitioners as academics are. Faculty have to learn by doing. The companies will probably gain very little immediate, concrete benefit from hosting Software Engineering faculty members at their organizations. That is why I suggest that funding agencies should underwrite faculty expenses. Reflecting on the early real success stories I mentioned: Basili and Zelkowitz s work, UNIX, C, Java, C++, etc, when we look at what all of these projects have in common, I think it becomes clear. They either came from industrial research labs done in collaboration with practitioners or, in the case of Basili and Zelkowitz s work, it was done by academic researchers in conjunction with NASA, an organizetion that produced production software. I do not believe this was a coincidence at all. I think these major successes happened because of the empirical evidence the researchers were able to provide by having practitioners use their ideas in practice and provide feedback on their ideas, and by researchers working side by side with practitioners. We still need significant empirical evidence to be part of every Software Engineering research project. We need to better understand the extent to which conclusions we can draw about open source and proprietary software systems are the same and different. Do different development paradigms lead to different results in terms of reliability, performance, ease of use and other important functional and non-functional characteristics? I think it is clear that the state of Software Engineering research has progressed by significant leaps during the last few decades, but there is still room to grow and improve. We are definitely out of the ugly, and mostly out of the bad, but we are not universally into the good, and we need to find ways to move into the excellent. REFERENCES [1] [2] E. Arisholm, L. Briand, E. Johannessen. A Systematic and Comprehensive Investigation of Methods to Build and Evaluate Fault Prediction Models. Journal Systems & Software, Vol 83, No 1, 2010.

9 [3] V.R. Basili, F.M. McGarry, R. Pajerski and M.V. Zelkowitz. Lessons Learned From 25 Years of Process Improvement: The Rise and Fall of the NASA Software Engineering Laboratory. Proc. Int Conf Software Engineering (ICSE02), Orlando FL, May [4] V.R. Basili and D.M. Weiss. Analyzing Software Data in the Software Engineering Laboratory. Fourth Minnowbrook Workshop on Software Performance Evaluation, Blue Mountain Lake, NY, Aug [5] G. Denaro and M. Pezze. An Empirical Evaluation of Fault- Proneness Models. Proc. International Conf on Software Engineering (ICSE2002), Miami, USA, May [6] H. Do, G. Rothermel, and S. Elbaum. Infrastructure Support for Controlled Experimentation with Testing and Regression Testing Techniques. Technical Report , Oregon State University, Jan [7] P. Frankl and E.J. Weyuker. Comparing Fault Detecting Ability of Testing Methods Proc. of ACM SIGSOFT 91, Conference on Software for Critical Systems, New Orleans, La., Dec 1991, pp [17] E.J. Weyuker, T.J. Ostrand, and R.M. Bell. Comparing the Effectiveness of Several Modeling Methods for Fault Prediction. Empirical Software Eng. Journal, June [18] E.J. Weyuker and T.J. Ostrand. Software Testing Research and Software Engineering Education. Workshop on the Future of Software Engineering and Research Santa Fe, NM, November [19] E.J. Weyuker, S. Weiss and D. Hamlet. Comparison of Program Testing Strategies. Proc. of the Fourth ACM SIGSOFT Symposium on Software Testing, Analysis and Verification (TAV4), Victoria, B.C., Canada, Oct 1991, pp [20] T. Zimmermann, R. Premraj, and A. Zeller. Predicting Defects for Eclipse. Proc. Third International Workshop on Predictor Models in Software Engineering (PROMISE 07). [21] T. Zimmermann and N. Nagappan. Predicting Defects Using Network Analysis on Dependency Graphs. Proc. 13th Int. Conference on Software Engineering (ICSE 08), p , [8] P. Frankl and E.J. Weyuker. A Formal Analysis of the Fault-detecting Ability of Testing Methods. IEEE Trans. on Software Engineering, Vol. 19, No. 3, March 1993, pp [9] M. Hutchings, H. Foster T. Goradia, and T. Ostrand. Experiments on the Effectiveness of Dataflow- and Controlflow- Based Test Data Adequacy Criteria. Proc. Int Conf Software Engineering (ICSE), pp , [10] T.J. Ostrand and E.J. Weyuker. Collecting and Categorizing Software Error Data in an Industrial Environment, J. Systems and Software, Vol.4, 1984, pp [11] N. Nagappan, T. Ball, and A. Zeller. Mining Metrics to Predict Component Failures. Proc 28th Int. Conference on Software Engineering (ICSE06), pp , [12] T.J. Ostrand, E.J. Weyuker, and R.M. Bell. Predicting the Location and Number of Faults in Large Software Systems. IEEE Trans. on Software Engineering, Vol 31, No 4, April [13] M. Pinzger, N. Nagappan, and B. Murphy. Can Developermodule Networks Predict Failures? Proc SIGSOFT FSE 2008, pp. 2-12, [14] S. Rapps and E.J. Weyuker. Data Flow Analysis Techniques For Test Data Selection. Proc. of the 6th IEEE/ACM International Conference on Software Engineering (ICSE), Tokyo, Japan, September 1982, pp [15] S. Rapps and E.J. Weyuker. Selecting Software Test Data Using Data Flow Information. IEEE Trans. on Software Engineering, Vol.SE-11, Apr 1985, pp [16] E.J. Weyuker, T.J. Ostrand, and R.M. Bell. Do Too Many Cooks Spoil the Broth? Using the Number of Developers to Enhance Defect Prediction Models. Empirical Software Eng. Journal, October 2008.