Refactoring Economics: An Empirical Study of Refactoring Effect on Development Costs

Transcription

1 Refactoring Economics: An Empirical Study of Refactoring Effect on Development Costs Quinten David Soetens, Serge Demeyer University of Antwerp, Belgium Abstract We will study the economical aspect of refactoring by investigating what effect refactoring has on the costs of adding new functionality to a system and by looking at the costs of the refactoring activity itself. We expect that the costs of adding functionalities will be lower after the system was refactored. Indeed it is generally accepted that refactoring has a positive influence on the maintainability of a system. However this is purely based on anecdotal evidence and still needs to be reinforced by a thorough scientific and systematic investigation. In our research we will be conducting a series of case studies on some real-life systems. On these cases we will determine where and when they were refactored and what influence these refactorings had on the development costs of the systems. Keywords: refactoring, economics, refactoring detection, software evolution, mining software repositories 1. Introduction The lion s share of any software engineering project is the so called maintenance. Indeed the rule states that 20% of the time is spent on the development of a system and 80% of the time on maintenance [20]. Moreover, the largest part of this maintenance (again 80%) is not spent on fixing bugs, but on other maintenance activities such as adding new functionalities [1]. However, as Lehman s laws teach us, adding new functionalities will become more and more difficult as the system evolves, since a system will gradually become more and more complex ( Law of Increasing Complexity ) [14]. Nowadays refactoring is generally accepted as part of the solution for the long term maintainability of a software system (cf. the Software Engineering Body of Knowledge [1]). Refactoring is defined as Transformations of the source code, without changing the observable behavior, to make the source code easier to understand and reuse [19]. (i.e., refactoring lowers the complexity of the code) It is generally agreed upon that refactoring improves the maintainability of a software system, however this is based purely on anecdotal evidence and has yet to be empirically validated [16]. We will therefore study the refactoring economics in a software development process. We ask ourselves the following questions: a. What are the costs of the individual refactoring steps? b. What effect does refactoring have on the maintenance costs? c. When will refactoring pay off its investment? Small and Medium Enterprises (SME s) will gain advantages from this research, as they will be better able to keep an eye on the costs and benefits of software maintenance which will give them a competitive advantage on much larger and more cumbersome companies. This research will also increase our knowledge on the costs and benefits of refactoring, which will give us a clearer image on when, where and how we need to apply refactorings. If we can show with this project that refactorings pay off their investment, then this can be used to convince software companies who do not refactor that refactoring is a useful and cost saving activity in the development of their software. Which addresses: quinten.soetens@ua.ac.be (Quinten David Soetens), serge.demeyer@ua.ac.be (Serge Demeyer) Preprint submitted to BENEVOL Proceedings of Short Papers December 3, 2009

2 will allow them to spend more resources on more innovating applications. If on the other hand we find that refactoring does not pay off its investment, then management will be able to do a better cost benefit analysis by taking the loss caused by refactoring into account when deciding if they will refactor. 2. Goals We have set ourselves two goals to examine the economical aspect of refactoring in a software development process.: to empirically validate whether refactoring has a positive effect on the maintainability of a software system. (i.e., Will the manpower needed to add new functionalities decrease?) to examine if and when refactoring will pay off its investment. The first goal can be achieved by showing that the cost to add new features after the system was refactored ($ After ) is substantially lower than the cost of adding new features to the system before it was refactored ($ Before ). In short, we will show that: $ After $ Before For the second goal we will need to look at the cost of the refactoring activities themselves ($ Refactoring ). We will know that refactoring pays back its investment if we can show that: $ Refactoring + $ AddingFeatures $ AddingFeatures with $ Refactoring + $ AddingFeatures the cost to add a set of new features to a system after it was refactored and $ AddingFeatures the cost of adding that same set of features to the system without first refactoring the system. To realize aforementioned goals we will need to lay some groundwork. We need to ascertain how we can determine where and when the software was refactored. In the past 10 years there has been a lot of research into techniques to do refactoring detection. However this was always done independently of one another. That is why we start our work with a comparative study of existing refactoring detection techniques, to determine which technique is most suitable for our purposes. 3. Related Work Some work has already been done towards understanding the economics of refactoring. Leitch and Stroulia for instance have proposed a framework to predict the return on investment for a refactoring using cost-benefit analysis [15]. Sullivan et al. have shown that options thinking can be used to value software restructurings [23]. Bahsoon and Emmerich use the flexibility of a system as a way to valuate the payoff of refactorings [3]. There have been many studies on the effect refactoring has on a Software System. Du Bois for instance has studied the effect on the comprehensibility and understandability of a system [9] [8]. He also studied the effect refactoring have on the coupling and cohesion of a software system [7]. Other noteworthy work is that of Geppert et al., who studied the effect of refactoring on changeability [10], that of Moser et al., who investigated the effect on reusability [18], that of Kataoka et al., who did some research involving the effect of refactoring on the maintainability of a system [12] and that of Higo et al., who analyzed the effect of refactoring on some complexity metrics [11]. Some more general studies on the effect of refactoring on software quality include the ones by Alshayeb, Wilking et al, Stroggylos and Spinellis and Moser et al. [2] [27] [22] [17] 4. Proposed Solution This research is split into two parts. The first one is the preparatory work and mainly consists of a comparative study of algorithms that search refactorings in the history of a system (i.e., refactoring detection techniques). The result of this study can then be used in the second phase of the project, which consists of a series of case studies on refactoring economics. 2

3 We will do our measurements on a few different cases (both open source and industrial projects). These cases need to be software systems where refactoring was actually applied. We will primarily be looking at projects written in Java, since this is a very popular programming language with a very good support for automated refactoring (JRefactory, Eclipse ID, Netbeans IDE,...). Later on we will also be looking at C and C++ programs to see if our conclusions can be made language independent. Our experiments will be done in a few iterations, increasing the difficulty with each iteration. This way we will be able to identify possible problems in the used techniques and tools early in the project. 1. In the first iteration we will perform our experiments and measurements on small to medium sized projects (i.e. projects ranging from 10 to 100 KLOC), that are written in Java and have known refactoring periods. 2. In the second iteration we will look at systems of the same size as in the first iteration, but this time we will look at projects written in C, C++ or C# and with unknown as well as known refactoring periods. 3. In the final iteration we will look at larger systems (i.e., projects ranging from 100 to 1000 KLOC). To insure the quality of our research we intend to follow the guidelines for empirical research set by Kitchenham et al. as well as the guidelines on conducting and reporting case studies set by Runeson and Höst [13] [21] Refactoring Detection Several refactoring detection techniques that have lead to good results have already been developed. Weißgerber and Diehl for instance presented a technique that uses signature-based analysis to identify possible refactorings and clone detection to rank the likelihood of it being a refactoring [26]. Danny Dig et al. used a combination of syntactical and semantical analysis to identify refactorings [4][5][24]. Demeyer et al. have developed a set of heuristics based on a combination of change metrics to identify refactorings. Van Rysselberghe and Demeyer have developed a way that uses clone detection [25]. And Xing and Stroulia developed the UMLDiff algorithm, which is capable of detecting some basic structural changes in a system [28]. We will compare these refactoring detection techniques based on a few criteria, that we have subdivided into primary and secondary criteria. Primary Criteria Precision is a criterion that measures to which extent the techniques are correct, in other words: Are the detected refactorings truly refactorings? Recall is a criterion that measures to which extent the techniques are complete, in other words: How many of the actual refactorings were found? Performance is a criterion that measures the speed with which the techniques detect refactorings. Secondary Criteria Scalability is a criterion that measures to which extent the technique is applicable on larger projects. Scalability and performance are important criteria to us, since we also want to detect refactorings an medium sized to large projects and projects with a long history. Scope: This means that we will determine whether the algorithm is language independent or specific and we will also look at which refactorings it is capable of finding Cost of a new functionality With this experiment we will empirically study the effect refactoring has on the costs of adding functionality to a system. To do this we shall determine what the costs are of adding a new feature before and after a refactoring period. The experiment will be performed on a few cases and consists of a few steps: 1. Using some refactoring detection technique, we can analyze the case s version control system to find a version that has clearly just been preceded by a refactoring period. (Call this version V R ) 3

4 2. We analyze the versions of the system before V R to find features that were added before V R, but after a possible previous refactoring period. For each of these features we determine the cost. This way we can determine the average cost of a new feature per version. 3. We do the same for the versions after V R, but before a possible subsequent refactoring period. 4. Then we can project how the average cost of adding a new feature changes over time (i.e., how this average cost changes with each version of the system) In this last step (step 4) we expect to see results like the ones presented in the sketch in figure 1. We expect to see the average cost of new features increase before the refactoring period. The average cost of new features in the versions following the refactoring period will be significantly lower than in the versions preceding the refactoring period. However the cost of new features will again increase over time. Figure 1: Cost of implementing a new feature should increase with time. After a refactoring period the cost should decrease PayOff of Refactoring In this part of the research we will address the following research questions: Will refactoring eventually pay off its investment? When does this payoff become apparent? e.g., How many new features need to be added before there is a profit? In this experiment we will look at a series of features that were implemented after the system was refactored and we will compare their cost to what they would have cost if the system had not been refactored. Since the latter cost can not be measured we will need to use a cost estimation model. The cost of the features as is, can be measured from the implementation, however in order to have comparable data it is best to also use the cost estimation model to determine this cost. The cost estimation model that we use, will be the one(s) that we concluded from the preparatory work to be the best for our purposes. To answer our first research question we need to measure (or estimate) the cost of the refactoring itself (call this $ Refactoring ). We also need to estimate what the costs are of adding a series of new features after the refactoring (call this $ AddingFeatures ). And finally we need to estimate what the costs are of adding this same series of features to the system before the refactoring (call this $ AddingFeatures ). The refactoring will then produce a profit only if $ Refactoring + $ AddingFeatures $ AddingFeatures. If we find that the refactoring produces a profit, we can project how $ Refactoring + $ AddingFeatures changes by increasing $ AddingFeatures with one feature at a time. This way we can find at which point in time (after how many features) the refactoring starts to return on its investment. 4

5 5. Summary In short we propose to investigate the economical aspect of refactorings by a series of case studies on some real-life systems. We will study the costs of refactoring and the influence refactoring has on the costs of adding functionality to an existing system. We expect to see that the cost of adding functionality is lower after the system has been refactored. After that we can also determine whether the refactoring activity is a profitable one. [1] Abran, A., Bourque, P., Dupuis, R., Moore, J. W., Tripp, L. L., Guide to the Software Engineering Body of Knowledge - SWEBOK, 2004th Edition. IEEE Press, Piscataway, NJ, USA. [2] Alshayeb, M., Empirical investigation of refactoring effect on software quality. Inf. Softw. Technol. 51 (9), [3] Bahsoon, R., Emmerich, W., Applying archoptions to value the payoff of refactoring. In: Proceedings of the the 6th International Workshop on Economic-Driven Software Engineering Research (EDSER-6). [4] Dig, D., Comertoglu, C., Marinov, D., Johnson, R., July Automated detection of refactorings for libraries and frameworks. In: Proceedings of Workshop of OO Reengineering (WOOR 05). [5] Dig, D., Comertoglu, C., Marinov, D., Johnson, R., Automated detection of refactorings in evolving components. In: Proceedings of European Conference on OO Programming (ECOOP 06). pp [6] Du Bois, B., September A study on quality improvements by refactoring. Ph.D. thesis, University of Antwerp, Middelheimlaan 1, B-2020 Antwerpen, Belgium. [7] Du Bois, B., Demeyer, S., Verelst, J., Refactoring improving coupling and cohesion of existing code. In: WCRE 04: Proceedings of the 11th Working Conference on Reverse Engineering. IEEE Computer Society, Washington, DC, USA, pp [8] Du Bois, B., Demeyer, S., Verelst, J., Does the refactor to understand reverse engineering pattern improve program comprehension? In: CSMR 05: Proceedings of the Ninth European Conference on Software Maintenance and Reengineering. IEEE Computer Society, Washington, DC, USA, pp [9] Du Bois, B., Demeyer, S., Verelst, J., Mens, T., Temmerman, M., Does god class decomposition affect comprehensibility? In: Proceedings of the IASTED International Conference on Software Engineering. IASTED/ACTA Press, pp [10] Geppert, B., Mockus, A., Rossler, F., Refactoring for changeability: A way to go? In: METRICS 05: Proceedings of the 11th IEEE International Software Metrics Symposium. IEEE Computer Society, Washington, DC, USA, p. 13. [11] Higo, Y., Matsumoto, Y., Kusumoto, S., Inoue, K., Refactoring effect estimation based on complexity metrics. In: ASWEC 08: Proceedings of the 19th Australian Conference on Software Engineering. IEEE Computer Society, Washington, DC, USA, pp [12] Kataoka, Y., Imai, T., Andou, H., Fukaya, T., A quantitative evaluation of maintainability enhancement by refactoring. In: ICSM 02: Proceedings of the International Conference on Software Maintenance (ICSM 02). IEEE Computer Society, Washington, DC, USA, p [13] Kitchenham, B. A., Pfleeger, S. L., Pickard, L. M., Jones, P. W., Hoaglin, D. C., Emam, K. E., Rosenberg, J., August Preliminary guidelines for empirical research in software engineering. IEEE Trans. Softw. Eng. 28 (8), [14] Lehman, M. M., Belady, L. A., Program evolution : processes of software change / edited by M.M. Lehman, L.A. Belady. Academic Press, London ; Orlando :. [15] Leitch, R., Stroulia, E., Understanding the economics of refactoring. In: In: Proceedings of the Fifth ICSE Workshop on Economics- Driven Software Engineering Research. [16] Mens, T., Demeyer, S., Software Evolution. Springer Publishing Company, Incorporated. [17] Moser, R., Abrahamsson, P., Pedrycz, W., Sillitti, A., Succi, G., A case study on the impact of refactoring on quality and productivity in an agile team, [18] Moser, R., Sillitti, A., Abrahamsson, P., Succi, G., Does Refactoring Improve Reusability? Vol. Volume 4039/2006. Springer-Verlag, pp [19] Opdyke, W. F., Johnson, R. E., September Refactoring: An aid in designing application frameworks and evolving object-oriented systems. In: SOOPPA 90: Proceedings of Symposium on Object- Oriented Programming Emphasizing Practical Applications. [20] Pfleeger, S. L., Atlee, J. M., Software Engineering Theory and Practice. Prentice Hall. [21] Runeson, P., Höst, M., April Guidelines for conducting and reporting case study research in software engineering. Empirical Software Engineering 14 (2), [22] Stroggylos, K., Spinellis, D., Refactoring does it improve software quality? In: WoSQ 07: Proceedings of the 5th International Workshop on Software Quality. IEEE Computer Society, Washington, DC, USA, p. 10. [23] Sullivan, K. J., Chalasani, P., Jha, S., Sazawal, V., Software design as an investment activity: A real options perspective. In: Real Options and Business Strategy: Applications to Decision Making. Risk Books. [24] Taneja, K., Dig, D., Xie, T., November Automated detection of API refactorings in libraries. In: Proc. 22nd IEEE/ACM International Conference on Automated Software Engineering (ASE 2007). pp [25] Van Rysselberghe, F., Demeyer, S., September Reconstruction of successful software evolution using clone detection. In: Tommi Mikkonen, M. W. G., Saeki, M. (Eds.), Proceedings IWPSE 03 (International Workshop on principles of software evolution). IEEE Computer Society, pp [26] Weißgerber, P., Diehl, S., Identifying refactorings from source-code changes. In: ASE 06: Proceedings of the 21st IEEE/ACM International Conference on Automated Software Engineering. IEEE Computer Society, Washington, DC, USA, pp [27] Wilking, D., Khan, U. F., Kowalewski, S., An empirical evaluation of refactoring. e-informatica Software Engineering Journal 1 (1), [28] Xing, Z., Stroulia, E., Refactoring detection based on umldiff change-facts queries. In: WCRE 06: Proceedings of the 13th Working Conference on Reverse Engineering. IEEE Computer Society, Washington, DC, USA, pp