Research on Formal Degree Evaluation and Analysis for Domain Software Based on Evidence

Transcription

1 Chinese Journal of Electronics Vol.25, No.5, Sept Research on Formal Degree Evaluation and Analysis for Domain Software Based on Evidence BAO Tie, LIU Shufen and HAN Lu (College of Computer Science and Technology, Jilin University, Changchun , China) Abstract Existing researches fail to involve formalized methods in evaluation and analysis of domain software and lack analysis on formal degree, this paper comes up with a formal degree evaluation approach for domain software based on evidence. Various levels of transformation models are mapped by formal analysis of evidence in life cycle of domain software so as to quantitatively measure degree of evidence. Evaluation model based on evidence is established by analyzing detailed evaluation requirement. A level model including mapping condition is established to describe formal degree at hierarchical level. This paper explains detailed evaluation process through an evaluation example. The approach stated in this paper can describe formal degree of domain software, evaluation data can support subsequent bottleneck analysis and trustworthy evolution, thus provide formal support for creditable construction and analysis. Key words Software evaluation, Domain software, Formal degree, Evidence metrics. I. Introduction Development of social informatization is increasing people s dependence for software day by day, thus software are also widely used in various industries, especially for those applied in control system of production equipment or management system of core businesses as they are playing a vital role in many enterprises. At present, an increasing demand for domain software may also lead to increasing scale and complicated structure of domain software day by day. Hence, creditable construction and reliability service become more and more difficult to be guaranteed. In case of a failure of core software operated in key application, this will bring the user great losses. In view of importance of domain software, trustworthy domain software construction and guarantee of its operating quality have been highly-focused by researchers in related fields. Therefore, the studies in related technologies, aiming at quality evaluation of domain software and formal analysis, are under way. Currently, relevant research covers two major parts: 1) evaluation and analysis of domain software, such as those of specific development properties (e.g., quality evaluation of ERP open source software) and those of specific application (e.g., modeling analysis of service-oriented software), and research on specific issues in evaluation of domain software; and 2) formalized methods are used to analyze specific issues in the software s lifecycle, such as analyzing strategy for test on software s effectiveness, improvement of software s resilience capability, and formalized computation of the system reliability. Since existing researches fail to involve formalized methods in evaluation and analysis of domain software and lack analysis on formal degree, this paper engages formalized tools and proposes a method to evaluate and analyze the formal degree of domain software. The paper concerns primarily with evaluation and analysis of formal degree of domain software, the major work involves in: carrying out formalized analysis to the evidence to generate quantitative measurement, carrying out analysis to evaluation demand and application demand to construct evaluation model based on the evidence, conducting analysis on the level definition and mapping condition to formal degree to construct levle model based on evaluation model, illustrating how to analyze bottleneck problems based on the evaluation data and illustrating the evaluation and analysis process of the formal degree of the domain software with an example. The first major contribution of the paper is to provide a systematic approach for evaluation of formal degree of the domain software, since it generates evaluation results for measuring formal degree of the domain software by formalized analysis to the evidence and measure- Manuscript Received Oct. 21, 2015; Accepted Feb. 5, This work is supported by the National Natural Science Foundation of China (No ), the National Key Technology Research and Development Program of China (No.2014BAH29F03), and the Jilin Province Science and Technology Development Program (No GX). c 2016 Chinese Institute of Electronics. DOI: /cje

2 794 Chinese Journal of Electronics 2016 ment, construction of evaluation model and level model as well as analysis to bottleneck problems based on evaluation data and further analysis can be carried out based on the evaluation result. The second major contribution is to introduce formalized tool to the evaluation and analysis process for the domain software, which makes the evaluation and analysis results more credible. The third major contribution of the paper is to illustrate how to evaluate and analyze the formal degree of the actual domain software with the approach in the paper. The remainder of the paper will be structurally organized as follows. Section II analyzes the existing evaluation and analysis approaches for domain software as well as formalized tools to the specific problems for the software. Section III illustrates the formalized analysis and measurement for evidence of the domain software. Section IV illustrates the process of constructing the evaluation model and level model. Section V illustrates how to apply the approach to carry out evaluation and analysis on the formal degree of the domain software. Section VI is the conclusion of the paper. II. Related Works For one thing, existing researches mainly concentrate on evaluation and analysis of domain software. Aversano et al. [1] proposed a framework for evaluating the quality and functionality of open source software systems. The customized framework was applied to the evaluation and comparison of five ERP open source software systems. Zhang et al. [2] proposed a systematic approach of modeling quality attributes in feature models based on domain experts judgments using analytic hierarchy process and conducting quality aware product configuration based on the captured quality knowledge. A Goal-driven software development risk management model (GSRM) was presented, and its explicit integration into the requirements engineering phase and an empirical investigation result of applying GSRM into a project were introduced [3].In Ref.[4], a mixed abstraction level modeling methodology was presented for the performance evaluation of NoC architecture. Then based on the methodology, they develop a full system mixed-level NoC evaluation and verification platform. Riccobene and Scandurra [5] introduced a framework for modeling and prototyping service-oriented applications. As proof of concept, a case study taken from EU research projects has been considered to show the functionalities and potentialities of the proposed framework. Bao et al. [6] proposed a trustworthiness evaluation method based on actual evidence which can be used to evaluate trustworthiness level of software in practical application. Based on rules, an evidence-driven framework for trustworthiness evaluation of software was proposed in Ref.[7]. Though such work researches domain software of specific properties, domain software used for specific application, and specific issues in evaluation of domain software, credibility of the evaluation results is reduced to some degree as no formalized methods are adopted. For another, existing researches concentrate on analyzing specific issues in the software s lifecycle with formalized methods. Zhang et al. [8] focused on reliability evaluation using hybrid methods. Based on the proposed model, the formal calculation of the system s reliability is given. In Ref.[9], Li et al. showed how the natural causeand-effect structures that can be found in non-formal requirements descriptions can be used systematically to arrive at a software specification. Lochau et al. [10] presented a combination of architecture-driven model-based testing principles and regression-inspired testing strategies for efficient, yet comprehensive variability-aware conformance testing of variant-rich systems. The contracts about exception handling and a way were set up to assess them automatically in Ref.[11], improved the resilience capabilities of software by transforming the source code. For this part, it primarily analyzes specific issues in software design, development, test and maintenance with formalized method. Nevertheless, it lacks full understanding of the software s features or comprehensive analysis method for formal degree of the domain software. In conclusion, as no formalized methods are currently adopted in research of domain software, credibility of the evaluation and analysis is thus affected. Though formalized tools are used for research on specific issues of some software, methods that can comprehensively analyze and measure the formal degree of software are still needed. And existing research on description and analysis methods for formal degree of domain software is insufficient. It is very difficult to provide strong formal support for credible construction and reliability analysis. Aiming at the above problems, this paper comes up with the formal degree evaluation method for domain software based on evidence, analyze the formal transformation process of software evidence, and establish the evaluation and level models, thus providing trustworthy support for formal analysis, construction and evolution processes. III. Formal Analysis for Evidence of Domain Software This paper makes a formal degree analysis for domain software based on evidence, formal analysis for evidence is also the basis for evaluation. First, it makes an overall explanation for formal analysis for evidence of domain software, as shown in Fig.1, the evidence is from software artifacts at each phase of life cycle of domain software, including domain norms, domain knowledge, industry rules,

3 Research on Formal Degree Evaluation and Analysis for Domain Software Based on Evidence 795 rules and regulations of enterprises, documents related to requirement analysis and architecture design, software testing report, user application feedback and other types of objective and subjective artifacts. Software artifacts are divided into three types of evidence: first, function evidence, it put forward all types of functions that shall be supported by software; second, restriction evidence, including restrictive conditions for software s life cycle during special phase; third, measure evidence, mainly including system testing report, function verification report, user evaluation report, etc. The first type of evidence includes all types of functions realized by software, which is also the most important part during domain software development, and function evidence can be transformed to multiple formal models according to sequence of software s life cycle by formal tools, as shown in Fig.1, the order from left to right: description transformation, featurebased transformation, model-based transformation, implementation transformation and result transformation. The second type of evidence can be transformed to the restrictive conditions for transforming of function evidence at special phase and formal tools can be used to verify if the conditions were matching with each other. The third type of evidence mainly includes some test and assessment reports related to domain software which can be transformed to results transformation. These three evidences can be treated with various modes, and require some automatic analysis transformation tools and participation of domain experts, software engineers and users during the treatment. Fig. 1. Formal analysis framework for evidence of domain software In this paper, the on-duty management module of production management software in power plant of power domain is taken as an example. Raise specification language (RSL) is used as the tool for formalized analysis. Major functions of the software are developed with Java. Module in the evaluation instance is also developed with Java. Based on RSL and Java, the formal analysis processes of three types of evidence for domain software are explained. As shown in Fig.2, three rectangles formed by solid lines located at upper left corner of the figure include three types of evidences used in this example respectively, due to space limitations, this paper selected part of them to explain. The first rectangle formed by solid lines on upper left corner includes three function evidences: the first one is knowledge of electric power industry, describing mode of troubleshooting by corporate on-duty personnel. The second one is about corporate on-duty rules, explaining all items of detailed on-duty provisions. The third one is a requirement analysis file about production management software in power plants, including function analysis for on-duty management module. The second rectangle at upper left corner includes three restriction evidences, and specifies fault record book shall perform cleanup and support multiple users parallel operations and also specifies Java language applied in on-duty management module. In fact, main parts of production management software in power plant have been implemented in Java language. The third rectangle at upper left corner includes two measure evidences, one is the feedback on on-duty management function made by the user, and the other is the test report for implementation code of such function. Test report can be generated by using testing tools to test the implementation code. The rectangle formed by imaginary lines from up to bottom at the right corner in Fig.2 covers evidence transformation process, in which an oval formed by imaginary lines presents the mapping relation between entities at two ends of the connecting line, and the entity at the head is transformed to the one at the end through applicable tools. The rhombus presents the mapping relation between the evidence and transformation at two ends of the connecting line. The evidence at the head is the restrictive condition for transformation at the end. There are three description transformations in the rectangle formed by imaginary lines on the top of Fig.2,

4 796 Chinese Journal of Electronics 2016 Fig. 2. Analysis process for evidence of software in the evaluation example which are from three function evidences, describing function requirement for on-duty management module, mainly including: 3 5 persons are arranged in on-duty group; on-duty personnel are going on a tour of inspection; in case of any failure of an equipment, these faults will be recorded on the fault record book and reported to the upper level, and the fault record book can support regular fault management function. The above second rectangle formed by imaginary lines includes two featurebased transformations, describing main particulars and functions of on-duty management module, and they are transformed from description transformations of the upper level. Corresponding relations are marked by using connecting line with arrows, two description transformations of which correspond to feature-based transformation at this level. The above third rectangle formed by imaginary lines includes two model-based transformations, describing mathematical model to realize on-duty management module, such as pipe-type model is used to support functional parallelism; basic data structure is used in the record book. The rectangle formed by imaginary lines at the right of the bottom includes two implementation transformations, Java language is used to realize the corresponding transformation at the above level. The rectangle formed by imaginary lines at the left of the bottom includes one result transformation, which is transformed by two measure evidences in the example, such transformation is mainly used to transform evaluation of function module and test report into standard formats. Three restriction evidences in the example are the restrictions to relevant transformations, and the corresponding relations are also marked by using connecting lines with arrows. IV. The Construction of Evaluation Model and Level Model The evidences of domain software are divided into three types, formal analysis is conducted for such evidences by different approaches, formal degree evaluation model and level model for domain software may be established based on the evidence after formal analysis, and then the detailed evaluation work can be conducted. As shown in Fig.3, it first illustrates how to conduct the

5 Research on Formal Degree Evaluation and Analysis for Domain Software Based on Evidence 797 quantitative analysis for the evidence, and then illustrates how to set up evaluation model and level model. Different types of evidences need quantitative analysis adopting different approaches; the formal degree evaluation for domain software will proceed based on these quantitative data. The function evidence at most corresponds to five levels of formal transformation (maybe less than five levels of transformation for some evidence, such as the evidence in the example of the above section), the formal level can reflect the reliability in the design, implementation and application process of function requirement included in the evidences, and generate scores based on the formal levels. The measure evidence can generate standard report through transformation, and generate scores based on the evaluation report of standard format. The restriction evidence is the restrictive conditions for each level transformation, it has to be judged through the satisfaction status of the restricted conditions, and scores may be generated based on the judgment. The generation of scores needs not only the analysis result based on formal tool but also the participation of domain experts, software engineers and users to some degrees. To guarantee objectivity of the scoring process, Analytic hierarchy process (AHP) or other similar methods [12] can be adopted to guarantee effectiveness of the quantitative evaluation results as far as possible. Fig. 3. Analysis process for evidence of software in the evaluation example The formal evaluation model is a tree model based on the detailed evaluation demand. Evidences for evaluation can be selected through the analysis for the evaluation demand, combining with the restrictions of evaluation cost and technical conditions, evidences may be classified and combined based on the evaluation logic, and the evidences may be mapped as the leaf nodes of models, and the non-leaf nodes of models represents different combining operations (logic operation or mathematic operation). During evaluation, each leaf node shall be evaluated firstly to generate evaluation scores, then operation may be conducted from down to up according to the combining operation of non-leaf node, the corresponding evaluation scores for each non-leaf node will also be generated. Based on weight-decided complexity (evaluation demand, branch quantity and clarification level of branches relative importance), direct assignment or methods like AHP are used to determine weight of each branch. If the branch number is smaller and the relative importance is more specific, the determination of weight of the branch is relatively simple, and we can use the method of direct assignment. If the branch number is bigger and the process to determine relative importance is complex, then we can consider methods similar to AHP to determine the branch weights [13]. The evaluation model established in this example is relatively simple. The nodes contain fewer branches, the process to determine the weight of the branch is relatively simple, and thus a direct assignment method will be acceptable. The level model can set up multiple orderly levels and mapping conditions according to the demand through different levels of description on formal degree of domain software, and the mapping conditions can be restricted based on the scores of leaf node and non-leaf node of models. The scores of leaf-node represents the formal degree of the specific evidence, while the scores of non-leaf nodes are the combining operation for multiple evidence scores, they can set up the corresponding relations between the domain software and level through judging the matching situation of evaluation instance and mapping conditions. V. The Construction and Application of Evaluation Framework In the previous paragraphs, we conducted formal analysis for the domain software evidence, and set up evalua-

6 798 Chinese Journal of Electronics 2016 tion model and level model based on the evidence. Thus the formal degree of specific domain software can be evaluated. Each tool and the model established for evaluation can be stored in the formal evaluation resource library, helping to develop formal analysis and creditable construction and evolution. This paper illustrated the formal degree evaluation and analysis process for domain software combining with Fig.4, the whole lifecycle of the domain software was monitored during the evaluation process, including the main periods of function analysis, architecture design, system implementation, test analysis and application. The rectangular at the bottom of this figure indicates the formal evaluation resource library, including various evaluation resources such as evaluation model, evidence measure tool, condition distinguishing tool, formal analysis tool, formal transformation model. The resources library will offer support for the formal degree evaluation of domain software; meanwhile, various resources generated from each evaluation will also be stored into the resources library. The middle dashed rectangular identifies the whole evaluation and analysis process. The left rectangular includes the evidence used in the evaluation, these evidences are various artifacts in the lifecycle of the software, and these evidences will be classified into three types to analyze according to the instructions in the previous sections. Meanwhile, through the analysis for the evaluation demand, an evaluation model can be established according to the actual needs, the established evaluation model in the resource library could be referred to. Through the various formal analysis tools in the resource library, refer to various transformation models established, the function evidence and measure evidence can be analyzed. Through the condition judgment tools in the resource library, the restriction evidence can be analyzed; finally the corresponding scores can get through evidence measure tools. The evaluation model instance can be formed through combining with the evaluate model and evidence scores, through matching the formal level mapping conditions in the level model, the domain software can be mapped into the corresponding level in the level model, thus the formal degree of domain software can be weighted. Problems can be analyzed based on the formal level and evaluation data of domain software, determine the bottleneck of the formal degree, thus give the suggestions for domain software, these suggestions will be as new requirement of iteration for domain software, directing the evolution of the domain software, thus continuously improve the formal degree of domain software. This paper continued to take the example duty management model of power plant production management software in the power domain to describe application of this approach in a simple way. The example includes three function evidences (the source of F 1, F 2, F 3 ), three restriction evidences (R 1, R 2, R 3 ) and two measure evidences (the source of M 1 ), according to the evaluation demand, the evidence is mapped as the leaf node on the evaluation model, the function evidences in the example are mapped as three leaf nodes of F 1, F 2 and F 3. The restriction evidences are mapped as three leaf nodes of R 1, R 2 and R 3, and the measure evidence is mapped as one leaf node of M 1, a triple evaluation model is established according to these nodes, the model combination operation is simple summation based on weight, more nodes and more complex evaluation models can be established according to requirements while evaluating actually. According to different types of evidences that the leaf nodes correspond to, the quantitative analysis can be done for the evidence, and the corresponding scores can also be generated referring to the factors including formal tools, the creditable degree of domain experts. As shown in Fig.1, F 1 function is relatively easy to implement in this example, so software support for the function is sufficient. And the score of F 1 is relatively high. The formal transformations of F 2 and F 3 are mapped to the same sequence, so they have the same score. But their function is more complex than F 1. Due to software limited support to F 2 and F 3,their score is slight lower. Similarly, the condition of R 3 is easier to meet than R 1 and R 2,sothescoreofR 3 is slight higher. And the score of R 1 and R 2 is slight lower. The score of M 1 is directly from the score field of evidence after pretreatment. As shown in Table 1, the second level of the evaluation model is the node combined by F, R and M, the third level is the root node, corresponding scores of non-leaf nodes may be calculated according to models, and mark of root node is the total score of this model. Similarly, according to the evaluation demand and the actual application, the level model can be established to weigh the formal degree of domain software. Each level includes a series of conditions which are based on evaluation model and restricted based on score of evidence and value of combination node. For example: minimum score of the necessary evidence may be set, average score of some optional evidences may be set, threshold of the total score may also be set, and so on. While instance specified in this paper is simple, decomposition of mapping conditions and set of thresholds will be a complex process if the evaluation demand and software scale are complex and great. Currently as full automation of decomposition of mapping conditions and set of thresholds is unavailable, experts are needed to some extent [14,15].In evaluation instance specified in this paper, the users keep a close eye on objective comment and subjective feeling to the software. While evidence M 1 corresponds to the subjective feeling, other evidences to objective, formalized comment to the software. Therefore, we regard the total score and score of M 1 as the core reference of level mapping, and the two reference values will be involved

7 Research on Formal Degree Evaluation and Analysis for Domain Software Based on Evidence 799 Leaf node (Score) Weight Table1.Weightandscoreforthenodeofevaluationmodelintheexample Non-leaf Remark node (Score) F 1 (90) 0.3 The father node of these three nodes is the non-leaf node F ; weight is F 2 (85) 0.3 given to them respectively according to their corresponding importance, and the score of node F is calculated based on the scores and weight of the three nodes. Among which, the follow-up transformation of F 2 and F 3 is F (86.5) 0.4 F 3 (85) 0.4 the same; therefore they have the same scores. Weight is given to F node according to the relative importance of this level. R 1 (85) 0.2 Thefathernodeofthesethreenodesisthenon-leafnodeR, weight is given R 2 (80) 0.3 to them respectively according to their own importance, and the scores of node R can be calculated ditto. Weight is given to R node according to the R (86) 0.2 R 3 (90) 0.5 relative importance of this level. ThefathernodeofM 1 node is M, it has only one node, therefore the weight M 1 (75) 1 is 1, ditto, the score of node M can be calculated. Weight is given to M M (75) 0.4 node according to the relative importance of this level. Weight Root node (Total) Root (81.8) in the mapping conditions. On the other side, the threshold is also set based on evaluation demand and application demand. While evidence M 1 corresponds to the user comment, a centesimal system scoring can be worked out through comprehensive analysis and treatment of all users comments. Here, thresholds of the total score and M 1 s score are set based on people s evaluation habits, 85 or above as outstanding level, as cut-off level, and that below 60 as failure level. Naturally, proper thresholds can also be set according to actual demands. A simple level model aiming at the example can be established. As shown in Table 2, the model mainly concerns about the total score of the model and user feedback (the score of M 1 evidence), therefore, these two indicators will be judged in the mapping conditions. It can be known combining with the data in Table 1, the formal degree of this model is mapped as level 1 in the example, and the bottleneck problem is that the score of M 1 evidence is very low, if you want to enhance the formal degree, then the score of M 1 evidence need to be increased. According to the evolution analysis conducted based on the connotation expressed by M 1 evidence, it is suggested to improve the support for the domain code, i.e., the module shall support the fault code of power equipment. The mentioned analysis on bottleneck issues of domain software also figures out a direction for software evolution. With reference to specific evaluation data, we can work out specific measures to improve the formal degree of software. To simplify the calculation, a simple instance is specified in this paper. Table 2. Level model in the evaluation example Level Mapping condition Remark Creditable level indicates the domain software has good formal fea- (total 85) 2 (M 1.score 85) ture. 1 0 (85 > total 60) (85 > M 1.score 60) (others) (unpredictable) Available level indicates the domain software can achieve the basic functions. The software has average formal feature. Unknown level indicates the domain software is not available or unable to be evaluated. Evaluation of large scale and complex domain software will involve considerably more workloads. VI. Conclusion and Future Work The domain software plays an important role in the industry informatization, this paper came up with a formal degree evaluation approach for domain software based on evidence aiming at the problems existing in the current quality evaluation and formal analysis of domain software. Through formal analysis for all kinds of evidence of the lifecycle of domain software, quantized evaluation results were generated based ontheformallevelofevidence. Through the analysis for evaluation demand and cost, an evaluation model based on evidence can be established. Through an evaluation example of software model in the domain of electricity, this paper illustrated the whole process of the formal degree evaluation. Through the approaches proposed in this paper, the formal degree of domain software can be weighted clearly, and further analysis for the bottleneck problems of software based on the evaluation data can be conducted, thus suggestions on improvements can be put forward. For instance, formal degree of domain software for the given evaluation instance is rated as Level 1. Specific indexes containing the level of domain software can be decided through analysis of specific data during the evaluation. Thus, proper measures can be taken to improve the rating of the index. All of these can offer formal support for the creditable construction, reliable analysis and creditable evolution for the domain software. This paper evaluated the domain software adopting formal approaches, made the evaluation more creditable as an important reference for the quality analysis of the domain software. However, the formal analysis for the domain software evidence has not formed automatic analysis approaches and framework, part of work, especially for the analysis of the evidence of multiple resources and different formats, has to be completed relying on experts

8 800 Chinese Journal of Electronics 2016 and labor participation. If we introduce various methods of formalized analysis and scoring mechanism in our method, a very complex transformation strategy needs to be established to form quantitative evaluation results in the same scale. Further studies are therefore necessary in the future work to solve these problems. References [1] Aversano Lerina and Tortorella Maria, Quality evaluation of floss projects: Application to ERP systems, Information and Software Technology, Vol.55, No.7, pp , [2] Zhang Guoheng, Ye Huilin and Lin Yuqing, Quality attribute modeling and quality aware product configurationin software product lines, Sofware Quality Journal, Vol.22, No.3, pp , [3] Islam Shareeful, Mouratidis Haralambos and R. Weippl Edgar, An empirical study on the implementation and evaluation of a goal-driven software development risk management model, Information and Software Technology, Vol.56, No.2, pp , [4] Yin Shouyi, Zhang Zhen, Hu Yang, et al., Mixed-levelmodeling methodology for network-on-chip architecture exploration, Chinese Journal of Electronics, Vol.23, No.3, pp , [5] Riccobene Elvinia and Scandurra Patrizia, A formal framework for service modeling and prototyping, Formal Aspects of Computing, Vol.26, No.6, pp , [6] Bao Tie, Liu Shufen and Wang Xiaoyan, Research on trustworthiness evaluation method for domain software based on actual evidence, Chinese Journal of Electronics, Vol.20, No.2, pp , [7] Wang Xiaoyan, Liu Shufen and Bao Tie, An evidence-driven framework for trustworthiness evaluation of software based on rules, Chinese Journal of Electronics, Vol.21, No.4, pp , [8] Zhang Xuejie, Wang Zhijian and Xu Feng, Reliability evaluation of cloud computing systems using hybrid methods, Intelligent Automation and Soft Computing, Vol.19, No.2 SI, pp , [9] Li Zhi, G. Hall Jon and Rapanotti Lucia, On the systematic transformation of requirements to specifications, Requirements Engineering, Vol.19, No.4 SI, pp , [10] Lochau Malte, Lity Sascha, Lachmann Remo, et al., Deltaoriented model-based integration testing of large-scale systems, Journal of Systems and Software, Vol.91, pp.63 84, [11] Cornu Benoit, Seinturier Lionel and Monperrus Martin, Exception handling analysis and transformation using fault injection: Study of resilience against unanticipated exceptions, Information and Software Technology, Vol.57, pp.66 76, [12] R. Dorado, A. Gomez-Moreno, E. Torres-Jimenez, et al., An AHP application to select software for engineering education, Computer Applications in Engimeering Education, Vol.22, No.2, pp , [13] M. Soycan, A quality evaluation of precise point positioning within the bernese GPS software version 5.0, Arabian Journal for Science and Engineering, Vol.37, No.1, pp , [14] Ayag Zeki, A combined fuzzy AHP-simulation approach to CAD software selection, International Journal of General Systems, Vol.39, No.7, pp , [15] A.A. Zaidan, B.B. Zaidan, Al-Haiqi Ahmed, et al., Evaluation and selection of open-source EMR software packages based on integrated AHP and TOPSIS, Journal of Biomedical Informatics, Vol.53, pp , BAO Tie was born in He received the Ph.D. degree from Jilin University in He is now an associate professor in College of Computer Science and Technology, Jilin University. His research interests include software evaluation and analysis, formal analysis and network management. ( baotie@jlu.edu.cn) LIU Shufen was born in She is now a professor and Ph.D. supervisor in College of Computer Science and Technology, Jilin University. Her research interests include computer supported cooperative work, software architecture and software programming method based on MDA. ( liusf@jlu.edu.cn) HAN Lu (corresponding author) was born in She received the M.S. degree from Jilin University in She is now a senior engineer in College of Computer Science and Technology, Jilin University. Her research interests include software evaluation and analysis, software testing. ( hanlu@jlu.edu.cn)