Software Reliability Estimation Based on Static Error Detection

Transcription

1 7 th Central and Eastern European Software Engineering Conference in Russia - CEE-SECR 2011 October 31 November 3, Moscow Software Reliability Estimation Based on Static M. Moiseev, M. Glukhikh, A. Karpenko, H. Richter

2 Importance of Software Reliability Analysis Modern software contains errors Errors can lead to disasters Software Reliability Analysis Error detection should be organized 2

3 Known Approaches Heuristics approaches Dynamic approach Architecturebased approach Program metrics Development process 3

4 Known Approaches Program Metrics Based on simple code properties, such as number of statements number of conditions number of loops number of functions... 4

5 Known Approaches Development Process Metrics Based on development process properties, such as duration of development number & qualification of developers number & qualification of testers methodology used automation tools used 5

6 Known Approaches Others Runtime Based on failures observed at run-time Architecture-based Based on known reliability of program components 6

7 Our Approach Based on source code static analysis Delivers Ranking of errors (based on failure probability) Reliability characteristics Limitations Single-threaded C programs Error types uninitialized variable use incorrect pointer dereference pointer out of bounds 7

8 Features of Our Approach Analysis of a program model Analysis of all possible execution paths Advantages Reliability estimations is based on real errors Results are applicable for any exploitation conditions Makes debugging more effective Drawbacks Does not consider quantitative time Does not consider normal program exploitation Execution path probability estimation False positives problem 8

9 Program Classes Computational Programs Server 9

10 Reliability characteristics used Computational programs Probability of whole program successful execution P( ) Server programs Probability of n statements successful execution P(n) Mean executed statement number before failure n 10

11 Algorithms Model building State determination Error detection Error ranking Reliability estimation 11

12 Program Model Features Control flow graph Three-operand assignment form A = B op C If and Phi statements If Phi If Phi 12

13 State Determination Algorithms State representation Control flow analysis Statement analysis Sequential If statement analysis Phi statement analysis Loop analysis Interprocedural analysis 13

14 Program State Representation Based on obects, values, and probabilities set of triples = state probability Obect values intervals pointers Q, resource descriptors {( o, vk p k )} P( Q) 14

15 Probability normalization Control flow normalization Q in Input ( in ) P Q ( s ) = State normalization o : Q out ( o, v, p ) k k P Output ( out Q ) ( s) Q p ( o, v p ), k k k Q = P( Q) 15

16 Sequential Statement Analysis 16 a = b + c ( ) ( ) ( ) ( ) =,1, ,,1, 1..2, c b Q in ( ) ( ) { },1, a, Q out = =, ,6,, 4 1,5,, 4 1,4,, 4 1,3, 2 1,2,, 2 1,1, c c c c b b =, ,8,, 4 1,7,, 4 1,6,, 4 1,5,, 8 1,4, a a a a a

17 If Statement Analysis True and false combination consideration ( true ) P Q ( false ) P Q = = c C p ( o, v, p ) c C true false p ( o, v, p ) k k k k k c, k c. Normalization of state probabilities Normalization of non-affected triples probabilities 17

18 If Statement Analysis Example 172 combinations where a < b 28 combinations where a >= b Q in = ( a, ( 1..10),1), ( b, ( ),1 ),... Q false true Q ( a,( 1..10),0.86) ( a, ( 4..10),0.14) ( b, ( ),0.86) ( b, ( 4..10),0.14) Normalization: 0.86 for true, 0.14 for false 18

19 Phi Statement Analysis Identical triples are added together o ( ) in o, v, p Q,( o, v, r ) ( ) out o, v, p + r Q in, vk : k k 1 k k Q2 k k k Control flow normalization ( out ) ( in ) ( in Q = P Q P Q ) P In 1 In 2 Phi 19

20 Based on incorrect values in state uninitialized variable use o, v, pointer dereference ( o, v, p ) null out of bounds correct if k otherwise error is detected ( p ) ( o, v, p ) noninit ( o, v, p ) invalid ( o,( o, offset ), p ) i 0 offset < sizeof k noninit k k ( ) o k 20

21 Error Inhibition ob use (ob, valid, p1) (ob, invalid, p2) P(Q)=p1+p2 (ob, valid, p1) P(Q)=p1 21

22 Error Ranking Errors are sorted according to probability of occurrence Most dangerous errors can be corrected first Probabilities are summarized for same errors in the same statement 22

23 Overall reliability estimation probability of successful execution ( n) = P P( Q) end statements probability of n statements successful execution ( n) = P P( Q) n statements executed mean executed statements number before failure n n ( P( n) P( n + 1) ) = max n= 0 n 23

24 Implementation AEGIS static analyzer analysis of C/C++ source code interval, points to, resource analysis loop & interprocedural analysis spread range of program errors detected Results error ranking table P(n) table P( ) mean executed statements number before failure 24

25 Experiments made Purpose Testing of our approach Debugging example Test programs Students' proects Real-world proects (embedded software) 25

26 Sample of reliability analysis while (!(feof(f))) // 0.5 { i = t = 0; // Failure in one of three cases prov(&t, strlen(st), st); } Probability of successful execution is 0.75 = * *

27 Amount of errors in real-world proects 100 More than 500 errors, 2/3 of considered types Density about 0.8/1KLOC 80 Error number A B C D E F G H I J K L Proect name 27

28 Distribution of error number Error number E-06 1.E-05 1.E-04 1.E-03 1.E-02 1.E-01 0,25 0,5 1,0 Error probability 28

29 Debugging results Original Corrected n, Mstatements 29

30 Directions for Future Work Reliability estimation Annotations for path probability estimations Run-time analysis for path probability estimation Execution time estimation Static analysis itself Soundness & precision Parallel program analysis Annotations for functional error detection 30

31 Conclusion Approach for software reliability estimation based on error detection using static analysis Implementation in AEGIS tool (prototype) ranking of errors by the probability of occurrence probability of successful execution probability of N statement successful execution mean number of executed statements before failure 31

32 Contacts Saint Petersburg State Polytechnical University Digitek Labs Mikhail Glukhikh, Mikhail Moiseev, Anatoly Karpenko Clausthal University of Technology Harald Richter 32