Cloud9 Parallel Symbolic Execution for Automated Real-World Software Testing

Transcription

1 Cloud9 Parallel Symbolic Execution for Automated Real-World Software Testing Stefan Bucur, Vlad Ureche, Cristian Zamfir, George Candea School of Computer and Communication Sciences

2 Automated Software Testing Automated Techniques λ Symbolic Execution Scalability Applicability Usability Industrial SW Testing Manual Testing Static Analysis Fuzzing Model Checking 2

3 Cloud9 - The Big Picture Parallel symbolic execution Linear scalability on commodity clusters Full symbolic POSIX support Applicable on real-world systems Platform for writing test cases Easy-to-use platform API 3

4 Automated Systems Testing λ Symbolic Execution Promising for systems testing: KLEE [*] High-coverage test cases Found new bugs... But applied only on small programs [*] C. Cadar, D. Dunbar, D. Engler, KLEE: Unassisted and automatic generation of high-coverage tests for complex systems programs, OSDI

5 Apache Memcached 5 GNU Coreutils

6 Symbolic Execution in a Nutshell void proc_pkt(packet_t* pkt) { } if (pkt->magic!= 0xC9) { } err(pkt); return; if (pkt->cmd == GET) {... } else if [C9 A0... ] 6

7 Symbolic Execution in a Nutshell void proc_pkt(packet_t* pkt) { } if (pkt->magic!= 0xC9) { } err(pkt); return; if (pkt->cmd == GET) {... } else if [C9 A0... ] 6

8 Symbolic Execution in a Nutshell void proc_pkt(packet_t* pkt) { } if (pkt->magic!= 0xC9) { } err(pkt); return; if (pkt->cmd == GET) {... } else if [C9 A0... ] pkt->magic!= 0xC9 6

9 Symbolic Execution in a Nutshell void proc_pkt(packet_t* pkt) { } if (pkt->magic!= 0xC9) { } err(pkt); return; if (pkt->cmd == GET) {... } else if [C9 A0... ] pkt->magic!= 0xC9 pkt->cmd == GET 6

10 Symbolic Execution in a Nutshell void proc_pkt(packet_t* pkt) { } if (pkt->magic!= 0xC9) { } err(pkt); return; if (pkt->cmd == GET) {... } else if [C9 A0... ] pkt->magic!= 0xC9 pkt->cmd == GET 6

11 Symbolic Execution in a Nutshell void proc_pkt(packet_t* pkt) { } if (pkt->magic!= 0xC9) { } err(pkt); return; if (pkt->cmd == GET) {... } else if λ 7

12 Symbolic Execution in a Nutshell void proc_pkt(packet_t* pkt) { } if (pkt->magic!= 0xC9) { } err(pkt); return; if (pkt->cmd == GET) {... } else if λ.magic == 0xC9 λ λ.magic!= 0xC9 7

13 Symbolic Execution in a Nutshell void proc_pkt(packet_t* pkt) { } if (pkt->magic!= 0xC9) { } err(pkt); return; if (pkt->cmd == GET) {... } else if λ.cmd == GET λ.magic == 0xC9 λ λ.cmd!= GET λ.magic!= 0xC9 7

14 Symbolic Execution in a Nutshell void proc_pkt(packet_t* pkt) { } if (pkt->magic!= 0xC9) { } err(pkt); return; if (pkt->cmd == GET) {... } else if λ.cmd == GET λ.magic == 0xC9 λ λ.cmd!= GET λ.magic!= 0xC9 2 program size paths 7

15 CPU Bottleneck Memory Exhaustion 8

16 Parallel Tree Exploration W1 W2 W3 8

17 Parallel Tree Exploration W1 W2 W3 Key research problem: Scalable parallel exploration 8

18 Linear Solution to Exponential Problem Time to Test Program Size 9

19 Linear Solution to Exponential Problem 1 worker Time to Test Program Size Testing target 9

20 Linear Solution to Exponential Problem 1 worker Time to Test 2 workers 4 workers 8 workers Program Size Testing target Bring testing time down to practical values 9

21 Throw Hardware at the Problem 10

22 Scalability Challenges Tree structure not known a priori?????????? 11

23 Scalability Challenges Static Allocation 12

24 Scalability Challenges 12

25 Scalability Challenges Anticipate Allocation 13

26 Scalability Challenges 13

27 Outline Scalable Parallel Symbolic Execution POSIX Environment Model Evaluation 14

28 Cloud9 Architecture Global Symbolic Tree 15

29 Cloud9 Architecture W1 s Local Tree W2 s Local Tree W3 s Local Tree Each worker runs a local sequential symbolic execution engine (KLEE) 15

30 Cloud9 Architecture Candidate nodes are selected for exploration Fence nodes Fence nodes bound the local tree Candidate nodes 16

31 Load Balancing W1 W2 W3 LB Hybrid distributed system: centralized reports, P2P work transfer 17

32 Load Balancing W1 W2 W3 LB Hybrid distributed system: centralized reports, P2P work transfer 17

33 Load Balancing W1 W2 W3 LB Hybrid distributed system: centralized reports, P2P work transfer 17

34 Work Transfer W1 Candidate Fence 18

35 Work Transfer W1 W2 Candidate Fence 18

36 Work Transfer Virtual W1 W2 Candidate Fence 18

37 Work Transfer Virtual W1 W2 Candidate Fence 18

38 Work Transfer Materialized W1 W2 Candidate Fence 18

39 Work Transfer W1 W2 Exploration disjointness + completeness Candidate Fence 18

40 Path-based Encoding Nodes are encoded as paths in tree Compact binary representation Two paths can share common prefix Small encoding size For a tree of leaves, a path fits in <128 bits (16 bytes) 19

41 Load Balancing in Practice Work done [% of total instructions] Continuous load balancing LB stops after 4 min LB stops after 1 min Time [minutes] Load balancing necessary to ensure scalability 20

42 Outline Scalable Parallel Symbolic Execution POSIX Environment Model Evaluation 21

43 Calls into the Environment if (fork() == 0) {... if ((res = recv(sock, buff, size, 0)) > 0) { pthread_mutex_lock(&mutex); memcpy(gbuff, buff, res); pthread_mutex_unlock(&mutex); }... } else {... pid_t pid = wait(&stat);... } 22

44 Environment Model fork() Program Under Test Cannot directly execute symbolically Environment (C Library / OS) 23

45 Environment Model fork() Program Under Test Model Code Equivalent functionality Executable symbolically Environment (C Library / OS) Symbolic Execution Engine 23

46 Starting Point Single-threaded utilities Single-threaded isolated nodes Files Network Stubs POSIX Symbolic Execution Engine 24

47 POSIX Environment Model Single-threaded utilities Multi-threaded programs Distributed systems Message passing Asynchronous events, IPC Servers and clients Files Threads pthread_* Processes Pipes Signals Network TCP/UDP/UNIX POSIX Symbolic Execution Engine 25

48 Key Changes in Symbolic Execution Multithreading and Scheduling Deterministic or symbolic scheduling Non-preemptive execution model Address Space Isolation Copy on Write (CoW) between processes CoW domains for memory sharing 26

49 Symbolic Engine System Calls Symbolic engine support needed for threads/processes 1. Thread/process lifecycle 2. Synchronization 3. Shared memory Symbolic Engine System Calls thread_create thread_terminate process_fork process_terminate get_context thread_preempt thread_sleep thread_notify get_wait_list make_shared 27

50 Outline Scalable Parallel Symbolic Execution POSIX Environment Model Evaluation 28

51 Testing Real-World Software Apache Memcached 29 GNU Coreutils

52 Time to Reach Target Coverage Time to achieve target coverage [minutes] printf 60% coverage 70% coverage 80% coverage 90% coverage Number of workers Faster time-to-cover, higher coverage values 30

53 Increase in Code Coverage Additional code covered [ % of program LOC ] Coreutils suite (12 workers, 10 min.) Index of tested Coreutil (sorted by additional coverage) Consistent code coverage increase 31

54 Exhaustive Exploration Time to complete exhaustive test [hours] memcached ( paths) Number of workers Scalability of exhaustive path exploration 32

55 Instruction Throughput Useful work done [ # of instructions ] 1.8e e e e e e e e e e+00 4 minutes 6 minutes 8 minutes 10 minutes Number of workers memcached Linear scalability with number of workers 33

56 Experimental Setup Symbolic State Client Process Symbolic cmd. TCP Stream Srv. response memcached/ Apache/ lighttpd Execute the whole world symbolically 34

57 Symbolic Test Cases Easy-to-use API for developers to write symbolic test cases Basic symbolic memory support POSIX extensions for environment control Network conditions, fault injection, symbolic scheduler 35

58 Symbolic Test Cases Testing HTTP header extension make_symbolic(hdrdata); // Append symbolic header to request strcat(req, X-NewExtension: ); strcat(req, hdrdata); // Enable fault injection on socket ioctl(ssock, SIO_FAULT_INJ, RD WR); // Symbolic stream fragmentation ioctl(ssock, SIO_PKT_FRAGMENT, RD); 36

59 Conclusions Parallel symbolic execution Linear scalability on commodity clusters Full POSIX environment model Real-world systems testing Use cases Increasing coverage Exhaustive path exploration Bug patch verification 37