Cloud9 Parallel Symbolic Execution for Automated Real-World Software Testing Stefan Bucur, Vlad Ureche, Cristian Zamfir, George Candea School of Computer and Communication Sciences
Automated Software Testing Automated Techniques λ Symbolic Execution Scalability Applicability Usability Industrial SW Testing Manual Testing Static Analysis Fuzzing Model Checking 2
Cloud9 - The Big Picture Parallel symbolic execution Linear scalability on commodity clusters Full symbolic POSIX support Applicable on real-world systems Platform for writing test cases Easy-to-use platform API 3
Automated Systems Testing λ Symbolic Execution Promising for systems testing: KLEE [*] High-coverage test cases Found new bugs... But applied only on small programs [*] C. Cadar, D. Dunbar, D. Engler, KLEE: Unassisted and automatic generation of high-coverage tests for complex systems programs, OSDI 2008 4
Apache Memcached 5 GNU Coreutils
Symbolic Execution in a Nutshell void proc_pkt(packet_t* pkt) { } if (pkt->magic!= 0xC9) { } err(pkt); return; if (pkt->cmd == GET) {... } else if...... [C9 A0... ] 6
Symbolic Execution in a Nutshell void proc_pkt(packet_t* pkt) { } if (pkt->magic!= 0xC9) { } err(pkt); return; if (pkt->cmd == GET) {... } else if...... [C9 A0... ] 6
Symbolic Execution in a Nutshell void proc_pkt(packet_t* pkt) { } if (pkt->magic!= 0xC9) { } err(pkt); return; if (pkt->cmd == GET) {... } else if...... [C9 A0... ] pkt->magic!= 0xC9 6
Symbolic Execution in a Nutshell void proc_pkt(packet_t* pkt) { } if (pkt->magic!= 0xC9) { } err(pkt); return; if (pkt->cmd == GET) {... } else if...... [C9 A0... ] pkt->magic!= 0xC9 pkt->cmd == GET 6
Symbolic Execution in a Nutshell void proc_pkt(packet_t* pkt) { } if (pkt->magic!= 0xC9) { } err(pkt); return; if (pkt->cmd == GET) {... } else if...... [C9 A0... ] pkt->magic!= 0xC9 pkt->cmd == GET 6
Symbolic Execution in a Nutshell void proc_pkt(packet_t* pkt) { } if (pkt->magic!= 0xC9) { } err(pkt); return; if (pkt->cmd == GET) {... } else if...... λ 7
Symbolic Execution in a Nutshell void proc_pkt(packet_t* pkt) { } if (pkt->magic!= 0xC9) { } err(pkt); return; if (pkt->cmd == GET) {... } else if...... λ.magic == 0xC9 λ λ.magic!= 0xC9 7
Symbolic Execution in a Nutshell void proc_pkt(packet_t* pkt) { } if (pkt->magic!= 0xC9) { } err(pkt); return; if (pkt->cmd == GET) {... } else if...... λ.cmd == GET λ.magic == 0xC9 λ λ.cmd!= GET λ.magic!= 0xC9 7
Symbolic Execution in a Nutshell void proc_pkt(packet_t* pkt) { } if (pkt->magic!= 0xC9) { } err(pkt); return; if (pkt->cmd == GET) {... } else if...... λ.cmd == GET λ.magic == 0xC9 λ λ.cmd!= GET λ.magic!= 0xC9 2 program size paths 7
CPU Bottleneck Memory Exhaustion 8
Parallel Tree Exploration W1 W2 W3 8
Parallel Tree Exploration W1 W2 W3 Key research problem: Scalable parallel exploration 8
Linear Solution to Exponential Problem Time to Test Program Size 9
Linear Solution to Exponential Problem 1 worker Time to Test Program Size Testing target 9
Linear Solution to Exponential Problem 1 worker Time to Test 2 workers 4 workers 8 workers Program Size Testing target Bring testing time down to practical values 9
Throw Hardware at the Problem 10
Scalability Challenges Tree structure not known a priori?????????? 11
Scalability Challenges Static Allocation 12
Scalability Challenges 12
Scalability Challenges Anticipate Allocation 13
Scalability Challenges 13
Outline Scalable Parallel Symbolic Execution POSIX Environment Model Evaluation 14
Cloud9 Architecture Global Symbolic Tree 15
Cloud9 Architecture W1 s Local Tree W2 s Local Tree W3 s Local Tree Each worker runs a local sequential symbolic execution engine (KLEE) 15
Cloud9 Architecture Candidate nodes are selected for exploration Fence nodes Fence nodes bound the local tree Candidate nodes 16
Load Balancing W1 W2 W3 LB Hybrid distributed system: centralized reports, P2P work transfer 17
Load Balancing W1 W2 W3 LB Hybrid distributed system: centralized reports, P2P work transfer 17
Load Balancing W1 W2 W3 LB Hybrid distributed system: centralized reports, P2P work transfer 17
Work Transfer W1 Candidate Fence 18
Work Transfer W1 W2 Candidate Fence 18
Work Transfer Virtual W1 W2 Candidate Fence 18
Work Transfer Virtual W1 W2 Candidate Fence 18
Work Transfer Materialized W1 W2 Candidate Fence 18
Work Transfer W1 W2 Exploration disjointness + completeness Candidate Fence 18
Path-based Encoding 0 1 0 0 1 0 0 1 1 Nodes are encoded as paths in tree Compact binary representation Two paths can share common prefix 0 0 1 0 Small encoding size For a tree of 2 100 leaves, a path fits in <128 bits (16 bytes) 19
Load Balancing in Practice Work done [% of total instructions] 100 90 80 70 60 50 40 30 20 10 0 Continuous load balancing LB stops after 4 min LB stops after 1 min 0 2 4 6 8 10 Time [minutes] Load balancing necessary to ensure scalability 20
Outline Scalable Parallel Symbolic Execution POSIX Environment Model Evaluation 21
Calls into the Environment if (fork() == 0) {... if ((res = recv(sock, buff, size, 0)) > 0) { pthread_mutex_lock(&mutex); memcpy(gbuff, buff, res); pthread_mutex_unlock(&mutex); }... } else {... pid_t pid = wait(&stat);... } 22
Environment Model fork() Program Under Test Cannot directly execute symbolically Environment (C Library / OS) 23
Environment Model fork() Program Under Test Model Code Equivalent functionality Executable symbolically Environment (C Library / OS) Symbolic Execution Engine 23
Starting Point Single-threaded utilities Single-threaded isolated nodes Files Network Stubs POSIX Symbolic Execution Engine 24
POSIX Environment Model Single-threaded utilities Multi-threaded programs Distributed systems Message passing Asynchronous events, IPC Servers and clients Files Threads pthread_* Processes Pipes Signals Network TCP/UDP/UNIX POSIX Symbolic Execution Engine 25
Key Changes in Symbolic Execution Multithreading and Scheduling Deterministic or symbolic scheduling Non-preemptive execution model Address Space Isolation Copy on Write (CoW) between processes CoW domains for memory sharing 26
Symbolic Engine System Calls Symbolic engine support needed for threads/processes 1. Thread/process lifecycle 2. Synchronization 3. Shared memory 1 2 3 Symbolic Engine System Calls thread_create thread_terminate process_fork process_terminate get_context thread_preempt thread_sleep thread_notify get_wait_list make_shared 27
Outline Scalable Parallel Symbolic Execution POSIX Environment Model Evaluation 28
Testing Real-World Software Apache Memcached 29 GNU Coreutils
Time to Reach Target Coverage Time to achieve target coverage [minutes] 60 50 40 30 20 10 0 printf 60% coverage 70% coverage 80% coverage 90% coverage 1 4 8 24 48 Number of workers Faster time-to-cover, higher coverage values 30
Increase in Code Coverage Additional code covered [ % of program LOC ] 50 40 30 20 10 0 Coreutils suite (12 workers, 10 min.) 0 10 20 30 40 50 60 70 80 90 Index of tested Coreutil (sorted by additional coverage) Consistent code coverage increase 31
Exhaustive Exploration Time to complete exhaustive test [hours] 6 5 4 3 2 1 0 memcached (7.4 10 4 paths) 2 4 6 12 24 48 Number of workers Scalability of exhaustive path exploration 32
Instruction Throughput Useful work done [ # of instructions ] 1.8e+10 1.6e+10 1.4e+10 1.2e+10 1.0e+10 8.0e+09 6.0e+09 4.0e+09 2.0e+09 0.0e+00 4 minutes 6 minutes 8 minutes 10 minutes 1 4 6 12 24 48 Number of workers memcached Linear scalability with number of workers 33
Experimental Setup Symbolic State Client Process Symbolic cmd. TCP Stream Srv. response memcached/ Apache/ lighttpd Execute the whole world symbolically 34
Symbolic Test Cases Easy-to-use API for developers to write symbolic test cases Basic symbolic memory support POSIX extensions for environment control Network conditions, fault injection, symbolic scheduler 35
Symbolic Test Cases Testing HTTP header extension make_symbolic(hdrdata); // Append symbolic header to request strcat(req, X-NewExtension: ); strcat(req, hdrdata); // Enable fault injection on socket ioctl(ssock, SIO_FAULT_INJ, RD WR); // Symbolic stream fragmentation ioctl(ssock, SIO_PKT_FRAGMENT, RD); 36
Conclusions Parallel symbolic execution Linear scalability on commodity clusters Full POSIX environment model Real-world systems testing Use cases Increasing coverage Exhaustive path exploration Bug patch verification 37