Real World Software Assurance Test Suite: STONESOUP Charles Oliveira/SAMATE <charles.deoliveira@nist.gov> Guest Researcher at Software and Systems Division, IT Laboratory NIST
Outline - Introduction STONESOUP program Test suite Test case sample TEXAS usage Documents and reports 2
Introduction - SOUP 3rd party software Open source libs libssl libxml libpq... Is this Software Of Unknown Provenance (SOUP) safe? Frameworks Java/ Spring C++/ Boost PHP/ Zend... Standalone apps Apache Postgres Drivers... Application 3
STONESOUP program Securely Taking On New Executable Software Of Uncertain Provenance (STONESOUP) http://www.iarpa.gov/index.php/research-programs/stonesoup 4
STONESOUP program The goal of STONESOUP program was to eliminate the effects of vulnerabilities in software applications by: - extending the scope and capability of approaches for analysis, confinement, and diversification; - addressing a wide range of security vulnerabilities within the same framework; - integrating approaches to leverage the strengths and weaknesses of each; - adding no more than 10% running time slowdown. 5
STONESOUP program Phase 1 Neutralize 75% of vulnerabilities of 2 weakness types in 10k SLOC programs Phase 2 Neutralize 80%+ of vulnerabilities of 4 weakness types in 100k SLOC programs Phase 3 Phase 3 performers were those that made significant progress in Phase 2 as measured by the program metrics. The three teams and the names of their developmental tools are: Kestrel Institute - VIBRANCE (video) Columbia University - Minestrone Grammatech - PEASOUP Neutralize 90%+ of vulnerabilities of 6 weakness types in 500k SLOC programs 6
STONESOUP program - Performers STONESOUP performers neutralize vulnerabilities in: 7
STONESOUP program - Test & Evaluation System - Test & Evaluation execution and Analysis System (TEXAS) was designed and developed to test Performer technology - Developed by STONESOUP team - Command Line Interface (CLI) to run and evaluate tests cases - Communication API to interact to Performer s tools 8
Test suite - Base programs 478 637 636 638 380 GNU Tree 380 637 637 GNU Grep 478 637 477 479 JTree 380 160 480 476 Number of test cases per base program in 9 red circles
Test suite - CWEs for C programs Weakness type CWEs (56) Concurrency handling(765) 363 367 412 414 543 609 663 764 765 820 821 833 831 828 479 Injection(701) 078 088 089 Number handling(725) 190 191 194 195 196 197 369 682 839 Resource drains(733) 400 459 674 774 789 834 835 401 771 773 775 Memory corruption(965) Null pointer(693) 120 124 126 127 129 134 170 415 416 590 761 785 805 806 822 824 843 476 10
Test suite - CWEs for C programs Weakness type CWEs (56) Concurrency handling(765) 363 367 412 414 543 609 663 764 765 820 821 833 831 828 479 CWE-363: Race Condition Enabling Link Following (2.8) 078(TOCTOU) 088 089 Race Condition (2.8) Injection(701) CWE-367: Time-of-check Time-of-use CWE-412: Unrestricted Externally Accessible Lock (2.8) 190 191 194 195 196 197 369 682 839 Number Missing handling(725) CWE-414: Lock Check (2.8) CWE-543: Use of Singleton Pattern Without Synchronization in a Multithreaded Context (2.8) 400 459 674 774 789 834 835 401 771 773 775 CWE-609: Double-Checked (2.8) Resource drains(733) Locking CWE-663: Use of a Non-reentrant Function in a Concurrent Context (2.8) CWE-764: Multiple Locks of a Critical 120Resource 124 126(2.8) 127 129 134 170 415 416 590 761 785 805 Memory corruption(965) CWE-765: Multiple Unlocks of a Critical Resource (2.8) 806 822 824 843 CWE-820: Missing Synchronization (2.8) CWE-821: Incorrect Synchronization (2.8) 476 Null pointer(693) CWE-833: Deadlock (2.8) CWE-831: Signal Handler Function Associated with Multiple Signals (2.8) CWE-828: Signal Handler with Functionality that is not Asynchronous-Safe (2.8) CWE-479: Signal Handler Use of a Non-reentrant Function (2.8) 11
Test suite - CWEs for C programs Weakness type CWEs (56) Concurrency handling(765) 363 367 412 414 543 609 663 764 765 820 821 833 831 828 479 Injection(701) 078 088 089 190OS 191 194 195 196 197 369 Number handling(725) CWE-078: Command Injection (2.8)682 839 CWE-088: Argument Injection or Modification (2.8) 400SQL 459Injection 674 774 (2.8) 789 834 835 401 771 773 775 Resource drains(733) CWE-089: Memory corruption(965) Null pointer(693) 120 124 126 127 129 134 170 415 416 590 761 785 805 806 822 824 843 476 12
Test suite - CWEs for C programs Weakness type CWEs (56) Concurrency handling(765) 363 367 412 414 543 609 663 764 765 820 821 833 831 828 479 Injection(701) Number handling(725) 078 088 089 190 191 194 195 196 197 369 682 839 CWE-190: Integer Overflow Wraparound (2.8) 400 459or 674 774 789 834 835 401 771 773 775 Resource drains(733) CWE-191: Integer Underflow (Wrap or Wraparound) (2.8) CWE-194: Unexpected (2.8) 120Sign 124 Extension 126 127 129 134 170 415 416 590 761 785 805 Memory corruption(965) CWE-195: Signed to806 Unsigned Conversion Error (2.8) 822 824 843 CWE-196: Unsigned to Signed Conversion Error (2.8) 476 Null pointer(693) CWE-197: Numeric Truncation Error (2.8) CWE-369: Divide By Zero (2.8) CWE-682: Incorrect Calculation (2.8) CWE-839: Numeric Range Comparison Without Minimum Check (2.8) 13
Test suite - CWEs for C programs CWE-400: Resource Exhaustion (2.8) CWE-459: Incomplete Cleanup (2.8) Weakness type Recursion (2.8) CWEs (56) CWE-674: Uncontrolled CWE-774: Allocation of File Descriptors or Handles Without Limits or Throttling (2.8) 363 367 (2.8) 412 414 543 609 663 764 765 820 821 833 831 CWE-789: Uncontrolled Memory Allocation Concurrency handling(765) CWE-834: Excessive Iteration (2.8) 828 479 CWE-835: Infinite Loop (2.8) CWE-401: Memory Leak (2.8) 078 088 089 Injection(701) CWE-771: Missing Reference to Active Allocated Resource (2.8) CWE-773: Missing Reference to Active File Descriptor or Handle (2.8) 190 191 194 195 196 197 369 682 839 Number handling(725) CWE-775: Missing Release of File Descriptor or Handle after Effective Lifetime (2.8) Resource drains(733) Memory corruption(965) Null pointer(693) 400 459 674 774 789 834 835 401 771 773 775 120 124 126 127 129 134 170 415 416 590 761 785 805 806 822 824 843 476 14
Test suite - CWEs for C programs Weakness type CWEs (56) CWE-120: Classic Buffer Overflow (2.8) CWE-590: Free of Memory not on the Heap (2.8) 363 367 412 414Free 543of609 663not 764 765 of 820 821(2.8) 833 831 CWE-124: Buffer Underflow (2.8) CWE-761: Pointer at Start Buffer Concurrency handling(765) 828 479 CWE-126: Buffer Over-read (2.8) CWE-785: Use of Path Manipulation Function without Maximum-sized Buffer (2.8) CWE-127: Buffer Under-read (2.8) CWE-805: Buffer Access with Incorrect Length Value (2.8) CWE-129: Improper Validation of Array Index CWE-806: Buffer Access Using Size of Source Buffer (2.8) 078 088 089 Injection(701) CWE-134: Uncontrolled Format String (2.8) CWE-822: Untrusted Pointer Dereference (2.8) CWE-170: Improper Null Termination (2.8) CWE-824: of Uninitialized Pointer (2.8) 190 191 194 195Access 196 197 369 682 839 Number handling(725) CWE-415: Double Free (2.8) CWE-843: Type Confusion (2.8) CWE-416: Use Afterdrains(733) Free (2.8) 400 459 674 774 789 834 835 401 771 773 775 Resource Memory corruption(965) Null pointer(693) 120 124 126 127 129 134 170 415 416 590 761 785 805 806 822 824 843 476 15
Test suite - CWEs for C programs Weakness type CWEs (56) Concurrency handling(765) 363 367 412 414 543 609 663 764 765 820 821 833 831 828 479 Injection(701) 078 088 089 Number handling(725) 190 191 194 195 196 197 369 682 839 Resource drains(733) 400 459 674 774 789 834 835 401 771 773 775 120 124 126 127 129 134 170 415 416 590 761 785 805 Memory corruption(965) CWE-476: NULL (2.8) 806Pointer 822 824Dereference 843 Null pointer(693) 476 16
Test suite - CWEs for Java programs Weakness type CWEs (50) Concurrency handling(568) 363 367 412 414 543 609 663 764 765 820 821 833 832 567 572 Injection(526) 078 088 089 564 Number handling(532) 190 191 194 195 196 197 369 839 Resource drains(532) 400 459 674 774 789 834 835 Error handling(532) Tainted data(498) 209 248 252 253 390 391 460 584 023 036 041 606 17
Test suite - CWEs for Java programs Weakness type CWEs (50) Concurrency handling(568) 363 367 412 414 543 609 663 764 765 820 821 833 832 567 572 CWE-363: Race Condition Enabling Link Following (2.8) 078 088 (TOCTOU) 089 564 Race Condition (2.8) Injection(526) CWE-367: Time-of-check Time-of-use CWE-412: Unrestricted Externally Accessible Lock (2.8) 190(2.8) 191 194 195 196 197 369 839 Number handling(532) CWE-414: Missing Lock Check CWE-543: Use of Singleton Pattern Without Synchronization in a Multithreaded Context (2.8) CWE-609: Double-Checked Locking 400 459(2.8) 674 774 789 834 835 Resource drains(532) CWE-663: Use of a Non-reentrant Function in a Concurrent Context (2.8) CWE-764: Multiple Locks of a Critical 209 248Resource 252 253(2.8) 390 391 460 584 Error handling(532) CWE-765: Multiple Unlocks of a Critical Resource (2.8) CWE-820: Missing Synchronization (2.8) 023 036 041 606 Tainted data(498) CWE-821: Incorrect Synchronization (2.8) CWE-833: Deadlock (2.8) CWE-832: Unlock of a Resource that is not Locked (2.8) CWE-567: Unsynchronized Access to Shared Data in a Multithreaded Context (2.8) CWE-572: Call to Thread run() instead of start() (2.8) 18
Test suite - CWEs for Java programs Weakness type CWEs (50) Concurrency handling(568) 363 367 412 414 543 609 663 764 765 820 821 833 832 567 572 Injection(526) 078 088 089 564 CWE-078: OS Command Injection (2.8) 190 191 194 195 196 197 369 839 Number handling(532) CWE-088: Argument Injection or Modification (2.8) 459 674 774 789 834 835 Resource drains(532) CWE-089: SQL400 Injection (2.8) CWE-564: SQL Injection: Hibernate (2.8) 209 248 252 253 390 391 460 584 Error handling(532) Tainted data(498) 023 036 041 606 19
Test suite - CWEs for Java programs Weakness type CWEs (50) Concurrency handling(568) 363 367 412 414 543 609 663 764 765 820 821 833 832 567 572 Injection(526) Number handling(532) 078 088 089 564 190 191 194 195 196 197 369 839 Integer Overflow or Wraparound 400 459 674 774 789 834(2.8) 835 ResourceCWE-190: drains(532) CWE-191: Integer Underflow (Wrap or Wraparound) (2.8) CWE-194: Unexpected Sign252 Extension 209 248 253 390(2.8) 391 460 584 Error handling(532) CWE-195: Signed to Unsigned Conversion Error (2.8) to 036 Signed Error (2.8) 023 041Conversion 606 TaintedCWE-196: data(498)unsigned CWE-197: Numeric Truncation Error (2.8) CWE-369: Divide By Zero (2.8) CWE-839: Numeric Range Comparison Without Minimum Check (2.8) 20
Test suite - CWEs for Java programs Weakness type CWEs (50) CWE-400: Resource Exhaustion (2.8) CWE-459: Incomplete Cleanup363 (2.8) 367 412 414 543 609 663 764 765 820 821 833 832 Concurrency handling(568) CWE-674: Uncontrolled Recursion (2.8) 567 572 CWE-774: Allocation of File Descriptors or Handles Without Limits or Throttling (2.8) 078 088 089 (2.8) 564 Injection(526) CWE-789: Uncontrolled Memory Allocation CWE-834: Excessive Iteration (2.8) Number handling(532) CWE-835: Infinite Loop (2.8) 190 191 194 195 196 197 369 839 Resource drains(532) Error handling(532) Tainted data(498) 400 459 674 774 789 834 835 209 248 252 253 390 391 460 584 023 036 041 606 21
Test suite - CWEs for Java programs Weakness type CWEs (50) 367 412Exposure 414 543 609 663 764 765 820 821 833 832 CWE-209: 363 Information Through an Error Message (2.8) Concurrency handling(568) 572 Exception (2.8) CWE-248: 567 Uncaught CWE-252: Unchecked Return Value (2.8) 088 089 564of Function Return Value (2.8) Injection(526)CWE-253: 078 Incorrect Check CWE-390: Detection of Error Condition Without Action (2.8) 191 194Error 195 196 197 369 839 Number handling(532) CWE-391: 190 Unchecked Condition (2.8) CWE-460: Improper Cleanup on Thrown Exception (2.8) 400 459 674 774 789 834 835 Resource drains(532) CWE-584: Return Inside Finally Block (2.8) Error handling(532) Tainted data(498) 209 248 252 253 390 391 460 584 023 036 041 606 22
Test suite - CWEs for Java programs Weakness type CWEs (50) Concurrency handling(568) 363 367 412 414 543 609 663 764 765 820 821 833 832 567 572 Injection(526) Number handling(532) Resource drains(532) Error handling(532) Tainted data(498) 078 088 089 564 190 191 194 195 196 197 369 839 CWE-023: Relative Path Traversal (2.8) CWE-036: Absolute Path (2.8) 400 459 674 774 789 834Traversal 835 CWE-041: Improper Resolution of Path Equivalence (2.8) 209 248 252 253 390 391 CWE-606: Unchecked Input460 for 584 Loop Condition (2.8) 023 036 041 606 23
Test suite - Base programs - Total of 7770 test cases which generates ~240GB compressed!!! - The STONESOUP Test and Evaluation team (T&E) used 277 independent virtual machines simultaneously on Amazon Web Services between April and December 2014 for performers to run the test cases. - The NIST VM is 22GB and contains test cases patched from the base program - The strategy was to patch the test cases, distributing.diff files instead of whole copies of each base program 24
Test suite - Virtual Machine (VMware) - Download (11x2GB) at http://samate.nist.gov/sard/testsuite.php#standalone - OS: Ubuntu 12.04 CPU: 4 VCPU recommended Memory: 4GB (8GB recommended) Storage: 59GB Total / 41GB Used / 16GB Available - Inside NIST_TT_VM folder there is a document with login and password for the VM - Important directories: - /opt/stonesoup: contains the entire NIST STONESOUP package including scripts and documents /opt/share: contains a TEXAS installation, test cases (diffs), base programs all their dependencies - Performers tools are not in the VM 25
Test case sample (from the virtual machine) - Pick a test case in /opt/share/testcases/ - Chosen: C-C476C-SUBV-03-ST01-DT05-DF10-CF22-01 26
Test case sample (from the virtual machine) - Pick a test case in /opt/share/testcases/ - Chosen: C-C476C-SUBV-03-ST01-DT05-DF10-CF22-01 C or Java 27
Test case sample (from the virtual machine) - Pick a test case in /opt/share/testcases/ - Chosen: C-C476C-SUBV-03-ST01-DT05-DF10-CF22-01 CWE-476: NULL Pointer Dereference 28
Test case sample (from the virtual machine) - Pick a test case in /opt/share/testcases/ - Chosen: C-C476C-SUBV-03-ST01-DT05-DF10-CF22-01 Algorithmic variant: refined CWEs mapped to a code snippet previously defined by T&E team 29
Test case sample (from the virtual machine) - Pick a test case in /opt/share/testcases/ - Chosen: C-C476C-SUBV-03-ST01-DT05-DF10-CF22-01 Base program: CMUD Coffee MUD CTRE GNU Tree FFMP FFMpeg GIMP Gimp GREP GNU Grep OSSL OpenSSL PSQL Postgres SUBV Apache Subversion WIRE Wireshark ELAS Elastic Search JMET Apache JMeter JENA Apache Jena JTRE Java Tree LENY Apache Lenya LUCE Apache Lucene POIX Apache POI 30
Test case sample (from the virtual machine) - Pick a test case in /opt/share/testcases/ - Chosen: C-C476C-SUBV-03-ST01-DT05-DF10-CF22-01 Injection point: represent specific locations in the base program that are guaranteed to be executed given the defined I/O pairs. Identifiers reference different injection points in each base program. 31
Test case sample (from the virtual machine) - Pick a test case in /opt/share/testcases/ - Chosen: C-C476C-SUBV-03-ST01-DT05-DF10-CF22-01 Taint source: 01 ENVIRONMENT_VARIABLE 02 FILE_CONTENTS 03 SOCKET 04 SHARED_MEMORY 32
Test case sample (from the virtual machine) - Pick a test case in /opt/share/testcases/ - Chosen: C-C476C-SUBV-03-ST01-DT05-DF10-CF22-01 Data type: 01 ARRAY 05 STRUCT 02 SIMPLE 06 TYPEDEF 03 VOID_POINTER 07 UNION 04 HEAP_POINTER 33
Test case sample (from the virtual machine) - Pick a test case in /opt/share/testcases/ - Chosen: C-C476C-SUBV-03-ST01-DT05-DF10-CF22-01 Data flow: 01 ADDRESS_ALIAS_1 11 BASIC 05 ADDRESS_AS_CONSTANT 12 VAR_ARG_LIST 06 ADDRESS_AS_FUNCTION_RETURN_VALUE 17 BUFFER_ADDRESS_POINTER 10 INDEX_ALIAS_50 18 JAVA_GENERICS 34
Test case sample (from the virtual machine) - Pick a test case in /opt/share/testcases/ - Chosen: C-C476C-SUBV-03-ST01-DT05-DF10-CF22-01 Control flow: 01 INTERCLASS_1 18 POINTER_TO_FUNCTION 08 INTERFILE_1 19 RECURSIVE 12 INTERPROCEDURAL_1 22 MACROS 16 INTERRUPT 26 FUNCT_INVOC_OVERLOAD 35
Test case sample (from the virtual machine) - Pick a test case in /opt/share/testcases/ - Chosen: C-C476C-SUBV-03-ST01-DT05-DF10-CF22-01 Unique increment: increment in case of multiple test cases are sharing the same parameters aforementioned. 36
Test case sample (from the virtual machine) - Pick a test case in /opt/share/testcases/ - Chosen: C-C476C-SUBV-03-ST01-DT05-DF10-CF22-01 - Browsing the test case - install/: this test case installation files scripts/: specific scripts to manage running process src/: the entire base program + files seeded with intentional weaknesses testdata/: input data which will [and won t] trigger the seeded weakness testoutput/: matching output data for each input data C-C476C-SUBV-03-ST01-DT05-DF10-CF22-01.xml: TEXAS makefile C-C476C-SUBV-03-ST01-DT05-DF10-CF22-01.yaml: benign and exploiting inputs 37
TEXAS usage Stage 1: standard compilation Stage 2: compilation with performer technology Analysis/ Compilation Execution Scoring I/O Pairs Analysis/ Compilation The source code or binary of a program is scanned looking for CWE code patterns and applying diversification techniques to harden the resulting binary. The output of the Analysis phase is a binary executable. Execution The Execution step is run for each I/O, and involves actually invoking the binary created in the Analyze step with known inputs. Performer technology may also monitor the execution of the binary to look for execution patterns indicative of an attack in progress or software vulnerability. Scoring Scoring executed immediately after the Execution step and looks at the environment for the known outputs defined in the metadata for the given I/O pair that was executed. 38
Documents & Reports Main STONESOUP documents provided at SARD website: - Test and evaluation phase 3 final report Performers reports Weaknesses documentation Test cases creation guide TEXAS user guides Visit: http://samate.nist.gov/sard/around.php 39
Questions? Charles Oliveira/SAMATE [charles.deoliveira@nist.gov] 40