Developers and the Software Supply Chain Andy Chou, PhD Chief Technology Officer Coverity, Inc.
About Andy CTO at Coverity since 2010 Co-founder at Coverity, 2003 From five guys in a garage to 280 employees worldwide PhD, Computer Science, Stanford University, 2003 Various roles: VP Engineering, Chief Scientist, Chief Architect, Product Manager, Security Research Director 2
This Talk is About Three Things 1. A developer perspective on the Software Supply Chain. 2. Static Analysis: what it s good at, and what it s not. 3. Shared experience of how to do the heavy lifting of Building It Right. 3
Software is Eating the World - Marc Andreessen Health Financial Automotive Software Communications Mobile Enterprise SCM / Logistics 81% of business leaders believe technology is a fundamental element of their business model Over 60 million tablets and 175 million smart phones will be in the workplace by the end of 2012 By 2016, open source software will be included in mission-critical applications within 99% of Global 2000 enterprises 4
Supply Chain Code Flow Producer Delivery Δ Δ Δ Δ Consumer Commercial developer Open source developer System Integrator Consultant IT department Appsec team Third party assessor User Developer! 5
Security Checkpoint Model Producer Checkpoint Delivery Consumer NIST SP 800-53 Static Analysis Dynamic Analysis Simulations White/grey/black box testing Fuzz testing Pen testing Signing Development process inspection 6
Security Checkpoint Model Producer Checkpoint Delivery Consumer NIST SP 800-53 Static Analysis Dynamic Analysis Simulations White/grey/black box testing Fuzz testing Pen testing Signing Development process inspection 7
More Complex Supply Chain A Hard Problem... Producer Consumer 8
More Complex Supply Chain Not any easier... Producer Producer Consumer Open Source 9
Static Analysis Core Technology Code Defects char *p Static Analysis Verification Engine true p = 0 true... if (x == 0) false p = foo() if(x!= 0) false s=*p return 10
Static Analysis Characterization Static Analysis (SA) inspects software artifacts (1) to identify defects without execution. Ranges in complexity from text pattern matching to full formal verification. Only SA has all of these desirable characteristics: Automatic Objective Scales to large code bases (>10MLOC) Coverage of every line of code Repeatable and deterministic 11
Static Analysis Drawbacks False Positives (FP): reported issues that turn out to be incorrect False negatives (FN): real defects that were not identified FN FP Real Defects Static Analysis Findings 12
Checkpoint Use Case Problems Producer Delivery Consumer Reject SA Defects Caught defects pre-accept Provided feedback to Producer False positives Highly inefficient looping cycle Source code access restrictions Strong incentive to accept defects 13
False Positives Producer Delivery Consumer SA Reject Issues Defects False Positives??? 14
False Positives False positives are hard to triage without understanding the code. Bugs get fixed. False positives accumulate. Over time, FP rate approaches 100%. False positives can be tracked, but it can become difficult to manage over many products and many versions. Defects Issues False Positives??? 15
Inefficient Looping cycle Each time defects go back to the developer, it causes a new testing cycle. This is likely to conflict with the developer s next release development and planning. This increases resistance to patching. Each consumer may have a different process or tool for review, leading to overlapping yet inconsistent results. Producer Reject Delivery Defects SA 16
Source code access Without source code, it is very difficult to triage correctly. It is exceedingly difficult to get most development organizations to give up their code. Even with the code, most consumers of software would not be able to understand the code well enough to triage defects correctly. 17
Organizational Incentives Developers are pressured by their organizations to deliver more, faster. Managers and Executives care about predictable delivery of differentiating features to market. Quality, Security, and many other attributes of software are not as easily measured as other attributes. 18
Straw Poll of Top Development Concerns Time to market, reduce cycle time Quality - Usability, Maintainability, Scalability, Security - CWE 25, OWASP 10 Development talent - Evaluating, Visibility Compliance - various standards Cost - Reduce development cost Open Source - Managing Quality and Security Cloud - Develop and testing applications Supply chain - quality and security 0 20 40 60
One of the Many Ways to Drive a Release Development start Feature Freeze Integration Freeze Release Freeze GA Issues: Feature Development W/o a clear metric for quality of features at FF, it is impossible to predict how long it will take to stabilize the release Features are complete 1 st round of QA Integration complete 2 st round of QA As a result, product is shipped when the GA date comes Final E2E QA
What s Wrong With This Picture? Producer Consumer Power to fix code. Accountable for broken software. Culture is the hardest, most important problem of all. 21
Only Developers Can be Highly Effective with Static Analysis 22
Developer-Centric Assurance Model Policy SA Producer Delivery Consumer Assurance Evidence Defects addressed in the inner loop of development 23
Static Analysis is in the SDL Every-sprint practices are developer processes SA is here. 24
for good reasons. Developers have the source code. (1) Developers can triage defects: they know the code. (2) Developers can fix defects. (3) Incentives Adoption Alignment Effort Resistance 25
The Developer s World A million-line program has ~20M characters (1M lines 20 characters/line), or about 40 novels 26
Code hardens over time and becomes harder to change 27
What developers need from a SA tool Accuracy. Few false positives. Clarity. Precise explanation of defect. What is it? Where is it? Depth. It needs to beat doing something else. Remediation. How do I fix it? Speed. Find defects while the code is still fresh in the mind. Scalability. Millions to tens of millions of lines of code. Workflow. Developers work in teams. 28
What you get out of a good SA deployment New developers learn faster. Grade A developers become more mindful of their code. We came to fix bugs; we stayed to become better software developers Grade B developers stop making the same mistakes. Grade C developers fix defects. Grade D developers are uncovered. 29
Getting the Most Effectiveness Out of SA Use it for what it s good at. Don t use it for what it s not good at. At first, focus on bottom-up adoption and integration. 30
Integration into SDLC High Static Analysis Deployment Maturity Model Level 1 Critical defects identified and addressed, no new defects introduced Level 2 Coverity integrated with existing development tools Level 3 Gates in release/ operational acceptance process Level 4 Coverity integrated into the desktop/ide Level 5 All legacy defects eliminated, build fails if new defects are introduced Adoption High 31 Confidential: For Coverity and Partner use only. Copyright Coverity, Inc., 2012
How Static Analysis Tools are Developed Analyze code We pray your code works like this code. Test for scalability+ more Refinement for every defect type Triage Most time consuming. Invent new algorithms Think real hard The easy problems are NP-complete. 32
How Static Analysis Tools are Developed Repeat for 30-60MLOC of open source code and a few commercial partners. Analyze code Most algorithms in academic papers only work for 100-1000 LOC. Test for scalability+ more Refinement for every defect type Triage Common real-world code idioms are baked in. If we know about them. Invent new algorithms Think real hard 33
Your Code Hasn t Been Through This Refinement Process. 34
Defect Density (defects / kloc) Open Source vs Commercial Defect Density (2012) 1 0.9 0.8 0.7 0.6 Commercial (260 Projects, 381 MLOC) 0.5 0.4 Open Source (118 Projects, 68 MLOC) 0.3 0.2 0.1 0 Under 100K 100K - 500K 500K - 1M Over 1M Code base size (LOC) 35
Issues detected Analysis of Android Gingerbread (11735 issues total) 7000 6000 5876 5000 4000 3000 2000 1843 1383 1000 730 437 351 304 247 222 167 93 82 0 Android Component 36
CWE Top 25: The Right Stuff. 37
CWE Top 25: What is SA Good At? 38
CWE Top 25: What You Need Brains For. 39
CWE Top 25: What Can Be Scanned 40
CWE Top 25: For Hired Guns. 41
SA for C/C++ Embedded Software 42
SA for Java Web Application 43
CWE Top 25 Static Analysis Focus Lists C/C++ embedded: 1. Buffer overflow 2. Buffer size 3. Integer overflow 4. Format string 5. Command injection 6. Hard-coded credentials 7. Path traversal 8. Dangerous functions Java web app: 1. SQL Injection 2. Command Injection 3. Cross-site Scripting 4. Hard-coded credentials 5. Path traversal 6. Dangerous functions 7. Open redirect 44
How to Create Your Own Focus List Learn what Static Analysis is good at. Learn what languages and technologies are used by your development team. Subset the CWE 25 by what is applicable to your system, and what SA is good at detecting. Make sure you get adoption! Fix rate is everything. Review the list as SA improves or your technologies or risks change. 45
Security is a Multidimensional Space No state of the art tool today covers everything well Depth of Analysis Context aware Remediation Model library View technologies Framework support Interprocedural C# Javascript Java SQLI XSS Command Path Injection Traversal CWE25/OWASP10 Coverage Language Coverage Python 46 Confidential: For Coverity and Partner use only. Copyright Coverity, Inc., 2013
What Can be Accomplished - CERN 47
Getting it Right! How to Talk to Developers. Search for Coverity Security Library Free, BSD-licensed remediation library for XSS. An exemplar. 48
Help developers get it done. What you want to get done. How to get it right. Code!!! 49
Help developers get it done. Developers don t care about how the attack works. Speak in their native language code. Generic advice is no good it needs to be specific to the programming language and technology they use. E.g. How to avoid Hibernate HQL injection in Java. We need to do a better job reaching them where they are: StackOverflow, github, google search. 50
Technologies on the Horizon 51
Stimuli Response Black-Box Testing Tests are constructed w/o insight and access to application internals. When is it done? Test SW Application Component 3 Leaf G Leaf H Leaf J User Input User Output Component 1 Component 2 Leaf A Leaf B Leaf C Leaf D Leaf E Leaf F 52 For Coverity and Partner use only. Copyright Coverity, Inc., 2012
Change Impact Analysis Change impact analysis enables understanding of the effect of a given code change beyond the place in the code where the change occurs f34 f25 f15 Foo f90 f34 Example: changing the behavior of a function might affect the behavior of other functions that call it Impacted code f56 f87......... f33... f77... f76 f89 f23... f54 f32 f67 Less obvious example: changing the type hierarchy might change the resolution of virtual methods, resulting in a change of behavior where those methods are called............... Changed code... Change ripple Legacy code 53 For Coverity and Partner use only. Copyright Coverity, Inc., 2012
Test Advice / Prioritization Test Advice Actionable work items to address risk due to inadequate testing Test Policy Evaluation Critical code analysis Change impact analysis Test execution analysis Customized Test Policy Test Monitoring Code Ownership and Change History Static Code Analysis 54 For Coverity and Partner use only. Copyright Coverity, Inc., 2012
SwA is a Process, Not a Push Button or Checkbox Today we re noticing specific vulnerabilities, but we ll never Build It Right unless we go deeply into development, understand it, and enact change from within the system. To really do SwA well requires deep insight into the processes, people, tools, skills, and especially culture of the organization producing the software. Developers play a key role, and we need to show them how to develop software right. Not merely point out what s wrong. 55
Questions
Thank You Dr. Andy Chou andy@coverity.com
Level 1 Centralized Build Analysis Improved process efficiency Identify & address most critical defects Memory Corruption Security Flaws Memory Leaks Crash causing defects Ensure no new defects are introduced Baseline code base Track & maintain code quality Notifications/alerts to developer/manager 58 Confidential: For Coverity and Partner use only. Copyright Coverity, Inc., 2012
Level 1 Centralized Analysis Process Automated daily or weekly build Assigned to relevant developers Check in Check in Check in Defects New defects Requirements definition Develop features Business Process Modeling Review defects Prioritize actions Make fixes Track progress Find & fix Check in Central source control management system or continuous integration server Defects
Level 2 Integrated into SDLC Integration with core development tools Improve developer productivity Notification via defect tracking systems SCM integration 1 Source file with defect 2 SCM query returns last person to edit the file 3 Owner automatically assigned to the defect #include stdio.h int main () { char * C = 0; *c = a ; Source Code Repository Latest rev: psmithers
Level 3 Governance/Quality Assurance Gates in release/operational acceptance process No new defects, legacy defects trend lower CCM and technical debt do not increase Validate that code has met your code governance goals before it is promoted to the next stage in the lifecycle
Level 4 Developer desktop analysis Integration via the IDE Identify & fix defects while working on code Development Environment Code Check-in Nightly build Fix defects locally on the developer machine Require clean code before check-in Continuous Integration/Nightly build
Level 5 - Coverity Clean Achieve Coverity Clean Legacy defects are eliminated Builds are failed if new defects are introduced Minimize late stage defects & churn from re-testing. Product ships with no critical defects Coverity used for testing all Life Critical software systems that are embedded in pacemaker devices. Zero Defect policy where all critical defects are prevented from making it to production. Major US Hospital Cardiac Rhythm Division 63 Confidential: For Coverity and Partner use only. Copyright Coverity, Inc., 2012
Scan project 64 Confidential: For Coverity and Partner use only. Copyright Coverity, Inc., 2012