Stories From the Front Lines: Deploying an Enterprise Code Scanning Program

Stories From the Front Lines: Deploying an Enterprise Code Scanning Program Adam Bixby Manager Gotham Digital Science 10/28/2010 YOUR LOGO HERE

Introduction Adam Bixby, CISSP, MS o Manager at Gotham Digital Science Penetration Tester Static Analysis/Code Auditing abixby {at} gdssecurity.com http://www.gdssecurity.com o Currently running an enterprise code scanning program at a large financial services organization Over 200 application integrated with code scanner in less than a year

Outline Why am I here? Lessons Learned Tips for a successful scanning program Code Scanning Integration Strategy Code Scanning Strategies - Pros and Cons Questions

Why am I here? Many companies try to invest in code scanning solutions only to fail o Common Reasons for Failure: Misguided perception of what a scanner can do Don t know how to use it properly Don t plan a pilot program Buy it without any training I ll figure it out Don t tune the product properly for their enterprise/application environment Fail to get support from upper management

What is not going to be discussed Comparisons of the different Static Analysis Tools/Solutions o Fortify SCA o Rational AppScan Source Edition (Ounce) o Vericode o Checkmarx o Coverity o Klockwork This talk is about the process

Lessons Learned

Lessons Learned Need IT/Upper Management support o Probably the most important takeaway from this presentation o Without upper level support or mandate, code scanning programs are doomed to fail Development teams will not be as cooperative Remediation of issues need to be mandated Issues will sit there scan after scan without being addressed o All Integrations I ve been involved in needed buyoff in order for their success

Lessons Learned Common excuses from upper management o Too expensive o We are already performing penetration tests o Application teams already have busy release schedules Business units want functionality Security scanning will only slow them down

Lessons Learned Common excuses from development teams o We don t have time o Busy release schedule o We can t fix all 1000 Cross-Site Scripting issues by our next major release o This code has been deployed with these issues for 10 years, why do we have to fix it now? Application makes us money, why change it?

Lessons Learned Common assumptions that are made by team performing code scanner rollout o All development teams are run the same o Development teams will be onboard Everyone wants to eliminate security issues, right? o Our 5 year old spare server will be able to run a static analysis scan without any problems Code scanners are very resource intensive and hog A LOT of memory o Here s a zip file with our code, this should scan properly Always missing libraries/dependencies

Tips for a Successful Scanning Program

Tips for a successful scanning program Get management support ASAP o Cannot stress this point enough o Will make the difference between success and failure o Who needs convincing? CIO CISO Application Owners IT Management o How do we go about convincing management?

Tips for a successful scanning program Arm yourself with solid fact about code scanning solutions o Help find and fix vulnerabilities o Cost effective because detection and remediation comes earlier in the Software Development Lifecycle (SDLC) Fixes can be incorporated into regular bug remediation schedule instead of having to be performed out of band Often the case when issues are found during pen tests If issue goes away in the code security scan, issue has been fixed. Not always necessary to allocated additional retesting time in release schedule

http://www.informit.com/articles/article.aspx?p=1357183

Actual cost benefit analysis at a large financial services firm based on real SDLC defect cost data Finding and fixing software security issues earlier in the SDLC makes economic sense! the cost of removing a software defect grows exponentially for each downstream phase of the development lifecycle in which it remains undiscovered Study by Gary McGraw (Cigital) and Jim Routh, CISO for KPMG US http://www.informit.com/articles/article.aspx?p=1357183

Want more evidence? Cost to repair a security vulnerability in an application increases later in the development cycle (Forrester, 2009):

Tips for a successful scanning program Arm yourself with solid facts about code scanning solutions, cont o Gets secure code to market faster o Makes your applications and your company more secure from outside and inside threats This is why we are all here and why we have jobs to keep bad guys out! o Secure company data/secrets o 75% of security breaches occur at the Application level (Gartner, 2005)

Tips for a successful scanning program Arm yourself with solid facts about code scanning solutions, cont o Exploited security vulnerabilities in critical externallyfacing applications can result in significant financial and reputation losses o Regulators, Compliance, Audit, customers, partners, and security policies are demanding security solutions Audit will often inquire if source code scanning for security is part of a development lifecycle From an audit report I was privy to: no requirements or guidance for regular security vulnerability assessments of source code and secure code development

Tips for a successful scanning program Arm yourself with solid facts about code scanning solutions, cont o Vulnerabilities in code can be difficult for developers to identify without proper tools and training o Penetration testing Will not find all issues More expensive to rely solely on pen tests Issues found in production are more expensive to fix Still should be performed since source code scanning will not find all issues (runtime bugs, server configuration bugs, business logic bugs, etc)

Tips for a successful scanning program Arm yourself with solid facts about code scanning solutions, cont o Our application team does not have time to setup and run a code scanning tool. Their schedule is full already. Investing some minimal time upfront will save time in the future Scans can be totally automated Scan setup is usually not that complicated Run in application build environment Process can be rolled into SDLC seamlessly

Tips for a successful scanning program Arm yourself with solid facts about code scanning solutions, cont o Easy to enforce company s existing coding practices and security policies without having to deviate from existing development lifecycle Finding security bugs can be done during normal build process Most code scanners allow for the creation of custom rules create a rule to look for existing coding practices that need enforcing Fixing security bugs can be rolled into normal bug remediation effort without having to stop the presses Testing and verification can occur during testing and rescans

Tips for a successful scanning program Learn how the tool works o Sounds obvious, however, many security teams fail to get a code scanner integrated because they just are not proficient enough with it Don t know how the scanner works and get frustrated Can t troubleshoot scan failures Don t know how to weed through the multitude of issues produced to identify most critical issues that need addressing

Tips for a successful scanning program Educate Application Owners o Armed with knowledge of the code scanner as well as facts about why you need a code scanner in your enterprise, inform app owners on what you are proposing Should get their buy-in as well Easy integration into development phase of SDLC Will find vulnerabilities during builds and help with remediation Facts stated previously o Demo actual exploits of common vulnerabilities to show how dangerous they can be XSS and SQLi demos are easy and effective

Tips for a successful scanning program Run a pilot program first o Will help gauge how successful a full rollout across all applications will be Will determine if major roadblocks exist that will prevent full rollout o Buy a few licenses upfront and purchase more in the future if needed

Tips for a successful scanning program Run a pilot program first, cont o Identify a small number of highly visible, externally-facing applications within your organization and target them Get app owner or PM buy-in If these apps are successful, will give you good ammunition for convincing management to do a full rollout Try to find applications of varying languages Want to test the code scanners ability against all of the types of applications run within your organization a) C++, C b) Java c) C#, ASP.NET d) PHP

Tips for a successful scanning program Develop a step-by-step guide for application teams to utilize when they are walking through code scanner integrations o This helps to make the integration process seamless and painless o If you can do most of the work upfront for a developer, they will be much more receptive

Code Scanning Integration Strategy Developed Web Site on company intranet portal o Identifies all steps needed from requesting code scanning integration to issue remediation help Steps we have broken the integration process into: 1. Introduction to Enterprise Code Scanning program Includes description of scanner 2. Request code scanner integration within application E-mail that is sent to us 3. Instructions on how to install code scanner on Build Server/Desktop 4. Signup application developer(s) to attend training Identify member from each development team who is responsible for all code scanning issues (security guru)

Code Scanning Integration Strategy Develop Web Site on company intranet portal o Identifies all steps needed from requesting code scanning integration to issue remediation help Steps we have broken the integration process into, cont : 5. Automate your code scans Automate the scans into existing build server 6. Scan code 7. Analyze scan results (validate issues) 8. Remediate issues 9. Submit feedback 10. Post Integration steps explained

Code Scanning Integration Strategy Develop Web Site on company intranet portal o Identifies all steps needed from requesting code scanning integration with the application to issue remediation help More on automating the code scans Types of code scanning integrations: Continuous integration build server Build server using autosys/cron-job Desktop Scheduler on Windows machine What happens to the scan results when scans are completed Automatically uploaded to a reporting management portal used for storing the results, generating reports and keeping track of key metrics Fortify 360 Server Rational AppScan Reporting Console

Code Scanning Integration Strategy When utilizing build server automation, develop scan templates (batch files or shell scripts) for the developers ahead of time o Allow them to download from company portal o Makes for much faster integration o Only a few details in scan script need to be replaced for scans to run successfully o Minimize the potential for scan failure

Tips for a successful scanning program Identify a member of your company s security team to be the code scanner guru o Needs a strong technical understanding of security concepts o Needs to be able to review the code scanner results and distinguish true positives from false positives Identify potential risk to the organization if issue is exploited o Ability to provide technical guidance to application teams Help understand and remediate security issues o Ability to present results to application teams and provide remediation support Help prioritize issue remediation and provide recommendations

Tips for a successful scanning program Develop a single scanning strategy and DO NOT allow for deviation o o o Identify how your organization would like to run your code scans and force all development teams to use this method Do not allow for one team to run their scans through build integration and another team to run ad hoc scans using their IDE code scanner plug-in Lack of continuity will make troubleshooting scan failures harder Will produce inconsistent results

Code Scanning Strategies - Pros and Cons Developer runs code scans from their IDE using the integrated plug-in o Typically, developers pulls latest code from version control repository into IDE and run scan o Pros o Cons Easy to run Usually different than what is in production Scanners are resource intensive Will hog up machine for long periods of time during the work day Developer needs to remember to run the scans and can only be done when the developer is using their workstation Cannot automate

Code Scanning Strategies - Pros and Cons Company Security Team runs scans for application teams from a dedicated machine o Pros o Cons Can build a powerful box that has a lot of hard drive space, memory and CPU power Need to pull latest code drop from version control repository Which version do you scan? Libraries/dependencies are not included in these code drops The proper libraries will be needed every time you run the scan. Can be very cumbersome Too much overhead to keep all scans accurate and running

Code Scanning Strategies - Pros and Cons Run code scans on application s existing build server and automate o Recommended approach o Pros Source is the most up-to-date and accurate to what is in production Code that is on the build server should be compile-able and therefore should generate the most accurate scans All libraries/dependencies should be present

Code Scanning Strategies - Pros and Cons Run code scans on application s existing build server and automate o Pros cont o Cons Least time consuming out of all the approaches mentioned Build server integrations are setup once Scans can be automated to run weekly/monthly and automatically uploaded to your scan repository Most cost effective Build servers tend to be beefier machines with high specs No need to acquire an additional machines to run the scans None identified to date

Tips for a successful scanning program Ensure scans are performed on a periodic basis o Helps determine if remediation is occurring o Does not have to been done during nightly builds o Ensure new issues are not being added to code o Tracks progress of remediation over time Good for management level reports o Ensure scans are actually being performed Developers will run the scan once and forget about it Setup windows desktop scheduler, autosys, or use a continuous integration server Send weekly reminders to developer in charge of scanning if not using build server integration

Tips for a successful scanning program Integrate security issues found by your code scanner into a bug tracking system o Bugzilla, JIRA, etc o Code scan issues become treated like any other bug found Hopefully given a higher priority o Can be resolved and closed during normal bug remediation process o Good place to track details of fix Can be shared across team/company o Make sure you analyze bugs first before checking into bug tracking system False positive will make there way in there otherwise

Tips for a successful scanning program Pay for Technical Support o Will help iron out kinks/snafu s that might arise o Do not solely rely on the manual Not written to give the user any more information than necessary to run against a simple application o May need custom rules to be written for unsupported libraries, policy enforcement, etc o Management usually wants custom reports Not the easiest to develop if you don t know what you are doing o Hiring a knowledgeable consultant to help with the integration goes a long way as well

Tips for a successful scanning program Sensitivity/Classification of source code that is scanned needs to be identified o Need to treat the machine that code is scanned on as having the same classification level as the source code Scanners leave translated versions of the source code on the scanning machine. Files need to be treated with same classification as the source code o Make sure the results file is given the same classification as well If code is SECRET, scan results need to be designated SECRET as well Code snippets, etc

Tips for a successful scanning program Send application teams feedback surveys o Feedback from the development teams can only help with making the code scanning program more successful

QUESTIONS? Adam Bixby abixby {at} gdssecurity.com