Managing Software Risk at the System Level. Making code quality work for the business

Transcription

1 Managing Software Risk at the System Level Making code quality work for the business

2 Architectural complexity cause business disruptions Architecturally Complex Violations are structural flaws involving interactions among components that reside in different application layers. Although they constitute only 8% of the vulnerabilities in an application, they represent: 52% of the repair effort and require 20 times more changes to fix! 8 times more likely to escape into testing and 6 times more likely to escape into operations. SYSTEM LEVEL FLAWS 8 % 52 % 90 % Downtime caused by system-level flaws! Effective Software Risk Prevention: Focus on critical violations that matter Focus resources on area of highest impact not highest volume 92 % of all defects 48 % of total repair effort 10 % UNIT LEVEL FLAWS Tracking programming practices at the Unit Level alone may not translate into the anticipated business impact, most devastating defects can only be detected at the System Level. - OMG

3 The goal of system quality Control the software characteristics that have the greatest impact on the business! Software Characteristic RELIABILITY PERFORMANCE EFFICIENCY SECURITY MAINTAINABILITY Rules/Best-Practices No error handling along the call chain Typecast mismatching along the call chain Mis-configured frameworks (e.g., update trigger in Hibernate) Expensive loops, with indirect calls involved Incorrect use of indices Multiple performance violations along the call chain Input validation, SQL injection, Cross-site scripting Failure to use vetted libraries or frameworks Secure architecture design compliance Excessive horizontal layers Code duplication business logic vs. exact code comparison Strict hierarchy of calling between architectural layers

4 SAM solution must match the business need Enterprise-wide solutions measure system level structural characteristics such as Reliability, Performance Efficiency, Security and Maintainability Developer-centric tools monitor code quality of individual components at unit level, these are language-specific and narrowly focused.

5 Architecture Compliance System & unit level analysis is a MUST for real insight System / Application Value Web Services APIs Java COBOL JSP Java ASP.NET C++ C# VB Integration quality Architectural compliance Risk propagation simulation Application security Resiliency checks Transaction integrity Automated Function Point / EFP Counting Effort estimation Data access control SDK versioning Calibration across technologies Risk Management Reliability Security Performance Productivity Measurement Vendor Management IT Measurement & Governance Hibernate Messaging Spring Struts.NET Module Value Oracle EJB PL/SQL SQL Server T/SQL COBOL Intra-technology architecture Intra-layer dependencies Module complexity & cohesion Design & structure Inter-program invocation Security Vulnerabilities Rules Compliance Defect Reduction Sybase DB2 IMS Program/Unit Transaction Risk Data Flow Code style & layout Propagation Risk Code documentation Class or program design Basic coding standards Value Comments Hygiene 4

6 Comparison of enterprise & developer-centric SAM solutions Developer-Centric tools are focused on code hygiene Research shows that 90% of defects found in production and 60% of defects found in QA are related to cross component or cross technology interactions which require Enterprise solutions to detect! SDLC Stage Type of Rules CAST AIP (JEE/RDBMS) Open Source at IDE 4 Results per KLOC Structural Defects 3 % in QA % in PROD Component level code hygiene 25 ~ s Development Component level best practices 109 ~ s 40% 10% Integration & System Testing System Level - Framework compliance - Architectural checks - Cross technology - Security - Transaction s 60% 90% 1) Majority of these are related to XML which don t result in any defects CAST Application Intelligence Platform 2) Naming conventions and basic code hygiene don t necessarily result in defects, may make it hard to maintain and change 3) Based on a study of defect profiles by CAST Research Labs from ten major corporations worldwide 4) This analysis adds all the checks in the popular tools PMD, Check Style, FindBugs, and JDepend

7 CAST AIP vs. code analyzer CAST Application Intelligence Platform Code Analyzers System level analysis Unit/program level analysis Analysis occurs at build or integration phase Analysis occurs locally at developer workstation Ensures architectural, structural and programming best practices Ensures basic programming best practices Prevention of serious reliability, performance, security issues Prevents TCO growth over time by proactive preventing the system from becoming brittle, complex, undocumented, and architectural unsound to modify Holistic view of system that pinpoints critical defect while imparting software engineering advice Checks for good code hygiene, readability and cleanliness Prevents error prone programming, complexity of unit algorithmic complexity at unit level..etc Code level view that flags poor coding hygiene and non-conformance to coding standards 6

8 Only CAST s systems level analysis can Provide deep architectural visibility through cross layer / technology analysis Analyze systems composed of multiple technologies: SAP,.NET, Business Objects, C/C++, COBOL, Borland Delphi, Java, PowerBuilder, IBM SQL-PSM, Oracle Forms, Oracle PL/SQL, J2EE Frameworks Struts, Hibernate, JavaScript, HTML,.ASP, Sybase T-SQL, T-SQL, Microsoft SQL Server, Tibco BusinessWorks, Visual Basic, PeopleSoft, Siebel, PL1, Fortran, Flex, RPG, EGL Detect critical vulnerabilities that will disrupt business critical systems and processes. Generate actionable metrics and remediation plans that make development teams better and systems more reliable. Aggregation & Consolidation Static Analysis Dynamic Analysis Function Points Transaction Finder System Level Analysis Dependencies Code Pattern Scanning Rule Engine Architecture Checker Data Flow

9 CAST AIP holistic system analysis Tech coverage: Support all technologies in the application context Behavior Simulation - emulate run-time behaviors of components Dependencies - analyze cross-layer / technology links between app components Code Pattern Scanning - scan pattern and anti-pattern in application control flow Content Updater - adjust analysis results to match application advanced behaviors Data Flow Analysis - track along static and dynamic call stacks the use of the content of variables Architecture Checking - identify invalid calls and references between architectural Layers Transaction Finder - identify cross-layer / technology transactions from UI to data entities Static Analysis Aggregation & Consolidation Dynamic Analysis Function Points Transaction Finder Rule Engine System Level Analysis Data Flow Dependencies Code Pattern Scanning Architecture Checker 8

10 "The Whole Is Greater Than The Sum Of Its Parts A CASE STUDY

11 Global insurance company situation Global Insurance company with +150k employees and USD 30 billion in revenues attempted to build an in-house system level software analysis and measurement solution. The goal Combat system reliability issues and increasing costs in critical systems. Management visibility and control of technical quality of in house and service providers IT Ecosystem Oracle, Peoplesoft and Java, VB,.NET, C, C++,C# applications, Multiple systems integrators: Oracle, Capgemini, Accenture Solution Develop in-house analysis and measurement capability with open source tools (Jdepend, PMD, CheckStyle, FxCop and commercial tools (DevPartner (Performance) and McCabe (Metrics). 10

12 Global insurance company result After 3 years, 4 in-house developers and systems integrator s support staff deployed a solution at a cost of $1.5M. Challenges Delivered a partial solution Lacked ability to analyze SQL, no inter-component analysis Unable to aggregate analysis at portfolio level and no diagnostics Partially adopted by organization because the solution Didn t cover all systems in the portfolio Provide an incomplete view of systems Unable to support decisions No training, documentation of deployed solution Partial funded only funded development Requires substantial resource and financial investment 4 FTEs to fix bugs and adopt technology evolution Unable to upgrade new version of open source tools No further R&D and evolution of the product. 11

13 Global insurance company hidden cost of free software Licensing In-house solution using open source* Free Open source based aggregator solution (Sonar)* $780,000 (2 instance of base product + 2 instances of plug ins for 5 years) Maintenance Free Included in base fee Administration $500,000 (1 FTE for 5 years) $500,000 (1 FTE for 5 years) Hardware Budget Development/ Integration $100,000 (Dev and production environments) $1,000,000 (2 FTEs for 5 years) $50,000 (Production Environment) $250,000 (.5 FTEs for 5 years) 5 Year Cost $1.5M $1.6M Costs not consider by the client Identify needs, research and investigate Open Source solutions Select analyzers and develop tools to integrate with OS analyzers Build and maintain dashboards Test and certify deployments to production Apply patches for bug fixes Maintaining the integration of all parsers into one dashboard Create custom rules Calibrate data across parsers Track of development/ roadmap of each parser Create training material and conducting training for various stake holders Ongoing support for developers and administrators 12

14 Global insurance company lessons learned Home grown efforts become another development project Difficult to contain costs, schedule, scope creep Developer level tools cobbled together does not result in system level capability or support enterprise level decision making. Grossly underestimated Total Ownership Cost Open source Licensing can create licensing infringement risks Brain Drain Organization spends time and money to train in-house to become experts, resources leave and take the knowledge with them Unreliable Technical Support Open source development is not accountable to respond to urgent technical issues (security breaches) Unpredictable development of future components (new analyzers, upgrades) 13

15 Implementing a Code Quality Management initiative HOW TO BEGIN

16 The recognized leader for SAM in IT for many years WE KNOW HOW TO INTRODUCE CODE QUALITY MANAGEMENT INTO LOW MATURITY ORGANIZATIONS Prioritize Identify Hotspots Automate Stabilize Measure Educate Optimize Eliminate Run an initial measurement across the affected applications. Find where the biggest structural weaknesses and architectural hotspots are in these applications. Baseline the key applications along the most important application health parameters. Ensure they don t deteriorate, especially with respect to critical violations. Implement a continuous measure of structural quality, with a direct feedback loop to developers. Track asset improvement and identify risk as early as possible in SDLC. Key Success Factor Prioritize output to only relevant software flaws, by: - Propagation risk - Transaction risk - Critical violation density Key Success Factor Focus efforts on the businessrelevant characteristics - Stability & resilience - Performance efficiency - Security & software risk Key Success Factor Rely on measurement platform that remains consistent - Over time, for trending - Across apps portfolio - With industry standards

17 CAST approach to code quality management in SDLC Senior architect for program oversight Leverage the architect office to scale reuse, frameworks, and best practice across teams Data aggregation & measurement Stabilize measurement model for inclusion into IT dashboards, vendor SLAs, and team tracking Build-stage Global Analysis Use CAST for rules that require system-level context Use Unit-level or CAST for all other rules Unit-level Local Online Analysis Analyze hygiene and simple best practice rules at the IDE, for direct feedback that doesn t require context 16

18 Don t wait for an outage to manage software risk The time to repair the roof is when the sun is shining. -John F. Kennedy 17