82-01-16 Certification of Externally Developed Software Craig A. Schiller Payoff Developers of large systems spend thousands of dollars ensuring that the software they create performs as expected, that no unauthorized changes have been introduced, that the individuals hired to develop the system are of good character and background, that the system complies with laws and regulations, and that security safeguards are present and functionally correct. However, many developers add such support programs as a data base management system or an operating system with little or no measures to ensure or determine that the preceding concerns have been addressed. This first in a series of two articles discusses an approach to gathering assurances to support the certification (for use in systems) of software developed outside of the organization's control. Problems Addressed For years, the software engineering community has used certification as a means of ensuring that large critical systems(usually government-related) are accurate, correct, and ready for operational use. Certification was used primarily to validate code that was developed locally. It was assumed that Commercial Off-The-Shelf software was not a threat to the system. As applications grew in size and complexity, pressure increased to reduce their time to market, and as the number of individuals involved in commercial software development increased, the number of errors and incidents of malicious code also increased. The problem is aggravated by an attitude called the shrink-wrapped syndrome that has been inherited from the non-programming world. It is usually assumed that a shrinkwrapped or sealed package is better than an open package. The food and drug industry is highly regulated to build confidence and to protect the public. Unfortunately, the message is so strong that it carries over into such unregulated industries as software development. The result is that usually cautious software engineers place unwarranted trust in shrinkwrapped software. When problems occur, the developer usually has little recourse because commercial off-the-shelf (COTS) rarely includes source code or warranties. If enough other users have experienced a similar problem, then the commercial off-the-shelf (COTS) developer may make an out-of-cycle fix available. Otherwise, if it is fixed, the developer must wait for the next official version release. The local software developer is ultimately responsible for the performance of the system being developed, regardless of whether the source of the problem is traced to developed code or a commercial off-the-shelf (COTS) product. Commercial Off-The-Shelf products represent a category of software that may be assumed trustworthy, but the issue exists for all externally developed software (i.e., software that was developed outside of the security professional's control). The difference between internally developed software and externally developed software is control of the development process, knowledge gathered during development, and the ability to perform detailed tests based on that knowledge. The controlled gathering of knowledge of an internally developed system is fundamental to the concept of certification. This article presents a body of collectable knowledge that may be used to guide the determination of the content and extent of tests to reduce the threat from externally developed software.
Certification Concepts Used on Developed Software Certification of software developed for government use is a well-documented process. The most recognized description of certification is in the Federal Information Processing Standard (FIPS)102 publication. The objective of the certification process is to ensure that a system is accurate and correct, that a system meets all applicable federal laws and directives, and that the system security safeguards work as intended. The certification process relies on a significant (but unspecified) development infrastructure. Software developers may respond in several different ways to meet these requirements. Most attempts to address certification include a formal software development life cycle that provides for the following: Clear, testable requirements. Traceability of requirements through specification, design, coding, testing, and integration phases. Internal controls to ensure that the system that is tested is the same system that was specified, designed, and coded. Test results demonstrating that the system met its requirements accurately and correctly, and tests of the security safeguards demonstrating that they work as intended. Although these processes directly address the requirements for certification, other less direct processes and controls are usually necessary. For example, Configuration Management processes are necessary to ensure that changes are not introduced without management and technical review. Subcontract management is necessary to ensure that subcontracted work meets requirements; this provides an avenue for recovery should the subcontractor performance fail to meet requirements. The assurances that software development processes provide to the technical management team making a certification recommendation to executive management should be considered. Executive management should solicit certification recommendations from the discipline experts or technical managers (e.g., certification for the physical environment from industrial security or the facility manager or system certification from the software project manager). Organizational Process Assurances Organizational process assurances monitor the product assurance processes for corruption. Common implementations of organizational processes include program management, audit, risk management, internal control systems, corrective actions systems, contract review, internal quality audits, and the use of statistical tools for metrics collection and use. For computer systems products, the acquiring organization may have little visibility into the organizational processes. If the systems developing organization and its processes are well-known (e.g., when the developer is another part of the organization), the product-using organization may be able to accept some of the risk for a reduced testing regime based on the knowledge of the developer's assurance and an examination of the testing results. Several indications may be drawn by the presence or absence of International Standards Organization 9000 certification (an international quality certification program)and whether the organization has been through a software capability evaluation (e.g., as a part of a government contract proposal). If the organization claims ISO 9000 (i.e., 9001, 9002, or 9003)certification, a copy of the scope statement should be obtained. The scope statement identifies the extent of the certification; the certification may include software development
activities. ISO 9000certification states that the organization performs the stated objectives for the processes covered in the scope statement. The certification process also provides for regular recertification to ensure that the quality implied by ISO 9000 is actively maintained. This certification does not mean that the company puts out a quality product, nor does it mean that the processes are necessarily productive. It does mean that the organization has processes and that it follows its documented processes. This implies that if a copy of the organizational assurance processes can be obtained from an ISO 9000 certified product developer whose scope statement includes software development, the developer may be following those processes. This adds a significant measure of evidence to support the decision made regarding the use and testing of the software product. For an independent evaluation of the effectiveness or maturity of processes, the security professional may be able to use the results of evaluations or assessments in relation to the Software Engineering Institute's (SEI) Capability Maturity Model (CMM). An Software Engineering Institute software capability evaluation is performed to support the selection of a contractor for some government contracts. An assessment is usually performed by the contractor to determine whether it meets thecmm standard. In general, most assessments have been found to be at least one level higher than the subsequently evaluated level. If a contractor claims to have been evaluated at SEI level 3, then the product user knows (if the evaluation can be substantiated) that the product developer at the time of the evaluation had a mature development organization that has organizational level standards and procedures for configuration management, QA, training, project management, subcontract management, and software development. Similarly, if the developer claims to have been assessed at SEI level 3, he or she is more likely to have project-oriented maturity with defined processes. Substantiating the results of an assessment can only be achieved by having the developer prove these findings. The developer may be unwilling to do that because it may pinpoint company weaknesses. This issue highlights a significant difference between ISO 9000certification and Software Engineering Institute evaluations and assessments. ISO 9000 results are intended to be shared with potential customers, and SEI evaluations and assessments are intended to be used only in relation to a specific contract proposal offering. Therefore, SEI evaluations represent only the state of the contractor in relation to the proposal that requires the evaluation. Although the criteria of the Software Engineering Institute evaluation and assessments are more useful to the user of a product in determining the details of key assurance processes, the static nature of the evaluations and assessments and the strong binding to specific contract proposals weakens the criteria s usefulness as a general assurance meter. However, if the system developer has ISO 9000certification and a previous SEI evaluation, the product user can confidently use the SEI evaluation to support decisions to use the product and reduce the extent of product testing with no perceived severe losses. In the case of products with potentially severe losses, another form of organizational assurance can be acquired through an external audit. This form of assurance requires the cooperation of the product developer; assurance is only possible if the nature of the product makes this a reasonable request. If little is known about the organizational assurances, the rigor and thoroughness of testing should be increased. Product Assurance Product assurance is the core of the quality task, with processes defined for each life cycle phase. Product assurance ensures that the product does what it is supposed to do, that the product does it as well as was expected, and that the product shipped is what it is supposed to be (e.g., all product components are present, it is not an older version, it is not a test version, and it does not contain a virus). Product assurance examples include life cycle reviews (e.g.,design review, inspections, and peer reviews) and such processes as
requirements management and configuration management. The results of these assurances are produced by the developers. The previous comments regarding ISO 9000 and SEI evaluations also apply to product assurance. Similarly, if the developer s product assurance measures are appropriate to the potential for loss, then the product user may be able to reduce the rigor and extent of product assurances, as long as the user can validate that the product received is the product tested by the developer. Traditional methods include the concept of bonded software (QA locking the certified product in a penetration-resistant case using a serial numbered metal band) or storage in a protected library. Newer technologies include the use of digital signatures or cryptographic checksums. Measures that are less expensive and less effective may be appropriate for Electronic Data Systems Corp products with less severe potential losses. These measures include site visits, sitting in on development meetings, discussions with the technical staff, and interviews. These assurances are more qualitative than quantitative, but they may add confidence or weight to other assurances. Personnel Assurances Personnel assurances include a comprehensive training program that provides instruction at the awareness level up to and including the cognitive level. To provide assurance, the training should be provided to meet clearly stated, testable objectives. To develop a comprehensive training program, the task and skills necessary to meet the organization s needs should be identified. The tasks and skills should also be expressed in terms of roles and responsibilities. Critical roles and responsibilities (including those that are critical during disaster recovery or emergency conditions) should be identified. Roles requiring special attention (e.g., separation of duty, position certification, or profession certification) should be identified. Clearances and personnel performance reviews also provide assurance. For securityrelated products with less severe potential losses, more credibility can be given to development teams lead by acertified Information Systems Security Professional (CISSP) or composed of personnel with other appropriate certifications. For systems products with severe potential losses, it is recommended that an official description of the development organization's roles, responsibilities, clearances required, internal controls guaranteeing separation of duty, and key position qualifications and credentials be identified. Financial Assurances Program management reviews address financial assurances. A company's inability to manage its finances can cause a development program to fail. Subcontract management is another facet of the financial assurances. The cost and schedule program data is used to project expenses to allow management to react to financial conditions and trends before they become program-threatening incidents. The source selection process may apply weight to such items as financial stability when making the selection. The loss of financial stability of either the prime contractor or its subcontractors could threaten the customer's ability to maintain the system after delivery, and it has the potential to reduce the quality of the product in response to financial pressures. If the intended use for the product does not require periodic updates or deficiency resolution from the developer, then financial assurances may not be necessary. If the product is easily replaced by another similar product, then financial assurance is unnecessary. However, if the product serves a critical function (e.g., availability), sensitive function (e.g., integrity), or if the developer provides a key management service to support confidentiality that could result in severe loss, then some financial assurances should be pursued, preferably before acquisition.
Some management aspects of financial assurance can be derived from the availability of SEI Capability Maturity Model evaluation results regarding project management and subcontract management. Other financial assurances can be gained by reviewing the developer's annual report, financial statements, stock ratings, and other indicators from Standard and Poor's, Dunn and Bradstreet, and court filings for bankruptcy. A knowledgeable CPA should be consulted for other available indicators of financial stability. Operations Assurances To preserve the quality delivered with the system, the operations community should have processes for receiving, handling, storing, installing, testing, and accrediting the system for operation. The operations community should conduct tests and inspections from an operations perspective to ensure that normal operations perform as expected. In addition, the operations community should develop, publish, and test plans for disaster recovery, continuity of operations, and end-user contingency planning. Physical Environment Assurances The physical environment of the operations and development sites can affect the safeguard selection process. A sensitive application, such as NASA's mission control, is usually located in a controlled access area (i.e., an area that is not accessible to the public). Additional safeguards required should be considered. A system's certification and accreditation to operate are only valid for a specific physical environment. For systems that are developed for general use or for many different user communities, the developers should clearly describe the physical environment assumptions and expectations used when developing the product. Logical Environment Assurances With logical environment assurances, the developer documents the environments in which the product is expected to be successfully executed. The logical environment includes restrictions (e.g., minimum memory required, maximum records supported, export restrictions), limitations (e.g., supports text files only, specific device requirements), devices tested on, devices not tested but expected to execute on, compatibility issues, and preferred, required, or alternative system parameter settings. Risk and Vulnerability Assurances It is not possible to protect a system against all potential threats and exploitable vulnerabilities. Managers use risk management to select the set of potential threats and vulnerabilities that should be addressed. Security professionals determine the value of assets by examining the nature of the asset and its actual or proposed use. This is to assess the value to the organization as well as the potential value to those who might exploit the organization, the existence of vulnerabilities, the frequency of each threat, the potential impact of each threat, and the exploited vulnerability. They may also determine and use a measure of the certainty of the data used and collected. From this information, security professionals may recommend to management a set of potential threats and vulnerabilities to address and a set of proposed safeguards. The contents of these two sets of data depends on the reliability of the data used to make the decisions. In the case of threat and vulnerability data, no certified sources of information are evident. Therefore, certifying the supporting data consists of documenting its source and selecting information to judge its credibility, currency, and accuracy.
External Interface Assurance Interfaces with other systems and organizations are a potential source of problems for system developers. To minimize the problems from external interfaces, developers create and maintain interface specification and interface control documents for rigorous interface control. Memorandums of agreement or understanding for less rigorous interface control are also created. Systems products may also have documentation or supporting code for developers that include interface definitions. This may be in the form of application programming interfaces or documents that describe file and field formats. Documented external interfaces may allow product users to conduct black box testing. External interface assurances may provide all forms of integrity, availability, and confidentiality assurance. Other forms of assurance may exist within an organization's development processes (e.g., maintainability, timeliness, responsiveness, documentation, useability, communications, and legality). Following the pattern used for the above assurances, an organization would develop equivalent measure that can be taken for systems products for each form of assurance that is unique to the organization (or otherwise missing from the preceding list). Applying Certification Concepts ISO 9000 describes a series of concerns that a manufacturer should address regarding the certification of materials acquired for use in the manufacturing process. In software, the corollary to acquired materials is externally developed software and data. As the International Standards Organization 9000standard recognizes, an organization cannot reasonably claim a product is certified unless all internally and externally developed components have been certified for a specified purpose. However, with planning it is possible to certify a product in stages. In the case of software, the product can be certified for general use within an organization(i.e., data has been gathered and some tests have been run and documented that do not have to be repeated when a department performs specific certification of the product). In some cases, general certification may cover all or most of a department's concerns. There may be no need for certification based on the general use of the product (e.g., a data base management system [DBMS]). However, if that DBMS is used to dispense drugs to patients, for example, the concerns for integrity, privacy, and availability fuel a need to certify the DBMS for this intended use. The security professional then determines the degree and type of certification to recommend by examining the general use of the product and the intended use of the product for the type and severity of potential loss. The following is a summary of the tasks essential to certifying systems-related products: Determining the type and degree of certification required for general use of the product. Determining the types of potential losses that could result from general use. Estimating the range of severity. Developing likely loss scenarios Determining the type and degree of certification required for the intended use of the product. Determining the types of potential losses that could result from the intended use. Estimating the range of severity.
Developing likely loss scenarios. Previous screen Gathering identification and assurance information about the Electronic Data Systems Corp product and developer.prescribing tests on the basis of the type of loss, the severity, likely loss scenarios, and the available assurance information.conducting tests.prescribing corrective action activities on the basis of test results, assurance data, potential types of losses, and potential severity.managing the proposed corrective actions for approval.executing the corrective action activities approved by technical management.preparing and delivering the certification briefing to management.changing the version designator for each product to indicate certification. Recommended Course of Action Software that has been developed outside of the organization's control can present certain concerns to the security practitioner. The certification concepts and different types of assurances contained in this article form a basis from which to evaluate this type of software. By reviewing the information provided, the security practitioner can ensure that certification for externally developed software follows the correct process. The second article in this series provides a checklist by which this software can be evaluated. The author would appreciate hearing about experiences in implementing this or related efforts regarding externally developed software. The author may be contacted at the following address and phone number: Craig A. Schiller 16018 Fathom Lane Houston, 77062 713.286.5351 Author Biographies Craig A. Schiller Craig A. Schiller is a senior analyst for PARANET, Inc. In addition, he is the cofounder of an automated information system security engineering team for NASA's Johnson Space Center, NASA's technology for information security conference, and the Texas Gulf Coast chapter of the ISSA.