Defining a Validation Process for End-user (Data Manager / Statisticians) SAS Programs Andy Lawton, Boehringer Ingelheim UK Ltd., Berkshire, England INTRODUCTION The requirements for validating end-user SAS programs are not defined by regulatory bodies. These programs are used for many purposes in clinical trials from data query generation to reporting. We can infer from 21 CFR Part 11 (electronic records) that validation is required, so what should a pharmaceutical / biotech company does to address this? Many companies address this by undertaking a QC as a final process but this is not sufficient. The use of system validation methods, which are well defined and widely accepted, would not meet our objectives (see below) on these end-user programs. The aim of validation in this area is that endusers will produce high quality programs that fit the purpose for which they are designed and provide accurate results with a style that promotes reliability, efficiency, portability, flexibility and ease of use. All of this has to be achieved without creating a significant increase in costs, as we must remember that the primary goal of validation is to meet a business need. This paper examines the methods introduced by BI to meet the demands of validation. DEFINITIONS We have defined the end-user program (not system) validation as establishing documented evidence which provides a high degree of assurance that a specific process will consistently produce a product meeting its predetermined specification and quality attributes. To create a structure in which we can validate programs requires that we define a number of items: Responsibilities - two key roles were defined for different aspects of validation: Owner Programmer Classification Method for Programs - define types of programs an what level of validation will be applied: Core Programs User Programs Ad-hoc Programs Define a Validation Process for End-user Programs - Program Development Life Cycle: The PDLC consists of four stages: 1. Requirements / Specification 2. Program Development 3. Testing 4. Maintenance and Documentation The type of SAS end-user programs covered are as follows Programs for producing trial-specific Data Quality listings, tables and graphs Programs to create analysis datasets Programs for the trial support (statistical analysis, subject data listings) including programs creating in-text and end-of-text tables and figures Programs for project / trial specific evaluations (e.g. interim or meta analysis, integrated summary of efficacy and integrated summary of safety programs) Programs distributed at the Project level by statisticians / Data Managers for trial level evaluations Programs for publications Programs for any external or internal presentation Programs for official internal use PROGRAM DEVELOPMENT LIFE CYCLE (PDLC) FOR SAS PROGRAMS
It is essential to have SOPs and Guidelines covering the processes and forms to document the process. The processes that need to be applied are summarized below in the Program Development Life Cycle. This has been adapted from the System Development Life Cycle. The following is a brief description of each step of the Program Development Life Cycle. These steps have to be documented and the form that we use within BI - the Program Validation Document - is the key to the whole process. 1. REQUIREMENTS/SPECIFICATION This is the key driver for the validation process, and the area most feared by so many end-user SAS programmers - their commitment is often "we know what we have to do so why do we have to document it". We sympathize with this view as we as an industry already provide copious documentation for everything that we do. Our process does not require that the end-user generate any more documentation then they have already completed. Out process states that the following must exist: Statistical Analysis Plan / Data Management Plan - providing documentation on the specification of individual tables, figures and listings required in a clinical trial General Guidelines on the Format and Layout of Output for BI - This prevents the need for continual repetition of the basic "look and feel" of output (e.g. titles, footers, column headers, etc.) The overall objective of this first stage is that the objectives of the program must be clear and agreed to, by both the owner and the programmer. The source of requested program has to be identified as it determines the Program Validation requirements. The source could be one of the following: use of an existing validated program - no Program Validation requirements exist amendment to an existing validated program - change control procedures have to be followed and re-testing might be necessary (see Step 3 of the PDLC) development of a new program - all 4 steps of the PDLC are required 2. PROGRAM DEVELOPMENT The second step of the Program Development Life Cycle cannot be started before the requirements / specification phase is completed and the resulting documentation is available. During the program development of end-user programs, good programming practices should be followed (a guideline / SOP must be available to address this). These practices optimize programming in efficiency, quality and understanding / readability. Program tools such as ASAP from ComplementSoft can aid in two ways. Using the template library can help in the standardization of code. Creating and outline shell to produce a flowchart from ASAP can aid in the further development of complex programs (e.g., analysis datasets). Together with the program, if necessary, the programmer has to provide documentation to the user such as: a copy of the output (listing, table, report or dataset) instructions how to use the programs (e.g. parameters to be passed to a program) This documentation only has to be generated if it cannot be ensured that the user understands and uses the program in the intended way. That means that the documentation to the user may vary depending on the experience / knowledge of the user and could be just notes / comments in the program itself. 3. TESTING Testing based on the requirements must be undertaken and documented. There are several tests that have to be performed and some that are optional which are only appropriate in some specific circumstances. The tests with their requirements are listed below.
Programs that are developed for more general distribution, such as Project level programs to be used in multiple trials, must be tested to a higher standard as the risk is higher. This may involve more extensive black-box testing or a white-box review. White-box testing is required for any programs that handle data for the primary endpoint. It is the owner's responsibility to assess the risk. 3.1 WHITE-BOX White-box testing involves the source code and the implementation details. It can include examination of the programming standards, style and control methods, source language, database design, and the like. White-box testing will be conducted only by a programmer or technical expert who is familiar with the computer language. It is important that white-box testing is an integral part of program development procedures. White-box testing can be accomplished using different methods. The two most robust are independent review by another programmer or using an interview technique. Using the interview technique - the author will explain line by line to another programmer / reviewer - the purpose of each line / statement. By using this method, errors are easier to find due to the interaction. It is also not as time consuming because the reviewing programmer does not have to take time to become familiar with unknown code. The following points need to be addressed during white-box review: 1. Look at the specifications and requirements for the program (e.g., Statistical Analysis Plan) 2. Examine the structure of the program (e.g. Flowchart) 3. Review the Program Log observations input to a data step observations created by a data step 4. Review the code program logic programming standards / style To create the flow chart of the program, tools should be available to aid in the source code review. An example of such a software tool is ASAP from ComplementSoft which is used for analysis of SAS programs and macros. 3.2 BLACK-BOX Black-box testing is conducted independently of the source code. This type of testing matches actual output against predicted output. Typically the test steps and the predicted result are predefined in test cases. The test cases / inputs and the development of test data are required when testing the core programs. For the Program Validation of user programs it would unreasonable to predefine and approved expected results for each and every user program. Therefore it is essential to document and predefine the expected tests and the results in a guideline / SOP so that this can act as a formal predefinition of the expected results by defining it in the respective application specific section (See Appendix 1). 4. MAINTENANCE The maintenance of a program includes the documentation, distribution, storage / archival, change management and version control. Documentation of every one of the previous Program Development Life Cycle steps should be complete and available. This is essential for core programs. Determine who is responsible for the distribution of the program and any updates. The programs must be stored in a defined area which are on a qualified / validated storage medium. Programs that have passed this testing phase and the Program Validation Document has been approved by both the owner and the programmer must not be overwritten any more. Any amendment must follow the necessary change control procedure. It is important to control changes to a program in order to ensure that it continues to function correctly. Changes to a program may be proposed either to correct an error or to enhance functionality or performance. For core programs systems / procedures must exists to track errors (bugs) and fixes. It is every user's responsibility to report any problems / errors of a program or even the application itself to the owner of the
program. At a minimum the following information has to be provided: program affected date of observation name of observer description of the problem / bug impact (severity, urgency, etc.) Any changes made to a program must be validated. The extent to which this is fulfilled depends on the extent ot the changes made. Version control is essential for tracking all changes made to the programs and associated documentation to provide a complete history of the program and its various versions, as well as to ensure that the version of the program in use at any given time can be uniquely identified and controlled. As mentioned previously, programs that have passed the testing must not be overwritten. The internal documentation (either within the program code or in the title) will indicate the version and will be distinct. ACKNOWLEDGEMENTS The SAS system is a registered trademark of the SAS Institute Inc., Cary, NC, USA. ComplementSoft ASAP is a registered trademark of ComplementSoft LLC, USA. S*Plus is a registered trademark of Insightful Corporation, Seattle, WA, USA. CONTACT ADDRESS Andy Lawton Clinical Data Management Department Boehringer Ingelheim Limited (UK) Ellesfield Avenue, Bracknell, Berkshire RG12 8YS UK lawtona@bra.boehringer-ingelheim.com CONCLUSION The validation of end-user SAS programs does require extra work, but if you are already testing your programs to an adequate level, and meeting requirements for Statistical Analysis Plans then the costs do not have to be too excessive. The use of programming tools such as ASAP can not only aid in the program development, via its template facility, but also it can prove to be invaluable to the review of code using the flow charts and in program review facilities.
APPENDIX 1 The table is an example of test methods that contains aspects of testing which are either mandatory or optional; however, at least one of the optional methods has to be chosen. The tester must assess which options are appropriate for the purpose of the program. The involved persons (e.g., owner, lead statistician, independent statistician or programmer) have to agree on the testing method and have to document this by singing the Program Validation Document. Aspects of Testing Description of Testing Expected / Acceptable Result Summary Statistics - Option 2 (SS2) Summary Statistics - Option 3 (SS3) The output from the program is compared with the output of the same data from a different system. For example, if a SAS program is used to create mean values by subgroups, then the output is compared to mean values by subgroups from a different statistical package (e.g., S*Plus ) The output from the program is compared with the data created manually. For example, if PROC TABLUATE is used to create mean values by subgroups, then the raw data is manually tabulated and the mean values are compared. The summary statistics created by the different systems match each other. The summary statistics created by the computer procedure and the manual method match. Mandatory (M) or Optional (O) O O