Best Practice in SAS programs validation. A Case Study CROS NT srl Contract Research Organisation Clinical Data Management Statistics Dr. Paolo Morelli, CEO Dr. Luca Girardello, SAS programmer
AGENDA Introduction Program Verification: a Business Approach Program Verification: some case studies
Headquarters in Verona (Italy) Founded in 1993 Offices in Milan and Munich 40 employees FACTS about CROS NT Data Management, Statistical, PhV and hosting services Services to Pharma, Biotech and CROs Cooperation with Universities of Padua, Bologna, Milan
Introduction Topic of the presentation: how to maximize the quality of programming while minimizing the time to verify program. In the first part of the presentation we will discuss about the business part: What is program verification? Why program verification is necessary? When is program verification done? Who performs program verification? How does the verification process work? In the second part of the presentation we will discuss about a case study
What is program verification Making certain that the program does what it is supposed to do, producing a documented evidence of this
Why program verification is necessary The aim of SAS validation in pharmaceutical research area is that end-users will produce high quality programs that fit the purpose for which they are designed and provide accurate results with a style that they promote: Reliabity Efficiency Portability Flexibility Ease of use
When is program verification done Program verification should performed as soon after the development of the SAS code, before putting the product in production Development and production environment should be clearly defined; Audit trail of program changes should be present as soon the program is released to production
Who performs program verification The SAS programmer who create the code should perform basic testing and follow coding rules, like: Error log search Warning evaluation Comments on critical steps Comments on Macro usage Details of the SAS program (datetime of creation, SAS programmer name, dataset used, datetime of verification, Name of second SAS programmer, etc) It should be emphasized to perform then a program verification by a second SAS programmer
How does the verification process work Biostatistician creates specs then Submits request Interactive Process SAS developer produces TLGs Then submits verification request Interactive Process Quality Control programmer verifies results
Different Verification Procedures SOP should define different verification procedures. Independent programming Reviewing results Random review of results Visually verify code Some of them should mandatory, other optional. The Document Containing the programming specs (for example the SAP) should define which approach to follow, illustrating program verification techniques (for example using alternative SAS programming procedures) The determination of the level of validation should follow a risk-based model. The key is to determine the effect on the process if the program does not produce the desired result.
Error Types Business strategy should identify common error types found in: Statistical tables Listings Graphs Data analysis files Header section of SAS programs Bad programming specifications Metric report related to error type should be analyzed in order to perform preventive action correction
Specific CDISC SDTM Validation specs Metadata Level Verifies that all required variables are present in the dataset Reports as an error any variables in the dataset that are not defined in the domain Reports a warning for any expected domain variables which are not in the dataset
Specific CDISC SDTM Validation specs - Metadata Level Notes any permitted domain variables which are not in the dataset Verifies that all domain variables are of the expected data type and proper length Detects any domain variables which are assigned a controlled terminology specification by the domain and do not have a format assigned to them
SAS Programming Rules when validating Emphasizing well commented programs. Macro in order to use programs repeatedly to verify different programs (re-usability) Using alternative SAS programming procedures when validating. Define a workflow if error are identified
How to optimize the process Good specs & Good standards & Good training = Good programming results
A Case Study
Example of Derived Datasets Validation (1/4) First Programmer programs all derived datasets Second Programmer programs all derived datasets PROC COMPARE Compare original derived datasets versus validation derived datasets
Example of Derived Datasets Validation (2/4) proc compare base=listing compare=validation listbase listcomp; id pt; run; The COMPARE Procedure Comparison of WORK.LISTING with WORK.VALIDATION (Method=EXACT) Observation Summary Observation Base Compare ID First Obs 1 1 pt=121 First Unequal 79 79 pt=201 Last Unequal 79 79 pt=201 Last Obs 89 89 pt=212 Number of Observations in Common: 89. Total Number of Observations Read from WORK.LISTING: 89. Total Number of Observations Read from WORK.VALIDATION: 89. Number of Observations with Some Compared Variables Unequal: 1. Number of Observations with All Compared Variables Equal: 88.
Example of Derived Datasets Validation (3/4) Values Comparison Summary Number of Variables Compared with All Observations Equal: 3. Number of Variables Compared with Some Observations Unequal: 1. Total Number of Values which Compare Unequal: 1. Maximum Difference: 1. Variables with Unequal Values Variable Type Len Label Ndif MaxDif age NUM 8 AGE (years) 1 1.000 Value Comparison Results for Variables AGE (years) Base Compare pt age age Diff. % Diff 201 41 40-1.0000-2.4390
Example of Derived Datasets Validation (4/4) The COMPARE Procedure Comparison of WORK.LISTING with WORK.VALIDATION (Method=EXACT) Observation Summary Observation Base Compare ID First Obs 1 1 pt=121 Last Obs 89 89 pt=212 Number of Observations in Common: 89. Total Number of Observations Read from WORK.LISTING: 89. Total Number of Observations Read from WORK.VALIDATION: 89. Number of Observations with Some Compared Variables Unequal: 0. Number of Observations with All Compared Variables Equal: 89. NOTE: No unequal values were found. All values compared are exactly equal.
Example of Tables Validation (1/3) First Programmer programs all tables applying the set of layout specifications and saves outputs in Word Second Programmer programs all tables avoiding to add additional SAS code to control output Compare of outputs
Example of Tables Validation (2/3) Tmt A Tmt B First Programmer - Output in Word Age (years) n 41 48 Mean (SD) 51.44 (10.39) 52.10 (11.00) Median 55.00 55.00 Min - Max 30.00-66.00 27.00-71.00 Gender Female 14 (34.15%) 21 (43.75%) Male 27 (65.85%) 27 (56.25%) Second programmer - Output SAS proc means data=demog n mean stddev median min max; var age; by tmt; run;
Example of of Tables Validation (3/3) Tmt A Tmt B First Programmer - Output in Word Age (years) n 41 48 Mean (SD) 51.44 (10.39) 52.10 (11.00) Median 55.00 55.00 Min - Max 30.00-66.00 27.00-71.00 Gender Female 14 (34.15%) 21 (43.75%) Male 27 (65.85%) 27 (56.25%) Second programmer - Output SAS proc freq data=demog; tables gender*tmt; run;
Example of Listings Validation (1/2) First Programmer programs all listings applying the set of layout specifications and saves outputs in Word Second Programmer prints derived datasets in SAS Compare listing output in Word versus output in SAS of derived dataset
Example of Listings Validation (2/2) Listing Output in Word Print of Derived Dataset Listing 1 Demographic Characteristics Subject ID Gender Age Race 121 M 50 3 122 M 34 3 123 F 58 3 124 M 64 3 125 M 57 3 126 F 64 3 127 M 39 3 128 M 55 2 129 M 41 3 130 M 44 3 131 M 32 3 132 M 37 3 133 M 61 3 134 F 56 3 135 M 34 3 136 M 34 3
Example of Registration Errors
Metrics on Programming Errors Specification not detailed 40% Output Structure 30% Wrong interpretation of specification 60% Specification 14% Display Variables 14% Output Writing 56% Calculation of variables 20% Layout 45% Selection of Variables 14% SAS Programming 66% Programming 41%
Examples of Errors Layout Writing of a note in table Incorrect: Percentagesare calculatednumberofpatients Correct: Percentagesare calculatedon number ofpatients
Examples of Errors Programming data age; set demog; if age<20 then age_c=1; else if 20<age<40 then age_c=2; else if age>=40 then age_c=3; run; data age; set demog; if age<20 then age_c=1; else if 20<=age<40 then age_c=2; else if age>=40 then age_c=3; run;
Examples of Errors Wrong interpretationofspecification Note of a table(in SAP): Note 1: Onlypatients withallvalueforprimary analysis are included in the table. In SAS Program: In the table, allpatientsare included
Thank you for your attention Questions?