ing Automated Notification of Errors in a Batch SAS Program Julie Kilburn, City of Hope, Duarte, CA Rebecca Ottesen, City of Hope, Duarte, CA

Transcription

1 ing Automated Notification of Errors in a Batch SAS Program Julie Kilburn, City of Hope, Duarte, CA Rebecca Ottesen, City of Hope, Duarte, CA ABSTRACT With multiple programmers contributing to a batch SAS program that runs nightly, we needed a way for the program to automatically notify the appropriate programmer if an error in the log occurred. Further we wanted to identify specifically what that error was and which contributing section of code caused the problem. This paper contains an example of one solution that uses sent from SAS and macro variables that you may not even know existed! INTRODUCTION The National Comprehensive Cancer Network (NCCN) Outcomes Database collects data on breast cancer, non- Hodgkin s lymphoma, non-small cell lung, ovarian, and colon/rectal cancers from NCCN member institutions and affiliated community sites across the United States. The Data Coordinating Center (DCC) of the NCCN Outcomes Database is located at the City of Hope and consists of a team of database programmers and statisticians lead by the DCC principle investigator. The primary responsibility of the DCC is to oversee database and web operations and data analysis for the project. For each disease, the DCC statisticians have developed algorithms to manipulate the data such as staging, drug therapy, and concordance; these programs are combined to produce reports that are available on the project s web application for project investigators and staff. SAS data is automatically generated and manipulated so that reports can be updated daily via a large batch program that runs automatically each night. The DCC statisticians can modify their algorithms and disease-specific portions of the programs that contribute to the large batch program. There is a constant need to verify that the multi-faceted program runs smoothly each night, but error checking can be tedious, and even easily forgotten because of the behind-the-scenes nature of batch SAS jobs. This paper illustrates a very simple way to keep our statisticians informed of the status of each contributing section of the batch program. First we capture the error information and tuck it away for safe-keeping using automatic macro variables and external text files. Next, we retrieve the information from the text files and analyze it. And finally, we send relevant information in an message from SAS to the program manager so she knows that all is well, or if not, where to begin searching to remedy the error. CAPTURE AND STORE ERROR INFORMATION We want to know if there is an error, which contributing program the error comes from, and what those errors are. There is a need to store the information outside of the SAS session so that it can be retrieved at any point, whether in the same SAS session or in a later one. AUTOMATIC MACRO VARIABLES Automatic macro variables (AMV) are created by the macro processor and supply a variety of information. They are useful in programs to check the status of a condition before executing code. Most automatic variables are global and are created when SAS is invoked. The values of AMVs are reset during each procedure or data step. We reference AMVs in the same way as with user-defined macros, with an ampersand (&) in front of the AMV name. We will use the following AMVs: SYSERR: SYSFILRC: SYSERRORTEXT: SYSWARNINGTEXT: numeric: used to detect major system errors, such as out of memory or failure of the component system when used in some procedures and DATA steps. A value of 0 indicates no error, any other number indicates some type of error. numeric: value of 0 indicates that the file referenced by the most recent FILENAME statement exists. character: contains the text of the last error message formatted for display in the SAS log character: contains the text of the last warning message formatted for display in the SAS log 1

2 SYSLAST: character: contains the name of the SAS data file created most recently. IDENTIFYING ERRORS MACRO Our macro, which will capture the desired information into text files, has three parameters, listed below. Within the macro we reference a macro variable tdata refers to the path to the directory where we would like to store the files we will be creating, and which we will assign at a later time. We use AMV SYSERR to determine if an error has occurred prior to the macro call. If no error exists, ie. &syserr=0, then we write none and the contributing section name to a simple text file. If an error is found, ie. &syserr^=0, then we write error and the contributing section name to the text file. We also create another separate text file with detailed error messages, using AMVs SYSERRORTEXT, SYSWARNINGTEXT, and SYSLAST. /* Fn = name of the text file we will create to contain whether and */ /* where the error exists. */ /* Fndetail = name of a separate text file for the detailed error message */ /* where the error an error. */ /* Contrib = specific name of the contributing section of code */ EXAMPLE %macro iderror(fn,fndetail,contrib); filename em "&tdata\&fn..txt"; /*if there is no error */ %if &syserr=0 %then %do; data _null_; file em; put "none &contrib"; %end; /*of no error section */ /*if there is an error */ %else %if &syserr^=0 %then %do; data _null_; file em; put "error &contrib"; /* separate text file for details */ filename detail "&tdata\&fndetail..txt"; data _null_; file detail; put "Last Error Message: &syserrortext"; put "Last Warning Message: &syswarningtext"; put "Last Dataset Created: &syslast"; %end; /*of error section */ %mend; Using the simple dataset class.sas that is available in the SASHELP files, we will generate some simple code to illustrate the use of the macro IDERROR. After each section of code that we want to check, we place a call to the macro, which gives unique names to the macro parameters for each check. libname w14 'c:\wuss14\data\'; %let tdata=c:\wuss14; /*section XXX: run a proc print */ proc print data=w14.class; title 'A Simple Data Set'; var name sex age height weight; 2

3 %iderror(er1,er1detail,xxx); /*section YYY: manipulate same data, output a frequency*/ data playmore; set w14.class; if name<"n"; proc sort data=playmore; by age; proc freq data=playmore; title 'A Simple Frequency'; table age; %iderror(er2,er2detail,yyy); /*section ZZZ: manipulate more, output something to a file*/ filename W14out 'c:\wuss14\wuss14out.txt'; proc format; value $ gender M = Male F = Female ; data wordgender; set w14.class; format sex gender.; file W14out /* <-- intentionally leave off a semicolon to produce an error*/ put name "'s gender is " sex; %iderror(er3,er3detail,zzz); Running this sample code will generate four text files: one for each call to IDERROR (Figures 1-3) with the section name and whether there is an error or not. These we will call error flag files. Because only one section has errors, there is only one detailed error message text file (Figure 4). These files are permanently stored, and will be accessible even after the current SAS session if they need to be. Figure 1. Text File er1 Figure 2. Text File er2 Figure 3. Text File er3 Figure 4. Text File er3detail 3

4 USING STORED ERROR INFORMATION We now need to retrieve our stored information. We can do this immediately following the original program, or in a separate SAS session. We will create a permanent dataset with variables to hold the error message (msg), the contributing section (cont), and the details of the error(s) (col1-col3). We want one observation per call of the iderror macro. Libname w14 c:\wuss14\data\ ; data w14.all_error; length msg $ 5 cont $ 4 col1-col3 $ 150; delete; The dataset starts out empty, and will be added to with each pass of the macro readtxt below. This macro also has three parameters, which coincide with those from the previous macro. The macro first reads in the error flag file using simple list input. Then if the error detail file exists, the information is read from this file as well. Note the use of the AMV SYSFILRC to determine if this detail file exists. Because we need to merge the message and detail datasets, we create a dummy dataset with empty variables if the detail file does not exist. The merged dataset contains all five variables for one of the contributing sections. We use a PROC APPEND to add this row of data to others with each call of the macro. %macro readtxt(ifn,ifnd,contrib); /* read from the error flag file */ Filename in_e "&tdata\&ifn..txt"; data messages; infile in_e; input msg $ cont $; /* read from the detail file if error exists */ filename in_d "&tdata\&ifnd..txt"; %if &sysfilrc=0 %then %do; data detail; length t $ 150; infile in_d; input t $ ; /* Need these in one row instead of three rows */ proc transpose data=detail out=detailrow; var t; data detailrow; set detailrow (drop=_name_); %end; /* if no error exists then just create an empty dataset for merging below */ %else %do; data detailrow; length col1-col3 $ 150; %end; data messages_all; merge messages 4

5 detailrow; proc append base=w14.all_error data=messages_all force; %mend; For our example, we call this macro three times, one for each section of code that we checked earlier. We need to use the same file names and contributing section names that were created in the first macro. %readtxt(er1,er1detail,'xxx'); %readtxt(er2,er2detail,'yyy'); %readtxt(er3,er3detail,'zzz'); The SAS dataset is displayed below as it appears in the table view from the explorer window. Figure 5. Dataset w14.all_error THE ERROR INFORMATION We can now easily set up an message that contains this information. For illustration purposes, we are using the most basic programming to from SAS. A FILENAME statement is required. We send the to the intended recipient, with an appropriate subject line. We will report on every contributing section, indicating the section name and whether an error has occurred. For the section(s) with errors, we give the detailed messages as well. filename errmail ; data _null_; set w14.all_error; file errmail to = "jkilburn@coh.org" subject = "Nightly Program ERROR CHECK" ; if msg='none' then do; put 'No Errors in the ' cont 'section'; end; else if msg='error' then do; put 'ERRORS OCCURRED in the ' cont 'section'; put 'ERROR/WARNING messages below: '; put col1; put col2; put col3; end; 5

6 Figure 6 below displays the received. Figure 6. SAS-Generated CONCLUSION As with most programming adventures, there is generally more than one way to approach the problem of knowing when and where to look for errors using a minimum amount of effort. This paper has illustrated a simple method that any SAS programmer can use, whether a master or a beginner. CONTACT INFORMATION Your comments and questions are valued and encouraged. Contact the author at: Name: Julie Kilburn Enterprise: City of Hope Phone: jkilburn@coh.org SAS and all other SAS Institute Inc. product or service names are registered trademarks or trademarks of SAS Institute Inc. in the USA and other countries. indicates USA registration. Other brand and product names are trademarks of their respective companies. 6