Macro Tabulating Missing Values, Leveraging SAS PROC CONTENTS Adam Chow, Health Economics Resource Center (HERC) VA Palo Alto Health Care System Department of Veterans Affairs (Menlo Park, CA) Abstract Metadata, in a broad context, is data about data. Executing a SAS PROC CONTENTS is a form of metadata. When you run a PROC CONTENTS on a SAS data set, the results tell you various attributes of the data set. Moreover, it details each variable in the data set, such as the name, the variable type (numeric or character), the variable label that makes the name s mnemonic a little bit more meaningful, and the variable format that instructs the SAS System on how the values are to be displayed externally. As insightful as PROC CONTENTS is, however, it does not tell you anything about missing values. Accordingly, by leveraging the results from PROC CONTENTS, the %CHK_MISSING macro described in this paper, tabulates missing values for each variable in a data set and reports the count and the corresponding percentage in a separate data set. This paper will: 1) summarize the macro specification 2) examine the design methodology and syntax and 3) demonstrate how it is used with an illustration. The %CHK_MISSING macro was developed and tested under Base SAS 9.1.3 Service Pack 4 in UNIX, SUN operating system 5.10 platform. Introduction Missing data is not uncommon in SAS data sets. A common technique to quickly count missing values in a data set is to directly employ PROC MEANS with the NMISS option. However, this method operates on numeric variables only. As an alternative, one simple way to count both numeric and character missing is to apply the NMISS aggregate function in PROC SQL, but this solution requires writing an aggregate NMISS function for every variable. If you have a data set with tens, hundreds, or even thousands of variables, doing this is very tedious and time consuming. Furthermore, this process is likely to produce mistakes resulting from mistyping or inadvertently skipping over certain variables. In view of this, the %CHK_MISSING macro eliminates this manual process by systematically leveraging information from a single PROC CONTENTS. PROC SQL creates a macro variable storing a list of variable names, followed by another macro variable holding the count of the variable names. With these two pieces of information, augmented with the NMISS summary function, code is dynamically generated in a %DO %UNTIL loop within another PROC SQL. Following execution, the tabulated result is reshaped and merged back to the original PROC CONTENTS. With a data step, the percent missing is calculated, followed by another data step to consolidate and cleanup the information in preparation for external representation. The final product is a SAS data set that looks similar to a PROC CONTENTS found in the SAS Output Window, but also includes the count and percentage missing for each variable. Macro Specification The entire SAS code for the %CHK_MISSING macro is in the Appendix. The full macro specification is listed as follows: %chk_missing(indsn=, outdsn= ) %CHK_MISSING can be called from within any executing SAS code. As presented in this paper, it is a keywords macro and needs just two parameters. Their meaning and usage are briefly explained below: 1) indsn (required): the input SAS data set name 2) outdsn (required): the output SAS data set name 1
Design Methodology And Syntax The %CHK_MISSING macro is separated into 8 major parts, of which only the first 5 will be discussed. The last three parts are not as critical, as parts 6 and 7 deal with data cleanup in preparation for another macro call if needed, while part 8 contains several ordinary macro labels placed right before the %MEND statement which stops macro execution. Part 1 %if %length(&indsn) = 0 %then %do %goto ENDIT1 %end %if %length(&outdsn) = 0 %then %do %goto ENDIT1 %end Part 1 quickly checks that two required parameters are entered into the macro. If a parameter is missing when %CHK_MISSING is executed, the %GOTO statement branches to macro label %ENDIT1 where an error message is written to the SAS Log before exiting the macro. Part 2 proc contents data=&indsn out= ac1 noprint run Part 2 is the most important piece of the macro where a PROC CONTENTS is deployed to find attributes of a data set. The NOPRINT option tells SAS not to print the results to the Output Window. Printing is suppressed because it is an extraneous step since most of the relevant information is already contained in data set AC1. Each row in AC1 represents a variable found in the macro s input data set (first macro parameter). The columns, of which there are 40, detail the structure of the input data set (e.g., number of observations). They also tell you the characteristics of each variable, such as the name, variable type (numeric or character), label that allows for a more descriptive verbiage than is possible with just the variable name, and format that instructs SAS on how the values are to be displayed externally. Part 3 proc sql noprint select name into :_v_list_ separated by '~' from ac1 select count(*) into :_numobs_ from ac1 quit %put _V_LIST_ = &_v_list_ %put _NUMOBS_ = &_numobs_ The above PROC SQL creates two macro variables. Macro variable _V_LIST_ is a list comprised of the values of an entire column in data set AC1. Basically, this is a text string of all the variable names in the input data set (first macro parameter). The variable names are separated by the tilde (~) which functions as a text delimiter. As such, the second macro variable _NUMOBS_ simply represents the count of the number of variable names. The %PUT statements immediately following PROC SQL writes the stored values of the two macro variables to the SAS LOG. This code is included so that the list can be visually inspected and validated. If the list is too long, the count simply tells you how many data elements are found on the list. 2
Part 4 proc sql create table ac2 as select %let i =1 %do %until(& i > &_numobs_) %if & i = &_numobs_ %then %do nmiss(%scan(&_v_list_,& i,%str(~))) as %scan(&_v_list_,& i,%str(~)) %end %else %do nmiss(%scan(&_v_list_,& i,%str(~))) as %scan(&_v_list_,& i,%str(~)), %end %let i = %eval(& i +1) %end from &indsn quit proc transpose data= ac2 out = ac3 (rename=(col1=miss_count)) name=vars run proc sql create table ac4 as select a.varnum as varnum label="#", a.name as name label="variable", a.type as type, a.length as length label="len", a.format as format label="format", a.formatl, a.formatd, a.label as label label="label", a.nobs, b.miss_count as miss_count from ac1 as a left join ac3 as b on a.name eq b.vars order by varnum quit Tabulating missing values is accomplished in first PROC SQL of this section. Using the macro function %SCAN within a %DO %UNTIL loop, an aggregating NMISS function is dynamically written for each data element on _V_LIST_. The loop stops when index i exceeds the value stored in _NUMOBS_. The result is a one observation data set AC2 where each column represents a variable name holding the count of the missing values. The NMISS aggregate function in PROC SQL counts missing values. Most of the summary statistics that PROC SQL calculates are numeric. However, NMISS is one of the few aggregating functions that can accept either numeric or character type variables. Next, a PROC TRANSPOSE reshapes the one observation data set, from a 1xN to Nx2 data set, where N=_NUMOBS_. In the reorganized data set, AC3, the first column is called VARS which contains all the variable names. The second column, MISS_COUNT, holds all the corresponding missing values counts. The main purpose of reshaping is to prepare data set AC3 to be merged with data set AC1 which was created in Part 2. 3
The joining of the data is controlled in the second PROC SQL. Although the original PROC CONTENTS has 40 columns, only a handful are really meaningful: VARNUM showing the variable s column position, NAME, TYPE, LENGTH, three formatting constructs displaying name/length/decimal, LABEL, and NOBS which shows the observation count of the input data set INDEMO. Data set AC4, sorted by VARNUM, is simply a matrix that has PROC CONTENTS information from AC1, as well as missing values counts from AC3. Part 5 data ac5 set ac4 miss_pct=miss_count/nobs format miss_count comma20. miss_pct percent8.3 label miss_count="obs Missing Count" miss_pct ="Obs Missing Pct(%)" run data &outdsn format varnum name type_ set ac5 length type_ $4 if type eq 1 then type_="num" else if type eq 2 then type_="char" if formatl ne 0 and formatd ne 0 then do format=strip(format) strip(put(formatl,3.)) "." strip(put(formatd,3.)) end else if formatl ne 0 then format=strip(format) strip(put(formatl,3.)) "." else if format ne "" then format=strip(format) "." label type_="type" drop type formatl formatd run The NOBS variable generated from PROC CONTENTS in Part 2 indicates the number of observations in a data set. Using NOBS as the denominator, calculation of the percent missing is performed in the first DATA step in Part 5. The subsequent DATA step cleans the data before outputting it as a final data set. The main purpose of the second DATA step is to make data set AC5 look like SAS Output when PROC CONTENTS is executed without the NOPRINT option, but with the VARNUM option included. The two right most columns have the raw counts of the missing values and the corresponding percentages. The column variable type in AC5 has numeric values of 1 or 2. This tells you the variable is either numeric or character, respectively. A new variable, TYPE_, is created to serve as a proxy to display Num or Char. Also, the three columns storing formatting information are consolidated into one. The STRIP function is used to remove leading and trailing blanks. Illustration The following illustrates the usage of the %CHK_MISSING macro. To simplify the demonstration, the example input data set INDEMO has 10 observations and 7 variables. The desired output data set is given the name OUTDEMO. Therefore, the %CHK_MISSING macro is submitted with the two parameters as follows: %chk_missing(indsn=indemo, outdsn=outdemo) 4
Figure 1 details the SAS View Table of input data set INDEMO. After invoking %CHK_MISSING with the two parameters, the output data set Figure 1 (Data set INDEMO: SAS View Table Parameter # 1 of Macro) OUTDEMO is shown in Figure 2. Notice the 7 variables in data set INDEMO correspond to the 7 observations in data set OUTDEMO. Patient ID which is a character variable, has no missing values, and thus, reports 0% missing, while ICD9 #2 which has 7 missing values, shows 70% missing. The Amount, which is a numeric variable, has one missing value, and therefore, displays 10% missing. Figure 2 (Data set OUTDEMO: SAS View Table Parameter # 2 of Macro) The %CHK_MISSING macro creates temporary data sets. A PROC CONTENTS is performed on input data set INDEMO. The OUT= option puts the results into data set AC1, which has 40 columns. Figure 3 below shows the abbreviated SAS View Table with some of the columns displayed. Notice that it is sorted by Variable Name, not by the Variable Number. This is the default ordering and is not an issue since it will be re-ordered later in PROC SQL when the missing counts are joined to this data set. Figure 3 (Data set AC1 : Has 40 Columns But Not All Columns Are Shown) 5
Using the information from PROC CONTENTS, two macro variables are created in PROC SQL. Both macro variables, including the stored values, are written to the SAS Log. _V_LIST_ = amt~cpt1~cpt2~dos~dx1~dx2~pat_id _NUMOBS_ = 7 Tabulating missing values is Figure 4 (Data set AC2 ) achieved with another PROC SQL. Within a %DO %UNTIL loop, the macro function %SCAN extracts a segment of the list holding the variables names. In each cycle through the loop, an aggregate NMISS function is written for each variable name. As such, Figure 4 shows the results in AC2, a one observation data set with 7 columns. Each column stores the missing values count for each variable in data set INDEMO. Figure 5 (Data set AC3 ) Minimizing SAS Log Output By Using SAS System Options The one observation in Figure 4 is immediately reshaped into a 7x2 data set using PROC TRANSPOSE. Figure 5 displays this outcome in AC3. The primary reason for reshaping is to have the count of missing values integrated with the original PROC CONTENTS represented by AC1. After merging data sets AC1 and AC3 together to build data set AC4 (as explained in Part 4), the percentage missing is calculated in a DATA step that produces data set AC5. The last action item is consolidating the format information before outputting the final data set called OUTDEMO. Depending on your SAS System Option setup, a lot of information can be written to your SAS Log when executing a macro containing %DO loops and %IF logic. If your preference is to reduce the amount of output, there are system options you can use, such as NOSYMBOLGEN, NOMLOGIC, and NOMPRINT. For example, using the last illustration, you can add the following OPTIONS statement before invoking the %CHK_MISSING macro. OPTIONS NOSYMBOLGEN NOMLOGIC NOMPRINT %chk_missing(indsn=indemo, outdsn=outdemo) NOSYMBOLGEN suppresses the display results as the macro variable references are resolved. NOMLOGIC, on the other hand, tells the macro processor not to trace macro execution with respect to parameter values and the TRUE/FALSE conditions behind the %IF logic. As the Macro language is just a mechanism to store and synthesize code, the NOMPRINT option instructs the Macro facility not to write SAS statements during a macro execution. Conclusion In this paper, the different components of the %CHK_MISSING macro syntax were examined. Moreover, the illustrations visually demonstrated how missing values are calculated and subsequently reported as an output SAS data set. This macro takes advantage of the PROC CONTENTS output. Two macro variables are created using PROC SQL: a list storing the variable names and the count. With these two pieces of information, NMISS aggregate functions are dynamically generated in a %DO %UNTIL loop for each variable inside a PROC SQL. Within each cycle of the loop, syntax is systematically written using the %SCAN macro function coupled with the list. Compared to manually typing individual NMISS summary functions in a text editor such as the SAS Program Window, using the %CHK_MISSING macro saves time and eliminates potential input errors. 6
Key Words PROC CONTENTS, PROC SQL Macro Variable, PROC SQL List, PROC SQL NMISS Aggregate Function, %DO %UNTIL, %SCAN, %GOTO, STRIP Function, NOSYMBOLGEN, NOMLOGIC, NOMPRINT, Metadata. Acknowledgments I would like to extend special thanks to the following person and organization: Colleague Jennifer Scott (MS) of HERC for proofreading and critiquing of this paper. HERC (Health Economics Resource Center) for its full support by giving me the opportunity to write this paper. Trademarks SAS and all other SAS Institute Inc. product or service names are registered trademarks or trademarks of SAS Institute Inc. in the USA and other countries. indicates USA registration. Other brand and product names are registered trademarks or trademarks of their respective companies. About The Author The author welcomes your comments, suggestions, and questions. You can contact the author at: Adam Chow Health Economic Resource Center VA Palo Alto Health Care System 795 Willow Road (152 MPD) Menlo Park, CA 94025 (650) 493-5000 x22061 adam.chow@va.gov achow94109sf@yahoo.com 7
Appendix %macro chk_missing(indsn = /* Input SAS data set name */, outdsn = /* Output SAS data set name */ ) /* 1. quick check of the 2 parameters entered */ %if %length(&indsn) = 0 %then %do %goto ENDIT1 %end %if %length(&outdsn) = 0 %then %do %goto ENDIT1 %end /* 2. PROC CONTENTS of input SAS data set */ proc contents data=&indsn out= ac1 noprint run /* 3. make a list of the var names, also show # of vars */ proc sql noprint select name into :_v_list_ separated by '~' from ac1 select count(*) into :_numobs_ from ac1 quit /* print out to LOG */ %put _V_LIST_ = &_v_list_ %put _NUMOBS_ = &_numobs_ /* 4. summarize, transpose, and merge back with AC1 */ proc sql create table ac2 as select %let i =1 %do %until(& i > &_numobs_) %if & i = &_numobs_ %then %do nmiss(%scan(&_v_list_,& i,%str(~))) as %scan(&_v_list_,& i,%str(~)) %end %else %do nmiss(%scan(&_v_list_,& i,%str(~))) as %scan(&_v_list_,& i,%str(~)), %end %let i = %eval(& i +1) %end from &indsn quit proc transpose data= ac2 out = ac3 (rename=(col1=miss_count)) name=vars run proc sql create table ac4 as select a.varnum as varnum label="#", a.name as name label="variable", a.type as type, a.length as length label="len", a.format as format label="format", a.formatl, a.formatd, 8
a.label as label label="label", a.nobs, b.miss_count as miss_count from ac1 as a left join ac3 as b on a.name eq b.vars order by varnum quit /* 5. calculate % missing, then output a data set */ data ac5 set ac4 miss_pct=miss_count/nobs format miss_count comma20. miss_pct percent8.3 label miss_count="obs Missing Count" miss_pct ="Obs Missing Pct(%)" run data &outdsn format varnum name type_ set ac5 length type_ $4 if type eq 1 then type_="num" else if type eq 2 then type_="char" if formatl ne 0 and formatd ne 0 then do format=strip(format) strip(put(formatl,3.)) "." strip(put(formatd,3.)) end else if formatl ne 0 then format=strip(format) strip(put(formatl,3.)) "." else if format ne "" then format=strip(format) "." label type_="type" drop type formatl formatd run /* 6. delete temporary data sets */ proc datasets nolist delete ac1 ac2 ac3 ac4 ac5 run quit /* 7. reset the macro variables */ %let _v_list_ = EMPTY LIST %let _numobs_ = 0 /* 8. Exit */ %goto ENDIT2 %ENDIT1: %PUT ERROR: MACRO STOPPED - MUST ENTER ALL PARAMETERS %ENDIT2: %mend chk_missing 9