Foundations and Fundamentals

Transcription

1 How to Monitor Production and Development Processing using the SAS Logging Facility Curtis E. Reid, U.S. Bureau of Labor Statistics, Washington, DC Abstract An innovative diagnostic feature was introduced in the release of SAS 9.2 called the SAS Logging Facility ( SLF ). The SLF provides a way to categorize and filter log messages from the server and the programs and to redirect them to a variety of output destinations. This paper shares how one can use the SLF during the execution of production jobs, in the development and maintenance of programs, and the associated benefits. The intended audience members for this paper are entry-level to intermediate SAS developers and system administrators who want to learn how to better support SAS applications. Introduction An innovative diagnostic feature was introduced in the release of SAS 9.2 called the SAS Logging Facility ( SLF ). It is a configurable logging framework for SAS on both the server and programming environments. Traditional SAS logs display the program processing details including notes, warnings and error messages. The SLF provides a way to categorize and filter log messages from the server and the programs and to redirect them to a variety of output destinations. This paper demonstrates how one can use the SLF during the execution of production jobs, and how to use it in the development and maintenance of programs; it will also suggest the benefits that can be derived by investing the time to do so. For the purpose of this paper, I will simplify the definition and usage of an SLF. I will not focus on configuration or the XML. This paper is intended for entry-level to intermediate SAS developers and systems administrators, and shares the information in ways that will enable them to become productive in a short period of time. To go beyond the scope of this intent, please see the References section at the end of this paper. SAS Logging Facility Concepts and Terminology All server and programming environments create events such as INFO:, WARNING:, etc. SLF is used to collect these events and write them out to a variety of output devices such as an external log file. Then these events can be categorized and filtered in support of diagnostic analyses. The main difference between SLF and traditional logging is that SLF provides much more detail than a traditional log output message such as the date and time and specific event. Specific events can be categorized and filtered based on some specific threshold. There are specific concepts and terminology that are associated with SLF that will require explanation. It is best understood if the concepts and terminology are listed in the order that SLF is used (rather than alphabetical order). SLF is invoked in one or both of two ways, via the server side and/or the application side. A server side invocation can be easily thought of as invoking the SAS application via the Windows Start Menu or from a command line by adding a system parameter -LOGCONFIGLOC. An application invocation is performed after start up using autocall macros or functions in SAS code. Please note that invocation for the server or application side is dependent on your specific platform. For this paper, Windows is used as the platform.

2 This is a sample invocation for the server side: From Windows, click START -> RUN -> cmd. This opens a command window. A sample run command: "C:\Program Files\SASHome\SASFoundation\9.3\sas.exe" -CONFIG "C:\Program Files\SASHome\SASFoundation\9.3\nls\en\sasv9.cfg" - LOGCONFIGLOC "basic:warn,fileappender,c:\temp\nesug2012\mylogs\logfacility.%s{hostna me}.log" This is a sample invocation for the application side: Open your SAS 9.3 in the usual way. Use the autocall macros %LOG4SAS, log4sas function, or declare logger object constructor. Specific instructions on invocation will be explained later in the paper. Below is a list of terminology: logger It is a named entity that defines a hierarchal level of the logging system for a message category. Root is the highest level logger. appender It represents a named output destination for messages. log event It is an event that is reported by SAS. level A diagnostic level associated with a log event. The levels from lowest to highest are: TRACE, DEBUG, INFO, WARN, ERROR and FATAL. message category It is a classification of messages produced by SAS. threshold A threshold is the lowest event level that is processed. Log events produced by levels that are below the threshold are ignored. filter A filter is a set of character strings or thresholds that you specify. Log events produced are compared to the filter to determine whether the events should be included or not. pattern layout A pattern layout is a template that allows you to format your messages for the output destination. Monitoring Production Processing One of the biggest issues with Production systems is that changes should not be introduced into production without going through an appropriate software life cycle process. Fortunately, there is an easy solution to this. You can monitor production processing using the server side of the SLF. As explained earlier, SLF is invoked by adding the parameter to your SAS invocation command line: -LOGCONFIGLOC "basic:warn,fileappender,c:\temp\nesug2012\mylogs\logfacility.%s{hostna me}.log This statement requests basic logging information without creating a logging configuration file; a specified level of WARN threshold logging output is written to a log file stored in the path c:\temp\nesug2012\mylogs with a file name of logfacility lt-didd.log. The %S{hostname} portion is called a S Conversion Character. The S represents system information based on a specific keyword such as hostname, etc. A detailed listing of these keywords can be found in the online documentation titled SAS 9.3 Logging: Configuring and Programming Reference. 2

3 After you ve run your production programs, open your log file and you will find detailed log events such as: T13:05:04,733 WARN [ ] - WARNING: SUMWGT was requested but no WEIGHT variable was specified. Weight of 1 will be assumed for each observation. As you can see, it provided the date, time, log event level, and thread identifier of the log event, username, and the message. Already, it has provided useful information to both the developer and the system administrator for diagnostic purposes. Notice that the log event entry is different from the WARNING: message that appears in a traditional log file. Monitoring Development Processing During the development process, you will have an opportunity to add code to your programs to take advantage of the SLF. The commonly understood concepts of SAS Macro Language and SAS Language are the focus of this paper. Beginning with SAS version 9.2, the OPTIONS MAUTOSOURCE is set by default. If you have a program that explicitly turns off MAUTOSOURCE, it needs to be turned on for this to work. Example #1 Program 1 filename mylog "c:\temp\nesug2012\mylogs\example1.log"; 2 %log4sas; 3 %log4sas_appender(myappender, "FileRefAppender", 'fileref=mylog'); 4 %log4sas_logger(mylogger, 'level=info appender-ref=(myappender)'); 5 %log4sas_info(mylogger, 'Executing chap21_means.sas'); 6 %include "c:\temp\nesug2012\chap21_means.sas"; 7 %log4sas_info(mylogger, 'Finished chap21_means.sas'); In line 1, a FILENAME reference needs to be created to capture the results from the SLF to an external file MYLOG. In line 2, the autocall macro %LOG4SAS must always be called the first time to establish the SLF within an application environment. In line 3, an appender named MYAPPENDER is needed to reference the output destination to MYLOG. In line 4, a logging environment is named as MYLOGGER which points to the appender MYAPPENDER. In line 5, a logging event INFO is created with a message Executing chap21_means.sas to MYLOGGER. In line 6, an included SAS program chap21_means.sas is called. In line 7, another logging event INFO with the message Finished chap21_means.sas is created. Opening the log file example1.log reveals: Executing chap21_means.sas Finished chap21_means.sas That is all that was written out to the log file from SLF. This is not very useful but this is a start. Let s continue to expand on this with the next example. 3

4 Example #2 Program 1 filename mylog "c:\temp\nesug2012\mylogs\example1.log"; 2 %log4sas; 3 %log4sas_appender(myappender, "FileRefAppender", 'fileref=mylog pattern="%d %-5p (%S{jobid}:%S{user_name}) %m"'); 4 %log4sas_logger(mylogger, 'level=info appender-ref=(myappender)'); 5 %log4sas_info(mylogger, 'Executing chap21_means.sas'); 6 %include "c:\temp\nesug2012\chap21_means.sas"; 7 %log4sas_info(mylogger, 'Finished chap21_means.sas'); The only difference between example #1 and example #2 is found in line 3 where a pattern is added to MYAPPENDER. As defined earlier, a pattern is a template layout for the message. There are four patterns which are %d, %-5p, (%F:%L) and %m. %d provides the datetime in ISO8601 format. %-5p provides the logging event level reported. If the level is less than five characters, it is padded on the right side. (%S{jobid}:%S{user_name})logs the job or process id and the username. Parenthesis and the colon are literal characters. Finally, %m provides the message that was reported to SLF for that log event. The following entries in the example2.log reveal: T12:54:11,279 INFO (4048:Reid_C@PSB) Executing chap21_means.sas T12:54:12,357 INFO (4048:Reid_C@PSB) Finished chap21_means.sas As you can see, it provided much more information than the first example. You can use this information to analyze how long a macro or program takes to run or to easily identify a specific logging event. In the third example (example3.sas), a SAS function is used within a data step. The full code of this example is attached at the end of this paper. I will just highlight some examples that are used to create a logging event for SLF. Create a logger and appender during the first iteration of the data step (_N_ = 1) as follows: rc=log4sas_appender("functionappender", "FileRefAppender", "fileref=nesug"); rc=log4sas_logger("functionlogger", "appender-ref=(functionappender) level=info"); rc=log4sas_logevent("functionlogger", "info", "Obtained today's date."); rc=log4sas_logevent("functionlogger", "info", "Determined the number of business days."); The first one creates an appender called functionappender. The second one creates a logger called functionlogger that references functionappender. Two logging events are written out to document that today s date and the number of business days were obtained. Now, for each iteration from the second observation going forward, three logging events/functions are created. These three functions are: rc=log4sas_logevent("functionlogger", "info", "Found date differences."); rc=log4sas_logevent("functionlogger", "info", "Made adjustments in days."); 4

5 rc=log4sas_logevent("functionlogger", "info", "Made adjustments in months."); Since I didn t specify any patterns, the logger showed the following entries: Obtained today's date. Determined the number of business days. Made adjustments in days. Made adjustments in months. Made adjustments in days. Made adjustments in days. Made adjustments in days. Made adjustments in days. Made adjustments in days. Finally, the fourth example (example4.sas) demonstrates the use of both INFO: and ERROR: messages using the SLF. This example attempts to create a libname reference named MYLIB which is saved into a non-existent directory. It generated ERROR: messages in both the logger and the traditional log files. The logger showed the following entries: T13:08:19,063 INFO (5256:Reid_C@PSB) Started example4.sas T13:08:19,142 ERROR (5256:Reid_C@PSB) Unable to create libname ref MYLIB T13:08:19,188 ERROR (5256:Reid_C@PSB) Unable to create dataset MYLIB.MYTEMP T13:08:19,204 INFO (5256:Reid_C@PSB) Finished example4.sas The traditional SAS log showed the following entries: 1 /* example4.sas */ 2 3 filename mylog "c:\temp\nesug2012\mylogs\example4.log"; 4 5 %log4sas; 6 %log4sas_appender(myappender, "FileRefAppender", 'fileref=mylog pattern="%d %-5p 6! (%S{jobid}:%S{user_name}) %m"'); 7 %log4sas_logger(mylogger, 'level=info appender-ref=(myappender)'); 8 9 %log4sas_info(mylogger, 'Started example4.sas'); NOTE: Started example4.sas %macro example4; 12 /* Notice that I do not have a subdirectory named nesug2013 */ 13 libname mylib "c:\temp\nesug2013"; 14 %if (&syslibrc ne 0) %then %do; 5

6 15 %log4sas_error(mylogger, 'Unable to create libname ref MYLIB'); 16 % 17 %else %do; 18 %log4sas_info(mylogger, 'Created a libname reference MYLIB'); 19 % data mylib.mytemp; 22 message = 'MYTEMP dataset created'; %if (&syserr ne 0) %then %do; 25 %log4sas_error(mylogger, 'Unable to create dataset MYLIB.MYTEMP'); 26 % 27 %else %do; 28 %log4sas_info(mylogger, 'Created a permanent dataset MYLIB.MYTEMP'); 29 % 30 %mend example4; 31 %example4; NOTE: Library MYLIB does not exist. ERROR: Unable to create libname ref MYLIB ERROR: Library MYLIB does not exist. NOTE: The SAS System stopped processing this step because of errors. NOTE: DATA statement used (Total process time): real time 0.01 seconds cpu time 0.01 seconds Conclusion ERROR: Unable to create dataset MYLIB.MYTEMP %log4sas_info(mylogger, 'Finished example4.sas'); NOTE: Finished example4.sas One of the biggest benefits of using the SLF is that it is very easy to analyze your logs based on logging event type such as INFO, WARN, or ERROR. Also, it provides additional information that is not otherwise available in a traditional log like the date time stamp, which allows you to track real time and other performance metrics. Invoking the SLF via a systems option at the start time creates a server side SLF. This allows you to easily track Production processing without any code modification to your production code. You can create some metrics based on the information in the SLF log file quickly and easily. Invoking the SLF using an autocall macro or SAS function allows you to further customize your SLF in development mode such as tracing the calls using messages and logging events. This paper only touches on the basic concepts of SLF; there are many other features that are available in SLF, such as the XML configuration file, in-filtering, rolling logging and much more. Getting beyond the basics is harder if the developer does not understand the concepts and terminology first. If you practice and play around with SLF, it will be easier to understand. The author hopes this paper provided a quick primer for you to get started using SLF. 6

7 References SAS 9.3 Logging: Configuration and Programming Reference, Sample Codes used for example1.sas and example2.sas, SAS 9.3 Logging: Configuration and Programming Reference, dmsf07o6d33.htm Sample Code used for function_example.sas, SAS 9.3 Logging: Configuration and Programming Reference, xpzdxddchhz.htm Sample Codes for Chapter 21 Means, Edited by Curtis E. Reid, Acknowledgments Racine Bell, Bryan Beverly, Steven Holmes, Jeannine Mercurio and Eric M. Winslow, U.S. Bureau of Labor Statistics, for assistance with proofreading. Jurgen Kropf and Eric M. Winslow, U.S. Bureau of Labor Statistics, for their support in this paper. SAS and all other SAS Institute Inc. product or service names are registered trademarks or trademarks of SAS Institute Inc. in the USA and other countries. Indicates USA registration. Other brand and product names are registered trademarks or trademarks of their respective companies. Contact Information Your comments and questions are valued and encouraged. Contact the author at: Curtis E. Reid, Information Technology Specialist Office of Employment and Unemployment Statistics Division of Industry Data Development PSB Massachusetts Avenue, N.E. Washington, D.C reid.curtis@bls.gov 7

8 filename mylog "c:\temp\nesug2012\mylogs\example1.log"; /* example1.sas */ %log4sas; %log4sas_appender(myappender, "FileRefAppender", 'fileref=mylog'); %log4sas_logger(mylogger, 'level=info appender-ref=(myappender)'); %log4sas_info(mylogger, 'Executing chap21_means.sas'); %include "c:\temp\nesug2012\chap21_means.sas"; %log4sas_info(mylogger, 'Finished chap21_means.sas'); filename mylog "c:\temp\nesug2012\mylogs\example2.log"; /* example2.sas */ %log4sas; %log4sas_appender(myappender, "FileRefAppender", 'fileref=mylog pattern="%d %-5p (%S{jobid}:%S{user_name}) %m"'); %log4sas_logger(mylogger, 'level=info appender-ref=(myappender)'); %log4sas_info(mylogger, 'Executing chap21_means.sas'); %include "c:\temp\nesug2012\chap21_means.sas"; %log4sas_info(mylogger, 'Finished chap21_means.sas'); filename nesug "c:\temp\nesug2012\mylogs\example3.log"; data a; dob mmddyy10.; format dob tod mmddyy10.; /* example3.sas */ /* In the first iteration of the DATA step, create an appender */ /* and a logger, and initialize variables tod and bdays. Then, determine */ /* the number of days in the month prior to the current month. */ if _n_ = 1 then do; rc=log4sas_appender("functionappender", "FileRefAppender", "fileref=nesug"); if rc ne 0 then do; msg = sysmsg(); put msg; ABORT; rc=log4sas_logger("functionlogger", "appender-ref=(functionappender) level=info"); if rc ne 0 then do; msg = sysmsg(); put msg; 8

9 ABORT; /* Get the current date from the operating system */ tod=today(); retain tod; rc=log4sas_logevent("functionlogger", "info", "Obtained today's date."); if rc ne 0 then do; msg = sysmsg(); put msg; ABORT; /* Determine the number of days in the month prior to current month */ bdays=day(intnx('month',tod,0)-1); retain bdays; rc=log4sas_logevent("functionlogger", "info", "Determined the number of business days."); if rc ne 0 then do; msg = sysmsg(); put msg; ABORT; /* end the processing for first iteration */ /* Find the difference in days, months, and years between */ /* start and end dates */ dd=day(tod)-day(dob); mm=month(tod)-month(dob); yy=year(tod)-year(dob); rc=log4sas_logevent("functionlogger", "info", ""); if rc ne 0 then do; msg = sysmsg(); put msg; ABORT; /* If the difference in days is a negative value, add the number */ /* of days in the previous month and reduce the number of months */ /* by 1. */ if dd < 0 then do; dd=bdays+dd; mm=mm-1; rc=log4sas_logevent("functionlogger", "info", "Made adjustments in days."); if rc ne 0 then do; msg = sysmsg(); put msg; ABORT; /* If the difference in months is a negative number add 12 */ /* to the month count and reduce the year count by 1. */ if mm < 0 then do; mm=mm+12; yy=yy-1; rc=log4sas_logevent("functionlogger", "info", "Made adjustments in months."); if rc ne 0 then do; 9

10 msg = sysmsg(); put msg; ABORT; datalines; 01/01/ /28/ /03/ /28/ /29/ /01/ /10/ /11/ /12/1974 ; proc print label; label dob='date of Birth' tod="today's Date" dd='difference in Days' mm= 'Difference in Months' yy='difference in Years'; var dob tod yy mm dd; filename mylog "c:\temp\nesug2012\mylogs\example4.log"; /* example4.sas */ %log4sas; %log4sas_appender(myappender, "FileRefAppender", 'fileref=mylog pattern="%d %-5p (%S{jobid}:%S{user_name}) %m"'); %log4sas_logger(mylogger, 'level=info appender-ref=(myappender)'); %log4sas_info(mylogger, 'Started example4.sas'); %macro example4; /* Notice that I do not have a subdirectory named nesug2013 */ libname mylib "c:\temp\nesug2013"; %if (&syslibrc ne 0) %then %do; %log4sas_error(mylogger, 'Unable to create libname ref MYLIB'); % %else %do; %log4sas_info(mylogger, 'Created a libname reference MYLIB'); % data mylib.mytemp; message = 'MYTEMP dataset created'; %if (&syserr ne 0) %then %do; %log4sas_error(mylogger, 'Unable to create dataset MYLIB.MYTEMP'); % %else %do; %log4sas_info(mylogger, 'Created a permanent dataset MYLIB.MYTEMP'); % %mend example4; %example4; 10

11 %log4sas_info(mylogger, 'Finished example4.sas'); /****************************************************************/ /* S A S S A M P L E L I B R A R Y */ /* */ /* NAME: BPG21R01 */ /* TITLE: MEANS Procedure, Chapter 21 */ /* PRODUCT: SAS */ /* SYSTEM: ALL */ /* KEYS: EXAMPLES FROM DOCUMENTATION, SKEWNESS, KURTOSIS, */ /* KEYS: DESCRIPTIVE STATISTICS, */ /* PROCS: MEANS SUMMARY */ /* DATA: */ /* */ /* SUPPORT: UPDATE: */ /* REF: SAS Procedures Guide, CHAPTER 21 */ /* MISC: */ /* */ /****************************************************************/ options ls=132; title; data gains; input name $ team $ age ; cards; Alfred blue 6 Alicia red 5 Barbara. 5 Bennett red. Carol blue 5 Carlos blue 6 ; proc means nmiss n; class team; data gains; input name $ height weight; cards; Alfred Alicia Barbara Bennett Carol Carlos ; proc means noprint; class name; output out=results; /* **************** */ /* chap21_means.sas */ /* **************** */ 11

12 proc print data=results; data gains; input name $ sex $ height weight school $ time; cards; Alfred M AJH 1 Alfred M AJH 2 Alicia F BJH 1 Alicia F BJH 2 Benicia F BJH 1 Benicia F BJH 2 Bennett F AJH 1 Bennett F AJH 2 Carol F BJH 1 Carol F BJH 2 Carlos M AJH 1 Carlos M AJH 2 Henry M AJH 1 Henry M AJH 2 Jaime M BJH 1 Jaime M BJH 2 Janet F AJH 1 Janet F AJH 2 Jean M AJH 1 Jean M AJH 2 Joyce M BJH 1 Joyce M BJH 2 Luc M AJH 1 Luc M AJH 2 Marie F BJH 1 Marie F BJH 2 Medford M AJH 1 Medford M.... Philip M AJH 1 Philip M AJH 2 Robert M BJH 1 Robert M BJH 2 Thomas M AJH 1 Thomas M AJH 2 Wakana F AJH 1 Wakana F AJH 2 William M BJH 1 William M BJH 2 ; proc means data=gains; var height weight; class sex; output out=test max=maxht maxwght maxid(height(name) weight(name))=tallest heaviest; proc print data=test; data homes; input name $ homeown $ age income ; cards; 12

13 rodrick n smith n freiss y garcia y williams n mason n lopez n gregory n reid n schulman y garrett y zingraff y ; proc means; class age homeown; var income; output out=stats mean=incmean; proc print data=stats; data gains; input name $ sex $ height weight school $ time; cards; Alfred M AJH 1 Alfred M AJH 2 Alicia F BJH 1 Alicia F BJH 2 Benicia F BJH 1 Benicia F BJH 2 Bennett F AJH 1 Bennett F AJH 2 Carol F BJH 1 Carol F BJH 2 Carlos M AJH 1 Carlos M AJH 2 Henry M AJH 1 Henry M AJH 2 Jaime M BJH 1 Jaime M BJH 2 Janet F AJH 1 Janet F AJH 2 Jean M AJH 1 Jean M AJH 2 Joyce M BJH 1 Joyce M BJH 2 Luc M AJH 1 Luc M AJH 2 Marie F BJH 1 Marie F BJH 2 Medford M AJH 1 Medford M.... Philip M AJH 1 Philip M AJH 2 Robert M BJH 1 Robert M BJH 2 Thomas M AJH 1 Thomas M AJH 2 Wakana F AJH 1 13

14 Wakana F AJH 2 William M BJH 1 William M BJH 2 ; proc means; title 'Statistics For All Numeric Variables'; proc means data=gains maxdec=3 nmiss range uss css t prt sumwgt skewness kurtosis; var height weight; title 'Requesting Assorted Statistics'; proc format; value timepr 1='Fall' 2='Spring'; proc means maxdec=3 fw=10; class school time; var height weight; format time timepr.; title 'Statistics With Two Class Variables'; proc sort; by school time; proc means maxdec=3 fw=10; by school time; var height weight; output out=new mean=hmean wmean stderr=hse wse; format time timepr.; title 'Statistics With Two By Variables'; proc print; title 'New Data Set'; format time timepr.; data relay; input name $ sex $ back breast fly free; cards; Sue F Karen F Jan F Andrea F Carol F Ellen F Jim M Mike M Sam M Clayton M ; 14

15 proc means data=relay noprint; var back breast fly free; class sex; output out=newmeans min=; proc print data=newmeans; by sex; title 'Using PROC PRINT with PROC MEANS'; proc summary data=relay print min; var back breast fly free; class sex; output out=newsumm min=; title 'Using PROC SUMMARY with the PRINT option'; proc print data=newsumm; by sex; title 'Using PROC PRINT with PROC SUMMARY'; 15