A robust and flexible approach to automating SAS jobs under Unix Mike Atkinson, with the Ministry of Health Services, British Columbia

Transcription

1 A robust and flexible approach to automating SAS jobs under Unix Mike Atkinson, with the Ministry of Health Services, British Columbia Abstract So you ve got a Unix server that is terrific for running big jobs, either right when you command or perhaps scheduled to run off-hours. What if those jobs could you to indicate their success or failure, with generated reports as attachments (when successful)? And if they could automatically store all of their logs, date and time stamped? This paper describes the use of a perl script that has been developed at the BC Ministry of Health Services to schedule SAS jobs on Unix. Here the term job refers to a set of SAS program(s) and/or perl module(s) to be run in sequence. In practice, most of the steps in our jobs are SAS programs, with perl used to do things like ing reports, and sending files via FTP. The Unix server is up and running 24/7, which makes it ideal for scheduling those massive jobs to run at night while everyone is sleeping. You may be aware that Unix comes with a nifty scheduling facility called crontab, which lets you run programs at specific times and dates, including the ability to run programs at regular intervals. Our system extends the power of crontab so that we can easily run multi-step jobs. Log and output files are automatically date-time stamped, so they won t be overwritten. Reports are typically ed to recipients as a final step of each job. Directory Tree The Unix file system uses a tree structure, and trees are a great way to organise your SAS code, logs, data, and output. In our system, the effective root of the tree indicates the environment: devl, prod, test, et cetera. Under each environment (devl, prod, etc) is an identical structure for storing all the items related to SAS jobs: programs, logs, reports, data. Here is a sample tree structure for an automated system: /path/to/devl /parms /bin /sascode /demo /monthly_reports

2 /shared_code /macros /logs /demo /monthly_reports /shared_code /reports /demo /monthly_reports /shared_code /sasdata /demo /monthly_reports Some of these sub-directories (shown in bold above) are required by our job scheduling system; others can be created as needed based on system-specific requirements. The /bin sub-directory contains all perl code, including the scripts that run jobs. Other useful perl modules (such as a module designed to FTP files) and any custom perl modules are stored here as well. The /parms sub-directory contains two files related to each job. Each job requires a.run file and a.mail file in the /parms sub-directory, for instance: demo.run and demo.mail. The.run file lists the steps (SAS programs and/or perl modules) to be run, one per line. Parameter values can be passed to perl modules or to SAS programs on the command line. The /sascode sub-directory in turn has a separate sub-directory for each distinct system being automated. Each of its sub-directories stores a group of SAS programs that might represent a system or sub-system of a larger system. (Using sub-directories in this way is a great way to organise SAS code, logs, and reports.) Our convention is to create a sub-directory of /sascode named /shared_code to store programs that cross system boundaries, and to put shared macros in a sub-directory beneath /shared_code. Other SAS code is placed under the appropriate sub-directory name. The /logs sub-directory contains the same set of sub-directories as the /sascode sub-directory; that is, there is a log directory corresponding to each sascode directory. When a job is run, the log for each executed program is automatically placed into the appropriate /logs sub-directory, with the job start date and time both contained within the file name of the log. The /reports sub-directory should also contain the same set of sub-directories as the /sascode sub-directory. Reports to standard SAS output will be automatically stored in the correct sub-directory under /reports, again date and time stamped. There is a bit more flexibility around the /sasdata sub-directory. This sub-directory (as might be expected) is used to store SAS datasets, and is allocated within SAS programs using a libname statement. Programs can choose whether to store each SAS dataset

3 directly under the /sasdata directory, or to store datasets under a lower level sub-directory (one level down from /sasdata). One of the goals when writing production SAS programs is to make the code completely portable between environments. It is much easier to manage migration from development to production when the migration does not require a single change to the code. We use a simple method for portable programs: the perl script that executes SAS programs passes the environment (e.g. devl) as a parameter to each SAS job it runs. SAS programs can easily make this into a macro variable indicating which environment the job is running in. Portability can be achieved if SAS programs use this macro variable within the path to all referenced files. Crontab The perl script will pass the environment to each executed SAS program, but how does the perl script know which environment it is running in? Our perl script determines the environment by checking the current directory at run time. A convention we follow for scheduling jobs (under crontab) is that two commands are always scheduled together: a cd command (change directory) to set the current directory as the /bin directory in one of the environments, and, separated by a semi-colon on the same line, a command to run the perl script. For instance, a crontab entry might look like the following: * * cd /path/to/prod/bin; run_job demo Up until now, we ve not given a name to the perl script - here it appears as run_job. This run_job is actually a small stub that performs some initialisations and then calls the real workhorse perl module job_control.pm. Each system using our job scheduler has its own small stub script to set a few system-specific items (such as the path to the system) before calling job_control.pm. Most of the details of the perl module job_control.pm are outside the scope of this paper. When the run_job perl script is executing, it can easily determine what the current directory is in this case /path/to/prod/bin. This system-specific script would likely be expecting the path to be of the form /path/to/environment/bin, and in this case the script would find that it is running in the prod environment. Crontab will execute these commands at the indicated time: 5am on the first of each month. The perl scripts will look in the directory /path/to/prod/parms for files named demo.run and demo.mail, and will start executing the steps in the demo.run file. It is also easy to run a job immediately by entering the run_job command at the prompt (while in the /bin directory of the environment we want to run in). This is often how the job is submitted to restart at the point of error - described later in the section Running jobs manually.

4 Defining Jobs As mentioned above, each job requires two files in the /parms directory, named job_name.run and job_name.mail where job_name is replaced by the name of a job. The.run file lists the steps of the job, in the order they are to be run. The.mail file lists the addresses of people to receive s of job success/failure notifications and s with reports as attachments. The.run file contains one line for each step in a job. This line simply names the program to be run and the name of the system the program belongs to, and optionally provides parameter values to be passed to the job. Either a SAS program or perl module can be called in each step: the file type (after the dot) indicates the type of each step. Comments start with a pound sign. Here s a short sample job.run file that has been named demo.run. # Demonstration job # demo/compile_data.sas demo/summary_report.sas demo/mail_reports.pm period_type=month_end You may have noticed that each line in this job starts with demo/. This names the system (or sub-system) the programs belong to: this also forms part of the path to sub-directories where the SAS code will be found, and where logs and reports will be placed. For instance, SAS code might be in the directory /path/to/prod/sascode/demo. When this job is executed (by the run_job perl script), one step will be run at a time. The file type on each line in the.run file (.sas or.pm) indicates whether the named code is SAS or perl. In this job, first the SAS program named compile_data.sas will be run. This program must be in the sub-directory named /demo, which in turn falls within the /sascode directory, e.g. path_to/devl/sascode/monthly_reports. This command line has a parameter named period_type. The perl script will pass this to the SAS program, where it can be turned into a macro variable (named &period_type) with the value MONTH_END. (Note: The sample code for demo_init.sasinc is just a demonstration of parameter passing, and does create any macro variables actually used by sample program named summary_report.sas.) The next SAS program in the job, summary_reports.sas, can create reports in one or many available formats: standard output listing, PDF, RTF, et cetera. If reports are written to the standard output destination, they will be created (in the reports sub-directory) as.lst files, with the program name and date/time stamp in the filename. For more refined output, the reporting program can specify a filename for a PDF or other type of output file.

5 The final step of this job is a perl module called mail_reports.pm. This perl module is commonly found at the end of.run files. Like other perl modules and scripts (but unlike SAS code in the system), the mail_reports.pm module does not reside in a sub-directory named /bin/demo it sits right in the /bin sub-directory. When this perl module is executed, it will look for reports within the correct reports sub-directory, in this case the directory /path/to/devl/reports/demo. The mail_reports.pm perl module will only mail reports associated with this particular execution of the job, which are identified by the date and time stamp that should appear in every report s file name. The mail_reports.pm module sends reports to recipients based on information in the.mail file. This file lists the addresses of report recipients, and includes one or more regular expression patterns for each recipient. Each recipient is sent all report files (created by this run of the job) that have names matching at least one of the listed patterns for that recipient. A regular expression consisting of a single period (.) will match all reports created by the job (and in the named sub-directory). In addition to regular expressions, there are a couple of special characters available as well, mainly for job administrators: exclamation mark (!) and greater-than sign (>). The exclamation mark indicates that a recipient will receive a notification of success or failure each time the job is executed. The greater-than sign indicates that the recipient will receive a list of which reports were sent to all recipients. Here is a sample.mail file, which in this case would be named demo.mail: # # Send all the reports, a list of who was sent what report, # and the job summary report IT.Guy@ABC.net! >. # # Send all reports, but nothing else Main.User@ABC.net. # # Gets all PDF reports only Other.User@ABC.net pdf$ PDF$ # # Gets reports containing one, two or three in the file name Another.User@ABC.net one two three SAS programs The first thing one of our SAS programs needs to do is to determine which environment it s being run in: development, production, et cetera. This has to be done first since if the program will %include other SAS code, it needs to know whether to include code file(s) from the devl, prod, or another environment. The perl script (that runs each SAS program) extracts the environment from the current directory, and passes this to the SAS program via sysparm. Also passed via sysparm (as one single, long value) are other parameters and values, including the date-time stamp.

6 The SAS program can use a single statement to set the macro variable &Env as the environment. * Find whether we are running in DEVL or PROD environment; %let Env = %scan(&sysparm, 1, %str(:)); Now that we have set &Env, we are ready to %include SAS code from the correct environment, by including &Env in the file path. Our SAS programs are likely to %include some code to read the remaining parameters from &sysparm, such as: %include "/path/to/&env/sascode/shared_code/initial.sasinc"; Here s what the command (created by the run_job script) might look like. This would be a single, long command it is only wrapped here because it is too long to fit on one line of this document. Notice that the sysparm parameter is one long string of characters the format will be described later. /usr/sas/sas -autoexec /path/to/site/autoexec/autoexec.sas -log /path/devl/logs/demo/compile_data log -print /demo/devl/reports/demo/compile_data lst /demo/devl/sascode/demo/compile_data.sas -sysparm devl:subdir=demo:date_stamp= :run_instance=1:period= MONTH_END This command ensures that the date and time stamp ( ) appears in the log and listing file names. The value is also passed to the SAS program via sysparm, to be placed into SAS macro variable &date_stamp. The string passed to sysparm is available within SAS as the system macro variable &sysparm. As mentioned, each SAS program extracts this from the start of &sysparm, as shown above. The SAS code in the file initial.sasinc reads the remainder of &sysparm (following the first colon), setting macro variables for each pair of variable=value found (between successive colons. In addition, initial.sasinc does some generic initialisation, such as allocating macro and template libraries, and determining the current date. Here s some of the code from initial.sasinc, which reads the parameter pairs from &sysparm. * Parse the passed sysparm value, setting macro values; %macro parse_sysparm; %global sysparm; %* Start with the 2nd parm the first indicated the environment; %let i = 2; %* Pairs are separated by colons; %let pair = %scan(&sysparm, &i, %str(:));

7 %do %while (%str(&pair) ne %str()); %end; %* Each parameter pair has a macro name and value, separated ; %* by the equals sign; %let var_name = %scan(&pair, 1, %str(=)); %let var_value = %substr(&pair, (1 + %index(&pair, %str(=)))); %* Assign value to global macro variable &var_name; %global &var_name; %let &var_name=&var_value; %* Get the next value, if any; %let i = %eval(&i + 1); %let pair = %scan(&sysparm, &i, %str(:)); %mend parse_sysparm; * Call the macro just defined; %parse_sysparm; The above macro could be stored in a macro library, with only the last line (calling the macro) required in the initial.sasinc code. If presentation is not important, a program might simply print to the standard text-based SAS output destination, and generate a report in a fixed width font. But for any program that will be run frequently, it might be worth a bit of effort to make reports more readable. Most often we will generate reports (using ODS) into PDF format. A sample SAS program (job step) Imagine that you need a SAS program named summary_report.sas to produce a monthly summary report. We ll look at a relatively simple sample program that creates PDF output (which is not really a monthly report), and then see how this program might be handled by the system. /********************************************************************** * summary_reports.sas: Report personal health expenditures * * for Canada * * * * Change History: * * * * ---Date--- Who What * * M.Atkinson Original * * * **********************************************************************/ * Find whether we are running in DEVL or PROD environment; %let Env = %scan(&sysparm, 1, %str(:));

8 * Include initialisation code; * (Set dates for processing based on the current date); %include "/path/to/&env/sascode/demo_init.sasinc"; * Print values of global macro variables into the log for reference; * Any global macro variables containing an underscore will be included; %print_macro_vals(inc_text=%str(_)); * Give file names for HTML and PDF output; * The macro variable &date_stamp should appear in file names; filename htmlfile "/path/to/&env/reports/&subdir/personal_health_expend_&date_stamp..html"; filename pdffile "/path/to/&env/reports/&subdir/personal_health_expend_&date_stamp..pdf"; * Assign a libname to the demo library; libname demolib "/path/to/&env/sasdata/demo"; * Create report in HTML format; options missing=' '; options orientation=landscape; ods listing close; ods markup file=htmlfile (title="personal Health Expenditure") style=my_helvetica tagset=tagsets.remtitle; ods printer pdf file=pdffile style=my_helvetica notoc; * Build a format for millions; proc format; picture million low-high='0,000,009m' (prefix='$' mult= ); run; * Report expenditures; proc report data=demolib.personal_health_expenditure nowd; column row_order description year, amount; define row_order / noprint group; define description / group ' ' style(header)={vjust=bottom}; define year / across ' ' style(header)={vjust=bottom};

9 define amount / analysis ' ' format=million. style(header)={vjust=bottom}; compute amount; if (substr(description, 1, 1) = '%') then call define(_col_, "format", "percent7.1"); endcomp; title1 "Personal expenditure on medical care and health services"; footnote1 j=l h=10pt f=helvetica "From Statistics Canada: quit; ods printer close; ods markup close; ods listing; Using the Current Date The following code is an example of setting various macro variables based on the current date. This is an artificial example, since the program summary_reports.sas above does not use the values created by this.sasinc file, except for those created by shared_code/initial.sasinc. /************************************************************************** * demo_init.sasinc Demonstrate initialisation processing, including * * the setting of dates based on the current date. * * * * Change History: * * * * ---Date--- Who What * * M.Atkinson Created * **************************************************************************/ * Include the shared initialisation code; * (this sets macro variable &current_date); %include "/hlth/sda/&env/sascode/shared_code/initial.sasinc"; * Set date cutoff value based on current date; data _null_; * Use 6 days after current date, allowing for the case that data for ; * the date becomes available a few days prior to the actual date; current_date_plus6_sas = 6 + input("&current_date", yymmdd8.); * If 6 days from current date is still prior to the 15th, ; * (i.e. we were prior to the 9th) then use the end of the prior month; if (day(current_date_plus6_sas) < 15) then do; call symput('mid_month', 'NO'); cutoff_sas = -1 + mdy(month(current_date_plus6_sas), 1, year(current_date_plus6_sas)); end;

10 * Otherwise, use mid-month (i.e. the 15th); else do; call symput('mid_month', 'YES'); cutoff_sas = mdy(month(current_date_plus6_sas), 15, year(current_date_plus6_sas)); end; call symput('cutoff_sas', cutoff_sas); call symput('cutoff_num', put(cutoff_sas, yymmddn8.)); run; * Print values of global macro variables into the log for reference; %print_macro_vals(inc_text=%str(_)); * Assign a libname to the demo library; libname demolib "/hlth/sda/&env/sasdata/demo"; One convention we have is to include an underscore in all macro variable names this makes it easy for our %print_macro_vals macro to pick out which macro variables to write to the log. We typically call this macro after initialising all macro variables, so that each log has a nice record of all the macro variable values that were in effect. For instance, the following might be written within the log: ************ User Macro Variable Values Shown Below ************ Note: Macro variables must contain one of: _ CURRENT_DATE = CUTOFF_NUM = CUTOFF_SAS = DATE_STAMP = MID_MONTH = NO PERIOD = MONTH_END RUN_INSTANCE = 1 TODAYS_DATE = ************ User Macro Variable Values Shown Above ************ A sample SAS step as part of a job Now suppose that this SAS program was one step of a job - the job shown in the section Defining Jobs. If a job fails, an is sent to the appropriate person or persons. In this case, the failure notification is sent only to IT.Guy@ABC.net (who has the! symbol in the.mail file). Presumably he will fix the error and re-submit. The indicating failure looks like this. (It tends to get your attention when you receive an with subject line containing failed in capital letters!) Subject: **run_job demo FAILED** run_job demo FAILED with return code 2 Command:...run_job demo

11 Environment:...devl Job Name:...demo Date Stamp: Start at step:...(beginning) End at step:...(end) Following steps were run: Start Return Time Code Command 18:23:46 0 demo/compile_data.sas period=month_end 18:23:57 ******2 demo/summary_reports.sas 18:23:57 Failure Time To run all steps from the failed step onward, you might use the following command: run_job demo summary_reports.sas & The last line of the failure message can be very handy. Once the error has been corrected, the job can be restarted at the step that had an error by using the command provided. Specifying the date-time stamp is key when restarting jobs. Since the mail_reports.pm perl module (usually the last step of the job) will be using the date-time stamp to identify all reports that apply to this job, it is important that the same date-time stamp value be used in the names of files created both in job steps run prior to the point of error, and in later job steps ran in the restarted job. Specifying the original date-time stamp as an argument to the perl script overrides the usual behaviour of using the current date and time, and the passed date-time stamp value is used instead. Oops if the same date/time stamp is used, won t the new log of the restarted step overwrite the log created by the failed execution? To prevent this from happening, old logs are kept in a separate file to the most recent log. Before a log file can be overwritten, it is renamed by the perl script, appending.old to the file name. If there have been two or more failures for the same step, logs for successive runs are appended to the log that ends with.old. The final, successful, log is always stored on its own, in the file ending with.log. When the job completes successfully, the final step (mail_reports.pm) will the report(s) to the indicated recipients. In the case of the demo job (as indicated by the demo.mail file), the recipients will be: IT.Guy@ABC.net (who receives all reports) Main.User@ABC.net (who also receives all reports) Other.User@ABC.net (who receives all reports of type PDF) Another.User@ABC.net (who receives reports where the file name contains one, two, or three ) All reports created during the job (by programs in the demo system), and which are targeted for a given recipient, will be mailed as multiple attachments of a single to

12 that recipient. Since the attached reports for each recipient may be different, each receives their own, separate, . Let s suppose that the program detail_reports.sas created reports named fancy_one.pdf, really_fancy.pdf, and summary_report.lst. All three reports would go to each of the recipients listed above, and the report named one.pdf will go to Other.User@ABC.net (and also to the other three recipients). But let s assume the job executed successfully the first time. An such as the following, showing the execution time for each step in the job, would be sent to IT.Guy. Subject: run_job demo was successful run_job demo completed with return code 0 Command:...run_job demo Environment:...devl Job Name:...demo Date Stamp: Start at step:...(beginning) End at step:...(end) Following steps were run: Start Return Time Code Command 18:18:35 0 demo/compile_data.sas period=month_end 18:18:46 0 demo/summary_reports.sas 18:18:55 0 demo/mail_reports.pm 18:18:55 Completion Time In addition to the job summary shown above, each recipient of report(s) would be sent an with applicable reports attached. Attachments to the could then be viewed on the screen or saved to disk. The with report attachments might look like the following: Subject: Reports from demo stream, Run date: This has the following 3 attachments: /path/to/devl/reports/demo/personal_health_expend_ html /path/to/devl/reports/demo/personal_health_expend_ pdf /path/to/devl/reports/demo/compile_data lst In addition to the text above listing these files, the would have the indicated files as attachments. Here s what the reports look like:

13 Personal_Health_Expend_ html: Personal expenditure on medical care and health services Total personal expenditure on consumer goods and services $596,009M $655,722M $719,917M $801,211M $891,197M Total personal expenditure on medical care and health services $28,986M $33,623M $38,032M $43,862M $50,589M Medical care $13,456M $15,395M $16,997M $19,657M $22,778M Hospital care and the like $1,388M $1,577M $1,798M $2,044M $2,380M Other medical care expenses $4,137M $4,546M $5,178M $5,480M $6,315M Drugs and pharmaceutical products $10,005M $12,105M $14,059M $16,681M $19,116M % of all personal expenditure on medical care and health services 4.9% 5.1% 5.3% 5.5% 5.7% From Statistics Canada: Personal_Health_Expend_ pdf: (on next page)

14 Personal expenditure on medical care and health services 18:18 Thursday, July 2, Total personal expenditure on consumer goods and services $596,009M $655,722M $719,917M $801,211M $891,197M Total personal expenditure on medical care and health services $28,986M $33,623M $38,032M $43,862M $50,589M Medical care $13,456M $15,395M $16,997M $19,657M $22,778M Hospital care and the like $1,388M $1,577M $1,798M $2,044M $2,380M Other medical care expenses $4,137M $4,546M $5,178M $5,480M $6,315M Drugs and pharmaceutical products $10,005M $12,105M $14,059M $16,681M $19,116M % of all personal expenditure on medical care and health services 4.9% 5.1% 5.3% 5.5% 5.7% From Statistics Canada:

15 compile_data lst: row_ Obs order year description Straight proc print with no frills 1 18:18 Thurday, July 2, Total personal expenditure on consumer goods and services Total personal expenditure on consumer goods and services Total personal expenditure on consumer goods and services Total personal expenditure on consumer goods and services Total personal expenditure on consumer goods and services Total personal expenditure on medical care and health services Total personal expenditure on medical care and health services Total personal expenditure on medical care and health services Total personal expenditure on medical care and health services Total personal expenditure on medical care and health services Medical care Medical care Medical care Medical care Medical care Obs amount Obs lag_description 1 2 Total personal expenditure on consumer goods and services 3 Total personal expenditure on consumer goods and services 4 Total personal expenditure on consumer goods and services 5 Total personal expenditure on consumer goods and services 6 Total personal expenditure on consumer goods and services 7 Total personal expenditure on medical care and health services 8 Total personal expenditure on medical care and health services 9 Total personal expenditure on medical care and health services 10 Total personal expenditure on medical care and health services 11 Total personal expenditure on medical care and health services 12 Medical care 13 Medical care 14 Medical care 15 Medical care Et cetera

16 IT.Guy would also receive a third , which lists which reports were sent to recipients. In this case, that might look like this: Subject: *Reports that the demo stream sent out, Run date: Recipient: IT.Guy@ABC.net /path/to/devl/reports/demo/personal_health_expend_ html /path/to/devl/reports/demo/personal_health_expend_ pdf /path/to/devl/reports/demo/compile_data lst Recipient: Main.User@ABC.net /path/to/devl/reports/demo/personal_health_expend_ html /path/to/devl/reports/demo/personal_health_expend_ pdf /path/to/devl/reports/demo/compile_data lst Recipient: Other.User@ABC.net /path/to/devl/reports/demo/personal_health_expend_ pdf Running jobs manually Jobs can be run on demand at the command prompt. First, we ensure that the current Unix directory is the /bin directory of the environment we wish to run in, such as /path/to/devl/bin. The command specify to run the entire job stream as a new run and using the current date and time in the date-time stamp, as in: run_job demo & or to run at a specific step and using a specific date-time stamp, as in: run_job demo summary_reports.sas & If all re-starts of a job specify the original date-time stamp (i.e. the time of the first attempt to run the job), all reports created by the job will have names which include this date-time stamp. If the failed step created reports, these will be overwritten by the new, successful, run of this step. At successful completion of the job, the latest version of all reports can be automatically mailed to recipients, regardless of whether the reports were generated prior to, within, or after the failed step. Note: the log file with this date-time stamp will be the log for the latest, successful, execution of each step. Earlier logs for failed steps are not lost however they are stored in a log file named with date-time stamp and also with the suffix.old.

17 Essential Perl Modules As mentioned earlier, the run_job perl script is just a stub that sets some parameters before calling the workhorse of the system, job_control.pm. The job_control.pm script reads both the.run file (to get the list of steps in the job) and the.mail file (for the list of recipients). Each line of the.run file is parsed to find the sub-directory, program name, and arguments (if applicable). Take for example, the following line: demo/compile_data.sas period=month_end Given this line (in the.mail file) the perl script will identify the sub-directory (demo) and the name of the program (compile_data.sas). Since in this case the program has the suffix.sas, the perl script will call the runpgm.pm perl module to execute the program using SAS. (If the suffix had been.pm, step would have been recognised as a perl module and executed as a perl subroutine.) The mail_reports.pm perl module is not required to be in every job, but is nearly always at the end of each job. When called, this perl module is given a system or sub-system name, such as demo in the call: demo/mail_reports.pm The mail_reports.pm module finds all files in the corresponding reports sub-directory (e.g. /path/to/prod/reports/demo) and performs pattern matching for each potential recipients as indicated by the information that was in the.mail file. An is sent to each recipient for whom one or more reports met the criteria, with those reports files as attachments. A summary of this information is sent to any recipient who has the pattern > (greater-than sign). Conclusion We ve developed a handy Unix-based framework for executing multi-step SAS jobs. Reports and logs are handled nicely, and it is straightforward to re-start failed jobs mid-stream. The system has the kind of reliability that helps one relax while on vacation. Contact Information Mike.Atkinson@gov.bc.ca working with the British Columbia Ministry of Health Services Victoria, British Columbia (250) SAS and all other SAS Institute Inc. product or service names are registered trademarks or trademarks of SAS Institute Inc. in the USA and other countries. indicates USA registration. Other brand and product names are registered trademarks or trademarks of their respective companies.