Foundations & Fundamentals

Transcription

1 10 Things a Beginner Should Know Before Writing a SAS Program Lisa Eckler, Lisa Eckler Consulting Inc., Toronto, ON ABSTRACT Writing and running one s first SAS program can be wonderfully quick and easy or not. Either way, there s a long route with many lessons to be learned between writing that first program and truly being a SAS programmer. This paper will address some of the things your best friend would tell you about SAS before you tried it, if your best friend were an experienced SAS programmer. An early awareness of some of these ideas may help you avoid having to discover them in the face of a looming deadline, with a program that has gotten unmanageably complicated and doesn t produce the desired result. This paper should be of value to beginner programmers or those making a transition to SAS from some other programming language. It will include some fundamentals of programming, an introduction to some tips and techniques for understanding and working with SAS code, and pointers to some classic SAS papers and other recommended resources. This is an updated version of the NESUG 2007 paper with the same title. INTRODUCTION Why is this important? Why not jump right in and learn by trying? Certainly you could crack open a basic SAS manual, or run through the Help available from the toolbar in an interactive SAS session, and learn to construct and string together some steps. The problem is that it s tough to progress from there to elegant and efficient program design without some exposure to well-designed programs. More simply, it s hard to know what you don t know. You probably have some idea about the concepts of program input and output, and know how to invoke SAS in your environment, whether that means knowing how to start and end an interactive SAS session or how to submit a batch program for execution and where the log and output will end up. If not, plan a time-out to learn about those things. (Refer to sections #9 and #10, below, for some advice on resources.) Most of the topics highlighted in this paper have been eloquently addressed by others in the SAS user community, so the intention here is not to explain in detail, but to plant a seed of curiosity and offer some suggestions for nurturing it, so that when the time is right, you will have some ideas on what to look for. Generations of SAS programmers have come before you, and you can benefit from their experience. #1 START WITH A PLAN It can be very quick and easy to produce something from your data using SAS code, but that isn t helpful if what you produce isn t what you need. While there is a time and place for casual exploration of data or iterative analysis using SAS, you need to have a plan before beginning to write a program. For this discussion, a program will be defined as an automated set of instructions that will produce consistent results when repeated under the consistent circumstances. Addressing variations in circumstances or data that can reasonably be anticipated adds to the robustness of the program. Before you start playing with some data, consider whether the exercise is to advance your knowledge of the data or to write a repeatable set of instructions to accomplish a result. The former can be very valuable if your goal is a one-time analysis. If your role is that of programmer, as opposed to analyst or researcher, your goal should be to develop automated, repeatable processes meaning programs so you should have a statement of purpose before you start writing code. The level of detail in your plan will vary with the complexity of the program. Include this statement as part of the comments in your code. For example, a very basic plan might be: 1

2 ** **; ** Read customer names and addresses from a flat file, sort by city, and create a **; ** report listing the customers in each city. **; ** **; A slightly more complex plan might be: ** **; ** Read customer names and addresses from a flat file, combine that data with **; ** city-to-region mappings from the region s database and regional manager data **; ** from an Excel spreadsheet. Create a report for each regional manager listing **; ** their customers by city. **; ** Include an Exception Report for addresses which don t appear to contain a city. **; ** **; With experience, the plan for your program may begin to sound like SAS instructions. That s not a bad thing, as long as it remains understandable to someone who doesn t write code. You see, commenting your code doesn t have to be tedious, especially if the comments help you to plan and structure your efforts. Why is a plan so important? It will clarify the objective and keep you focused. Meandering strolls may be enjoyable, but there s no guarantee that you will reach your destination if you haven t identified what the destination is. This advice, of course, applies to any programming language. #2 FIGURE OUT HOW TO ACCESS YOUR STUFF A wise colleague has said only partly jokingly that the five most important SAS statements are: LIBNAME, LIBNAME, LIBNAME, LIBNAME and LIBNAME. While that might be a slight overstatement, understanding how to refer to the data source(s) you wish to work with is very important. Understanding the power of LIBNAME, FILENAME and path specification will allow you to take advantage of what SAS can do for you. This facilitates program input and output. It s simple mechanical stuff like connectivity to remote SAS sessions on other platforms, or working with data from a non-sas database or file format, which opens up the opportunities. Find out what SAS products you are licensed for and what databases or data formats you may need to work with, and then read about SAS/ACCESS, SAS/CONNECT, and other SAS products which may help you. Some of the choices you make about how to approach your programming task will depend on what SAS tools are available to you. The simplest situation occurs when you have data already stored in a SAS dataset and you can begin working with that dataset. SAS datasets can be referred to using a single name, such as EXAMPLE1. In a typical environment, SAS will assume the default library of WORK and member type of DATA, so the statement set EXAMPLE1; is really a short-form of set WORK.EXAMPLE1.DATA; (although you will very rarely see it described explicitly). Under normal conditions, the contents of the WORK library are lost when a SAS session ends. To work with data from a permanent SAS data library, the statement would be data set PERM_LIB.EXAMPLE1; 2

3 where libname PERM_LIB <some dataset name recognizable to your operating system> ; had been defined prior to the DATA step. #3 THINGS ARE NOT ALWAYS AS THEY APPEAR Things are not always as they appear in SAS. This can be a great strength or great weakness, depending on the circumstance. One situation you may encounter is when SAS attempts to help you by allowing arithmetic operations on character variables which happen to be the character form of a numeric value. For example, you may have been taught that the literal A (a quoted string) is not the same as the letter A (without quotes) in a program. It would logically follow that the literal 1 is not the same as the number 1. If you specify an arithmetic operation on a variable containing the value 1, however, SAS will attempt to perform arithmetic as if 1 were the number 1. Yes, this can be very confusing! Sometimes this automatic conversion is helpful and other times not. SAS log notes will tell you if this has occurred during program execution, so a thorough review of the log is necessary. Consider how SAS treats the values of the variables in the following code as something you hope not to encounter: data EXAMPLE; var_one = 1; result = 2 * var_one; put result= ; var_a = 'A'; result = 2 * var_a; put result= ; var_a = '1'; result = 2 * var_a; put result= ; A portion of the SAS log resulting from running this code: result=2 NOTE: Invalid numeric data, var_a='a', at line 698 column 18. result=. result=2 var_one=1 result=2 var_a=1 _ERROR_=1 _N_=1 In another situation, you may look at the results of a procedure or DATA step, either in report form or by browsing a dataset, and see something that differs from the values of the variables because a format has been applied. Formats don't alter the underlying stored data, but may affect what you see. For example, you may know that what was entered (and stored) was numeric, but what you see appears to be character. Or, what you see may be a different format of number than what was captured. One way to recognize when data have been formatted is to run the CONTENTS procedure on the dataset. For example, if you looked at only the following PROC PRINT output, you might assume that dt is an alpha-numeric variable, and that avg_precip_inches is a numeric variable with one decimal place: 3

4 avg_ avg_precip_ Obs dt high_f inches 1 APR APR APR APR APR Those assumptions would be wrong: dt is actually stored as a numeric SAS date which happens to be formatted as monyy. and avg_precip_inches is stored with two decimal places but displayed with one. #4 THE PDV IS KEY SAS, and particularly the DATA step, isn t magic and doesn t do magic. The DATA step allows for tremendous flexibility, but you need at least an elementary understanding of the workings of the Program Data Vector (PDV) and the SAS Interpreter in order to take advantage of it with confidence. This is sometimes described as understanding How SAS Thinks (Howard, 2004). Just because you don t (usually) see what s in the PDV doesn t mean it isn t important. A great way to become familiar with the PDV and the inner workings of the DATA step is the judicious use of PUT _ALL_; during the development or test phase to display what SAS is seeing and doing as you process each observation in a DATA step. By default, PUT statements will write to the program log (or log window). You need to be aware of the number of lines you will be writing, in order not to fill the log with a huge volume of displayed data. Some of the very powerful features of DATA step processing are also the most likely to cause confusion for a beginner. These would include SET statement options, the RETAIN statement, or MERGE, or FIRST.byvariable and LAST.byvariable processing. These would be helpful things to experiment with in a test environment, using PUT _ALL_;. #5 HARNESS THE POWER If it feels like your program is getting too long or overly-complicated, it is! Keep in mind that no one else who reads your code is going to find it easier to understand or maintain than you do. Even though you may recognize that the complexity has gotten out of hand, it s hard to know what tools or techniques are available to you if you ve never seen or heard of them. Here are some thoughts which are simple, but perhaps not intuitive: You can specify which variables to DROP, KEEP or RENAME on a SET statement, a DATA statement, or as part of the specification of input to, or output from, a procedure. Applying a SAS format, or a user-defined format created using PROC FORMAT, is an easy way to classify data for display or for testing. Unexpected values will stand out if you ve grouped the expected values with a user-defined format when summarizing data. There are many powerful functions which allow you adjust the appearance of data, derive or increment intervals of date or time, manipulate character strings, or perform mathematical or statistical operations. General awareness of the sorts of functions which are available which you can look up as needed -- may save you the tedium of trying to code these operations the hard way. (See Cody, 2006 for examples.) OPTIONS which are set for a program or session as opposed to within a step stay in effect until the end of the session, unless they are switched again. For example, at first glance it might seem that the two pieces of code shown here have the same effect: 4

5 proc print data = WEATHER(obs=10); options obs = 10; proc print data = WEATHER; Consider, however, what would happen in the second and fourth occurrences of PROC PRINT shown here: proc print data = WEATHER(obs=10); title1 'First Print Example with limited number of obs'; proc print data = WEATHER; title1 'Second Print Example with limited number of obs'; options obs = 10; proc print data = WEATHER; title1 'Third Print Example with limited number of obs'; proc print data = WEATHER; title1 'Fourth Print Example with limited number of obs'; #6 THERE ISN T JUST ONE RIGHT WAY There s always more than one way to achieve your objective using SAS code. In fact, there may be more possibilities than there are SAS programmers. For example, there are ongoing debates over the relative merits of PROC SQL versus the DATA step, or what is the most efficient way to perform table look-ups. Decisions regarding the optimal approach for any program will involve trade-offs in resource utilization for both human and machine resources. In your environment, emphasis may be placed on the readability or maintainability of what you write, or you may face constraints in processing -- CPU time, elapsed time, memory or storage space. As you develop a program, immediate timelines may dictate using whatever method works as a short-term solution and revisiting the code to streamline it or improve the efficiency later. An over-riding factor will be whether you are comfortable with, and are able to verify the results of, whatever you re coding. There s lots of room for enhancement as you develop your personal repertoire of techniques over time. Exposure to quality works of others, such as in conference papers, should inspire your development. One of my favorite ways to review a complex program is to map the flow of data into and through the program steps. There are flow-charting software packages available, but I do it the low-tech way -- on paper drawing something like a crude flowchart with a node for each step or procedure and for each 5

6 dataset. Nothing makes opportunities for streamlining of code and processing more obvious. Of course, this would be helpful at the planning stage if a complex program were anticipated. #7 WALLPAPER BELONGS ON WALLS You may feel good the first time you develop a really long program and it achieves its purpose. Take a moment to celebrate your accomplishment, but then remember that bigger is not necessarily better. Developing a program iteratively by capturing interim steps from the Program Editor window may lead to a correct result at the end of a meandering path. It may be tempting, under time pressure, to stick with what works even though it may not be the most direct, efficient or comprehensible route. When time allows, you can review your code to look for wallpaper, or repeating blocks of code. There are two potentially significant opportunities for improvement here: to eliminate unnecessary code which can improve processing efficiency, and to streamline repetitive code which will make your program easier to read and maintain. This is especially important for repeatable processes over large volumes of data. Sometimes efficiency considerations will dictate a reorganization of the program to streamline, or the use of different procedures to transform data or produce statistics. If those things don t apply, or you still have repeated patterns of the same or similar code, then one or more macros may be appropriate. There are many ways to use macros to reduce the volume of code and make the program more readable and maintainable. At the most elementary level, macros can be used for text substitution, for anything from a single variable to a large chunk of repeating code. Here are a couple of uses of macros to simplify programs and the workload of whoever is running or maintaining those programs. Macro Variables Consider the situation where you have date selection or identification in several places in a program, all based a date which varies for each run, and each of the uses requires a slightly different format. You might use a macro variable %LET REPORT_DATE = at the top of the program for a single-source update, and then run through a DATA step to create all the necessary forms of the date. Macro Code A first step toward streamlining a program would be to move one set of all of the code which is in common to a macro which is called repeatedly from steps which contain only the lines of code which are different. In this case, a macro is treated like an INCLUDE module, where the same code gets included in multiple places. Similar to the way a macro variable can be assigned once and then used throughout a program, a block of code can be assigned a name and then used throughout the program by referring to it by name. In this way, macros can help to avoid repeating blocks of code. The block of code should sit outside of the boundaries of a PROC or DATA step. It should begin with a %MACRO statement and terminate with %MEND (for Macro End). Of course the macro needs to be defined prior to the point where it is first referenced in the program. Consider the situation you where you need print a listing for all observations in a dataset, then print a listing of the subset of observations where one condition is true, then another listing of the subset of observations where a different condition is true. You might end up with a lengthy program where most of the code is common, like PROC PRINT or PROC REPORT, with just a WHERE clause specific to the step. This brief discussion of Macro Variables and Macro Code should give you an idea that the topic of macros is a huge one, from individual variables through to a comprehensive macro language. Much about design and development of macro programs is an advanced topic, and many User Group papers, books and courses have been developed to showcase the amazing potential for using SAS macros. Refer to Dunn and Whitlock (2006) for a discussion of macro design considerations. If you aren t ready to tackle macro programming yet, just remember that it exists as a very valuable tool for future use. 6

7 #8 YES, YOU DO HAVE TO TEST Your first program which runs without syntax errors may feel like a reason to celebrate, but it s only a true victory if it also produces a correct result. Syntactical correctness doesn t guarantee logical correctness, so there s no reason or excuse for not validating the results from a newly-written program. There are many simple procedures which can be run against a SAS dataset to analyze your results. PROC FREQ or PROC MEANS can provide a quick tabulation of key variables. Data can be subset for more detailed review using an option to limit the number of observations or a WHERE clause, random samples can be selected, or data including automatic variables within a DATA step can be displayed using a PUT statement. PUT statements can be especially helpful for displaying automatic variables or the results of functions used in the DATA step. The statement PUT _ALL_; will dump the contents of the Program Data Vector to the log. A great way to explore functions is to create a small test dataset containing known values and then run through that, using PUT statements before and after the functions are invoked in the DATA step. My two favorite testing tools are the PUT statement in a DATA step, and the use of PROC FORMAT and a subsequent PUT function to classify data which has been fed into PROC FREQ. Another approach, if you can access your interim or result datasets in an interactive environment and they aren t too large, is filtering with a WHERE clause to quickly see the range of values and learn about your data. This might be particularly helpful if you re doing preliminary data analysis and programming at the same time. Reviewing and understanding the program log is always important, but especially so for a newly-written program. Error messages in the log are almost always a cause for concern. Log notes and warning messages may or may not be indicative of a problem. They warrant careful consideration: you need to understand your data to know whether a dataset created with zero observations is a problem, or a merge with multiple occurrences of the by-variables is reasonable or not. Consider including some errorhandling code or modifying the program to reduce notes and warnings and improve processing efficiency. Even syntactical and logical correctness of your program don t guarantee that the program achieves its purpose. Checking back to your statement of purpose will help ensure that you have reached your goal. If the goal has changed because of what you ve learned about the data or requirements during development, you can restate the purpose or at least the program documentation. #9 YOU ARE NOT ALONE Help is available; you are not alone in your quest for SAS programming knowledge even if you aren t working directly with other SAS programmers or you re at your computer at an hour when no one else in your time zone would be expected to be working. You ll find that there are lots of online documents, plus an active community of knowledgeable and helpful people who can and will provide advice with no expectation beyond your word of thanks. If you found this paper, you probably have some awareness of SAS user groups. They exist at the local level in major metropolitan areas of North America, as well as regional, global, special-interest and inhouse groups. User Group papers and presentations may be available in person or through web sites or conference Proceedings. To find a public SAS User Group, you can check the SAS Institute support website or To find papers from the past several years worth of some SAS User conferences, you can search by conference and author at If you peruse the Proceedings from a number of SAS User conferences, you may notice that there is often a Beginner Tutorials or Foundations track which collects more in-depth papers on topics you would be well-advised to explore, and that there are a number of authors who write and present repeatedly on related material, which means their papers are being refined over time. If you find an author whose paper is helpful, be sure to look for similar but updated work by the same person. The recently-enhanced search facility on this site allows a simultaneous search of multiple conferences and years, by subject or author. Of course SAS Institute provides a range of excellent resources for learning to use SAS. In the nearest form, an extensive Help library can be accessed by typing HELP in the command box of the Program 7

8 Editor Window. (You may, for example, type HELP CONTENTS PROCEDURE in the command box to locate syntax and examples for PROC CONTENTS.) Learning opportunities also include instructor-led or Web-based courses, and as a portal to resources. SAS-L is an list, giving you access a virtual community of real SAS users with a huge range of experience and expertise who may be able to answer your questions. Activity on the list spans the globe and the clock. Several members of this community are remarkably generous with their time, patience and near-encyclopedic knowledge of common -- or sometimes obscure -- topics. Anyone can search the archives of past questions and answers or post questions to the list. Threads of responses to past questions may include respectful disagreements on the best way to solve a problem. While this may be beyond the scope of what you want when you re facing a deadline, these may containing enlightening nuances. It further supports the point that there isn t just one right way. Don t be afraid to ask questions on SAS-L, it is open to anyone. In the interests of being a good citizen in the virtual community, though, you may want to search the archives to see whether your question has already been answered, and to learn a bit about the etiquette of the list. If you are going to post a coding or data manipulation question, be as specific as you can about your data and your requirements. Including code to create some test data which illustrate your question will help you explain the situation you re asking about and make it more convenient for someone to answer your question. Access to SAS-L is available via There are some excellent papers describing SAS-L and how to make use of it. (See Schreier, 2004.) is a Wiki, a web-based application which allows any community member to access and also contribute content. This application is maintained by the user community for the user community. It was launched by the SAS Global Users Group Executive Board in 2007, for the benefit of the international SAS community. It is a rapidly-growing collaborative repository. If you don t have an immediate need for SAS help or information, but want to bookmark just one link to SAS resources just in case you ever need it, consider That will give you links to the major online resources for all things SAS, on one page. Reading conference papers, using SAS Institute Help facilities, searching SAS-L and sascommunity.org happen to be solitary activities. That is not to suggest that you need to tackle all your SAS questions alone with your computer. Don t forget to take advantage of the skills of your colleagues and people you may meet at User Groups as well. Just as with the online community, though, you will want to respect the time and efforts of others by considering whether the answer to your question is readily accessible elsewhere. There are lots of options, so where you seek information depends on your preferred learning style, timeline, budget and whether you are seeking something as specific as syntax or help with a particular product or option on a particular platform, or as general as design advice. #10 SOMETIMES YOU HAVE TO ASK FOR IT Now that you have some advice on getting started, you should be feeling empowered to learn with some independence. There s just one more step before you can apply what you ve learned. Before you can start programming, you need to know some things about your operating environment. This atmospheric stuff is like the technical equivalent of understanding a little bit about the local language and culture when you travel. It s important to recognize what you couldn t possibly be expected to figure out on your own, and when you need to ask for help locally. While you may get some guidance from members of the SAS community outside of your organization, some things are very site-specific. Since very few machines are stand-alone and unsecured, you need to be told -- or ask -- what your user ID and initial password are, and to make sure you have access to SAS and to the data you ll need for your work. This can be a huge stumbling block, because it s not intuitive: if you don t already have a user ID for the system, it s not likely you ll be able to guess what it is, even if some has already set it up for you. Similarly, you d probably never be able to guess what the file names, library names or database names are. 8

9 So, if you only have three questions which you may ask of the proverbial genie -- or the harried technical advisor at your site -- a good use of those questions would be: What is my user ID and what is the initial password for it? Where can I safely store my programs and data? Can you direct me to any existing programs I can look at as examples? Those are absolutely reasonable questions, for which you can t be expected to know the answers. The third question may seem like cheating, but it isn t. It s not to suggest that you re going to copy what s been done before, although there s no harm in reusing some well-written code that belongs to your organization. The answer may provide invaluable clues to data sources, naming conventions or utility libraries. The answers to all three questions would be relevant to even the most experienced SAS programmer who is faced with a new environment, whether it s a new organization or a new operating system or database. CONCLUSION You should now have some ideas on how to proceed and where to turn when you encounter inevitable obstacles or are ready to try something new. The community of SAS users is strong and there are many generous members who have contributed free content and continue to provide more. Look around and learn from the best of what s out there. REFERENCES AND RESOURCES This is a very small subset of recent personal favorites from the hundreds of beginner-level User Group papers which are available publicly available on the internet. Depending on where you are in your own skills development, you may find some of these to be helpful: Bilenas, J. The Power of PROC FORMAT; Updated for SAS 9 Proceedings of the NorthEast SAS Users Group Inc. Eighteenth Annual Conference Cody, R. An Introduction to SAS Character Functions Proceedings of the NorthEast SAS Users Group Inc. Nineteenth Annual Conference Dunn, T and Whitlock, I. "Beginning Macro Design Issues" Proceedings of the NorthEast SAS Users Group Inc. Nineteenth Annual Conference Howard, N. How SAS Thinks or Why the DATA Step Does What It Does Proceedings of the NorthEast SAS Users Group Inc. Seventeenth Annual Conference Mitchell, R.M. "Finding Your Mistakes Before They Find You: A Quality Approach For SAS Programmers" Proceedings of SAS Global Forum 2007 Conference Schreier, H. "Ask This Old Newsgroup" SAS Users Group International Proceedings Whitlock, M. How To Get Around the SAS Customer Support Web Site Proceedings of the NorthEast SAS Users Group Inc. Nineteenth Annual Conference

10 Whitlock, I. How to Think Through the SAS DATA Step Proceedings of the NorthEast SAS Users Group Inc. Nineteenth Annual Conference ACKNOWLEDGEMENTS Thanks to Tim Trussell of SAS Canada and my fellow members of the TASS leadership team for informal input, Isabelle Hastie of RBC for asking challenging questions, and the many colleagues and clients I ve worked with who have learned their lessons the hard way and then been willing to share their knowledge. SAS and all other SAS Institute Inc. product or service names are registered trademarks or trademarks of SAS Institute Inc. in the USA and other countries. indicates USA registration. Other brand and product names are registered trademarks or trademarks of their respective companies. CONTACT INFORMATION Your comments are welcomed. Happy programming! Lisa Eckler lisa.eckler@sympatico.ca 10