Preparing your data for analysis using SAS. Landon Sego 24 April 2003 Department of Statistics UW-Madison

Transcription

1 Preparing your data for analysis using SAS Landon Sego 24 April 2003 Department of Statistics UW-Madison

2 Assumptions That you have used SAS at least a few times. It doesn t matter whether you run SAS in interactive mode (Windows) or in batch mode (Unix/Linux).

3 Interactive SAS for Windows

4 Editing SAS code in EMACS for Batch mode execution

5 Executing SAS in batch mode at the Linux prompt

6 Where we re going Rarely does data come to you in a form that is analyzable. As a best case, all you need to do is clean your data and check it for consistency. As a worst case, extensive manipulation of the data is needed in order to analyze. We want to get familiar with some tools in SAS used to check, clean, and manipulate data.

7 The dark toolbox SAS is like a toolbox as big as a garage with thousands of tools. For many SAS users, this toolbox is dark and there is a monster lurking inside. Let s turn on the light and meet a few of the tools available in SAS. No guarantees about the monster

8 Published resources Lots available I ve learned on my own and with the SAS documentation Google searches

9 SAS Online Documentation or

10 What I use most often in the SAS Online Documentation Base SAS Software SAS Language Reference: Concepts SAS Language Reference: Dictionary SAS Macro Language Reference SAS Procedures Guide SAS/STAT SAS/STAT User s Guide

11 SAS Language Reference: Dictionary

12 SAS Procedures Guide

13 SAS/STAT User s Guide

14 Conventions, terminology, and options

15 Conventions SAS terminology in red SAS code in blue

16 Basic terminology Data in SAS exists as a data set, with variables and observations. variables are the columns. observations are the rows. Two types of variables: Character and numeric. Character variables can range in length from 1 to 32,767 = 2 15 characters. Numeric variables can be virtually any size (within the limitations of the computer)

17 My favorite options On the first line of almost every SAS program I write, I include the following: options nodate nocenter nonumber ps=3000 ls=200 mprint mlogic symbolgen; These options control the format of the output and make macro code easier to debug

18 Importing data into SAS

19 Importing data into SAS Data can exist in many forms (text file, Excel spreadsheet, permanent SAS data set, etc.) Excel spreadsheets are probably the most common form. Can use DDE (dynamic data exchange) (Windows version of SAS only) But for Excel files, I like to use CSV file format (comma separated value). Works on any platform.

20 Excel CSV text file SAS data set Column of j s provide a buffer at the end of each line of text in the CSV file. If you are running SAS on a Linux or UNIX machine, you need to add the j s (or use the dos2unix command to convert the CSV file to the text formatting used by UNIX).

21 Save Excel spreadsheet in the CSV format

22 How the CSV file looks (when viewed with a text editor) Location,Type,Length,j Albuquerque,1,1.414,j Albuquerque,1,2.000,j Albuquerque,1,1.414,j Albuquerque,1,2.236,j Albuquerque,2,2.000,j Albuquerque,2,2.236,j Albuquerque,2,2.236,j Albuquerque,2,2.236,j Lexington,1,2.000,j

23 SAS code to import the CSV file data fake; infile c:\mydata\fake.csv dsd firstobs=2; input location :$14. type length; proc print; run; Note, if I were to use: input location $ type length; it would truncate the location variable to 8 characters in length.

24 Results from the proc print Obs location type length 1 Albuquerque Albuquerque Albuquerque Albuquerque Albuquerque Albuquerque Lexington Lexington Lexington Lexington Lexington Lexington Lexington Johannesburg Johannesburg Johannesburg Johannesburg Johannesburg Johannesburg

25 Checking and summarizing data

26 Checking and summarizing data This is IMPORTANT! Don t take for granted that there aren t mistakes in your data. proc contents data=fake; proc freq data=fake; run;

27 Result from proc contents Data Set Name: WORK.FAKE Observations: 19 Member Type: DATA Variables: 3 Engine: V8 Indexes: 0 Created: 11:46 Wednesday, April 16, 2003 Observation Length: 32 Last Modified: 11:46 Wednesday, April 16, 2003 Deleted Observations: 0.. File Name: /tmp/sastmp_sego/sas_workb6b _ gstat201.stat.wisc.edu/fake.sas7bdat Alphabetic List of Variables and Attributes----- # Variable Type Len Pos length Num location Char type Num 8 0

28 The FREQ Procedure Results from proc freq Cumulative Cumulative location Frequency Percent Frequency Percent Albuquerque Johannesburg Lexington Cumulative Cumulative type Frequency Percent Frequency Percent Cumulative Cumulative length Frequency Percent Frequency Percent

29 Selecting subsets of the data

30 Selecting observations (rows) A large contiguous group of observations Specific observation numbers Using selection criteria e.g. when the location is Lexington or when the length is between 1 and 2.

31 Selecting a group of contiguous observations data smallfake; set fake (firstobs=10 obs=15); proc print; Data set options Obs location type length 1 Lexington Lexington Lexington Lexington Johannesburg Johannesburg Selects observations 10 through 15

32 Selecting specific observation numbers data smallfake; set fake; if _n_ in (7,11,16); proc print; Obs location type length 1 Lexington Lexington Johannesburg Selects observation numbers 7, 11, and 16.

33 Selection criteria: where statement data smallfake; set fake; where location = Lexington ; or where location ne Lexington ; or where location in ( Lexington, Albuquerque ); or where (1 le length le 2); or where (length > 2.3);

34 Selection criteria: if statement data smallfake; set fake; if location in ( Lexington, Albuquerque ); or if location = Lexington location = Albuquerque ; or if location = Johannesburg then delete; These three if statements produce identical results.

35 Some comparison operators ne or ^= not equals to eq or = equals to ge or >= greater than or equal to gt or > greater than le or <= less than or equal to lt or < less than in if contained in a group not in if not contained in a group and or & and logical operator or or or logical operator

36 Selecting and managing variables

37 Selecting variables using keep data smallfake (keep = location length); set fake; where type = 1; (type is available for processing, but not written to smallfake data set) Data set options data smallfake; set fake (keep = location length); (type is not available for processing)

38 Selecting variables using drop data smallfake (drop = type); set fake; where type = 1; (type is available for processing, but not written to smallfake data set) Data set options data smallfake; set fake (drop = type); (type is not available for processing)

39 Renaming variables location type data fake1; set fake (rename=(location=place type=trt)); where trt = 1; trt place data fake2 (rename=(location=place type=trt)); set fake; where type =1; data fake3 (drop = location type); set fake; where type = 1; place = location; trt = type; These three pieces of code achieve the same result. Look closely at the where statements.

40 Concatenation

41 Concatenation (stacking) SAS can stack multiple data sets on top of one another. Pay attention whether or not the variables and their attributes (length and variable type) match among the different data sets. Can use the set statement or proc append to concatenate data sets.

42 Using the set statement to concatenate data sets Suppose you wanted to stack the three data sets fake, faux, and fraud on top of one other: data fantastic; set fake faux fraud;

43 Using proc append to concatenate data sets proc append concatenates only two data sets at a time and typically these data sets must have the same variable names with the same attributes. proc append base=fake data=faux; Here the observations in faux are tacked onto the end of the fake data set. The combined data set is called fake.

44 Splitting data into several data sets Supposed we want all Albuquerque observations to go into a data set called albuq, the Lexington observations to go into the data set lexing, and observations that have lengths larger than 3.0 into the data set large. data albuq lexing large; set fake; if location = Albuquerque then output albuq; else if location = Lexington then output lexing; if length gt 3 then output large;

45 Merging data

46 Merging (combining) data Merging data sets places two or more data sets side by side into a single data set. If you simply want place two data sets side by side (1 to 1 merging): data faux; set faux (rename=(location=location1 type=type1)); data fantastic; merge fake faux; proc print data = fantastic;

47 Results of 1 to 1 merge Obs location type length location1 type1 weight 1 Albuquerque Lexington Albuquerque Lexington Albuquerque Lexington Albuquerque Lexington Albuquerque Lexington Albuquerque Lexington Lexington Lexington Lexington Johannesburg Lexington Johannesburg Lexington Johannesburg Lexington Johannesburg Lexington Johannesburg Lexington Johannesburg Johannesburg Albuquerque Johannesburg Albuquerque Johannesburg Albuquerque Johannesburg Albuquerque Johannesburg Albuquerque Johannesburg Albuquerque

48 Merging data with a by variable All data sets that will be merged must first be sorted by the linking variables: proc sort data = fake; by location type; proc sort data = faux; by location type; data fantastic; merge fake faux; by location type; proc print data = fantastic;

49 Results of merging with by variables Obs location type length weight 1 Albuquerque Albuquerque Albuquerque Albuquerque Albuquerque Albuquerque Johannesburg Johannesburg Johannesburg Johannesburg Johannesburg Johannesburg Lexington Lexington Lexington Lexington Lexington Lexington Lexington

50 More about merging When you merge with a by statement, you may only want observations that have by-variable matches in both data sets. fake fraud

51 Using (in =) data set option Assume both fake and fraud are sorted by location and type. data fantastic; merge fake (in = tmp1) fraud (in = tmp2); by location type; from_fake = tmp1; from_fraud = tmp2; proc print;

52 Identifying obs from both data sets from_ from_ Obs location type length thickness fake fraud 1 Albuquerque Albuquerque Albuquerque Albuquerque Albuquerque Albuquerque Albuquerque Albuquerque Albuquerque Johannesburg Johannesburg Johannesburg Johannesburg Johannesburg Johannesburg Johannesburg Johannesburg Lexington Lexington Lexington Lexington Lexington Lexington Lexington Lexington Lexington

53 Using (in =) data set option Now select observations that are common to both data sets: data fantastic; merge fake (in = tmp1) fraud (in = tmp2); by location type; if tmp1=1 and tmp2=1; proc print;

54 After selecting for observations in common: Obs location type length thickness 1 Albuquerque Albuquerque Albuquerque Johannesburg Johannesburg Johannesburg Johannesburg Johannesburg Johannesburg Lexington Lexington Lexington

55 Merge mania data fantastic; merge fake fraud; by location type; data fantastic; merge fake (in=tmp1) fraud; by location type; if tmp1 = 1; data fantastic; merge fake (in=tmp1) fraud (in=tmp2); by location type; if tmp1 =1 and tmp2 = 1;

56 Creating new variables

57 Creating new variables data fantastic; merge fake faux; by location type; newcode = substr(location,1,1) '-' trim(left(type)); growth_index = length + weight**2; if (growth_index gt 15) then large = '*'; else large = ' ';

58 Results of new variables growth_ Obs location type length weight newcode index large 1 Albuquerque A Albuquerque A Albuquerque A Albuquerque A Albuquerque A Albuquerque A Johannesburg J Johannesburg J Johannesburg J Johannesburg J Johannesburg J Johannesburg J * 13 Lexington L Lexington L * 15 Lexington L * 16 Lexington L Lexington L * 18 Lexington L Lexington L *

59 Common functions used to manipulate text strings compress index left scan substr trim Refer to SAS Online Docs: Base SAS Software SAS Language Reference: Dictionary Dictionary of Language Elements Functions and Call Routines

60 by-group processing

61 by-group processing Suppose you wanted a subset of the data that contained the observation with the smallest length from each location. proc sort data = fake; by location length; data shortest; set fake; by location length; first = first.location; last = last.location;

62 output from by-group processing Obs location type length first last 1 Albuquerque Albuquerque Albuquerque Albuquerque Albuquerque Albuquerque Johannesburg Johannesburg Johannesburg Johannesburg Johannesburg Johannesburg Lexington Lexington Lexington Lexington Lexington Lexington Lexington

63 by-group processing proc sort data = fake; by location length; data shortest; set fake; by location length; if first.location = 1; Obs location type length 1 Albuquerque Johannesburg Lexington

64 Basic macros

65 Basic macros SAS macros allow you to easily program repetitive tasks. On the surface, creating a SAS macro is very similar to creating a function in R or S-Plus. SAS Macro is actually a text generation tool.

66 Name macro, begin macro definition, identify macro variables Code to be generated by macro Note the use of proc append %macro analyze(dataset=,response=); proc mixed data = &dataset; class location type; model &response = location type; lsmeans location type; ods output lsmeans=model_means; data model_means; set model_means; variable = "&response"; Macro example proc append base=results data=model_means; End macro definition Call macro Print results %mend analyze; %analyze(dataset=fake,response=length) %analyze(dataset=faux,response=weight) proc print data = results;

67 Results from macro code Obs Effect location type Estimate StdErr DF tvalue Probt variable 1 location Albuquerque _ <.0001 length 2 location Johannesburg _ <.0001 length 3 location Lexington _ <.0001 length 4 type <.0001 length 5 type <.0001 length 6 location Albuquerque _ <.0001 weight 7 location Johannesburg _ <.0001 weight 8 location Lexington _ <.0001 weight 9 type <.0001 weight 10 type <.0001 weight

68 proc transpose

69 Rearranging data with proc transpose Consider this output from proc mixed: Obs year cultivar Effect trt Estimate StdErr DF tvalue Probt Intercept _ trt calevel _ calevel*trt Intercept _ trt calevel _ calevel*trt Intercept _ trt calevel _ calevel*trt

70 Results from proc transpose proc transpose data=solution out=tsolution; by year cultivar; var estimate; id effect; calevel_ Obs year cultivar _NAME_ Intercept trt calevel trt Estimate Estimate Estimate Estimate Estimate Estimate Estimate Estimate Estimate

71 Parting words of advice

72 Attributes of SAS SAS is read/write intensive. Every time you create a data set, the data set is written to the disk. Where does it get written? To the SAS Work Library, which is assigned to a directory somewhere..use proc contents to find out. For CALS HP users and PC users, the SAS Work Library resides on the actual machine.

73 Attributes of SAS Users of the AFS system beware! (Stat department, CS department) The SAS Work Library is assigned to your account in AFS not to the local machine that is running SAS. Your local computer running SAS Network traffic AFS recording and reading your SAS data sets

74 Assigning the SAS Work Library To assign the SAS work library to a local directory (when running in batch mode on a Linux or Unix system): $ sas mysascode.sas -work /scratch

75 Synthesis Most of what we ve covered today involves the data step. Many of the techniques shown in this presentation can be applied together in a single data step. Now that you know the names of some of the tools, use the Online Documentation!