SAS and SQL "Join" Forces to Perform a Many-to-Many Merge Ginny P. Lai, Eric P. Elkin ICON Clinical Research, San Francisco, CA

Transcription

1 SAS and SQL "Join" Forces to Perform a Many-to-Many Merge Ginny P. Lai, Eric P. Elkin ICON Clinical Research, San Francisco, CA ABSTRACT Many-to-many merge refers to specific instances where there is a need to merge two or more data sources with multiple observations for key identifiers. For example, it would be important to merge a dataset containing one record per patient per medical visit and a dataset containing one record per patient per lab test for the purpose of identifying the medical visit closest to each lab result for each patient. SAS generally can not handle a many-to-many merge properly by a simple MERGE-BY statement (although a complex MERGE accompanying an advanced DATA step can be used). In this situation, an easier and more straightforward alternative is to employ PROC SQL under SAS/Base. This talk will focus on basic PROC SQL syntax for non-sql programmers in order to perform a many-tomany merge. INTRODUCTION At times it is necessary to perform a many-to-many merge in contrast to a one-to-one or one-to many merge. In this case, a programmer needs to combine all the rows from one table with all the rows from another table. A MERGE accompanying an advanced DATA step can be used but this is complex and time consuming. PROC SQL presents a powerful alternative tool which is much simpler and more straightforward for performing a many-to-many merge. The purpose of this paper is to demonstrate many-to-many merging using basic PROC SQL syntax for SAS users with knowledge of DATA step but little-to-no PROC SQL experience. SAMPLE DATA In this paper, two datasets (shown below) will be used as our example to illustrate a many-to-may merge. Imagine we have a dataset of medical visit data and a dataset of lab result data. First, we want to find all visit records within five days of each lab date for each patient. Note that both datasets have a common patient ID (PATID) with multiple visits or lab results per patient. PATID VID VISDATE VISTYP E Jan Jan Feb Feb Mar Mar Mar Apr Jan Jan Feb Mar Mar Mar Mar Jan Jan Apr Feb

2 Feb Mar Mar Apr Apr Apr May Jan Jan Feb Feb Feb Mar Mar Mar Apr Apr Apr Apr May patid labid labdate pcs mcs Jan Feb Mar Apr May Jan Feb Mar Apr May Jan Feb Mar Apr May Jan Feb Mar Apr May Jan Feb Mar Apr May

3 MERGING VS. JOINIG The following output shows what happens when you combine datasets with multiple records per unique key using a traditional MERGE-BY statement. Take patient 3 for example: the two datasets match one-on-one in the original order per observation until the medical visit dataset runs out of observations and carries over the last record through the rest of the merging with the lab data. patid vid visdate vistype labid labdate pcs mcs JAN JAN JAN FEB APR MAR APR APR APR MAY By using PROC SQL, the code presented below demonstrates the Cartesian product via a JOIN-ON clause. JOIN allows us to see all the possible combinations which satisfy the ON expression between two source tables VISIT01 and LAB01 (more details about different types of JOIN can be found in the recommended reading.) Here, the asterisk (*) after SELECT selects all the variables from each dataset. The QUIT statement then turns off the SQL facility at the end. Note that the datasets do not need to be sorted beforehand. proc sql create table temp01a as select * from visit01 join lab01 on visit01.patid = lab01.patid Observe the differences between the listings produced using a MERGE-BY statement versus PROC SQL. TEMP01A now has 15 rows for patient 3 due to the merger of 3 records from VISIT01 with each of the 5 records from LAB01. patid vid visdate vistype labid labdate pcs mcs JAN JAN JAN JAN APR JAN JAN FEB JAN FEB APR FEB JAN MAR JAN MAR APR MAR JAN APR JAN APR APR APR JAN MAY JAN MAY APR MAY ALTERNATIVE SYNTAX Another way to perform a many-to-many merge is through a comma separated list and a WHERE statement. This works in the specific case when you want each record in one dataset merged to each record in another dataset (restricted only by the WHERE clause.) However, this syntax many not work for more complex joins. proc sql create table temp01b as select * from visit01, lab01 where visit01.patid = lab01.patid 3

4 A short temporary nickname could also be given in a FROM clause using the keyword AS after the actual dataset name. This is especially useful in later demonstrations for simplifying the code. proc sql create table temp02 as select * from visit01 as v, lab01 as l where v.patid = l.patid Notice here SQL table TEMP01A, TEMP01B, and TEMP02 are identical. KEEPING ONLY CERTAIN VARIABLES One method to limit our variables in our final table is through the SELECT clause. Namely, use the dataset name followed by the variable of interest. Using a nickname here eliminates the number of keystrokes. proc sql create table temp03 as select v.patid, v.visdate, l.patid, l.labdate, l.mcs, l.pcs from visit01 as v, lab01 as l where v.patid = l.patid REMAINING STEPS Adding another restriction in the WHERE clause instructs SAS to keep the rows only if visit dates and lab dates are within 5 days of each other. This restriction can also be applied in a dataset after creating TEMP03 however, it is more efficient using PROC SQL. proc sql create table temp04 as select v.patid, v.visdate, l.patid, l.labdate, l.mcs, l.pcs from visit01 as v, lab01 as l where (v.patid = l.patid) and ( abs(l.labdate - v.visdate) <= 5) The code produces the following TEMP04 table: patid labdate visdate mcs pcs 1 15JAN JAN FEB FEB MAR MAR MAR MAR APR APR JAN JAN FEB FEB MAR MAR MAR MAR JAN JAN JAN JAN APR APR FEB FEB MAR MAR APR APR

5 4 15APR APR MAY MAY JAN JAN FEB FEB FEB FEB MAR MAR MAR MAR APR APR APR APR MAY MAY Finally, we come to our last step to find the closest visit date to the lab date. The steps below ensure that we take the visit date prior to the lab date if there is a tie. data temp05 set temp04 datediff = visdate-labdate datediff_abs = abs(datediff) proc sort data=temp05 out=temp05_s by patid labdate datediff_abs datediff data temp06 set temp05_s by patid labdate datediff_abs datediff if first.labdate The resulting listing for TEMP06 should look like the following. datediff_ patid labdate visdate abs datediff mcs pcs 1 15JAN JAN FEB FEB MAR MAR APR APR JAN JAN FEB FEB MAR MAR JAN JAN APR APR FEB FEB MAR MAR APR APR MAY MAY JAN JAN FEB FEB MAR MAR APR APR MAY MAY CONCLUSION When it comes to combining data sources, the MERGE-BY statement and PROC SQL joins are very different and are viewed as two alternatives. The MERGE method was designed to manipulate sorted datasets. It basically matches records one-to-one. On the other hand, PROC SQL was designed to return a result table from non-sorted data with an all-to-all record match. The ability of PROC SQL to create Cartesian products and its simplicity of coding makes it preferable in some cases as a stepping stone to a more meaningful result. Accordingly, this paper serves as an introduction to familiarize the programmer with the basic SQL syntax to perform a many-to-many merge. ACKNOWLEDGMENTS 5

6 Many thanks to Sarah Short and George Sen for reviewing drafts of this paper. RECOMMENDED READING SAS Institute Inc., SAS 9.1 SQL Procedure User s Guide, Cary, NC: SAS Institute Inc., CONTACT INFORMATION Your comments and questions are valued and encouraged. Contact the authors at: Ginny P. Lai ICON Clinical Research 188 Embarcadero, Suite 200 San Francisco, CA Work Phone: (415) ginny.lai@iconplc.com Web: Eric Elkin ICON Clinical Research 188 Embarcadero, Suite 200 San Francisco, CA Work Phone: (415) eric.elkin@iconplc.com Web: SAS and all other SAS Institute Inc. product or service names are registered trademarks or trademarks of SAS Institute Inc. in the USA and other countries. indicates USA registration. Other brand and product names are trademarks of their respective companies. 6