Taking Almost Raw Lab data and Turning it into Data Fit to be Analyzed

Transcription

1 Taking Almost Raw Lab data and Turning it into Data Fit to be Analyzed ABSTRACT By Jeffrey K. Magouirk The purpose of the Data Coordinating Center at National Jewish Medical and Research Center is to collect data for patient, drug and medical device studies. These studies examine compliance issues, quality of life measures and clinical trials. Usually, the increase in workload for a new study, especially a clinical trial, means that the workload placed upon the human resources available is greater than a proportional increase of just adding one study. As a method to use these scarce resources in a more efficient manner, processes that previously taken human intervention were automated. A process that needed automation was the test results from the Pulmonary Function Test (PFT) or Spirometry data. This test measures different types of lung volumes. Before this process was automated, the data was hand entered onto a paper case report form (CRF). The paper forms were scanned and made into electronic images. These images were inputted into optical reading software that turned the fields on the form into numbers and characters or question marks were the software could not read the handwriting. Operators then verified the numbers and characters and corrected the question marks. After verification was completed, the data was submitted to a database. The purpose of this paper is to show how SAS and the use of DOS scripting are used to automate this example of data collection. The automating of the results of the PFT data saves both the time and effort of the scarce human resources. These savings of time and effort were from the reduction of hand copying the data from a machine screen print and the now non-existence need for verification. There were also increases in the quality of the data, since errors from transcription of the data and verification were now a non-issue. INTRODUCTION Data collection performed for a long-term study of lung functions has proven to be dynamic effort. Part of efforts performed can be attributed to the changes in technology and people overtime. Since changes have occurred the process of collecting this data needed to evolve almost continually. In the mid-eighties the first data collected was accomplished via a double-data entry system from paper CRFs. This system worked well until the late nineties when it was replaced by to a hand filled-in electronic CRFs that an operator verified. The verification process would check for errors of handwriting and a software would help to find some of the handwriting errors. After verification was done, the data was then inputted into a database. Again, this was good, but data errors were also part of the process, and errors wanted to be reduced or brought to zero. In early 2001 a new Pulmonary Testing Machine also know as a Spirometry machine, was purchased and its use began. Since this machine captured and stored the data electronically, it was decided that instead of transferring the data to paper and then 1

2 scanning the results into a database after human verification. The accuracy of the data would be greater if the data from the Pulmonary Testing machine could be done electronically. To accomplish this task took a great deal of effort and thought went into doing it. The savings in effort after the data collection was done electronically and the reduction in errors provided a greater savings in time and effort than were expended to automate the task. DATA FLOW Patient is given a Pulmonary Function Test Data Flows into machine which patient is tested on PFT results are requested to be put into SQL database SQL-Server Database sends request for PFT data PFT results are a colon delimited file DOS scripting reads the names of PFT results and creates a file. This file is read by SAS to parse the data to be placed into a SQL database TRANSFERRING THE DATA The process of transferring the data from the Spirometry machine to a table in MS SQL- Server took two steps. The first step was moving the data from the servers that were used by the Pulmonary Testing Function machine onto servers that the database was located. 2

3 The next step was to have SAS read the data into the SQL-server. To accomplish this task DOS scripting and a SAS program were used. DOS Scripting The DOS scripting used in the process of transferring the data was simple and easy to write. The script would copy the data from the servers that held the results from the Spirometry machine. A command would copy the testing results onto a directory on servers in the Biostatistics area. The script would then create the names of those files in that directory into a text file. After the text file was created, the script would call and start SAS. Below is the DOS script used to start the transferring of data from the Spirometry machines to a database. DOS Copy commmand Copy \\Mt_YALE\PPU\SharedDATA\ILD\*.csv \Shared\Data\Spirometry_PFT\ILD" Directory were copied files are to be put "\\Longs_Peak Orginal Data directory DOS command to display the contents of a directory of all csv files DIR "\\Longs_Peak\Shared Data\Spirometry_PFT\ILD\*.csv" /b >> "\\Longs_Peak\shared Data\Spirometry_PFT\ILD\ildpft.txt" DOS command to redirect the output from the DIR command to a file named ildpft.txt Calls the SAS program Spirometry_Read_yager.sas and excutes it. "C:\Program Files\SAS\SAS 9.1\sas.exe" -sysin "\\Bios_data\shared data\spirometry_pft\programs\spirometry_read_yager.sas" 3

4 SAS Coding The file named ildpft.txt is the start of bringing the data into SAS. The data is brought into SAS so it can be formatted correctly and then put into an SQL-Server table. Below is an example of the ildpft.txt file csv csv csv csv These are the names of the files that need to be brought into SAS and processed. Regular code and a macro are used to bring the files into SAS. After the spirometry files are brought into SAS they are reformatted so these can be put into SQL-Server. The first part of the program is regular code that reads in the ildpft.txt file with the names of the spirometry results. filename spiro"\\longs_peak\shared data\spirometry_pft\ild\ildpft.txt"; libname ILDSQL odbc dsn='ild'; data files; informat namefiles $char40.; infile spiro end=done missover; namefiles ; i+1; filex=scan(namefiles,1,.); call symput('file2read' left(put(i,8.)),trim(namefiles)); call symput('nameoffile' left(put(i,8.)),trim(filex)); call symput('num',i); run; 1) Explanation of the code shown above. Filex is a variable created by using the scan function that leaves out the file extension, in this case csv. The first call symput trims, left justifies it and assigns the macro variable named file2read from the namefiles variable. The second call symput trims, left justifies and assigns the macro variable named nameoffile from the filex variable Both of these first two call symput actions are done to make the data look better to the human eye when in the dataset, but more import is to create macro variables used later in the program. The last call symput function will number each record in the file named files and be used in the next modular of code. 4

5 /***********************************************************/ /******Reading in the data of the PFT data ***************/ %macro readfiles; %do i=1 %to # data n&&nameoffile&i; infile "\\Longs_Peak\shared data\spirometry_pft\ild\&&file2read&i" delimiter=':' MISSOVER DSD lrecl=32767 ; informat labval best32. ; informat pred $15. ; informat baseline $4. ; informat precpred $4. ; informat post $4. ; informat precpredpost $4. ; informat change $4. ; format labval best12. ; format pred $15. ; format baseline $4. ; format precpred $4. ; format post $4. ; format precpredpost $4. ; format change $4. ; input labval pred baseline precpred post precpredpost change ; if _ERROR_ then call symput('_efierr_',1); /* set ERROR detection macro variable */ %end; run; %mend readfiles; %readfiles; 2) Explanation of the creation of the macro code shown above. The %do i=1 %to # command will run the macro readfiles until the number of times through the code is equal to the number of records on the files dataset. This is also known as an iterative do loop. The data n&&nameoffile&i; statement will write new files with the naming convention that is used in the directory from which the ildpft.txt file was created. The doubleampersand macro variable, shown here as n&&nameoffile&i, is the macro equivalent of an array or vector of values. i The &&file2read&i is the macro variable that is the names of the raw data files that are read into SAS to be reformatted. Again, the double-ampersand macro variable, &&file2read&i, is the macro equivalent of an array or vector of values. 5

6 Example of the how the double-ampersand works in the case of reading in the raw data files &I (subscript) &&nameoffile&i (arrary element) element value or name read of file read into dataset 1 &file2read csv 2 &file2read csv 3 &file2read csv Inserting the data into SQL-Server To insert the test results into a table on SQL-Server the PROC SQL command was used. PROC SQL was used to instead of the PROC APPEND statement since all of the variables wanted in the SQL-Server table could be shown, unlike with the other PROC. The form of it was PROC SQL; INSERT INTO datatable (variable1,.variablen) SELECT variable1, varablen FROM datasetx; QUIT: The actual code used for this example is below. proc sql; insert into ildsql.pft_data_uco(uco_num,chk_dig,visit_nu,testdate,age,ht,weight, Test_LOC,BLANKFRM,TESTMeth,PIMXPRE0,PEMXPRE0, prbfev1,prbfv1fc,prbfvc,prbff25,prbff50,prbff75, prff2575,pbpvcur,prefrctg,prerv,pretlc,presvc,prbraw, prbsgaw,pbpvccr,pbpvca,pbpvcb,predlco,predlcov,pimxpred, pemxpred,pimaxobs,pemaxobs,predtlc,predfrct,predrv,predsvc, predfvc,predfev1,predfv1f,predff25,predff50,predff75, predraw,predsgaw,ppdpcvck,ppdpvccr,psbtlc,psbfrctg,psbrv,psbsvc,srbfvc, psbfev1,psbfv1fc,psff2575,psbff25,psbff50,psbff75,psbraw,psbsgaw,ppdlco,ppdlcova,inputdate) select uco_num,chk_dig,visit_nu,testdate,age,ht,weight, Test_LOC,BLANKFRM,TESTMeth,PIMXPRE0,PEMXPRE0, prbfev1,prbfv1fc,prbfvc,prbff25,prbff50,prbff75, prff2575,pbpvcur,prefrctg,prerv,pretlc,presvc,prbraw, prbsgaw,pbpvccr,pbpvca,pbpvcb,predlco,predlcov,pimxpred, 6

7 pemxpred,pimaxobs,pemaxobs,predtlc,predfrct,predrv,predsvc, predfvc,predfev1,predfv1f,predff25,predff50,predff75, predraw,predsgaw,ppdpcvck,ppdpvccr,psbtlc,psbfrctg,psbrv,psbsvc,s rbfvc, psbfev1,psbfv1fc,psff2575,psbff25,psbff50,psbff75,psbraw,psbsgaw, ppdlco,ppdlcova,inputdate from work.njc_uco_numb_b; quit; Conclusions Using DOS Scripts to develop a file that SAS could then read which contained the names of the raw data files was very useful. This was a major savings in time and effort when the number of files became large. When the number of files that needed to be read into SAS was large it became overwhelming for an analyst to hand input the files that needed to be read into SAS. The use of the double-ampersand was, again, a great savings in time and effort on the analyst part. Having SAS read the file that was created by the DOS scripting so the files could then be brought into SAS and then be put into SQL-Server was great. The use of PROC SQL was advantageous since only the variables that were to be put into SQL-Server could be identified and this was another savings in time and effort. i Art Carpenter, Carpenter s Complete Guide to the SAS Macro Language, Cary, NC: SAS Institute Inc., pp., pp73 Contact Information Jeffrey K. Magouirk Coordinator Data Coordinating Center Department of Biostatistics National Jewish Medical and Research Center 1400 Jackson Street Denver, Colorado Work Phone: magouirkj@njc.org SAS and all other SAS Institute Inc., product or service names are registered trademarks or trademarks of SAS Institute Inc. in the USA and other countries. indicates USA registration. 7