Open-source software for geodetic imaging: ROI_PAC for InSAR and pixel tracking

Transcription

1 Open-source software for geodetic imaging: ROI_PAC for InSAR and pixel tracking Authors: Matthew E. Pritchard; and contributors to the ROI_PAC wiki Version: 0.4 May, 2014 Deformation Discovery: Interferogram from southern Peru showing ground displacement from at least 3 different origins within 50 km. The deformation in the upper left was expected as this interferogram spans an earthquake swarm that occurred in October, However, this data show that the locations from some global earthquake catalogs are in error by 10 km or more. The other two areas of deformation are newly discovered but based on their characteristics in a volcanic region (center) and near a Lake (lower right), volcanic and groundwater related sources of deformation are suggested. The dates of the two images used are 17 June 2006 and 4 December 2004 are from the Envisat spacecraft of the European Space Agency. 1

2 Overview of Contents Section 0. Acknowledgements Section I. Introduction Section II. Getting started: Data & software Raw data format Available software for InSAR and pixel tracking Appendix 0: Some basic LINUX commands Appendix 1: Getting and installing ROI_PAC Appendix 2: ROI_PAC file types and file formats Appendix 3: ROI_PAC processing flow: Script-by-Script Appendix 4: User defined options for ROI_PAC processing Appendix 5: Which scripts created each file generated by ROI_PAC? Appendix 6: What is the information in the RSC files? Appendix 7: Access to data and special processing considerations for each satellite Appendix 8: Viewing ROI_PAC output files and moving them to other programs Appendix 9: ROI_PAC Troubleshooting Appendix 10: What ROI_PAC can t do (yet) Appendix 11: Annotated bibliography of useful books and websites References 2

3 Section 0. Acknowledgements The impetus for this textbook was a realization that although there are many useful review papers and a few textbooks about InSAR and pixel tracking (e.g., Hansen, 2001; Kampes, 2006; Simons & Rosen, 2007; Massonnett & Feigl, 1998; Burgmann et al., 2000; Rosen et al., 2000; Chapter co-authored by Zhong Lu in Dzurisin, 2007; Feretti, 2014) there is no existing textbook on the rapidly developing field of imaging geodesy that explicitly helped new users get started using real data and software. Demand for such a textbook is high from attendees at the UNAVCO short courses and others. So, in 2009, Matt Pritchard proposed to write such a document as an online, free textbook as part of the education plan in a National Science Foundation CAREER that was funded in The focus here is on practical concerns of using the software, and we refer the reader to the literature for theoretical developments. In particular, a chapter from the Ph.D. thesis of Sean Buckley (a modified version is available online: documents some of the equations used in the software described in this text. Further corrections and additions to the textbook are planned at least until the end of the grant in 2016, so please send any suggestions or comments to Pritchard@cornell.edu. The materials in this book have been merged from many sources and many authors web pages, scientific papers and presentations, theses, classroom lectures and short courses, one-onone training and so on. We thank all of the students and postdocs over the years who have commented on these documents and presentations and contributed to their development. We especially thank the approximately 170 attendees of the annual day UNAVCO short courses from entitled InSAR: An introduction to Processing and Applications for Geoscientists (with an optional module for pixel tracking) that were taught by Eric Fielding (JPL, ), Paul Rosen (JPL, ), Yuri Fialko (UCSD, 2008); Matt Pritchard (Cornell, ), Scott Baker (UNAVCO ), Walter Szeglia (Central Washington University, ), and Piyush Agram (JPL, 2013) we further thank the UNAVCO staff for helping to organize these courses. The ROI_PAC software (Rosen et al., 2004) that is discussed extensively in this book was developed over several decades by researchers at the Jet Propulsion Laboratory (JPL) and the California Institute of Technology (Caltech) in Pasadena, CA. Some parts of ROI_PAC date back to the early 1980's and the pioneering spacebourne InSAR work done by Dick Goldstein, Howard Zebker and others (Goldstein et al, 1986). During the 1990's, Gilles Peltzer and colleagues started to put together the programs into a set of scripts. Through the collective efforts of the following people, ROI_PAC was created: Paul Rosen (JPL), Scott Hensley (JPL), Gilles Peltzer (JPL/UCLA), Francois Rogez (JPL), Frederic Crampe (JPL/UCLA), Eric Fielding (JPL), Mark Simons (Caltech), Rowena Lohman (Caltech undergrad/grad student, now at Cornell), and Matt Pritchard (Caltech grad student now at Cornell). 3

4 Section I. Introduction For centuries, measurement of the shape of the Earth (called the science of geodesy) was necessarily time consuming. Even with new technologies like the Global Positioning System (GPS), vast portions of the Earth remain infrequently monitored for movement. Over the past 20 years, a new form of geodesy has rapidly developed whereby satellite images can be compared to infer movements of the Earth's surface. Called imaging geodesy, the synoptic aircraft or satellite views allow large regions to be surveyed densely without any human setting foot in the area. Imaging geodesy encompasses several different types of methods including Interferometric Synthetic Aperture Radar (InSAR, e.g., Rosen et al., 2000) as well as the automated comparison of SAR and optical images via the techniques of pixel or feature tracking (e.g, Leprince et al., 2007). InSAR can image sub-centimeter deformation of the Earth's surface every 20 meters over areas spanning hundreds to thousands of kilometers. InSAR has allowed vast areas of the Earth's surface to be monitored frequently for deformation for the first time and led to discoveries in a variety of fields (including signals caused by earthquakes, volcanoes, landslides, glaciers, groundwater, and human-induced ground deformation). Pixel tracking is a very complementary tool to InSAR-- although the sensitivity to deformation is less (decimeter instead of subcentimeter in a given image pair) and the horizontal spacing is coarser, it can be applied to both radar and optical images and often works when InSAR does not for example, in areas that have large displacements or changes to the radar scattering properties of the ground. Section II. Getting started with InSAR and pixel tracking: Data & software Radar is a commonly used word, but it is actually an acronym for RAdio Detection And Ranging and involves the active transmission and reception of microwave energy to measure distance. The waves in the radar portion of the electromagnetic spectrum have frequencies from GHz or wavelengths of 1 cm 1 m. Different portions of this spectrum were divided into code names during World War II and these codes are still widely used. The most commonly used codes by satellite radars are X-band (~ cm wavelength), C-band ( cm), and L-band (15-30 cm), although the S band ( cm) was used on NASA Magellan mission to Venus, the Cassini mission to Saturn uses Ku band ( cm), and P-band (> 1 m) is sometimes used on airplane imaging radar systems. For reference, ground-penetrating-radars are typically in the UHF to VHF bands of.05-1 GHz (30 cm to 6 m), weather radars can be in the K, X, C, and S bands, air traffic control radars can include L, S, and X bands and cell phones, Wi- Fi, television, and Bluetooth typically are in the UHF band that overlaps mostly with the P but partly with the L band. Radars have a wide variety of uses, but we will be considering imaging radars that are able to distinguish fine details in two-dimensional arrays that constitute images of the ground. Specifically, we use Synthetic Aperture Radar (SAR), a common imaging technique used on a moving platforms (like an air- or spacecraft) to produce high resolution images in the along-track direction. SAR Raw Data Format The format of raw SAR data varies depending on the satellite and a compilation of documentation of several different formats is available here: SAR raw data is often delivered in the Committee on Earth 4

5 Observation Satellites (CEOS) format. The CEOS-format SAR raw data nominally consists of a Volume Directory File (ASCII), SAR Leader File (mixed ASCII and Binary), Raw Data File (Binary), and a Null Volume File (ASCII). The Volume Directory File describes the arrangement of the data on the storage media (tape, CD, file directory, etc.). The SAR Leader File provides pertinent information about the specific SAR data set: Raw Data File size, spacecraft height and velocity, scene center latitude, longitude and time of acquisition, etc. The Raw Data File contains a header record plus the SAR raw data nominally stored as one data line per record. Each value in the header contains useful information about the parameters of the data collected (the satellite clock time, line number, etc.) that are documented by the space agencies ( Each record line consists of a prefix (header in the case of ERS, this is 412 bytes), raw data (see next paragraph), and a suffix (not always used). The binary SAR raw data (excluding the header and suffix on each line) is commonly stored in two formats as a complex number in I/Q (in-phase (real) and quadrature (imaginary) components) or offset video (real numbers). We focus on the I/Q format, as it is the most common for the satellites we discuss here (the offset video was used for SEASAT). The choice of data formats is governed by how the radar signal returned to the satellite from the ground (the echo) is converted from an analog to digital. The I/Q format uses a complex converter while offset video requires a more complicated converter to save as real numbers but the total number of data points in each format is the same. Each format has different advantages and requires different algorithms to process. For each return sample, the received voltage is recorded as an integer byte values -- each byte is made up of 8 bits and each bit has a value of 0 or 1. For example, ERS SAR raw data is 5-bit quantized, i.e., the voltage is recorded as one of 32 values (2 5 ) -- between 0 and 31, and the JERS-1 data are quantized to 3 bits (or between 0 and 7). Thus, JERS and ERS data are significantly smaller than the equivalent binary files composed of floating point numbers that are typically made up of 4 bytes each (each byte is 8 bits), and the data can be easily read on all types of computers (no matter if they are big endian or little endian). In order that negative voltages can be recorded, a bias is added to all returned voltages. For ERS, this bias is about 15.5 (small deviations from this nominal value can be determined from the raw data and are typically calculated by the satellite downlink facility and included in the leaderfile). To convert the raw data number (dn) to a voltage (V), use the expression: V = A * (dn-15.5) where A is a scaling constant depending on the gain of the radar receiver and analog to digital quantizer. Since the data is quantized directly as complex numbers, each returned sample is recorded as a real and imaginary component for the I/Q format thus the signal from a given sample is complex(dn 2*n ,dn 2*n -15.5) where n is the column number. The raw data file can be viewed using the program mdx (Appendix 8) using the following command: Mdx i1 FILENAME LINELENGTH (show example figure). Available software Pixel tracking: 5

6 Pixel tracking with ROI_PAC: A set of python scripts (download here: have been released that use ROI_PAC to generate dense sub-pixel offsets of optical or SAR using the method of normalized cross correlation (NCC), which computes the offsets in the spatial domain (like the program IMCORR). Cosi-Corr (Leprince et al., 2007) An open-source program to generate sub-pixel offsets is called COSI-CORR (available at that generates subpixel offsets of optical images that can use either NCC or a different algorithm called the Fourier Shift Theorem. COSI-CORR requires the use of the GIS package ENVI and the commercial software package IDL, which are usually available for $2000-$3000 to academic users. IMCORR (Scambos et al., 1992), Open source software (available at that uses NCC to generate sub-pixel offsets between optical images. InSAR software: ROI_PAC: The Repeat Orbit Interferometry PACkage developed at JPL and Caltech, is a collection of fortran and C programs bound together with perl scripts to perform certain routine but important tasks in synthetic aperture radar (SAR) repeat orbit interferometry. Specifically, the individual programs perform everything from raw data conditioning, SAR image processing, interferogram formation, correlation estimation, phase unwrapping, baseline determination, estimation of topography, removal of topography from a deformation interferogram, and geocoding. The perl scripts control certain prescribed ways of combining these programs to specifically create a geocoded deformation phase or topographic phase image from two satellite radar images and a digital elevation model, or create a deformation phase image from three radar images without a digital elevation model. The perl scripts allow considerable control over every step of the processing, such that individual programs or combinations of programs can be manipulated to produce a more favourable output without direct interaction with the program itself. ROI_PAC does not perform well with airborne sensors because it does not include motion compensation. The software is described in Rosen et al., (2004) and this is the article that should be cited when using the software. The software package is the most widely used for InSAR processing in the peer-reviewed scientific literature (over 3000 unique downloads since 2003 and 150 peer-reviewed publications, roipac.org/bibliolist) ISCE: InSAR Scientific Computing Environment is a new Stanford/Caltech/JPL software package that builds on some of the FORTRAN and C programs in ROI_PAC but is faster, more accurate, and provides better geolocation (Gurrola et al., 2010; Rosen et al., 2009; Zebker et al., 2010). For example, it uses a motion-compensated geodetic coordinate system (used to process SRTM and UAVSAR data, for example) instead of the range/doppler coordinate system of ROI_PAC. Using this motion-compensated 6

7 coordinate system from the start facilitates time series analysis because co-registration of images before they can be combined is not required (as in ROI_PAC). Another difference is that ISCE uses python instead of Perl Scripts. Licensing for ISCE is available to WInSAR members through UNAVCO. GMTSAR: Open source InSAR processing software based on the open source Generic Mapping Tool (GMT) package (Sandwell et al., 2011) DORIS ( is another open source suite of InSAR processing software that is being integrated in the NEST (Next ESA SAR Toolbox) software packaged ( Training in the use of the NEST software is available: ( Open source software to create InSAR time series using multiple methods (Agram et al., 2013): GIANT ( Open source software for Atmospheric Phase Screen removal (Jolivet et al., 2011) Pyaps ( Other notable InSAR processing software that is commercially available includes: GAMMA Nigel Press Associates (NPA) ( DIAPASON ( Training.php); 7

8 Appendix 0: Some basic LINUX commands ROI_PAC requires users to use UNIX or LINUX and interact with their computer via the command line. For many people, use of the command line requires a period of adjustment usually lasting a day or two. The key point to remember is that LINUX is based on a directory structure so you need to constantly be aware of what directory you are currently in. There are many good resources on the web that provide a quick introduction to LINUX -- for example: and Here we provide a very short list of some essential commands. The Top 16 Most Useful LINUX commands you will use everyday 1) ls List the files and subdirectories ls -alf will list the files and directories with additional useful information (size, date of last modification, and who has permission to read/write) For example, here is some example output: -rw-r--r-- 1 matt staff Mar myfile.pdf drwxr-xr-x 6 matt staff 204 Jan Stephen/ An important trick in LINUX is the use of wildcards. If I wanted to display all the interferogram files, I would type ls *.int On many computers, different types of files show up as different colors. Directories are one color, files that you can execute are another, files that are linked (see below) another, etc. 2) man Display the manual pages for any command. For example, typing man ls would allow you to see all of the options you can type with the ls command including ls -alf 3) cd change directory cd by itself takes you to your home directory (usually /home/yourname) cd.. moves you one directory up cd./name moves you down into directory NAME cd /FULL/PATH moves you to the specific directory /FULL/PATH that may neither be directly up or down from your current location 4) cp Will copy a file a new version will be made and the original will be saved 8

9 The usual syntax is cp OLDFILE NEWFILE where OLDFILE is the existing file and NEWFILE is the name of the file you wish to create. You can create the NEWFILE in a different directory if you include the full path 5) mv Will move a file. The original version will not be saved. Useful when you do not want to have two copies of a large file. 6) ln -s Sometimes a file is so big that you do not want to move the whole thing, but instead want to link a large file from one directory to another. For example, ROI_PAC uses this command with the large single look complex files. The syntax is ln -s ORIGINAL_FILE /NEW/FILE/DESTINATION 7) rm Will remove or delete a file. USE WITH CAUTION Useful flags are rm f to force without having to be asked permission and rm r to remove all files and subdirectories in a directory 8) pwd Will show you the path of the current directory you are in. It stands for print working directory. 9) Edit text files. There are many ways to do this depending on what computer you are on. If you are on a Mac, you can use the program TextEdit. Many machines have a program called emacs which allows you to interact with the text file in a familiar menu based manner (like Word). If you want to go old school you can learn how to edit using a program called vi that works on all platforms. A web search for vi will reveal all the commands you need to use this program. 10) more Prints the contents of a text file to screen. Use the spacebar to scroll forward and type q to exit. 11) chmod Allows you to change the permissions on a file (to give others the ability to read, write or execute the file). 12) source ROI_PAC sets up a list of file name shortcuts in a file called SAR_CONFIG. Any time you change this file you need to type source SAR_CONFIG to make the changes active. Every time you log in, your computer runs a program called.bash_profile or something similar if you are using a shell other than bash. You can set your computer to run this command automatically for you every time you log in by typing this command in your.bash_profile file: source /PATH/SAR_CONFIG where PATH is the directory where the SAR_CONFIG file is located. 13) df -k Displays the amount of disk space currently being used (in terms of bytes and %) and how much is available. Important so you don t overfill the disk. 14) du -ks./* This command allows you to see the size of the all the files within a set of subdirectories. The ls command will show you the size of individual files but will not sum the 9

10 contents of an entire directory. This command is useful to see which directories are taking up the most disk space. 15) grep Extremely useful command for extracting information from text files by doing a search. For example, if I want to find the pixel size of a certain file, I would type grep PIXEL_SIZE FILENAME where it would search for the phrase PIXEL_SIZE in all the files named FILENAME 16) mkdir NAME Makes a new subdirectory with NAME 10

11 Appendix 1: Getting and installing ROI_PAC Getting ROI_PAC ROI_pac is copyrighted software that requires a license. Licenses are available at no charge for non-commercial use from Open Channel Foundation ( Installation The new release (v ) has a very different installation system compared to releases before v The easiest way to build it is explained in the ROI_PAC_3_0_1/ROI_PAC/AAREADME_BUILD_ROIPAC file. 1. Expand Archive Uncompress and untar the archived file. (Command line input is in red) % tar -xzf ROI_PAC_3_0_1.tgz 2. Read the documentation Read through the text file ROI_PAC_3_0_1/AAREADME and the additional documentation found in ROI_PAC_3_0_1/ROI_PAC/DOC. The instructions for installing the software and running the scripts are described in ROI_PAC_3_0_1/ROI_PAC/DOC/Setup and ROI_PAC_3_0_1/ROI_PAC/AAREADME_BUILD_ROIPAC. The most important steps are summarized below. % cd ROI_PAC_3_0_1 % less AAREADME 3. Compile Executables Follow the instructions in the file ROI_PAC_3_0_1/ROI_PAC/AAREADME_BUILD_ROIPAC. Copy and paste each command individually into a shell. These commands will build and test the software package for multiple compilers. If problems are encountered, refer to the bottom of AAREADME_BUILD_ROIPAC for platform specific issues or look through prior posts on the forum ( at Open Channel Foundation. To see what shell you are using, type echo $SHELL on the command line. % cd ROI_PAC_3_0_1/ROI_PAC % less AAREADME_BUILD_ROIPAC % which ifort g95 f90 pgf95 f95 xlf gfortran cc gcc icc % touch aclocal.m4 Makefile.in configure %./contrib/install-fftw.sh CC=cc % export FFTW_LIB_DIR=/software/ROI_PAC_3_0_1/ROI_PAC/NetInst/fftw /lib % export FFTW_INC_DIR=/software/ROI_PAC_3_0_1/ROI_PAC/NetInst/fftw /include %./contrib/multibuild.sh etc. 11

12 When testing the build, you will need to gunzip the provided test data set prior to running the script multitest.sh. % ls ROI_PAC_3_0_1 ROI_PAC_3_0_1.tgz roi_pac_testdir.tar.gz % gunzip roi_pac_testdir.tar.gz % cd ROI_PAC_3_0_1/ROI_PAC/test-runs %../contrib/multitest.sh /full_path/roi_pac_testdir.tar /full_path/roi_pac_3_0_1/roi_pac/multibuild /installs/share/roi_pac /full_path/roi_pac_3_0_1/roi_pac/multibuild /installs/defaults/bin /full_path/roi_pac_3_0_1/roi_pac/multibuild /installs/gfortran/bin Each test run should take ~40 minutes depending on your computer. If the test fails while processing the DEM contained in roi_pac_testdir.tar, you may have to manually swap this DEM file with one that has the proper byte order for your machine (i.e., little-endian versus big-endian, see TEST_DIR/AAREADME). If the test is successful, the finished product is the file geo_ unw which should be buried within the directory structure (check in testruns/defaults/test_dir/int_930110_950523). You can view the image file using the program mdx, if you already have it installed. Otherwise, you can try to view the file with a different program that can display binary image files (i.e., Matlab, ENVI, ERmapper). The file is composed of floating point values in a band-interleave format, i.e. alternating lines of magnitude and phase. The size of the file should be just under 10 MB. The meta-data for the file can be found in geo_ unw.rsc. If you have progressed this far, congratulations! You have a working version of ROI_PAC. 4. Install Executables Make a new directory called INT_BIN where you will deposit the final executables. % mkdir ROI_PAC_3_0_1/ROI_PAC/INT_BIN You can copy the set of executables from the multi-build operation. Choose one of the successful builds found in ROI_PAC_3_0_1/ROI_PAC/multibuild*/installs. % cd ROI_PAC_3_0_1/ROI_PAC/multibuild*/installs/defaults/bin % cp * ROI_PAC_3_0_1/ROI_PAC/INT_BIN/. NOTE: If you plan to use ODR orbit files, then the getorb ( executable (installed separately) must be located in INT_BIN. You can copy the executable into this directory or make a symbolic link. NOTE: After finishing this step, please check the Patches ( page to review/install any version specific corrections. 12

13 5. Configure SAR_CONFIG The shell script SAR_CONFIG controls the location of the code and scripts. Once this is set properly the tools of ROI_PAC should be available to the user. SAR_CONFIG describes how to establishes required environment variables pointing to the programs and perl scripts that ROI_PAC uses, as well as the location of orbit data and other ancillary data. ROI_PAC will absolutely not work if these are not set properly. Create your personal copy with the settings your like and source the file at login. Example config variables are provided in csh and bash style. If you use a different shell you may need to change the SAR_CONFIG syntax. For example: csh: setenv SAR /home/insar bash: export SAR=/home/insar Read the instructions in ROI_PAC_3_0_1/ROI_PAC/DOC/SAR_CONFIG. Copy this file into the directory ROI_PAC_3_0_1/ROI_PAC and edit the appropriate lines. You will have to source this file every time that you want to use ROI_PAC. It will set your path environment so that the necessary scripts will be available on the command line. % cp ROI_PAC_3_0_1/ROI_PAC/DOC/SAR_CONFIG ROI_PAC_3_0_1/ROI_PAC/. % vi ROI_PAC_3_0_1/ROI_PAC/SAR_CONFIG (edit paths) % source SAR_CONFIG In addition to the paths to ERS orbit files (SAR_PRC_DIR and SAR_ODR_DIR) or Envisat orbit files (POR_DIR and VOR_DIR), you will also need to define the INT_BIN and INT_SCR directories. To do this, add the following lines to SAR_CONFIG. export INT_BIN="/full_path/ROI_PAC_3_0_1/ROI_PAC/INT_BIN" export INT_SCR="/full_path/ROI_PAC_3_0_1/ROI_PAC/INT_SCR" Instead of using export for the bash shell, use the setenv syntax if you are using the csh or sh shell. You can also add the path to mdx, getorb, or other related software that you might install. export MDX="/software/mdx/bin" export FFTW_LIB="/software/ROI_PAC_3_0_1/NetInst/fftw /lib" export PATH="$PATH:$INT_BIN:$INT_SCR:$MDX" 6. Update Path To Perl The path to the perl executable may be different on your machine than is used in the ROI_PAC scripts. You can check if this step is necessary by comparing the paths in the following outputs: % cd ROI_PAC_3_0_1/ROI_PAC/INT_SCR % grep perl make_raw.pl #!/usr/bin/perl 13

14 % which perl /usr/bin/perl The example shown above shows that no changes are necessary. If the paths are different, there is a simple script in ROI_PAC_3_0_1/ROI_PAC/INT_SCR that will edit this path for you in all of the perl scripts. % cd ROI_PAC_3_0_1/ROI_PAC/INT_SCR % chgperlpath.pl 7. Celebrate You are now ready to process your own SAR data. Create a workspace where you will process future data. You may also want to create a new directory elsewhere on your system where you can archive your DEM and orbit files. Update SAR_CONFIG with the new location of the orbit files. Remember to source your SAR_CONFIG in each new shell before you start to process data. Alternatively, you can add a line to the file that initializes your shell environment upon startup (such as.cshrc or.bash_profile). Some notes on compiling ROI_PAC for Mac 10.6.x (Snow Leopard) I recently compiled ROI_PAC on a 64-bit Mac (OS , little endian). Here are notes on the procedure. 1. Obtain Compilers The Snow Leopard operating system does not include any compilers by default. In order to obtain compilers, you must install XCode, which is the Mac developer software. The newest version of XCode (4.2.x) is not available for free it is $99 if you don t already have a license. The older XCode (3.2.x) was free, but is no longer downloadable. If you can obtain a.dmg for XCode 3.2.x, this can be installed for free. All versions of XCode include a gcc compiler. Therefore, simply installing any version of XCode will allow you to install and compile the FFTW library. However, you will get stuck after this point if you have not also installed a separate fortran compiler (in other words, no fortran compiler is included with XCode 3.2.x). The following website has resources for installing gfortran: The first.dmg (4.2.3) on the website will not work with XCode you will not be able to get ROI_PAC to link to FFTW during compilation if you install gfortran You must instead install the gfortran-42-5xxx version of gfortran (whichever matches your XCode version). 2. Build Without Multibuild I found that building ROI_PAC with these compilers was easier without using multibuild.sh. Following the instructions in /ROI_PAC_3_0_1/ROI_PAC/INSTALL, I did the following: % cd /ROI_PAC_3_0_1/ROI_PAC/ 14

15 % LDFLAGS=-L/$FFTW_LIB_DIR F77=gfortran./configure -- prefix=/roi_pac_3_0_1/roi_pac/int_bin % mkdir /ROI_PAC_3_0_1/ROI_PAC/INT_BIN % sudo make % sudo make install I found that specifying the fortran compiler was necessary, as was setting the LDFLAGS option (even though $FFTW_LIB_DIR was already exported). Finally, I put the prefix option in order to direct the executables to INT_BIN (otherwise I m not sure where they would end up). After this, I proceeded with the instructions above to set up SAR_CONFIG and so forth. 15

16 Appendix 2: ROI_PAC file types and file formats In outline, the files output by ROI_PAC have 4 basic types: 1) ASCII files used for input or output by programs (suffixes:.out,.in,.off: more details below) 2) Binary files as real or complex files (see below) RMG format (Names for JPL radar pioneer Richard M. Goldstein) are made up of real*4 (32 byte) numbers in two arrays side-by-side. The two arrays often show the magnitude of the radar image and the phase although not always (sometimes the phase is the correlation. The length and width of each array are given as lines in the metadata (.rsc) file. Thus the total width of the binary file is (2 * width) and the data are stored as (Magnitude, magnitude, magnitude, Phase, Phase, Phase, ) complex or c8 files contain real and imaginary values that are stored in the binary file as (real, imaginary, real imaginary, etc.) 3) ASCII files associated with each binary file containing metadata (end with.rsc resource file); and a record of the changes made to the metadata file (.rsc.hst history file) 4) ASCII log files that record the commands that have been run in each directory where ROI_PAC is run. The log files provides a record of the commands that have been run and the times that these were run. During debugging you can often just copy and paste these commands onto the command line modifying one or more values. The log1 file shows the information that was written out by the scripts to the screen, but unfortunately it usually does not include error messages. We now describe the specifics of these files in categories of input file (user-created) and output files (generated automatically from ROI_PAC) divided by the suffix to the file name: Input files: *.proc = ASCII text file specifying parameters used in processing and there are two main types: int*.proc files for creating the interferograms (required and the suffix is allowed to be different) and optional roi.proc or (DATE).proc files that are used to create the slc s. See Appendix 4 for details on the different parameters that can be set. *.dem = binary file with digital elevation model in lat-long or UTM coordinates, signed integer 2-byte values (in meters unless scaling specified). An important question to consider when comparing InSAR results to GPS or others is whether the heights in the DEM file refer to heights relative to the ellipsoid or the Geoid (SRTM refers to Geoid heights). The.dem and.dem.rsc file can be automatically generated by the script get_srtm.pl available at or you can create your own files. The format for the.rsc file is as follows: WIDTH number of columns FILE_LENGTH number of rows 16

17 X_FIRST longitude of upper left corner (does it refer to center of upper left pixel?) Y_FIRST latitude of upper left corner (does it refer to center of upper left pixel?) X_STEP longitude pixel size in degrees for latlon, meters for UTM Y_STEP latitude pixel size in degrees for latlon, meters for UTM X_UNIT degrees or meters Y_UNIT degrees or meters Output files: *.off = ASCII text file of offsets measured between two images columns 1=X, 2=ΔX, 3= Y, 4=ΔY, 5=Signal-to-noise ratio, other columns are three components of the match covariance *.in = ASCII text file created by ROI_pac scripts used as input for a compiled program *.out = ASCII text informational output of a program *.hgt = binary file with simulated SAR amplitude image and elevation in radar coordinates, real 4-byte values band interleaved by line (BIL or rmg ) *.slc = complex real 8-byte binary file containing real and imaginary parts of the singlelook complex (SLC) image (and multilooked versions of the SLC) *.cor = binary file with average amplitude of SAR images used to form an interferogram and correlation measure of coherence, real 4-byte values band interleaved by line (BIL or rmg ) *.raw = binary file of raw data, I,Q (see section II for explanation) 1-byte integer values for each echo sample *.aff = ASCII text file of affine transformation to map simulated image to actual SAR image *.unw = binary file with SAR amplitude image and unwrapped phase, real 4-byte values band interleaved by line (BIL or rmg ). The units in the file are radians and can be converted to cm by multiplying by the radar wavelength and dividing by 4*pi. NOTE: phase change values in this file are relative that is zero radians of phase change could actually be -1 m, +1 m, or some other value that must be otherwise constrained. *.flg = binary file with flags used and resulting from unwrapping with standard unwrapper, 1-byte values with flags set in bits * int = complex real 8-byte binary file containing real and imaginary parts of the interferogram (can also be read as real or float values band interleaved by pixel) 17

18 .amp = binary file with amplitudes of the two SAR images used to form an interferogram, real 4-byte values band interleaved by pixel (can also be read as complex 8-byte values) *.msk = binary file with SAR amplitude image and coherence with zeros in masked out areas, real 4-byte values band interleaved by line (BIL or rmg ) *.trans = binary file with inverse mapping transformation from SAR to DEM coordinates, two bands are range and azimuth pixel locations of SAR for each DEM pixel, real 4-byte values band interleaved by line (BIL or rmg ) The file names before the suffix are descriptive of the steps that were taken to create that file. For example: *rlks* Specifies the number of range looks taken from the full resolution interferogram or, for a geocoded file, from the resolution of the full resolution geocoded file filt_* means that the file has been filtered by the filter.pl script *-sim* means that the simulated topographic interferogram has been removed flat_* the file has been flattened (a.k.a. orbital effects have been removed) geo_* file has been geocoded SIM_* The DEM has been converted to radar coordinates (simulated interferogram) using orbital information but no fine adjustment of the position has been made (see radar_*) radar_* The DEM has been converted to radar coordinates (simulated interferogram) using orbital information and a fine adjustment of the position has been made (this file is created after SIM_*) radar_*.unw is created by diffnsim.pl and includes the orbital & topographic effects radar_*.hgt is created by rect.pl and includes only the topographic effects ramp_* The orbital phase ramp created by diffnsim.pl with no topographic effects *SIM* or *ODR* or *PRC* or *HDR* means that the orbital effects have been removed using the given type: ODR = Delft PRC = DPAF SIM = re-estimated baselines HDR = header file (or for Envisat, some precise orbits are still called HDR) 18

19 Appendix 3: ROI_PAC processing flow: Script-by-Script Before you get started, you need the following 5 things: 1) Orbit files may need to be downloaded directly from a space agency or other institutions it depends on which satellite you are using. See Appendix 6 2) InSAR processing software: see: Appendix 1 for installing ROI_PAC. To make sure it is working properly it is recommended to try the test data and ensure that the processing results are correct. 3) Digital Elevation Model (Definitely try SRTM if available (between 60 degrees north and 56 degrees south); The script get_srtm.pl will allow you to download the software and convert it to ROI_PAC format by specifying a few simple parameters (lat/lon, global vs. US versions, etc.) The latest version is available at: 4) Viewing software: also see Appendix 8 5) Raw data: for information on how to download, see Appendix 7 Step 1: Create the proper directory structure (two subdirectories with the raw data from the two different dates to be combined. Format could be YYMMDD). Then within each subdirectory, run the following command for ERS data, where DATE = YYMMDD (or whatever date format you use for the directory name). Run make_raw.pl ORBIT_TYPE LEADER_FILE DATE There are different scripts if you are using JERS, RADARSAT, ENVISAT, or ERS data downloaded in CEOS format from ASF. Those that are not available in ROI_PAC are available at: However, once you've made raw, the following steps are more-or-less the same for all data types. Data ingestion: make_raw_(sat).pl The make_raw script will prepare the files for processing in ROI_PAC, performing an initial check of the file, reformatting the data, locating the appropriate orbit file, and performing a Doppler computation. There are different ingestion programs for each satellite type ERS-1 and ERS-2: use make_raw.pl For data formatted at the Alaska Satellite Facility use make_raw_asf.pl Envisat: use make_raw_envi.pl ALOS: use make_raw_alos.pl JERS-1: use make_raw_jers.pl 19

20 COSMO-SkyMed (CSK): (to be released in version 3.1) raw, use make_raw_csk.csh SLC, use make_slc_csk.csh TerraSAR-X and TanDEM-X (TSX & TDX): (to be released in version 3.1) SLC, use make_slc_tsx.csh Radarsat-1 has two data products from the Alaska Satellite Facility CEOS: use make_raw_rsat-ceos.pl STF: use make_raw_radarsat_swath.pl (Not yet released) You can run make_raw_(sat).pl with no arguments to see a detailed usage page. As an example, to run with ERS, cd to the directory containing the master SAR scene and run the following command (using Delft ODR orbits, you could also use the HDR, header file orbits or PRC orbits): $ make_raw.pl ODR SARLEADER(DATE1) (DATE1) Repeat the steps: cd to the directory of the slave SAR scene, and run make_raw_xxx.pl adjusting the date and filename appropriately. You are now ready to produce an interferogram. NOTE: make_raw_(sat).pl can concatenate multiple frames together you do not need to run each frame separately. For the concatenation to work, you need to have all of the frames together in the same directory and have their file names be in numerical order so that the program can figure out which goes first, second, third, etc. based just on the file names. For Envisat and ALOS data, the files usually come in the correct format for concatenation the file time (Envisat) or frame number (ALOS) usually appear in the file name. For ERS, depending on how you receive the data, you may need to rename the binary data and leader files to have each frame be in the correct order. The output files are: $date.raw unpacked raw data 8 bits I,Q for each sample, one line per echo record, ERS has 412 bytes extra at start of lines $date.raw.rsc metadata for raw data hdr_data_points_$date.rsc orbit data reformatted other intermediate files, including dop.unw output of programs to measure Doppler centroid of data (fit put into.raw.rsc) The other programs called by make_raw_(sat).pl include (information here may not be completely up-to-date, please read the scripts themselves for the more details); make_raw.pl: new_parse finds missing lines and adds duplicates, finds overlapping lines and removes 20

21 delay_shift shifts the lines in the raw data to remove offset (called swst) due to satellite geometry leader2rsc takes information from the leader file to put into the.rsc file state_vector.pl -- calculates satellite position & velocity at requested epoch dopiq GetPeg.pl make_raw_envi.pl: asa_im_decode state_vector.pl -- calculates satellite position & velocity at requested epoch scan_doppler.pl -- make_raw_jers.pl autofocus.pl JoinJData concatenate scenes and remove raw data header Calc_squint.pl leader2rsc takes information from the leader file to put into the.rsc file mlcc used in Doppler centroid estimation dopiq used in Doppler centroid estimation make_raw_alos.pl: ALOS_pre_process ALOS_merge ALOS_fbs2fbd ALOS_fbd2fbs state_vector.pl -- calculates satellite position & velocity at requested epoch make_raw_rsat-ceos.pl: state_vector_rsat.pl -- calculates satellite position & velocity at requested epoch intert2terrest 21

22 extract_ut1_utc parse_rsat.le or parse_rsat.be modify rough raw data into conditioned raw data dopiq-new used in Doppler centroid estimation Step 2: Create the proper processing file called "date1-date2.proc" Run process_2pass.pl "date1-date2.proc" [DoItFrom] [EndItAt] (see table below for some possible [DoItFrom] or [EndItAt] points) process_2pass.pl is the most commonly run script, but others ( CS/process_index/perl_scripts.html) are sometimes needed [this list is incomplete and a little out of date, but it's better than nothing!]. If you are exceedingly lucky, process_2pass.pl can go from the raw data to making an interferogram to removing the orbital effects ("flattening") to removing the topographic effect to unwrapping to geocoding in a single run. More likely, you are going to start and stop this flow at various points to fix errors or otherwise change the parameters (see the table below). There are numerous such "start" and "stop" points listed below, but we highlight (in purple bold) the ones that you will use most frequently. To run the process to the end, use done for [EndItAt]. 22

23 [DoItFrom] [EndItAt] point raw roi_prep orbbase slcs offsets resamp flatorb full_res done_sim_off done_sim_removal begin_filt filtered make_mask unwrapped redo_base sim_removal_bsim unwrapped_bsim done Comments Always start here after running make_raw.pl Create input files for SAR processor Calculate baselines Create single-look-complex interferograms Co-register two slcs: Frequently restart here after (problem #2a on this list) Resample slave scene to master coordinates "flatten" scene and remove orbital ramp Good stopping point: have created interferograms and correlation files Calculate offsets between interferogram and DEM Good stopping point: have removed the topographic effects from the flattened interferogram filter the interferogram done filtering create mask for unwrapping Good stopping point: Flattened interferogram with topography removed is unwrapped Empirically re-estimate the baselines remove topo from newly re-flattened interferogram unwrap newly re-flattened interferogram Final stopping point: Put flattened unwrapped interferogram with topography removed in geographic coordinates How long will it take? Processing a single frame interferogram takes on the order of an hour (using the default parameters, although it can take longer if you have a noisy interferogram) from raw data to the final interferogram with the orbital and topographic effects removed, unwrapped and placed in geocoded coordinates. Of course this does depend on your computer processing speed and whether you use $concurrent_roi=yes among other options. All output files and directories will be placed in the directory where param.proc tells them to. If the script terminates successfully, the final line of terminal output will be That s all folks. How much memory do I need? For most satellites besides ALOS, you can process most data with 2 Gb of RAM. For ALOS you need at least 4-8 Gb. How much disk storage do I need? For a single frame of ERS data (other satellites may be bigger or smaller), the slc s are each about 1.5 Gb, the full resolution interferograms are about 1 Gb. Thus, the complete int* directory is of order 6 Gb and thus including raw data and slc s a single frame completely processed to done is about 10 Gb. Multiply this number by the number of frames to get an estimate. 23

24 process_2pass.pl calls secondary scripts that are listed here in the order with which they are called and a brief description of their purpose (information here may not be completely up-todate, please read the scripts themselves for the more details): [DoItFrom= raw] raw2ampintcor.pl is a script that calls other scripts to go from the steps of making the slc s, interferograms and correlation files dopav.pl -- checks to make sure the bandwidth of the two slcs are within reasonable limits. If not, it removes the slc files which kills the process_2pass.pl Calculates the mean Doppler centroid for the pair of images and common bandwidth uses: (date).raw.rsc files makes: roi.dop.rsc files roi_prep.pl -- Creates the input files for the roi.pl script uses: (date).raw; (date).raw.rsc; roi.dop; (date).proc [optional] roi.proc [optional] makes: (date).slc.rsc (date)_roi.in -- Input file for the roi program [EndItAt= roi_prep] calls: GetPeg.pl calculates peg point on the ground baseline.pl -- calculates baseline between image pair from orbits or header information uses: (date1).slc.rsc; (date2).slc.rsc makes: (date1)_(date2)_baseline.rsc [EndItAt= orbbase] calls: state_vector.pl (calculates satellite position & velocity at requested epoch) roi.pl -- Creates the slcs from the raw data uses: (date).raw; (date).raw.rsc; roi.dop; (date).proc [optional] roi.proc [optional] makes: (date).slc; (date).slc.rsc (date)_roi.out output file for the roi program calls: the roi -- FORTRAN program creates the slc s; length.pl which updates the rscfile with the new length of the file after roi is complete 24

25 look.pl -- Takes specified number of looks of an imagefile. This script is called several times throughout process_2pass.pl, but we will only refer to it at this first usage uses: (date).slc; (date).slc.rsc [EndItAt= slcs] [DoItFrom= slcs] makes: (date)_rlks.slc; (date)_rlks.slc.rsc make_offset.pl -- computes offset between two slcs as a function of x, y in the reference image. The program calculates the offsets twice, once at large spacing between searches with a large search window in order to get a rough estimate of the offset (*ampcor_gross*) and then with a denser spacing between searches and a smaller search window (*ampcor*) uses: both (date).slc and (date).slc.rsc files makes: gross_x; gross_y; (date1)_ (date2)_ampcor_gross.in; (date1)_ (date2)_ampcor_gross.off; (date1)_ (date2)_cull_gross.off; (date1)_ (date2)_ampcor_gross.out; (date1)_ (date2)_ampcor.in; (date1)_ (date2)_ampcor.off; (date1)_ (date2)_cull.off; (date1)_ (date2)_ampcor.out; fitoff_ampcor_gross.out fitoff_ampcor.out; [EndItAt= offsets] [DoItFrom= offsets] calls: offset.pl -- which actually calculates the offsets once all files are set up (this is the program that is called twice for the gross and fine offsets) fitoff selects good points from the ampcor file and puts them in the *cull* file resamp.pl -- makes the interferogram and amplitude files uses: slc and slc.rsc files; (date1)_ (date2)_cull.off; makes: (date1)_ (date2).int; (date1)_ (date2).int.rsc (date1)_ (date2).amp; (date1)_ (date2).amp.rsc (date1)_ (date2)_resamp.in; (date1)_ (date2)_resamp.out calls: resamp_roi -- FORTRAN program that actually makes the interferogram after putting the slc s in the same reference frame. Fits a second order polynomial to the culled offsets and uses this to resample the 25

26 [EndItAt= resamp] [DoItFrom= resamp] slave slc to the master. Interferogram formed by multiplying the complex pixel of master scene by the complex conjugate of the slave scene add_rmg.pl -- This is a utility program used throughout the scripts that works on two files of the same size and will add or subtract rmg them together or you can not add/subtract but mask certain values. You have the choice of outputting zero at a given pixel when one of the images has a zero in it or both do. In this location in the script, the program creates a file that is the size of the interferogram but that is all set to $ref_height (default is zero) to be used by diffnsim.pl to flatten the interferogram uses: junk.unw makes: reference.hgt; reference.hgt.rsc diffnsim.pl -- are then used to flatten the interferogram, meaning that the orbital effects are removed but not the topographic effects uses: (date1)_ (date2).int; makes: flat_(date1)_ (date2).int; ramp_orbit_type.unw diffnsim_flat*.int.in [EndItAt= flatorb] [DoItFrom= flatorb] calls: diffnsim -- computes the differential interferogram (in this instance, it only removes the orbital effects) make_cor.pl -- creates an rmg file with average amplitude and correlation between images uses: flat_(date1)_ (date2).int; (date1)_ (date2).amp makes: (date1)_ (date2).cor; calls cchz_wave -- which calculates the correlation using a 5x5 triangular weighted window replace mag.pl -- replaces the magnitude from infile1 with the magnitude from infile2. Replaces the magnitude of the correlation file with the average from the two scenes uses: flat_(date1)_ (date2).int makes: (date1)_ (date2).cor; calls: rmg2mag_phs; cpx2mag_phs; mag_phs2rmg; mag_phs2cpx; -- These are utility programs written in C used throughout the perl scripts to 26