CHANGE DETECTION IN MULTITEMPORAL VERY HIGH RESOLUTION IMAGES

LOGO Remote Sensing Laboratory Dept. of Information Engineering and Computer Science University of Trento Via Sommarive, 14, 38050 Trento, Italy CHANGE DETECTION IN MULTITEMPORAL VERY HIGH RESOLUTION IMAGES Lorenzo Bruzzone Francesca Bovolo E-mail: lorenzo.bruzzone@ing.unitn.it Web page: http://disi.unitn.it/rslab/ Outline 1. Background on Change Detection in Multitemporal Images; 2. Change Detection in Very High Resolution Multispectral Images; 3. Change Detection in Very High Resolution SAR images; 4. Operational Issues and Conclusions. 2

LOGO Background on Change Detection in Multitemporal Images Introduction Change detection (CD): process that analyzes multitemporal remote sensing images acquired on the same geographical area for identifying changes occurred between the considered acquisition dates. Landsat TM (Burned Area) ERS-1 (Burned Area) Quickbird (Tsunami) ange After Cha Before Change 4

1. Binary change detection Introduction: CD Problems Goal: production of binary maps in which changed and unchanged areas are separated; Number of images: 2 (or pairs of images extracted from a series); Application domain: detection of abrupt (step) changes. Mexico, May 2002 Mexico, April 2000 Map of burned areas 5 2. Multiclass change detection Introduction: CD Problems Goal: generation of a change-detection map in which land-cover transitions are explicitly identified; Number of images: 2 (or pairs of images extracted from a series); Application domain: updating thematic maps, detection of multiple changes. May 1995 (Landsat) July 1995 (Landsat) Thematic Map URBAN URBAN BARE SOIL SUGAR BEET WHEAT BARE SOIL BARE SOIL CORN BARE SOIL SOYBEAN 6

Introduction: CD Problems 3. Change detection in long temporal series of images Goal: detection of changes associated with modifications of the behavior of the temporal signature of a land cover between two time series (detection of long term changes); Number of images: 2 time series made uponn images (n>>2); Application domain: monitoring seasonal/annual changes. -1.0 0 +1.0 NDVI NDVI 200 180 160 140 120 100 198 9 200 180 160 140 120 100 Loss of vegetation Re-vegetation 199 7 7 CD: General Architecture X 1 t 1 image Ancillary Data Data Collection X 2 t 2 image Data Pre-processing 1 Data Analysis N Change-Detection ti Products X N t N image 8

Data Collection: Sensors and Satellites Main issues to be considered Geometrical resolution: the geometrical detail required from the application is of major importance for the choice of the sensor and of both the pre-processing and data analysis algorithms. Spectral resolution: the complexity of the addressed problem and the end-user s requirements involve constraints on the choice of the kind of sensor (active or passive) and on the necessary spectral resolution. Acquisition frequency: high acquisition frequency often results in a high probability to include in the analysis data taken from different sensors (with important consequences in the data analysis phase). Whether conditions: the constraints on the robustness of the system to clouds or atmospheric conditions (e.g. related to the latitude of the investigated area) results in constraints for the choice of both the sensor and the satellite. 9 Data Collection: Application Requirements Ground truth: it is important to understand if it is possible to collect ground truth data for the training of the processing algorithms or if the system should be completely unsupervised. Prior information: it is important to collect all the prior information available on the application in order to translate it in possible useful constraints in the phase of design of algorithms. Ancillary data: it is important to identify possible ancillary data that can be used in the development of the system according to a multisource processing architecture. 10

Data Collection: User Needs and Requirements User expectation: given an application, it is important to properly understand the final goal of the users. Different specific goals result in the choice of different processing strategies and algorithms. Cost: the economical cost of the development of the architecture and of the operational use of the system should be properly understood. Processing time: it is important to identify the requirements (if any) related to the expected processing time for the developed service. 11 Data Collection: Requirements Most common assumptions of the techniques present in the literature Images uniformly sampled over the period of interest in corresponding periods on different years. Radiometric issues Sensor: should be possibly the same for all the images of the series; Acquisition season: Should be possibly the same for applications associated to the detection of abrupt changes; Should be properly sampled over the year for applications associated to the monitoring of the seasonal evolutions of phenomena. Atmospheric conditions: Different conditions in different portions of multitemporal images should be compensated (non-stationary effects of the atmosphere); Clouds should be properly detected and removed; Light conditions should be as similar as possible. 12

Data Collection: Requirements Geometrical issues Sensor: should be possibly the same for all the items of the temporal series to guarantee a similar acquisition geometry; Satellite orbit: should be possibly the same (ascending or descending); View angle: should be possibly the same, but it may change in multisensor sequences, or in series acquired by sensors with variable incidence angle. Topography: a complex topography makes ortho-rectification and geometrical corrections more difficult and strongly dependent on the quality of available digital elevation model (critical with different view angles). 13 Change Detection: General Architecture X 1 t 1 image Ancillary Data Data Collection X 2 t 2 image Data Pre-processing 1 Data Analysis N Change-Detection ti Products X N t N image 14

Data Pre-processing Original Images X 1 X N Very important with optical images t 1 image Radiometric Corrections t N image Mandatory in all change-detection techniques Depends on the specific sensor considered and on the quality of the considered images Depends on the requested acquisition frequency and data availability (careful application, see information theory!) Geometric Corrections & Image Registration Image Filtering Interpolating missing data X 1 X M 15 Data Pre-processing: Atmospheric Corrections Differences in light and atmospheric conditions between the two acquisition times can be mitigated by applying radiometric calibration to the images. Two different approaches can be applied: Absolute calibration: digital numbers are transformed into the corresponding ground reflectance values (radiometric transfer models, regression algorithms applied to ground-reflectance measurements collected during the data acquisition phase). Relative calibration: modification of the histograms, so that the same gray-levels values in the two images can represent the same reflectance values, whatever the reflectance values on the ground may be (histogram matching). The choice of one of the two approaches depends on the particular application considered and on the specific information available. 16

Data Pre-processing: Image Registration Generally it is not possible to obtain a perfect alignment between multitemporal images. This is mainly due to local defects in the geometries of the images. Residual misregistration results in a very critical source of noise, which is called registration noise Pixel at t 1 Pixel at t 2 Shift vs. error for 1 x 1 meter pixels ε Misregistration error Generic effect of misregistration Error in area as a function of the displacement in high resolution sensors nge imum error in chan area [m2] Maxi 1 0,8 0,6 0,4 0,2 0 0,00 0,25 0,5 0,75 Shift betw een images [m] Error 17 Data Pre-processing: Registration Noise Effects Elba Island, Landsat-TM4 1992 August t 1 image Change-detection map ember 19 994 Sept t 2 image Registration noise t 1 image 18

Data Pre-processing: Registration Noise In the remote-sensing literature, different approaches have been proposed to mitigate the effects of residual misregistration between images: Image smoothing approaches and adaptive filtering; Use of spectral bands where investigated changes are not visible for identifying the effects of registration noise on the specific scene analyzed and to reduce them in spectral channels where changes can be detected; t d Modeling approaches: assume that misregistration effects are locally uniform and can be estimated and compensated by analyzing the spatial brightness gradients. Adaptive approaches: estimates the distribution of registration noise in the specific scene of interest and exploits the estimation in the decision process. L. Bruzzone and R.Cossu, An Adaptive Approach to Reducing Registration Noise Effects in Unsupervised Change Detection, IEEE Trans. Geosci. and Rem. Sens., vol. 41, no.11, 2003. L. Bruzzone and S.B. Serpico, Detection of changes in remotely-sensed images by the selective use of multi-spectral information, Int. J. Rem. Sens.,, vol. 18, no. 18, pp 3883-3888, 1997. 19 CD: General Architecture X 1 t 1 image Ancillary Data Data Collection X 2 t 2 image Data Pre-processing 1 Data Analysis N Products X N t N image 20

Data Analysis: Taxonomy Supervised techniques Ground truth information required for all the images of the temporal series Stochastic techniques Based on the analysis and exploitation of statistical properties Partially supervised techniques of data (domain adaptation) Ground truth information not available for all the images of Deterministic techniques the temporal series Based on a priori i defined d rules based on the physical meaning of the calibrated signal Unsupervised techniques Ground truth information not available 21 Data Analysis: Binary CD Goal: To produce a binary change-detection map in which each pixel is associated with one of two possible classes: the class of changed pixel, ω c, or the class of unchanged pixels, ω n. Assumption: A significant ifi variation in the spectral response of corresponding pixels in the two images can be associated with a land-cover change. Operational conditions: the change detection problem can be solved either with supervised or unsupervised techniques. The second one is the most interesting from an operational point of view. 22

Data Analysis: Binary CD Elba Island, true color composite t 1994 August t 1 image Area affected by changes due to a forest fire tember 1 Sept 994 t 2 image Binary change-detection map 23 Data Analysis: Binary CD City of Bam, Iran Septemb ber 2003 t 1 image Changes related to damages after an earthquake January 200 04 t 2 image Binary change-detection map True color composite, Quickbird 24

Binary CD: Problems Binary change detection in remote-sensing images is characterized by several peculiar factors that render ineffective some of the multitemporal image analysis techniques typically used in other application domains. Some of these factors are: Differences in light conditions, sensor calibration, and ground moisture at the two acquisition dates considered; Absence of a reference background; Lack of a priori information about the shapes of changed areas; Non-perfect alignment (registration noise) between the two considered images; Different acquisition conditions of multitemporal images (view angle, shadows, etc.). 25 Binary CD: Methods Supervised Methods Binary classification techniques, one-class classifiers. Unsupervised Methods Techniques based on the generation and the analysis of the difference/ratio image (DI) (land cover changes can be detected by applying a decision threshold to the histogram of the DI); Techniques based on a multiscale analysis of the multitemporal images; Techniques based on multitemporal PCA; Image regression techniques; Other techniques based on linear data transformations (e.g., MAD). Operational Issues Only unsupervised methods satisfy the requirements of most of the applications. 26

Binary CD: Typical Architecture Corrected t 1 image X 1 Comparison X D Analysis X 2 Corrected t 2 image Difference/Ratio Image Change-detection map Ω ={ω c, ω u } 27 Binary CD: Comparison Operators Technique Feature vector at the time t k Computation of b diff i f k = X k XD = f1 f2 + C Univariate image differencing Vegetation index differencing k k f f k =V X D 1 2 X D = f f + C Change vector analysis 1 m fk = [ Xk,.., Xk ] X D = f1 f2 Regression f1 = X1 b and ˆ b f2 = X 2 X D = f f + C 1 2 Principal component Analysis 1 fk = [ Pk,.., Pk m ] X D = f1 f2 28

Binary CD: Comparison Operators f Technique Feature vector at the time t k Computation of k X D Image rationing b fk = X k X D = f2 / f1 Similarity measures = [ px ( )] fk k KL( X2 X1)= log ( f1/ f2) f1 VV SAR backscattering coefficient differencing Difference of scattering matrix element products Difference of scattering matrix correlation coefficients f k f = [ σ ] X D = f1 f2 k VV * fk = [ SHH SVV ] X D = f1 f2 S S * HH VV = 2 2 SHH S VV X = f f D 1 2 29 Binary CD: Comparison Operators MOST COMMONLY USED COMPARISON OPERATORS Difference Vector difference Ratio Log-ratio Information theoretical similarity measures Multispectral Images SAR Images SAR Multispectral Images Multisensor 30

Change Vector Analysis (CVA) Pixel (i,j) Magnitude Image Multispectral image t 1 X 21 Multispectral image t 2 X 1 X 1 x 2 Vector X Difference D1 X 2 Multispectral difference image X 1 X D ρ = + 2 2 ( X1, D ) ( X 2, D ) ϑ = tan x 1 1 X 1, D X 2,D Direction Image 31 Polar Change Vector Analysis Polar Domain D = ρ, ρ< ρ max < { ϑ : 0 and 0 ϑ 2π} Circle of unchanged pixels = T C n c { ρ, ϑ : 0 ρ < and 0 ϑ < 2π } Annulus of changed pixels A = ρ, T ρ < { ϑ : ρ max and 0 ϑ < 2 π } A c C n D T ρ max ρ ϑ 0 Sector of changed pixels S = ρ ρ T < < < {, ϑ: T and ϑ ϑ < ϑ, 0 ϑ < ϑ < 2π 1 2 1 2 } k k k k k S k 32

Polar Change Vector Analysis: Example Study area: Lake Mulargia, Sardinia Island (Italy). Multitemporal data set: a portion of 412 300 pixels of two images acquired by the TM sensor of Landsat-5 satellite in September 1995 and July 1996. Before Change After Change Reference Map 33 Polar Change Vector Analysis: Example Corrected Images (Ideal case) Registration noise effects Radiometric difference effects Optimal threshold value on the magnitude variable: ideal case Registration noise effects Threshold value on the magnitude variable: radiometric distortion case F. Bovolo and L. Bruzzone, A Theoretical Framework for Unsupervised Change Detection Based on Change Vector Analysis in Polar Domain, IEEE Trans. Geosci. Remote Sens., vol. 45, no. 1, 2007. 34

Binary CD: Typical Architecture Corrected t 1 image X 1 Comparison X D Analysis X 2 Corrected t 2 image Difference/Ratio Image Change-detection map Ω ={ω c, ω n } 35 Binary CD: Difference Image Analysis Different techniques have been proposed in the remote-sensing literature for the analysis of the difference/ratio image: Trial-and-Error Techniques - Non-automatic ti procedures based on the interactive ti selection of the threshold (carried out manually by an operator); Empirical Thresholding Techniques - In optical data a single hypothesis testing procedure is adopted (threshold is fixed at nσ from the mean value of the difference image). In SAR data decision threshold is fixed to satisfy a given value for the probability of false alarms; Unsupervised Bayesian Framework - Exploits automatic techniques based on both implicit or explicit parameter estimation in an unsupervised framework, and performs change detection either with a context sensitive or insensitive method. 36

Binary CD: Difference Image Analysis Fuzzy set techniques Model the ambiguity within the pixels that arises from the overlapping of classes and perform threshold selection. Context-sensitive techniques Assume that pixels are likely to be surrounded by pixels belonging to the same class. Therefore takes advantage from the use of spatial interpixel dependence. Multiscale analysis Models changes at different resolution levels for properly representing complex homogeneous objects in the investigated scene as well as borders of the changed areas and change details. As an example, in the following we focus the attention on the unsupervised Bayesian framework. 37 Difference Image Analysis: Bayesian Framework The discrimination between changed and unchanged classes (in the histogram of the magnitude image) can be carried according to algorithms inspired to the Bayes classification criterion Unchanged (ω n ) h(x D ) Changed (ω c ) T Gray-level value ( XD ) = P ( ωc ) ( XD ωc ) + P ( ωn ) ( XD ωn ) p X P p X P p X Problem: the estimation problem should be solved in an unsupervised way. 38

Difference Image Analysis: Bayesian Framework Pixel based Direct detection of the decision threshold with implicit estimation of the statistical terms of classes Explicit estimation of the statistical terms of classes with the Expectation-Maximization (EM) algorithm Bayes rule Minimum error Bayes rule Minimum cost Context based Markov Random Fields (MRF) based regularization strategy 39 Bayesian Framework: EM-Based Technique The problem can be solved with a 3-step procedure: Model definition and Initialization; Iterative estimation of the statistical terms of the difference image; Decision. X D Initialization Iterative estimation (EM algorithm) Decision M Difference Image ε Change-detection map L. Bruzzone, D. Fernández Prieto, An adaptive semi-parametric and context-based approach to unsupervised change detection in multitemporal remote sensing images, IEEE Transactions on Image Processing, Vol. 11, No. 4, April 2002, pp. 452-466. L. Bruzzone, D. Fernández Prieto, Automatic analysis of the difference image for unsupervised change detection, IEEE Trans. Geosci. Rem. Sens., Vol. 38, No.3, May 2000, pp. 1171-1182. 40

Bayesian Framework: Statistical Models 1 2πσ D Gaussian model: p ( XD ωi ) = exp 2 i 2 ( X D μ i ) 2σ i Mean value Variance Rayleigh model: X X px ( D ωi) = exp σ 2 D D 2 2 i 2σ i Rice model: 2 2 XD XD + Mi XDMi p ( XD ωi ) = exp I 2 2 0 2 σ i 2σ i σ i Non centrality parameter Semi-parametric mixture model: R 2 1 i 1 ( XD Yr) px ( D ωi) = k R i r= 1 h k Kernel function 41 Bayesian Framework: EM-Based Technique Initialization of the parameters of the selected statistical model: A priori knowledge available: define two sets on the basis of the expected structure of the histogram such that pixels in them have a high probability to belong to ω n and ω c, respectively, and compute initial parameter values. A priori knowledge unavailable: initialize parameters by applying a technique based on genetic algorithms. Y. Bazi, L. Bruzzone, F. Melgani, Image Thresholding Based on the EM Algorithm and the Generalized Gaussian Distribution, Pattern Recognition, Vol. 40, No. 2, 2007, pp 619-634. 42

Bayesian Framework: EM-Based Technique P At each iteration of the EM algorithm the value of the likelihood function L(θ) between the difference image distribution and the estimated mixture distribution increases. A local maximum of the log-likelihood function is reached at convergence. It is possible to show that the EM iterative equations for the Gaussian model are as follows: t t t t P P ωn p X i, j ( ωn) p ( ) / ωn ( X i, j / ωn ( ) ( ( ) ) ) X i (, j) t t X(, i j) X D p ( ) ( ) ( ) X(, i j) X D ( X i, j ) p X i, j t+ 1 μn = ( ωn) t t P ( ωn) p ( X ( i, j) / ωn) IJ t X(, i j) X D p ( X ( i, j) ) t t P ( ω ) ( ( ) ) n p X i, j / ω 2 n X ( i, j) (, ) ( (, ) 2 1 ) t t n X i j p X i j μ X t+ D ( σ n ) = t t P ( ωn) p ( X ( i, j) / ωn) t X(, i j) X D p X ( i, j) t+ 1 = ( ) 43 Bayesian Framework: EM-Based Technique At the convergence of the iterative algorithm, the parameters estimated can be used according to three different strategies: Bayes rule for the minimum error (BRME) (pixel based): P P ( ωc ) p ( X D / ωn ) = ( ωn) p( XD / ωc) Bayes rule for the minimum cost (BRMC) (pixel based): Ratio between the costs of missed and false alarms P kp ( ωc ) p ( XD / ωn ) = ( ωn) p( XD/ ωc) 44

Bayesian Framework: EM-Based Technique Bayes rule for the minimum error with Markov Random Fields (MRF) for modeling the prior terms of classes (context based). Minimization of the following oo genergye function: cto ( ) ( ) context l (, ) { l (, ), (, ) N(, ) } U (X D,C l ) = Udata XD (, i j ) Cl ( i, j ) + U C i j C g h g h i j 1 i I 1 j J Grey levels statistics energy term Interpixel class dependence energy term Neighborhood system (i-1,j) (i,j-1) (i,j) (i,j+1) (i+1,j) First Order Neighborhood system Second Order Neighborhood system (i-1,j-1) (i-1,j) (i-1,j+1) (i,j-1) (i,j) (i,j+1) (i+1,j-1) (i+1,j) (i+1,j+1) 45 Data Analysis: Multiclass CD Target: Generate a change-detection map in which changed areas are separated from unchanged ones and each land-cover transition type can be explicitly identified (i.e., the pair of classes associated with a generic pixel at two times are detected). Assumption: Each data class (or each set of data classes) in the timespectral domain of the two images can be associated with a specific landcover transition (or with a specific unchanged land-cover class). Operative conditions: only supervised or semi-supervised techniques result in an explicit labeling of land-cover transitions. 46

Multiclass CD: Example URBAN URBAN BARE SOIL SUGAR BEET WHEAT BARE SOIL BARE SOIL CORN BARE SOIL SOYBEAN False color composition of multispectral images acquired in May 1995 and July 1995 on an agricultural area in Italy Multiclass change-detection map (map of land-cover transitions) 47 Multiclass CD: Example SEVERE DAMAGED AREAS MEDIUM DAMAGED AREAS UNDAMAGED AREAS Pre and post-tsunami SAR images of two different test sites (December 2006, Indonesia) Multiclass change detection map 48

Multiclass CD: Example Before Changes A c T ϑ 1 2 Lake surface S 1 ϑ ϑ 1 2 2 Burned area Lake surface enlargement No-change C n S 2 Burned area ϑ 2 1 Reference Map After Changes F. Bovolo and L. Bruzzone, A Theoretical Framework for Unsupervised Change Detection Based on Change Vector Analysis in Polar Domain, IEEE Transactions on Geoscience and Remote Sensing, Vol. 45. No. 1, pp. 218-236 236, Jan 2007. 49 Multiclass CD Supervised Methods (ground truth is available for all the acquired temporal images): Post-Classification Comparison (independent classification maps are compared); Direct Multidata Classification (each transition is considered as a specific class); Compound Classification (exploits the maximization of the posterior joint probability of classes). Partially-Supervised Methods (ground truth is not available for all the images of the temporal series): Lacking information is reasonably inferred from theanalysis of thespatial and/or the statistical distribution of unlabeled pixels (domain adptation methods). 50

Multiclass CD Unsupervised Methods (ground truth is not available): Change vector analysis technique with proper analysis of the phases of the spectral change vectors (different phases intervals are associated with different changes); Operational Considerations: Supervised (and in some cases partially supervised) algorithms provide high accuracies; Only supervised and partially supervised techniques provide an explicit label to detected land-cover transitions; Collection of ground truth is a complex and time-consuming task. 51 Multiclass CD: Post-Classification Comparison X 1 t 1 image Supervised classifier Supervised Estimation of Classifier Parameters Land-cover Map t 1 Map Comparison Different kinds of changes X 2 Training set Y 1 Supervised classifier Map of land-cover transitions t 2 image Supervised Estimation of fclassifier Parameters Training set Y 2 Land-cover Map t 2 Problem: uncorrelated classification errors increase the overall change detection error 52

Multiclass CD: Direct Multidate Classification X 1 t 1 image Supervised Classifier X 2 t 2 image Supervised Estimation of Classifier Parameters Map of land-cover transitions Training set Y 1 Training set Y 2 Problem: ground truth is required for both images and for the same pixels in the two images 53 Multiclass CD: Compound Classification X 1 t 1 image Different kinds of changes Supervised Compound Classifier Land-cover Map t 1 Map Comparison Map of land-cover transitions X 2 px/ 1 ( j ω k ) px / p(ω n,ω k) 2 ( j ω n) Partially-Supervised Estimation of fclassifier Parameters Training set Y 1 Training set Y 2 Land-cover Map t 2 L. Bruzzone, D. Fernández Prieto, S.B. Serpico, A neural-statistical approach to multitemporal and multisource remote-sensing image classification, IEEE Transactions on Geoscience and Remote Sensing Sensing, Vol. 37, No. 3, pp. 1350-1359. 54

Multiclass CD: Partially-Supervised Compound Classification X 1 t 1 image Different kinds of changes Supervised Compound Classifier Land-cover Map t 1 Map Comparison Map of land-cover transitions X 2 px/ 1 ( j ω k ) px / p(ω n,ω k) 2 ( j ω n) Partially-Supervised Estimation of fclassifier Parameters Training set Y 1 Training set Y 2 Land-cover Map t 2 L. Bruzzone, R. Cossu, G. Vernazza, Detection ti of landcover transitions by combining multidate classifiers, Pattern Recognition Letters, Vol. 25, No. 13, 1 October 2004, pp. 1491-1500. 55 CD in Long-Time Series Target: Detect tchanges in the seasonal (or yearly) behavior of the temporal behaviour of a pixel (e.g., comparison of the temporal signatures of two series of images acquired by a sensor in the same season at different years). Assumptions Modifications in the behavior of the seasonal temporal signature of corresponding pixels are associated with changes; The number of observations in the time series should be sufficient for the characterization of the investigated phenomenon (theoretically related to the Nyquist sampling rate). Operative Conditions: Supervised and unsupervised signal processing techniques can be used to extract change information by comparing temporal signatures. 56

CD in Long-Time Series 200 180 Loss of vegetation ND DVI 160 140 120 100 1989 1997 200 Re-vegetation 180 DVI 160 N140 120-1.0 0 +1.0 100 1989 1997 57 CD in Long-Time Series Pixel-Based Methods Fourier analysis; Standardized PCA of time series; Kalman filter; Regression techniques; Vegetation Cover Conversion (VCC); Neural networks based approaches (MLP and RBF); Autumn colour monitoring method; Contrast of two times series. Contex-Based Methods Integration of neighboring pixels information and time series transition probabilities; Three-dimensional clustering; Maximum-likelihood detectors; Region-based approaches. 58

CD: Integrated Analysis In some applications the time series can be analyzed by integrating ti the results obtained from change detection applied to pairs of images: High levelel joint analysis of binary change detection maps extracted from long time series; Analysis of maps of land-cover transitions for identifying temporal patterns related to land-use and/or land-cover dynamic; Joint analysis of similar products extracted from images acquired by satellites with different sensors; Joint analysis of short term and long term products to perform more robust and reliable results. 59 LOGO Change Detection in Very High Resolution Multispectral Images

CD in Multitemporal VHR images What are the main differences between change detection in medium resolution (MR)/high resolution (HR) images and change detection in very high resolution (VHR) images? What are ae the most critical ca problems pobe in the analysis ayss of multitemporal VHR images? Can change detection techniques developed for MR/HR images be applied to VHR images with satisfactory results? 61 CD in Multitemporal VHR images July 2006 October 2005 Quickbird images of the city of Trento (Italy) 62

CD in Multitemporal VHR images: Problems In VHR images it is particular important t to carefully distinguish i between: Radiometric changes; Semantic changes. Radiometric changes: statistically significant changes between the measured radiometry of corresponding pixels (or objects) in two images acquired at different times. Semantic changes: changes occurred on the ground between the acquisition of the two considered images. In VHR images often radiometric changes do not represent semantic changes (effect much more critical than in MR/HR images). 63 CD in Multitemporal VHR images: Problems Why radiometric changes in corresponding pixels (or objects) of VHR images often do not represent semantic changes? Different geometry (view angle) that results in the acquisition of different parts of the objects on the ground; Difficult co-registration (high amount of residual registration noise); Different extensions of shadows (different acquisition seasons and/or view angles); Saturation effects and artefacts. If not properly modelled these effects lead to a sharp increase of the number of false alarms in the change-detection map. 64

CD in VHR Images: Data Pre-processing Raw Images Registration Ortho-rectification i Radiometric corrections Cloud detection Topographic corrections Pan-sharpening Image filtering Feature extraction Characterization of the noise sources Mosaiking Pre-processed data 65 CD in VHR Images: Additional Pre-processing Pan-sharpening: fusion between PAN and MS images. Geometric corrections and co-registration: corrections based on the acquisition architecture and alignment based on the exploitation of structured primitives. Modeling of the residual registration noise: adaptive estimation of the distribution (and the expected effects) of registration noise. Shadow detection and modeling of sources of noise: it is important to model precisely all the sources of noise that result in spectral changes for increasing the reliability and the accuracy of the change detection process. F. Bovolo, L. Bruzzone, L. Capobianco, A. Garzelli, S. Marchesi, F. Nencini, Change detection from pan-sharpened images: a comparative analysis, ESA-EUSC 2008: Image Information Mining: pursuing automation of geospatial intelligence for environment and security, ESRIN, Frascati (Italy), March 4-5, 2008. 66

CD in Multitemporal HR Images: Example August 1994, TM band 4 September 1994, TM band 4 Magnitude Difference Image Pixel-Based Change Detection Map 67 CD in Multitemporal VHR Images: Example October 2005 July 2006 QB band 4 QB band 4 Magnitude Difference Image Pixel-Based Change Detection Map 68

CD in Multitemporal VHR images Standard pixel-based change-detection techniques developed for the analysis of medium resolution images fails when dealing with VHR images as they do not explicitly considers: the residual misregistration (registration noise) among multitemporal images; the high spatial correlation between pixels in a neighborhoods; the intrinsic multiscale nature of objects present in VHR images. 69 Multitemporal VHR images: Registration Noise Even after proper registration and geometrical corrections, VHR multitemporal images generally show a significant ifi residual registration ti noise (RN). Original i Quickbird Image Multitemporal False Color Composite Problem: the comparison between non perfectly aligned pixels leads to a sharp increase of the number of false alarms in border regions. 70

Multitemporal VHR images: Registration Noise How identifying and mitigating the registration noise in the change detection process? This can be done only understanding in detail the behaviour of RN in VHR images: What are the properties of RN in VHR images? Is it possible to model the RN distribution ib ti between multitemporal l VHR images? Can the estimated distribution used in change-detection technique? 71 Registration Noise Properties In order to derive RN properties p we applied the polar CVA to multitemporal datasets defined in different ways. Available data: portion (984 984 pixels) of an image acquired by the Quickbird satellite in July 2006 over the City of Trento (Italy). July 2006 72

Registration Noise: Property 1 Effects of increasing misregistration on unchanged pixels. Multitemporal datasets are made up of the available image X 1 and misregistered copies of it (X 2 ) (different amount of residual misregistration were considered). Simulated multitemporal false color composites Misregistration=2 pixels Misregistration=6 pixels Reference Image X 1 73 Registration Noise: Property 1 Misregistration = 2 pixels Misregistration = 6 pixels C n C n 74

Registration Noise: Property 1 PROPERTY 1 RN affects unchanged pixels by: Increasing the spread of the cluster in the circle of unchanged pixels C n with respect to the case of perfectly aligned images. The spread increases by increasingi the amount of RN. Generating clusters of dominant registration noise in the annulus of changes A c, which have properties very similar to those of changed pixels. The number of patterns in these clusters increases by increasing the amount of RN. 75 Registration Noise: Property 2 Effects of increasing misregistration on changed pixels. Multitemporal datasets are made up the available image X 1 and misregistered copies of it (X 2 ) including simulated changes (different amount of residual misregistration were considered). Simulated multitemporal false color composites Perfectly aligned Misregistration = 6 pixels Simulated Image X 2 76

Registration Noise: Property 2 PROPERTY 2 Clusters associated with changed pixels in A c exhibit nearby stationary statistical properties versus the amount of RN. Perfectly aligned Misregistration = 6 pixels Perfectly aligned Misregistration = 2 pixels Misregistration = 6 pixels 77 Registration Noise: Properties 3 and 4 Effects of misregistration at different scales. Multitemporal datasets are made up of the available image X 1 and a 4-pixel misregistered copy of it (X 2 ) including changes; the Daubechies-4 stationary wavelet transform was used for decreasing the resolution. Simulated multitemporal false color composite (full resolution) Simulated multitemporal false color composite (low resolution level) 78

Registration Noise: Properties 3 and 4 Multitemporal data set (Misregistration = 4 pixels) Full resolution (Level 0) Reduced resolution (Level 3) 0 X 1 0 X 2 3 3 X 1 X 2 C n C n A c T A c T ρ 0 ρ 3 79 Registration Noise: Properties 3 and 4 PROPERTY 3 Clusters of dominant registration noise in A c exhibit non-stationary statistical properties versus the scale (resolution) of the images. PROPERTY 4 Clusters associated with changed pixels in A c exhibit nearby stationary statistical properties versus the scale (resolution) of the images. 80

Estimation of Registration Noise Distribution X 11 VHR Multispectral image t 1 X 21 VHR Multispectral image t 2 Multiscale Decomposition Multiscale Decomposition M u l t i C s V c A a l e X X 1 0 D X X 1 N D Multiscale Difference Images Differential Analysis for RN Estimation Estimated RN Distribution ib ti in the direction domain F. Bovolo, L. Bruzzone, S. Marchesi, Analysis and Adaptive Estimation of the Registration Noise Distribution in Multitemporal VHR Images, IEEE Transactions on Geoscience and Remote Sensing, submitted. 81 Estimation of RN Distribution: Example Study area: South part of Trento (Italy). Multitemporal data set: portion (984 984 pixels) of two images acquired by the Quickbird satellite in October 2005 and July 2006. October 2005 July 2006 Reference Map 82

Estimation of RN Distribution: Example Le evel 0 ρ = T A c Normalized conditional densities of the direction of pixels in A c P0( ρ T) pˆ 0( ϑ ρ T) PN ( ρ T) pˆ N( ϑ ρ T) C n Le evel N ρ = T C n Adopted estimation technique: Parzen windows A c pˆ n ( ϑρ T) Number of SCVs in A c at level n M 1 n 1 ϑ ϑm = γ Mn m= 1 h n h n Smoothing parameter at level n 83 Estimation of RN Distribution: Example We define the marginal conditional density of registration noise in the direction domain as : ( ) ( ) 0 0 pˆ RN ( ϑρ T) = K P ρ T p ( ϑρ T) P N ρ T p N ( ϑρ T) contstant 84

Estimation of RN Distribution: Example Use of the estimated registration noise distribution for detecting sectors of dominant registration noise: S RN1 A c C n Le evel 0 S RN2 T RN S RN1 { ϑ ϑ } { ϑ ϑ } S RN 1 = ρ, : ρ 220 and 40 < 120 SRN 2 = ρ, : ρ 220 and 225 < 270 A c S RN 2 C n Lev vel n 85 Estimation of RN Distribution: Example Use dominant registration noise sectors to produce a map of registration ti noise: M RN ω if [ ρ( i, j), ϑ( i, j)] S (, i j) ωc ωn, otherwise RN RN k S RN1 A c C n S RN2 Registration-noise map 86

CD in VHR Images The estimation of the registration noise distribution can be used in the change detection algorithm for mitigating the effects of the residual misalignment between images. However, we still have to consider two important issues: the high spatial correlation between pixels in a neighborhood; the intrinsic multiscale nature of objects present in VHR images. How modeling high spatial correlation between pixels in a neighborhood in the change-detection process? 87 Context-Sensitive CD in VHR Images CVA at Full Estimation of the RN Resolution Distribution X 1 X 2 1 X 1 VHR Multispectral image t 1 X 2 1 VHR Multispectral image t 2 Multiscale Decomposition Multiscale Decomposition M u l t C i V s A c a l e F. Bovolo, L. Bruzzone, S. Marchesi, A Context-Sensitive Technique Robust to Registration Noise for Change Detection in Very High Resolution Multispectral Images, IEEE 2008 Int. Geosci. Rem. Sens. Symp.,,(IGARSS '08), Boston, Massachusetts, USA., 6-11 July 2008 X X0 D1 X N D 1 Multiscale Difference Images E s t i R m N a t i o n Vector Difference Context Sensitive Analysis Multitemporal segmentation Parcel-based Context Sensitive e Analysis XX X 1 X 1 X 2 CD Map 88

Context-Sensitive CD in VHR Images XXX 1 Segmentation VHR Multispectral image t 1 Segmentation Map t 1 Multitemporal Fusion X 2 1 Segmentation Multitemporal- Parcel Map VHR Multispectral image t 2 Segmentation Map t 2 k i R i R i number of pixels Ri that belong to a cell of RN or to C ki non-change if 0.5 ni change otherwise n number of pixels R i 89 Context-Sensitive CD in VHR Images: Example Study area: South part of Trento (Italy). Multitemporal data set: portion (984 984 pixels) of two images acquired by the Quickbird satellite in October 2005 and July 2006. October 2005 July 2006 Reference Map 90

Context-Sensitive CD in VHR Images: Example Change Detection False Missed Total Procedure Alarms Alarms Errors Overall Accuracy (%) Standard approach (supervised) sed) Multiscale parcel-based (unsupervised) 55921 5808 61729 93.63 29616 3382 32998 96.59 Reference map Standard approach Multiscale context-sensitive technique 91 CD in VHR Images There is still an open issue to be considered d and discussed: d the objects in the scene have different scale factors. Thus it is important to introduce multiscale or multilevel strategies in the change detection process. Different techniques can be used for a multiscale decomposition of multitemporal VHR images and for a multiscale analysis, e.g., Wavelets; Gaussian decomposition; Morphological filters. In the following, as an example, an adaptive method based on a multilevel hierarchical segmentation is described. M. Dalla Mura, J.A. Benediktsson, F. Bovolo, L. Bruzzone, An Unsupervised Technique Based on Morphological Filters for Change Detection in Very High Resolution Images, IEEE Geosci. Rem. Sens. Lett., Vol. 5, No. 3, pp. 433-437, July 2008. 92

Multilevel Binary CD in VHR Images X1 X 1 Hierarchical segmentation 0 SM 1 L SM 1 t 1 VHR image 0 SM 2 Multitemporal hierarchical segmentation maps fusion X1 Hierarchical 2 segmentation L SM 2 Spatio-temporal hierarchical segmentation t 2 VHR image P 0 P L Multilevel parcel-based feature extraction x 1 x 2 Multilevel parcel-based CVA M Ω ={ω c, ω n } l SM h S t ti M t l ti l l l t d f i iti d t h SM : Segmentation Map at resolution level l computed for acquisition date h. l l l P : Parcel Map at resolution level l computed by fusing SM 1 and SM 2. x : Multilevel parcel-based feature vectors computed at time h. M h : Change detection map. 93 Hierarchical multilevel segmentation Objective: obtain a set of segmentation maps (one for each level) characterized from hierarchical non ambiguous relationships among them. 3 O 1 2 O 1 1 O 1 pixel O 1 L 3 L 2 L 1 Pixel level e 94

Hierarchical multilevel segmentation The segmentation map at a generic level l is a partition P l (X) in a set of N l disjoint regions ol s (i.e., P l (X) = { 1 l l o,, on }), with X and s,r = 1,2,, N l l ol s, and such that l = 0,, L-1, and r s: 1. pixels in l are spatially contiguous; N l l 2. ; o s = X s= 1 l l 3. o ; o = s r o s o s 4. H l ( l) = true; 5. for each pair of regions l,, H l l l o o ( o o ) = false; 6. l 1 N k q= 1 o l 1 q = o l k s l r s Hierarchical constraint r 95 Hierarchical multilevel segmentation Level 3 Pixel Level Level 1 (L 1 ) Level 2 Level 1 Pixel Level Level 2 (L 2 ) Level 3 (L 3 ) 96

Multitemporal Hierarchical Segmentation The procedure for generating parcel maps is based on two steps: Independent hierarchical segmentation of the two considered images; Fusion of the two segmentation maps (SM). 1X1 X 1 1 Homogeneity in the spatial domain Homogeneity in the temporal domain X 1X2 1 Parcel map 97 Multitemporal Hierarchical Parcels: Example Pixel level Parcel level 4 Parcel level 5 Parcel level 6 Parcel level 7 Parcel level 8 98

Multilevel Change Vector Analysis Perform hierarchical multilevel feature extraction for each multemporal parcels (hierarchical parcel-based feature extraction). For each pixel, comparison between X 1 and X 2 is carried out by applying the Change Vector Analysis to the spectral values and the related hierarchical context-based features (mean vectors) computed at the two dates: X D ( i, j ) = x ( i, j ) x ( i, j ) 2 1 The difference image M D contains information about changes at different resolution levels. The change-detection map is produced according to a decision threshold (Th) obtained either manually or automatically: ωc M( i,j) ωu X ( i, j) > Th D X ( i,j ) Th u D,j 99 Multilevel Binary CD in VHR Images: Example Study area: urban/rural area, south zone of Trento (Italy). Multitemporal data set: portion (1090 740 pixels) of two scene acquired by Quickbird on a Trentino area (Italy) in October 2005 and July 2006; Objective: identify the changes associated to building and road yards. Image X 1 Image X 2 100

Multilevel Binary CD in VHR Images: Example Pixel-based and single-level parcel based technique Considered Levels False Missed Overall Level Alarms Alarms Error Pixel 32811 22389 55200 1 23373 19663 43036 2 23957 6497 30454 3 19524 10805 30329 4 16414 7701 24115 Multilevel parcel-based technique False Alarms Missed Alarms Overall Error Pixel,1 23564 16757 40321 Pixel,1,2 21231 7944 29175 Pixel,1,2,3 16906 6549 23455 Pixel,1,2,3,4 10112 5598 15710 101 Multilevel Binary CD in VHR Images: Example Change-Detection Maps Sta andard pix xel-based Si ingle-leve l parcel-ba ased Mu ultilevel Pa arcel-base ed Referen ce map 102

Multilevel Binary CD in VHR Images: Discussion Multilevel segmentation seems a better strategy than other multiscale decomposition methods to model spatio-temporal context and multiscale properties of VHR images as: Adaptively models spatial context of each pixel according to segments; Models the intrinsic multilevel spatial context information present in VHR images; Shows both high sensitivity to geometrical details and high robustness to noisy components in homogeneous areas. Is intrinsically robust to the presence of noisy components. 103 Multiclass CD in VHR Images Multiclass change detection in VHR images can exploit the same architectures adopted in MR and HR change detection, i.e. Post-Classification Comparison (PCC); Direct Multidate Classification (MDC); Compound Classification (CC). The main issue in VHR images is related to the design of classification techniques suitable to the properties of VHR data. 104

Multiclass CD in VHR Images with PCC Ω ={ω 1,, ω N } X 1 Hierarchical segmentation Hierarchical Feature extraction Classification Module t 1 VHR image Land cover transition map Map Comparison X 2 Hierarchical segmentation Hierarchical Feature extraction Classification Module Ω ={ω 1 ω 2, ω 1 ω 3, ω N ω N-1 } t 2 VHR image Ω ={ω 1,, ω N } 105 Hierarchical multilevel segmentation Level 3 Pixel Level Level 1 (L 1 ) Level 2 Level 1 Pixel Level Level 2 (L 2 ) Level 3 (L 3 ) 106

Hierarchical Multilevel Feature Extraction Level 3 Spectral features Mean, Standard deviation Spatial features Area, Shape factor Relational features Number of sub-objects Level 2 Spectral features Mean, Standard deviation Spatial features Area, Shape factor Level 1 Spectral features Mean, Standard deviation Pixel Level Spectral features Digital number values 107 Hierarchical Multilevel Feature Extraction The size of the feature vector is related to the depth of the tree and to the number and kind of feature extracted for each level. L3 Level 1 Level 2 Level L {,...,,...,...,,... 1,1 1, n 2,1 2, n L,1 L, n } x= f f f f f f 1 2 2 L2 L1 Hierarchical Feature extraction For each level the number and kind of extracted features may be different. At lower levels we can choose spectral features (spatial context is less relevant). L0 At higher levels we can choose spatial features (because the spectral disomogeneity). Problem: very high number of features can result in the Hughes phenomenon in the classification phase. L. Bruzzone, L. Carlin, A Multilevel Context-Based System for Classification of Very High Spatial Resolution Images, IEEE Transactions on Geoscience and Remote Sensing, Vol. 44, No. 9, 2006, 2587-2600. 2600 108

Classification Module Solution: use the SVM classification approach, which was proven robust to work in high dimension feature space. Class 1 Support Vector Class 2 Transformed space A pattern x is associated with one of the two classes according to the function sign[f(x)], where f(x) is the discriminant function associated with the optimal hyperplane and is defined as: f(x) = w Φ(x) + b* SVMs map the data into a higher dimensional feature space, where classes are separated by a hyperplane defined by a vector w andabiasb. 109 CD in Temporal Series of VHR Images What about change detection in long temporal series of VHR remote sensing images? Some techniques are presently under development for addressing this kind of challenging and important problem (see the part of this presentation on VHR SAR images for an example). However the main problem (now almost solved) is related to the very limited data availability. 110

Conclusion on CD in Multispectral VHR Images Automatic change detection in multispectral VHR images is very important for a wide range of applications. Some promising techniques have been proposed in the scientific literature for analyzing VHR data. However, there are many open issues and a significant amount of work to carry out in the next years. Some interesting research directions are related to: Improve pre-processing techniques (e.g., automatic co-registration procedures); Define models and techniques for discriminatingi i spectral (radiometric) i changes from semantic changes; Define advanced multiscale and multilevel modeling of multitemporal data. Develop data fusion techniques and change detection on hybrid multitemporal VHR images. 111 LOGO Change Detection in Very High Resolution SAR Images

CD in Multitemporal SAR images What are the main differences between change detection in optical and SAR images? Can change detection techniques developed for optical images be applied to SAR images? What are the main differences between change detection in medium resolution (MR)/high resolution (HR) SAR images and change detection in very high resolution (VHR) SAR images? Can change detection techniques developed for MR/HR SAR images be applied to VHR SAR images with satisfactory t results? 113 CD: General Architecture X 1 t 1 image Ancillary Data Data Collection X 2 t 2 image Data Pre-processing 1 Data Analysis N Products X N t N image 114

Data Pre-processing: SAR Images Raw Images Focusing Geometric corrections Radiometric corrections Radiometric normalization Digital Elevation Model Single-image and/or Mutitemporal despeckling Mosaiking Pre-processed SAR data 115 Binary Change Detection: Block Scheme Corrected t 1 image X 1 Comparison X D Analysis X 2 Corrected t 2 image Log/Ratio Image Change-detection map Ω ={ω c, ω n } 116

Binary CD: Comparison Operators Technique Feature vector at the time t k Computation of Image rationing Similarity measures = [ px ( )] f k b fk = X k X D = f2 / f1 X D fk k KL( X2 X1)= log ( f1/ f2) f1 VV SAR backscattering coefficient differencing Difference of scattering matrix element products Difference of scattering matrix amplitude correlation coefficients f k f = [ σ ] X D = f1 f2 k VV f * k = SHHSVV X = f f * S HH S VV = 2 2 SHH S VV D D 1 2 X = f f 1 2 117 Comparison Operators Difference operator: Equivalent number of looks Radar cross section L L 1 L 1 ( 1 )! 1 1 2 ( 1, 2) I L LX D L + j L j σσ PX D σ σ = exp X D ( L 1! ) σ1 + σ2 σ 2 j= 0 j!( L 1 j)! L ( σ 1 + σ 2 ) j The statistical distribution of the difference image depends on both: the relative change between the intensity values in the two images; a reference intensity value. This leads to a higher change-detection error for changes occurred in high intensity regions of the image than in low intensity regions (i.e., changes are detected in a different way in dark and bright regions. 118

Comparison Operators Ratio operator: reduces the multiplicative distortions common to the two considered images due to speckle noise; makes the distribution of the ratio image depending only on the relative changes between images. Log-ratio operator: Equivalent number of looks ( 2 L 1 )! L X D 2 ( L 1! ) [ 2 1 X D ] L 1 σ 2 PX (, ) D σ σ = σ 1 σ σ + 1 2 2L Radar cross section produces more symmetrical statistical distribution of the classes of changed and unchanged pixels; transforms the residual multiplicative li noise model into an additive noise model. log( X ) = log( f / f ) = log( f ) log( f ) D 2 2 2 1 119 Binary CD: Log/Ratio Image Analysis The discriminationi i between changed and unchanged classes (in the histogram of the log/ratio image) can be carried according to algorithms inspired to the Bayes classification criterion. Changed (ω c1 ) Unchanged (ω n ) h(x D ) Changed (ω c2 ) T 1 T 2 Gray-level value N px ( ) = P ( ω ) px ( ω ) + P ( ω ) px ( ω ) N = 0,1, 2 D n D n ci D ci i= 0 PROBLEM: the estimation problem should be solved in an unsupervised way. 120