Knowledge Discovery From Global Remote Sensing and Climate Data: Results from Supervised and Unsupervised Data Mining

Transcription

1 Knowledge Discovery From Global Remote Sensing and Climate Data: Results from Supervised and Unsupervised Data Mining Mark Friedl Department of Geography and Environment, Center for Remote Sensing, Boston University, 675 Commonwealth Avenue, Boston MA, Carla Brodley Department of Computer Science, Tufts University, Haligan Hall, 161 College Avenue Medford, With contributions from Douglas McIver, Alex Lotsch, Annemarie Scneider Support from NASA (TEP, MODIS, and Ames)

2 Context Introduction Global observation systems now a reality EOS/GEOSS, remote sensing, in-situ, sensor webs. Large, complex data sets High dimensional, noise, missing data, space, time. Earth system is not in steady state Climate, ecosystems, change at multiple space & time scales, etc Require tools to fuse, explore, understand and discover patterns in disparate data sets!

3 Problem Domains Supervised Classification of Global Land Cover Goal: map global land cover from remote sensing Annual time scales 1-km spatial resolution Repeatable processing; maximum accuracy. Unsupervised Learning of Joint Climateecosystem variability Goal: search, mine, discover patterns that are indicative of coupled climate-ecosystem dynamics Identify anomalies, trends,etc.

4 Supervised Classification of Global Land Cover MODIS Global Land Cover Database: Internally Consistent Maps IGBP Secondary label Classification confidence Cereal vs Brdlf crops UMD Decid vs Evergrn LAI/FPAR PFT BGC IGBP: International Geosphere-Biosphere Programme;UMD: University of Maryland LAI/FPAR: Leaf Area Index/Fraction Absorbed Photsynthetically Active Radiation PFT: Plant Functional Types; BGC: Biome BGC

5 MODIS Moderate Resolution Imaging Spectroradiometer Onboard EOS Terra (10:30 AM descending); and EOS- Aqua (1:30 PM ascending) local solar equatorial crossing Sun synchronous, near polar orbit; km 36 spectral bands, VNIR, SWIR, TIR ( μm) Spatial resolutions at 250-, 500-, and 1000-m (nadir) depending on waveband Scan angle: +/-55 o ; 2330 km swath 2-day global repeat, 1-day or less poleward of 30 o Onboard calibration; Band-to-band registration, etc.

6 Classification Algorithm Decision Tree C4.5 + Boosting Supervised Approach Required to provide robust, repeatable results Relies heavily on input training database Mature algs for missing data

7 Training Site Global Sampling Live Database: currently ~2300 sites, ~48,000 1-km instances

8 The Land Cover Input Database Temporal and spectral features from MODIS 32-day composites View-angle corrected surface reflectance 7 land bands Enhanced Vegetation Index (EVI) Annual Metrics Min, max, mean for each band, plus EVI 108 features total! Extract Exemplars From STEP Database Estimate Classification Apply Classification to Global Data Fuse Results With Ancillary Data Maps

9 Conditional Probabilities Boosting equivalent to additive logistic regression Friedman et al, Ann. Statist., 28(2): , 2000 Allows estimates of class membership probabilities to be estimated from boosted DT s Weighted vote for class y from boosting: F y (x) = log (1 / Β t ) t=1:t If prediction from DT is y, 0 otherwise Class membership probability P( y = j x) = exp(f j (x)) K k=1 exp(f k (x)) McIver and Friedl, 2001 IEEE TGARS; McIver and Friedl, 2002 RSE; Friedl et al., 2002 RSE

10 Fusing Results w/ancillary Information Issue: Good cross-validated accuracies, but maps were poor in many areas Particular problems wrt specific classes: agriculture, urban (wetlands) Solution Use spatial priors based on available coarse resolution maps/data sets Fuse w/boosted DT results via Bayes Rule Class conditional probs from boosted DTs e.g., for urban areas; agriculture Bayes Rule Posterior probabilities Spatially explicit prior probabilities

11 Sample Results for Urban Class (Priors Estimated Using DMSP and Gridded Population Data) MODIS Data Only After Bayes Rule Scneider et al., PERS 2003

12 Unsupervised Analysis of Joint Climate-Ecosystem Variability A. Independent Component Analysis (ICA) Analysis of Northern Hemispheric Sea Surface Temperatures (SST) Anomalies Feature Extraction from NDVI time series B. Principal/Canonical Correlation Analysis (PCA/CCA) Joint Variability of Global Vegetation and Precipitation Analysis of NH drought and SST patterns

13 Independent Component Analysis of Time Series NDVI Independent signals are convoluted and recorded by a sensor (e.g. microphones, satellite instrument) ICA separates the signal mixtures into the original source signals Independent, not just uncorrelated Blind Source Separation no a priori knowledge about the sources

14 FASIR-NDVI Mean Monthly FASIR NDVI Fourier Adjusted Solar zenith angle corrected Interpolated Reconstructed Normalized Difference Vegetation Index NOAA(7,9,11,14)-AVHRR Monthly x1 spatial resolution Significant Linear Trends in NDVI Los et al. (1994), Tucker et al. (2001)

15 Aerosols IC Residual Aerosol signal in tropics Covariation with Stratospheric Aerosol and Gas Experiment (SAGE) II data Not revealed via PCA El Chichon Mt. Pinatubo Lotsch et al., IEEE TGARS, 2003

16 Orbital Drift IC Discontinuities coincide with AVHRR sensor changes Reflect changes in sensor view geometry & orbital drift Limited to southern latitudes Lotsch et al., IEEE TGARS, 2003

17 Joint Variability in Climate & Vegetation (GIMMS-NDVI vs Standardized Precipitation Index 7/1981-3/2003)

18 Northern Hemisphere Mid-Latitude Browning June-August Standard Anomalies relative to mean Motivated by Hoerling and Kumar 2003, Perfect Ocean for Drought, Science Lotsch et al. (2005) Geoph. Res. Letters

19 NDVI and SPI Anomalies May-September North America W CSW Asia W NDVI SPI06

20 Canonical Correlation Example Eurasia (CF1) Correlation Map NDVI Correlation Map SPI06

21 Ocean-Drought Teleconnections II. Eurasia & Australasia Correlation with SST (MAM) SPI pattern dry wet Pacific Warm Pool PC1 of SPI06 warm cold

22 Conclusion Unprecedented reduction of plant photosynthetic activity is linked to synchronous patterns of sea surface temperature fluctuations and geographically extensive patterns of drought in the Northern Hemisphere midlatitudes during ΔSST Pacific + Atlantic + Indo-Pacific NH Precip Regimes Red. Plant Photosynthesis

23 Discussion: Technical Supervised Learning It s not about the learning algorithm.. Data and biases associated with training data are what count Unbalanced training data Feature selection Active Sampling or identifying redundant training data How to stabilize classification results across years Unsupervised Linear vs non-linear methods; Gaussian vs non-gaussian Danger of fishing expeditions Analyses need to be hypothesis driven Toolkit feels less mature, esp for very large data sets. Clustering, PCA, CCA, etc. (may reflect my ignorance) Dimensionality, feature selection key challenges.

24 Discussion: General Data Mining in Earth Sciences Hard Looking for causal relations, not just patterns Need teams to prevent natural scientists from doing naïve analysis and computational scientists from doing naïve science Chicken and the egg problem at last workshop. What is data mining? What is data mining for Earth Sciences? Informatics, not data mining? How can NASA get behind this without a clear definition? But, NASA should be supporting this they have the interests in both measurements and science Need to foster community Publishing Where to publish this stuff? Is it technical or is it science? Where to present? What meetings? 24