Analyzing Multidimensional Scientific Data in ArcGIS

Transcription

1 Federal GIS Conference February 9 10, 2015 Washington, DC Analyzing Multidimensional Scientific Data in ArcGIS John Fry Solution Engineer, Esri jfry@esri.com Esri FedGIS 2015 Technical Workshop

2 UNCLASSIFIED//FOR OFFICIAL USE ONLY Fishing Performance Surface ESRI Federal User s Conference 2/10/2015 Christopher Henton Christopher.henton.ctr@navy.mil Naval Oceanography 2

3 UNCLASSIFIED//FOR OFFICIAL USE ONLY Background Naval Oceanographic Office NOAA for the NAVY Survey ships Oceanographic models Requirement: Various customers have requested a performance surface model to predict ideal locations for fishing based on biological and environmental data. GOAL: To identify potential fishing zone (PFZ) forecasting. Best Areas where fish could be! Naval Oceanography 4

4 UNCLASSIFIED//FOR OFFICIAL USE ONLY ENVIRONMENTAL ENVELOPE Species Environmental Envelope (HSPEN): In Aquamaps, this represents a response curve describing the habitat usage of a species or its preferences with respect to specific environmental parameters ; Environmental parameters include: Depth (m) Surface Temperature ( C) Surface Salinity (PSU) Primary Production (mgc m-² day -1) Sea Ice Concentration (% cover) Distance to Land (km); The environmental envelope is presented as a range of values for each parameter: Minimum, Preferred minimum (Min P ), Preferred maximum (Max P ), Maximum). 10 th Percentile 90 th Percentile Naval Oceanography Source: K. Reyes, AquaMaps: Algorithm and Data Sources for Aquatic Organisms 6

5 UNCLASSIFIED//FOR OFFICIAL USE ONLY Normalization In Aquamaps, this represents a response curve describing the habitat usage of a species or its preferences with respect to specific environmental parameters ; Environmental parameters include: Depth (m) Surface Temperature ( C) Surface Salinity (PSU) Primary Production/Chlorophyll (mgc m-² day -1) Sea Ice Concentration (% cover) Distance to Land (km) 10 th Percentile 90 th Percentile a + (x - A) x (b - a) / (B - A) The environmental envelope is presented as a range of values for each parameter: Minimum, Preferred minimum (MinP), Preferred maximum (MaxP), Maximum). Species Envelope (HSPEN): Min Pref Min (10th) Pref Max (90th) Max Common Name Depth (m) Abe's flyingfish Temperature ( C) Abe's flyingfish Salinity (psu) Abe's flyingfish Primary Production Abe's flyingfish Naval Oceanography Source: K. Reyes, AquaMaps: Algorithm and Data Sources for Aquatic Organisms 8

6 UNCLASSIFIED//FOR OFFICIAL USE ONLY Normalization Examples Temperature ( C) Abe's flyingfish Depth (m) Abe's flyingfish 1 0 A B C A B C Naval Oceanography 9

7 UNCLASSIFIED//FOR OFFICIAL USE ONLY Fusion Probabilities of Occurrence The relative probabilities of occurrence are presented as probability values ranging between A probability of occurrence is first calculated for the each of the individual environmental predictors. Pc = Pbathymetry c Ptemperature c Psalinity c Pprimary production c This multiplicative approach allows each environmental predictor to act as a knock-out criterion. = Kesner-Reyes, K., K. Kaschner, S. Kullander, C. Garilao, J. Barile, and R. Froese AquaMaps: algorithm and data sources for aquatic organisms. In: Froese, R. and D. Pauly.Editors FishBase. World Wide Web electronic publication. version(04/2012). Naval Oceanography 10

8 10 Depths UNCLASSIFIED//FOR OFFICIAL USE ONLY Multidimensional Processing Time E p S s = D i x Chl S x D T D x S D + F D S s F s F s = S s F S (max) 0 X Weight P o = F s P O (max) X Weight D 20 X Weight S S = Species Sum D i = Distance from Shore Chl S = Chlorophyll at surface T D = Temperature at depth S D = Salinity at depth F D = Frontal boundaries at depth F S = Fish Family Sum P O = Probability of Occurrence Naval Oceanography 12

9 UNCLASSIFIED//FOR OFFICIAL USE ONLY Experiment Environmental Factors Salinity (PSU): by depth / forecasted time Temperature (ºC): by depth / forecasted time Chlorophyll (µg/l): Remote Sensing (monthly) Distance from Shore Depth Frontal Boundaries (MGET/Modeled Not included yet) Fish Habitat Preferences (AquaMaps) 10 Fish Families 78 Species Maritime Boundaries, EEZ, Fish Closure Areas, Marine Protected Areas (layers) Temp Salinity Chlorophyll Fish Preference Raw Data Normalize / Fuse HTML Shapefile KML/Image Output Naval Oceanography

10 UNCLASSIFIED//FOR OFFICIAL USE ONLY Data Management 78 Species 96hr forecast 200 Depths Water Temp & Salinity Results 5,331,200 Raster Layers!!!!!!!!! ~1TB ~5 Days Extraction Environmental Normalization Depth Normalization F s Sum [(T m D r ) + [(T m D r ) (N p ) (F s )] + [(T m D r ) (F s )] + (T m D r ) ] E p = R l [(34 200) + [(34 200) (4 ) (78)] + [(34 200) (78)] + (34 200) ] 2 = 5,331,200 Raster Layers T m = Time Range D r = Depth Range N p = Normalized Preference F s = Fish Species E p = Environmental Parameter R l = Raster Layers Naval Oceanography 15

11 UNCLASSIFIED//FOR OFFICIAL USE ONLY Summary Multidimensional problems are complex and require new ways of processing and thinking. Data Management Analytics Visualization Broader temporal scales Climatology Predictive (Months/Years/Decades) Naval Oceanography 16

12 UNCLASSIFIED//FOR OFFICIAL USE ONLY REFERENCES Jonathan Sharples, Jim R. Ellis, Glenn Nolan, Beth E. Scott, Fishing and the oceanography of a stratified shelf sea, Progress in Oceanography, Volume 117, October 2013, Pages , ISSN , ( Kaschner, K., J. Rius-Barile, K. Kesner-Reyes, C.Garilao, S.O. Kullander, T. Rees and R. Froese (2013). AquaMaps: Predicted range maps for aquatic species. World wide web electronic publication, Version 08/2013. K. Reyes, AquaMaps: Algorithm and Data Sources for Aquatic Organisms Pauly, D. V. Christensen, S. Guénette T.J. Pitcher, U.R. Sumaila, C.J. Walters, R. Watson and D. Zeller Towards Sustainability in World Fisheries, Nature 418: ( Jason J. Roberts, Benjamin D. Best, Daniel C. Dunn, Eric A. Treml, and Patrick N. Halpin Marine Geospatial Ecology Tools: An integrated framework for ecological geoprocessing with ArcGIS, Python, R, MATLAB, and C++. Environ. Model. Softw. 25, 10 (October 2010), DOI= /j.envsoft Additional Reading: Eli Agbayani. TCOM Meeting, Basel, Switzerland (9/2008) ( Josephine "Skit" Rius-Barile. AquaMaps - behind the scenes ( Kathleen Kesner-Reyes. AquaMaps: An Overview ( Jones, M.C., S.R. Dyeb, J.K. Pinnegar and W.W.L. Cheung (2012). Modelling commercial fish distributions: Prediction and assessment using different approaches. Ecological Modelling 225(2012): ( Daniel Pauly Publications: ( Naval Oceanography 17

13 Outline ArcGIS and Scientific Data Multidimensional Tools Ingest and aggregation Visualization and Analysis Services, Ready-to-Use Maps, Web Applications

14 Communicating Science to Decision Makers Winds in the Area of Interest will be sustained at 20kts with gusts to 35kts. ArcGIS Waves in the Area of Interest will between 2.5 m and 3 m. What does that mean??? You can conduct X operation at these locations, but not at these locations. Will my mission be a success? What platform can I use?

15 Communicating Science to Decision Makers Reduction in upwelling and mixing not beneficial in these locations. Stratification occurring at these locations ArcGIS What does that mean??? Will it be too costly to fix? What techniques should I use? Pollutant runoff in surrounding regions needs to be addressed Fish yields will be low in these areas

16 Demo Visualize Scientific Data Clearly Esri 2015 FedGIS Analysis of Multidimensional Data in ArcGIS

17 Scientific Data Oceanographic data Sea temperature Salinity Ocean current Meteorological data Temperature Humidity Wind Land Soil moisture NDVI Land cover

18 1. Latitude (y) 2. Longitude (x) 2D

19 1. Latitude (y) 2. Longitude (x) 3. Height (z) 3D

20 1. Latitude (y) 2. Longitude (x) 3. Time (t) 3D

21 1. Latitude (y) 2. Longitude (x) 3. Depth (d) 4. Time (t) 4D

22 Scientific Data in ArcGIS direct ingest share data management ArcGIS analysis visualization

23 Scientific formats in ArcGIS 10.3 GRIB (General Regularly-distributed Information in Binary form) - A gridded format used operationally worldwide by most meteorological centers. netcdf (Network Common Data Form) - A self-describing, machine-independent data format that support the creation, access, and sharing of array-oriented scientific data. HDF (Hierarchical Data Format) - A scientific format that supports a proliferation of different data models, including multidimensional arrays, raster images, and tables. Variable / Dimension Temperature / Time Salinity / Depth Pressure / Altitude

24 CF Convention Climate and Forecast (CF) Convention - Initially developed for: - Climate and forecast data - Atmosphere, surface and ocean model-generated data - Also for observational datasets CF is now the most widely used conventions for geospatial netcdf data. Current version 1.6 According to CF conventions, variables representing time must always explicitly include the units attribute. You can use Compliance checker utility to check a netcdf file. CF-Convention Compliance Checker for netcdf Format

25 Using Scientific Data in ArcGIS Behaves the same as any layer or table Display o Same display tools for raster and feature layers will work on multidimensional netcdf raster and netcdf feature layers. Graphing o Driven by the table just like any other chart. Animation o Multi-dimensional data can be animated through time dimension Analysis Tools o Will work just like any other raster layer, feature layer, or table. (e.g. create buffers around netcdf points, reproject rasters, query tables, etc.)

26 Ingesting Scientific data in ArcGIS Directly reads netcdf file using o o o Make NetCDF Raster Layer Make NetCDF Feature Layer Make NetCDF Table View Directly reads HDF and GRIB data as raster Multidimension Supplemental Tools Download from ArcGIS Online

27 Changing Time Slice Y Time Time = X

28 Spatial and Temporal Analysis Several hundreds analytical tools available for raster, features, and table Temporal Modeling o Looping and iteration in ModelBuilder and Python

29 Demo Working with Multidimensional Data Direct Ingest Analysis Geoprocessing Models Predictive Analysis Toolset Esri 2015 FedGIS Analysis of Multidimensional Data in ArcGIS

30 N-dimensional capability in ArcGIS 10.3 Visualization Observations Modeling Processing on-the-fly Raster Functions Dissemination Mosaic Data Set Server Native METOC Data Formats Exploitation netcdf GRIB HDF ArcGIS Desktop

31 Raster Mosaic Dataset for N-dimensional Data (ArcGIS 10.3) Resolves Traditional Raster Data Management Issues Processing Time Reduced exponentially Overlapping Imagery Pick Best raster Storage Reduces storage by removing redundancy Multi-resolution Data No need to sample up or down Disparate Datasets Handle large NoData areas Maintenance Add Rasters as required Data Quality Reduces resampling Maintain Metadata Appended directly upon registration

32 Scientific data support in Mosaic Dataset Supports netcdf, HDF and GRIB o Spatial Aggregation o Temporal Aggregation o On-the-fly analysis Accessible as Map Service Accessible as Image Service Supports direct ingest Eliminates data conversion Eliminates data processing Improves workflow performance Integrates with service oriented architecture

33 Multidimensional Mosaic Datasets Aggregate (mosaic) spatial, time, and vertical dimensions Raster Types for netcdf, HDF & GRIB Define variables when adding Rasters Each Row is a 2D Raster with variables and dimension values Define on-the-fly processing Serve as Multidimensional o Image Service o Map Service o WMS

34 Mosaic Dataset - Visualization Visualize variables at any time and vertical dimension Visualize flow direction and magnitude variables Visualize temporal change of a variable

35 Visualization of Raster as Vectors New Vector Field renderer for raster o Supports U-V and Magnitude-direction o Dynamic thinning o On-the-fly vector calculation Eliminates raster to feature conversion Eliminates data processing Improves workflow performance

36 Modeling with Raster Function Templates (RFT) o o o o

37 Fish suitability New Dimensions in ArcMap 10.3 Blog Post Shown at 2014 UC Plenary

38 Demo Using the Mosaic Dataset for Analysis Creating Mosaic Datasets Adding functions to mosaic datasets Esri 2015 FedGIS Analysis of Multidimensional Data in ArcGIS

39 Create Space-Time Cube & Emerging Hot Spot Analysis

40 Demo Using ArcGIS Pro with Multidimensional data -Time Aggregation of Multi-dimensional data -Update a Mosaic Dataset Esri 2015 FedGIS Analysis of Multidimensional Data in ArcGIS

41 Sharing Multidimensional Datasets 2 Ways to Serve Data Directly from individual raster dataset From mosaic dataset (requires image extension) Publish to Server Mosaic Dataset Publish to Server Image Extension

42 Publishing a WMS on ArcGIS Server Enable WMS capabilities on Service Editor or Manager

43 Ready-To-Use Analysis Services Esri hosted analysis on Esri hosted data - Simplify job of GIS Professionals - Can be used in models and scripts just like any other tool - Extend spatial analysis to a much broader audience - Available in Desktop or as REST service

44 Ready-to-Use Scientific Data Maps ArcGIS Online GEOSS Water Services: GLDAS Soil Moisture Data o Evapotranspiration o Soil Moisture o Snow Pack

45 Web Application Samples - GitHub - WMS Multi Dimensional Esri Viewer - NetCDF MultiDimensional Template with - ArcPy-NetCDF-Web-Charting-Tools

46 Tell the story of your scientific data Create Story Maps Storymaps.esri.com Connecticut Department of Energy Dead Zones in our Oceans

47 Demo Web Resources Esri 2015 FedGIS Analysis of Multidimensional Data in ArcGIS

48 Things to Consider Embrace the Common Data Model (netcdf, HDF etc.) Use Data and metadata standards (OGC, CF etc) Provide mechanism so that we can access scientific data using a single set of APIs. and can expect data to be CF complainant Make your data spatial (by specifying geographic or a projected coordinate system) Clearly define workflow and requirements Create sample tools / templates where possible ArcGIS Online / Solutions Pages / GitHub

49 Scientific Data Resources ArcGIS Multidimensional Supplemental Tools - Esri Raster Function Templates GitHub - Esri/WMSMultidimensionalEsriViewer - Working with multidimensional data Help: `

50 Federal GIS Conference February 9 10, 2015 Washington, DC Don t forget to complete a session evaluation form! Esri FedGIS 2015 Technical Workshop

51 Federal GIS Conference February 9 10, 2015 Washington, DC Print your customized Certificate of Attendance! Printing stations located on L St. Bridge, next to registration Esri FedGIS 2015 Technical Workshop

52 Federal GIS Conference February 9 10, 2015 Washington, DC GIS Solutions EXPO, Hall D Monday, 12:30pm 6:30pm Tuesday, 10:45 AM 4:00 PM Exhibitors Hands-On Learning Lab Technical & Extended Support Demo Theater Esri Showcase Esri FedGIS 2015 Technical Workshop

53 Federal GIS Conference February 9 10, 2015 Washington, DC Networking Reception: National Museum of American History Tuesday, 6:30 PM 9:30 PM Bus Pickup located on L Street Esri FedGIS 2015 Technical Workshop

54 Federal GIS Conference February 9 10, 2015 Washington, DC Interested in diving deeper into Esri technology? Add a day to your Fed GIS experience and register to attend the Esri DevSummit Washington DC. Stop by the registration counter to sign up. Esri FedGIS 2015 Technical Workshop

55