Display Management Processing Serving

Transcription

1 MAGIC 2014 Symposium April 27 May 1, Kansas City, Missouri Technical Workshop Display Management Processing Serving Working with Imagery and LiDAR in ArcGIS J.D. Overton Copyright 2014 Esri 1

2 ArcGIS is a Comprehensive Imagery System Integrating All Types, Sources, and Sensor Models... Imagery is a Fundamental Part of the Systems Copyright 2014 Esri 2

3 ArcGIS is a Comprehensive Imagery System Comes with Imagery Provides Advanced Imagery Tools Manages Massive Image Collections Copyright 2014 Esri 3

4 ArcGIS Is a Strong Imagery Platform Introduces Intelligent Imagery on the Web Search Imagery on the Web Web Analysis Image Table Web Map Pop-Ups New Sensors SPOT 6 Landsat 8 Pleiades DMCii Chinese (3) Premium Services DigitalGlobe AccuWeather RapidEye Copyright 2014 Esri 4

5 ArcGIS The Platform for Fully Integrated GIS and Imagery Integrating Imagery is core to GIS Management Visualization Dissemination Analysis Maximizes the value of imagery Copyright 2014 Esri 5

6 ArcGIS For Image Data Management Storage, Catalog, Metadata & Process Workstation User What tools do I have? How can I easily work with it? Organizations with collections of processed imagery How do I serve all our ortho images? How can I serve my elevation data to multiple users? Enterprises collecting new imagery How do I process and serve imagery that we acquire? Catalog all available imagery Make it quickly accessible in the required form Copyright 2014 Esri 6

7 Direct Access to Imagery Workstation User Knowledge Workers Executive Access Public Engagement Work Anywhere Enterprise Integration ArcGIS Imagery Display Management Processing Serving Professional GIS Server Online Content and Services Copyright 2014 Esri 7

8 Direct Image Access Workstation Accesses Pixels and Performs Required Processing ArcGIS Desktop provides direct access Raster Datasets Nearly all image formats TIF, IMG, NITF, JPEG2000,... ArcGIS provides direct access to nearly all imagery Copyright 2014 Esri 8

9 Specify the Source Type of Raster Data Source Type = PROCESSED This informs ArcGIS not to apply automatic stretch. GP Tool Raster Dataset Properties Copyright 2014 Esri 9

10 ArcMap Settings related Imagery and Raster Data Setting ArcMap options Copyright 2014 Esri 10

11 Turn OFF Raster Types you don t have Displaying specific raster formats Copyright 2014 Esri 11

12 Geoprocessing Settings (ArcGIS Desktop 10) 10 ArcGIS 10.1 SP1 or 10.2.x Desktop Backgroud Geoprocessing (64-bit) Using geoprocessing options to control tool execution Copyright 2014 Esri 12

13 Why build Pyramids? Images with >5,000 columns Pyramids are down sampled versions of the image. Without pyramids, the entire dataset must be read from disk and re-sampled to a smaller size. The pyramid file are named: imagename.rrd (8/9.x) imagename.ovr (10.x) Size +/- 8% of the original uncompressed file size. Size +/- 2% of the original if compression is applied x Raster pyramids Copyright 2014 Esri 13

14 Build Pyramids (*.OVR) Build for images that have > 5000 cols Significantly improve performance on large TIFs Draw speed and faster Statistics Calculations Enables Mosaic methods to work at smaller scales Use JPEG Compression when building pyramids Y C b C r Source: YCbCr - Wikipedia, the free encyclopedia Copyright 2014 Esri 14

15 Image Formats (Pixel Source) Tiled TIFF with JPEG compression This is the recommended storage format for source imagery because it performs the fastest in ArcGIS. All percentage estimates below will be compared to this. TIFF raw On average, large* files will be 20 percent slower. MrSID Approximately 100 percent slower. JPEG 2000 Approximately 200 percent slower. IMG with RRD Approximately 50 percent slower. JPEG On average, large* files will be 400 percent slower because there is no internal tiling. *A large file is approximately x pixels uncompressed. This is approximately a 100 MB panchromatic (1-band) image or a 300 MB color (3-band) image. Copyright 2014 Esri 15

16 Apply JPEG Compression to TIFs (continued) Original 239 MB Compressed 37 MB Copyright 2014 Esri 16

17 Calculate Statistics Not required but highly recommended Auto generated when needed Copyright 2014 Esri 17

18 Statistics Skip factor Process less data Spatial Analyst Tools can ignore skip factors Display only applications can divide columns/rows by 1000 Use Area of Interest to Avoid clouds Avoid lakes Copyright 2014 Esri 18

19 Drag multiple objects from Catalog window Show different view Select multiple objects Drag and drop on map Batch operations (right click) Copy/Delete Build Pyramids Calculate Statistics Working with the Catalog Window Copyright 2014 Esri 19

20 Imagery Community Imagery Community Copyright 2014 Esri 20

21 Image Management Workflows Copyright 2014 Esri 21

22 Image Management Workflows Image Management Guidebook Copyright 2014 Esri 22

23 ArcGIS Comes With Imagery Esri Curated From Imagery Providers From ArcGIS Users Oceans Terrain Topographic Imagery Canvas Copyright 2014 Esri 23

24 Basemap (Add Data) for direct access to ArcGIS Online Copyright 2014 Esri 24

25 Customize Basemaps with ArcGIS Online Organization Customize basemap layers Copyright 2014 Esri 25

26 Basemap Layer (Create your own) Mr SID Cached Map Services Vector data Basemap Layer can contain both Vector and Image/Raster data Working with basemap layers Copyright 2014 Esri 26

27 Lock ArcMap zooming to AGO/Bing/Google Map Scales Good way to limit local cache with Basemap Layers 2 1 Working with map scales Copyright 2014 Esri 27

28 Export Map Cache Looks like a Raster to ArcGIS Desktop and ArcGIS Mobile Behaves like Map Cache Exporting and importing map caches Export Map Server Cache Use with Disconnected editing and sharing map cache tiles Copyright 2014 Esri 28

29 Create Tile Package Provision devices and applications that run offline (applications that can run disconnected from ArcGIS Server or ArcGIS Online), such as applications built with ArcGIS Runtime SDK for Java, ios, Windows Mobile, and WPF Generate Tile Package Copyright 2014 Esri 29

30 Image Analysis Window Options Layer List Display Processing What is the Image Analysis window? Copyright 2014 Esri 30

31 Clip (Inside) 1 Graphics 2 Selections Using the Clip button on the Image Analysis window Copyright 2014 Esri 31

32 Mask (Outside) 1 Graphics 2 Selections Using the Mask button on the Image Analysis window Copyright 2014 Esri 32

33 Elevation (DEMs or DTMs) 6 ArcInfo GRIDs Notice the seams or lines (Elevation Min/Max differences) Copyright 2014 Esri 33

34 On-the-fly Mosaic (created in memory) 1. Select rasters 2. Select Mosaic type 3. Click Mosaic command Using the Mosaic button on the Image Analysis window Copyright 2014 Esri 34

35 Apply Functions (on-the-fly processing) 1. Dynamic 2. Easy to Add/Remove Rasters with functions Copyright 2014 Esri 35

36 Shaded Relief (created in memory) 1. Select Mosaic (or a DEM) 2. Select Color Ramp 3. Click Mosaic command Shaded Relief function Copyright 2014 Esri 36

37 Hillshade function Slope function Aspect function Raster Functions Copyright 2014 Esri 37

38 Accessing Imagery As Direct, Static or Dynamic ArcGIS Desktop: Full function GIS & Imagery Direct - Raster Datasets, Mosaic Dataset Static Internet Layers - ArcGIS Online, Cached Imagery,.. Image Services ArcGIS Online: Imagery on any device Imagery and Elevation layers as a subscriber to ArcGIS Online Static Internet Layers Image Services - LYR Web APIs: Web accessibility SOAP/REST JavaScript, Silverlight, Flex Static Internet Layers Image Services Open Interoperability/Standards WMS, WCS, KML Copyright 2014 Esri 38

39 Organizations with collections of processed imagery Enterprises collecting new imagery Storing, Organizing and Structuring Imagery Display Management Processing Serving Copyright 2014 Esri 39

40 Forms of Imagery Ortho Images Imagery processed and mosaicked into tiles Image Strips Orthoimages along corridors Processed Rasters Elevation models Analysis results Thematic maps Scenes From satellites Georeferenced Used for analysis Sensor Images Raw / minimal pre-processing Non rectified Simple Mgmt. Primarily as Background Data is procured from a source Advanced Mgmt. Primarily for Analysis or further processing Data is acquired Copyright 2014 Esri 40

41 Use Case 1 Tiled Orthos Images are stored as many regular tiles Tiles are edge-matched Tiles most commonly have 3 bands Dynamic range most commonly is 8 bit Color correction has already been applied No regions of NoData are present (except possibly a few tiles along project boundary) Optimum format for Orthos is TIFF* with and JPEG YCBCR compression * Tiled in blocks of 128 x 128 or 256 x 256 Copyright 2014 Esri 41

42 Use Case 2 Multi-image mosaics in large files Images are stored as many irregular tiles. Valid imagery area will typically have irregular boundaries. Large areas of NoData. Data is typically compressed (MrSID files). NoData is typically zero, but after compression is near 0. Color correction has already been applied. USDA National Agricultural Imagery Program [NAIP] program County Compressed Mosaics (CCMs) Copyright 2014 Esri 42

43 Use Case 3 Individual orthorectified image frames Least commonly encountered Individually orthorectified images No cropping (NoData around each image) Not color balanced NOTE: If cropped and color balanced, becomes Use Case 1 Copyright 2014 Esri 43

44 Where to store your image/raster data? Database Pro: Highly structured In-house experts Con: Load time 1+ step from source image Update process Files Pro: Uses native images Reduces duplication Dynamic mosaic Maintain source image metadata Con: Loosely structured File system The choice is yours, we support both Copyright 2014 Esri 44

45 Enhancing Processing Methodologies Conventional Image Processing Workflows are Linear Multiple processes create intermediate results Products created as static mosaic Mosaic Datasets Enable Transactional Workflows Processes applied on demand Products created on demand Copyright 2014 Esri 45

46 On-the-fly Processing & Dynamic Mosaicking Resolves Traditional Image Management and Processing Issues Processing Time Reduces processing Overlapping Imagery Maintain information Disparate Datasets Large NoData areas Image Quality Reduces re-sampling t1 t2 t3 t4 Storage Reduces storage by removing redundancy Multi-resolution Data No need to sample up or down Maintenance Add imagery as required Maintain Metadata Retain valuable information Copyright 2014 Esri 46

47 What is a Mosaic Dataset? Catalog/Library of Imagery and rasters Associated metadata Processing functions to be applied Stored in Geodatabase Authored using ArcGIS Desktop Standard (ArcEditor) Advanced (ArcInfo) Enables Dynamic Mosaicking On-the-fly processing What is a mosaic dataset? Copyright 2014 Esri 47

48 Mosaic Dataset Workflow Create File Geodatabase Create Mosaic Dataset Add Rasters The workflow and tools to create mosaic datasets Copyright 2014 Esri 48

49 Mosaic Dataset Layer *Boundary extent of all the raster datasets *Footprint boundary and attributes for each raster in mosaic dataset *Image layer controls the rendering of the mosaicked image similarly to a raster layer. *Editable in ArcMap Types of layers when using a mosaic dataset layer in ArcMap Copyright 2014 Esri 49

50 Web Services can supplement your Mosaic Dataset World Imagery from ArcGIS Online Add to ArcMap and Save As Layer File World Imagery Local 6 Orthos Creating a mosaic dataset containing raster data from multiple dates Scroll down to section Add a Web map as the overview" Copyright 2014 Esri 50

51 Mosaic Dataset Overviews Consider building overviews over raster pyramids when using: Preprocessed tiled imagery Non-overlapping imagery, no use of mosaic methods Imagery that will be processed on-the-fly, but the parameters and mosaicking method will not be changed Default service overview factor is 3 Web Maps (i.e. AGO/Bing/Google) use scale factor of 2 Web Map Scales Factor of 2 Mosaic dataset overviews Copyright 2014 Esri 51

52 Use Transmission Compression None Results in no compression being applied to the imagery, providing the highest quality, but results in the maximum data transfer across the network. JPEG An efficient compression method that can often compress imagery by about three to eight times with little degradation in the image quality. When using this method, a compression quality can also be set between 1 and 100 percent. A value of 80 tends to retain image quality while providing approximately 8x compression. LZW An efficient, lossless compression method that is often used for imagery with similar pixels such as scanned maps or classification results. This is recommended for discrete data. Mosaic dataset properties Copyright 2014 Esri 52

53 Mosaic Dataset Managing Raster catalogs of DEM/DTM/LiDAR in Geodatabase Seamless mosaic of file-based Elevation Data Apply functions Copyright 2014 Esri 53

54 Mosaic Dataset Color Orthos Multiple resolutions (0.5 ft Urban, 1 ft Semi-Urban, 2 ft Rural) Publish Mosaic Datasets of multiple years of Orthos Copyright 2014 Esri 54

55 Managing collections of Imagery from Multiple Dates Create Mosaic Dataset for each year Create the master mosaic dataset Add Rasters selecting Mosaic Datasets Add Year field (Text) to Footprint (mosaic dataset) Calculate Values to populate value for each year Examine the temporal mosaic dataset Orthos2008 Orthos2010 Master Creating a mosaic dataset containing raster data from multiple dates Copyright 2014 Esri 55

56 Optimizing Performance Storage and Data Structure Image Formats (Pixel Source) Apply JPEG Compression to TIFs Dramatic size reduction vs. Uncompressed TIFs Build Pyramids (*.OVR) Pyramids will reduce the number Overviews Mosaic Dataset Overviews Use Transmission Compression Copyright 2014 Esri 56

57 Storage and Data Structure Organize Imagery into folders containing no more than 1000 images. Make sure the storage disks are not filled beyond 95 percent. Set permissions on images as read only. Set permissions on folders containing images to read/write. Consider a separate File Geodatabase for each Mosaic Dataset. See Image Management Guide Book section on Data Structure Copyright 2014 Esri 57

58 Setting Default Transmission Compression for Mosaic Dataset Mosaic dataset properties Copyright 2014 Esri 58

59 (Client-side) Transmission Compression in ArcGIS Desktop Works for both Mosaic Datasets (authoring) and Image Services (consuming) Copyright 2014 Esri 59

60 Dissemination Providing Image Accessibility Imagery Display Management Processing Serving Sharing: Direct, Static & Dynamic Access: Multiple Clients Copyright 2014 Esri 60

61 Dynamic Image Access Image Services - Server based access and processing ArcGIS Server Image Services Directly from a raster dataset User defined: Projection Sampling Changeable compression On-the-fly processing Multiple products from a single source Return image or pixel values Export of area of interest ArcGIS Server Image Extension Extends Image Services to serve mosaic datasets Serve large collections of imagery As Image with dynamic mosaicking As Catalog with selection and query Download source pixels with clipping 1 raster to many clients 1:M Mosaic Dataset to many clients M:M Copyright 2014 Esri 61

62 Static Image Access Web Optimized Imagery Map Cache for Imagery Provide static background image Tiles for web caching Most scalable web delivery Example: ArcGIS Online Imagery Created and served using ArcGIS Server Pre-Processing / On Demand Recommended Workflow Create Mosaic Dataset Generate Cache Maintain Mosaic Dataset Update Cache ArcGIS Server Tile Analysis Modeliing Update ArcGIS Server Cache Copyright 2014 Esri 62

63 MAGIC 2014 Symposium April 27 May 1, Kansas City, Missouri Technical Workshop Working with LiDAR and LAS Datasets Copyright 2014 Esri 63

64 LAS Dataset - Overview LAS Dataset File based data structure Stores references to LAS files on disk Treats a collection of LAS files as one logical dataset Optionally reference break-line data Copyright 2014 Esri 64

65 LAS Dataset and Index creation. Folder file1.las file2.las Add LAS Dataset Compute Stats Folder file1.las file1.lasx file2.las. file2.lasx LAS auxiliary file (.lasx) Improves performance of spatial queries Efficient use of larger files and files over a network Benefits both visualization and data processing Index auto-updates after editing operations Copyright 2014 Esri 65

66 Usage Recommendations Classified LAS files organized as tiles 200 MB file size limit All participating LAS files have the same Spatial Reference (Projected CS) Data location (preferably on a local disk) Lidar - Working with LAS Datasets Copyright 2014 Esri 66

67 Licensing Standard, 3D, or Spatial Analyst License - Create LAS dataset - Modify LAS dataset schema - Include in Mosaic dataset Publishing requires Image Server extension - View in 2D - Rasterize 3D Analyst License - Edit class codes - View in 3D - TIN based surface analysis - Rearrangement Lidar - Working with LAS Datasets Copyright 2014 Esri 67

68 LAS Dataset - Visualization In 2D (ArcMap) or 3D (ArcScene) As points Symbolize using LAS attributes (Class Codes, RGB etc.) Copyright 2014 Esri 68

69 LAS Dataset - Visualization As a surface Symbolize & filter (Bare-earth, First-return, etc.) Control the number of points shown Copyright 2014 Esri 69

70 LAS Dataset - Editing Manual editing (data errors, mis-classifications) Using Geo-processing tools Copyright 2014 Esri 70

71 LAS Dataset - Surface definition Support for break-lines (Surface constraints) Copyright 2014 Esri 71

72 LAS Dataset GP Tool Summary 3D Analyst Conversion Data Management Conversion To Raster LAS Dataset Data Management Layers and Table Views Tools enhanced to support LAS Dataset Copyright 2014 Esri 72

73 LAS Dataset - Analysis Derive surfaces - LAS Dataset As Raster - LAS Dataset As TIN Direct analysis - Interpolate Shape - Add Surface Information - Line Of Sight - Locate Outliers Rasterize on point metrics - LAS Point Statistics As Raster Copyright 2014 Esri 73

74 LAS Dataset - Spatial Reference Support Assign spatial reference (if missing) Override using a.prj file (if already present) Project Clip using an area-of-interest File extent Study Area Clipped file extent Copyright 2014 Esri 74

75 LiDAR data - Sharing Mosaic Dataset Manage LiDAR data like imagery Add / remove LAS files Add / remove LAS datasets Share the data as Image services Easy for others to see available LiDAR data Easy to download for project work Copyright 2014 Esri 75

76 LAS Optimizer Optimize and Compress LAS files Esri compressed LAS (*.zlas) Lossless Transparently treated as LAS but much better for use and exploitation No ArcGIS License required for use Library for 3rd party use LAS Optimizer v 1.1 Copyright 2014 Esri 76