data has become increasingly important in many industries

Transcription

1

2 ing Spatial Databases to Efficiently Mana Objects at a National Level a Geospatial Forum 2015 Viehmann uct Manager EMEA CLE Corporation st 19, 2015 Copyright 2014 Oracle and/or its affiliates. All rights reserved. Screenshot courtesy of: FUGRO BV

3 tivation data has become increasingly important in many industries sed on small scale in engineering & construction, comms, utilities in the past oving into continuous site surveying or asset management now ublic Services driven by topics such as renewable energy, risk management,... ny business cases have become economically viable ost of 3D data collection has gone down significantly ased on LiDAR or Image Matching ading to country-wide initiatives g. in Poland, the Netherlands (3D Pilot NL, AHN-2), Germany (AdV), UK, Ireland,..

4 ling from 2D to 3D or from regional to nationallevel e possible options (or a combination thereof) se more sophisticated software uy faster computers ove to new architectures

5 first choice using a scalable database ires a spatially enabled database ta integration with other sources nline availability eo-referenced imagery, existing 3D structures, attributes,... st access to arbitrary part of data set or processing or visualization neral benefits of mature DBMS nformation lifecycle management data administration, tuning caleability multi-processor support, clustering,... xecuting data-intense logic where the data resides

6 cle Database with Spatial and Graph Option Points Web Services (OGC) SPARQL End Point Lines Oracle Spatial and Graph Polygons Geocoding, Routing Inferencing Rasters Network Graphs e1 f1 e2 n1 n2 f2 e4 e3 Topologies RDF Semantic Graphs 3D

7 cific Datatype for Point Clouds Logical structures Contains point cloud metadata and footprint Also contains a pointers to one or more block tables pc 1 pc 2 pc 3 pc 4 pc 5 pc 6 Physical structures Point cloud block tables Contain the points Can be very large Could be partitioned Add new tables as necessary pc 1 blocks pc 2 blocks pc 3 blocks pc 4 blocks pc 5 blocks pc 6 blocks

8 sible 3D processing flows in Oracle 12c LiDAR Files LiDAR loader SDO_PC tables or flat table Convert to Geometries Query and Clip Generate TIN Georaster Generate DEM TIN tables Flat files Load point cloud Point Tables Query and Clip Calculate Contour Lines Query and Clip Convert to Geometries

9 ÖBB-InfrastrukturAG, R&D Oracle Spatial and Graph: Railway Network Management ctives ize railway planning, construction and enance efficient routing of railway lines ion and processesmore than 8 billion of objects along railway tracks es LiDARdata to be viewed with existing tructure vector data es comprehensive metadata about y tracks through CSW rs data through open WebServices [This technology ] is indispensable to pr geospatial data with high efficiency at low cost, Dr. Michaela Haberler-Weber ÖBB-Infrastruktur AG, R & D Image courtesy of: IQso

10 second option Oracle ExadataDatabase Machine ase offloadin storage intensive query operations offloaded to ge CPUs GB/sec SQL data throughput age Index data skipping ase optimized compression id Columnar for 10x DB size reduction faster analytics ase optimized PCI Flash rt caching of database data Million Database IOs/sec rt Flashlog speeds transactions Database optimized QoS End-to-end prioritization from application to DB and storage Database optimized availabilit Fastest recovery of failed database server, storage or switch Fastest backup. Incremental offloa Exachk top-to-bottom validation o hardware, software, settings Database optimized messagin SQL optimized InfiniBand protocol high throughput low latency SQL

11 ch escienceresearch project on Massive Point Clouds ct Consortium, led by Peter v. Oosterom, TU Delft Delft: GIS technology TU Delft Library 3TU.Datacentrum TU Delft Shared Service Center ICT LeSC(Netherlands escience Center) racle Corporation (NEDC) ijkswaterstaat gro B.V.

12 ch escienceresearch project on Massive Point Clouds ation of user requirements ased on structured interviews with: Government community: Rijkswaterstraat Commercial entity: Fugro B.V. Scientific community: TU Delft Library asis for conceptual benchmark: Tests for functionality Classified by importance sting single-user environments to avoid load-generator complexities

13 formats and tested datasets al Height Model of the Netherlands (AHN2) overing surface of the entire country -10 pts/m billion pts 0,185 LAZ files, 987 GB in total, 1.64 TB uncompressed, Y, Z) only ture plans AHN3 at even higher resolution Cyclorama-based photogrammetric datasets (50x AHN2, and with RGB)

14 formats and tested datasets hmarks nning 30 queries against full dataset estigating different storage methods Flat tables Block storage Compressed and uncompressed ing engineered system racle Exadata Database Machine

15 chmark results (I) rmance and best practices ta loading se external table mechanism in conjunction with create table as select to tream LAZ files directly into database oading all 640bn points from LAZ files in 4:39h on Exadata X4-2 (Full Rack) brid Columnar Compression compress for query high setting delievers best compromise between storage eduction and query performance equires 2.24TB of storage in database, compared to 0.97TB in LAZ files or 12TB uncompressed) LAS

16 chmark results (II) rmance and best practices ta management se partitioning for better performance and improved parallelism 0655 partitions with between 10,000,000 and 50,000,000 rows lso helps administration and information lifecycle management ta model lat table storage model o indexes, making use of Exadata internal indexing on storage cells ery results ostly sub-second second response times for small-area queries

17 can this be used? rivation of 3D models lassification, conflation with data from other sources b-based or service-based rendering isual inspection, etc. sing the full resolution of the dataset or parts thereof (pyramiding) rge scale data dissemination database processing and analytics otential basis for change detection in multi-temporal point clouds articularly for buildings and vegetation

18 a Production Workflow Capital Region of Brussels entation at Geosptial World Forum 2014

19 a models for City Modeling tydb(open source) is widely used LOD1 Building LOD2 Building LOD3 Building LOD4 Building mantically structured model uctures at multiple levels of detail tures and facades thophotos rsioning Source: Research Center

20 intaining City Models eration with Hasso Plattner Institute Graphics courtesy of Rico Richter, HPI

21 nt Cloud Change Detection Screenshot courtesy of Rico Richter, HPI

22 nt Cloud Change Detection Screenshot courtesy of Rico Richter, HPI

23 intenance of 3D City Models Screenshot courtesy of Rico Richter, HPI

24 intenance of 3D City Models Screenshot courtesy of Rico Richter, HPI

25 nge detection process g massively parallel algorithms in graphics cards Graphics courtesy of Rico Richter, HPI

26 third option use new architectural components ing use of Big Data technologies ploying massively parallel algorithms on commodity hardware ypically using Hadoop platform with MapReduce technology quires spatial processing capabilities patial datatypes and algorithms patial indexing W PRODUCT: Oracle Big Data Spatial and Graph patial libraries and indexing for 2D, 3D, projected or geodetic data aster data processing (mosaic, reprojection, format conversion, analysis,...) ertified on Cloudera Distribution (CDH 5.4)

27 Big Picture Oracle Big Data Management System DATA RESERVOIR DATA WAREHOUSE Cloudera Hadoop Oracle Big Data SQL Oracle NoSQL Oracle R Distribution Oracle Big Data Spatial and Graph Oracle Event Processing Big Data Appliance Apache Flume Oracle GoldenGate Oracle Big Data Connectors Oracle Data Integrator Oracle Oracle Database Database In-Memory, Oracle Industry Multi-tenant Models Oracle Industry Models Oracle Advanced Analytics Oracle Advanced Oracle Spatial Analytics & Graph Oracle Spatial & Graph Oracle Data Integrator Exadata Oracle GoldenGate Oracle Even Processing

28 mary aging 3D Data in an Oracle Spatial and Graph Database e access to 3D data for visualization and analysis egrated management of all kinds of 2D and 3D data ntegration of geospatial and associated attribute data -situ processing bject recognition, data analysis,... nagement and publishing of data and metadata aking data useable through open standards tabase functionality easily complemented with Big Data platform

29 our local community in South Africa uth African Oracle Users Group OUG) pecial Interest Group for Spatial and raph atial SIG Meeting Aug. 20 :00h 12:00h racle Office, Woodmead SAOUG Connect Conference Annual User Conference, Sep Cape Town, Cape Sun Hotel

30 re resources rther information on oracle.com ww.oracle.com/goto/spatial gs ttps://blogs.oracle.com/oraclespatial veloper forums on OTN ttps://community.oracle.com/community/database/oracle-database-options/spa kedin community Oracle Spatial and Graph group ogle+ community Oracle Spatial and Graph SIG

31

32