Analysing Big Data in ArcGIS

Transcription

1 OIL & GAS Analysing Big Data in ArcGIS AIS based risk modelling Esri European Petroleum GIS Conference 2014 Karl John Pedersen 7th November DNV GL th November 2014 SAFER, SMARTER, GREENER

2 Introduction Large amounts of data are produced from monitoring systems Processing and analysing this data provides a challenge How does DNV GL use Big data AIS vessel tracking Background and challenges Solution Implementation examples Q & A 2

3 Background Environmental Modelling and GIS unit within Oil & Gas, DNV GL, Oslo DNV GL - International company headquartered in Norway Global classification, certification, technical assurance and advisory company GIS used in risk modelling and visualisation for customers Many projects use AIS data How have we adapted to increasing amounts of AIS data Examples from Oil & Gas and Maritime sectors ,000 years offices countries employees 3

4 What is AIS AIS - Automatic Identification System - vessel tracking data Create lines append lines spanning multiple days Join to ship database to get attributes e.g Vessel type & size, engine & fuel type 4

5 Analysis examples Examples where AIS data are used in risk modelling 5

6 View vessel traffic around an offshore installation 6

7 Calculate vessel density around installation collision risk 7

8 Risk to pipelines from fishing activity Installation 8

9 Number of fishing vessels per grid cell quantifies risk 9

10 Fishing activity vs. depth risk from vessel anchors 10

11 Developing risk models in ArcGIS AIS data used in further analysis AIS data from ArcGIS often used as input to DNV GL risk models Results visualised/mapped in ArcGIS Risk models increasingly developed in ArcGIS: ArcGIS AIS Database Process AIS data in ArcGIS Run risk model Map/ Visualisation Data export for users Excel/Access 11

12 ArcGIS modelling example Input accident frequencies from Excel Run risk model - Python Arctic Shipping Risk 12

13 How we previously handled AIS data Data in SQL Server accessed via IBM Cognos Imported text files from Cognos into ArcGIS Challenges: Large text files, lacking data structure Many steps - possibility of errors Limited geographic selection Time consuming Increasing data quantities Historical data Global data Need to improve access to AIS Big data 13

14 What is Big data? Big data a collection of data sets so large and complex difficult to process using traditional data processing applications (Wikipedia) Example from DNV GL: Database of > 2,5 billion AIS points 14

15 Solution for handling AIS data IBM Netezza data warehouse implemented to improve data access Implemented in existing Cognos environment What is Netezza: Data warehouse appliance from IBM Provides rapid access and analysis of Big data Includes a Spatial Esri package AIS data readable directly in ArcGIS 15

16 Making data available to ArcMap from Netezza Netezza Spatial Esri package AIS data imported from supplier Spatial data created in Netezza Polygon data loaded via WKT format 16

17 Accessing data in Netezza from ArcGIS Install IBM Netezza ODBC driver Add data as normal geographic data has own icon Data access via: ArcToolbox Query Layer Python 17

18 ArcToolbox Query Layer Processing carried out on server Select geographical area Points mapped directly in ArcGIS Data downloadable to ArcGIS Work reduced from days to minutes 18

19 Allows analysis of global AIS data 19

20 Serving maps to users - ArcGIS Server ArcGIS Server service for AIS data created Issues creating service for whole dataset 20

21 Serving maps to users - Cognos Cognos use spatial data from Netezza Esri Maps for Cognos Implemented simple mapping tool 21

22 Limitations ArcGIS cannot write to Netezza Issues creating ArcGIS service No support for raster Limited user base 22

23 Summary Advantages of implementing solution: Enables significantly improved access to AIS data Allows analysis which was not previously possible Part of existing data warehouse Ability for existing non-gis users to analyse data using familiar tools 23

24 That s all! Questions? Contact details: Karl John Pedersen Principal Specialist, Environmental Modelling and GIS, Environmental Risk Management, Høvik, Norway [email protected] Mobile SAFER, SMARTER, GREENER 24