Big Data Spatial Analytics An Introduction

Transcription

1 2013 Esri International User Conference July 8 12, 2013 San Diego, California Technical Workshop Big Data Spatial Analytics An Introduction Marwa Mabrouk Mansour Raad Esri iu UC2013. Technical Workshop op.

2 Agenda What is Big Data? What is Hadoop? How does Spatial integrate with Big Data and Hadoop? How do I get started? Esri UC2013. Technical Workshop. Big Data Spatial Analytics An Introduction

3 New Challenges for Organizations Better Decision making Intelligence Insight/ Foresight Social Data Analysis Log files analysis Fraud Detection Esri UC2013. Technical Workshop. Big Data Spatial Analytics An Introduction

4 Collect Data!!! Esri UC2013. Technical Workshop. Big Data Spatial Analytics An Introduction

5 Is it a large network of a million points??? Is it Satellite & Aerial Imagery??? What is Big Data??? Is it a simple Geodatabase with Billions of records??? Is it this technology called hadoop? Esri UC2013. Technical Workshop. Big Data Spatial Analytics An Introduction

6 Big Data, a new Data Type Big Data Volume - Ever growing data, petabytes / zetabytes - Use 350 billion annual meter readings to better predict power consumption Velocity - Time-sensitive, two mins can be too late - Turn 12 TBs of tweets into updates about people s conditions during a storm for emergency response Variety - Any type of data, text / audio/ video / click streams / other - Exploit documents, images, voice recordings, videos in customer experience analysis programs to improve customer satisfaction Esri UC2013. Technical Workshop. Big Data Spatial Analytics An Introduction

7 What is hadoop? Common HDFS MapReduce Esri UC2013. Technical Workshop. Big Data Spatial Analytics An Introduction

8 Esri UC2013. Technical Workshop.

9 Esri UC2013. Technical Workshop.

10 Parallel Framework Executes Map Reduce Task Reads HDFS Java/Python/Awk Esri UC2013. Technical Workshop.

11 $> cat input Map sort Reduce > out Esri UC2013. Technical Workshop.

12 Esri UC2013. Technical Workshop.

13 Apache Hive Esri UC2013. Technical Workshop.

14 SQL MapReduce Job Esri UC2013. Technical Workshop.

15 hive> select * from cities where country= lebanon ; HDFS CSV TSV JSON BINARY MapReduce Esri UC2013. Technical Workshop.

16 Esri UC2013. Technical Workshop. What About Spatial?

17 Big Data A new data type for Geospatial Maps Spreadsheets Social Media Big Data Services DBMS Sensor Networks Imagery Esri UC2013. Technical Workshop. Big Data Spatial Analytics An Introduction

18 Geospatial in Big Data Big Data with location in Enterprise Big Data warehouse Run spatial queries on data where it resides Pull Results in ArcGIS: Visualize results as a map; Include in a report; Publish in a web or mobile app Big Data Esri UC2013. Technical Workshop. Big Data Spatial Analytics An Introduction

19 GIS Tools for Hadoop Support Running geometry-based spatial queries inside Hadoop Open Source - Apache 2.0 license Two types of users - Developers - ArcGIS users Esri UC2013. Technical Workshop. Big Data Spatial Analytics An Introduction

20 GIS tools for Hadoop libraries ArcGIS Geoprocessing Tools Connect From ArcGIS to Hadoop using GP GIS Tools for Hadoop (Use samples with Hadoop & Hive to get started) Spatial Framework Run Hive Queries with spatial operators Esri Geometry API Build Map/ Reduce Spatial Apps in Java Esri UC2013. Technical Workshop.

21 GIS Tools for Hadoop Github repos GIS tools for Hadoop ArcGIS User Starting Point Develop er Starting Point GeoProcessing tools for Hadoop Spatial framework for Hadoop Esri Geometry API for rjava a (EGA) Esri UC2013. Technical Workshop.

22 Spatial Storage CSV,TSV Lat,Lon Esri JSON format -{geometry:{x:-123,y:45},attributes:{}} Custom Esri UC2013. Technical Workshop.

23 User Defined Functions select tolower( ESRI ); select * from mytable where cos(rad) < 0.1; Esri UC2013. Technical Workshop.

24 Esri UC2013. Technical Workshop. Spatial UDF!

25 select * from cities where near (x,y,-84.2,39.4); Esri UC2013. Technical Workshop.

26 select * from cities where contains (x,y, #mypolys ); Esri UC2013. Technical Workshop.

27 How to get started? Technology Download GIS Tools for Hadoop -Sample (jars + sample data/ query) Hadoop cluster -Amazon Web Services EMR -Cloudera Manager and VM, CDH Run sample Esri UC2013. Technical Workshop. Big Data Spatial Analytics An Introduction

28 How to get started? Data and Query Big Data Use Case, problem definition Dataset - Formats - Conversion - Location - Loading Query - Hive Script Esri UC2013. Technical Workshop. Big Data Spatial Analytics An Introduction

29 Cloud AWS Elastic Map Reduce GP Tools Esri UC2013. Technical Workshop. Big Data Spatial Analytics An Introduction

30 On Premise Cloudera Map Reduce Esri UC2013. Technical Workshop. Big Data Spatial Analytics An Introduction

31 Big Data The Data warehouse Esri UC2013. Technical Workshop. Big Data Spatial Analytics An Introduction

32 Esri UC2013. Technical Workshop. Big Data Spatial Analytics An Introduction

33 Big Data Partners Esri UC2013. Technical Workshop. Big Data Spatial Analytics An Introduction

34 Thank you Marwa Mabrouk - mmabrouk@esri.com Mansour Raad - mraad@esri.com, thunderheadxpler.blogspot.com/ Please fill out the session evaluation First Offering ID: 2333 Esri UC2013. Technical Workshop. Online Paper pick up and put in drop box

35 Esri UC2013. Technical Workshop. Big Data Spatial Analytics An Introduction