THE STATE OF GEO BIG DATA IN OPEN SOURCE. Rob Emanuele

Transcription

1 THE STATE OF GEO BIG DATA IN OPEN SOURCE Rob Emanuele

2 Who am I? open source geospatial developer working with big geo data. developer at Azavea in Philadelphia, US. maintainer of the GeoTrellis project.

3 GEOBIGDATA FROM A DEVELOPER S PERSPECTIVE

4 Frank Warmerdam PHOTO CREDIT: IAN TURTON

5 Frank Warmerdam Inventor of GDAL Founding director of OSGeo Worked at Google on geospatial systems using MapReduce

6 PlanetLabs

7 PlanetLabs

8 PlanetLabs

9 PlanetLabs

10 PlanetLabs

11 PlanetLabs Processes over 100,000 scenes per day 3-5 meter resolution

12 PlanetLabs - Pipeline position spatially apply geometric corrections apply radiometric corrections apply cloud masking ortho-rectify

13 PlanetLabs - Pipeline GDAL GRASS OSSIM OpenCV

14 BIG DATA IS ABOUT ORCHESTRATION

15 MapReduce (Hadoop) Inflexible: Everything must be a MapReduce job Running locally is painful Debugging is painful

16 PlanetLabs - JobServer PostgreSQL database for job management PostGIS for storing indexed imagery metadata Tasks are orchestrated by machines receiving imagery and the next stage in the pipeline.

17 PlanetLabs - JobServer Allows pipeline operations to be written with C++/python tooling Running batch is very similar to running local Easier to debug

18 PlanetLabs - JobServer workers hitting database causes slowness Postgres/PostGIS is amazing, robust and very fast, but has its limit. It has sharding capabilities for horizontal scalability, but I haven t seen it used in geospatial (is anyone using this?)

19 Horizontal vs Vertical Scalability

20 MIXING HORIZONTAL AND VERTICAL SCALING IS GOING TO CAUSE PAIN.

21 PlanetLabs - JobServer Managing resource allocation is difficult Fault tolerance is hard Advanced orchestration like complex prioritization and task specification are tough, non-geo problems to be solving.

22 ORCHESTRATION IS HARD

23 PlanetLabs - JobServer Managing resource allocation is difficult Fault tolerance is hard Advanced orchestration like complex prioritization and task specification are tough, non-geo problems to be solving.

24 BIG DATA IS ABOUT DEPLOYMENT

25 DEPLOYMENT IS HARD

26 DEPLOYMENT IS HARD; CLOUD PROVIDERS HELP

27 Cloud Providers Amazon Web Services (AWS) Google Cloud Platform OpenStack (e.g. RackSpace)

28 Cloud Providers Amazon Web Services (AWS) Google Cloud Platform OpenStack

29 AWS A set of services for running software on the cloud Many services. SQS, CloudFormation, ECS, EFS, EBS, SWF, Elastic Beanstalk, DynamoDB, Redshift

30 AWS - EC2 Virtual machines that run a variety of hardware specs and operating systems. Spot Instances are cheap! Open source tooling for devops

31 AWS - S3 Object store High availability, distributed access Can share publicly or based on authentication

32 Landsat 8 on AWS Landsat 8 images are published to a public s3 bucket Over 85 TB worth of imagery ts/landsat/

33 Nasa NEX on AWS Downscaled Climate Projections (NEXDCP30) Global Daily Downscaled Projections (NEX-GDDP) MOD13Q1 (Vegetation Indices 16-Day L3 Global 250m) Landsat GLS (Global Land Survey)

34 Nasa NEX on AWS Downscaled Climate Projections (NEXDCP30) Global Daily Downscaled Projections (NEX-GDDP) MOD13Q1 (Vegetation Indices 16-Day L3 Global 250m) Landsat GLS (Global Land Survey)

35 Downsampled Climate Projections Monthly temperature and precipitation data over contiguous US Historical from models, 4 RCP scenarios from netcdf files Over 5 TB of data

36 Local Climate Impact Assessment Modeling Funded by US Department of Energy Azavea in cooperation with Nature Conservancy Goal to make climate model data useful to local regional planners

37

38

39 Hadoop

40 Matei Zaharia

41

42 Apache Spark Open sourced in 2010 under BSD license Formally maintained by UC Berkeley s AMPLab Donated to the Apache Software Foundation in 2013 and relicensed as Apache 2.0 Graduated to a top level Apache project in 2014

43 Apache Spark a distributed computation engine. An API that lets you work with distributed data as a collection. Language bindings for use with Java, Python, and R.

44

45

46

47

48 GeoTrellis a Scala library for geospatial data types and operations. enables Spark with raster capabilities. storage and bounded retrievals from HDFS, Accumulo, and S3

49

50 Accumulo BigTable clone (columnar database) Records stored on HDFS Lexicographically sorted table index

51

52

53

54 Space Filling Curves

55 Space Filling Curves github.com/locationtech/sfcurve

56 Other projects using SFCurve GeoMesa GeoWave

57 Zonal Summaries

58 Zonal Summaries

59 Benchmark Results Yearly Average, 2006 to 2100 Single Layer, GB uncompressed

60 Benchmark Results Yearly Average, 2006 to 2100 Single Layer, GB uncompressed 40 m3.xlarge instances (estimated $2.00 USD per hour on spot market)

61 Summary big data is about orchestration. big data is about deployment. the state of geo big data is the state of big data, with work towards enabling geospatial data types. use Apache Spark! spatial indexing of distributed data is a hot topic.

62 LET S DEVELOP AND USE THE BEST TOOLS POSSIBLE

63 THANK gitter.im/geotrellis/geotrellis github.com/geotrellis/geotrellis