Use of Cloud Computing for scalable geospatial data processing and access

Transcription

1 Use of Cloud Computing for scalable geospatial data processing and access Andrew Turner CTO, FortiusOne Partner: U.S. Federal Geographic Data Committee

2 What is GeoCommons? A Brief History

3 Vulnerability Identification Chicago Denver Atlanta Fiber Density Route 2 Los Angeles Route 1 Electric Transmission Line Density

4

5

6 Columbus Circle Holland Tunnel WTC Baseline connectivity of a fiber network provider in NYC. This particular provider is a good proxy for the structure of the entire island of Manhattan since they have about 80% of the right of ways on the island and a large number of egress points off the island. The higher the peak in the map the more frequently used the path is as a possible routing path.

7 Lastly a scenario is run where just 10,000 sq ft. of damage is done to the Holland Tunnel and the impact calculated. The result is a 8.6% loss of network connectivity, 134 times the impact of the WTC simulation. The dramatic impact is seen in the image from the loss as well as the stress put on the GW Bridge route out of the city.

8 GeoCommons: Version 1

9

10 Find interesting data

11 Find interesting data Map a relevant area

12 Find interesting data Map a relevant area Visualize to find meaning

13 Find interesting data Map a relevant area Visualize to find meaning Layer, Modify, and Analyze

14 Find interesting data Map a relevant area Visualize to find meaning Collaborate with others Layer, Modify, and Analyze

15 Find interesting data Map a relevant area Visualize to find meaning Publish and share results Collaborate with others Layer, Modify, and Analyze

16

17

18

19 Visualization

20 Analysis

21 Applying Lessons Learned

22 Modularize Application Programming Interface Finder Maker RESTful Core Interfaces

23 Relational Databases Don t Scale Well

24 Datasets as Databases KML Shapefile CSV (Excel) GeoRSS Documents Finder Maker Core

25 Datasets as Databases Upload KML Shapefile CSV (Excel) GeoRSS Documents Finder Maker Core

26 Datasets as Databases Upload KML Shapefile CSV (Excel) GeoRSS Documents Finder Maker Parse & Store Core

27 Datasets as Databases Upload KML Shapefile CSV (Excel) GeoRSS Documents Finder Maker Parse & Store Core

28 Datasets as Databases Upload KML Shapefile CSV (Excel) GeoRSS Documents Finder Maker Parse & Store Core

29 Datasets as Databases Upload KML Shapefile CSV (Excel) GeoRSS Documents Finder Maker Parse & Store Core

30 Datasets as Databases Upload KML Shapefile CSV (Excel) GeoRSS Documents Download Finder Maker Parse & Store Core

31 Datasets as Databases Upload KML Shapefile CSV (Excel) GeoRSS Documents Download Finder Maker Parse & Store Analyze Core

32 Datasets as Databases Upload KML Shapefile CSV (Excel) GeoRSS Documents Download Visualize Finder Maker Parse & Store Analyze Core

33 Geospatial Catalog and Server

34 Delivery Mechanisms

35 Appliances Sun 4150 RAID Array

36 Web Scaled Racks 3 Appliances Network File Storage Load Balancer Monitoring and Tunnels Production & Staging racks Racks in office for development

37 Limits in Scaling Limits in Development

38 Limits in Scaling Limits in Development People

39 Limits in Scaling Limits in Development People Power

40 Limits in Scaling Limits in Development People Power Size

41 Limits in Scaling Limits in Development People Power Size Cost

42 Limits in Scaling Limits in Development People Power Size Cost Time

43 Limits in Scaling Limits in Development People Power Size Cost Time

44 Limits in Scaling Limits in Development People Power Size Cost Time Testing on clean machines

45 Limits in Scaling Limits in Development People Power Size Cost Time Testing on clean machines Deployment testing of upgrades

46 Limits in Scaling Limits in Development People Power Size Cost Time Testing on clean machines Deployment testing of upgrades Controlled Environments

47 url Leveraging the Cloud

48 Amazon Web Services

49 Management Consoles

50 Processing via MapReduce

51 Launching New Instances

52 Elastic Computing Cluster - EC2 Virtual Servers Machine Images (AMI) On-Demand CentOS AMI

53 Elastic Computing Cluster - EC2 Virtual Servers Machine Images (AMI) On-Demand CentOS AMI build

54 Elastic Computing Cluster - EC2 Virtual Servers Machine Images (AMI) On-Demand CentOS AMI register bundle build

55 Elastic Computing Cluster - EC2 Virtual Servers Machine Images (AMI) On-Demand CentOS AMI register bundle instantiate build

56 Elastic Computing Cluster - EC2 Virtual Servers Machine Images (AMI) On-Demand CentOS AMI register bundle instantiate build

57 Elastic Computing Cluster - EC2 Virtual Servers Machine Images (AMI) On-Demand CentOS AMI register bundle instantiate build

58 Elastic Computing Cluster - EC2 Virtual Servers Machine Images (AMI) On-Demand CentOS AMI register bundle instantiate build

59 Elastic Computing Cluster - EC2 Virtual Servers Machine Images (AMI) On-Demand CentOS AMI register bundle instantiate build

60 Elastic Block Store - EBS Create EBS 100 GB

61 Elastic Block Store - EBS Create EBS 100 GB attach

62 Elastic Block Store - EBS Create EBS 100 GB attach snapshot

63 Elastic Block Store - EBS Create EBS 100 GB attach snapshot S3 Diff v1

64 Elastic Block Store - EBS Create EBS 100 GB attach snapshot S3 Diff v1 Diff v2

65 Elastic Block Store - EBS Create EBS 100 GB attach snapshot S3 Diff v1 Diff v2 Create & Attach

66 Elastic Block Store - EBS Create EBS 100 GB attach snapshot S3 Diff v1 Diff v2 Create & Attach

67 Elastic Block Store - EBS Create EBS 100 GB attach snapshot S3 Diff v1 Diff v2 Create & Attach

68 Elastic Block Store - EBS Create EBS 100 GB attach snapshot S3 Diff v1 Diff v2 Create & Attach

69 Public Datasets

70 Additional Benefits Federation Tile generation Content-delivery System Simple Queue System (SQS) tiles/openstreetmap/9/74/97.png S3 Storage tiles/openstreetmap/9/74/98.png tiles/bluemarble/9/74/97.png tiles/bluemarble/9/74/98.png

71 Cloud Architecture EC2 image of current system architecture EBS image stored to S3 of default database Current application release in S3 Start an EC2, attach data, attach code, startup v1.4.3 Default Datasets

72 Cloud Architecture EC2 image of current system architecture EBS image stored to S3 of default database Current application release in S3 Start an EC2, attach data, attach code, startup create instance v1.4.3 Default Datasets

73 Cloud Architecture EC2 image of current system architecture EBS image stored to S3 of default database Current application release in S3 Start an EC2, attach data, attach code, startup create instance v1.4.3 Default Datasets

74 Cloud Architecture EC2 image of current system architecture EBS image stored to S3 of default database Current application release in S3 Start an EC2, attach data, attach code, startup create instance v1.4.3 Default Datasets attach data

75 Cloud Architecture EC2 image of current system architecture EBS image stored to S3 of default database Current application release in S3 Start an EC2, attach data, attach code, startup create instance Default Datasets v1.4.3 attach data Snapshot Backup Backup Backup

76 Cloud Architecture EC2 image of current system architecture EBS image stored to S3 of default database Current application release in S3 Start an EC2, attach data, attach code, startup create instance Cache Downloads S3 Default Datasets v1.4.3 attach data Snapshot Backup Backup Backup

77 Scaling RESTful architecture Caching for speed, and CDN support Amazon Web Services CloudWatch Elastic Scaling Load Balancer

78 Private Instances

79 First Users: Meedan, Media

80 Repeatable

81 Repeatable

82 Data Federation community

83 Geospatial Federated Search Search

84 Geocoding

85 Geocoding - Scale as Required Upload CSV Geocode Cache Results API Geocoding Engine TIGER/Line SQLite

86 Geocoding - Scale as Required Upload CSV Geocode Cache Results API Geocoding Engine TIGER/Line SQLite

87 Best Practices Applied to the Government Built using open, established tools Full choice - Linux, Windows Full Control Repeatable processes Continual backup Scaling dynamic and large datasets Synchronous and Asynchronous analysis

88 Level of Maturity Widely adopted Broad support and ecosystem Full stack support

89 Perceived Impediments to Adoption Single Vendor (open-source alternatives arising) Maintenance and Location Data Security

90 Thank you Andrew Turner