InGeoCloudS: open Cloud-based services for Geospatial Data management in an INSPIRE context Benoit BAURENS, InGeocloudS Project Coordinator benoit.baurens@akka.eu www.akka.eu
OUTLINE Introducing words Why InGeoClouds? InGeoCloudS ID-card What does it feature? How? What s next? InGeoCloudS, GIC Open Meeting @ BRGM, Orléans, June 3rd 2013 2
OUTLINE Introducing words Why InGeoClouds? InGeoCloudS ID-card What does it feature? How? What s next? InGeoCloudS, GIC Open Meeting @ BRGM, Orléans, June 3rd 2013 3
Introducing words Benoit BAURENS, Project Manager and InGeocloudS Project Coordinator representing the whole InGeoCloudS Team!! benoit.baurens@akka.eu www.akka.eu InGeoCloudS, GIC Open Meeting @ BRGM, Orléans, June 3rd 2013 4
Our expertise a global offer InGeoCloudS, GIC Open Meeting @ BRGM, Orléans, June 3rd 2013 5
Our services -flexible and tailored to your needs Experts on demand Reactivity, flexibility Our experts support you on location through A deep understanding of each business sectors A strong master of existing and coming technologies A concrete ability to create innovation Engineering Expertise, know-how Our team support you on your work-packages and tailored turnkey solutions through our Mechanical design centers Systems design centers Software design centers Nearshore/offshore facilities Consulting Result & implementation oriented Our services enable you to sustainably optimize the entire value chain through our entities Casciope MBtech Consulting InGeoCloudS, GIC Open Meeting @ BRGM, Orléans, June 3rd 2013 6
A strong international foothold InGeoCloudS, GIC Open Meeting @ BRGM, Orléans, June 3rd 2013 7
OUTLINE Introducing words Why InGeoClouds? InGeoCloudS ID-card What does it feature? How? What s next? InGeoCloudS, GIC Open Meeting @ BRGM, Orléans, June 3rd 2013 8
Why InGeoCloudS? InGeoCloudS, GIC Open Meeting @ BRGM, Orléans, June 3rd 2013 9
About INSPIRE Data quantity and quality Visibility, accessibility and sharing. InGeoCloudS, GIC Open Meeting @ BRGM, Orléans, June 3rd 2013 10
About INSPIRE In order to ensure that the spatial data infrastructures of the Member States are compatible and usable in a Community and transboundary context, InGeoCloudS, GIC Open Meeting @ BRGM, Orléans, June 3rd 2013 11
About INSPIRE InGeoCloudS, GIC Open Meeting @ BRGM, Orléans, June 3rd 2013 12
Why InGeoCloudS? InGeoCloudSservices intend to support data providers in fulfilling their obligations with regards to INSPIRE. InGeoCloudS, GIC Open Meeting @ BRGM, Orléans, June 3rd 2013 13
About Cloud Computing Towards perfect capacity management? InGeoCloudS, GIC Open Meeting @ BRGM, Orléans, June 3rd 2013 14
About Cloud Computing Towards perfect capacity management? App App App Operating System Hardware Traditional Stack InGeoCloudS, GIC Open Meeting @ BRGM, Orléans, June 3rd 2013 15
About Cloud Computing Towards perfect capacity management? App App App Hardware Operating App System Operating App App App System App App Operating App System App App Traditional App Hardware App Operating Stack App System Hardware Operating App System App App Hardware Operating App System App App Operating Traditional App System Hardware App Traditional Stack App Hardware Operating Stack System Traditional Hardware Operating Stack System Hardware Operating Traditional App System App Traditional Stack App Hardware Stack Traditional Stack Hardware Traditional Stack App App App Operating App System App App App App App Traditional Hardware Operating Stack System Traditional Stack Traditional Stack Hardware Traditional Stack App App App Operating App System App App Hardware Operating App System App App Traditional Hardware Operating Stack System Traditional Stack Hardware Traditional Stack App App App Operating System Hardware Traditional Stack InGeoCloudS, GIC Open Meeting @ BRGM, Orléans, June 3rd 2013 16
About Cloud Computing Towards perfect capacity management? App App App Hardware Operating App System Operating App App App System App App Operating App System App App Traditional App Hardware App Operating Stack App System Hardware Operating App System App App Hardware Operating App System App App Operating Traditional App System Hardware App Traditional Stack App Hardware Operating Stack System Traditional Hardware Operating Stack System Hardware Operating Traditional App System App Traditional Stack App Hardware Stack Traditional Stack Hardware Traditional Stack App App App Operating App System App App App App App Traditional Hardware Operating Stack System Traditional Stack Traditional Stack Hardware Traditional Stack App App App Operating App System App App Hardware Operating App System App App Traditional Hardware Operating Stack System Traditional Stack Hardware Traditional Stack App App App Operating System Hardware Traditional Stack InGeoCloudS, GIC Open Meeting @ BRGM, Orléans, June 3rd 2013 17
About Cloud Computing Towards perfect capacity management? Starting costs can remain reasonable Adaptive capacity: scaling-down is as important as scaling-up InGeoCloudS, GIC Open Meeting @ BRGM, Orléans, June 3rd 2013 18
Cloud Computing Whatare wetalkingabout? Cloud computing is a model for enabling ubiquitous, convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, servers, storage, applications, and services) that can be rapidly provisioned and released with minimal management effort or service provider interaction. (NIST) http://csrc.nist.gov/publications/nistpubs/800-145/sp800-145.pdf InGeoCloudS, GIC Open Meeting @ BRGM, Orléans, June 3rd 2013 19
Cloud Computing What are we talking about? Cloud computing comes from the convergence of: service oriented architectures... loose coupling of services with operating systems and technologies... parallel computing large scale data analysis, up to thousands of machines virtualization independence from physical hardware App App App OS OS Hypervisor OS Hardware Virtualized Stack InGeoCloudS, GIC Open Meeting @ BRGM, Orléans, June 3rd 2013 20
Cloud Computing Expected Cost Reductions Pay-as-you-go: Operations Costs versus Infrastructure Costs Reduced Total Cost of Ownership Technical staff, power supply, physical space hardware, cables, Scale economy among partners Share of Databases, servers, CPUs, InGeoCloudS, GIC Open Meeting @ BRGM, Orléans, June 3rd 2013 21
Cloud Computing Expected Cost Reductions An example of an Amazon EC2 instance: Large Instance: 7.5 GB of memory, 4 EC2 Compute Units (2 virtual cores with 2 EC2 Compute Units each), 850 GB of local instance storage... 0.26 per hour Data Transfers IN... 0.00 Data Transfers OUT: 1 GB free,... 0.12 per GB Elastic Block Store:... 0.11 GB/month... 0.11 per Mi. I/O req.s (May2012) You can define rules to rent new machines as needed automatically! InGeoCloudS, GIC Open Meeting @ BRGM, Orléans, June 3rd 2013 22
Cloud Computing Not onlyabout money A cloud platform shall also provide Large computing power with ad-hoc machines and network Various up-to-date Operating Systems and technologies Ubiquitous access and Quality of Service having the data all in one place shall: Improve interoperability Foster successful collaborations Ease the adoption of international standards Open the door to inter-disciplinary studies InGeoCloudS, GIC Open Meeting @ BRGM, Orléans, June 3rd 2013 23
Why InGeoCloudS The project wants to demonstrate that a Cloud infrastructure can be used by public organisations to provide more efficient, scalable and flexible services for creating, sharing and disseminating spatial environmental data InGeoCloudS, GIC Open Meeting @ BRGM, Orléans, June 3rd 2013 24
OUTLINE Introducingwords Why InGeoClouds? InGeoCloudS ID-card What does it feature? How? What s next? InGeoCloudS, GIC Open Meeting @ BRGM, Orléans, June 3rd 2013 25
Partnership Profile 5 Geological Surveys bringing in 6 initial Use Cases (datasets and applications) Ground Water Management Geo-Hazards GeoData Publication and Mapping EC Support 3 ICT organizations bringing key-expertise Cloud Computing Semantic Web and Linked Data GIS Software architecture and integration InGeoCloudS, GIC Open Meeting @ BRGM, Orléans, June 3rd 2013 26
Key Dates Feb 2012 March 2013 Sept. 2013 July 2014 InGeoCloudS, GIC Open Meeting @ BRGM, Orléans, June 3rd 2013 27
OUTLINE Introducingwords Why InGeoClouds? InGeoCloudS ID-card What does it feature? How? What next? InGeoCloudS, GIC Open Meeting @ BRGM, Orléans, June 3rd 2013 28
Pilot1 Achievements Fundamental scalable services for data management: Elastic Database Server and Elastic File Server Data publication modules Linked Open Data facilities An API: Web Services upon a REST-based architecture Data providers data and applications in the cloud InGeoCloudS, GIC Open Meeting @ BRGM, Orléans, June 3rd 2013 29
Pilot1 Achievements InGeoCloudS, GIC Open Meeting @ BRGM, Orléans, June 3rd 2013 30
OUTLINE Introducingwords Why InGeoClouds? InGeoCloudS ID-card What does it feature? How? What s next? InGeoCloudS, GIC Open Meeting @ BRGM, Orléans, June 3rd 2013 31
How? Our Components Requirements The Basics Apache and Tomcat, MapServer, PostgreSQL+PostGIS, Linux OS, Web Server Map Server Spatial Data Storage Operating System Geo-Spatial Stack What else? GeoNetwork Portal + Data Access/Search + Services definition Framework (Sitools2) Triplestore and SPARQL endpoint (Virtuoso) Generalized REST Architecture for all InGeoCloudS components InGeoCloudS, GIC Open Meeting @ BRGM, Orléans, June 3rd 2013 32
How? The Architecture InGeoCloudS, GIC Open Meeting @ BRGM, Orléans, June 3rd 2013 33
How? Focus GlusterFS at work Transparent access for applications Similar to NFS. Automatic set-up on IGC instances. InGeoCloudS, GIC Open Meeting @ BRGM, Orléans, June 3rd 2013 34
How? Focus on Elastic DataBase Server PostgreSQL (+PostGIS) PgPool Load balancer Master/Slave architecture Streaming replication Scalability Parallel read operations Can add/suppress servers as needed at any time. Reliability Automatic fail-over A slave replaces the Master InGeoCloudS, GIC Open Meeting @ BRGM, Orléans, June 3rd 2013 35
How? Focus on Data Publication Component Transforming geo-data into geo-services HTTP API OGC:WMS OGC:WFS HTTP load balancer Data publication API Mapserver Server Mapserver Server Mapserver Server ELASTIC GEOSPATIAL SERVER CLUSTER ElasticFS and DB Cloud infrastructure InGeoCloudS, GIC Open Meeting @ BRGM, Orléans, June 3rd 2013 36
How? Some lessons Learnt so far. Identify a common software stack Homogeneous infrastructure Scale economies Cloud compliance Define data volumes requirements Capacity planning Scalability challenges and requirements High-throughput services Reliability Guarantee QoS for INSPIRE/OGC services Web Server Map Server Spatial Data Storage Operating System Geo-Spatial Stack It is not simply about portingexisting applications to the cloud, but rather to integratethem in a scalable geo-spatial framework InGeoCloudS, GIC Open Meeting @ BRGM, Orléans, June 3rd 2013 37
How? Some lessons Learnt so far. Focus GlusterFS at work 800 GlusterFS - write GlusteFS - read 730 700 600 Throughput (MB/s) 500 400 300 342 344 200 125 210 100 78 0 77 55 1 2 4 8 Number of Servers InGeoCloudS, GIC Open Meeting @ BRGM, Orléans, June 3rd 2013 38
How? Some lessons Learnt so far. Elastic Web Server at work Issued Requests System Load No. Servers Load Threshold 12000 10000 100 90 80 Requests/ min 8000 6000 4000 3 servers 4 servers 70 60 50 40 30 Average CPU Utilization 2000 2 servers 20 1 server 10 0 1 6 11 16 21 26 31 36 41 46 51 Time 0 InGeoCloudS, GIC Open Meeting @ BRGM, Orléans, June 3rd 2013 39
How? Volumes so far Estimated Data volumes and Throughput Web Req/s: BRGM: 9 (avg) to 1300 (max) Transfer volume GB/month: EPPO 6 -> BRGM 1275 Data volume GB: EKBAA 2 -> BRGM 380 Growth rate GB/year: GEUS 0.2 -> BRGM 50 Required CPUs: GEOZS 2 -> BRGM 29 QoSand Resilience of information: Shake-maps available in case of earthquake Largerequest volume in presence of specific event (e.g., water pollution) InGeoCloudS, GIC Open Meeting @ BRGM, Orléans, June 3rd 2013 40
OUTLINE Introducingwords Why InGeoClouds? InGeoCloudS ID-card What does it feature? How? What s next? InGeoCloudS, GIC Open Meeting @ BRGM, Orléans, June 3rd 2013 41
What s next? Excerpt of current Work IGC Management and Administration High level management and monitoring Including billing and accounting Preserving data-providers ownership and responsibility with own workspace (not a central portal) InGeoCloudS, GIC Open Meeting @ BRGM, Orléans, June 3rd 2013 42
What s next? Open trials : Pilot2 More Datasets expected External institutions (you are) invited to use and try the platform External data providers (you) to push their data in InGeoCloudS More Linked Data Opensearch Sample applications for cross-domains correlation of data Larget set of services Built-in Geoprocessing tools Your services? InGeoCloudS, GIC Open Meeting @ BRGM, Orléans, June 3rd 2013 43
Thanks for your attention Thanksfor yourattention and do not hesitateto jointhe adventure! www.ingeoclouds.eu contact@ingeoclouds.eu www.facebook.com/ingeoclouds @ingeoclouds InGeoCloudS, GIC Open Meeting @ BRGM, Orléans, June 3rd 2013 44