Big Data and evolution of the Ground System EO ENG and the imarine case

Transcription

1 Big Data and evolution of the Ground System EO ENG and the imarine case Andrea Manieri Engineering R&D Lab. Rome, 26/11/

2 AGENDA The Big data challenges seen from the space Engineering and (some) big data challenges The imarine success story: a (no trivial) big (and heterogenous) data platform Demo (optional) 2

3 Internet of the Future 3 Source IDC Prediction 2013: competing on the 3d Platform

4 EO Ground System as enabler for Big Data Platforms Several Challenges ahead: Fast and Secure access to such Data Torrent Smart analysis of data for scientific and civil purposes Powerful visualisation tools Integration of satellite with ground produced data (IoT) Several Opportunities as well More missions, more data, more value Borderless cooperation of societies enhancement Open Access and value generation 4 Source IDC Prediction 2013: competing on the 3d Platform

5 Cloud and Big Data a paradigm shift Cloud Computing and the IT as an utility: Illusion of infinite resources; No up front investment; Pay per use 1. Tools and Techniques enabling Clouds Virtualisation Infrastructure provisioning automation Deployment automation Commodity IT (homogenous, simple, cheap) 5 1.Cfr. Armbrust et.al Above the Clouds Berkeley Univ

6 Perspectives on Big Data Smart sample set and low effort algorithm Vs. Low effort data collection and smart algorithms 6

7 Education and Jobs creation New big data platforms need not ony training but new expertise Curricula are needed for Data Scientists and Data Curators Domain Knowledge portfolio need to supported by Data Management skills 7

8 AGENDA The Big data challenges seen from the space Engineering and (some) big data challenges The imarine success story: a (no trivial) big (and heterogenous) data platform Demo (optional) 8

9 Engineering a world wide ICT System Integrator 7% of domestic market about 7,000 professionals a global player with more than 1,000 large accounts on all vertical sectors business integration & consulting, outsourcing, products and solutions 9

10 Big Data Analytics and Visualization The Spago World 10 Courtesy Grazia Cazzin ENG, Spago World Initiative

11 New generation of (data) scientists The ENG Education path Scholar : Data Management Engineering training school Enrico Della Valle Industrial partner: 2 level Specialising Master in Big Data Engineering - Politecnico of Turin Industrial partner: Big Data Observatory - Politecnico of Milan 11 Courtesy Grazia Cazzin ENG, Spago World Initiative

12 Big Data and Cloud Services CLOE PONT SAINT MARTIN TORINO VICENZA MILANO PADOVA ROMA Portale di Gestione Infrastructure as a Service Gestione e Automazione Platform as a Service Software as a Service Virtualizzazione Server Network Storage Apps 12 Courtesy Manfredi Caizzi Engineering.MO, Innovation Director

13 Safety and Health Applications ENGINEERING RESEARCH & DEVELOPMENT Partecipatory Sensing Urban Context Awreness Gruppo Engineering The data torrent from Internet of things An EU JP initiative

14 AGENDA The Big data challenges seen from the space Engineering and (some) big data challenges The imarine success story: a (no trivial) big (and heterogenous) data platform Demo (optional) 14

15 The System, The Infrastructure, The Initiative 15 Courtesy Pasquale Pagano CNR ISTI, imarine Technical Manager

16 Social Interactions: notifications The Social Portal offers a familiar view of what is happening on their VREs A single place to Get status and updates from applications and other users they are interested in; Get notifications about messages, jobs completion, new generated products, etc. 16 Courtesy Pasquale Pagano CNR ISTI, imarine Technical Manager

17 Social Interactions: the top bar The Social Portal offers a familiar view of what is happening on their VREs A single place to Manage all the portal extension. Messages Workspace Notifications Page Search in your Workspace Home Social 17 Courtesy Pasquale Pagano CNR ISTI, imarine Technical Manager

18 Social Interactions: the workspace The Social Portal offers a familiar view of what is happening on their VREs A single place to Manage data, store and preserve them Share data 18 Courtesy Pasquale Pagano CNR ISTI, imarine Technical Manager

19 Concepts: Virtual Research Environment Virtual Research Environment (VRE) is a distributed and dynamically created environment where subset of data, services, computational, and storage resources regulated by tailored policies are assigned to a subset of users via interfaces for a limited timeframe at little or no cost for the providers of the participatory data e- infrastructures 19 Courtesy Pasquale Pagano CNR ISTI, imarine Technical Manager

20 Storage as a Service 20 Courtesy Pasquale Pagano CNR ISTI, imarine Technical Manager

21 Computing as a Service 21 Courtesy Pasquale Pagano CNR ISTI, imarine Technical Manager

22 The initiative: how gcube installed by D4Science is used with Fishery Communities A BUNDLE is a set of services and technologie s grouped according to a family of related tasks for achieving a common objective 22 Courtesy Pasquale Pagano CNR ISTI, imarine Technical Manager

23 Accounting and the pay per use appraoch Service Data hosting Infrastructure Unlimited users, Infrastructure support, helpdesk, back-up, security Validation (records) Workspace Hardware Default Processing (<1MB) Social Tool Community Management Storage 1TB Validation (Datasets) Custom Processing (> 1MB) User Management Unlimited VRE s Cloud Resources Custom Data Resources Spatial Data integration Large and Active Storage Hour/Day Month Year 23 Courtesy Pasquale Pagano CNR ISTI, imarine Technical Manager

24 The System Flexible 24 Software platform to abstract over differences in location, protocols, and models by It turns resources and technologie s into a utility by offering a single registration Featurerich Courtesy Pasquale Pagano CNR ISTI, imarine Technical Manager

25 The (current) Architecture Physical view OCCI LibVirt IS Not IS IC IS Reg Sw Repo Res Mgr Res BrK IS gl B IS Not IS IC IS Reg Sw Repo ghn Mgr Res Mgr VRE Mod Shib Usage T. Res BrK Deployer SOA3 ARGUS gcore gcore gcore gcore gcore gcore gcore gcore gcore Hypervisor (XEN) Hypervisor (XEN) Hypervisor (XEN) Information System VRE Manager AAA Infrastructure Enabling Services VO (tenant) dedicated Services 25

26 Key take away messages ESA EO ground system is a good candidate for guiding the creation of a multidisciplinary collaborative scientific platform EO data sources (satellites) when complementing ground sensors and signals sources represent the most important (Big) Data sets for civil and scientific initiatives. As such it s urgent to conceive a Public (and Private?) Initiative for exploitation of such values and ensuring long term sustainability Engineering and the imarine experience may represent a success story onto which ESA and ASI may leverage to pave the way for the future of EO ground system. gcube can represent the valid starting point for the next phase of EO ground system, being: Open Source but well documented and supported Cloud Ready for pay per use and federated models Supporting storage, access and manipulation of any data type 26

27 AGENDA The Big data challenges seen from the space Engineering and (some) big data challenges The imarine success story: a (no trivial) big (and heterogenous) data platform Demo (optional) 27

28 Walkthrough in imarine Thanks! Acknowledge Pasquale Pagano, CNR ISTI imarine Technical Manager Ermanno Travaglino ENG R&D, imarine Federated Accounting Grazia Cazzin, ENG R&I Spago World, Big Data Courses and Observatory Manfredi Caizzi, ENG.MO CLOE 28

29 Walkthrough in imarine imarine Gateway D4Science Demo Accounting (Dev) 29

30 Back up slides Log in 30

31 Back up slides Notification 31

32 Back up slides Chat/Forum social networking 32

33 Back up slides Workspace: Content Querying and Browsing 33

34 Back up slides Download 34

35 Back up slides Accounting 35