320473 Databases & Web Applications Lab 320454 Big Data Project A Instructor: Peter Baumann email: p.baumann@jacobs-university.de tel: -3178 office: room 88, Research 1 320302 Databases & Web Applications (P. Baumann)
Big Science Data [OGC Ocean Science Interoperability Experiment; image source: Mbari] 2
OGC Coverage Types Coverage = digital representation of space/time varying phenomenon n-d MultiSolid Coverage MultiSurface Coverage MultiCurve CoverageMultiPoint Coverage «FeatureType» Abstract Coverage Referenceable GridCoverage Grid Coverage Rectified GridCoverage 3
Facing the Coverage Deluge sensor feeds [OGC SWE] coverage server 4 4
Taming the Coverage Deluge sensor feeds [OGC SWE] coverage server 5 5
Let s Take a Closer Look... t Divergent access patterns for ingest and retrieval Server must mediate between access patterns 6
Our Research Large-Scale Scientific Information Services (L-SIS) Research Group flexible, scalable services on massive multi-dimensional scientific data Particular focus: n-d arrays Massive = multi-tb multi-pb per object Results: rasdaman array DBMS (www.rasdaman.org), demo at www.earthlook.org Geoservice standards: OGC WCS suite, http://external.opengeospatial.org/twiki_public/coveragesdwg/webhome ISO 9075 SQL Part 15: MDA (under work) 7
rasdaman: Scalable Array Analytics raster data manager : Array Database = SQL + n-d arrays select img.green[x0:x1,y0:y1] > 130 from LandsatArchive as img tile streaming architecture: scaling from laptop to cloud rasdaman Web visitors www.rasdaman.org 8
Use Case: Satellite ImageTime Series [Diedrich et al 2001] 9
EarthServer Big Earth Data Analytics Up to 130 TB databases for all Earth sciences + planetary science EU FP7-INFRA, 3 years, 5.85 meur Platform: rasdaman; strictly open standards Cryospheric Science landcover mapping Airborne Science high-altitude drones Atmospheric Science climate variables Geology geological models Oceanography marine model runs + in-situ data Planetary Science Mars geology 10
Database Visualization select encode( struct { red: (char) s.image.b7[x0:x1,x0:x1], green: (char) s.image.b5[x0:x1,x0:x1], blue: (char) s.image.b0[x0:x1,x0:x1], alpha: (char) scale( d.elev, 20 ) }, "image/png" ) from SatImage as s, DEM as d [JacobsU, Fraunhofer 2012; [data courtesy BGS, ESA] [JacobsU, Fraunhofer; data courtesy BGS, ESA] 11
Parallel / Distributed Query Processing ad-hoc federation mixed hardware Dataset D select max((a.nir - A.red) / (A.nir + A.red)) - max((b.nir - B.red) / (B.nir + B.red)) - max((c.nir - C.red) / (C.nir + C.red)) - max((d.nir - D.red) / (D.nir + D.red)) from A, B, C, D Dataset C Dataset A Dataset B 12
Secured Archive Integration First-ever direct, ad-hoc mix from protected NASA & ESA services in OGC WCS/WCPS Web client (EarthServer + CobWeb) 13
Demo 14
Next: On-Board Query Intelligence [OPS-SAT: ESA CubeSat] Democratize direct data access [imagery courtesy ESA, NASA] 15
Summary Project work embedded in international projects & collaborations Present Publish 16
Big Picture 320302 Databases and Web Applications Fall lecture, undergrad + grad Advanced course in spring: Information Architectures 320473 Databases and Web Applications Lab Lab, grad 320454 Big Data Project A Project, grad New meeting slot: Tue 09:45, Research 1, room 88 17
Project Task Pick a topic http://www.faculty.jacobs-university.de/pbaumann/iubremen.de_pbaumann//courses/researchtopics/ Perform task planful: Spec document 20% -- Sep 26 Oct 03 Prototype 1: breakthrough implementation 20% -- Oct 17 Prototype 2: ready for benchmark 20% -- Oct 31 Benchmark results 20% -- Nov 14 Publication 10% -- Nov 28 Prototype 3: ready for handover 10% -- Dec 05 18
Resources rasdaman website www.rasdaman.org demo www.earthlook.org Our publications http://www.faculty.jacobs-university.de/pbaumann/iu-bremen.de_pbaumann//pubs.php Instructor: p.baumann@......and the rasdaman team 19
Main Evaluation Criteria complete wrt. requirements Solid engineering bug-free, project & code documentation, coding quality,... user-friendliness and appealing look&feel complexity (in absolute terms and in comparison to other teams' work) Good writeup Specification, documentation, paper (no particular order) 20
Project/Lab Topics http://www.faculty.jacobs-university.de/pbaumann/iubremen.de_pbaumann/teaching.php -> course list -> list of topics 21