Databases & Web Applications Lab Big Data Project A

Transcription

1 Databases & Web Applications Lab Big Data Project A Instructor: Peter Baumann [email protected] tel: office: room 88, Research Databases & Web Applications (P. Baumann)

2 Big Science Data [OGC Ocean Science Interoperability Experiment; image source: Mbari] 2

3 OGC Coverage Types Coverage = digital representation of space/time varying phenomenon n-d MultiSolid Coverage MultiSurface Coverage MultiCurve CoverageMultiPoint Coverage «FeatureType» Abstract Coverage Referenceable GridCoverage Grid Coverage Rectified GridCoverage 3

4 Facing the Coverage Deluge sensor feeds [OGC SWE] coverage server 4 4

5 Taming the Coverage Deluge sensor feeds [OGC SWE] coverage server 5 5

6 Let s Take a Closer Look... t Divergent access patterns for ingest and retrieval Server must mediate between access patterns 6

7 Our Research Large-Scale Scientific Information Services (L-SIS) Research Group flexible, scalable services on massive multi-dimensional scientific data Particular focus: n-d arrays Massive = multi-tb multi-pb per object Results: rasdaman array DBMS ( demo at Geoservice standards: OGC WCS suite, ISO 9075 SQL Part 15: MDA (under work) 7

8 rasdaman: Scalable Array Analytics raster data manager : Array Database = SQL + n-d arrays select img.green[x0:x1,y0:y1] > 130 from LandsatArchive as img tile streaming architecture: scaling from laptop to cloud rasdaman Web visitors 8

9 Use Case: Satellite ImageTime Series [Diedrich et al 2001] 9

10 EarthServer Big Earth Data Analytics Up to 130 TB databases for all Earth sciences + planetary science EU FP7-INFRA, 3 years, 5.85 meur Platform: rasdaman; strictly open standards Cryospheric Science landcover mapping Airborne Science high-altitude drones Atmospheric Science climate variables Geology geological models Oceanography marine model runs + in-situ data Planetary Science Mars geology 10

11 Database Visualization select encode( struct { red: (char) s.image.b7[x0:x1,x0:x1], green: (char) s.image.b5[x0:x1,x0:x1], blue: (char) s.image.b0[x0:x1,x0:x1], alpha: (char) scale( d.elev, 20 ) }, "image/png" ) from SatImage as s, DEM as d [JacobsU, Fraunhofer 2012; [data courtesy BGS, ESA] [JacobsU, Fraunhofer; data courtesy BGS, ESA] 11

12 Parallel / Distributed Query Processing ad-hoc federation mixed hardware Dataset D select max((a.nir - A.red) / (A.nir + A.red)) - max((b.nir - B.red) / (B.nir + B.red)) - max((c.nir - C.red) / (C.nir + C.red)) - max((d.nir - D.red) / (D.nir + D.red)) from A, B, C, D Dataset C Dataset A Dataset B 12

13 Secured Archive Integration First-ever direct, ad-hoc mix from protected NASA & ESA services in OGC WCS/WCPS Web client (EarthServer + CobWeb) 13

14 Demo 14

15 Next: On-Board Query Intelligence [OPS-SAT: ESA CubeSat] Democratize direct data access [imagery courtesy ESA, NASA] 15

16 Summary Project work embedded in international projects & collaborations Present Publish 16

17 Big Picture Databases and Web Applications Fall lecture, undergrad + grad Advanced course in spring: Information Architectures Databases and Web Applications Lab Lab, grad Big Data Project A Project, grad New meeting slot: Tue 09:45, Research 1, room 88 17

18 Project Task Pick a topic Perform task planful: Spec document 20% -- Sep 26 Oct 03 Prototype 1: breakthrough implementation 20% -- Oct 17 Prototype 2: ready for benchmark 20% -- Oct 31 Benchmark results 20% -- Nov 14 Publication 10% -- Nov 28 Prototype 3: ready for handover 10% -- Dec 05 18

19 Resources rasdaman website demo Our publications Instructor: the rasdaman team 19

20 Main Evaluation Criteria complete wrt. requirements Solid engineering bug-free, project & code documentation, coding quality,... user-friendliness and appealing look&feel complexity (in absolute terms and in comparison to other teams' work) Good writeup Specification, documentation, paper (no particular order) 20

21 Project/Lab Topics -> course list -> list of topics 21