Forschungskolleg Datenbanken

Transcription

1 Forschungskolleg Datenbanken

2 Wer für Wen? Verschiedener Mitarbeiter der Professur Datenbanken Leitung: Martin Hahmann, Informatik/Medieninformatik Bachelor: B-510, B-520, B-530, B Vertiefung zur Bachelorarbeit im FS 5 Informatik/Medieninformatik Master/Diplom: VERT-4, PM-FOR, PM-ANW - Vertiefung zur Masterund Diplomarbeit 2 SWS Vorlesung für Bachelor 2 SWS Vorlesung, 2 SWS Übung für Master 2

3 Information zur Veranstaltung Folien werden unter zum Ausdrucken zur Verfügung gestellt (Zugriff von außerhalb der TUD. Login: tud Passwort: dbs und umgekehrt) Weitere Informationen Kontakt per Mail an: 3

4 Inhalt und Ziel der Vorlesung Heute: Vorstellung der Arbeiten am DB-Lehrstuhl im Überblick Vorstellung der Forschungsthemen im Detail Präsentation möglicher Themen für eine Bachelor-, Master- bzw. Diplomarbeit 1-2 Vorlesungen zum Thema Wissenschaftliches Arbeiten Übungen für Master Studenten Wahl eines Forschungsgebiets Kontaktaufnahme mit Betreuer Vorbereitung/Start Bachelor-, Master-, bzw. Diplomarbeit 4

5 5

6 Dresden Database Systems Group 6

7 The Zettabyte Age 20 PB processed every day at Google (2008) 200 million photos are uploaded Facebook every day 2,314 photos/second (2010) 60 hours of video uploaded to YouTube every minute (2012) By 2015, 1 Zettabyte of data will flow over the internet (Cisco Visual Networking Index, June 2011) One zettabyte = stack of books from Earth to Pluto 20 times 7

8 The End of Science The quest for knowledge used to begin with grand theories. Now it begins with massive Everything is data amounts of data. Rise of data-driven culture Zetta High-performant data analytics Welcome to the Petabyte Age. Exploit sophisticated statistical methods 8

9 10

10 Our Research Projects Architecture 12

11 13

12 Time Series Forecasting Sophisticated models Advanced strategies High-dimensional data Real-time requirements 14

13 Model-Based Forecasting Process Statistical time series description (model) New Time Series Values U i Forecasting Values F i+h Model Evaluation Model Identification Model Estimation Forecasting and Model Update Model Evaluation Model Adaptation Time Series Data Model Type AR(2) Model Parameters φ 1 =0.55, φ 2 =0.45 Model Estimation Forecasting Model Adaptation Model Identification Model Creation Model Usage Model Maintenance 15

14 Forecasting Sales Data Data ascertainment Reports Time series analysis Metering technology Exploit latest technological trends Sneak Peak Reporting Missing data: fill gaps Outlier Detection GfK Headquarter Nürnberg 16

15 Renewable Energy Forecasting Increasing share on total energy production Fluctuating output depending on exogenous influences Decentralized allocation Stability requires foresightful balancing of demand and supply Surplus electricity cannot be stored efficiently Access to technical details of RES installations is limited Many meters are still non-profiled Self-consuming households ( ProSumers ) 17

16 18

17 Clustering Clustering is the task of assigning a set of objects into groups (called clusters) so that objects in the same cluster are more similar (in some sense or another) to each other than to those in other clusters Find a similarity measure Clusters may have different size, shapes and densities Clusters can be nested hierchically

18 Clustering: A Users Dilemma multiple / different parameters per algorithm algorithm setup no general definition of quality zoo of measures & visualizations result interpretation Which algorithm fits my data? Which parameters fit my data? How good is the obtained result? How to improve result quality? >500 algorithm variants algorithm selection adjustment take a guess heuristics Algorithmics Feedback 20

19 Augur: A Generalized Process Aggregation techniques Modular clustering algorithms Ensemble generation Visualization of high-dimensional data Convenient interaction on portable clients 21

20 22

21 System Architecture Challenges OLTP OLAP 23

22 ERIS Architecture Overview Frontends SQL SPARQL nicekiki µkiki User Code Operator Pool Extensible Core Programming Language Custom Operators Advanced Data Objects Storage Domain Local Memory Storage Domain Local Memory Monitoring and Adaption Storage Domain Local Memory Dynamic Routing Table Programmable Schema-Free Storage In-Memory Optimized NUMA-Aware Latch-Free Self-Organizing Runtime and Application Code Cyclic Logical Logging 24

23 25

24 Database Evolution Takes time because of number of installation Application Version 1 Application Version 2 Application Version 3 Application evolution Version of the schema Data Takes time because of data volume 26

25 Database Evolution Application Version 1 Application Version 2 Application Version 3 CONNECT TO db VERSION v n ; Schema versions Data Other database objects 27

26 28

27 Compression of Intermediate Results In-memory DBMS dominate analytical data processing Accessing intermediate results as expensive as accessing base data optimization potential Employ compression Base Data CPU Main Memory Intermediate Results Reduced transfer times Better cache utilization Fast operations directly on compressed data Computational overhead Decision for one format 4-Gamma Null Suppression Bit Packing Frame-of-Reference Run Length Encoding VByte NewPFD PFOR VarInt-G8IU Delta Coding Dictionary Coding 4-Wise NS Exploitation modern Hardware, e.g. SIMD instructions a b c d e f g h a+e b+f c+g d+h one instruction From one compressed format to another Without inefficient complete decompression 29

28 Databases on low-energy many-cores Parallel computing on energy-efficient heterogeneous HW, i.e. the Tomahawk Close to the hardware (no underlying os) Data-flow oriented overcome SQL Feedback may end up in new HWfeatures Close cooperation with e-engineers and other chairs of the faculty of computer science 30

29 VAVID Input Relations Datatransfer is (the) slow Memory hierarchies are complex to optimize Databases can control and optimize access globally A B C X Y Z SQL is fast because of advanced optimization - Known datastructure: Relation - Known operations: Relational Algebra Arbitrary Programs + Speed of SQL? - Domain specific languages for data processing - Optimizable datastructures and operations - E.g. PACTs: composable processing functions à la MapReduce Known data dependencies for automatic parallelism User function Cross AX BX CX AY BY CY AZ BZ CZ 31

30 32

31 Everyday Life Manufacturer Role Role Role Person Applicant Employee Customer Time 33

32 Role Types and Natural Types Separating the functional core of an object from its role-based features Role types linked with can-play relation to Natural Types Sharing Role Types over several Natural Types Relation between role types for foundation Represents relationship between entities Person Student School Employee Customer Employer Retailer Company University Owner Accomod. Building 34

33 35

34 Example: Product Recalls 36

35 Property Graph Data Model 37

36 Graph Processing (Old World) Data already in RDBMS SQL as interface Data transfer to application Efficient processing in GDBMS Processing on replicated data No combination with relational data possible 38

37 Graph Processing (New World) SQL/GQL Storage Separate graph engine next to relational engine Query plans with relational and graph operators Support for relational and graph query language Relational and graph data can be addressed together in the same query

38 Forschungskolleg Datenbanken