Data Warehouses and NoSQL Sharing Administra6ve Informa6on

Transcription

1 Data Warehouses and NoSQL Sharing Administra6ve Informa6on Carmen Barandela So-ware Engineer CERN / GS AIS October 24 28, 2011 JINR/CERN Grid and Management Informa6on Systems

2 Agenda Data Warehouses in Administra6ve Compu6ng Recap: Data Warehouses Theory Data Warehouses and Informa6on Systems in AIS Founda6on, HR and FI Informa6on Systems Complex Data Extrac6on Processes Pixel Perfect Repor6ng Dashboards NoSQL... 2

3 Agenda Data Warehouses in Administra6ve Compu6ng Recap: Data Warehouses Theory Data Warehouses and Informa6on Systems in AIS Founda6on, HR and FI Informa6on Systems Complex Data Extrac6on Processes Pixel Perfect Repor6ng Dashboards NoSQL... 3

4 Ca. 16,000 People 4

5 Mankind s Largest Machine 5

6 Enormous Amount of Data Jan Janke: "Data Warehouses and Analy6cal Data Processing..." 6

7 Administra6ve Compu6ng Provides means to administrate CERN Enables physicists to focus on their work Allows management to make the right moves 7

8 Why Data Warehouses? Heterogeneous compu6ng landscape Various specialised OLTP systems Planning needs Legal Requirements Support administra6ve staff Enforce security and safety on site Allow management to make decisions 8

9 Example: Keep Finances Under Control Specialised Systems Accoun6ng, ERP for CERN stores External contracts management Payroll, treasury management, Specialised small user groups Dis6nct databases Systems only accessible to authorised specialists High availability and performance, real 6me data 9

10 Example: Keep Finances Under Control General Financial Informa6on System Single system Access to data from mul6ple sources Different levels of complexity Specialised small user groups Dis6nct databases Systems only accessible to authorised specialists High availability and performance, real 6me data 10

11 Example: Keep Finances Under Control General Financial Informa6on System Single system Access to data from mul6ple sources Different levels of complexity Users from all areas of CERN Single data warehouse Security is extremely important! System is accessible CERN wide. High availability and performance, but no necessity for real 6me data 11

12 AIS Financial Data Warehouse Keep data in sync with data providers Master complex data extrac6on process Ensure high query performance Base for detailed data analysis Technologies: o ORACLE RAC database o Java Enterprise web applica6ons o In house developed frameworks o Third party BI and repor6ng tools 12

13 Agenda Data Warehouses in Administra6ve Compu6ng Recap: Data Warehouses Theory Data Warehouses and Informa6on Systems in AIS Founda6on, HR and FI Informa6on Systems Complex Data Extrac6on Processes Pixel Perfect Repor6ng Dashboards Detailed Data Warehouse Example Management Data Layer (MDL) 13

14 Find the Needle in the Hay 14

15 OLTP vs OLAP OLTP OLAP Data source Opera6ons OLTP (consolidated) Data purpose Run the business Repor6ng, analysis Inserts, updates High Periodic batch jobs Query complexity Low High DB design Normalized Star, snowflake Availability Cri6cal Less cri6cal Target Opera6onal staff Middle/higher Mgmt. 15

16 OLTP vs OLAP OLTP OLAP Data source Opera6ons OLTP (consolidated) That s theory! Data purpose Run the business Repor6ng, analysis Real world is not that easy Inserts, updates High Periodic batch jobs Query complexity Low Depends DB design Normalized Snowflake and others Availability Cri6cal May be very criocal Target Opera6onal staff Mgmt. + OperaOons 16

17 Normalisa6on (Codd/Boyce) 1NF 1 table = 1 rela6on, no repea6ng groups or duplicate rows 2NF All non prime anributes depend on all parts (anributes) of a composite key 3NF All non prime anributes depend only on the (whole) key 17

18 Normalisa6on (Codd/Boyce) 1NF 1 table = 1 rela6on, no repea6ng groups or duplicate rows 2NF Not in 3NF, why? All non prime anributes depend on all parts (anributes) of a composite key 3NF Course Category Winner Origin Monaco 10 Formula 1 M. Webber Australia Japan 10 Formula 1 S. Venel Germany Japan 10 Rally S. Ogier France All non prime anributes depend only on the (whole) key) 18

19 Star Schema Branch branch_key branch_name branch_type Sales Fact Table item_key branch_key loca6on_key units_sold dollars_sold avg_sales item item_key item_name brand type supplier_type loca6on loca6on_key Street city state_or_province country Measures Source: hnp:// warehouse/star snowflake schema.php (16/10/2010) 19

20 Branch branch_key branch_name branch_type Measures Snowflake Schema Sales Fact Table item_key branch_key loca6on_key units_sold dollars_sold avg_sales item item_key item_name brand type supplier_key loca6on loca6on_key street city_key supplier supplier_key Supplier_type city city_key city state_or_province country Source: hnp:// warehouse/star snowflake schema.php (16/10/2010) 20

21 From Opera6ons to Repor6ng FI ERP HR Source: hnp:// is.php (16/10/2010) 21

22 Analysis Data Mining Drilldown Finer detail granularity (e.g. add a group by column) Slice & dice Play with the dimensions Combine different dimensions Remove/add a dimension Analyse fact changes 22

23 Agenda Data Warehouses in Administra6ve Compu6ng Recap: Data Warehouses Theory Data Warehouses and Informa6on Systems in AIS Founda6on, HR and FI Informa6on Systems Complex Data Extrac6on Processes Pixel Perfect Repor6ng Dashboards NoSQL... 23

24 CERN/AIS Business Map 24

25 Founda6on Common data layer for various AIS services Data interfaces for other CERN services Common applica6ons (e.g. mgmt. of roles) OperaOve systems HR InformaOon System (HRT) FI InformaOon System (CET) more domain specific informaoon systems 25

26 Various Specialised Systems ORACLE HR CERN Training Applica6on Safety & access systems EDH (Electronic Document Handling) Accoun6ng Applica6on ERP system for CERN stores Contract follow up 26

27 Technical Environment Source databases: ORACLE 10/11g Microsor Excel HR/FI Informa6on Systems: ORACLE 10/11g Java Enterprise web applica6ons SAP Business Objects tool family 27

28 Data Extrac6ons Nightly scheduled batch jobs Extrac6ons organised in SQL scripts Run by self developed batch runner Controls Order of execu6on (sequen6al, parallel) Cri6cality Logging Problem escala6on (automa6c s) 28

29 Defini6on of Extrac6on Process (1) General defini6ons 29

30 Defini6on of Extrac6on Process (2) Batches & commands 30

31 Importance of Monitoring New hardware for DEV databases (gain > 1h) 31

32 Turtle or Leopard? 32

33 ORACLE Materialised Views Pre aggregated summaries Benefit from query rewrite Source: ORACLE 10g Documenta6on / Data Warehousing Guide 33

34 ORACLE Materialised Views Pre aggregated summaries Benefit from query rewrite Source: ORACLE 10g Documenta6on / Data Warehousing Guide 34

35 Materialised (Summary) Views Don t use remote tables if you need query rewrite Create materialized view log on all source tables 35

36 Materialised (Summary) Views Don t use remote tables if you need query rewrite Create materialized view log on all source tables 36

37 Snapshots Use snapshots to efficiently access remote tables Syntax: CREATE SNAPSHOT AS [Your Query] Refresh op6ons: FAST COMPLETE FORCE 37

38 Snapshots Use snapshots to efficiently access remote tables Syntax: CREATE SNAPSHOT AS [Your Query] Refresh op6ons: FAST COMPLETE FORCE 38

39 Pipelined Func6ons PL/SQL is data source instead of a table May increase performance in environments with heavy PL/SQL use 39

40 Pipelined Func6ons PL/SQL is data source instead of a table May increase performance in environments with heavy PL/SQL use 1 CREATE OR REPLACE TYPE mytableformat AS OBJECT( col_a NUMBER, col_b DATE, col_c VARCHAR2(25) ) / CREATE OR REPLACE TYPE mytabletype AS TABLE OF mytableformat / 40

41 Pipelined Func6ons 2 CREATE OR REPLACE FUNCTION myfunc RETURN mytabletype PIPELINED IS BEGIN FOR i in LOOP PIPE ROW ( mytableformat( i, SYSDATE+i, 'Row ' i ) ); END LOOP; RETURN; END; END; 41

42 Pipelined Func6ons 3 SELECT * FROM TABLE( myfunc() ); col_a col_b col_c /10/2010 Row /10/2010 Row /10/2010 Row /10/2010 Row /10/2010 Row 5 Use a pipelined func.on if you require a data source other than a table! 42

43 Database Design Star schema like Highly de normalised incl. duplica6on of data Use single anribute keys wherever possible Performance maners! Be careful when extrac6ng over database links Certain tables from opera6onal systems are copied Dele6on & recrea6on of indexes Use par66ons Manual control of sta6s6cs collec6on Op6mizing execu6on plans very 6me consuming 43

44 Repor6ng Applica6on Framework Column and ordering selec6on Sub reports Various output formats (e.g. HTML, PDF) Charts Self service repor6ng Automated scheduled report execu6on Row and column based access control 44

45 Data Access Which data (columns) am I allowed to see? As a supervisor I may not be en6tled to see the health insurance category. A safety or medical officer may not see the salary, etc. Which rows are visible to me? Unit leader of B only sees persons from Unit B. Name Unit Tel Salary Category Meyer A $ 4,900 3 Schmidt B $ 6,400 1 Cook B $ 5,

46 User Interface 46

47 Pixel Perfect Forms Use of Apache FOP library Examples: Employment & training anesta6ons Swiss / French card applica6on forms Business Objects XI Enterprise Direct use Indirect use via Business Objects Java SDK Examples: Salary slips Car s6ckers Work orders 47

48 Business Objects Commercial tool family from SAP Advantages Rich repor6ng possibili6es (interac6ve or via SDK) Appealing dashboards using Xcelsius Only a few users need the knowledge to design reports Drawbacks Two way data storage (file system & database) Some6mes stability problems Time intensive administra6on and maintenance Expensive 48

49 Management Dashboards Designed locally using MS Office and Xcelsius. Data comes from the MDL data warehouse. Published as Flash to the BO Server. 49

50 Remember: High data volumes + analysis = data warehouse OLTP vs. OLAP Use the facili6es the tool provides Materialized views, snapshots, pipelined func6ons Keep things extensible and simple! Par66ons are very helpful 50

51 Agenda Data Warehouses in Administra6ve Compu6ng Recap: Data Warehouses Theory Data Warehouses and Informa6on Systems in AIS Founda6on, HR and FI Informa6on Systems Complex Data Extrac6on Processes Pixel Perfect Repor6ng Dashboards NoSQL... 51

52 NoSQL Not Only SQL 52

53 Why NoSQL?(1) Tradi6onal RDBMS systems: Defini6on of a rigid schema Guarantees ACID transac6ons:. Atomicity. Consistency. Isola6on. Durability 53

54 Why NoSQL?(1) Tradi6onal RDBMS systems: Defini6on of a rigid schema Guarantees ACID proper6es:. Atomicity. Consistency. Isola6on. Durability Scalability 54

55 Why NoSQL?(2) New web apps different needs:. Scalability & elas6city (at low cost!). High availability. Flexible schemas. Geographic distribu6on Do not necessarily need:. Strong consistency/integrity. Complex queries 55

56 NoSQL use cases 1. Massive data volumes. Massively distributed architecture. Google, Amazon, Yahoo, Facebook Extreme query workload. Impossible to efficiently do joins at that scale with an RDBMS 3. Schema evolu6on. Schema flexibility is not trivial at large scale. Schema changes can be gradually introduced with NoSQL. 56

57 NoSQL Taxonomy Key value stores Document databases Column oriented databases Graph databases 57

58 NoSQL Taxonomy Key value stores 1 key (unique id) > 1 value (binary object, blob) The DB does not understand it: no data model Very fast Examples: Amazon Dynamo, Memcache DB 58

59 NoSQL Taxonomy Document databases Key value store: the value is usually structured Querying with more than a key is possible Examples: Amazon SimpleDB, CouchDB... 59

60 NoSQL Taxonomy Column oriented databases Distributed, persistent mul6dimensional sorted map. Map indexed by: row key, column key and a 6mestamp Examples: Google BigTable, Cassandra (Facebook) Google paper: hnp://sta6c.googleusercontent.com/external_content/untrusted_dlcp/labs.google.com/en//papers/bigtable osdi06.pdf 60

61 NoSQL Taxonomy Graph databases Based on the graph theory: Ver6ces: like en66es Edges: rela6onships between the en66es Examples: Neo4j, FlockDB (Twiner) 61

62 CAP Theorem Consistency Availability ParOOon Tolerance 62

63 CAP Theorem Consistency Availability Only 2 can be saosfied at the Ome ParOOon Tolerance 63

64 No Consistency?

65 No Consistency? Eventual Consistency!!!

66 Eventual Consistency The storage system guarantees that if no new updates are made to the object eventually all accesses will return the last updated value. 66

67 RDBMS vs NOSQL Strong Consistency Big Datasets Scaling is possible SQL Good availability Consolidated tech. Eventual Consistency Huge Datasets Scaling is easy API High availability S6ll immature tech. 67

68 Ques6ons? More Info Спасибо!