Evolution of Information Management Architecture and Development

Transcription

1 Evolution of Information Management Architecture and Development Stewart Bryson Chief Innovation Officer, Rittman Mead! Andrew Bond Head of Enterprise Architecture, Oracle EMEA

2 Oracle Information Management Reference Architecture

3 Oracle Information Management Reference Architecture

4 Oracle Information Management Reference Architecture

5 Nothing Changed High costs Top Technology Priorities 1. Analytics and Business Intelligence 2. Mobile technologies 3. Cloud computing (SaaS, IaaS, PaaS) 4. Collaboration technologies (workflow) 5. Legacy modernisation 6. IT management 7. CRM Source: Gartner Exec Program survey of 2,000+ CIO s worldwide (Jan 2013) Poor performance Fragile / Poor agility No clear business executive sponsorship Lack of true business alignment of projects Poor user engagement Poor data quality Data governance procedures insufficient

6 Some Things Have Changed

7 De-Mystifying Schema on Read ETL DQ Bus. Rules Mapping Data pools Traditional Schema on Write Data quality managed by formalised ETL process Data persisted in tabular, agreed and consistent form Data integration happens in ETL Structure must be decided before writing Big Data Schema on Read Interpretation of data captured in code for each program accessing the data Data quality dependent on code quality Data integration happens in code

8 Everything is About to Change! Memory is getting cheap

9 Conceptual View Actionable Events Actionable Insights Actionable Information Structured Enterprise Data Input Events Event Engine Data Reservoir Data Factory Enterprise Information Store Reporting Other Data Execution Innovation Events & Data Discovery Lab Discovery Output

10 Discovery Lab Actionable Events Actionable Insights Actionable Information Structured Enterprise Data Input Events Event Engine Data Reservoir Data Factory Enterprise Information Store Reporting Other Data Execution Innovation Events & Data Discovery Lab Discovery Output

11 Information Platform Actionable Events Actionable Insights Actionable Information Structured Enterprise Data Input Events Event Engine Data Reservoir Data Factory Enterprise Information Store Reporting Other Data Execution Innovation Events & Data Discovery Lab Discovery Output

12 Data Application Actionable Events Actionable Insights Actionable Information Structured Enterprise Data Input Events Event Engine Data Reservoir Data Factory Enterprise Information Store Reporting Other Data Execution Innovation Events & Data Discovery Lab Discovery Output

13 Information Solution Actionable Events Actionable Insights Actionable Information Structured Enterprise Data Input Events Event Engine Data Reservoir Data Factory Enterprise Information Store Reporting Other Data Execution Innovation Events & Data Discovery Lab Discovery Output

14 Real-Time Events Actionable Events Actionable Insights Actionable Information Structured Enterprise Data Input Events Event Engine Data Reservoir Data Factory Enterprise Information Store Reporting Other Data Execution Innovation Events & Data Discovery Lab Discovery Output

15 Information Management Logical View Data Sources Data Ingestion Access & Performance Layer Enterprise Performance Management Data Engines & Poly-structured sources Structured Data Sources Content Docs Operational Data COTS Data Master & Ref. Data Streaming & BAM SMS Web & Social Media Foundation Data Layer Raw Data Reservoir Immutable raw data reservoir Raw data at rest is not interpreted Past, current and future interpretation of enterprise data. Structured to support agile access & navigation Immutable modelled data. Business Process Neutral form. Abstracted from business process changes Information Interpretation Virtualisation & Query Federation Pre-built & Ad-hoc BI Assets Information Services Discovery Lab Sandboxes Project based data stores to support specific discovery objectives Rapid Development Sandboxes Project based data stored to facilitate rapid content / presentation delivery Data Science

16 Technical architecture assessment

17 Complex Event Processing Expert System Decision Engine Social Social

18 Delivering the Information Management Reference Architecture Stewart Bryson

19 Three Versions of the BI Development Process What IT thinks it should be Requirement Analysis High Level Design Low Level Design Coding Testing Acceptance Testing What normally happens Excel Spreadsheet Shared linked spreadsheets Local Access Database Shared Access Server SQL Server Database Oracle Datawarehouse What we are trying to acheive Discovery & Profile Model Exploit

20 Actionable Events Actionable Insights Actionable Information Structured Enterprise Data Input Events Event Engine Data Reservoir Data Factory Enterprise Information Store Reporting Other Data Execution Innovation Events & Data Discovery Lab Discovery Output

21 The data factory is our conduit between our Data Reservoir and our Enterprise Model Actionable Events Actionable Insights Actionable Information Structured Enterprise Data Input Events Event Engine Data Reservoir Data Factory Enterprise Information Store Reporting Other Data Execution Innovation Events & Data Discovery Lab Discovery Output

22 How do we build the Data Factory?

23

24 ETL is the traditional choice for building a Data Factory

25 Does this look familiar?

26 Typical Waterfall Approach Interview users Construct requirements Create logical data model SQL prototyping of source model Document source-to-target mapping ETL development Front-end development Performance tuning

27 Alternative: Iterative Waterfall? Design iterations around smaller chunks Iteration 1: Interviews and user requirements Iteration 2: Logical modeling Iteration 3: ETL Development Iteration 4: Front-end development Requires 4 iterations before we get any usable content

28 Manifesto for Agile Software Development

29 Manifesto for Agile Software Development We are uncovering better ways of developing software by doing it and helping others do it.! Through this work we have come to value:! Individuals and interactions over processes and tools! Working software over comprehensive documentation! Customer collaboration over contract negotiation! Responding to change over following a plan! That is, while there is value in the items on the right, we value the items on the left more.!

30 Manifesto Applied to Reference Architecture User Stories instead of requirements documents User asks for content or functionality through a narrative Typically includes current version of the report Time-boxed iterations Iteration has a standard length Choose one or more user stories to fit in that iteration Rework is part of the game There are no missed requirements... only those that haven t been delivered yet.

31 Build Now, Refactor Later

32 Model-Driven Development: An Agile approach to building the Data Factory

33 Rapid Development Sandbox Staging layers exist to support batch load scenarios Oracle GoldenGate can bypass the Staging layer to trickle-feed the Foundation layer

34 Rapid Development Sandbox We use our Foundation layer combined with any Access & Performance layer content to close stories immediately

35 Rapid Development Sandbox Developers can use independent development sandboxes to work on their issues in isolation

36 Model-Driven Development: Relational Only Populate the Foundation Layer directly using Oracle GoldenGate We use the OBIEE Semantic Layer to virtually build dimensional models

37 Oracle GoldenGate Architecture

38 GoldenGate: Loading the Foundation Layer INSERTALLRECORDS makes the Foundation Layer possible Map SOURCE and TARGET map SUGARCRM.ACCOUNTS, target SUGARFND.ACCOUNTS, KEYCOLS(ID, FND_SCN), INSERTALLRECORDS, colmap( USEDEFAULTS, KEYCOLS defines matching criteria USEDEFAULTS defines column for column mapping Pull built-in attributes from the token and the header "CSN"), "COMMITTIMESTAMP"), "OPTYPE") );

39 Model-Driven: Mapping Physical to Logical

40 Model-Driven: Mapping Physical to Logical True or False OBIEE is only effective with dimensional models ( star schemas )? False Logical Table Sources (LTS) provide the functionality for mapping transactional schemas as dimensional models. Siebel initially acquired nquire and the technology that would become OBIEE to report against Siebel CRM.

41 ETL Development: Collateral from Model-Driven Development 1

42 ETL Development: Collateral from Model-Driven Development 1 2

43 ETL Development: Collateral from Model-Driven Development 1 2 3

44 ETL Development: Collateral from Model-Driven Development 1 WITH SAWITH0 AS (select count(t2196.id) as c1, sum(t2196.amount) as c2, concat(concat(t3853.first_name, ' '), T3853.LAST_NAME) as c3, T2196.LEAD_SOURCE as c4, T2196.SALES_STAGE as c5 from SUGARFND.ACCOUNTS T2041 /* Accounts (Standard) */, SUGARFND.USERS T3853 /* Users (Account Owner Role) */, SUGARFND.OPPORTUNITIES T2196 /* Opportunties (Standard) */, SUGARFND.ACCOUNTS_OPPORTUNITIES T2221 /* Accounts Opportunities (Standard) */ where ( T2041.ASSIGNED_USER_ID = T3853.ID and T2041.ID = T2221.ACCOUNT_ID and T2196.ID = T2221.OPPORTUNITY_ID ) group by T2196.LEAD_SOURCE, T2196.SALES_STAGE, concat(concat(t3853.first_name, ' '), T3853.LAST_NAME)), SAWITH1 AS (select distinct 0 as c1, D1.c3 as c2, D1.c4 as c3, D1.c5 as c4, D1.c2 as c5, D1.c1 as c6 from SAWITH0 D1) select D1.c1 as c1, D1.c2 as c2, D1.c3 as c3, D1.c4 as c4, D1.c5 as c5, D1.c6 as c6, D1.c7 as c7, D1.c8 as c8 from ( select D1.c1 as c1, D1.c2 as c2, D1.c3 as c3, D1.c4 as c4, D1.c5 as c5, D1.c6 as c6, sum(d1.c5) over (partition by D1.c3) as c7, 2 sum(d1.c6) over (partition by D1.c3) as c8 from SAWITH1 D1 order by c1, c7 desc, c3 ) D1 3 where rownum <=

45 Loading Access and Performance: Relational Staging Layer Access and Performance Foundation Layer

46 Loading Access and Performance: Relational GoldenGate handles Staging and Foundation layers ODI from Staging for transactional table loading Source System Staging Layer Access and Performance Foundation Layer ODI from Foundation for complex calculations and integration

47 How do we build the Data Factory for Big Data sources?

48 How do we build the Data Factory for Big Data sources?

49 Code

50 De-Mystifying Schema on Read ETL DQ Bus. Rules Mapping Data pools Traditional Schema on Write Data quality managed by formalised ETL process Data persisted in tabular, agreed and consistent form Data integration happens in ETL Structure must be decided before writing Big Data Schema on Read Interpretation of data captured in code for each program accessing the data Data quality dependent on code quality Data integration happens in code

51 Can we build a Data Factory against Big Data in an Agile way?

52 Actionable Events Actionable Insights Actionable Information Structured Enterprise Data Input Events Event Engine Data Reservoir Data Factory Enterprise Information Store Reporting Other Data Execution Innovation Events & Data Discovery Lab Discovery Output

53 Sandboxes facilitate Agile Sandbox delivery options Separate Data Lab environment Delivered as part of Information Management architecture Self Service Sandboxes Self service provisioning of new sandboxes for Discovery phase Automation of data access rights, resources and tools provisioning Data provision Quickly take on new data to rapidly make available to Analysts Tools such as Data Factory can fully automate data flows

54 Agile Data Factory for Hadoop Building an OBIEE metadata repository against Hadoop using Hive or Impala Moving sandbox-based R models to Oracle R Enterprise Using Oracle Direct Connector for Hadoop

55 Using OBIEE Against Hive/Impala

56 Rules of Thumb for Data Organized information leads to better analyses Information needs to be organized in order to analyse it RDBMS are great when information is organized Hadoop minimizes the penalty for disorganization The closer you are to insight, the more complete and organized information needs to be Data needs to be organized to monetize it effectively

57 What that really means is We need to apply structure to data in order to analyze it Schema on read works well for us in Discovery as we can be Agile about interpretation As we move into Discovery schema on read can causes Governance & quality issues Key lesson: The cost to store & manage is distinct from structural considerations between Big Data and RDBMS technologies

58 Conclusions Nothing has changed Architecture principles remain the same -Best practice -Reference architectures Some things have changed New technologies mean more opportunities for silos and religion The genius of AND and the tyranny of OR -Check out TDWI Chicago 2013 Keynote Bigger Impact by Ken Rudin Director of Analytics, Facebook Everything is about to change In-memory technologies will provide massive opportunity for data exploitation and huge architectural simplification as well as collaborative development

59 Call To Action Work with us to collaborate on your next architecture revision