JBoss Data Services. Enabling Data as a Service with. Gnanaguru Sattanathan Twitter:@gnanagurus Website: bushorn.com

Transcription

1 1

2 Enabling Data as a Service with JBoss Data Services Prajod Vettiyattil Gnanaguru Sattanathan Twitter:@gnanagurus Website: bushorn.com 2

3 What this session is about v The why and what of data services v How data services work v Use cases v JBoss Data Services Platform 3

4 4 Why

5 Proliferation of data Data Consumers Custom er portal Employee portal ERP CRM Accounting Billing Finance Marketing Sales Partner Management Vendor Management SQL File Mainfra me NoSQL Content Manage ment System ERP 5 Data Sources and Data Managers

6 Proliferation: so what? Multiplicity of connections High development cost Huge operational overhead Difficult and risky to change Data Sources/Managers Dispersed data connectors Data duplication Too much ETL Lines of Business copies data Duplicated data aggregation Impossible to create Single source of truth Data ownership issues No comprehensive view No data movement dashboards Location of data and its status 6

7 7 What

8 Data Services and DSP DSP = Data Services Platform Abstracts the data managers/ sources The basic view Data Consumers Presents the data as a service to the consumer ETL++ C1 C2 C3 C4 C5 C6 Data Service 1 Data Service 2 Data Services Platform Data Service 3 Data Service 4 8 Data Managers D1 D2 D3 D4 D5 D6

9 Dashboard in a DSP Data Connections Error Corrections Data movement status Data Dashboard Failures Errors Alerts 9

10 10 How it works

11 Features of a DSP Enables architecture principles Separation of concerns Protected variations Data adapters Data mapping tools and standards Data caching Local and distributed Service search and reuse Data security and data usage audit Data access control Central channel for all data requirements Data dashboard Configurable performance and reliability 11

12 12 Use cases

13 Auto manufacturing supply chain: Requirements Vehicle ownership experience Business Process Automation Disparate data sources Multiple data feeds Parts catalog Prices Dealer updates Parts consumed Parts replaced Part failure statistics Customer feedback Post purchase Breakdown support Service Quality Dashboards Integration solutions based on batch transfers Unreliable Not traceable 13

14 Auto manufacturing supply chain: Layer Diagram Customer Experience Dashboards Business Activity Monitoring Business Processes Enterprise Service Bus Data Services Platform Breakdown reports Parts supplier feeds Customer feedback Parts Catalog Dealer feeds Dealer Info Customer Master 14

15 Enterprise Data Access Layer: Requirements Golden copy / System of Record / Single source of truth Shared services team for Enterprise Data Management Data usage audit Data access control Reduce request load on Data Management team Reduce data maintenance costs 15

16 Enterprise Data Access Layer: Layer Diagram Enterprise Data Consumers Data Services Virtual DB Data base drivers Data Services Platform Auditing Data Aggregation Metadata Data Access Control Mainframe Partner Data Content Management System Partner Info Customer Master Employee Info 16

17 Reporting risk for securities: Requirements Internal and external reporting Risk and margin Centralized risk capture and management Calculate risk from different customer activities Report consolidated data to comply with regulation Dodd Frank Sarbanes Oxley Act (SOX) Dashboards for higher management 17

18 Reporting risk for securities: Architecture without DSP COTS Trading Systems Customer facing Apps Partner Apps Government Systems Reporting Applications Order Mgmt Execution Mgmt Liquidity Mgmt Position Mgmt Order Book Price feeds Enterprise Middleware Systems (MQ, ESB, FTP, File shares) Trade feeds Margin Mgmt Trade Matching Clearing Ref Data Feeds Payment Systems Custom built Apps Risk Management Settlement Ref Data Mgmt Accounting 18

19 Reporting risk for securities: Patterns in this requirement Regulatory requirement for transparency Cannot be met by opaque internal systems Data Sources Large number of them Internal and external Reports are read heavy No real time data requirements once a quarter or once a year No excuses for incorrect data in reports Non-discretionary spending 19

20 JBoss Data Services Platform 20

21 Architecture Data consumers (Custom Applications, COTS products, Business Processes, Business Services ) The EDS platform v5 Runs on SOA-P Teiid ModeShape Data interfaces (JCR API, Web service, JDBC, ODBC, OData,..) Data virtualization Data Adapters SOA Platform Metadata repository Data Services Platform Parts of the architecture Data interfaces Data adapters Data virtualization Metadata repository SAP Sybase Flat file XML SalesFo rce Data Sources Oracle Cassan dra Mongo DB 21

22 Data sources Oracle DB IBM DB2 MS SQL Server MySQL PostgreS QL Sybase Greenpl um Teradata Netezza Ingres Mondria n MetaMa trix LDAP Salesfor ce Delimite d file XML file Web services Apache Hive MS Excel MS Access JBoss Messagi ng JBoss HornetQ TIBCO IBM MQ 22

23 Data Mapping Teiid Designer Map actual data tables using transforms to virtual tables MDD; use Data Models, not SQL Semantic mapping Virtual procedures A set of SQL statements, similar to DB stored procedures 23

24 Data Standards JCR Java Content Repository(JSR-283) OData Open Data Protocol JDBC ODBC Others S-RAMP An SOA repository spec, OASIS Web Services REST JMS 24

25 Access control and Audit Teiid passwords MembeshipDomains for authentication Data roles Fine grained access and visibility control of tables CRUD level permissions for VDB LDAP integration ModeShape LoginContext AuthenticationProvider Role to Action mapping 25

26 Teiid and ModeShape Data type Teiid ModeShape Approach Relational Hierarchical Metadata repository Not suitable Yes Content repository Not suitable Yes ACID transactions Yes Yes SQL queries Yes Yes(JCR-SQL) Flat file data source Yes Not suitable Relational DB data source Yes Not suitable 26 Schema Fixed Optional NoSQL data sources Not suitable Yes Stores data No Yes

27 Summary Data Services Why What How Use cases Auto Manufacturer Enterprise Data Access Layer Regulatory Reporting JBoss DSP Data virtualization 27 Teiid ModeShape

28 Questions Prajod Vettiyattil Gnanaguru Sattanathan Website: bushorn.com Our Open Source Middleware Group on LinkedIn 28