Anzo Smart Data Integra/on

Transcription

1 Anzo Smart Data Integra/on Cambridge Seman-cs Contact: Marty Loughlin Vice President, Financial Services Cambridge Seman<cs 141 Tremont St., 6 th Floor, Boston, MA marty@cambridgeseman<cs.com (o) Cambridge Seman2cs Inc. All rights reserved. Company Confiden2al.

2 Introduc/on to Cambridge Seman/cs Anzo, a so8ware suite driven by Seman/c Web standards to execute Smart Data Solu/ons for diverse data from varied sources Company: Founded by senior members of IBM s Advanced Internet Technology Group So5ware: Our Anzo sosware suite is built on W3C Seman<c Web open data standards Currently 3 rd genera<on of the product in produc<on use Select Customer Presenta/ons & Interviews SearchCIO, Using Seman<c Technology to Turn Big Data into Smart Data, Interview with David Saul, CSO, State Street Semtech 2012: Athena: Using Seman<cs to Help Manage Staples Enterprise Service Bus, Presented by Staples Select Media Coverage on Cambridge Seman/cs & Seman/c Technology Strategic Finance cover story: Business Tool The Next Quantum Leap Data- Informed.com - Big Pharma Tracks Compe<<on with Seman<c Web Tools Informa<onWeek, Big Data + Seman<c Web: Love At First Terabyte? CMSWire, The Seman<c Web and the Modern Enterprise 2014 Cambridge Seman2cs Inc. All rights reserved. Company Confiden2al Page 2

3 Anzo Smart Data Integra/on Overview Anzo Smart Data Integra2on uses common, conceptual models with exis<ng ETL tools to increase the speed and decrease the cost of comple<ng high- quality, governed data integra<on projects by 10 <mes or more Cambridge Seman2cs Inc. All rights reserved. Company Confiden2al Page 3

4 Smart Enterprise Data Management Anzo Smart Data Integra<on is a prac<cal example of Smart Enterprise Data Management, a new, sensible paradigm for managing enterprise data at the conceptual level Cambridge Seman2cs Inc. All rights reserved. Company Confiden2al Page 4

5 The Data Integra/on Challenge Lead (SFA system) Source Systems S x T ETL Jobs Target Systems Customer 360 Quote (Quote system) Enterprise Warehouse Order (OMS system) Business Data Marts Contract (CMS system) Each ETL Project: Manually coded Requires source & target SMEs Many hand- offs Each Job Business Analyst Developers QA & Ops Define Mapping Requirements Code ETL Job Test & Deploy Compliance and Regulatory Repor<ng 2014 Cambridge Seman2cs Inc. All rights reserved. Company Confiden2al Page 5

6 The Common Model is Data Glue Source Systems Lead (SFA system) Quote (Quote system) Order (OMS system) Contract (CMS system) Common Model ( Data Glue ) Many common concepts across disparate systems Seman<c data science connects these common concepts Data is glued together by its underlying business meaning Poten<al to use industry standard models, e.g., FIBO Business Analysts and IT can use conceptual models to: Create data services Understand the data landscape Track data lineage Conduct downstream analy<cs 2014 Cambridge Seman2cs Inc. All rights reserved. Company Confiden2al Page 6

7 Anzo Smart Data Integra/on Lead (SFA system) Quote (Quote system) Source Systems S + T Maps Common Model ( Data Glue ) Target Systems Customer 360 Enterprise Warehouse Order (OMS system) Business Data Marts Contract (CMS system) Each Job Compliance and Regulatory Repor<ng Each ETL Job: Generated from map Only source SME required Hours, not months Business Analyst Map Source to Conceptual Model Anzo SDI Automa<cally Generate ETL 2014 Cambridge Seman2cs Inc. All rights reserved. Company Confiden2al Page 7

8 ASDI Use Case: Crea/ng a Data Lake Sources Common Conceptual Model Anzo provides a planorm wide common conceptual model Anzo Smart Data Integra<on Data Lake Self- service Analy<cs Self- service Data Extracts & Marts ASDI streamlines and automates data inges<on Anzo enables end- user self- service using models 2014 Cambridge Seman2cs Inc. All rights reserved. Company Confiden2al Page 8

9 Self- service Analy/cs via a Conceptually Described Data Lake Source 1 4 Map Source Schema to CCBM Generate P2P ETL Job Common Conceptual Model 3 Generate Conceptual to Physical Map Data Lake 2 5 Generate DW Schema Data Feed Catalog Register Feed in Catalog 6 Model Driven Consump<on Conceptually described data lake and Anzo SDI enable self- service: Analy/cs, Extracts and Data Marts 2014 Cambridge Seman2cs Inc. All rights reserved. Company Confiden2al Page 9

10 Anzo Smart Data Integra/on Capabili/es Map to/from conceptual models Automa<cally generate ETL jobs Create Data Marts and Extracts On- demand Support data lineage and governance 2014 Cambridge Seman2cs Inc. All rights reserved. Company Confiden2al Page 10

11 ASDI: Map to & from Conceptual Models ASDI Data Mapper Built for Business Analysts Uses an Excel- based interface to map systems to/from conceptual models Supports mappings: Physical- to- Conceptual Conceptual- to- Conceptual Physical- to- Physical 2014 Cambridge Seman2cs Inc. All rights reserved. Company Confiden2al Page 11

12 ASDI: Automa/cally Generate ETL Jobs Mappings are combined and reused to define projects Projects are automa<cally compiled into ETL jobs to run on Pentaho or Informa<ca Support for SSIS, DataStage, Ab Ini<o, Talend, and others to follow Generated jobs include automated quality checks and error handling 2014 Cambridge Seman2cs Inc. All rights reserved. Company Confiden2al Page 12

13 ASDI: Create Data Marts and Extracts On- demand Publish well- described data services into a shared, searchable data catalog Analysts use an itunes- like interface to select data elements needed for analysis Populate exis<ng or new data marts and extracts in minutes, as needed 2014 Cambridge Seman2cs Inc. All rights reserved. Company Confiden2al Page 13

14 ASDI: Support Data Lineage and Governance Automa<cally generated as a by- product of data integra<on projects Searched and browsed through governance dashboards Track and audit revisions of models and mappings Detect changes in the schemas of source databases 2014 Cambridge Seman2cs Inc. All rights reserved. Company Confiden2al Page 14

15 Anzo Smart Data Integra/on Architecture Conceptual Model Editor Data Mapper (Excel- based) Smart Data Integrator (web applica<on) includes: Project manager Schema manager Model manager Data feed manager Governance dashboards Anzo Smart EDM Server Conceptual Model Registry Schema & Sample Data Registry Mapping Registry ETL Compiler Data Source Registry Data Feeds Catalog Services: Sample data service Data feed persistence service Revision & audit service Access control service SQL CSV/TSV XML Proprietary ETL Engine SQL CSV/TSV XML Proprietary 2014 Cambridge Seman2cs Inc. All rights reserved. Company Confiden2al Page 15

16 ASDI User Roles Full User Defines projects and mappings Configures data sources & schemas Publishes projects to ETL tools Populates Data Catalog with Data Feeds Data Consumer Search and browse Data Catalog Creates on- demand data marts and extracts from Data Catalog Governance User Manage models Browse and search projects Browse and search data lineage Administrator Configures users and roles Configures dashboards and templates 2014 Cambridge Seman2cs Inc. All rights reserved. Company Confiden2al Page 16

17 Example Smart Data Integra/on Use Cases Customer Onboarding A large financial ins<tu<on offering data analy<cs as a service to its customers needed to streamline the mul<- month process of combining data from internal systems with data from a customer s own systems. Regulatory Repor/ng A large consumer lending organiza<on faces rapidly evolving regulatory repor<ng requirements that place an unsustainable burden on their compliance teams. They need a flexible, self- service Data Lake with interac<ve repor<ng capabili<es. Data Migra/on A large consul<ng company helping its customers migrate from mul<ple legacy systems to a new planorm looks to leverage their domain exper<se to accomplish high- quality migra<ons an order of magnitude faster than compe<tors Cambridge Seman2cs Inc. All rights reserved. Company Confiden2al Page 17

18 ASDI Lead Customer Value Projected value from a lead customer for a complex data integra<on project: Exis/ng Process 1 Business Analyst for 6 Months 4 Developers for 6 Months 3 Testers for 6 Months Total Cost: ~$1.5M New Anzo SDI Process 1 Business Analyst for 1 Month 1 Developer for 1 Month 1 Tester for 1 Month Total Cost: ~$100K Es<mated development cost reduced from $1.5M to ~ $100K Es<mated development <me reduced from over 6 months to ~1 month 2014 Cambridge Seman2cs Inc. All rights reserved. Company Confiden2al Page 18

19 Anzo Smart Data Integra/on Demo Source Systems Target Systems Holdings Conceptual Asset Model MySQL Security Master Oracle QA Scenario 1 Map Holdings database to Conceptual Asset Model Map Conceptual Asset Model to MySQL target database Publish and run Pentaho job Scenario 2/3 Add Oracle QA database as an addi<onal target Add Securi<es Master (SMF) as an addi<onal source Publish and run Pentaho jobs 2014 Cambridge Seman2cs Inc. All rights reserved. Company Confiden2al Page 19

20 ASDI Timeline & Roadmap Ongoing: Release of addi<onal ETL engine plug- ins July 2014: Restricted Beta program August 2014: Open Beta program Q3 2014: GA release Q4 2014: Version 2 Beta release Manage and track business and data requirements Manage and track data integra<on project issues Create project and ETL requirements documents Lifecycle work flows for projects, models, mappings Impact analysis Streamlined cloud deployment SDK 2014 Cambridge Seman2cs Inc. All rights reserved. Company Confiden2al Page 20

21 Other Anzo Products & Solu/ons Smart Data Analy<cs Rapid, flexible integra<on and analysis of structured and unstructured data Search, dashboards, search, visualiza<on, text analy<cs, Big Data connec<vity, spreadsheet integra<on, collabora<on, data cura<on Smart Enterprise Data Management Informa<on Landscape: holis<c view of data lineage across systems, applica<ons, projects, and people SOA/ESB Governance: manage service interfaces across an enterprise bus; support change control and impact analysis Search & Discovery Combine enterprise search, analy<cs, and symbolic compu<ng to deliver tailored answers to users searches 2014 Cambridge Seman2cs Inc. All rights reserved. Company Confiden2al Page 21