Value Realization at Johnson Controls using SAP HANA smart data integration Steve Carpenter Johnson Controls Ryan Champlin - SAP

Transcription

1 Value Realization at Johnson Controls using SAP HANA smart data integration Steve Carpenter Johnson Controls Ryan Champlin - SAP

2 Agenda What is SAP HANA smart data integration? Use cases for Smart Data Integration at Johnson Controls Value realized at Johnson Controls using Smart Data Integration

3 SAP HANA smart data integration and smart data quality Simplified landscape, accelerated performance and an open framework Transactions Traditional SAP HANA ETL Mix of Potentially Stale and Current Data SAP HANA In Memory Platform Current Data SDA Aggregate HANA EIM Services SDQ SDI Multiple Data Sources Replication On Premise and Cloud sources of Data Separate ETL, Replication and SDA Op RDBMS Other Sources Integrated ETL, Replication and SDA

4 SAP HANA SDI/SDQ Overview SDI: smart data integration Batch extraction and target-based CDC (HANA) Automated delta extraction -> real-time push of changes Data distribution to target (Hana->Target) Advanced data transformation capabilities Custom source/target development through adapter SDK (Java) Fully integrated HANA development experience (Studio/WebIDE, Extends SDA, Cockpit, Lifecycle Management) SDQ: smart data quality Global address cleansing that has support for over 240 countries Geocode enrichment for countries around the globe Matching and deduplication Data quality metadata to give insight and clarity into data quality changes applies

5 SAP HANA smart data integration Data Replication Overview WebIDE based replication editor Choose one or many tables Initial load only or with ongoing real time replication Available Configuration Target Columns Customize the target table to your needs Filter Limit what s being replicated Replication Behavior Change the default replication behavior o Identify changed records for consuming applications e.g. SAP BW, SAP Data Services, etc. o Create a history table Logical Partitions Decrease initial load times

6 SAP HANA smart data integration Data Integration Batch data integration HANA Studio or HANA Web-based Development Workbench design-time UI Design simple or complex data flows Available Transformations Basic SQL Aggregation, Filter, Join, Sort and Union Advanced SQL Case, Lookup, Pivot and Unpivot Data Lifecycle Date Generation, History Preserving Map Operation, Row Generation and Table Comparison Data Quality Cleanse, Match and Geocode Code Execution AFL, Procedure and R Script

7 SAP HANA smart data integration Real-time push to replicate sources Sources with change data capture (CDC) capability: HANA, Teradata, ECC, DB2, MS SQL Server, Oracle Transaction based: Reading DBMS transaction logs and pushing committed changes on table to be replicated Trigger-based: for each table to be replicated, DB trigger queues record changes to be pushed Guaranteed delivery for real-time push replication stream is disrupted or HANA temporarily goes down. Provides batch pull mode for all types of sources Supported SDI adapters (sources): SAP ECC, DB2, Oracle, MS SQL Server, File, OData, Teradata, HANA, Sybase ASE Distributes data in real-time or batch Supported targets from HANA: Sybase ASE, File, HANA, Teradata, DB2, Oracle, MS SQL Server

8 Product Differentiation for Data Integration SAP LT Replication Server SAP Sybase Rep Server SAP HANA smart data integration Real-time replication SAP Sources Primarily Impact to source systems NW Stack SAP Data Services Real-time replication Non-SAP Sources No transformations CEP/SDS Real time replication (log-based) Batch SAP and Non-SAP Sources Data transformations Data quality Natively built in HANA Batch Data transformation Data quality SAP and Non-SAP Sources Real-time Event streams

9 Agenda What is SAP HANA smart data integration? Use cases for Smart Data Integration at Johnson Controls Value realized at Johnson Controls using Smart Data Integration

10

11 Wherever they live, work or travel, people all over the world are touched by Johnson Controls products and services every day.

12 Johnson Controls Overview* (*prior to 2015 results) $42.8 billion Johnson Controls achieved record full-year revenues in ,000 employees serving customers in REVENUE $BILLIONS countries

13 Johnson Controls IT Landscape

14 Enterprise Data COE Charter Identify & Standardize the use and governance of information in support of the overall global business strategy Enterprise Master Data Hubs (Key Hubs: Supplier, Customer, Direct Materials, Indirect Materials, Employees, Finance, Reference) SAP MDG Develop and maintain controls on data quality, interoperability, and sources to effectively manage corporate risk High Velocity Data Cleansing & Migration SAP IS, DS, BOA Accelerators Decommissioning & Archiving System Data SAP ILM Ongoing Data Quality Operations SAP IS Collaborate w/business to define maintenance processes and teams SAP MDG Develop methods to ensure consistent application and use of analytics Enterprise Data Warehouse (Key Tenets: Performance, Enterprise Model, Data Federation, Agility, Reactive & Predictive, Compliance Forensic Discovery) SAP HANA, SAP HANA EIM, Hadoop, Legacy Data Warehouses

15 EIM Logical Architecture

16 EIM Conversion Flow & Risks SAP SAP Copy EBS MFGPRO EBS Copy BODS SAP Ref OTHERS FLAT FILES HDFS HIVE LAND HIVE TRXFRM TRANSFORM LOAD SAP Target MAPICS SQOOP BOA LAWSON Problem Statement Master Data is typically cutover 4-6 weeks prior to business go-live (no concerns) Transactional Tables that are large in nature (~100MM+ records) require special handling during cutover Goals Extract data in a timely manner from the source Analyze and POC alternatives to current techniques and tools (SDI, SLT, SRS, extractors, ABAP Data Flows, Golden Gate) targeting large data tables

17 SDI POC Overview Business-Relevant Tables (1016) Master, Transaction, Configuration Tables Views (1); Transparent (925), Pool (86), Cluster (4) Tables ECC Instances Size Analysis Automotive Experience 16TB Building Efficiency 8TB Power Solutions NA 4TB Power Solutions EU 4TB Log Switching Analysis (below) PS Hourly _00_02_04_06_08_10_12_14_16_18_20_ BE Hourly ECC Instances Table Analysis Record Count AE BE PSNA PSEU < 50, ,000 to < 100, ,000 to < 500, ,000 to < 1 mil mil to < 10 mil mil to < 100 mil mil to 500 mil mil to 1 bil >= 1 bil Total Tables > 50, > 1 Million AE Hourly _00_02_04_06_08_10_12_14_16_18_20_22

18 SDI POC Results Performance Testing on pre-production environment (application and database tiers co-located) Baseline Tests x2 (B1, B2) No tables being replicated to HANA No supplemental logging enabled 94 & 47 tables Test with SDI (T94, T47) Replication to HANA Supplemental logging enabled 100 users running for 100m (5m ramp-up, 90m steady state, and 5m ramp-down) Executed using 4 Load Generators located in Holland (MI, USA), Burscheid (DE) x2, and Monterrey (MX) Log switch rate of ~21 mins, log size ~660 mb Test Response Time CPU Memory IO Resp. Time (ms/op) IO/sec Rows Transactions B1 3.7% 0.7% 0.01% ,824 37,289 B2 17.7% 6.4% 0.01% ,459 37,718 T47 8.8% 1.7% 0.01% 1.85* ,851 38,036 T94 Lowest Lowest Lowest 1.62* ,213 38,154 *Storage Response <8 ms/op is desired

19 Revised Data Ingestion Framework Incremental Extraction of all data from all business-relevant tables from each source system Hadoop (HDFS/HIVE) Data Warehouse Data Migrations Master Data Hubs Data Scientists Lowest Cost Storage SDI-Based Framework for Oracle, HANA, SQL Server Sqoop-Based Framework for non-traditional databases System Current Tool Future Tool Oracle DBs (SAP, Oracle EBS) MS SQL Server DBs (iscala, FourthShift, Concorde, etc...) Progress DBs (MFGPRO/QAD, Symix) AS400 DBs (MAPICS, Lawson, MACPAC) Hadoop Sqoop, SAP Data Services SAP SDI (December) Hadoop Sqoop SAP SDI (2016) Hadoop Sqoop, SAP Data Services Hadoop Sqoop Hadoop Sqoop Hadoop Sqoop IIDP Update BOA with IDP results Ingestion Process Start Get Metadata Create Hive Table Ingest Update Hive Table End Update HBASE with Ingestion stats Reconciliation Update HBASE with reconciliation results

20 Summary SAP HANA Smart Data Integration HANA native data integration and data quality Loading/Distributing data into/out of HANA Consumable data quality services for applications Value Realization at Johnson Controls Bottlenecks around data ingestion removed, real-time Slight impact to ERP Systems but benefits > costs Next Steps: SDI implementation (December for SAP ECC) Next Next Steps: SDI Transformation Evaluation (2016)

21 Summary Steve Carpenter Director, Enterprise Data Center of Excellence Ryan Champlin Sr. Director of Engineering, SAP Enterprise Information Management & HANA

22 Further Readings and Links Roadmaps Customer Facing Roadmap Site (and how to find EIM Products) NOTE: Go to: Database and Technology -> Enterprise Information Management

23 Further Readings and Links Product Availability Matrix (platforms, connectivity, etc.) SAP HANA Smart Data Integration: E

24 Further Readings and Links Documentation EIM Products: SAP HANA smart data integration & smart data quality:

25 Further Readings and Links BEST EIM Resource Links: SAP HANA smart data integration and smart data quality: Overview: NOTE: See many additional videos on specific capabilities on the right on YouTube after selecting the above link. A few that might be interesting: Real-time replication from ECC: Eh5 Data Cleansing: W0Eh5 Geocoding: ex=31