MDM Architecture Deep Dive: Implementing Real-time Bi-Directional Synchronization Ron Matusof, VP MDM Solutions Informatica

Transcription

1 MDM Architecture Deep Dive: Implementing Real-time Bi-Directional Synchronization Ron Matusof, VP MDM Solutions Informatica Dmitri Korablev, VP Strategy and Planning Informatica 1

2 Agenda Introduction Understanding Synchronization Requirements Real World Use Cases Best Practices Summary 2

3 Introduction 3

4 Who We Are? Dmitri Korablev VP, Strategy & Planning Almost 20 years designing, architecting and building enterprise solutions Favorite quote: Simple things should be simple and complex things should be possible. Alan Kay Ron Matusof VP, Solution Architecture Almost 30 years architecting and integrating complex systems Favorite quote: Nothing is as simple as it seems. Corollary to Murphy s Law 4

5 Why is Synchronization an issue? It may be complex, but it should be possible. and it is not as simple as it seems. Theme for Today s Presentation Make everything as simple as possible, but not simpler. Albert Einstein 5

6 Synchronization Design Tradeoffs Consistency Performance Coherence Correlation Accuracy Fidelity Quality Timeliness Throughput Latency 6

7 Bi-Di Synchronization Complex MDM Scenario 5 Legacy CIF Steward Business Users Policy DB 4 Read/Update Hub GSDN Integration B2B Transforms Warehouse 1 Marts 3 Call Center 2 Legacy Systems Federated Query 7

8 Case 1: Hub as System of Record Legacy CIF Policy DB Steward Business Users Warehouse 1 Marts Call Center 8

9 Case 1: Hub as System of Record Questions you should ask Consistency Policy Do I Synch or do I Access in the Hub? Performance Is Real-time Always Better? Quality Do I want a Batch Window? Solutions Look at Consumption Scenarios Legacy CIF Wareh ouse Marts Call Center Synthesize appropriate keys for downstream use Leverage Existing Interfaces and Transport Don t reinvent the infrastructure Understand the Full Supply Chain Propagation, Syndication, or Synchronization? 1 DB Dat a Ste war d Busi ness User s 9

10 Synchronizing Multiple Systems Party Name Home Office ACME Products Taos Spacely Space Sprockets Orbit City Dunder Mifflin Scranton Party Name Home Office ACME Products Taos Spacely Space Sprockets Orbit City Dunder Mifflin Scranton Party Name Home Office ACME Products Taos Spacely Space Sprockets Orbit City Dunder Mifflin Scranton Classical approach focuses on data latency No emphasis on data correlation Limited metrics for what is good enough 10

11 Synchronizing Multiple Systems Party Name ACME Products Spacely Space Sprockets Dunder Mifflin Home Office Flagstaff Orbit City Scranton Party Name ACME Products Spacely Space Sprockets Dunder Mifflin Party Name ACME Products Home Office Flagstaff Orbit City Scranton Home Office Taos Spacely Space Sprockets Orbit City Dunder Mifflin Scranton Latency causes discrepancy in targets At t = 1, System 1 is synchronized with Hub At t = 3, System 2 is synchronized with Hub Between t = 1 and t = 3, no downstream correlation 11

12 Bi-Di Synchronization Case 2: R/T Synch with an application Legacy CIF Policy DB Steward Business Users 2 Legacy Systems Federated Query 12

13 Case 2: R/T Synch with an application Questions you should ask Consistency How do I correlate with a other systems? Performance What do I do about feedback loops? Quality Can I catch DQ Issues at source? Solutions Develop Appropriate Logical Models Consider handling changes through abstraction & insulation Understand both direct and indirect feedback loops Use Delta Detection and/or CDC for loop suppression Integrate with the application business workflow Event Driven vs. Process Driven (or both) 13

14 Synchronizing Multiple Systems Party Name ACME Products Spacely Space Sprockets Dunder Mifflin Home Office Flagstaff Orbit City Scranton informatica data services Party Name ACME Products Spacely Space Sprockets Dunder Mifflin Party Name ACME Products Spacely Space Sprockets Dunder Mifflin Bi-Directional Synch causes race conditions Need to characterize the latency, throughput and correlation impacts and tolerances. Home Office El Paso Orbit City Scranton Home Office Taos Orbit City Scranton 14

15 Case 3: Synch On-Premise with Cloud Legacy CIF Policy DB Steward Business Users 3 15

16 Case 4: Global Synchronization Questions you should ask Consistency How do I transform the data? Performance How do I handle simultaneous updates? Quality How do I handle End-To-End Lineage? Solutions Use composite services to handle transformations Use Hub Objects with the B2B transform as a facade. Put complexity in the transform & not the data model Watch for the costs of serialization/deserialization and IO Integrate with the application business workflow Event Driven vs. Process Driven (or both) 16

17 Case 5: Global Distribution of Hubs 5 Legacy CIF Policy DB Steward Business Users 17

18 Case 5: Global Distribution of Hubs Questions you should ask Consistency How do I handle one hub being off-line? Performance How do I handle simultaneous updates? Quality How do I govern data across the Hubs? Solutions Develop connection agnostic processes Implement Queued Updates Design appropriate Replication and Failover strategy Use embedded or external CDC to generate change lists Develop integrated workflow across the Hubs Event Driven vs. Process Driven (or both) 18

19 Bi-Di Synchronization Complex MDM Scenario 5 Legacy CIF Steward Business Users Policy DB 4 Read/Update Hub GSDN Integration B2B Transforms Warehouse 1 Marts 3 Call Center 2 Legacy Systems Federated Query 19

20 Best Practices 20

21 Application Integration Design for Application Consumption Pattern(s) Analyze and understand the business use of the data Socialize the application use of Master Model for the Business Usage of the Master Consider using the application data model as a starting point Reconcile with the data models for contributing sources Optimize the model for application use cases Architect for application security/performance Review regulatory constraint for distributing and sharing data Develop methodology for creating test data and testing the app Architect to achieve application performance requirements 21

22 Bi-Directional Synchronization Optimize the Synchronization Architecture Determine Requirements Based on Consumption Patterns Evaluate the Cost/Benefit of overachieving on the SLA Architect solution to meet the optimal cost/benefit Analyze Round trips and Feedback Loops Consider loops that pass through more than two systems Use CDC/Delta detection to suppress loops Create Transactionality/Compensation Strategies Architect solutions for handling cross platform transactions Consider how to compensate for discrepancies 22

23 Global Distribution Architect to a Common Core Model Ensure that synchronized attributes are formatted identically Account for local differences only in non-synchronized fields Account for Differences in 3 rd party software Date/time Currency Unique processing approaches Make adding additional countries/regions easy Ensure the core data model considers future requirements 23

24 Summary Do not over-engineer the solution. Architect the Synchronization Design to meet the SLA. Design performance for the whole solution, not the individual parts. Synchronization has the potential for introducing additional bottlenecks. Design synchronization solution to maintain appropriate correlation between target systems. Perceived synchronization issues are typically related to correlation between downstream systems. 24

25 Marketplace Overview A Trusted, Open Ecosystem Virtual Marketplace for Integration Apps Solutions across all technology areas DI, DQ, MDM, Cloud, etc. Open Ecosystem Apps from Partners, ISVs, Consultants, and Developers Seal of Approval ensures App quality R More than 600 Apps, over 200 Free! M 15k visits per month, 2k downloads 25

26 Questions? 26