iway Roadmap: 2011 and Beyond Dave Watson SVP, iway Software

Transcription

1 iway Roadmap: 2011 and Beyond Dave Watson SVP, iway Software

2 iway Software Products DataMigrator Core Integration Server iway Service Manager Information Management/Data Governance B2B Gateway Managed File Transfer Complex Event Processing/Business Rules Real-Time Data Replication Archiving/Information Lifecycle Management Adapters DataMigrator 7.7

3 iway Service Manager 6.1 Development iway Integration Tools Deployment iway Integration Applications Upgraded standards Xquery, JSON Management and Monitoring SNMP APIs Map Reduce parallel processing Hadoop CFR since March 2011 GA since June 2011 Download from onbuilders.com/ More details available on What s New Page Detailed features 6.1 and on /agenda ers/433

4 iway Service Manager August 2011 New Features/Improvements: Specification compliant iwxpath Copy/Rename Providers Dynamic Adapter Targets Debugging aid commands Scheduler Simplified DSIG Kerberos for nhttp Scripting support XML Archive Hadoop Full list available on What s New page. iway Service Manager and m/events/summit/summit11/agen da m/events/seminars/developers/4 33

5 BAM and TPM Near term Converge 3 BAM-TPM versions into 1 common version Performance/Scalability/Data Volumes Functionality (Resubmit, Message Management) Database (DB2, Oracle, Microsoft SQL Server) Additional functionality All queries in stored procedures Support user defined columns Predicate specification included user defined columns Use of authentication providers for log-on Use of ISM principal/acl security for BAM users Longer Term BSON database logging (10X faster than RDBMS) JSON query processing (5X faster than RDBMS)

6 Map Reduce Style Processing in iway Service Manager

7 What is Map Reduce? Map reduce is a concept made popular by Google, in which an action known as a map is applied to each item in a list. The outcome of each map can then be accumulated or changed in a process known as a reduce. The work of each map and reduce can be distributed across a cluster for parallel processing. Map reduce style processing implementations: Hadoop mongodb Amazon Elastic Google

8 Big Data Node Chunk Manager Map & Reduce Result Node Chunk Node Chunk

9 Big Integration Node Map & Reduce Connector Manager Result Node Combine Result Map & Reduce Connector Node Map & Reduce Connector

10 MapReduce Architecture in iway Service Manager Parallel Execution Items Map Optional Reduce Document Combine Result

11 Parallel Control Agents Parallel Count Agent Executes a flow N number of times in parallel passing a copy of the input document to each execution. An index is passed to each instance of the flow to allow it to manage its purposes. Parallel XML Split Agent Executes a flow for each node in the input matching an XPATH statement passing a matching node to each execution. The passed document passed to the instance of the flow helps it manage its purposes.

12 Simple Example There are a number of files in the file system that have the following structure: You want to read each of these files and compute the sum of the <val> elements.

13 Without the Parallel Control Agents Using a list of file locations you could read each document and calculate the values in sequence and accumulate the resulting documents.

14 With Parallel Control Agents Parent Flow with call to Parallel Control Agent. Flow executed in parallel for each item in the list.

15 Resulting Document

16 More Complex Uses Database Federation Attach Point Request Parallel Control Flow Accumulation Result Work Distribution Flow Gateway Node Gateway Node Gateway Node

17 Hadoop and iway Service Manager

18 Hadoop Developed by Yahoo Incorporates Google Map Reduce Designed for specific uses processing large data sets with parallel operations Parallel Operations Huge Sequential Immutable Files Support is available for ism Called Hadoop Distributed File System

19 HDFS What is HDFS? Hadoop s file system Designed for redundancy of data Files are immutable Usually stores huge files Gigabyte, Terabyte and Petabyte is common 64MB blocks for fast read Fast sequential access is not a design criteria Hive adds a SQL processing layer on HDFS

20 iway and HDFS iway offers sequential operations File iterator Each record passes down the flow Parallel File Iterator Records are processed in parallel in the flow Emit Agent Writes a file: from document, stream or external file

21 iway and Hadoop Iterator In standard Process Flow Each record in the HDFS is presented in sequence iway Service Manager processes the record Map Reduce can be used

22 Hadoop Iterator HDFS Each record passes through flow Synchronous access to file One at a time

23 Hadoop Parallel Iterator HDFS Each record passes through flow Several at a time Synchronous access to file Be careful of memory in accumulation

24 iway and Hadoop Iterator Hadoop Service Manager Pflow packages as a Hadoop map/reducer Input (listener) meets Hadoop interface Output (reduce) emitter meets Hadoop interface Each record passes through the flow The flow provides map/reduce functionality Login in pflows == NO 3GL programming

25 Hadoop Service Manager HDFS Flow distributes to Hadoop Flow acts as a Mapper/Reducer Lets users employ ism as a Hadoop Accessor Locality of reference Single stream operation Installed HSM into Hadoop Packaged HSMs

26 Enterprise Information Management and Data Governance

27 What s new in Data Quality Center Enhanced XML support SOAP Integration (Web Services) Multi-Language Support (Japanese, German, Spanish, ) Deploys to WebFOCUS and DataMigrator Server Data Quality Portal New Issue Types (Linking and Synchronization Issues) iway Service Manager Integration for real-time processing Master Data Center Web Services Integration Industry Model Healthcare Banking Government

28 What s new in iway Data Profiler 3.1 iway Data Profiler Distributed Architecture Oracle Support Multi-Language Support (Japanese, German) Porting to Mobile Devices ipad Tablet Android Tablet

29

30

31

32

33

34

35

36

37 What s new in DQC Web Service connectivity DQC can now connect to external Web services with multi-record output Matching engine The matching process handles up to 50 quadrillions (5*10 16 ) of records. Previous versions could only handle up to 4 billions (4*10 9 ) of records. Improved performance of matching process for data with complex linkage patterns Address solutions Improved US and Canadian Address patterns, leading to more precise address identification Database support Additional support for Teradata and Sybase IQ databases Support for MS Excel 95 files Distribution DQC for Windows OS ships with both 32-bit and 64-bit executables. The proper executable is chosen depending on processor and OS architecture. Expected GA date June 2011

38 What s new in DQ Portal, MDC DQ Portal New issue type for cross-record inconsistency Workflow can now trigger iway Service Manager processes based on what transition was executed Support for custom issue metadata attributes Configurable issue list screen columns Importer can run in different synchronization modes (upsert and full synchronization) MDC Integrated execution workflow engine with scheduler Better execution monitoring for long-running batch import/export processes Migration/upgrade of MDC project metadata

39 V8 DQC Web services UI for web service testing Support for JSON-based services JMS transport for web services UI for Soap Call step configuration using a supplied WSDL Database support Pre-defined database commands for custom tasks select where..., create table/index/view, truncate, drop, analyze Faster DB schema investigation Monitoring Web console monitoring overhaul JMX monitoring Performance reports in visual plan monitoring Plan file editor Analysis of attribute usage and dependency/impact analysis Remote plan file launching

40 V8 MDC End-user UI Data browsing Relationship visualization Integrated issue tracking Engine improvements Mixed workload batch and online access at the same time High availability Load distribution batch updates and read-only online requests Metadata and lookup file services Reference data management UI Maintenance of reference data used across the organization Cross-system translation tables Organizational structure Lookup files used in plan files Defining and maintaining hierarchies in reference data Version control and workflow

41 iway CEP Enable: Complex Event Processing Business Rules Engine

42 iway Enable: Conceptual Overview Integration Capture Tap Creation Configuration Control Rules Script CEP BAM Discovery Event Cloud Term Cloud Roll Ups Drill Downs Search/Filters Design InfoSet Feeds Resource Creation Configuration Web Application Portlets Enable API Copyright 2009, Information Builders. Slide 42

43 What s new in Enable 1.3 Enable Dashboard and Portlets Out-of-the-box dashboard tool to create ad hoc dashboards Ability to integrate Enable portlets and 3 rd party web 2.0 gadgets (Google, Twitter, etc) Broader portlet platform and browser support

44 What s new in Enable 1.3 Enable Login Manager Customizable set of web components for user authentication Enable Business Rules Engine Usability Updated GUI Business Rules editor Rules and Rule sets engine updates Rules Optimization Execute rules more efficiently If first rule in a rule set is true then stop evaluating

45 What s new in Enable 1.3 New Actions Framework Expanded action types , twitter, salesforece chatter, process, social, IFL services RESTful style servies support Ability to call any ism process flow via RESTful style servcies Ability to call Enable via RESTful style servcies

46 What s new in Enable 1.3 Server Side Scripting using Java API MetaData Modeling Adapter (MDMA) Used to expose adapter metadata to enterprise Enabled for Salesforce

47 What s new in Enable 1.3 Social Service Updates Enabling Salesforce Chatter Enabling Twitter

48 Chatter on Android with iway Feed/WebFOCUS Report

49 WebFOCUS report on Android Phone

50 Chatter on ipad with iway Feed/WebFOCUS Report

51 iway Adapters

52 iway Adapters Support latest IBM, Oracle and SAP middleware (application servers, ESBs ) Support Vmware Spring Integration Framework Support IWAF deployment WebFOCUS Reporting Server Upgraded SWIFT, HL7 and HIPAA standards Upgraded SAP, Siebel, RDBMS,.NET, CICS and JAM adapters New adapters for Healthcare EMR and Core Banking Applications

53 iway Roadmap: 2011 and Beyond Dave Watson SVP, iway Software