Information Integration Market trends and directions and IBM s Xperanto initiative

Transcription

1 Information Integration Market trends and directions and IBM s Xperanto initiative Nelson M. Mattos IBM Software Group Distinguished Engineer Director, Information Integration Integrating diverse business information across and beyond the enterprise Application s DB Client Web Service Message Workflow Information Information Integration Integrator

2 Agenda Market Trends Market opportunity Market Shifts Proof points Information Integration Approaches Domain characteristics Technologies Standards IBM Xperanto Project Vision Technologies Customer scenarios Summary

3 Business Integration: Trends Market Business opportunity for integration middleware is growing significantly Customer Diversity of solutions driving need for an integration platform for unification of disparate business processes, applications, and information sources E-business deployment requiring smooth integration with existing infrastructure Technology Major vendors building integration platforms Application providers moving to integration infrastructure by leveraging installed base Database vendors extending their current portfolio ETL, Portal, Content Mngmt, Messaging vendors all moving into the integration space Standards being defined and implementations being delivered XML enabling information exchange Web Services driving customer interest in time to value reduction Data Federation becoming a reality

4 40% of I/T budgets are spent on integration... I/T Spend Goal: reduce the customer integration burden... IBM Business Apps & Technology 25% Infrastructure & Integration 75%...40% is focused strictly on Integration...to help focus the I/T spend back on technology Source: Gartner... implies that customers will spend $9 B on integration in 2002

5 Integration Middleware Market ' CGR ($) Total Integration 24% 14.4 B 15 Application Server 47% 4.7 B ($ Billions) 10 Business Process 41% 1.0 B Integration Message Based 15% 3.5 B 5 User Interface 6% 1.7 B Source: IDC Data Integration 22% 3.5 B

6 The Data Deluge... Common Database Sizes Source: IBM Almaden Research Transactions 1s TB X GB 100s GB - 1 TB 1-20 GBs 100s MB 100s KB Warehouses Marts 100s TB 1s TB 100X 100X The world produces 250MB of information every year for every man, woman and child on earth. Mobile Pervasive 85% of digitized information unstructured. 10s GB 1s GB 1,000X 10,000X

7 Integration Market - Shifts Large IT investment in integration software and services Market categories are morphing, expanding, overlapping, blurring Current state is confusing Broad ranging customer requirements Significant overlap between products Market boundaries somewhat artificial: artifacts of recent technology innovations vs.. true buying intent Niche orientation makes benefits of one investment difficult to share with another Anticipate consolidation and market re-definition Application/process integration platform: EAI, business processes, application servers Data integration platform: Federation, ETML, hybrid XML-relational DBMS, structured and unstructured content Anticipate a major shift in the data management industry From data management systems that manage real data to those managing virtual data Federated/Virtual database mngmt systems

8 Customer Scenario Information Integration Challenge Enhance effectiveness of telesales Solution Record sales conversation and store it in text format Combine data and text mining techniques over sales transaction data and sales conversation data. Value Improve sales offerings and strategies by analyzing patterns that would not be visible had the structured and unstructured data remained isolated. Technical Requirements Information Integration Integration of structured and unstructured data Application of mining techniques over the integrated data

9 Customer Scenario Process Integration Challenge Become the partner of choice for deliveries resulting from B2B and B2C transactions. Solution Extract shipping data directly from their customers orders and shipping databases. Quickly return delivery status information to customers Use XML and WebServices for process integration and messaging across enterprises. Value Improved customer satisfaction by shortening the delivery process Increase cash flow and that of its customers Reduce operational cost. Technical Requirements Business Process Integration (workflow, business models) Application integration (messaging) Information Integration Integration/access to structured and unstructured data including application data, e.g., SAP Storage of XML messages)

10 What is Enterprise Information Integration? Enterprise information integration refers to a category of middleware which lets applications access data as though it were in a single database, whether or not it is. It enables the integration of structured and unstructured data to provide real-time read and write access, to transform data for business analysis and data interchange, and to manage data placement for performance, currency, and availability.

11 Comparing Different Approaches Application connectivity and Process Integration communicates events between applications A new customer event must be shared across sales, marketing, and service application domains A product sale must be shared across customer, finance, and inventory applications Information Integration is about capturing, integrating, and sharing the state of the business as embodied in its stored data An enterprise balance sheet must consolidate financials across multiple divisions A customer s portfolio must integrate banking accounts, brokerage accounts, stock holdings, and insurance policies across an enterprise.

12 Information Integration Approaches Centralizing data for consolidated access Data warehouses Operational data stores Production applications Typically managed by ETL (Extract, Transform, and Load) or replication technologies Integrated access to distributed sources Real time data, e.g., stock quotes Mixed format data, e.g., customer ODS plus related contract documents Represents an emerging category of technology EII (Enterprise Information Integration)

13 Information Integration Key Technologies Information integration should enable integrated, real-time access to traditional and emerging data sources, transformation of information to meet the needs of business analysis, and manage data placement for performance, currency, and availability. Heterogeneous Information Unified View of All Content Structured, semi-structured, and unstructured Legacy, packaged,and Web Real-time and point-in-time Data Placement Management Federation, Caching, Replication, ETML, Web Crawling Meta data management Rich Analytics Search and Crawl OLAP, Mining, Categorization, Summarization, Statistics,... XML/RDBMS store Traditional client or Web service SQL or XQuery Relational or XML native storage Infrastructure to complement application and process integration Common development environment Messaging, workflow

14 Information Integration Key Values Information integration technology enables integrated, real-time access to traditional and emerging data sources, transforms information to meet the needs of business analysis, and manages data placement for performance, currency, and availability. Heterogeneous Information Unified View of All Content Information correlation System-managed optimization Simplified programming and maintenance Data Placement Management High speed access Point-in-time access High availability Rich Analytics Comprehensive search and analysis Relevant results, Real-time analysis XML/RDBMS store Persistent, embeddable XML store Standards-based High performance Scalability, availability, integrity Infrastructure to complement application and process integration One-stop shopping Integrated environment Open and extensible

15 IBM Xperanto Project Integrating diverse business information across and beyond the enterprise Applications DB Client Web Service Message Workflow WebSphere Information Integration Data federation ƒread/write access across diverse data and content sources ƒdatabase flavor (SQL) ƒcontent flavor (CM API) ƒxml flavor (XML API) Data placement ƒcaching and replication over heterogeneous information Data transformation ƒsql, XML ƒadvanced search and mining ƒmetadata management Part of a complete business integration solution ƒbest data store for XML data - DB2 ƒwebsphere business integration ƒopen platform based on industry standards

16 Federation Technology Single database image UDFs Transparency Appears to be one source Heterogeneity Integrates data from diverse sources Structured, XML, unstructured, messages, Web, Function and Extensibility SQL/XQuery plus backend specific functions Autonomy Non-disruptive to data sources Performance Distributed optimization

17 Web Services Technology Provider support Access resources from Web clients Consumer support Extend reach of database to nontraditional, real-time data sources UDDI Catalog Web services for public of private use XML Registry Manage XML artifacts such as XML schemas, style sheets, DTDs

18 Extended Search: Unstructured Access IBM Extended Search ƒbrokered search architecture for searching thousands of existing data sources ƒresults are aggregated, ranked, and returned in a single hit list ƒeasily embeddable into any application ƒlotus databases, document systems, full text indexes, , directories, WWW, syndicated content, relational, file systems Combined with federation ƒgenerated search arguments ƒsophisticated ordering ƒxml document generation DB2 NetSearch Extender ƒover federated backend, relational sources Institute for Search and Text Analysis ƒmajor investment area Agent Relational Databases Web Content Mgmt WebSphere Servlet Agent Broker Content Management Search Customer name+2002 Agent Taxonomy & Indices Document Repositories Agent Internet Web Crawler

19 Multi-tiered Caching Improve query performance and availability Administrator defines Materialized Query Table ƒprecomputed or frequently used values ƒany data from the federated system ƒapplication indicates ability to use cache ƒimplicit or explicit use Developer or administrator enable cache use Informix ƒif enabled, reads are handled from the cache, writes passed through to the source ƒif not, reads and writes passed through to source Cache refresh managed: ƒmanually ƒby replication Cached in relational or XML format Flexible caching topologies supported A B DB2 C D A B C DB2 Cache D E F text WebSphere Oracle E F

20 XML Technology Object-relational implementation Store, retrieve, compose, decompose, validate, extract, transform Storage options Store intact Store as a collection of columns CLIENT Data management client Customer client application SQL(X) With XML Extender UDFs and SPs Relational Interface SERVER DB2 Server XML Extender Relational Storage DAD/DADX Hybrid XML-relational store SQL or XQuery XML specific storage, query, indexing, privileges, transformation, schema, interfaces, search DB2 engine core attributes: scalability, availability, reliability, manageability CLIENT Data management client Customer client application SQL(X) XQuery Relational Interface XML Interface SERVER DB2 Server Relational Storage XML Storage

21 Replication Applications Warehouse and ODS applications Consolidation and distribution Application integration Availability management Heterogeneous replication DB2, Oracle, Sybase, Informix, Microsoft, Teradata Table-based or transaction-consistent Point-in-time or continuous operation Embedded transformation Key investments Transaction-based replication over MQ Publish changes over MQ Information Integration CAPTURE Transactions in Memory Microsoft UOW TABLE BASE CD TABLE DB2 Microsoft staging tables MQ Send Queue transactions on a queue

22 Metadata Management Categorize & Group Discover Find, correlate, collaborate, modify, iterate Import interesting XML xsd, xslt, wsdl, into XR Design Extract obvious relationships Register dependencies Create new xsd, xslt, wsdl, views, maps, etc in XR Export artifacts and register dependencies lock,update, version, group Search, relate Administer, detect, notify, report Deploy

23 Administration Tooling Strategy Discover Data sources: databases, search engines, content providers, service providers Data: Tables, XML files, spread sheets, queues, users, etc. Transformations (XR) XSLT, XSD, etc Integrate/Filter Design Federate,ETL,Cache, Migrate/Replicate Schemas Transform (Clio?) Transform one source to another Create unified view of multiple sources Configure Define Topology Roll out Administer Monitor Deploy

24 MQ Integration MQSeries Networks II MQListener DB Workload MQ UDF / SPs Send Receive Read Publish Subscribe MQSeries Networks DB Clients

25 Asynchronous Access Workflow process II Client A Client B Async Listener Flow Wrapper Other External Data Client access based on wsdl: Interface describes µflow signature Target described DB2 server instance Binding gives transport protocol

26 Access to Data App SQL Client XML Client Query Metadata to get summary info & url for access 2 Meta Data Catalog stored as relational w/ XML view or XML w/ relational view 2. Select data of interest and use url to get real data Tightly Federated Data Relational Data XML Data Loosely Federated Data External Data Sources SQL Wrappers Messages Messages Web Services Virtual Tables Portlets Portlets Flows Flows Virtual Documents Connectors Connectors Other RDBs Other RDBs Files Files XML Wrappers... CM Stores

27 Programming Model Integration Facilitate access and exploitation from all key programming environments Leverage information integration capabilities from WebSphere and MQSeries environments Develop seamless XML query, handling, storage DB2/390 JSPs WebSphere Application Servlets EJBs WebSphere Servier Application Web Servier Services Flows EJBs Web Services MQSeries Information Integration Layer Analytic Functions JDBC/SQLJ/MQSeries/.. SQL Query Stored Procedures Cache XML Query User Defined Functions Text Search Message Queues Customer Orders Informix DB2/ Unix Oracle Silver Customers Gold Customers Suppliers Inventory Contracts IMS CM V8 Web Services Current Exchange Rates, Commodity Prices, etc.

28 Xperanto Demo Raise awareness of the information integration challenges in the industry Highlight key technologies in addressing those challenges XML - lingua franca for interchange Web services extends the reach of the database to non-traditional data Federation key for data volume, rate of change, and diversity Text search core technology for XML-based world Refresh mid November

29 Summary Information integration is an emerging technology category with tremendous market opportunity. It enables integration of data and content sources for Real-time read and write access Transformation for business analysis and data interchange Data placement for performance, currency, and availability It is expected that Information Integration will cause a significant shift in the data management industry. IBM is well positioned to lead the market with its Xperanto Initiative.

30 Thank you! Integrating diverse business information across and beyond the enterprise Applications DB Client Web Service Message Workflow Information Integrator Integration Xperanto