Global Data Integration with Autonomous Mobile Agents. White Paper

Transcription

1 Global Data Integration with Autonomous Mobile Agents White Paper June 2002

2 Contents Executive Summary... 1 The Business Problem... 2 The Global IDs Solution... 5 Global IDs Technology... 8 Company Overview Contact Information... 12

3 Executive Summary "Information is being generated at such a rapid rate that the amount created this year and next will exceed all the previous information produced throughout human history." How Much Information? Hal Varian and Peter Lyman. UC Berkeley School of Information Management and Systems Business Information Integration Most large corporations do not stand a chance of keeping pace with the onslaught of data and information coming from both inside and outside the organization. Organizations desperately need a way to manage and structure their data to gain maximum value from it. Global IDs is the first Business Integration Software Company to specialize in global data integration using autonomous agents. Our software products utilize mobile agents to collect, aggregate and integrate data from diverse computer systems. Global IDs Data Integration Server can help you glue together the data that resides in your systems, by creating a virtual integration layer across existing databases. We can achieve this on a global scale, without disrupting existing business operations. Using our technology, companies can 1. Gain transparency into their operational activities around the globe 2. Use this information to optimize business performance 3. Create competitive advantage through the better use of information. 1

4 The Business Problem Data Integration is Key Companies require a holistic view of their enterprise data, before they can employ advanced analytics and datadriven decision making. To create competitive advantage, companies must be capable of turning data into insight, and be capable of extracting business value from this insight. A simple example; Most companies are capable of looking at their sales and profitability information at a summarized level, but very few have the ability to look at the same information at a detai level across their whole organization. The opaqueness of a diverse computer environment prevents analysis of business performance 1. At a granular level ( by product and customer) 2. Across the whole organization, at all locations around the world 3. At any point in time ( current or historical ) Business Performance Optimization This lack of transparency results in embedded inefficiencies in both the demand and the supply chain, that are nearly impossible to root out. In the worstcase scenario, it leads to catastrophic situations like the Firestone tire defect problem. In the bestcase scenario, it leads to impressive financial results as evidenced by Walmart s climb to the top of the Fortune500. So how do companies solve the problem created by opaque information environments? By supporting Business Intelligence / Data Warehouse Initiatives Data integration, which is at the heart of these efforts, involves the identification, collection, pooling and joining of diverse data for the purpose of business analysis and decisionmaking. Enterprise Data Integration allows 1. Measurement of business process metrics at a detailed level across the organization 2. A detailed identification of process inefficiencies through Key Performance Indicator (KPI) variance. 3. A prioritized list of actionable improvements that lead to tangible business results ( cost reductions/avoidance or revenue generation ) 2

5 Companies that have embarked on such data warehousing initiatives have generally achieved impressive ROI s, if successful. Unfortunately, the risks of failure for such initiatives are also high, especially for large, enterprise integration efforts. Why are the risks high? Because the data landscape of most large companies is very complex. Enterprise Data Integration is nontrivial Given that most companies have a large number of databases distributed at many locations around the world, integration can be a nontrivial task. It is only by combining the information from these systems that the enterprise can realize the full value of the data they contain. The primary technical obstacles to largescale global data integration are: Heterogenity Heterogeneity is the degree of differentiation in the various data sources. 1. Environment Heterogenity Differences in hardware platforms, operating system environments, network protocols, and different database management software 2. Query Heterogenity refers to different ways of querying the data due to different query languages, different query capabilities, and error handling. Complexity 3. Data Complexity Complexity due to the number of tables (thousands), number of schemas, customization, errors etc. 4. Metadata Complexity Differences in data models, data types, schemas, semantics, standards and language. Inflexibility 5. Lack of openness Inclusion of new data sources into the integrated data environment cannot be accomodated, without extensive manual involvement. 6. Lack of Dynamic Adaptability Most data warehouse environments are created on the basis of static data models, that cannot change over time. Modifications of existing data sources, can corrupt the integrity of the data environment, unless there manual oversight of all source systems. Volume 7. Data Volume is increasing exponentially To further complicate the matter, companies are getting inundated with data every day, as the data content of the Internet grows. Other obstacles to largescale global data integration include: 8. Prohibitive costs Departmental data integration efforts typically cost $13 M. The cost of enterprise or global integration efforts, typically an order of magnitude higher, making such unaffordable by most companies. 9. Organizational Resistance It should be noted the organizational resistance can often be the greatest obstacle to integration efforts, since the results of such efforts can potentially affect the power wielded by those who control access to the information. 3

6 Companies need Automated Data Integration Solutions To overcome these barriers, and become learning organizations, there is a deep need for software that can simplify the problem of global business data integration. The following list reflects the desires of many IT managers at Fortune 500 companies. Automated Integration Most integration tasks are manual, requiring significant human involvement. Complex, enterprise integration projects require armies of people internal staff and external consultants. If integration tasks could be automated, there would be significant benefits. Reduced Implementation Cost Reduced Maintenance Costs Reduced Dependence on External Expertise Easy Access to Enterprise Information Most organizations are organized in productcentric ways. This organizational model tends to create vertical silos where information from one part of the company does not crossover easily to another part of the company. Globalization and increased competition have forced companies to migrate to customercentric models, where access to enterprise information is key. If there was easy access to enterprise information, there would be a: Reduced need to cross political barriers to access information Greater Visibility to Business Performance Metrics across entire organization Easier Migration to Customer Centric Model Flexible, Open Solutions Most integration solutions are rigid, closed and tend to resist change. If software was dynamic and open the benefits would be: Reduced Maintence Improved Performance over time Global IDs Data Integration Solutions allow organizations to achieve these goals, through the use of autonomous mobile agent technologies. 4

7 The Global IDs Solution Integration with Intelligent Mobile Agents Global IDs is the first technology company to specialize in global data integration using intelligent mobile agents. Our solutions use software agents to collect, aggregate and integrate data from diverse computer systems, that are geographically distributed. Our solution creates a virtual integration layer that ties together the data and metadata from disparate data sources on a global basis. Our solutions give companies the ability to handle large scale data integration projects. We enable companies to: 1. Integrate data at a enterprise or global level 2. Provide informationondemand from across the enterprise 3. Automate the cleansing and validation of corporate data. 4. Address the complexity of global data ( different languages / standards ) 5. Automate the stewardship of corporate data 6. Provide a scalable integration environment for bringing in data from outside the organization ( from suppliers, customers, 3 rd party data providers etc) While our solutions mimic the functionality of large federated databases, the use of mobile agents gives our solutions three distinct advantages. Low Cost We can reduce the cost of a largescale data integration effort significantly (by an estimated 60%70%) because of Agent Automation Our intelligent agents can perform most of the integration tasks that could previously be done only by experts (e.g. consultants) High Scalability Since our integration tasks are largely automated, we are agnostic about the size or complexity of the integration domain Complexity Our software agents use semantic classification algorithms originating from computationallinguistics literature to structure complex metadata, allowing automation of various integration tasks. 5 I

8 Scalability Our software agents are not impacted by the number of computer systems in the integration domain. Our software agents use autonomic reasoning for their integration activities, and that perspective, there is no difference between a single system, 10 systems or a 100 systems. Dynamic Adaptatibility Awareness of Change Our agents are cognizant of change in both data and metadata at the time when it occurs. This the first step towards complex dynamic behavior of the agent environment. Evolvability Our technology permits both the application logic and the integration schema to evolve over time. The following table outlines the broad data access functionality that is contained in our solution ( Note that all the features described below are not available currently ) Feature / Functionality Heterogenous Data Access 1. Hardware Platforms 2. Operating Systems 3. DBMS Platforms 4. Networks Heterogenous Data Sources 1. Structured Data 2. Unstructured Data Flexibility 1. Extensibility 2. Openness Description Agents can access data across a wide technology landscape Servers from IBM, Sun, HP, Compaq etc. AIX, Solaris, HP/UX, Windows, Linux etc. running Java VM Oracle, DB2, SQLServer etc. (DBMS is accessible via JDBC) TCPIP Supported Networks Agents access a majority of digital data/information types OLTP, OLAP Documents and Content Agents evolve with new functionality Add new data sources dynamically Interoperable via published API In addition, our solutions have the following functionality The Ability to Discover, Store and Monitor Metadata To search and locate metadata from across the enterprise, we use AgentBased Metadata Discovery Our intelligent agents can scan the entire network for metadata sources, including data dictionaries, metadata repositories and document based metadata. AgentBased Metadata Storage Our mobile agents can transport the metadata and store it in a common metadata repository. AgentBased Metadata Monitoring Our applications is cognizant of change in the metadata, in realtime. This allows us to dynamically adapt to changing metadata. The Ability to Discover, Store and Monitor Data Agents are used to discover data sources from across the enterprise, and transport the data, ondemand, to a central data store. AgentBased Data Discovery Our intelligent agents can scan the entire network to locate sources of structured data ( in DBMS systems ) and unstructured data ( in file systems or document management systems). 6

9 AgentBased Data Storage Our mobile agents can transport data ondemand and store it in a common metabase. AgentBased Data Monitoring Agents are capable of monitoring any data across the enterprise, at the table, column or field level. The Ability to detect, classify and integrate Identity data Agents are used to detect the ids that are used for all major business dimensions (customer, product, supplier, partner, employee, shareholder, location, currency etc) AgentBased ID Discovery Agents can scan and detect all the IDs that are used across enterprise databases. AgentBased Classification Using semantic classification methods, we can categorize the ids into the corresponding enterprise dimensions. ID Integration Using our patented globalids technology, we can integrate the ids across all important business dimensions. For example, in the financial services industry, where mergers and acquisitions are common, there is often the requirement to integrate the customer information of the merged / acquired companies. Our software can be used to automatically create a group of common customer ids from the customer ids that are being used by hundreds of databases in the two companies. The Ability to import and export Metadata Our agents can be used to interchange enterprise metadata with other applications via a published API Import Our software agents extract metadata directly from the data itself, through inference techniques. In the future, we will support metadata import from metadata repositories. Export The application can export metadata to other applications using standard formats (e.g. CWMI). To summarize Global IDs Data Integration Solutions enable companies to automate the integration of data across the enterprise, leading to better visibility into operational activities and supporting business performance optimization initiatives. 7

10 Global IDs Technology Intelligent Mobile Agent Technology Global IDs uses mobile agent technology for implementing the data integration functionality that it provides. Agents are distributed software components that are able to migrate between computer systems and carry out complex tasks. Global IDs uses entire systems of such agents to perform the tasks associated with data integration activities. Our software agents are created using objectoriented and componentbased development techniques, and are written in 100% pure Java. Our Solutions employ an architecture with 3 distinct layers User Interface The user interface is the mechanism via which the user interacts with the application. Our architecture supports multiple interfaces, including a. SWING interface ( Java FrontEnd ) b. Web Browser interface c. Instant Messaging (IM) Interface and Simple Messaging Services (SMS) Interface will be supported in the future d. Any application interface that supports XML communication Interface Broker The interface broker handles all communication between the Global IDs application and the user interface. The Interface Broker uses agents to translate the communication between the User Interface into a compliant XML message that is understood by the application. Data Integration Server The majority of the integration functionality resides in these components. The primary components are: a. Agents for discovery, collection, aggregation and integration of data and metadata b. Agents for administration and maintenance c. A relational database management environment to store data/metadata d. Third party applications 8

11 The following schematics shows the Global IDs Server architecture User Interface XML Interface Broker Agents Interfaces Applications RDBMS Agent Habitat Global ID s uses the following agents to perform specific integration activities Agents Data Discovery Agents A general class of agents that can a. Discover all the systems on the network b. Install an agent habitat on remote systems ( having obtained security permissions) c. Discover databases, applications and unstructured content on remote systems Data and Metadata Collection Agents These agents, after obtaining readonly access to the database environment, can retrieve any data or metadata from a. OLTP databases b. OLAP databases c. File Systems d. Document Management Systems ( future support) Data and Metadata Monitoring Agents These agents monitor any changes in the data or metadata environment of the source systems. Data and Metadata Analysis Agents These agents monitor analyze the data and metadata for patterns that may be use to during the integration phase. At present, we have the following analysis agents. 9

12 a. ID detection / classification agent b. ID integration agent Benefits Our distinctive approach to data integration, allows our customers to reap benefits from both the business perspective and the technology perspective. Business Perspective: 1. Optimize business performance through the rigorous use of KPI s Improved datadriven decision making Valuechain Analysis and Optimization Organizational performance measurement 2. Reduce costs of business integration efforts Significant reduction in data integration costs Retire redundant applications and reporting systems 3. Endusers Productivity Transparent access to business information from across the enterprise Less time spent looking for useful information Technology Perspective: 1. For data warehouse developers: Automation of many data integration activities Access to a enterprise level metadata repository Ability to meet enterprise business reporting / analysis needs 2. For application builders and system integrators: Adherence to Standards (Java, J2EE, XML) Improved fulfillment of customer needs Compression of delivery schedules Straightforward maintenance procedures, 10

13 Company Overview Global IDs was founded in 2001, by executives from IBM, with backgrounds in sales, marketing and technology. The company was created specifically to address the problem of large scale information integration. Our mission is to develop agentbased integration solutions that can accomplish integration across hundreds of systems. In our first six months, we have created the world s first data integration software that employs intelligent mobile agents for largescale integration tasks. Our solutions make it possible for large global companies to address the integration tasks that are required to improve business performance analysis. Our management team brings experience from leading systems integration companies like Accenture and KPMG. The management also brings existing relationships with many Global 1000 clients, ensuring a receptive market for the product portfolio. 11

14 Contact Information For further details on Global IDs products and services, please contact us via , mail, telephone or fax at: 18 Legion Court Rockaway, NJ USA Phone: +1 (973) Fax: +1 (973)