Web Services: A BI Perspective Philipose Mathew Rajarajeswari N Visalachi Rameshkumar

Transcription

1 Web Services: A BI Perspective Philipose Mathew Rajarajeswari N Visalachi Rameshkumar A Patni White Paper

2 COPYRIGHT Copyright Patni Computer Systems Ltd. All Rights Reserved. October 2004 Restricted Rights This document may not, in whole or in part, be copied photocopied, reproduced, translated, or reduced to any electronic medium or machine readable form without prior consent, in writing, from Patni Computer Systems Ltd. Information in this document is subject to change without notice and does not represent a commitment on the part of Patni. This document is provided "as is" without warranty of any kind including without limitation, any warranty of merchantability or fitness for a particular purpose. Further, Patni does not warrant, guarantee, or make any representations regarding the use, or the results of the use, of the written material in terms of correctness, accuracy, reliability, or otherwise. All other brand and product names are trademarks of their respective companies. Patni Computer Systems Limited India North America UK & Europe Japan Akruti, MIDC Cross Road No.21 Andheri (E), Mumbai Tel: Fax: Main Street Cambridge MA Tel: Fax: Vistacentre, 50 Salisbury Road Hounslow, Middlesex, UK. TW4 6JQ Tel: Fax: th floor, Aoyagi Building, Chuo , Nakano-ku, Tokyo Tel: Fax:

3 Table of Contents Overview... 4 What are Web Services?... 4 Value-adds of Web Services... 5 Web Services in BI Tools... 6 Database and Data Integration... 6 Reporting Application Metadata Management Process Management Implementing Web Services...11 The Road Ahead...12 References...12 About the Authors...13 About Patni...13 Copyright Patni Computer Systems Ltd., All rights reserved. 3

4 OVERVIEW Businesses have steadily deployed Business Intelligence (BI) applications to equip decision makers with better and more reliable data and the payoffs have been rewarding. By and large, BI provides an insight into the historical data but provides very less addressing of real time business needs. Also data integration from multiple sources on different platforms and technologies has been one of the main BI project hurdles. Web Services technology, which is fast addressing the Application Integration needs of enterprises, has the capability to help BI realize its potential as an enabler for proactive decision-making driven from accurate and real-time information from all corners of the enterprise. This paper discusses the Web Services technology, its applications and current offerings in BI. It also visits the BI tools stack to learn about the Web Service initiatives of BI products. Details of a Proof Of Concept carried out by the Patni BI CoE on the Web Services functionality provided by an ETL tool vendor, additionally forms part of this paper. WHAT ARE WEB SERVICES? Web Services are being deployed within businesses to enable better integration of operational processes. Business-to-Business (B2B) application integration over the extranet using web services is being explored. Most BI applications today are web based and web services technology promises to take BI beyond the extranets. Web Services can now enable multiple corporations to communicate BI information through standard program-toprogram protocols and centralized directory services, without any pre-established point-to-point connections required by BI extranets. Web Services enable BI application components to wrap their functionality and allow them to be rendered as a service to other applications using standard Webbased protocol. This would allow components in different languages and platforms to communicate with each other dynamically. Figure 1. BI Web Service Architecture Copyright Patni Computer Systems Ltd., All rights reserved. 4

5 It is thus possible for a program in Java running on UNIX platform to communicate with a program written in Microsoft code, which runs on Windows platform using Web Services methods. This functionality makes it possible for businesses to bridge the gaps between different applications and systems. They can be used to provide access to business processes, applications, BI cubes, reports, queries and data integration functions, databases etc. Web services are exposed using Simple Object Access Protocol (SOAP), -- an XML-based, industry standard communication mechanism, transportable over Internet protocols such as HTTP. By sending SOAP messages to a Web service, clients can invoke its business functionality. A Web Service, once invoked, returns XML documents, which themselves are packaged as a SOAP message. Web Services are described using the industry standard Web Services Description Language (WSDL), which is simply a set of XML tags. In order for people to find out about what web services are available, it is necessary to publish them in a registry called as Universal Description, Discovery and Integration (UDDI) registry. VALUE-ADDS OF WEB SERVICES Web Services have emerged as a powerful mechanism for data interchange and application Integration, both within and beyond the enterprise boundaries. The much-debated issue is how much can web services contribute in the BI processes, which spans from data integration to reporting and analysis. Two key technological offerings of Web Services make it stand out as the possible future directions of BI networks: Web Services happen over standardized Internet technology. This would enable multiple corporations to communicate BI information through common, centralized directory services, using standard program-to-program protocols. For companies that share BI information, BI web services promise to enable an elegant, fast, and easily repeatable integration experience. Almost all of the leading BI vendors are part of the Web Service Interoperability Consortium, which has been instrumental in defining the Web Service standards. This standardization would enable BI application components to communicate with each other easily and dynamically rather than wait for data integration to happen in order to communicate between systems. Web Services have a big say in moving towards Real time and Closed Loop BI solutions through effective Data Integration. Alert architectures in Real Time BI framework are increasingly being implemented using web services. As per a Data Warehousing Institute (TDWI) study, 19 percent of the companies surveyed indicated that web services were being used to route alerts to operational systems. Other common applications of web services in BI would be to integrate BI functionality into new serviceoriented applications that include the need to schedule and execute queries or reports, making available reports on portals, passing parameters to reports, etc. Area of Application Think of a purchase department getting notified on the poor quality of parts supplied by a vendor. How is this notification possible efficiently? As the shop floor database is fed with the defective part information, the web services of the database sends this information in real time to the data integration service of the data warehouse and enables a real time feed. The web services associated with the Data warehouse (DWH) has business logic embedded and kicks off the reporting service to notify the purchasing department users with detailed reports that convey that the vendor has been repeatedly providing poor quality parts. Consider all applications of an enterprise passing real time information using web services to the data warehouse and data integration and reporting tool web services being put to such business use! Web Services here would definitely make the Business Process Management initiatives simpler and faster. Copyright Patni Computer Systems Ltd., All rights reserved. 5

6 WEB SERVICES IN BI TOOLS Some of the BI vendors are now offering web service solutions that allow BI content to be integrated with other information, or fed into other applications. This creates exciting opportunities for building BI into the service-oriented architecture, and conversely for using the service-oriented architecture to short-circuit the labor of traditional BI solutions such as data warehousing. Today, most BI vendors offer Webservicesbased application programming interfaces (APIs) and software development kits (SDKs). The result is more-open access to BI functionality and capabilities for integration and extension into new applications based on Java 2 Platform, Enterprise Edition (J2EE) and.net. How ready are the BI players for web services and what are their offerings and their implementation successes? Web Services are being offered at the Database level, the Data Integration product level, Reporting & Analysis product level and also for Metadata Management and Process Management levels. DATABASE AND DATA INTEGRATION Usage of web services in the database will be a powerful addition to a developer's tool kit. Database vendors are also increasingly adding web services functionality in their products. Oracle RDBMS 10g allows one to define the functionality for a web service using PL/SQL packages, procedures, functions, and triggers. Accessing other web services from within the database is possible and this allows the database to perform tasks such as tracking, aggregating, and refreshing data generated by a remote web service. As databases are central to IT architectures, scenarios involving web services based on database activity can be imagined for nearly any business function. Area of Application Consider a bank, which has to upload data for its corporate clients. Using web services, it just needs to call the ETL web service of its clients through their own banking applications to upload the daily cash flow details. The ETL vendors provide APIs that can be invoked from applications based on Java 2 Platform, Enterprise Edition (J2EE) and.net. On invoking the web service, it triggers an ETL job to run on their client's ETL server and thereby update the client Data warehouse. F i Figure 2. ETL Web Services Copyright Patni Computer Systems Ltd., All rights reserved. 6

7 A sound Data Integration process would be the main success factor in a Web Services implementation and so the Web Services offerings provided by Data Integration product vendors are gradually picking up on this. Most of the ETL tool vendors provide features like Data Integration, Transformation Services and Web Services call outs during a data load. Informatica enables other applications to call its data loading WorkFlows as Web Services. Upon a particular Business Event, the WorkFlow can be invoked and the database loading is done. Informatica also allows its Transformation capability in Data Cleansing to be invoked as a web service. Informatica 7 offers two types of web services namely Web Services Provider and Web Services Consumer. Web Services Provider A Web Services Provider provides some of the PowerCenter Server functionality through service-oriented architecture. Web Services Hub forms the backbone of this architecture and it hosts the web services. The hub, which is installed on an application server, acts as a gateway processing SOAP requests from external clients. The requests are sent to the PowerCenter Server or Repository Server and their responses are sent back to the clients. Web Services Hub hosts Batch Web Services, Metadata Web Services and Real-time Web Services. Batch Web Services: Power Center Server and Work Flow Monitor services are web enabled and available as web services. Administration of the PowerCenter Server and running and monitoring of workflows can be performed using Batch Web services. Metadata Web services: They provide services to retrieve metadata from the repositories. Also provides authentication requests to authenticate users for accessing repositories. Real-time Web Services: Transformation logic is exposed as web services using real time web services. Service mappings are created which can receive a message from a web service client, transform it and update the PowerCenter repository. A service workflow is created after creating the mapping, sessions added and then published as a web service by the Web Services Hub. Web Services Consumer A Web Services Consumer allows Informatica mappings to access other web services providing business data and uses the data for transformation and loading. Business Object s Data Integrator ETL tool can publish any job as a callable web service and also call external web services from within its jobs using a built-in web services adapter. Copyright Patni Computer Systems Ltd., All rights reserved. 7

8 ETL Web Services Proof of Concept We have chosen Informatica PowerCenter 7.1 for our ETL Web Services Proof of Concept evaluation. Informatica Web Services Architecture The Web Services Hub 7.1 comes along with PowerCenter 7 and is a separate installable. The hub is installed on an application server (our hub install was on a Jboss Application server), which acts as the gateway for interacting with external client applications. A client web service application based on the Axis web services toolkit interacts with the hub to obtain Batch and Metadata web services. REAL-TIME SERVICES POWERCENTER SERVER WEB SERVICES CLIENT (Axis) BATCH SERVICES METADATA SERVICES REPOSITORY SERVER WEBSERVICES HUB (JBoss) Figure 3. Informatica Web Services PoC System Architecture Figure 4 below shows the web services hub console. The Batch Web Services functions available are listed in the figure. Figure 4. Informatica Web services Hub Copyright Patni Computer Systems Ltd., All rights reserved. 8

9 Accessing Web Services The client application calls the Batch Web Services to login to PowerCenter. The Workflows were accessed using the functions provided by Batch Web Services and then executed to load data. We were also able to fetch the WorkFlow logs. Without having any Informatica components installed and with just using HTTP, running and administering Workflows becomes a reality using Batch Web Services. Through our client application we connected to Metadata web services and were able to browse the PowerCenter repository metadata including Workflows and Tasks in the repository. For the Real time web services, we created service mappings & workflows and the workflows were published on the hub as services. The mappings created were one-way services, which receives updates from a web service client but does not send back a response. The client application accesses the hub and runs the published service workflows loading data on to the database tables. Figure 5. Snapshot of Application logs Copyright Patni Computer Systems Ltd., All rights reserved. 9

10 REPORTING APPLICATION Churning Business Intelligence out of large volumes of data would need more efficient and supportive reporting and analysis tools and web services promise hope. The reporting and analysis products have come up with SDKs, which cater to enabling of the reporting service as web services. Web Services opens up a lot of features on the BI reporting front. Web Service Requests on the reporting side could be to run a daily stock level query or execute a summary report or even to access a cube for analysis. The response data from executing the web services is then rendered in XML format and passed back to the calling application for display or further processing. BI OLAP and reporting tools can publish reports, cubes, and mining models as web services. Partners and sales people can invoke these web services across the globe in their applications. Conversely, consuming external web services by the BI tools to generate reports is gradually picking up. Business Objects reporting tool can consume an external web service and include this data in the reports generated. Cognos provides the capability to incorporate its BI content into any application or environment. Cognos Web Services provide means of delivering BI in whatever protocol or format is required for custom applications. Using explorable multidimensional reports can be delivered to PDAs, cell phones or to other wireless devices. Data can also be delivered in technologies like Macromedia Flash, which offer a good visual appeal to the user. METADATA MANAGEMENT As organizations grow and evolve, business processes and applications keep on changing making BI metadata management crucial. By managing metadata effectively, organizations are able to maintain enterprise-wide data integrity, deploy BI applications with faster turnaround times and in general manage enterprise information more efficiently. Web services promise to make distributed architecture based metadata management simpler and efficient. In this architecture, BI tools will expose their metadata as a web service that can be accessed in real time by any metadata management tool. This along with emerging standards like Common Warehouse Metamodel (CWM) can be a very powerful tool for enabling sharing of information among disparate BI tools. In fact in the future, whenever new dimensions, hierarchies, measures, business rules and reporting formats are defined or modified anywhere in the enterprise, web services will define them instantly in the metadata repository. PROCESS MANAGEMENT Application functionality and Business Rules can be encapsulated within a web service enabling better Business Process Management (BPM). The web service can initiate business processes on the occurrence of a particular event on a given point of time. A Gartner study points out that Business process management is complex and difficult, and enterprises look at web services to solve difficult issues in their interconnections. Gartner predicts the adoption speed to be five to ten years. Copyright Patni Computer Systems Ltd., All rights reserved. 10

11 Business Activity Monitoring (BAM) gets much support from web services as actions can be triggered to route information to business users or processes by a web service on a specific event. The Table below lists various BI Tool web services offerings: Database and Data Integration Level Oracle 10G Web services support is integrated with J2EE Server and allows developing web services with Java. The web service components automatically inherit the features of Oracle Application server Informatica Business Objects Data Integrator Power Center 7 offers: Web Services Provider Expose ETL functions as web services. Includes Batch, Metadata and Real-time web services Web Services Consumer Consume external web service for Source, Target or Transformation options Data Integrator 6.5 allows jobs to be published as callable web service and jobs to call external web services Reporting Application Level Cognos Cognos Web Services SDK which is part of Cognos Series 7 provide connections between Cognos software and multiple applications Business Objects MicroStrategy BusinessObjects, WebIntelligence and BusinessObjects Developer Suite - Cross-platform API information can be accessed and formatted in XML for use in other systems; XML can be used as an information source for creating reports within BusinessObjects MicroStrategy Web Universal and MicroStrategy SDK uses XML natively for all API communications, allowing all MicroStrategy BI functionality to be exposed through Web services IMPLEMENTING WEB SERVICES BI requires data integration across the enterprise and this is a complex process. Service Oriented Architecture (SOA) eliminates the headaches of protocol and platforms and applications integrate seamlessly. Web services, which are based on SOA, provides the much needed solution for application integration. Businesses looking at incorporating web services in their BI platform should initially limit the scope to applications within the enterprise. Data integration from legacy applications is one of the pain areas in a BI framework and it could be taken up as one of the processes to start with in deploying a web service. In other words, this could serve as a Proof Of Concept (PoC) too. Web services provided by databases and ETL services should be tried out in this integration work. This initial deployment would provide fair amount of guidance to future full-fledged web services enabled within and outside the enterprise network. Performance, data quality and data security issues are evaluated from the PoC and incorporated in the later phase deployments. Copyright Patni Computer Systems Ltd., All rights reserved. 11

12 THE ROAD AHEAD Gartner predicts that most systems will adopt web services interfaces to embed BI functionality into the new service-oriented applications by However, till 2007, the primary users of BI web services will not be traditional BI end users, but will be system integrators and application developers seeking to embed basic BI functionality into new business applications. Industry adoption of BI web services will be dependent on the acceptance of web services technology. Web Services Security Standards are still evolving and for businesses to provide and consume web services, securing their data would be one of the main concerns. But the potential of Service Oriented Architecture is amazing and the reality of making BI available easily in the applications of an enterprise without bothering about the complicated integration involved would make businesses adopt web services quickly. Definitely enterprises will be sharing BI with their customers, suppliers and the rest of the value chain using the web service driven BI networks and it promises to be easier, faster and in real time. REFERENCES 1. Web Services for BI Applications Have Yet to Be Exploited Gartner, April Hype Cycle for Web Services, 2003 Gartner, Techniques for Integrating BI into the Enterprise Mike Ferguson, BusinessIntelligence.com 4. Web Services in Oracle Products Oracle Technology Network 5. Informatica PowerCenter 7 Documentation Copyright Patni Computer Systems Ltd., All rights reserved. 12

13 ABOUT THE AUTHORS Philipose Mathew is a BI technical specialist who has been deploying BI solutions to Customers for nearly three years. He is part of the Technology Focus Group of Patni Business Intelligence COE and evaluates new BI technologies and products. Philip brings with him his prior experiences in developing and maintaining business applications in Leasing, Quality and Manufacturing verticals over multiple technologies. Rajarajeswari. N and Visalachi Rameshkumar are involved in developing BI applications for Customers across the globe and they are currently into developing BI applications as part of the Proof of Concept initiatives for evaluating new products and services. ABOUT PATNI Patni Computer Systems Limited is a global IT Services provider servicing Global 2000 clients in the Manufacturing, Insurance, Banking & Financial Services, Telecom, Retail, and Energy & Utilities, Logistics & Transportation, Media & Entertainment industries. With an employee strength of over 9,000; multiple offshore development facilities across eight cities; and 24 international offices across the Americas, Europe and Asia-Pacific; Patni has registered revenues in excess of US $250 million for the year Patni's technology focus spans e-business solutions, enterprise applications, embedded technology solutions and enterprise systems management. Its service offerings include application development and reengineering, application management and business process outsourcing. Committed to quality, Patni adds value to its client's businesses through well-established and structured methodologies, tools and techniques. Patni is an ISO 9001:2000 certified and SEI-CMMi Level 5 organization, assessed enterprise wide at P- CMM Level 3. In keeping with its focus on continuous process improvements, Patni adopts Six Sigma practices as an integral part of its quality and process frameworks. Patni is one of the Top 6 Indian-based companies who operate optimized global delivery models for customers, fronted by a strong local consulting capability. As industry leaders, Patni introduced Offshore Development Centers, and pioneered 'follow the sun' development and support frameworks. Patni s Business Intelligence Center of Excellence Patni has been delivering Business Intelligence (BI) and Data Warehousing (DW) solutions to Fortune 1000 global organizations over the last six years. Our service offerings range from Consulting Services, Customized BI/DW Development to Maintenance. Our solution has a foundation of deep domain knowledge, technological expertise, proven implementation frameworks and effective knowledge management. We have formed alliances to enable faster implementations and fully exploit the product features. These comprehensive solutions have delivered tangible benefits to our customers across industry verticals. Copyright Patni Computer Systems Ltd., All rights reserved. 13