Business Intelligence The ESB and Microsoft BI The role of the Enterprise Service Bus in Microsoft s BI Framework Gijsbert Gijs in t Veld CTO, BizTalk Server MVP gijs.intveld@motion10.com
About motion10
About me CTO at motion10 20+ years experience in integration architecture, design Working with BizTalk Server since beginning (2001) Architected and designed parts of BizTalk Server 4-time BizTalk Server MVP Member of the BizTalk Server Virtual Technical Specialist team Technical Editor of the new-to-be-released BizTalk book: BizTalk Server 2009 R2 Unleashed (July 2010)
Agenda What is Business Intelligence (BI)? Microsoft s BI Framework explained What is an Enterprise Service Bus (ESB)? How does an ESB fit in the BI Framework? What is the value add of the ESB to BI? Summary
What is Business Intelligence? Gartner describe it as A broad stack of applications and technologies for gathering, storing, analyzing, sharing and providing access to data to help enterprise users make better business decisions...the process of extracting raw data from operational business applications and databases and analyzing the data to make long term, high level business decisions.
What is Business Intelligence? Improving organizations by providing business insights to all employees leading to better, faster, more relevant decisions Delivered through a familiar environment Integrated into a business productivity infrastructure Built on a trusted & extensible platform
Three levels of BI Strategic Tactical Operational Analysis Data Access Collaboration 24x7 BI Enterprise ETL Embedded New Form Factors Business Impact Timeliness Ease of Use High-Level View # of Decisions
Microsoft s BI Framework Complete & integrated BI and Performance Management offering Leverages Microsoft Office investments and knowledge Enterprise grade and affordable
BI Topology
BI Platform: SQL Server Integrate Store Analyze Report
BI Layers Source: Gartner (april 2006)
What is an Enterprise Service Bus? An ESB is usually the center of a Service Oriented Architecture (SOA) SOA is the architecture style, ESB is the plumbing It is responsible for: Business process orchestration Protocol adaptation Transformation Routing It creates an abstraction layer that provides much flexibility It can be used for A2A and B2B integration In a SOA it can host your composite services It is basically a broker
SOA revisited Consume User Directed Compose User Experience and Interaction People using Content, BI, Collaboration and Communication SOA as mechanism to interact Standards based Interoperability Compose Business Process Integration, Automation and Optimization, Information Integration SOA as mechanism to transact Expose Existing Systems
Enterprise Service Bus visualized 1. Transform my message 2. Determine which endpoint I need 3. Route my message 4. Route the response to a second service 5. Return the final result to me Transform my message Routing Transform Service Process Orchestration On Ramp Off Ramp Service Consumers Protocol Adaptation Pub/Sub Service End Point Resolution Service Providers Resolve a service end point address for me
Typical ESB scenario: Composition
How does an ESB fit in the BI stack? DELIVERY SharePoint Server Reports Dashboards Excel Workbooks Analytic Views Scorecards Plans PERFORMANCE MANAGEMENT PLATFORM Excel Services PerformancePoint Services BI PLATFORM SQL Server Reporting Services SQL Server Analysis Services SQL Server DBMS SQL Server Integration Services BizTalk Server ESB
ESB value add 1: Operational BI ESB is all about (near) real-time messaging in a SOA Fast and current information Process-driven BI Especially in situations where the ESB is responsible for service composition and business process orchestration, the ESB is the ideal location to record business level information on transactions: BAM (Business Activity Monitoring)
Operational BI - Topology PORTAL BIZTALK SERVER ESB SHAREPOINT B2B REPORTING SERVICES ANALYSIS SERVICES LINE OF BUSINESS SYSTEMS
BAM lifecycle Business Process Modeling and Documentation Visual design of electronic forms Visual design of collaborative applications Real-time tracking of end-to-end business process performance Management visibility into business process performance Real-time process optimization Development of Business Processes Develop new composite processes from reuse of existing systems Deployment of highly distributed processes Intuitive end-user task interaction through Office Interaction with collaborative processes Effective management and control of distributed processes
BAM - How it works
Examples Operational BI 1. Analyst designs view template 2. BizTalk captures process events 3. SQL Server aggregates events to cubes 4. Excel requests SQL data and renders 5. Accessible through SharePoint portal BAM is real-time Business Intelligence with a business process context
ESB value add 2: Service Provider Because the ESB is the center of your SOA, it can provide all kinds of services to the applications connected to it The ESB can provide the following functionalities as services to the Data Integration Layer in your BI stack: Transformation services Validation services Access to legacy applications
Transformation Services Building these transformation maps is done by using the graphical, drag-n-drop mapper These graphical maps are compiled to the XSLT standard BizTalk Server executes these XSLT maps in realtime during processing of transactions The BizTalk Server ESB provides the transformation feature as a service that can be consumed by ESB users such as the Data Integration Layer in your BI stack
Validation Services The ESB provides services to validate transactions against: Schema Business Rules
BI s data integration layer Unstructured data Legacy data: Binary files Application database Extensive Connectivity High speed connectors for: Oracle, Teradata and SAP BW Standards based support Excel, XML, flat files, binary files Connection to Applications via BizTalk Server ESB MS Message Queues OLTP Change Tables Change Data Capture Transparently captures changes Real-time Integration DW
Access to legacy applications MQ MSMQ WSE HTTP SMTP File FTP POP3 SOAP SWIFT HL7 RosettaNet HIPAA SQL WCF Adapter SharePoint Pluggable Adapters In the Box PeopleSoft JD Edwards OneWorld XE JD Edwards Enterprise1 Oracle ODBC Siebel TIBCO Rendezvous TIBCO EMS SAP EDI/AS2 X12 and EDIFACT support Drummond Interoperability Certification Host Applications IBM mainframe zseries (CICS and IMS) Midrange iseries (AS/400) IBM DB2 Mainframe DB2 for z/os Midrange DB2/400 DB2 Universal Database for open platforms (AIX, Linux, Solaris, and Windows) Host Files
The use of canonical data models The introduction of a canonical data model as part of the implementation of an ESB is a good design principle It gives the following advantages to any ESB implementation, but especially to the ESB in a BI stack: Uniform description of what your data means Canonical schemas, if well designed, are self-documenting Easier to understand what the data means and how to map it to datawarehouses to create meaningful information that really serves analysis Data validation at the canonical level is much more efficient. Business Rules apply to information you understand well. Switching back-end applications does not impact your BI stack and data integration layer configuration
When to choose what for integration? SSIS when: Batch oriented, scheduled, bulk data loads Very high performance integration needed Integration is purely needed for BI Data is relational Data needs to be pulled ESB when: Near real-time data integration / low latency operational reporting Changes to data in datawarehouse are incremental SSIS does not provide access to the needed data source out-ofthe-box and BizTalk Server does Data structures are mixed and data coming from mixed platforms Data is mainly pushed
Summary There are 3 levels of Business Intelligence: Operational, Tactical and Strategic ESB s first value add to BI is providing input to Operational BI An important layer in the BI Stack is the Data Integration Layer, this is where the data is gathered, transformed and cleaned ESB s second value add to BI is providing and brokering reliable services for data transformation, validation and access to legacy applications BizTalk Server is the ESB that is part of Microsoft s Application Platform and perfectly ties in with the BI Stack
Contact Details Come to our Booth Register to win a Flip Camera Gijs in t Veld Gijs.intveld@motion10.com www.indafield.com/biztalknotes www.motion10.com