Emerging Technologies Shaping the Future of Data Warehouses & Business Intelligence

Transcription

1 Emerging Technologies Shaping the Future of Data Warehouses & Business Intelligence Service Oriented Architecture SOA and Web Services John O Brien President and Executive Architect Zukeran Technologies 1

2 SOA Topics Introduction Business Drivers Putting it all together and definitions SOA and Web Services Data Warehousing in a SOA World 2

3 Evolutionary Not Revolutionary Mainframe Era Core business functions become automated Client/Server Era Departmental applications grow Internet Era Network oriented applications Thin browser (standards based) clients Enterprise Services Era Applications become services within network oriented applications 3

4 Setting the Stage Parallel development of platforms and toolkits OS, Languages, IDE, Databases Enterprise Applications Integration (EAI) drives automation Platform components Application and Web Servers Communication standards evolve HTML, HTTP, XML from SGML 4

5 SOA Definitions It is not a product but a philosophy that guides the evolution of IT architecture along a path toward increased business value An architectural platform for rapidly adapting and solving business problems The ability to change the IT infrastructure faster than the competitors to support evolving strategy and to adapt to business conditions will lead to significant competitive advantage 5

6 Gartner s 5 Levels of Architectural Maturity 1. Chaos: Technology choices within the enterprise or across partners are made in isolation 2. Reactive: Businesses work to remove the proliferation of technologies through joint efforts and design approaches 3. Proactive: Each of the major participants in a multi-enterprise architecture effort actively manages architecture even though there is still a lack of coordination across all participants 4. Service: Some consistent architectural processes are established across enterprise partners, and architectural work aligns with key business objectives 5. Value: Architectures are fully leveraged, aligned, and oriented to provide competitive business leverage 6

7 Business Value from SOA Move from monolithic, custom-coded applications to choreographed, scripted components Agility and flexibility to reconfigure business functions and try new process models Move from tightly coupled to loosely coupled systems to deal with inevitable change 7

8 Key Concepts of SOA Abstraction to Business Process Modularity Encourages Reuse Connect through a Service Service is a description of a specific interface to a component that performs some function Loose Coupling Expose as little as possible of the service implementation Coarse Granularity Asynchronous Communication 8

9 What SOA Brings Application silos Tightly coupled Object oriented Known implementation Function oriented Built to Last Prolonged dev cycles Orchestrated Loosely coupled Message oriented Abstract Process oriented Built to Change Incremental build/deploy 9

10 SOA Technology Layers User Interface Process Services Displays information and manages interaction with the user Orchestrates the flow of work & keeps track of what s been done Services that perform specific chunks of work Objects Collection of data & closely related services Persistence Storage of data 10

11 Fabric SOA Enterprise Service Bus Components Service Client Service Client Service Client Service Client WSDL/SOAP Interface XML Interface HTTP Infrastructure WSDL/SOAP Interface EAI Infrastructure XML Interface WSDL/SOAP WSDL/SOAP XML Interface Interface Interface Various proprietary and standard interface or adaptors Services provided from Legacy, Application EAI Middleware 11

12 Enterprise Service Bus SOA Enterprise Service Bus Components Service Client Service Client Service Client Service Client WSDL/SOAP Interface XML Interface WSDL/SOAP Interface Service Gateway Protocol Translation, Service Address Mapping, Security, Logging XML Interface WSDL/SOAP WSDL/SOAP XML Interface Interface Interface Various proprietary and standard interface or adaptors Services provided from Legacy, Application EAI Middleware 12

13 Web Services and SOA It should be understood that Web services does not equal service-oriented architecture. Web services are a collection of technologies that let you build programming solutions for message oriented and application integration platforms Web services help facilitate the exposure of the service as ubiquitously as possible 13

14 Interoperability Web Service standardized specifications are making software from different vendors work XML: A powerful and universal way of representing any data, making integrations simple to implement SOAP: Uses XML as messages to define a universal Web service request, making process integration simple WSDL: Specifies all information needed for integration, making universal application assembly tools possible UDDI: A special Web Service which allows users and applications to locate Web Services 14

15 Web Service Technologies BPEL4WS Business Processes Reliable Messaging Security Coordination Transactions Quality of Service WSDL, UDDI, Inspection Description SOAP (Logical Messaging) XML, Encoding Transports Other protocols Other services Messaging Transport 15

16 WS-I Basic Profile 1.0 XML 1.0 (second edition) XML Schema Part 1: Structures XML Schema Part 2: Data Types TLS 1.0 Transport Layer Security X.509 Digital Certificate Specification HTTP 1.1 HTTP over TLS HTTP State Management Mechanism SSL

17 XML 1.0 XML (Extensible Markup Language) The beauty of HTML, derived from SGML XML is the key to interoperability and is the basis for all Web Services. With XML, we can exchange data between any application regardless of: Operating system Programming language Hardware platform Delivery device Software vendors 17

18 SOAP - Simple Object Access Protocol Request and response messages Request invokes a method on a remote object Response returns result of running the method SOAP specification defines an envelope Envelope wraps the message itself Message is a different vocabulary Namespace prefix is used to distinguish the two parts 18

19 WSDL - Web Service Description Language From W3.org WSDL 1.2 Primer: a machine-processable document that describes the expectations and functionality of a particular Web Service, so that a potential client can read the description and understand how to correctly interact with the service. Service provider defines the WSDL document for a service Service requestor uses the WSDL document to guide implementation of message creation 19

20 WSDL Data Types WSDL document has complete message details for data types of request/response content Data types defined by XML Schema Spec 1.0 Rich type definitions, simple to complex Complete type definitions allow for automatic validation of XML content 20

21 What is UDDI? Universal Description, Discovery and Integration (UDDI) How does the requestor get the WSDL? Standards based specification for service description and discovery Publish Service Registry Discover Service Provider Bind Service Requestor 21

22 Granularity A single exposed service may be comprised of n other existing services to provide the necessary business function Portal Takes Order Create Order Return Code Ship Date You re more interested in the result than how they perform their function Some atomic services could be exposed too Validate Customer Validate Order Check Inventory Check Pricing Calculate Delivery Start Workflow Cust Product Inventry WFM 22

23 Versioning Common issue in the design of distributed systems Not designed into the Web Services architecture, and organizations have to define best practices Changes that are backward compatible Addition of operations Addition of XML schema types Nothing in the WSDL conveys change Changes that are NOT backward compatible Removing operations Renaming operations Changing the structure of a complex type 23

24 Data Warehousing in a SOA World Data Acquisitions Real-Time Processing SOA Bus Architecture Rise of Shared Data Domains 24

25 Impact on Acquisitions Should you subscribe to the ESB for acquisitions? Yes & No. Be sure that you represent transactional data as just that. It may not be persisted the same in all places. 25

26 Impact on Acquisitions Object relational or Agile database modeling techniques allow for dynamic attribution rather than Entity-Relationship modeling. Dynamic attribute models store null-valued pairs that allow the persistence of any data structure in a table Be sure that metadata models store the key to correlating data back into dynamic length columns of rows 26

27 Impact on Acquisitions More data stores that map to Service Namespaces than the current application/database paradigm Services will persist data in local data stores that preserve object models In some cases, enterprise data domains will finally serve as the centralized subject areas in operational systems, but not likely 27

28 Real-Time Enabled Yes! But real-time transaction data enabled if you subscribe to the bus Transaction management will lead to real-time operations and alerts such as: Web Service calls waiting in queue too long High transaction loads indicate high database activity for data warehouse batch acquisitions Web Service calls failing Inspect the payload order over $1M just sent! 28

29 DW Part of SOA Bus If SOA is a philosophy and not a technology, what would the data warehouse look like under this approach? Bus architecture Granular services Interoperable across platforms Receive and send service calls Clients and portals utilizes services to retrieve data Key Point: SOA is an architecture that is transactional in nature. Is the data warehouse transactional or bulk? 29

30 Data Services Besides transactional data services (CRUD), bulk process services are created also DW services could be developed and called getcustomer passing in last fetch timestamp Challenge will be around XML document size Acquisition pattern could be developed as well getchangesfordw pass table name & ts 30

31 BI Services A layer of services around the data warehouse and data marts Standardizes calls for information getcustomer, not 10 different SQL statements All clients, applications, portals, Excel could access 31

32 SOA Management Possible Metrics: Quality of service Average response time Successful completions Data inflow and outflow rates Number of queued events Error logging Centralized ESB or gateway architectures will provide a single point in which to collect statistics and perform realtime analytics Great opportunity for new real-time data mart! 32

33 Class Discussion 33