Applied Business Intelligence Iakovos Motakis, Ph.D. Director, DW & Decision Support Systems Intrasoft SA
Agenda Business Drivers and Perspectives Technology & Analytical Applications Trends Challenges & Methodology Issues
Business Drivers & Perspectives
Business Drivers Better understanding of Customers, Costs and Revenue Streams Customer-Centric model of operation Analyze the profitability of products and market segments Customer Relationship Management Crossselling, Up-selling, etc. Targeted marketing campaigns Improve quality of service
Information Requirements Customer (Atomic & Demographic data) Products and Packages Sales Channels Supply Chain information Activity-Based Costing, Order Fullfilment, etc. Promotions, Discounts Customer Contact information Call Centers, Operational CRM systems
The Issues Needed information is scattered across different operational systems Huge amounts of data Difficult to extract and consolidate Widely different analytical functionality required by different users
Data Warehousing Exploits the information that is produced or captured in production systems Provides an infrastructure of processes, procedures and tools (software and hardware) that supports: Access to information (Data Acquisition) Centralized maintenance of consolidated & complete views of data Fast analysis of data and presentation of results (Data Exploitation & Presentation) Enables decision making and planning
Business Needs (Analysis) Consolidated view of data. Single version of the truth, consistent use of business rules Cross-functional analysis, improved management communication Enablement of analytical processes Facilitation of user decision making, by supporting User-Driven Reporting and train-of-thought Identification of hidden trends
Business Needs (IT) Centralized access to data Access to information in a timely manner Off-Load Complex Queries and Reports from key operational (production) systems Improved common procedures for data quality Uniform methods for data flow/distribution
Terms and Phrases that Scream DW Easy Access to Information Analysis of Data (especially Interactive or Ad hoc) Support for Train-of-Thought Automated Reporting Off-Load Processing from Legacy Systems Common Business Terms Consistent Information (Data) Data Consolidation/ Reconciliation/Rollup Data Quality Discover or Predict Business Trends
Summary Business Intelligence is the link between analytical business processes and available information It is a prerequisite for growing the business in a competitive environment Very high ROI for successful projects Required for CRM, cross-channel selling, e-business support, etc.
Technology & Analytical Applications Trends
OLTP vs OLAP OLTP (On-Line Analytical Processing) Systems Designed to manage day-to-day operations Typically organized by departmental applications Normalized database design to deal with frequent updates and data contention Designed to support large numbers of relatively short transactions. Less history retention
OLTP vs OLAP (Cont.) OLAP (On-Line Analytical Processing) Systems Analytical queries that facilitate decision making and planning Typically organized by subjects, e.g., Customer Profiling, Product Profitability, Inventory Mgmt, etc. Database design to support complex queries that retrieve and manipulate data Simpler requirements for transaction management Intensive historical analysis
Evolution of DW/BI Early 90 s Database Performance» MPP DB Architectures (e.g., NCR/Teradata, IBM DB2)» Dimensional Modeling Techniques ~Mid 90 s MOLAP Servers, OLAP Tools Understanding of ETL (Extraction, Transformation, Loading) Issues Late 90 s: Focus on Analytical Applications and Vertical Solutions
Data Warehousing Components Data Marts MOLAP Server OLAP Replication Data Warehouse Repository Batch Extract Load Data Staging Area Cleanse & Transform Data Browsing & Reporting Extract - Consolidate O p e r a t i o n a l D a t a S t o r e s
Current Status of Technology Maturity of DB, OLAP, ETL Technologies Entrance of ERP Vendors & Microsoft Consolidation of Market Vendors investing on consulting practices Small room for new purely technology-based players
Current Trends: Analytical Applications Trend towards productizing & verticalizing BI applications Tighter integration with operational applications BI Modules on top of ERP systems Closed-Loop, real-time requirements» Very hard, technology not very mature yet CRM, Balanced Scorecard, etc. Trend towards Buy vs. Build
End-to-End vs. Best-of-Breed Big players are trying to complete end-to-end solutions IBM, Oracle, SAS, etc. No clear superiority over best-of-breed solutions yet Technology landscape is continuously changing
E-Business Intelligence Business-to-Customer(B2C) and Business-to-Business are very data intensive B2B consolidates information across entire supply chain Needs for Understanding customers Analyze the profitability of partner relationships Optimizing supply chain management Many opportunities for analytical applications Much more data, but no fundamental differences
Enterprise-Wide Data Warehouse (EDW) vs. Data Marts EDW Guarantees complete and consistent information Facilitates cross-functional analysis Ensures scalability Dramatically reduces future integration efforts Data Mart Targeted towards a specific business need, or limited to specific data sets Higher success rates Usually higher ROI
EDW vs. Data Marts Both approaches are valid EDW is the ideal, but often practice dictates building isolated BI applications Architected Data Marts is often the best way Using a common Staging Area is always a good practice Achieves data sharing Facilitates integration of Analytical Applications
Challenges & Methodology Issues
The Challenges Providing merely a DW/BI infrastructure achieves no differentiation Prerequisites for Success Identify ways to add value, beyond data collection Implement analytical applications to address specific business pains Establishment of analytical business processes to exploit the DW/BI infrastructure Take into account People, Process & Organizational Issues
Key Understandings Data Warehousing is a process, not a product This process is evolutionary and iterative in nature The design of a DW is both user and data driven with an emphasis on analysis Cross-functional user involvement is highly-desirable Normalizing, cleansing, and validating data requires business user participation Data Warehousing does not fix operational system issues
Critical Success Factors Executive Sponsorship & Support People Process & Organization Issues Implementation of Analytical Business Processes Cooperation of key business users Dedicated IT Resources Prioritization & Scoping of Projects Domain knowledge Significant expertise required High failure rates for non-business driven efforts
Realities... DW/BI projects rarely receive highest priority, not mission-critical systems Need to continuously work on maintaining support, publishing results, trying to quantify ROI Conflicts of interest among key stakeholders Difficult to deliver as a fixed-time, fixed-cost project
Scoping The Scoping Phase is of utmost importance. Will cover: Interviews with business users from different functional areas t identify key business drivers What to build first Analysis of data sources & Data Quality Assessment Organizational Readiness Study Selection of key partners, which architecture to adopt Evolution Plan
Implementation Plan Phase the DW using: Subject Areas/Analytical Processes Selected Source Systems Phased User/Departmental Rollout Rapid Prototype Development for a high-payback Area Flexible and scalable Technical and Data Architectures New Subject Areas, Data Sources New BI applications, OLAP & Data Mining tools Larger volumes of data Higher frequency of analytical queries
Implementation Plan (Cont.) Business Domain knowledge is very important Solid Data Acquisition/ETL processes Effort is very often underestimated Typically takes >50% of total effort Tools do not automate data source analysis
Organizational Preparation Dedicated Team of Business and IT resources Min. 3 persons Business-Driven initiatives Preparation for continuous evolution ~ 6-12 months for first iteration, 3-6 months for subsequent iterations Data Quality issues need to be addressed as early as possible
Conclusion Technology offers many choices DW/BI applications will never be completely packaged, but many components are available Require significant business domain knowledge, technical expertise and project management skills DW/BI Apps cannot be produced massively Estimation of fixed-time, fixed-cost projects can be very challenging Rewards are fantastic for successful projects!!