Best Practices in Creating a Successful Business Intelligence Program Wayne W. Eckerson Principal, BI Leader Consulting www.bileader.com 1
Wayne Eckerson BI thought leader Founder, BI Leadership Forum Director, BI Leadership Former Director of Education and Research at TDWI Author Wayne Eckerson weckerson@bileadership.com www.bileader.com 2
Fact-based Decisions Data Developers Embedded Analysts Development Methods Top-down Internal Agenda BI Success Framework CULTURE PEOPLE ORGANIZATION PROCESS ARCHITECTURE Structured DATA Unstructured Sandboxes External Cross-functional Collaboration Analytical Center of Excellence Casual and Power Users Data Treated as a Corporate Asset Analysts Business-oriented BI Project Management Bottom-up Performance Measurement Appendix Self-service BI Evolving DW architecture Designing dashboard displays www.bileader.com 3
Business value of BI Progressive Personalized recommendations based on history Custom auto premiums based on actual driving habits via sensors Personalized online games based on playing habits Customized energy management for customers Proactive health insurance that identifies at-risk patients Optimize the siting of wind turbines by mining larger volumes of data Best time to buy ; average fare by airline, date & market Analyzes data from viral listening posts to prevent pandemics. www.bileader.com 4
What is business intelligence? Strategic view Use data to make smarter decisions Tactical view Reporting and analysis Process view Data Refinery Data is the new oil www.bileader.com 5
BI workflow DATA ACQUISITION ETL, data modeling, data quality, data warehousing DATA IN DATA INSIGHTS ACTION DATA OUT Reports, analysis, dashboarding, predictive modeling, DATA DELIVERY www.bileader.com 6
Evolution of BI Get the data Data Warehousing Use the data Business Intelligence Desktop query/reporting Extract, transform, load tools Data warehouses Improve the business Performance Management Dashboards and scorecards Business intelligence suites Web query/reporting On-line analytical processing (OLAP) 1990s 2000s 2010 2015 www.bileader.com 7 Drive the business Analytics Mobile BI Visual discovery Operational BI Data integration suites Packaged analytic applications Data virtualization Hive/Pig Hadoop Text analytics Cloud BI Predictive analytics
High = Reporting = Analysis Business Value Low Users analysts Execs, Mgrs, Workers Statisticians Waves of BI All users Business What happened? Reporting Static & Interactive Reports Why did it happen? Analysis Query, Excel, OLAP, Viz analysis What s happening? Monitoring Dashboards, Scorecards What will happen? Prediction Statistics, data mining, optimization 1980 s www.bileader.com 8 Tools 1990 s 2000 s 2010 s
Content Intelligence Keyword search, BI tools, Xquery, Hive, Java, etc. MapReduce, XML schema, Key-value pairs, graph notation, etc. HDFS, NoSQL databses BI Framework 2020 Business Intelligence End-User Tools Reports and Dashboards Design Framework MAD Dashboards Architecture Data Warehousing Data Warehousing Reporting & Analysis Analytic Sandboxes Analytic Sandboxes Ad hoc SQL, MDX, Java, Perl, Python Event-driven CEP, Streams Dashboard Alerts Event-Driven Alerts and Dashboards Event detection and correlation Continuous Intelligence Top down Bottom-up Excel, Access, OLAP, Data mining, visual exploration Analytics Intelligence www.bileader.com 9 Exploration Power Users
Pros: - Alignment -Consistency Cons: - Hard to build - Politically charged - Hard to change - Expensive - Schema Heavy Data Warehousing Architecture Top-down vs. Bottom-up BI TOP DOWN- Business Intelligence Corporate Objectives and Strategy Reporting & Monitoring (Casual Users) Predefined Metrics Non-volatile Data Reports Beget Analysis Analysis Begets Reports Pros: - Quick to build Analytics - Politically uncharged Architecture - Easy to change -Low cost Cons: - Alignment - Consistency - Schema www.bileader.com Light 10 Ad hoc queries Analysis and Prediction (Power Users) Processes and Projects 10 Volatile Data
Culture Requires strong leaders! Who deliver value fast! And manage change Requires purple people! www.bileader.com 11
Analytical Leaders Dan Ingle, Kelley Blue Book 1. Incremental development 2. Teamwork 3. One size doesn t fit all Amy O Connor, Nokia 1. Data is a product 2. Create an ecosystem 3. Change management Darren Taylor, Blue KC 1. Create the right team 2. Get executive support 3. Deliver a quick win Eric Colson, Netflix 1. Eliminate coordination costs 2. Work fast, cohere later 3. Build with context www.bileader.com 12 Tim Leonard, USXpress 1. Talk language of business 2. Let business present 3. Deliver quick wins Kurt Thearling, CapitalOne 1. Curate the data 2. Statisticians are craftsmen 3. Manage model production Ken Rudin, Zynga 1. Questions, not answers 2. Impacts, not insights 3. Evangelists, not oracles
People TOP-DOWN Business Intelligence Corporate Objectives and Strategy BI/DW Developers (Centralized) Reporting & Monitoring (Casual Users) Data Warehousing Architecture Predefined Metrics Casual Users Data architects, ETL developers, report developers, data administrators, DW administrators, technical architects, requirements specialists, trainers, etc. Analytics Architecture Ad hoc queries Power Users Analysts (Decentralized) Analysis and Prediction (Power Users) Processes and Projects BOTTOM-UP www.bileader.com 13 Super users, business analysts, statisticians, data scientists, data analysts
Users 24% Top down Bottom up 61% Casual Users Executives/Managers Salespeople Operations staff Customers & suppliers 80% Power Users (Bottom up) Super users Business analysts Analytical modelers Data scientists 80% Monitor metrics Analyze anomalies Drill to detail Explore data Model data Source data Top Down Reports/Dashboards Bottom up Excel, OLAP, Visual Analysis, Mining www.bileader.com 14
80/20 rule 80% of the time 20% of the time CASUAL USERS Task Tools Task Tools Executives Monitor Create queries Super users Managers Analyze MAD Dashboard Create plans (Excel, BI search, Workers Drill to detail Create reports voice-based BI) POWER USERS Task Tools Task Tools Super users Ad hoc reports Self-service BI Business analysts Explore, plan, viz Viz, Excel, SQL Statisticians Create models Data mining tools Data scientists Explore Hadoop Java, Perl, Hive, Pig Monitor Analyze Drill to detail MAD Dashboard Tailored Reporting Ad hoc Analysis www.bileader.com 15
EXERCISE: Map your users to tools 80% of the time 20% of the time CASUAL USERS Task Tools Task Tools POWER USERS Task Tools Task Tools Tailored Reporting Ad hoc Analysis www.bileader.com 16
Organization (BICC) Business sponsors Executive team (Business sponsors) - Approves roadmap - Secures funding - Prioritizes projects Departments Business team Purple team BOBI Team - Evangelizes BI/analytics - Coordinates super users and depts Super Users/ Analysts - Defines best practices - Defines and document metrics Data governance User support - Gathers requirements Statisticians Director of BI - Governs reports Technical team - Builds and maintains the EDW - Builds semantic layer for BI tools - Creates complex reports and dashboards - Develops model management platform www.bileader.com Data developers - 17 Coordinates databases and servers w/ IT
BICC organizing principles BI is a program, not a project Unique people, organization, and processes Multi-level organization Sponsors: executive committee Business: BI director, BOBI, Super users/analysts Technical: BI/DW developers Federated organization Centralized BI director, BOBI, statisticians Decentralized - Supers users and analysts www.bileader.com 18
Process TOP DOWN Monitor the Business Business Intelligence Corporate Objectives and Strategy BI/DW developers Reporting & Monitoring (Casual Users) Data Warehousing Architecture Predefined Metrics Casual Users 1. Start with a business process 2. Gather requirements 3. Build reports/dashboards 4. Test and deploy Analysts Analytics Architecture Ad hoc queries Power Users Analysis and Prediction (Power Users) Processes and Projects BOTTOM UP Explore the business www.bileader.com 19 1. Business problem or opportunity 2. Hypothesize 3. Explore 4. Publish
Architecture Machine Data Streaming/ CEP Engine Casual User Operational System Operational System ETL ODS Data Warehouse Virtual Sandboxes Logical or Physical Data Mart BI Server Interactive dashboards Top-down BI Bottom-up BI Web Data Hadoop Cluster Visual discovery tools Audio/video Data Free-standing Analytical sandbox External Data Power User www.bileader.com 20 KEY: Classic BI New Stuff
Power User Sandboxes Operational Systems (Structured data) Machine Data Streaming/ CEP Engine Casual User Operational System Operational System ETL ODS Data Warehouse Data Mart BI Server Top-down BI Virtual Sandboxes Bottom-up BI Web Data Hadoop Cluster In-memory Sandbox Audio/ video Data External Data www.bileader.com 21 Documents & Text Free-standing sandbox or analytical data mart Analytic platform or NoSQL database Power User
Analytical workflows Capture only what s needed Source Systems 1. Extract, transform, load Analytical database (DW) Capture in case it s needed Hadoop 6. Parse, aggregate 5. Explore data 8. Report and mine data Analytical tools www.bileader.com 22
Mining BI FUNCTIONALITY Analysis Dashboards Reporting BI Tools Market Desktop Analysis (e.g. Excel) Ad hoc Reports/ Dashboards Visual Discovery Data Mining Workbench Operational Reports/ Dashboards Multidimensional OLAP Pixel Perfect Reporting Relational OLAP Big Data Analytics Top-down Bottom-up TYPES OF USERS Casual Users Power Users Analyst Department Enterprise www.bileader.com SCOPE OF DEPLOYMENT 23
Mining BI FUNCTIONALITY Analysis Dashboards Reporting BI Tools Market Desktop Analysis (e.g. Excel) Ad hoc Reports/ Dashboards Analyst Visual Discovery Data Mining Workbench Operational Reports/ Dashboards Department Multidimensional OLAP Pixel Perfect Reporting Enterprise Relational OLAP Big Data Analytics Casual Users Top-down Bottom-up Power Users TYPES OF USERS Analyst Department Enterprise www.bileader.com SCOPE OF DEPLOYMENT 24
Mining Analytics BI FUNCTIONALITY Analysis Dashboards Real-time Reporting Vectors Desktop Analysis (e.g. Excel) Ad hoc Reports/ Dashboards Visual Discovery Data Mining Workbench Operational Reports/ Dashboards Multidimensional OLAP Pixel Perfect Reporting Relational OLAP Big Data Analytics Casual Users Top-down Bottom-up Power Users TYPES OF USERS Analyst Department Enterprise www.bileader.com SCOPE OF DEPLOYMENT 25
Data Low Latency Summarized Data High cost per TB High Latency Detailed Data Low cost per TB Hadoop (Archive, staging area for unstructured data, data preprocessing, batch reporting and mining, other) General Purpose RDBMS (Data marts, small DWs, ODSs) Analytic Platform (Terabyte data warehouses, free-standing sandboxes) Structured Semi-Structured Unstructured Adapted with permission from Hortonworks www.bileader.com 26
Challenges: Reconcile opposites Top Down Business IT Dept Bottom Up www.bileader.com 27
Analytical Maturity Reporting Analysis Dashboards Modeling Spreadsheets and Access Databases Independent Data Marts & Warehouses Data Maturity Enterprise Data Warehouse Big Data Ecosystem Pockets of Analytics Moderate Business Value Analytical Competitor High Business Value Strategic resource Flying Blind Low Business Value Analytical Potential Moderate Business Value Mission critical Tactical resource Analytical Culture Cost center Individual Departmental Enterprise Enterprise+ Scale and Scope www.bileader.com 28
Analytical Maturity Reporting Analysis Dashboards Modeling Spreadsheets and Access Databases Independent Data Marts & Warehouses Data Maturity Enterprise Data Warehouse Big Data Ecosystem Strategic resource Analytical Culture Mission critical Tactical resource Cost center Individual Departmental Enterprise Enterprise+ Scale and Scope www.bileader.com 29
Appendix Self-service BI Evolving DW architecture Designing dashboard displays www.bileader.com 30
Self-service BI www.bileader.com 31
Self-service BI Not so fast! www.bileader.com 32
Self service or self serving? REPORT CHAOS LOW ADOPTION www.bileader.com 33
The truth about self-service BI Self-service BI requires a lot of hand-holding! - Kevin Sonsky, Senior Director, Business Intelligence, Citrix Systems www.bileader.com 34
Types of self-service BI tools TOP DOWN- Business Intelligence Corporate Objectives and Strategy Reporting & Monitoring (Casual Users) Data Warehousing Architecture Predefined Metrics Non-volatile Data BI objects BI mashboards (IT DRIVEN) Visual discovery (ANALYST DRIVEN) Analytics Architecture Ad hoc queries Volatile Data Analysis and Prediction (Power Users) Processes and Projects www.bileader.com 35
Self-service BI tools Mashboard Visual analysis www.bileader.com 36
More Analytical More Interactive Self-service hierarchies CONSUMERS PRODUCERS View Personalize Navigate Modify Explore Expose on Demand Assemble Craft Source More IT-oriented More complex Model Develop www.bileader.com 37
Power Users Power Users Casual Users Casual Users Self service BI CONSUMERS View PRODUCERS Personalize Navigate Modify Assemble Craft Explore Model Source Develop BI Developers www.bileader.com 38
EXERCISE #2: Map users to self-service hierarchies CONSUMERS View Navigate Modify Explore Model PRODUCERS Personalize Assemble Craft Source Develop www.bileader.com 39
Best practices Ask right people Ask right questions Map processes Understand incentives Role mapping Tool fitting MAD Ad hoc Composite Scrums Sandboxes Prototypes BI Roadmap Councils Newsletters Town Halls Campaigns Shut down legacy Manage Expectations Marketing Leadership Requirements Change Mgmt Roles Framework Design Adoption Architecture Agile Support Training Tailored Super users Support Feedback Numeracy Help desk Mentoring Monitoring Surveys Certified reports Use the tools Flexibility Data Delivery Performance Post-mortems Layers of Abstraction Atomic data Data access Metadata/ Reuse Coverage Quality Timeliness Web Email Mobile Response times User Query concurrency complexity www.bileader.com 40
DW Architectures www.bileader.com 41
Strategic DW evolution 1990s Local data warehouses, spreadmarts in each BU 2000-2007 Fully centralized enterprise data warehouses BU 1 BU 2 BU 3 BU 4 BU 1 BU 2 BU 3 BU 4 Reports Reports Reports Reports Reports Reports Reports Reports Data Whs 1 Data Whs 2 Spread Mart 1 Spread Mart 2 Enterprise DWs Benefits: Rapid deployment Local control over priorities, resources Customization meets high % of requirements Challenges: Duplication of effort across BUs Redundant costs (HW, SW, support staff) Silo mentality, lack of comm across Bus Data integration difficult without scalable environment www.bileader.com 42 Benefits: Reduce data redundancy Promotes communication between Bus Resource efficiency (HW, SW, FTEs) Challenges: BUs compete over centralized DW resources One size fits all solution meets lower % of business requirements for each BU Data integration difficult due to limited resources
Hybrid DW architecture 2008+ Enterprise DW foundation with context-specific flexibility BU 1 BU 2 BU 3 BU 4 Reports Reports Reports Reports Ent DM 1 Ent DM 1 Enterprise Data Marts BU DM 1 BU-owned Data Marts BU-specific data, filters, biz rules DW Foundation ODS tables, shared dimensions Enterprise DW BU DM 1 BU DM 1 Hybrid model leverages benefits of both centralized & decentralized models Challenges from both models still exist to a lesser degree but consciously accepted given the benefits Crucial focus on easier data integration to support growth of various businesses Requires a robust social architecture - lots of communications and education, a strong BICC, a clear roadmap, strong business governance, and frequent meetings. www.bileader.com 43
Designing Dashboard Displays www.bileader.com 44
Design keys Less is more! Make every pixel count Avoid decoration Set standards Tell the story of the data www.bileader.com 45
Avoid decoration www.bileader.com 46 46
Tell the story Courtesy Stephen Few www.bileader.com 47
Tell the story (cont) First Iteration Second Iteration www.bileader.com 48
User feedback at Guess Wow, it s so easy to see how different patterns are selling, how different colors are taking off, it s so great to have visibility into other sides of the business, because there s lot of competition across our divisions. www.bileader.com 49
Set standards It s a rare type of chart, so when people see a spiderweb chart, I want them to associate it with patient satisfaction. It creates a mental shortcut for people if there s some variation and a personality that makes a metric stand out visually. -- Daniel Gerena, Director of BI and Analytics, Kaleida Health www.bileader.com 50
EXERCISE: Redraw this chart What is your ROLE? 60% 50% 40% 30% 20% 10% 0% Series1 Software vendor representative Business sponsor or user Consultant or systems integrator Academic BI or IT professional www.bileader.com 51
Questions?? I m listening! www.bileader.com 52