Best Practices in Creating a Successful Business Intelligence Program

Transcription

1 Best Practices in Creating a Successful Business Intelligence Program Wayne W. Eckerson Principal, BI Leader Consulting 1

2 Wayne Eckerson BI thought leader Founder, BI Leadership Forum Director, BI Leadership Former Director of Education and Research at TDWI Author Wayne Eckerson 2

3 Fact-based Decisions Data Developers Embedded Analysts Development Methods Top-down Internal Agenda BI Success Framework CULTURE PEOPLE ORGANIZATION PROCESS ARCHITECTURE Structured DATA Unstructured Sandboxes External Cross-functional Collaboration Analytical Center of Excellence Casual and Power Users Data Treated as a Corporate Asset Analysts Business-oriented BI Project Management Bottom-up Performance Measurement Appendix Self-service BI Evolving DW architecture Designing dashboard displays 3

4 Business value of BI Progressive Personalized recommendations based on history Custom auto premiums based on actual driving habits via sensors Personalized online games based on playing habits Customized energy management for customers Proactive health insurance that identifies at-risk patients Optimize the siting of wind turbines by mining larger volumes of data Best time to buy ; average fare by airline, date & market Analyzes data from viral listening posts to prevent pandemics. 4

5 What is business intelligence? Strategic view Use data to make smarter decisions Tactical view Reporting and analysis Process view Data Refinery Data is the new oil 5

6 BI workflow DATA ACQUISITION ETL, data modeling, data quality, data warehousing DATA IN DATA INSIGHTS ACTION DATA OUT Reports, analysis, dashboarding, predictive modeling, DATA DELIVERY 6

7 Evolution of BI Get the data Data Warehousing Use the data Business Intelligence Desktop query/reporting Extract, transform, load tools Data warehouses Improve the business Performance Management Dashboards and scorecards Business intelligence suites Web query/reporting On-line analytical processing (OLAP) 1990s 2000s Drive the business Analytics Mobile BI Visual discovery Operational BI Data integration suites Packaged analytic applications Data virtualization Hive/Pig Hadoop Text analytics Cloud BI Predictive analytics

8 High = Reporting = Analysis Business Value Low Users analysts Execs, Mgrs, Workers Statisticians Waves of BI All users Business What happened? Reporting Static & Interactive Reports Why did it happen? Analysis Query, Excel, OLAP, Viz analysis What s happening? Monitoring Dashboards, Scorecards What will happen? Prediction Statistics, data mining, optimization 1980 s 8 Tools 1990 s 2000 s 2010 s

9 Content Intelligence Keyword search, BI tools, Xquery, Hive, Java, etc. MapReduce, XML schema, Key-value pairs, graph notation, etc. HDFS, NoSQL databses BI Framework 2020 Business Intelligence End-User Tools Reports and Dashboards Design Framework MAD Dashboards Architecture Data Warehousing Data Warehousing Reporting & Analysis Analytic Sandboxes Analytic Sandboxes Ad hoc SQL, MDX, Java, Perl, Python Event-driven CEP, Streams Dashboard Alerts Event-Driven Alerts and Dashboards Event detection and correlation Continuous Intelligence Top down Bottom-up Excel, Access, OLAP, Data mining, visual exploration Analytics Intelligence 9 Exploration Power Users

10 Pros: - Alignment -Consistency Cons: - Hard to build - Politically charged - Hard to change - Expensive - Schema Heavy Data Warehousing Architecture Top-down vs. Bottom-up BI TOP DOWN- Business Intelligence Corporate Objectives and Strategy Reporting & Monitoring (Casual Users) Predefined Metrics Non-volatile Data Reports Beget Analysis Analysis Begets Reports Pros: - Quick to build Analytics - Politically uncharged Architecture - Easy to change -Low cost Cons: - Alignment - Consistency - Schema Light 10 Ad hoc queries Analysis and Prediction (Power Users) Processes and Projects 10 Volatile Data

11 Culture Requires strong leaders! Who deliver value fast! And manage change Requires purple people! 11

12 Analytical Leaders Dan Ingle, Kelley Blue Book 1. Incremental development 2. Teamwork 3. One size doesn t fit all Amy O Connor, Nokia 1. Data is a product 2. Create an ecosystem 3. Change management Darren Taylor, Blue KC 1. Create the right team 2. Get executive support 3. Deliver a quick win Eric Colson, Netflix 1. Eliminate coordination costs 2. Work fast, cohere later 3. Build with context 12 Tim Leonard, USXpress 1. Talk language of business 2. Let business present 3. Deliver quick wins Kurt Thearling, CapitalOne 1. Curate the data 2. Statisticians are craftsmen 3. Manage model production Ken Rudin, Zynga 1. Questions, not answers 2. Impacts, not insights 3. Evangelists, not oracles

13 People TOP-DOWN Business Intelligence Corporate Objectives and Strategy BI/DW Developers (Centralized) Reporting & Monitoring (Casual Users) Data Warehousing Architecture Predefined Metrics Casual Users Data architects, ETL developers, report developers, data administrators, DW administrators, technical architects, requirements specialists, trainers, etc. Analytics Architecture Ad hoc queries Power Users Analysts (Decentralized) Analysis and Prediction (Power Users) Processes and Projects BOTTOM-UP 13 Super users, business analysts, statisticians, data scientists, data analysts

14 Users 24% Top down Bottom up 61% Casual Users Executives/Managers Salespeople Operations staff Customers & suppliers 80% Power Users (Bottom up) Super users Business analysts Analytical modelers Data scientists 80% Monitor metrics Analyze anomalies Drill to detail Explore data Model data Source data Top Down Reports/Dashboards Bottom up Excel, OLAP, Visual Analysis, Mining 14

15 80/20 rule 80% of the time 20% of the time CASUAL USERS Task Tools Task Tools Executives Monitor Create queries Super users Managers Analyze MAD Dashboard Create plans (Excel, BI search, Workers Drill to detail Create reports voice-based BI) POWER USERS Task Tools Task Tools Super users Ad hoc reports Self-service BI Business analysts Explore, plan, viz Viz, Excel, SQL Statisticians Create models Data mining tools Data scientists Explore Hadoop Java, Perl, Hive, Pig Monitor Analyze Drill to detail MAD Dashboard Tailored Reporting Ad hoc Analysis 15

16 EXERCISE: Map your users to tools 80% of the time 20% of the time CASUAL USERS Task Tools Task Tools POWER USERS Task Tools Task Tools Tailored Reporting Ad hoc Analysis 16

17 Organization (BICC) Business sponsors Executive team (Business sponsors) - Approves roadmap - Secures funding - Prioritizes projects Departments Business team Purple team BOBI Team - Evangelizes BI/analytics - Coordinates super users and depts Super Users/ Analysts - Defines best practices - Defines and document metrics Data governance User support - Gathers requirements Statisticians Director of BI - Governs reports Technical team - Builds and maintains the EDW - Builds semantic layer for BI tools - Creates complex reports and dashboards - Develops model management platform Data developers - 17 Coordinates databases and servers w/ IT

18 BICC organizing principles BI is a program, not a project Unique people, organization, and processes Multi-level organization Sponsors: executive committee Business: BI director, BOBI, Super users/analysts Technical: BI/DW developers Federated organization Centralized BI director, BOBI, statisticians Decentralized - Supers users and analysts 18

19 Process TOP DOWN Monitor the Business Business Intelligence Corporate Objectives and Strategy BI/DW developers Reporting & Monitoring (Casual Users) Data Warehousing Architecture Predefined Metrics Casual Users 1. Start with a business process 2. Gather requirements 3. Build reports/dashboards 4. Test and deploy Analysts Analytics Architecture Ad hoc queries Power Users Analysis and Prediction (Power Users) Processes and Projects BOTTOM UP Explore the business Business problem or opportunity 2. Hypothesize 3. Explore 4. Publish

20 Architecture Machine Data Streaming/ CEP Engine Casual User Operational System Operational System ETL ODS Data Warehouse Virtual Sandboxes Logical or Physical Data Mart BI Server Interactive dashboards Top-down BI Bottom-up BI Web Data Hadoop Cluster Visual discovery tools Audio/video Data Free-standing Analytical sandbox External Data Power User 20 KEY: Classic BI New Stuff

21 Power User Sandboxes Operational Systems (Structured data) Machine Data Streaming/ CEP Engine Casual User Operational System Operational System ETL ODS Data Warehouse Data Mart BI Server Top-down BI Virtual Sandboxes Bottom-up BI Web Data Hadoop Cluster In-memory Sandbox Audio/ video Data External Data 21 Documents & Text Free-standing sandbox or analytical data mart Analytic platform or NoSQL database Power User

22 Analytical workflows Capture only what s needed Source Systems 1. Extract, transform, load Analytical database (DW) Capture in case it s needed Hadoop 6. Parse, aggregate 5. Explore data 8. Report and mine data Analytical tools 22

23 Mining BI FUNCTIONALITY Analysis Dashboards Reporting BI Tools Market Desktop Analysis (e.g. Excel) Ad hoc Reports/ Dashboards Visual Discovery Data Mining Workbench Operational Reports/ Dashboards Multidimensional OLAP Pixel Perfect Reporting Relational OLAP Big Data Analytics Top-down Bottom-up TYPES OF USERS Casual Users Power Users Analyst Department Enterprise SCOPE OF DEPLOYMENT 23

24 Mining BI FUNCTIONALITY Analysis Dashboards Reporting BI Tools Market Desktop Analysis (e.g. Excel) Ad hoc Reports/ Dashboards Analyst Visual Discovery Data Mining Workbench Operational Reports/ Dashboards Department Multidimensional OLAP Pixel Perfect Reporting Enterprise Relational OLAP Big Data Analytics Casual Users Top-down Bottom-up Power Users TYPES OF USERS Analyst Department Enterprise SCOPE OF DEPLOYMENT 24

25 Mining Analytics BI FUNCTIONALITY Analysis Dashboards Real-time Reporting Vectors Desktop Analysis (e.g. Excel) Ad hoc Reports/ Dashboards Visual Discovery Data Mining Workbench Operational Reports/ Dashboards Multidimensional OLAP Pixel Perfect Reporting Relational OLAP Big Data Analytics Casual Users Top-down Bottom-up Power Users TYPES OF USERS Analyst Department Enterprise SCOPE OF DEPLOYMENT 25

26 Data Low Latency Summarized Data High cost per TB High Latency Detailed Data Low cost per TB Hadoop (Archive, staging area for unstructured data, data preprocessing, batch reporting and mining, other) General Purpose RDBMS (Data marts, small DWs, ODSs) Analytic Platform (Terabyte data warehouses, free-standing sandboxes) Structured Semi-Structured Unstructured Adapted with permission from Hortonworks 26

27 Challenges: Reconcile opposites Top Down Business IT Dept Bottom Up 27

28 Analytical Maturity Reporting Analysis Dashboards Modeling Spreadsheets and Access Databases Independent Data Marts & Warehouses Data Maturity Enterprise Data Warehouse Big Data Ecosystem Pockets of Analytics Moderate Business Value Analytical Competitor High Business Value Strategic resource Flying Blind Low Business Value Analytical Potential Moderate Business Value Mission critical Tactical resource Analytical Culture Cost center Individual Departmental Enterprise Enterprise+ Scale and Scope 28

29 Analytical Maturity Reporting Analysis Dashboards Modeling Spreadsheets and Access Databases Independent Data Marts & Warehouses Data Maturity Enterprise Data Warehouse Big Data Ecosystem Strategic resource Analytical Culture Mission critical Tactical resource Cost center Individual Departmental Enterprise Enterprise+ Scale and Scope 29

30 Appendix Self-service BI Evolving DW architecture Designing dashboard displays 30

31 Self-service BI 31

32 Self-service BI Not so fast! 32

33 Self service or self serving? REPORT CHAOS LOW ADOPTION 33

34 The truth about self-service BI Self-service BI requires a lot of hand-holding! - Kevin Sonsky, Senior Director, Business Intelligence, Citrix Systems 34

35 Types of self-service BI tools TOP DOWN- Business Intelligence Corporate Objectives and Strategy Reporting & Monitoring (Casual Users) Data Warehousing Architecture Predefined Metrics Non-volatile Data BI objects BI mashboards (IT DRIVEN) Visual discovery (ANALYST DRIVEN) Analytics Architecture Ad hoc queries Volatile Data Analysis and Prediction (Power Users) Processes and Projects 35

36 Self-service BI tools Mashboard Visual analysis 36

37 More Analytical More Interactive Self-service hierarchies CONSUMERS PRODUCERS View Personalize Navigate Modify Explore Expose on Demand Assemble Craft Source More IT-oriented More complex Model Develop 37

38 Power Users Power Users Casual Users Casual Users Self service BI CONSUMERS View PRODUCERS Personalize Navigate Modify Assemble Craft Explore Model Source Develop BI Developers 38

39 EXERCISE #2: Map users to self-service hierarchies CONSUMERS View Navigate Modify Explore Model PRODUCERS Personalize Assemble Craft Source Develop 39

40 Best practices Ask right people Ask right questions Map processes Understand incentives Role mapping Tool fitting MAD Ad hoc Composite Scrums Sandboxes Prototypes BI Roadmap Councils Newsletters Town Halls Campaigns Shut down legacy Manage Expectations Marketing Leadership Requirements Change Mgmt Roles Framework Design Adoption Architecture Agile Support Training Tailored Super users Support Feedback Numeracy Help desk Mentoring Monitoring Surveys Certified reports Use the tools Flexibility Data Delivery Performance Post-mortems Layers of Abstraction Atomic data Data access Metadata/ Reuse Coverage Quality Timeliness Web Mobile Response times User Query concurrency complexity 40

41 DW Architectures 41

42 Strategic DW evolution 1990s Local data warehouses, spreadmarts in each BU Fully centralized enterprise data warehouses BU 1 BU 2 BU 3 BU 4 BU 1 BU 2 BU 3 BU 4 Reports Reports Reports Reports Reports Reports Reports Reports Data Whs 1 Data Whs 2 Spread Mart 1 Spread Mart 2 Enterprise DWs Benefits: Rapid deployment Local control over priorities, resources Customization meets high % of requirements Challenges: Duplication of effort across BUs Redundant costs (HW, SW, support staff) Silo mentality, lack of comm across Bus Data integration difficult without scalable environment 42 Benefits: Reduce data redundancy Promotes communication between Bus Resource efficiency (HW, SW, FTEs) Challenges: BUs compete over centralized DW resources One size fits all solution meets lower % of business requirements for each BU Data integration difficult due to limited resources

43 Hybrid DW architecture Enterprise DW foundation with context-specific flexibility BU 1 BU 2 BU 3 BU 4 Reports Reports Reports Reports Ent DM 1 Ent DM 1 Enterprise Data Marts BU DM 1 BU-owned Data Marts BU-specific data, filters, biz rules DW Foundation ODS tables, shared dimensions Enterprise DW BU DM 1 BU DM 1 Hybrid model leverages benefits of both centralized & decentralized models Challenges from both models still exist to a lesser degree but consciously accepted given the benefits Crucial focus on easier data integration to support growth of various businesses Requires a robust social architecture - lots of communications and education, a strong BICC, a clear roadmap, strong business governance, and frequent meetings. 43

44 Designing Dashboard Displays 44

45 Design keys Less is more! Make every pixel count Avoid decoration Set standards Tell the story of the data 45

46 Avoid decoration

47 Tell the story Courtesy Stephen Few 47

48 Tell the story (cont) First Iteration Second Iteration 48

49 User feedback at Guess Wow, it s so easy to see how different patterns are selling, how different colors are taking off, it s so great to have visibility into other sides of the business, because there s lot of competition across our divisions. 49

50 Set standards It s a rare type of chart, so when people see a spiderweb chart, I want them to associate it with patient satisfaction. It creates a mental shortcut for people if there s some variation and a personality that makes a metric stand out visually. -- Daniel Gerena, Director of BI and Analytics, Kaleida Health 50

51 EXERCISE: Redraw this chart What is your ROLE? 60% 50% 40% 30% 20% 10% 0% Series1 Software vendor representative Business sponsor or user Consultant or systems integrator Academic BI or IT professional 51

52 Questions?? I m listening! 52