Athens, 26 th June The Mega Trends that are changing the Economics of Data Warehousing & Business Intelligence

Transcription

1 Athens, 26 th June 2012 The Mega Trends that are changing the Economics of Data Warehousing & Business Intelligence

2 DWH is undergoing the most radical transformation seen since it was first conceived in the 1970s The DWH will remain a key component (and possibly the largest) of your IT infrastructure (Source: Gartner 2011) Not only you need to manage and store Big Data but also the need to find ways to extract the maximum value from it The DWH is reaching a major inflection point, which is highlighted by specific trends in the business environment technologies and implementation and delivery practices There are many diversions in making key decisions re emerging technology solutions You need to be aware of several key trends and how they will impact the cost-benefit balance of technology deployed to deliver business value Data Quality BIaaS appliances Mega Vendors Open Source BIG Data Cloud Mobility In-memory Hadoop Operational BI Visualization 2

3 BI & Analytics top enterprise technology Top Three Enterprise Technologies in Order of Importance, (Source: Gartner 2011) 3

4 Current Situation Mature market. Installations started at late 90 resulting into large and cost-heavy DWHs The vast majority of Big/Mid size organizations in GR have some form of DWH or datamart The architecture and technology selection in many cases is driven by the ERP vendors More than 70% of so-called EDWH actually serve only back-office or limited departmental use cases rather than enterprise standards (Source: Gartner) Only 10% of enterprises have managed to deploy BI to a majority of their users (Source: Business Application Research Council, BARC) Need to retrofit existing DWH some time between 5-7 years (Source: Gartner) and recalculate TCO and ROI Key Issues include: evolution usually without blueprint and strategy defined data quality weak analytics departmental version-of-truth and lack of proper Governance and ongoing Optimization and Consolidation 4

5 Market Dynamics At the high end, DWH is now mission-critical Data warehouse DBMSs have evolved to a broader analytics infrastructure Exploration of alternative DWH infrastructures is increasing mainly due to the cost (Source: Gartner), accept higher first cost in exchange for lower admin and mgt costs Performance optimization is driving multi-tiered data architectures and a variety of deployment options e.g. a large central DWH and in-memory data mart deployments Renewed interest in virtual DWH using federation technology that defers cost or shifts it to another category or owner For all vendors 60%+ of DWH accounts also have a competitor supporting DWH workloads, often for the same end-user community. Organizations should not accept procurement's arguments for fewer vendors as the only justification for eliminating a specialized solution (Source: Gartner) Everybody is talking about Big Data, usually in connection with some kind of advanced analytic application Social networking is incorporated into BI/DW (inc. searching, matching and combining, tagging, sharing, commenting and rating) into an analytic context ipad has accelerated the Going Mobile BI notion 5

6 DWH Architecture Considerations DWH is a persistent solution with a requirement for constant updating in both design and technology applied to the solution architecture (Source: Gartner) No matter how fast HW and how efficient memory become, the bottleneck in storage rates will always drive solutions based on an n-tier data architecture DWH data architecture and SLAs should be addressed in common Create or plan for additional data architecture tiers to meet specific service-level requirements In 2010, the market accepted the implementation of two copies of the DWH to resolve SL conflicts in mixed workloads towards optimization and performance Resurgence of Data Marts. Due to increasing variety DWH-driven applications and workload, data mart proliferation has become an acceptable, tactical solution (Source: Gartner) Also, need to plan for ongoing data mart consolidation and governance In-Memory DBMSs introduces a higher probability that analytics and transactional systems can share the same database Organizations should balance the analytic workloads and the storage growth DB optimization techniques vs. another application tier. First answer the SLA data architecture requirements By 2014, 85% of currently deployed DWH will not scale to meet volume and complexity demands 6

7 BI / DWH Mega Trends As these grow continuously, so scalability is key Datasets / Resources, Frequency data load Big Data Operational BI End-users Report refresh Reports Real-time analytics Integration & Processing Complexity Frequency is increasing, thus pushing closer to real time Size Speed Inter/ty is key to real-time & operational BI, plus SOA e.g. DQ, cust. 360 etc. BI layers Integration tools Operational BI SOA Interopera bility Economics Do more with less (headcount, cost) Data mart consolidation Cost optimization is healthy but don t fail to invest in BI 7

8 Technology Trends DWH Appliances Big Data Hadoop Operational BI In-memory Computing Trends in Data Integration Trends in Data Quality Trends in BI Data Stores Information Delivery Trends Trends in Business Analytics Trends in Data Visualization New BI Users and Mobile BI Cloud and aas Open Source DWH Mega Vendors 8

9 DWH Appliances By 2015, at least 50% of enterprises with DWH in production will include a DWH appliance (Gartner) A DWH appliance is A dedicated device combining HW (SMP and MPP elements and storage), DB and BI A platform configured based on the size of the DB and tuned for analysis workloads Certified by the vendor for predictable performance Great antidote to the numerous data analysis BU shadow systems Value Proposition Simplicity Speed installation and setup (avoid time-consuming HW balancing, "no assembly required ) Competitive price/performance offerings Predictable performance (in some cases with vendor-supplied SLAs) Lower total cost of acquisition (TCA) and/or TCO, Less mgt cost (single entity, lower resources) Ease of maintenance and support (one vendor to call) 9

10 The challenge and opportunity of big data Executive Breakfast Meeting Q1 CY12 The proliferation of data has always been part of the impact of ICT. But, the amount of data available for analysis is exploding and the scale and scope of the changes that such big data are bringing about have reached an inflection point collecting, storing, and mining big data for insights can create significant value for the world economy, enhancing the productivity and competitiveness of companies (Source: McKinsey Global Institute) Shortage of 140K 190K people in US with analytical and managerial expertise and 1.5M million managers and analysts with the skills to understand and make decisions based on the study of big data 10

11 How Big? What s the approximate total data volume that your organization s data warehouse environment manages, both today and in three years or so? 10-terabyte club will double over the next three years, 17% to 34% of DWs Reaching 10Tb may require a change of DW platform (Source: TDWI 2009) Data is growing by a factor of 10 every five years, a compound annual growth rate at almost 60%, IDC This growth rate is likely to accelerate given new and evolving information generating technologies (web info, social media, RFID tags, etc.) Telco, Banks, Insurance, Fuel, Retailers, utilities, airlines, 11

12 Data Integration Trends, and not ETL anymore DI survived a scalability crisis, but it now faces another with speed and interoperability: Mega Trends Drivers Targets Size Speed Interoperability Economics Data volumes and the complexity of transformations are increasing dramatically Overnight batch processing just isn t soon enough any more. Some business processes demand fresher information Integration solutions are converging Federated and virtual approaches are rising ETL for DW assumes a time-consuming and expensive dev process, but budgets are reduced and businesses must react quickly Data integration servers scale via parallel processing Users apply distributed architectures to gain scalability ETL now supports micro batches. Users also use federation, app integration buses, services, and replication time-sensitive data Low latency Seamlessly integrate BI with business processes to create faster decision making Diverse data requirements lead to multiple integration technologies Users are open to cheap ETL tools, federation, replication and Open Source Hand-coding is fading, due to high payroll costs. 12

13 Trends in Data Quality Most data-quality trends lead to broader enterprise use Mega Trends Drivers Targets Size Speed Departmental solutions Technical users Multinational data Batch operation Enterprise initiatives Business users (data stewardship) International data (Global postal standards) Real-time operation Interoperability Economics Point products Data profiling Departmental budget Tool suites Data monitoring Complexity on unstructured data mgt. (social media) Enterprise investment 13

14 Trends in BI Data Stores (operational BI, DM, spreadsheets,..) The DWH is in good economic shape, but faces speed and interoperability issues Mega Trends Drivers Targets Size Speed Operational and transactional apps increase data output daily More analytics, XML, unstructured data Business practices like performance mgt, ondemand, and real-time enterprise require very fresh data Data volume captured and stored for BI has increased DW size Users move to heftier DW platforms, distributed architectures Overnight refresh of a warehouse is still the norm. But some users also refresh timesensitive data multiple times daily Interoperability Refreshing a DW multiple times daily still requires ETL, but also ELT, federation, services, CDC, replication, micro batches An operational or real-time DW requires the BI framework to interoperate with itself and with other app stacks at an unprecedented level Economics Prices of DW platforms are down. Cheap DW appliances arrived Too many BI data stores More small & mid-size organizations can afford a (large) data warehouse Data mart and ODS consolidations 14

15 Information Delivery Trends Information Delivery continues to struggle with size and speed issues, due to the rise of enterprise BI and performance management, respectively: Mega Trends Drivers Targets Size Speed Interoperability Economics Enterprise BI (EBI) means more report consumers/producers and more reports to refresh/distribute Performance management gets more granular all the time Business processes need a way to push timesensitive info to users Reporting & analysis tools don t interoperate much or deeply Operational BI joins tactical and strategic BI as a viable practice Redundant BI solutions, typically for departmental use To survive EBI, IT must add more BI staff and deploy scalable BI tools, which requires increased budget, at a time when budgets are cut/frozen BI must refresh dashboard and scorecard-style reports frequently More firms push real-time alerts to end-users via a BI platform Some vendors are finally making their products interoperate usefully More users do operational BI, which demands greater interoperability Centralize/consolidate into fewer BI tools for less admin cost, more reuse Enterprise rather departmental budgets 15

16 Trends in Business Analytics Although OLAP is still the most common technology for business analytics, the users are moving to use more advanced forms, like data mining, statistical tools, natural language processing etc. Mega Trends Drivers Targets Size Many users have crossed the line from information delivery to business analytics Big Data OLAP is mature and the preferred approach to data analysis for BI More advanced analytics coming Speed Interoperability Users progressively need to query and study larger datasets, know as big data Reporting & analysis tools don t interoperate much or deeply Need for in-line analytics to guide customer facing apps Great query optimization improvement Users now keep more data live Need for near real-time info., web next best offer Seamless integration into operational workflows (data/event/content analytics) Some apps now have predictive algorithms embedded Economics In the current recession, firms are under pressure to reduce costs. Increasing revenue is harder than ever Data analytics into DWH Event analytics into web space Content analytics into unstructured content space Business analytics helps companies dig deeper than ever before for costs and revenue opportunities. Customer and operations analysis are booming 16

17 Trends in Data Visualization As the 2-dim tables cannot accommodate the results of adv d analytics, then it is expected the user interfaces of dashboards, scorecards, analyses, and reports, and portals to become visual Mega Trends Drivers Targets Size Speed Interoperability Economics More users demand dashboards Big data is now the norm Analytics is booming Dashboards, scorecards, and portals need frequent refresh Ad hoc queries need speed, especially for analytics As report/analysis varies, users need to access new data easily Need for in-line analytics to guide customer facing apps, etc. In the current down economy, capital budgets for enterprise BI are frozen or cut Dep t budgets relatively liquid Data vis tion supports growing user communities Visualizations must scale to data size Analytic relations are best viz d. Visualization tools are optimized for fast queries, even when queries are distributed, multidimensional, ad hoc, and repetitive Viz tools have optimized interfaces to go directly at source data Visualizations tend to be Web or service based; hence easy to embed Data viz tools are inexpensive compared to large multi-tool platforms for business intelligence Data viz adapts well to dep t use 17

18 Mobile BI The adoption and maturity of mobile BI fall behind other mobile enterprise applications, mainly due to the lack of specific business use cases and tangible ROI, and inadequate smartphone screen and keyboard forms However, larger form factor devices such as tablets and innovative approaches to online/offline BI technical architecture will boost mobile BI adoption and maturity in the near future (Source: Forrester) ipad-like interface, with touch-screen interactivity and vivid LED-backlit display, shows visual elements such as dashboards to great effect Knowledge workers will soon expect all key business processes and relevant information to be available wherever they are Source: Aberdeen

19 Open Source Software A new-but-promising technology The recession has driven-up interest in low-cost OSS. The OS software are being used at an unprecedented level, but within a minority of committed organizations. This is typical of a new-but-promising technology; it s hard to get users to try it but they tend to grow its usage once they try it. OPEN SOURCE SOFTWARE TYPE GROWTH PLAN TO USE (COMMITMENT) DBMS 48% 17% Data Integration (DI) Tools Business Intelligence (BI) Tools 41% 27% 41% 26% Nowadays, more than half of organizations seriously consider low-cost options for data warehouse platforms. Three types of OSS are poised for good growth in terms of usage with data warehouse platforms. Open source software used so far for DW/DI/BI, in most cases does complement and not replace traditionally licensed enterprise software. Source: TDWI Technology Survey 19

20 Gartner - Data Warehousing Trends for the CIO, Optimization and Performance Data Warehouse Appliances The Intensive POC Data Warehouse Mixed Workloads The Resurgence of Data Marts Column-Store DBMSs In-Memory DBMSs Data Warehouse as a Service and Cloud Using an Open-Source DBMS to Deploy the Data Warehouse 20

21 Forrester - Top 10 BI Predictions for It s all about getting things done. Standards, a single version of the truth, and enterprise-grade platforms continue to be important, but individual BI tools with the functionality to get things done trumps standards 2. Enterprises will learn to live with multiple BI tools. Forrester client inquiries about how to live with multiple BI tools far exceed inquiries about platform consolidations 3. More BI will move into the hands of end users. IT will learn not fight it or risk becoming irrelevant. IT will also learn not to fight spreadmart/spreadsheet wars or risk becoming irrelevant. Why? See prediction #1 4. Those BI platforms that support managed end user self-service will become more popular. It s like the story of Goldilocks and the three bears. No management or control is not acceptable, but too much control does not work. Finding the right win-win combination that combines the flexibility and agility that self-service brings with behind-the-scenes monitoring and adjusting will become the name of the game 5. Mobile BI will go mainstream. One needs to make decisions when and where they need to be made. Not when I get back to the office, which may be too late 21

22 Forrester - Top 10 BI Predictions for Cloud BI will continue to chip away at on-premises BI, but it s still a long road ahead. Heavy customization and integration of enterprise BI platforms, tools, and applications done by subject-matter experts and consultants will not go away 7. BI-specific DBMSes (in-memory, others) will go mainstream 8. Big data will start to move out of silos and into enterprise IT. IT will start to learn how to live with it 9. Exploration (without preconceived notions, prebuilt fixed data models, or up-front specific questions in mind) will be the new bread and butter of BI suites in addition to reporting, querying, OLAP, and dashboards/data visualization. 10. BI users will start demanding and vendors will start delivering BI tools integrated with and collaboration platforms. Just integrating BI with Excel is no longer enough. 22

23 Thank you!