Evolving Data Warehouse Architectures

Transcription

1 TDWI research Second Quarter 2014 BEST PRACTICES REPORT Evolving Data Warehouse Architectures In the Age of Big Data By Philip Russom tdwi.org

2 Research Sponsors Research Sponsors Actian Cloudera Datawatch Corporation Dell Software HP Vertica MapR Technologies

3 TDWI research BEST PRACTICES REPORT Second Quarter 2014 Evolving Data Warehouse Architectures In the Age of Big Data By Philip Russom Table of Contents Research Methodology and Demographics 3 Executive Summary 4 Introduction to Data Warehouse Architectures 5 Various Forms and Components of Data Warehouse Architectures 5 Business and Technology Drivers behind Architectural Evolution 8 Why Is DW Architecture Important? 10 Problems and Opportunities for Data Warehouse Architectures 12 Benefits of Effective Data Warehouse Architectures 13 Barriers to Success with Data Warehouse Architectures 14 Multi-Platform Data Warehouse Environments 16 Workload-centric DW Architecture 16 Distributed DW Architecture 17 A Distributed DW Architecture Is Both Good and Bad 17 From the Single-Platform EDW to the Multi-Platform DWE 18 Architectural Strategies for EDWs versus DWEs 18 Hadoop s Roles in Data Warehouse Architectures 20 Promising Uses of Hadoop that Impact DW Architectures 20 Integrating HDFS with an RDBMS Alleviates the Limitations of Both 22 Analytics and Reporting have Different Requirements for DW Architecture 24 DW Architectures for Reporting 24 DW Architectures for Analytics 24 Trends in Data Warehouse Architectural Components 25 Components Used Most Commonly Today 27 Components to be Adopted Within Three Years 28 Components Excluded From Users Plans 29 Vendor Platforms and Tools as DW Architectural Components 31 Top Ten Priorities for Data Warehouse Architectures by TDWI (The Data Warehousing Institute TM ), a division of 1105 Media, Inc. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. requests or feedback to info@tdwi.org. Product and company names mentioned herein may be trademarks and/or registered trademarks of their respective companies. tdwi.org 1

4 Evolving Data Warehouse Architectures About the Author PHILIP RUSSOM is a well-known figure in data warehousing and business intelligence, having published over 500 research reports, magazine articles, opinion columns, speeches, Webinars, and more. Today, he s the TDWI Research Director for Data Management at The Data Warehousing Institute (TDWI), where he oversees many of the company s research-oriented publications, services, and events. Before joining TDWI in 2005, Russom was an industry analyst covering BI at Forrester Research and Giga Information Group. He also ran his own business as an independent industry analyst and BI consultant and was contributing editor with leading IT magazines. Before that, Russom worked in technical and marketing positions for various database vendors. You can reach him at on Twitter, and on LinkedIn at linkedin.com/in/philiprussom. About TDWI TDWI, a division of 1105 Media, Inc., is the premier provider of in-depth, high-quality education and research in the business intelligence and data warehousing industry. TDWI is dedicated to educating business and information technology professionals about the best practices, strategies, techniques, and tools required to successfully design, build, maintain, and enhance business intelligence and data warehousing solutions. TDWI also fosters the advancement of business intelligence and data warehousing research and contributes to knowledge transfer and the professional development of its members. TDWI offers a worldwide membership program, five major educational conferences, topical educational seminars, role-based training, on-site courses, certification, solution provider partnerships, an awards program for best practices, live Webinars, resourceful publications, an in-depth research program, and a comprehensive website, tdwi.org. About the TDWI Best Practices Reports Series This series is designed to educate technical and business professionals about new business intelligence technologies, concepts, or approaches that address a significant problem or issue. Research for the reports is conducted via interviews with industry experts and leading-edge user companies and is supplemented by surveys of business intelligence professionals. To support the program, TDWI seeks vendors that collectively wish to evangelize a new approach to solving business intelligence problems or an emerging technology discipline. By banding together, sponsors can validate a new market niche and educate organizations about alternative solutions to critical business intelligence issues. Please contact TDWI Research Director Philip Russom (prussom@tdwi.org) to suggest a topic that meets these requirements. Acknowledgments TDWI would like to thank many people who contributed to this report. First, we appreciate the many users who responded to our survey, especially those who agreed to our requests for phone interviews. Second, our report sponsors, who diligently reviewed outlines, survey questions, and report drafts. Finally, we would like to recognize TDWI s production team: Jennifer Agee, Michael Boyda, and Denelle Hanlon. Sponsors Actian, Cloudera, Datawatch Corporation, Dell Software, HP Vertica, and MapR Technologies sponsored the research and writing of this report. 2 TDWI research

5 Research Methodology and Demographics Research Methodology and Demographics Report Purpose. This report educates users about the many directions data warehouse architectures are evolving. Big data is a major driver of change with its burgeoning size, sources, frequency of delivery, and diversity of structures. In addition, the adoption of advanced analytics and real-time operation is equally influential on DW architectures. To assist users, many new products and technologies have arrived recently from software vendors and the open source community. This report describes all of the above and more. Terminology. For the purposes of this report, big data is characterized as very large data sets (multi-terabyte or larger) that usually consist of a wide range of data types (relational, multi-structured data, text, and so on) from numerous sources, including relatively new ones (Web applications, machines, sensors, and social media). Survey Methodology. In November 2013, TDWI sent an invitation via to the data management professionals in its database, asking them to complete an Internet-based survey. The invitation was also posted in Web pages, newsletters, and publications from TDWI and other firms. The survey drew responses from 720 survey respondents. From these, we excluded incomplete responses and respondents who identified themselves as academics or vendor employees. The resulting completed responses of 538 respondents form the core data sample for this report. Research Methods. In addition to the survey, TDWI Research conducted many telephone interviews with technical users, business sponsors, and recognized data management experts. TDWI also received briefings from vendors that offer products and services related to data warehouse architectures and big data. Survey Demographics. The majority of survey respondents are IT professionals (76%), whereas the others are consultants (18%) and business sponsors or users (6%). We asked consultants to fill out the survey with a recent client in mind. The financial services industry (15%) dominates the respondent population, followed by consulting (14%), software/internet (10%), healthcare (8%), insurance (8%), and other industries. Most survey respondents reside in the U.S. (47%) or Europe (21%). Respondents are evenly distributed across all sizes of organizations. Position Corporate IT professionals 76% Consultants 18% Business sponsors/users 6% Industry Financial services 15% Consulting/professional services 14% Software/Internet 10% Healthcare 8% Insurance 8% Telecommunications 6% Manufacturing (non-computers) 6% Retail/wholesale/distribution 6% Education 5% Government: state/local 3% Utilities 3% Other 16% ( Other consists of multiple industries, each represented by 2% or less of respondents.) Geography United States 47% Europe 21% Asia 9% Canada 9% Australia / New Zealand 6% Mexico, Central or South America 5% Middle East 2% Africa 1% Company Size by Revenue Less than $100 million 19% $ million 15% $500 million $1 billion 12% $1 5 billion 15% $5 10 billion 8% More than $10 billion 15% Don t know 16% Based on 538 survey respondents. tdwi.org 3

6 Evolving Data Warehouse Architectures For years, data warehouses have been trending toward multiplatform environments. New platforms in the DW environment include Hadoop, NoSQL databases, columnar databases, and DW appliances. Most architectures are built as two primary layers: logical and physical. Multi-platform DW environments still have an EDW, but it s complemented by other, newer platform types. TDWI expects Hadoop and similar platforms to be in common use in DW environments within three years. Executive Summary In the early days of data warehousing, most data warehouses (DWs) were centered around a singleinstance database, plus a few edge systems for data marts, operational data stores (ODSs), and data staging. Over the years, TDWI has seen a strong trend toward more and more edge systems, many of them designed by vendors or users for workloads for which the average DW is not optimized, especially workloads for new forms of big data, processing for advanced analytics, managing tens of terabytes of source data, and data for real-time operations. The modern data warehouse environment (DWE) still includes an enterprise data warehouse (EDW), but the EDW is complemented by several other types of data platforms. A DWE includes the usual marts, ODSs, and staging areas, as well as newer standalone platform types for DW appliances, columnar databases, NoSQL databases, Hadoop, real-time technologies, and various analytic tools. Of course, some organizations have multiple warehouses, as well. Given the rising complexity, data warehouse architecture is more critical than ever in order to make sense of, govern, and optimize the complicated multi-platform DWEs that many user organizations are building. When asked what components a DW architecture should include, users answering this report s survey identified data standards (71%), a logical design (66%), and a physical plan (56%). The user consensus is that DW architecture includes multiple layers and components, and success usually entails balancing a logical design (mostly data models and standards) with a physical plan (mostly a portfolio of servers and how they integrate). The leading drivers for change in DW architectures are advanced analytics (57%), big data management and leverage (56%), and real-time operations (41%). Eighty-four percent feel that DW architecture is an opportunity when it achieves the leading drivers. The top barrier to success is the current lack of skills with big data and advanced analytics that most DW teams suffer. Completely pure DW architectures are rare. For example, only 15% of respondents have a single EDW with no additional data platforms. Even when a DW has a well-defined architecture, users will ignore the plan for some data domains, platforms, or workloads. That s why 31% have a core EDW, but with additional platforms in the DW environment, typically for analytic, big data, and real-time workloads. An increasing number of DW architectures are hybrids; for example, it s common that Inmon and Kimball approaches are applied within the same DW environment. A small but prominent community of DW professionals is integrating the Hadoop Distributed File System (HDFS) and other Hadoop tools into their DW environments. There are many areas within a DW architecture where HDFS shows promise, namely data staging areas, data archives, and repositories of big data for analytics. Hadoop is promising because it handles data types that the average DW cannot, namely unstructured data (text, audio, video) and semi-structured documents (XML, JSON). Yet, relational technologies are still required for most reporting, OLAP, and performance management functions supported by the average DW, so relational databases, SQL, and other relational technologies are not imperiled by Hadoop. Instead, TDWI expects them to be commonly integrated with Hadoop technologies in DW environments as early as To help users prepare for new DW architectures, this report quantifies trends in data warehouse architectures and catalogs newly available, relevant technologies. The report also documents how successful organizations are evolving their architectures to leverage new business opportunities for big data. The goal is to provide data warehouse professionals and their business counterparts with the information they need before planning the next generation of their logical data warehouse architecture and its physical deployment. 4 TDWI research

7 Introduction to Data Warehouse Architectures Introduction to Data Warehouse Architectures Various Forms and Components of Data Warehouse Architectures Architecture means many things to many people, especially when the term is applied to IT systems. The multiple definitions and forms of architecture are a natural consequence of the fact that IT systems almost always have an explicit overall architecture, and many system layers and components have their own architectures, as well. For example, many data warehouses are designed as multiple architectures that layer atop each other and work together: DWs are often designed as multiple architectural layers and components that work together. Logical architecture mostly data models and their relationships, with a focus on how these represent organizational entities and processes Data standards including standards for data modeling, data quality metrics, interfaces for data integration, programming style, format standards (e.g., 5-digit versus 9-digit ZIP codes), etc. Physical architecture mostly a plan for deploying data and data structures on the servers of the system architecture System architecture a topology of hardware servers and software servers, plus the interfaces and networks that tie them together In addition, DW architectures usually integrate with or overlap with related architectures for data integration (ETL, data sync, replication), business intelligence (reporting and analytics), and enterprise data (typically source data that exists within the application architectures of operational systems). Note that all these forms of architecture are complementary and never mutually exclusive. This means that it s normal for multiple architectures to apply simultaneously, even in a single IT system, such as a data warehouse. It s linguistically convenient for data warehouse professionals to call the combination of logical and physical layers the data warehouse architecture. Whether you consider it one architecture or two or four is a matter of semantic hairsplitting. The important point is that all four layers and components serve different purposes, have unique requirements, and must work together for a successful warehouse. To get a sense of how data warehouse professionals and related personnel define architecture nowadays and with which priorities this report s survey asked: What do you think DW architecture is? (See Figure 1, page 7.) Standards and rules. This is the most common component of DW architecture identified by survey respondents (71%). As one respondent put it, DW architecture must include: standards and guidelines for the interaction of different platforms and data structures. The standards typically guide how data is modeled, the formation of data quality metrics, metadata requirements, development methods, programming style, and which interfaces are to be used in which data integration situations. Some standards are more like business rules that ensure data integrity, such as inventory quantity cannot be negative or a patient cannot be both male and pregnant. Data standards and business rules are the leading priorities for data architecture. Not all architects agree that data standards and rules are truly architectural. To some, they are an adjunct to architecture, whereas others consider them integral to the logical layer of architecture. tdwi.org 5

8 Evolving Data Warehouse Architectures The logical layer is the foundation that other architectural layers are built atop. The physical layer is often portrayed as just a collection of servers, but it s really about how a logical data design is deployed on servers. The fact that architecture-free DWs are rare indicates architecture is a critical success factor. Logical design. Every well-designed data warehouse begins with a carefully planned logical design that focuses on the DW s data structures and the patterns or relationships among them (66%). Likewise, all updates and versions of a DW should start with a revision of the logical design, for two reasons: The logical design is where the architecture is aligned to business requirements, and the physical and system layers are loosely based on the logical design. According to a respondent, DW architecture is a structure aimed at giving the users multi-sectional views of the information through both relational and dimensional models. That comment reminds us that most DW professionals consider data modeling to be a primary skill for architecting a data warehouse perhaps the only skill. Although data modeling is very important and it s especially prominent in the logical layer it s not all there is to designing a data warehouse architecture. Physical plan. Focused on deploying data, data structures, and data platforms (56%), this is where the architect decides where the components of the logical design will go in the physical world. It s not necessarily a one-to-one mapping; many tables or marts may be specified in the logical plan, but they need not all be in one database or on one server in the physical deployment. Likewise, a single data structure on the logical layer may be normalized into multiple components on the physical level for the sake of performance or to avoid atomic redundancy. 1 As a different example, some data is updated so frequently that any aggregates or calculations based on it should be done at run time, as needed, in a federated or virtual manner, instead of ahead of time. This is sometimes called virtual data warehousing or the logical data warehouse, because it seems like the logical layer exists without the physical one. In a sense, physicality still exists, though not instantiated until run time. Some DW architects prefer to divide the physical plan into two layers: a plan for how data structures will be deployed and a separate but closely related plan or topology for hardware and software servers (which also includes how the servers integrate and interoperate). Isolating the data plan from the server plan makes sense when both are very complicated, when the two layers are owned by separate teams, or when you need to avoid server vendor lock-in. Much attention has been focused on server topology in recent years and that s where this report focuses for the most part because this is where most of the action is happening relative to DW architecture. For example, consider the many new types of platforms that have arrived recently from vendors and the open source community, plus new practices on the part of user organizations. These include DW appliances, columnar DBMSs, NoSQL DBMSs, Hadoop, and various analytic tools. They supply users growing demands for multi-platform system architectures under the assumption that having a software portfolio of diverse data platform types enables users to move data and workloads to the platforms that are best suited to them. The bulk of this report drills into multiplatform DW environments as manifestations of the physical layer of DW architecture. Collection of data and platforms without a plan. Some data warehouses work despite being a disorganized bucket of inconsistent data sets on disconnected data platforms. Certainly, this is not a recommended practice. These are rare cases (12%) because the likelihood of success is low a disorganized bucket provides limited value for an organization. For that reason alone, most DWs have a functioning architecture (as we ll see later, in the survey data for Figure 2). In a related issue, note that some areas within a data warehouse architecture demand more rigorous application of data standards than others. For example, compare the squeaky clean and carefully modeled data that goes into financial reports to the barely scrubbed data that s appropriate to exploratory analytics. By comparison, data for analytics sometimes seems like a collection without a plan. But the plan is still there; it s expressed in the analytic layer at run time, instead of in a priori processing in the data layer, as is typical of data for reports. 6 TDWI research 1 In a different survey question (not charted in this report), respondents reported that DW architecture is the job of a data warehouse architect (38%), everyone on the BI and DW team (23%), or a BI or DW director (19%).

9 Introduction to Data Warehouse Architectures Inmon versus Kimball. In the early days of data warehousing, influential architectural concepts were developed by two important pioneers in the field: Bill Inmon 2 and Ralph Kimball. 3 Throughout the 1990s, data warehouse professionals hotly debated which was better, Inmon s top-down approach focused on operational data in third normal form (3NF) or Kimball s bottom-up approach focused on dimensional models expressed in integrated data marts. Most architectures and approaches end up being hybrids after evolving. The Inmon versus Kimball dichotomy is still with us (12% in Figure 1), but the argument has cooled considerably in recent years because most DW architects nowadays employ multiple approaches, selecting the one that best fits a certain data structure, data domain, data platform server, or development team. Some DWs incorporate approaches by both Inmon (e.g., with 3NF data for exploratory analytics) and Kimball (e.g., with multidimensional data marts for business performance management). Furthermore, most architectures, standards, and approaches are a matter of degree. Just because you have a plan doesn t mean you should follow it all the time, as explained in this report s next User Story (below). Other. One respondent who selected the answer Other typed in his formula for a DW architecture, which stresses the combination of logical and physical plans: Business process architecture (metadata) + logical plan + data acquisition and distribution infrastructure topology [physical plan]. Another respondent expressed a similar sentiment: A coherent set of principles, models, and frameworks that guide the design and implementation of the (extended) enterprise data warehouse, its components, and their interrelationships at the conceptual, logical, and physical level. Yet another respondent expressed the opinion that DW architecture includes data integration and BI/reporting architectures, as well. Key to most users conception is the combination of logical and physical architectures. What do you think DW architecture is? Select all that apply. Standards for data models, quality, metadata, interfaces, development methods, etc. A logical design for data structures and the patterns or relationships among them 66% A plan for physically deploying data, data structures, and data platforms 56% A collection of data structures and data platforms, with little or no overall plan 12% The old Inmon versus Kimball argument 12% Other 5% 71% Figure 1. Based on 1,197 responses from 538 respondents; 2.2 responses per respondent, on average. USER STORY Most architectures end up being hybrids, despite standards, plans, and preferences. According to a user interviewed for this report: I ve worked on a lot of data warehouses, and I ve never seen one implemented 100% Inmon or 100% Kimball, although there s usually a tendency toward one or the other. Centralized and distributed architectures are rarely 100% either. And most ETL hub architectures are polluted with a few point-to-point interfaces that skirt the hub. I think all IT systems benefit from a guiding architecture for the sake of consistent standards, data integrity, ease of maintenance, depth of integration, business alignment, and the usual dogma. Even so, pedantic adherence to architecture isn t desirable, because a few unique situations inevitably arise that are best served by a minority of non-standard solutions. 2 Bill Inmon s approach is well documented in his books Building the Operational Data Store, Managing the Data Warehouse, and Data Warehouse Performance. 3 Ralph Kimball s approach is well documented in his books The Data Warehouse Toolkit, The Data Warehouse Lifecycle Toolkit, and The Data Warehouse ETL Toolkit. tdwi.org 7

10 Evolving Data Warehouse Architectures Business and Technology Drivers behind Architectural Evolution First, let s establish that data warehouse architectures exist and are in use by the majority of organizations that have a data warehouse. See Figure 2, where a whopping 79% of survey respondents report having a data warehouse with an architecture. Second, let s corroborate that data warehouse architectures are, indeed, evolving. (See Figure 3.) Roughly half of users surveyed (54%) report experiencing a moderate level of evolution, whereas roughly a quarter (22%) are seeing dramatic evolution. Another quarter are experiencing minimal change. Does your primary enterprise data warehouse have an architectural design? Is the architecture of your data warehouse environment evolving? Don t know 4% Don t know 2% No 18% 79% Yes No except when 22% DW updates adjust architecture slightly 54% Yes moderately Yes dramatically 22% Figure 2. Based on 538 respondents. Figure 3. Based on 538 respondents. What s driving change in data warehouse architectures? To get at the heart of the matter, this report s survey asked: What technical issues or practices are driving change in your DW architecture? (See Figure 4.) Most technical changes to data warehouse architectures are to provide better platforms for analytics, big data, and real-time operation. Advanced analytics. Most changes being made in DW architectures are to make a warehouse (or its extended environment) more conducive to advanced forms of analytics (57%), especially those enabled by technologies for data mining, statistics, or complex SQL. Note that online analytic processing (OLAP) is still the leading method and technology for analytics, but it s being complemented by other forms that are more focused on information discovery. Big data. As a close second, almost as many changes are being made to accommodate the increasing data volumes (56%) and non-relational data (25%) of big data. In a related data management issue, data virtualization is on the rise (23%). Real time. To keep pace with accelerating business practices, BI and DW infrastructure is being retrofitted with technology for real-time operation (41%), and sometimes for streaming data (15%). This is especially apparent in a popular practice called operational BI, which gathers very fresh data from operational applications and presents it in management dashboards and similar reports all within seconds or milliseconds. 8 TDWI research

11 Introduction to Data Warehouse Architectures What technical issues or practices are driving change in your DW architecture? Select all that apply. Advanced analytics (mining, statistics, complex SQL; not OLAP) 57% Increasing data volumes 56% Real-time operation based on fresh data 41% Business performance management 38% Online analytic processing (OLAP) 30% Non-relational data 25% Virtualization of data 23% Cloud adoption 21% Streaming data 15% Other 9% Figure 4. Based on 1,688 responses from 538 respondents; 3.1 responses per respondent, on average. Technical drivers aside, a number of business issues and evolving practices are also driving change in DW architectures. (See Figure 5.) Market pressures. Enhancements to DW architectures can contribute to competitiveness (45%) and fast-paced business processes (43%), as more organizations compete on analytics or simply run the business based on facts developed in BI/DW infrastructure. Compliance (29%). Many organizations have beefed up data security and role-based user access functions to comply with privacy and security regulations laid out in federal legislation such as HIPAA, Sarbanes-Oxley, and the USA PATRIOT Act. Others are scrambling to support the data standard known as the electronic medical record (EMR) mandated in the federal Affordable Care Act. Yet other organizations are altering their DW architectures to meet other definitions of compliance, such as service-level agreements (for performance and high availability) and internal data governance guidelines. Departmental BI. In some organizations, there is a shift toward funding (29%) and sponsorship (26%) at the departmental level. This is because departmental budgets are sometimes more fluid than capital ones, and most analytic applications are departmentally focused. One way to respond is to restructure a central DW into a componentized architecture, so that departmental assets are distinct from shared enterprise assets (perhaps on standalone servers, although more and more on clouds). An equally common response, however, is for a department to build its own redundant infrastructure for BI, DW, and analytics. In a similar vein, TDWI regularly sees departmental BI deployed atop a third-party cloud funded by the department, not IT. Organizational issues. A variety of changes to an organization can drive changes to IT systems and data warehouses, including reorganizations (25%), centralizing business control (30%), departmental power struggles (19%), and mergers and acquisitions (18%). Most business-driven changes to DW architectures are in response to markets, compliance, and departmental BI. tdwi.org 9

12 Evolving Data Warehouse Architectures What business issues or practices are driving change in your DW architecture? Select all that apply. Competitiveness 45% Fast-paced business processes 43% Centralizing business control 30% Compliance 29% Funding 29% Sponsorship 26% Reorganizations 25% Departmental power struggles 19% Mergers and acquisitions 18% Other 6% Figure 5. Based on 1,453 responses from 538 respondents; 2.7 responses per respondent, on average. USER STORY Success comes from a good logical design based mostly on business structures. Gary DeBoer has been a data warehouse professional and data architect at Saks Fifth Avenue, California Power Exchange, Boeing, Perot Systems, EDS, and Clearwire. To me, data warehouse architecture borrows structures from the business, applications, IT infrastructure, analytics, enterprise data, and so on. Pieces of these combine into an information architecture for the warehouse, which is sometimes called a conceptual data model or a logical design. In my experience, the best success comes when the logical design is based primarily on the business how it s organized, its key business processes, and how the business defines prominent entities like customers, products, and financials. Start with these and similar entities as the core business data; the core doesn t change, because the essence of business doesn t change. Once a logical data architecture is designed, you can start considering how it will be deployed physically and virtually. How enterprise is the data warehouse? With what enterprise applications will the warehouse integrate? How does it fit with IT infrastructure and enterprise data standards? In short, recognize that logical design and physical deployment are two different but related architectures; always give priority to the logical over the physical. The vote is in: the vast majority of users consider DW architecture important. Why Is DW Architecture Important? Anecdotally, TDWI has seen users concern for data warehouse architecture increase noticeably since about Many users are actively focused on architecture, as seen in the increased use of job titles such as data warehouse architect and enterprise data architect. To gauge the urgency of this activity, this report s survey asked: How important is architecture for the success of your data warehouse and related platforms today? (See Figure 6.) A mere 2% of respondents feel that data warehouse architecture is not a pressing issue. To the contrary, the vast majority consider DW architecture to be extremely important for success (79%), while many others consider it moderately important (19%). 10 TDWI research

13 Introduction to Data Warehouse Architectures How important is architecture for the success of your data warehouse and related platforms today? Moderately important 19% Not a pressing 2% issue currently 79% Extremely important Figure 6. Based on 538 respondents. Clearly, the data warehouse professionals and others who responded to the survey are nearly unanimous in their zeal for DW architectures. But why? To get their unvarnished opinions, the survey asked the open-ended question: In your own words, why is the architecture of a data warehouse important? The respondents comments are loaded with both wisdom and wit, as seen in the representative insights reproduced below: In your own words, why is the architecture of a data warehouse important? We need the structure; otherwise it s chaos. VP of IT, education, Asia Like anything to be done correctly, having a blueprint is necessary, and this is what architecture implies. Without it would be similar to building a home without a blueprint, which informs each of the subcontractors what they are required to do, from putting in the foundation to laying a roof. In other words, the potential for failure is high without a sound architecture that includes not just the blueprint but also the standards, processes, applications, and infrastructure. DW architect, telecommunications, U.S. DW architecture is a critical foundation for structure, performance, future expansion, business alignment, and user value. Architecture helps to save money in development and maintenance and to leverage knowledge yield for users. CEO, consulting firm, Europe Every DW solution is tailored to suit the business needs of the organization in question. Architecture is imperative to ensure that the objectives are being met and that all enhancements and changes to the DW are inclusive to the general business objective. Environmental scientist, federal government, U.S. Users are increasingly expecting the ability to access large volumes of data across a variety of data sources, and they expect to be able to do it fast. A robust data warehouse architecture is needed to support structured, unstructured, and multi-source data, plus to support real-time, operational, and predictive analytics. BI manager, manufacturing, U.S. It is a model or representation that enables testing the DW design against business needs. IT professional, software/internet, Canada As the Cheshire Cat noted in Alice in Wonderland, if you don t know where you re going, any road will get you there. We do not want to end up with an architecture like that of the Winchester Mystery House. Sales consultant, software/internet, U.S. tdwi.org 11

14 Evolving Data Warehouse Architectures Without an architectural plan there is little or no understanding of the content, structure, and relationships of the data within the warehouse. Without that understanding, the data is of limited value to the enterprise. IT specialist, utilities, U.S. It provides a plan and framework for how you are going to organize data in your environment so that it is understandable by the business, achieves the goals of data integration, stores the information efficiently and securely, and makes sense as a cohesive whole, rather than just a store of operational data. VP and technology manager, financial services, U.S. Data without architecture is just a mess. Director, hospitality and travel, Europe User perception says that DW architecture is mostly an opportunity, not a problem. Problems and Opportunities for Data Warehouse Architectures Establishing and sustaining data warehouse architecture takes time, specialized skills, team collaboration, and organizational support. With these requirements in mind, this report s survey asked: Is designing and maintaining data warehouse architecture mostly a problem or mostly an opportunity? (See Figure 7.) The vast majority consider DW architecture an opportunity (84%). Conventional wisdom today says that architecture gives a data warehouse and dependent programs for reporting and analytics greater data quality, usability, maintainability, and alignment with business goals. A small minority consider DW architecture a problem (16%). Data warehouse architecture faces a number of technical and organizational challenges, as noted in the next section of this report, but few organizations find that the challenges outweigh the benefits of architecture. Is designing and maintaining data warehouse architecture mostly a problem or mostly an opportunity? Select only one. Problem 16% 84% Opportunity Figure 7. Based on 538 respondents. 12 TDWI research

15 Problems and Opportunities for Data Warehouse Architectures Benefits of Effective Data Warehouse Architectures The strongest trend in DW architecture right now is the movement toward even more data platforms and related tools within the physical layer of the extended data warehouse environment. To test users perceptions of data warehouse platform diversification, this report s survey asked respondents: If your organization were to further diversify the platform types in your data warehouse architecture, which business and technology tasks would improve? (See Figure 8.) Most forms of analytics benefit from having more types of data platforms. This includes all data analytics, in general (61%), which bubbled up to the top of the list in Figure 8. In fact, many of the new platforms available are for analytics (although sometimes described as DW platforms), including data warehouse appliances, columnar databases, and Hadoop. Information exploration and discovery (43%) is also at the top of the list. This makes sense because the kinds of analytics on the rise right now (based on technologies for SQL, NoSQL, mining, statistics, and natural language processing) are all about discovering facts about the business that were previously unknown. Likewise, common types of analytic applications benefit from a multi-platform DW environment, including customer-base segmentation (19%), definitions of churn (18%), sentiment analytics (18%), and fraud detection (16%). The top beneficiaries of multi-platform DWs are analytics, business value, data breadth, and realtime operations. A more diverse platform portfolio can aid a business. Additional platforms are key to addressing new business requirements (36%), especially data-oriented ones such as analytics (61%), more numerous and accurate business insights (34%), business optimization (30%), and understanding business change (20%). A diverse platform portfolio can handle a diverse range of data types. This is key to embracing the unstructured and schema-free data types found in most big data. Once these new data types are addressed, an organization gets greater business value from big data (34%), broader data sourcing for analytics (33%), and more data for data warehousing (22%). Handling data in real time usually requires an additional purpose-built system. Traditional relational databases and batch-oriented Hadoop systems were not built for real-time operations (33%), although many organizations need faster business processes (26%). Along with retrofitting real-time functions onto platforms that lack them (e.g., Apache Storm on Hadoop or in-memory analytics on RDBMSs), many users add additional standalone tools for event processing to their warehouse environments. Adding low-cost platforms to a DW environment makes big data more affordable. Hadoop and NoSQL platforms give more favorable economics to big data management practices, such as data staging for data warehousing (20%) and data archiving (16%). In a similar vein, many TDWI members have extended their DW environments with new brands of analytic databases (typically appliances and columnar RDBMSs). These are optimized for analytics to a degree that the average report-oriented DW isn t, and the analytic databases are more affordable for most configurations. tdwi.org 13

16 Evolving Data Warehouse Architectures If your organization were to further diversify the platform types in your data warehouse architecture, which business and technology tasks would improve? Select seven or fewer. Data analytics 61% Information exploration and discovery 43% Addressing new business requirements 36% Greater business value from big data 34% More numerous and accurate business insights 34% Broader data sourcing for analytics 33% Real-time operations 33% Business optimization 30% Faster business processes 26% Recognition of sales and market opportunities 24% More data for data warehousing 22% Data staging for data warehousing 20% Understanding business change 20% Segmentation of customer base 19% Definitions of churn and other customer behaviors 18% Sentiment analytics and trending 18% Data archiving 16% Fraud detection 16% Cost containment 15% Identification of root causes 15% Understanding website visitor behavior 13% Effective use of machine data 12% Quantification of risks 10% Other 4% Figure 8. Based on 3,083 responses from 538 respondents; 5.7 responses per respondent, on average. Success requires solid skills, sponsorship, data management, and funding. Barriers to Success with Data Warehouse Architectures As we just saw, diversifying the types of data platforms in an extended data warehouse architecture has its benefits, at least in users perceptions. Diversifying also has a few barriers. To get a sense of the challenges of such multi-platform DW environments, this report s survey asked respondents: In your organization, what are the top potential barriers to further diversifying the platform types in your data warehouse architecture? (See Figure 9.) A skills gap is the largest barrier to success with new DW platforms and data types. Back in the day, building a multi-platform DW environment simply involved deploying a few data marts and operational data stores on their own standalone database instances. An equivalent environment nowadays is far more challenging, because it entails working with the new stuff data management tools and data types that are new to an organization. Due to the newness, many organizations that attempt a modern, diversified DW environment find that they have inadequate staffing or skills (47%). Their immaturity with new data types and sources (23%) plus new technologies for Hadoop, event processing, and so on make them unprepared for the complexity of multi-platform designs (25%) TDWI research 4 In a different survey question (not charted in this report), 67% of respondents report that their current data warehouse team does not have the skills for big data, new data types, new methods, and emerging technologies.

17 Problems and Opportunities for Data Warehouse Architectures As usual, organizational and business issues should be settled first. These include data ownership and other politics (43%), a lack of business sponsorship (38%), and a lack of a compelling business case (25%). A number of data management issues can get in the way. Technical users must prepare to address data integration complexity (36%), poor quality of data (34%), lack of enterprise data architecture (29%), and data security, privacy, and governance issues (25%). As with any new IT initiative, you won t get far without proper funding. In particular, account for the cost of acquiring multiple platforms (25%) and the cost of administering multiple platforms (27%). In your organization, what are the top potential barriers to further diversifying the platform types in your data warehouse architecture? Select seven or fewer. Inadequate staffing or skills 47% Data ownership and other politics 43% Lack of business sponsorship 38% Data integration complexity 36% Poor quality of data 34% Lack of enterprise data architecture 29% Cost of administering multiple platforms 27% Poor integration among multiple platforms 26% Complexity of multi-platform designs 25% Cost of acquiring multiple platforms 25% Data security, privacy, and governance issues 25% Lack of compelling business case 25% Existing data warehouse architecture 24% Immaturity with new data types and sources 23% Lack of metadata and schema in some big data 17% Real-time data handling 16% Getting started with the right project 15% Loading large data sets quickly and frequently 14% Scalability problems with big data 10% Processing queries fast enough 9% Other 3% Figure 9. Based on 2,758 responses from 538 respondents; 5.1 responses per respondent, on average. tdwi.org 15

18 Evolving Data Warehouse Architectures USER STORY The proliferation of departmental BI may work against a central DW and its architectural standards. Our central data warehouse is still fairly new and a bit under-populated with data, which explains why many departments have set up their own tools and data platforms for BI and analytics, said the BI director at a metals recycling firm. In particular, a number of departments have bought their own tools for reporting, analysis, and data exploration. The tools work okay without a warehouse, because most departmental users are happy looking at data from one application at a time. Of course, departmental BI is problematic, because it sees incomplete views of data, outside the context of the rest of the enterprise, and without standard definitions of business entities like customers and products. We have approval and funding for a major upgrade to our data warehouse and reporting tools, and that will be in place within a year. After that, we ll begin consolidating departmental systems into the new, central BI and data warehouse infrastructure. That way, everyone will have a much broader view of the business, based on consensusdriven definitions of business entities. Logical DW architecture must integrate multiple physical platforms. Multi-Platform Data Warehouse Environments Many enterprise data warehouses (EDWs) are evolving into multi-platform data warehouse environments (DWEs) as users continue to add standalone data platforms to their warehouse tool and platform portfolios. The new platforms don t replace the core warehouse, because it is still the best platform for the data that goes into standards reports, dashboards, performance management, and OLAP. Instead, the new platforms complement the warehouse because they are optimized for workloads that manage, process, and analyze new forms of big data, non-structured data, and realtime data. Users need additional platforms to get new value from new data. The catch is to build a multi-platform data environment without being overwhelmed by its complexity, which a good architectural design can avoid. Hence, for many user organizations, a multi-platform physical plan coupled with a cross-platform logical design is a DW architecture that s conducive to the new age of big data. 5 Workload-centric DW Architecture One way to characterize a data warehouse s architecture is to count the number and types of workloads it supports. According to a 2012 TDWI Research survey on high-performance data warehousing, a little over half of user organizations surveyed (55%) support only the most common workloads, namely those for standard reports, performance management, and online analytic processing (OLAP). The other half (45%) also support workloads for advanced analytics, detailed source data, various forms of big data, and real-time data feeds. The trend is toward the latter. In other words, the number and diversity of DW workloads is increasing due to organizations embracing big data, multi-structured data, real-time or streaming data, and data processing for advanced analytics. The catch is that some data warehouses (whether defined as a vendor s product or a user s design) can handle multiple, concurrent workloads of various types, whereas others cannot. Hence, the mix of workloads and data types that you manage on a central warehouse (or offload to other platforms) often depends on how capable the central warehouse is with multiple, concurrent workloads. Other determining factors include the ownership of data sets and platforms, plus their relative economic costs. 16 TDWI research 5 Some of the material in this section is drawn from a TDWI blog post by Philip Russom, Evolving Data Warehouse Architectures: From EDW to DWE, online at

19 Multi-Platform Data Warehouse Environments Distributed DW Architecture The issue in a multi-workload environment is whether a single-platform data warehouse can be designed and optimized such that all workloads run optimally, even when concurrent. More and more DW teams are concluding that a single-platform DW is no longer desirable. Instead, they maintain a core DW platform for traditional workloads (reports, performance management, and OLAP), but offload other workloads to other platforms. For these organizations, the DW is not going away; it s just being complemented by additional data platforms tuned to workloads that can and should be offloaded from the core warehouse. The diversification of DW workloads leads to distributed architectures for DWs. For example, data and processing for SQL-based analytics are regularly offloaded to DW appliances and columnar DBMSs. A few teams offload workloads for big data and advanced analytics to HDFS, MapReduce, and other NoSQL platforms. It s not just DWs; some users are offloading ETL jobs from an expensive data integration server to less expensive Hadoop. The result is a strong trend toward distributed DW architectures, where many areas of the logical DW architecture are physically deployed on standalone platforms instead of the core DW platform. A Distributed DW Architecture Is Both Good and Bad A distributed architecture is good if your fidelity to business requirements and DW performance leads you to deploy another data platform in your DW environment and the new platform integrates well with others in the distributed architecture, on both logical and physical levels. But it s bad when disconnected systems proliferate uncontrolled, like the errant data marts we all fear. So far, the newest generation of analytic databases and data management platforms is controlled by users far better than the marts of yore. Still, it s essential to be diligent with marts to avoid abuses. Note that the architectural distinctions made here have always been a matter of degree, and will continue to be so. In other words, no architecture is 100% monolithic or 100% distributed. Many are hybrids, and the percentage mix that s right for you depends on the details of your business and technology. DW architectures have always been distributed to some degree. It s just that the degree is more pronounced today. Furthermore, the trend toward distributed DW architectures is not new not by a long shot. For decades, warehouses have wended their way through a variety of edge systems that are deployed on standalone servers off to the side of the warehouse, but integrated with it. This has been true from the dawn of warehousing (as with data marts and operational data stores [ODSs]), although it has recently expanded (with DW appliances and columnar DBMSs), and is now continuing with new types of data platforms (namely NoSQL and Hadoop). Hence, even the new platforms fit comfortably into the well-established tradition of DW edge systems. USER STORY Pure monoliths are rare; most have at least a few satellite systems. According to a user interviewed for this report: We trend toward a monolithic warehouse architecture in that almost all our data for BI and analytics resides in a single RDBMS instance. The RDBMS we just migrated to is known to excel with monoliths, so it works for us from a performance and scalability viewpoint. As with any architecture, there s always a hybrid touch, and so we have a number of secondary platforms, too, as satellites that integrate with the central, monolithic warehouse. For example, all data staging is done on satellite platforms. It s not that our monolith can t handle staging; it s that capacity on the monolith s RDBMS is way too expensive for storing terabytes of detailed source data and pushing down ELT processing. tdwi.org 17