Data Virtualiza on: Percep ons and Market Trends

Transcription

1 Data Virtualiza on: Percep ons and Market Trends BI Leadership Benchmark Report By Wayne Eckerson Director of Research Business Applications and Architecture Media Group TechTarget April 2013

2 Table of Contents Executive Summary... 3 Overview... 4 Agile Data Development... 4 What is Data Virtualization?... 4 Challenges... 6 Use Cases... 7 Perceptions... 8 Survey Results... 9 Market Adoption... 9 Reasons for Not Deploying Data Virtualization Benefits Use Cases Scale Data Sources Challenges Vendors Respondent Advice Conclusion... 18

3 Execu ve Summary Data virtualiza on accelerates me to insight while making it easier for administrators to move, change and consolidate back-end data systems without affec ng downstream applica ons. DATA VIRTUALIZATION SOFTWARE IS NOT NEW it has existed for more than 20 years in one form or another under a variety of names but it is now gaining trac on in a variety of market segments, including business intelligence (BI), applica on development and big data. Despite the vision of an enterprise data warehouse, the reality in most organiza ons is that a mul plicity of data sources exist and cannot (or will not) be physically consolidated. This is where data virtualiza on comes in. Just like its virtualiza on cousin in the data center, data virtualiza on hides the physical implementa on of data behind a generic set of business objects (i.e., metadata). These objects, in effect, form the basis of a data service that business users and applica ons can access without having to go through the me-consuming and costly exercise of physically consolida ng data from heterogeneous systems. As a result, data virtualiza on accelerates me to insight while making it easier for administrators to move, change and consolidate back-end data systems without affec ng downstream applica ons. Today, data virtualiza on is s ll in the early adopter phase, but many more BI professionals are evalua ng the so ware than in the past. Meanwhile, the so ware has improved significantly, overcoming performance and scalability issues that dogged earlier versions of the so ware. Many BI professionals may not even realize they are using data virtualiza on so ware since many BI and ETL vendors have embedded these capabili es into their products. As the big data movement increases the volume and variety of data that business users want to access and examine, data virtualiza on so ware will play an increasingly important role in corporate intelligence architectures. 3

4 Overview Agile Data Development DATA VIRTUALIZATION SOFTWARE IS THE KEY to crea ng an agile, costeffec ve data management infrastructure. That s a bold statement since few organiza ons today have implemented data virtualiza on so ware, at least as an independent set of middleware that connects users with any data throughout an organiza on using a uniform interface. Data virtualiza on creates a logical view of distributed data. Bold, yet true. That s because the data in organiza ons is hopelessly distributed across mul ple opera onal and analy cal systems and, increasingly, external data sources, such as social media and syndicated data services. Tradi onally, organiza ons physically consolidate data within a single environment (e.g., a data warehouse) before building applica ons that query data, but this approach is not very agile. According to The Data Warehousing Ins tute, it takes organiza ons an average of 7.8 weeks to add a new data source to their data warehouses and seven weeks to build a complex dashboard or report. (See TDWI BI Benchmark Report, 2011). Given the fast-pace of business today, this is too slow to meet business needs. This is where data virtualiza on so ware can help. Once implemented, data virtualiza on so ware can accelerate applica on delivery because developers no longer have to source, integrate, and clean data on their own or wait for the IT department to do the work. Instead of hun ng down relevant data using a variety of tools and access methods, developers can use a single tool with a uniform interface to access data both inside and outside the organiza on. Data virtualiza on creates a logical view of distributed data and eliminates the need to always have to physically consolidate it in a local database. All this speeds delivery mes and liberates developers from data collec on grunt work. What is Data Virtualiza on? VIRTUAL VIEW. Data virtualiza on so ware makes data spread across physically dis nct systems appear as a set of tables in a local database a virtual view, if you will. Data virtualiza on so ware consumes virtually any type of data, including SQL, MDX, XML, Web services, flat files, and unstructured data in Hadoop and NoSQL databases, and publishes the data as SQL tables or Web services. When users submit a query, the data virtualiza on so ware calculates the op mal way to fetch and join the data on remote, heterogeneous systems. It then queries the relevant data, performs the 4

5 necessary joins and transforma ons, and delivers the results to users all on the fly without developers knowing anything about the true loca on of the data or the mechanisms required to access and merge it. It s almost too good to be true. Data virtualiza on so ware enables business users to get the applica ons they want faster and with less cost. AGILE DEVELOPMENT. With data virtualiza on, organiza ons can integrate data without physically consolida ng it (although data virtualiza on o en caches data to op mize performance). In other words, they don t have to build a data warehouse or data mart to deliver an integrated view of data, saving considerable me and money. If business requirements change, users don t have to wait for the IT department to populate a new data source into the data warehouse. IT administrators simply point the applica on to new virtual tables in the data virtualiza on layer (once the data source is modeled) to make the required changes. In other words, data virtualiza on so ware enables business users to get the applica ons they want faster and with less cost. A DATA SERVICE. With data virtualiza on, developers can use a single tool to access any data no ma er where it s located, what system it runs on or what interface is required to access it. And the so ware presents users with a unified view of the business, expressed as inter-related data objects defined in business terms. This canonical model or seman c layer greatly simplifies data access. It encapsulates back-end systems behind a unified interface (e.g., SQL, MDX, Web services), providing an important layer of abstrac on between applica ons and data. Essen ally, data virtualiza on so ware turns data into a service, hiding the complexity of back-end data structures behind a standard informa on interface. This layer of abstrac on not only provides business users and applica ons a uniform view of the business, but it also makes it easy for administrators to swap out, redesign, consolidate or move back-databases without affec ng business users or applica ons that make use of the underlying data. The upshot is that IT project teams can significantly reduce the me they spend sourcing, accessing, and integra ng data. In other words, data virtualiza on speeds project delivery, increases business agility, reduces costs, and improves customer sa sfac on. What s not to like? Data virtualiza on so ware consists of four basic components: 1. Data discovery tools that assist developers in understanding the structure and contents of remote data sources. 2. Business modeling or abstrac on tools that create business-oriented models of data that resides on different systems in various loca ons. 5

6 3. A distributed query op mizer and execu on engine that calculates the most efficient way to join remote data sets, perform transforma ons and deliver results to users or applica ons. 4. Data adapters that provide access to major databases, applica ons, files, and services. The me is right for more organiza ons to implement data virtualiza on so ware. FROM DATA CENTER TO DATA OBJECTS. Most companies have already implemented data center virtualiza on so ware using tools, such as VMWare, as well as So ware-as-a-service tools, such as Salesforce.com, which virtualize applica ons by making them accessible via a subscrip on service in the cloud. Similarly, data virtualiza on so ware encapsulates a complex opera ng and processing environment behind a simple user interface that provides greater agility, transparency, and efficiency. The me is right for more organiza ons to implement data virtualiza on so ware. Challenges HISTORICALLY, DATA VIRTUALIZATION SOFTWARE which first appeared in the market more than two decades ago had difficulty delivering high levels of performance when querying large volumes of data or applying complex transforma ons to data on the fly. The engineering required to query two or more databases is complex, and this complexity increased as data volumes and query complexity grew. As a result, data virtualiza on tools were o en used in niche applica ons involving small volumes of clean, consistent data that required few joins and li le to no transforma on. But given recent advances in hardware notably network and CPU speeds and larger memory footprints as well as be er query op mizers in data virtualiza on tools and the op on to cache data, performance and scalability constraints are disappearing. Few companies that implement data virtualiza on today cite performance or scalability as a top issue. To create a data service that makes sense to business users, data virtualiza on so ware does require organiza ons to spend me up front modeling their data for business consump on. This requires businesspeople to come to consensus on the meaning and defini on of key data elements, and also requires technical people to corral these defini ons into a linked model of the organiza on that makes sense to business users. This takes me and poli cal will, which are some mes in short supply in organiza ons. A final challenge is that source systems owners typically don t permit users to submit ad hoc queries against opera onal systems. This is a major reason that 6

7 companies build data warehouses in the first place. But as system performance increases, these objec ons are star ng to diminish. IT administrators are beginning to recognize that advances in hardware performance are causing the worlds of opera onal and analy cal processing to reconverge. Use Cases GIVEN THE ABOVE BENEFITS AND CHALLENGES, there are many use cases for data virtualiza on so ware. Another good use case for data virtualiza on is to augment a data warehouse or business applica on with real- me data maintained in elsewhere. 1. Data services layer. The most enlightened companies use data virtualiza on as a services layer for developers and applica ons to access any data. This creates a universal interface to data, no ma er where the data is stored (e.g., data warehouse, opera onal system, file, Web service, cloud) or what the performance requirements are. If a source system or network link is slow, administrators can use the data virtualiza on so ware to cache the specific data in a database to meet systems level agreements. 2. Emergency (SWAT) applica ons. When the business has an urgent need for a data-intensive applica on and can t spend the me or money to physically consolidate the data in a data warehouse or data mart, it can use a team of highly trained developers armed with data virtualiza on so ware to create the applica on quickly and with minimal cost. 3. Real- me applica ons. Another good use case is to augment a data warehouse or business applica on with real- me data maintained in elsewhere. Virtualiza on so ware queries the data warehouse for historical data and joins it on the fly with real- me data maintained in an opera onal system. 4. Virtual enterprise data warehouse. In decentralized organiza ons with rela vely autonomous business units, companies o en use data virtualiza on so ware to create an enterprise view of corporate data for decision making by dynamically querying data held in various business unit and departmental data warehouses and data marts. 5. External data. Another good use for data virtualiza on so ware is to augment an applica on or data warehouse with external data gleaned either from public websites or a subscrip on service. 6. ETL source. Data virtualiza on can also be used by ETL tools and other applica ons as a source of data to populate target systems. Rather than building direct interfaces to each system, the ETL tool relies on the data 7

8 virtualiza on layer to access remote data and feed it to the ETL tool. 7. Systems migra on. Data virtualiza on is also a good way to migrate source systems or data warehouses without disrup ng downstream applica ons. As long as the applica ons are pointed at the data virtualiza on tool, administrators can swap out backend sources without impac ng those applica ons. A classic use case for data virtualiza on so ware is to create a 360- degree view of customers. 8. Prototyping. Data virtualiza on is a great way to test the efficacy of a data-driven applica on before se ng it in stone. With data virtualiza on as a source, developers can experiment with different sets of data before physically consolida ng data in a data warehouse. 9. Analy cal sandboxes. Data virtualiza on so ware liberates analysts from having to write their own SQL and create their own data feeds when analyzing data. This frees them to spend more me analyzing data and less me collec ng it and crea ng data silos. 10. Support diverse BI tools. Many companies have mul ple, redundant BI tools that businesspeople use to create overlapping and conflic ng reports. With data virtualiza on so ware, business users can use whatever BI tool they want and s ll access the same data elements in the same way using iden cal seman cs degree view of customers. A classic use case for data virtualiza on so ware is to create a 360-degree view of customers by pulling data from different sources, such as order entry system, call center logs, and a provisioning database. Percep ons TODAY, THE BIGGEST OBSTACLE TO THE GROWTH of data virtualiza on is percep on. Given the innate bias among data warehousing professionals to persist data, many s ll doubt that data virtualiza on so ware offers adequate query performance and scalability. So, it takes me to introduce data virtualiza on tools into an exis ng data warehousing architecture. The tools must prove their worth in an ini al applica on and build from there. Since enterprise-caliber data virtualiza on tools cost several hundred thousand dollars, they need a well-respected visionary to advocate for their usage. But nonetheless, the tide is turning toward data virtualization. Most companies have given up the notion that they can populate all their data into a data warehouse. The value of creating a logical view of distributing data is gaining mindshare. At the same time, advances in hardware eliminate performance and scalability concerns. So the 8

9 BI professionals have heard about data virtualiza on and are inves ga ng its poten al uses. future looks bright for data virtualization, although market growth for the segment of software is slower than many might have expected. Survey Results THE REMAINDER OF THIS REPORT PROVIDES MARKET PERSPECTIVES on data virtualiza on. The following charts are based on a survey of 192 BI professionals conducted in March 2013 by the BI Leadership Forum, a LinkedIn group for BI directors and their teams that I moderate. Almost three-quarters (72%) of the respondents are BI professionals, 21% are BI consultants and 7% are business users or sponsors. About half (48%) come from large companies with more than $1 billion in revenue, while the rest are split between mediumsize companies with between $100 million and $1 billion in revenue and small companies with less than $100 million in revenue. Almost half (47%) rated their BI maturity as intermediate while 29% rated their maturity as advanced and 24% said they were beginners. Market Adop on According to the survey, data virtualiza on is s ll an early adopter market since only a bit more than a quarter of the respondents (27%) have either par ally deployed (18%) or fully deployed the so ware. However, almost one-third have data virtualiza on under considera on, which is good news as it shows that a good por on of BI professionals have heard about this technology and are inves ga ng its poten al uses for their organiza ons. It s difficult to sell middleware to companies that tend to fund projects that directly touch business users, not infrastructure. Data virtualiza on vendors need to devise new ways to deliver their technology so it is more closely associated with mee ng the needs of business users rather than improving IT infrastructure. Some have already done that by licensing their tools to BI and ETL vendors who use it to support query federa on capabili es, as we shall see later in this report. Figure 1: What is the status of data virtualiza on at your company? 9

10 No plans 35% Under consideration 30% Under development 8% Partially deployed 18% Fully deployed 9% Reasons for Not Deploying Data Virtualiza on Not knowing enough about data virtualiza on so ware is the main reason cited by respondents for not deploying it. Companies that have yet to deploy data virtualiza on so ware cite a variety of reasons. The top reason is knowledge: they simply don t know enough about data virtualiza on so ware, cited by 37% of respondents who have yet to deploy data virtualiza on so ware. Since the data virtualiza on market is s ll small, if vendors can convert a small frac on of these prospects into customers, they will enjoy considerable growth and have money to further evangelize the market. However, not all prospects make good customers. Thirty percent (30%) of respondents said they don t have funds to purchase so ware, and 23% said they already have a suitable data infrastructure. Only 20% men oned performance and scalability issues and only 15% said there is not enough value for the price. In ad hoc comments, respondents suggested other reasons for not purchasing data virtualiza on so ware, ranging from lack of source cleanliness and poor data governance across applica ons to it s a lower priority for us at this stage and our organiza on isn t large enough to jus fy the cost. Figure 2: What prevents you from deploying data virtualiza on so ware? (Respondents were asked to select all that apply.) 10

11 Don't know enough about data virtualization 37% Our budget is tapped out 30% We already have a suitable data infrastructure 23% Not enough value for the price 15% Clearly, data virtualiza on is not a one-trick pony. It delivers a wealth of benefits.. Benefits Performance and scalability issues Other Organiza ons that have deployed data virtualiza on, however, cite a host of benefits. Chief among these are simplicity (69%), agility (66%), and integra on (66%), followed by consistency (48%) and transparency (46%). Clearly, data virtualiza on is not a one-trick pony. It delivers a wealth of benefits, many of which are not truly appreciated during the purchasing process or fully exploited un l several projects have been implemented. 20% 20% For the purposes of this report, simplicity refers to giving users access to any data through one interface; agility refers to the ability to deploy new applica ons faster; integra on is about crea ng a unified BI architecture from disparate systems; consistency refers to properly formed queries that deliver the iden cal answers to the same ques ons; transparency is the ability of administrators to change or modify data sources without affec ng downstream applica ons. Survey respondents also cited other benefits, including selfservice explora on, use for enterprise architecture, and security and governance. Figure 3: What are the primary benefits that data virtualiza on provides your organiza on? (Respondents were asked to select all that apply.) 11

12 Simplicity 69% Agility 66% Integration 66% Consistency 48% The predominant use case for data virtualiza on among survey respondents is to augment the data warehouse. Transparency Other Use Cases 7% Given the BI background of the surveyed audience, it s not surprising that the predominant use case for data virtualiza on among survey respondents is to augment the data warehouse (77%). This is followed by prototyping (45%), visualizing real- me data within an exis ng applica on (39%), drill into detailed data in another system, such as a data warehouse (30%), and querying nonrela onal data sources (25%), which is gaining in popularity due to the rise in Hadoop usage. 46% Other reasons to adopt data virtualiza on so ware include crea ng an enterprise view of mul ple data warehouses (24%), querying external data (19%), suppor ng ETL processing (18%), querying external data (16%), and delivering a 360-degree view of customers (15%). Unfortunately, the ques on didn t ask specifically about data services, but collec vely the responses can be considered a data services layer. Figure 4: What has caused you to deploy data virtualiza on so ware? 12

13 Access data not available in the DW 77% Protoptye a BI application Visualize current data 39% 45% Drill through to detailed data Query non-relatonal sources 25% 30% Data virtualiza on so ware works best if it addresses all the data within an organiza on as well as the external data business users need. Create a virtual view of multiple DWs Query external data via APIs or Web services Use virtual views as a source for ETL tools Query data that can't be copied for security Scale Deliver a 360-degree view of customers A majority of the companies (59%) that have deployed data virtualiza on have implemented it on an enterprise scale, while one-quarter (25%) have deployed it for a business unit and just 14% have deployed it at the departmental level for one or more departments. Obviously, data virtualiza on so ware works best if it addresses all the data within an organiza on and a good bit of external data that business users look at on a regular basis. Of course, crea ng an enterprise model for data virtualiza on takes some me. 19% 18% 16% 15% 24% Most organiza ons that I ve spoken with that have implemented data virtualiza on start small by enabling a single project with data virtualiza on and then expand from there, adding new data sources for subsequent applica ons. Once the virtualiza on layer address a majority of data in an organiza on, it reaches a pping point whereby it becomes viewed as a corporate resource without ques on. Figure 5: Which best describes the scale and scope of your data virtualiza on so ware? 13

14 Enterprise 59% Business unit 25% Departmental 14% It is expected that more organiza ons will look to data virtualiza on to extend their query reach beyond rela onal databases. Not deployed Other Data Sources 0% 2% The most common type of system accessed by data virtualiza on so ware is a rela onal database, selected by 89% of respondents. This makes sense since most corporate applica ons run on rela onal databases. The second most accessed source is Web services (55%), which suggests that organiza ons that implement data virtualiza on are likely to have encapsulated applica ons in a services-oriented architecture (SOA). Respondents also use data virtualiza on to directly access applica ons (48%), file systems (42%), and mainframe databases (31%). Another 11% use data virtualiza on access other data sources, including NoSQL databases and Hadoop, according to open-ended comments wri en by respondents. Given the rise of NoSQL and Hadoop, it is expected that more organiza ons will look to data virtualiza on to extend their query reach beyond rela onal databases to non-tradi onal sources. Figure 6: What data sources do you virtualize with data virtualiza on so ware? (Respondents were asked to select all that apply.) 14

15 Relational databases 89% Web services 55% Applications 48% File systems 42% Data virtualiza on is sophis cated so ware that requires training and exper se. Mainframe databases Other 11% 31% Challenges Despite the many benefits data virtualiza on offers, there are challenges. Organiza ons that have deployed data virtualiza on so ware say challenges include ge ng other departments to adopt the so ware (46%), obtaining funds (44%), and managing performance (43%). These are followed by managing the complexity of the so ware (36%) and ensuring adequate scalability (33%). A few others men oned the challenge of governing diverse sets of logic and rules used by systems across the organiza on, among other things. Data virtualiza on is sophis cated so ware that requires training and exper se to use adroitly. It is also suscep ble to unexpected changes in source systems that can hamper performance or alter query results. Although this is true in tradi onal data warehousing environments as well, data virtualiza on so ware exposes changes immediately to end users. In some sense, this direct connec on to source systems is a good thing because it alerts business users to problems that only they can fix through poli cal pressure; but on the other hand, it exposes them to unwanted delays and distrac ons. 15

16 Figure 7: What have been your primary challenges in deploying data virtualiza on so ware? (Respondents were asked to select all that apply.) Adoption 46% Funding 44% Performance 43% Complexity 36% Scalability 33% Other 7% Vendors Interes ngly, most survey respondents use data virtualiza on so ware that is embedded in other tools, most notably BI tools (47%) and ETL tools (30%). Only a quarter (25%) use pure-play data virtualiza on tools, while 16% have built their own data virtualiza on layer. These results make sense given the nature of the survey pool (BI/DW professionals). However, embedded data virtualiza on so ware prevents organiza ons from crea ng a universal data service accessible by all tools, users, and applica ons. Ironically, the widespread use of data virtualiza on so ware inside other tools may impede its use as a universal data services layer that most companies desperately need. Figure 8: Which type of vendor supplies you with data virtualiza on so ware? (Respondents were asked to select all that apply.) 16

17 BI tool vendor 47% ETL vendor Pure-play data virtualization vendor 25% 30% Built our own data virtualization capabilities 16% Other 9% None 5% Respondent Advice Survey respondents that have deployed data virtualiza on were asked to provide advice to organiza ons that are just beginning the journey. Here are some of their more salient words of wisdom: Give it a try. You would be surprised to see it working in cases you were hesitant. Design for scale and reusability. The earlier in the project that you focus on how data layers and sources will be exposed to a variety of business applica ons, the easier the deployment and adop on. Consumers are looking for the biggest bang for the buck, both figura vely and literally. If abstrac on can be done in a way that makes layers reusable and portable for other projects, then you have gained a much needed edge. Define high value, high exposure use cases to develop first. Do performance and capacity planning up front. Adop on of data virtualiza on usually takes off quickly. All services developed should have enterprise wide value. Use common business en es across the enterprise. Make sure data virtualiza on has been included as part of the enterprise technology stack and that its role is well defined. See the big picture, prepare and plan change management. Carefully consider metadata and governance needs and impacts. Data virtualiza on so ware that isn t governed is trouble. Be realis c and clear about the requirements data virtualiza on is best suited to address. 17

18 Conclusion DATA VIRTUALIZATION SOFTWARE CREATES DATA SERVICES that can help IT departments accelerate the deployment of data-driven applica ons while improving the accessibility and maintainability of data resources throughout the company. Although there are many things to like about data virtualiza on so ware, few companies have implemented these tools today. A er 20 years, it s s ll an early adopter market. BI professionals increasingly understand that federa on is the future; they can no longer deliver enterprise data by physically consolida ng all data. Unfortunately, most organiza ons already have an exis ng data infrastructure and don t have the me or resources to augment or replace it with data virtualiza on so ware. As a result, the majority of the organiza ons that have implemented the so ware have done so in a niche fashion, by deploying BI or ETL tools with embedded data virtualiza on features. Although this embedded approach enhances these applica ons with query federa on capabili es, it doesn t provide a universal data access layer that glues together data throughout an organiza on. However, market forces are marching in the direc on of data virtualiza on. Advances in the hardware and be er query op mizers have drama cally improved the performance and scalability of data virtualiza on so ware. In addi on, BI professionals increasingly understand that federa on is the future; they can no longer deliver enterprise data by physically consolida ng all data. As more IT managers learn about this unique category of so ware, it will grow its footprint within corporate data infrastructures. 18

19 ABOUT THE AUTHOR ABOUT TECHTARGET: TechTarget publishes media for informa on technology professionals. More than 100 focused websites enable quick access to a deep store of news, advice and analysis about the technologies, products and processes crucial to your job. Our live and virtual events give you direct access to independent expert commentary and advice. At IT Knowledge Exchange, our social community, you can get advice and share solu ons with peers and experts. WAYNE ECKERSON has been a thought leader in the business intelligence field since the early 1990s. He has conducted numerous research studies and is a noted speaker, blogger, and consultant. He is the author of two widely read books: Performance Dashboards: Measuring, Monitoring, and Managing Your Business (2005, 2010) and The Secrets of Analy cal Leaders: Insights from Informa on Insiders (2012). Wayne is currently director of BI Leadership, an educa on and research service run by TechTarget that provides objec ve, vendor neutral content to business intelligence (BI) professionals worldwide. Wayne s consul ng company, BI Leader Consul ng, provides strategic planning, architectural reviews, internal workshops, and long-term mentoring to both user and vendor organiza ons. For many years, Wayne served as director of educa on and research at The Data Warehousing Ins tute (TDWI) where he oversaw the company s content and training programs and chaired its BI Execu ve Summit. him at weckerson@bileadership.com. Data Virtualiza on: Percep ons and Market Trends is a BI Leadership e-publica on. Wayne Eckerson Director, BI Leadership Jean Schauer Editor in Chief Ed Laplante Director of Sales elaplante@techtarget.com TechTarget 275 Grove Street, Newton, MA TechTarget Inc. No part of this publica on may be transmi ed or reproduced in any form or by any means without wri en permission from the publisher. TechTarget reprints are available through The YGS Group. 19