HiTech. White Paper. Data Virtualization: Enabling Next-Generation Business Intelligence

Transcription

1 HiTech White Paper Data Virtualizatio: Eablig Next-Geeratio Busiess Itelligece

2 About the Authors Vijay Raj Sigh Vijay Raj Sigh is the Ceter of Excellece (CoE) lead for the Busiess Itelligece ad Performace Maagemet (BIPM) practice withi the Techology Excellece Group (TEG) at Tata Cosultacy Services (TCS). Over a experiece of more tha 11 years, he has successfully achored several customer egagemets; helpig orgaizatios across idustries desig ad develop solutios ivolvig busiess itelligece tools ad techologies. I his curret role, Raj provides architectural guidace ad coducts techical reviews for TCS' HiTech projects, assists the presales teams i proposal maagemet, ad supports the learig ad developmet teams i orgaizatioal competecy buildig. He holds a Bachelor's degree i electroics ad istrumetatio egieerig from the Visvesvaraya Techological Uiversity, Karataka. Shobha Basal Shobha Basal is the Ceter of Excellece (CoE) lead for the Aalytics, Big Data, ad Iformatio Maagemet (ABIM) practice withi the Techology Excellece Group (TEG) at Tata Cosultacy Services (TCS). She has over 15 years of experiece, through which she has bee ivolved i umerous cosultig ad solutio developmet egagemets for some of TCS' leadig customers, across idustries. Shobha also supports the various presales teams durig the proposal maagemet process. She holds a Bachelor's degree i electrical egieerig from the Natioal Istitute of Techology, Karataka (NITK).

3 Abstract Data is a vital asset for eterprises ad is touted to become eve more importat i the times to come. Orgaizatios rely o data warehousig (DW) to maage ad mie gathered data ad busiess itelligece (BI) techiques to draw isights for busiess support, but the capabilities of existig systems eed to be augmeted to accommodate the chagig ature of data. Owig to globalizatio, digitizatio, ad a rapidly chagig ecoomic climate, its volume ad complexity are growig at a staggerig rate. There is a pressig eed for orgaizatios to effectively ad efficietly maage data ad trasform it ito valuable iformatio i a timely maer. To profitably sustai i the dyamic busiess eviromet that exists today, HiTech compaies are heavily ivestig i tools ad techiques based o advaced aalytics to derive actioable isights that eable timely ad effective decisio makig. These ew age systems should have the ability to tap ito the existig ad ew data sources structured ad ustructured, ad should be user-friedly with easy self-service optios, while esurig real-time access to the required iformatio. To cater to these demads, the BI ad DW systems eed to evolve quickly ad adopt moder data itegratio practices. I this paper, we have discussed how data virtualizatio ca ehace the traditioal BI ad DW systems, ad provide eterprises with the required iformatio i a timely fashio to meet the evolvig busiess requiremets with greater agility, ad at affordable costs.

4 Cotets 1. Data Virtualizatio: The Key to Agile Busiess Itelligece 5 2. Usig Data Virtualizatio for Developig Next Ge Busiess Itelligece Systems 6 3. How Data Virtualizatio Based Systems Will Help Eterprises 7 4. Curret State of Adoptio ad Future Outlook 9 5. Coclusio 10

5 Data Virtualizatio: The Key to Agile Busiess Itelligece Busiess itelligece (BI) helps orgaizatios make iformed decisios pertaiig to products ad processes, as well as optimize the supply chai, thereby helpig them to urture successful relatioships with customers ad parters. It also drives greater agility ad resposiveess with regard to global evets ad risks. Data warehouse (DW) helps i maagig data itegrity ad trasactio history, thereby esurig iformatio-cetric eeds of busiess stakeholders ad regulators are successfully met. However, the very architecture of a data warehouse itroduces redudacy ito the system sice the same data poits are stored i differet formats across various layers such as stagig server, operatioal data store (ODS), base layer, ad data mart. As ew sources of data emerge, busiesses require flexibility ad agility to itegrate them with eterprise data. For HiTech compaies, equipmet like pritig machies ad storage devices are very useful sources, data from which, if itegrated with eterprise data, ca provide busiess critical iformatio. It provides ear real-time isights o device performace ad its availability. Similarly, for the huma resource fuctio i a professioal services idustry, sourcig data from the social ecosystem ca be useful i assessig a potetial cadidate, thereby improvig the cost effectiveess of the recruitmet process. However, covetioal methods of data itegratio follow a pre-defied process that usually takes a couple of days to sometimes weeks, to itegrate ewer sources of data. Additioally, the existece of multiple aalytical systems such as a aalytics sadbox, Hadoop or Big Data store, ad so o, adds to the complexity. Eterprises therefore eed to brig about a paradigm shift i the way they store, trasform, ad retrieve data. To facilitate real-time decisio-makig processes ecompassig the etire busiess ecosystem, orgaizatios must take their BI capabilities to the ext level. Basic buildig blocks of a ext geeratio BI system are: Rapid respose aalytics egie to eable agile decisio-makig processes by meas of a aalytics sadbox, o-demad data itegratio, ad self-service aalytics tools Data virtualizatio ad federatio to combie disparate data sources i real-time to a sigle virtual data layer that provides itegrated data services to applicatios Big Data maagemet to itegrate data which may be structured or ustructured, iteral or exteral, i batches or i real-time or by streamig, o premise or through the cloud from differet sources Data visualizatio ad discovery to visually query the data, spot importat treds ad coditios, ad set alerts ad threshold values o dyamic data Data virtualizatio is a importat prerequisite to deployig the ext-geeratio BI system, ad offers a viable optio to augmet traditioal data itegratio architectures. Goig by emergig idustry treds, data virtualizatio is a must-have compoet i every eterprise's data itegratio portfolio, more so for HiTech compaies. 5

6 Data virtualizatio combies disparate sources, i real-time, ito a sigle 'virtual' data layer that provides itegrated data services to cosumig applicatios. I the era of Iteret of Thigs (IoT), Big Data, ad cloud, orgaizatios ted to accumulate huge volumes of ustructured data, 'warehousig' which, is ot viable. Data virtualizatio ca haress value from all types of data to provide a itegrated view i a stadard format to dowstream systems. Usig Data Virtualizatio for Developig Next Ge Busiess Itelligece Systems A comprehesive data virtualizatio platform should provide a orgaizatio with a virtualized access capability to eable easy access to data from differet sources. It should also help to collate data from diverse sources ad preset a uified view that is easily accessible. I additio, the platform should facilitate the abstractio of techical itricacies of data such as locatio, storage structure, accessibility, ad storage techology, ad allow maipulatio, trasformatio, quality, ad stadardizatio of source data. Additioally, the platform should esure a seamless delivery of iformatio to relevat stakeholders, by publishig result sets i a stadard format of ope database coectivity (ODBC) views ad data services, or web services such as simple object access protocol (SOAP), represetatioal state trasfer (REST), ad XML, for o-demad cosumptio by a variety of data-cosumig cliet applicatios. To achieve the aforemetioed capabilities, a data virtualizatio platform should essetially have the followig compoets (refer Figure 1): Data sources: These refer to the structured ad ustructured physical iformatio assets that exist withi ad outside a orgaizatio. These sources may be databases, sesors, iteret, cloud applicatios, packaged applicatios such as SAP, web services, data warehouse or data mart, ad spreadsheets. Data itegratio layer: Lyig at the heart of a data virtualizatio platform, this layer offers comprehesive data maipulatio services by itegratig data i real-time from a variety of sources as per predefied busiess rules. It also provides the ability to access, trasform, profile, stadardize, ad federate data usig a advaced query egie. Busiess sematic layer: This layer defies a set of logical or caoical views that represet a busiess etity grouped ito subject areas, defied by the orgaizatio s eterprise iformatio model. These views or models are reusable compoets that ca be used by multiple cosumers across the various busiess uits i a orgaizatio. Applicatio layer: This layer provisios the busiess sematics ito formats customized to requiremets of differet ed users. It ca iclude formattig ito XML for web services, or creatig views uder various aliases that match the way the users view data relevat to them. It also provides a mechaism to publish views, data, or web services, ad to host these services for dowstream cosumptio. 6

7 Data cosumers: Applicatio users require the delivery of data through differet protocols that they uderstad, such as JDBC, ODBC, SOAP or Hypertext Trasfer Protocol (HTTP), ad REST. These stadard protocols exted the cosumptio of data to a variety of dowstream applicatios such as BI, master data maagemet (MDM), spreadsheets, web service applicatio programmig iterfaces (APIs), ad other eterprise object cosumig applicatios. Data sources Structured data Ustructured data Real-time data Legacy data Iteret Big Data MDM Relatioal Data Web Services Maiframe Data File Social Media Cloud Web Logs Machie Device Coectors Data Itegratio Data Itegratio Layer Data Trasformatio Data Maskig Data Profilig Data Quality Data Stadardizatio Data Federatio Data Referece Busiess Sematic Layer Virtual Schema Virtual Tables Virtual Stored Procedures Data Services Applicatio / Hostig Layer Web Services (REST/SOAP/HTTP/JSON) XML Data Cachig Publishig Services Data Security/ Access Cotrol BI ad Aalytics Stadard Reportig Plaig Forecastig Budgetig Score cards Dashboards Aalytics Self Service Query Adhoc Reportig Data Cosumig Apps Eterprise Applicatios Dowstream Systems Cube Data Mart Goverace Metadata Maagemet Data Mart / OLAP MS Office EDW High Availability ad Optimizatio Mobile Audit ad Moitorig Figure 1: A Overview of Data Virtualizatio Referece Architecture How Data Virtualizatio Based Systems Will Help Eterprises Data virtualizatio greatly ehaces the covetioal BI/DW architecture improvig implemetatio flexibility ad busiess agility. Followig are some importat use cases that leverage data virtualizatio: Data warehouse augmetatio: Data virtualizatio ca sigificatly augmet the capabilities of a data warehouse. With the icreased uptake of digital techologies, a huge volume ad a wide variety of data is geerated i a orgaizatio's ecosystem. For example, the trasactio data geerated from machies like priters provide key isights of customer usage patters that ca help fie tue the sales strategy, idetify cross- 7

8 sell or up-sell opportuities, ad stregthe key features that are heavily used by cosumers. However, physical cosolidatio of such eormous volumes of ugovered ad ustructured data ito a warehouse or a mart is ot a feasible optio. Therefore, data from multiple systems, tied with data services ad metadata, will exted a existig data warehouse to become the 'logical eterprise data warehouse'. Real-time data itegratio: Real-time access to data from disparate sources provides faster ad comprehesive busiess isights for buildig a competitive advatage. Hece, orgaizatios eed to extract data at the poit of geeratio without waitig for it to get replicated to a cetral repository. To esure real-time ad curret data for actioable aalytics, a virtual layer ca be created over the recetly geerated data i trasactioal systems, alog with the historical data stored i data marts. HiTech compaies like chip maufacturers ad storage providers are explorig meas to leverage real-time data geerated by equipmet for prevetive maiteace ad timely fault detectio, thus eablig progosis, diagosis, ad health maagemet of machies. Data warehouse cosolidatio: Data maagemet programs coducted over a period of time, by decetralized orgaizatios with relatively autoomous busiess uits, almost always result i iformatio silos. Eterprises realize that effective sharig of data across busiess uits is vital to the smooth fuctioig of a orgaizatio. Data virtualizatio ca help create a eterprise view by itegratig siloed data warehouses ad marts. I case of mergers ad acquisitios (M&A), a data virtualizatio solutio ca provide a itegrated view of customer as well as employee data of the participatig compaies, without havig to wait for data warehouse cosolidatio. Self-service BI methods: I the pursuit of agility ad self-reliace, busiess users are o loger satisfied with just aalyzig the give iformatio. Users wat to perform data discovery, exploratio, ad aalytics with miimal support from IT teams. The data virtualizatio layer ca facilitate this 'self-service' BI by creatig data services with all the required iformatio, which are the itegrated with differet data sources. Busiess users ca sped more time i buildig their ow reports ad aalytics rather tha searchig for the right iformatio ad tryig to itegrate it. This provides more flexibility, faster results, ad ivolves less effort, leadig to costeffectiveess. Self-service BI is a highlight of the ext-geeratio BI systems as orgaizatios try to empower busiess users with govered data discovery capabilities as a expasio of the existig BI capabilities. Reuse across differet BI platforms: Eterprises ofte have multiple reportig tools i their BI ladscape resultig i redudat ad overlappig reports. The sematic or model created by differet BI tools is limited to the tool's ow platform. Usig data virtualizatio, a logical layer of caoical busiess etity model ca be provided to eable cosistet access by diverse dowstream cosumers across differet busiess groups. It ca reder the sematics or models created by various data discovery ad BI tools redudat. Busiess users ca use ay BI tool as desired, ad still access the same data usig cosistet sematics. It also reduces the complexity ad cost while migratig from oe BI platform to aother. Data acquisitio platform: I a typical data warehouse architecture, as data flows from the source to the mart layer, it creates various copies of redudat datasets. With data virtualizatio, we ca miimize this redudacy by creatig a virtual acquisitio layer based o extracts of differet source systems. A data virtualizatio platform has the ability to coect to a variety of sources with a API-based query iterface, such as web 8

9 services (SOAP ad REST), websites, ad packaged applicatios like Salesforce. This layer acts as a sigle stagig source for extractio, trasformatio, ad loadig (ETL) tools to extract data ad process for dowstream loads. It also elimiates the complexity of data acquisitio process by abstractio of differet data sources. Prototypig ad sadboxig: Buildig a ew data warehouse or ehacig a existig oe are timecosumig exercises sice the implemetatio team eeds to follow a set of pre-defied tasks. Moreover, there is o process i place for the busiess team to validate the frequetly chagig requiremets util the trasformatio is complete. This results i multiple iteratios somethig that ca be successfully avoided by usig data virtualizatio techiques. Data virtualizatio techiques help i prototypig requiremets faster, thus takig relatively lesser time for solutio developmet, ad perform experimetal aalytics with differet data sets, to realize timely beefits. The prototype thus developed ca be validated i time by the busiess stakeholders, with the feedback gettig icorporated i the subsequet iteratio, thereby eablig progressive improvemets. Moreover, oe-time busiess requests that do ot warrat the ivestmet of either time or effort for a permaet solutio, ca be quickly accomplished through this methodology. Curret State of Adoptio ad Future Outlook With growig iformatio maagemet challeges, compaies are cosiderig ewer ways to itegrate ad maage data. Most orgaizatios are therefore explorig data virtualizatio techiques to reap the umerous beefits of this approach that have bee discussed i the previous sectio. Despite the icreased focus o the data virtualizatio cocept, the techical improvemets, ad growig use cases, the techology is i is still i earlier stages of adoptio. The growig iformatio maagemet challeges are compellig compaies to cosider ewer ways to itegrate ad maage data data virtualizatio is oe such. The adoptio of the data virtualizatio techology depeds o busiess requiremets, with a special focus o future busiess eeds. Most compaies already have data maagemet ifrastructures i place ad as a result, may be hesitat to make ivestmets i this area. The use cases metioed i the paper demostrate the potetial beefits of data virtualizatio based busiess itelligece systems. Hece, irrespective of the curret state, busiesses will soo make the shift towards a ehaced system. To embark upo this trasformatio jourey, orgaizatios should keep the followig poits i cosideratio: 1. Coceptualize: Start with a maageable project that eables learig ad lays a strog foudatio for progress. Cosolidate small-scale data sources ito a sigle poit of iformatio to realize quick beefits. Defie the guidelies, stadards, ad referece architecture for solutio implemetatio. 2. Develop: Expad the usage across departmets by broadeig the ivolvemet of IT ad busiess teams. The existig BI/DW ladscape ca be optimized by: a. Implemetig data virtualizatio to elimiate or reegieer redudat stagig or acquisitio data layers 9

10 b. Augmetig data warehouses ad marts by itegratig exteral data sources (like web, Big Data, social media, ad so o) to improve the system's aalytical capabilities c. Replacig or reegieerig data marts or etities as data services 3. Operate: Adopt data virtualizatio across the eterprise as 'Iformatio as a Service' (IaaS) i a govered ad a optimized way. Build logical caoical etity models for reuse across busiess uits. Measure, moitor, report, ad cosume key metrics ad iformatio o a real-time basis. Coclusio With most orgaizatios already havig data maagemet ifrastructures i place, the adoptio of ewer techology faces a obvious roadblock hesitace i makig the humugous ivestmet of time ad effort. Data warehousig, over a period of time, has also proved its utility as the basis for a eterprise's decisio support system, especially, i the followig scearios: Whe historical time series aalysis of data across various dimesios is required Whe data quality is a issue, multiple passes ad huma itervetio are required to fix it before loadig it to a physical storage of a data warehouse Whe source systems are ofte offlie due to maiteace activities or ureliable etwork coectivity Whe data itegratio requires trasformatio of huge volumes of data, which may cause performace issues if doe i real-time However, irrespective of curret setups, the evolvig eeds of busiesses will make it madatory for techology to keep up with such trasformatios. With the emergece of i-memory computig, we are lookig at the possibility of a sigle physical database that ca cater to operatioal requiremets, as well as aalytical eeds of a busiess. Data virtualizatio will play a importat role i providig busiesses with sematic ad virtual models over the imemory databases, as a platform for busiess itelligece based reportig. Eterprise data warehouses will still exist, ad data virtualizatio will augmet ot replace the traditioal data itegratio systems. The scearios metioed i this paper demostrate the potetial of data virtualizatio to brig sigificat beefits to a orgaizatio, makig this techology worth a serious cosideratio. Give the varyig iformatio eeds, the quest for agility, itegratio challeges with ew ad emergig data sources, ad advacemet i techologies, data virtualizatio will be a key compoet i the overall data itegratio portfolio of a eterprise. 10

11 About TCS' High Tech Busiess Uit Accelerated idustry growth, rapid techological obsolescece, ad the eed for faster time to market compel High Tech orgaizatios to improve busiess agility. High Tech solutios from TCS address fudametal idustry problems, improve process efficiecy, ad ehace productivity ad collaboratio across busiesses, while optimizig overheads. From software product egieerig ad supply chai, to leveragig Iteret of Thigs, digital reimagiatio, cloud, Big Data, mobility ad others, our solutios empower idustry players - computer platform ad services compaies, software firms, electroics ad semicoductor compaies, ad professioal services firms to compete effectively. Complemetig our comprehesive service portfolio of IT solutios, busiess cosultig, product egieerig services, ifrastructure services, ad busiess process services, are our parterships ad alliaces with leadig idustry vedors. Dedicated iovatio labs, ifrastructure support, ad the Tata Research Developmet ad Desig Ceter (TRDDC) offer our cliets access to cuttig edge techologies, advaced systems egieerig methodologies, storage optimizatio, ad covergece solutios. Cotact For more iformatio about TCS High Tech Busiess Uit, visit: HiTech.marketig@tcs.com Subscribe to TCS White Papers TCS.com RSS: Feedburer: About Tata Cosultacy Services (TCS) Tata Cosultacy Services is a IT services, cosultig ad busiess solutios orgaizatio that delivers real results to global busiess, esurig a level of certaity o other firm ca match. TCS offers a cosultig-led, itegrated portfolio of IT ad IT-eabled ifrastructure, egieerig ad TM assurace services. This is delivered through its uique Global Network Delivery Model, recogized as the bechmark of excellece i software developmet. A part of the Tata Group, Idia s largest idustrial coglomerate, TCS has a global footprit ad is listed o the Natioal Stock Exchage ad Bombay Stock Exchage i Idia. For more iformatio, visit us at IT Services Busiess Solutios Cosultig All cotet / iformatio preset here is the exclusive property of Tata Cosultacy Services Limited (TCS). The cotet / iformatio cotaied here is correct at the time of publishig. No material from here may be copied, modified, reproduced, republished, uploaded, trasmitted, posted or distributed i ay form without prior writte permissio from TCS. Uauthorized use of the cotet / iformatio appearig here may violate copyright, trademark ad other applicable laws, ad could result i crimial or civil pealties. Copyright 2015 Tata Cosultacy Services Limited TCS Desig Services I M I 03 I 15