ISSN: Page 120

Transcription

1 Satellite Conference ICSTSD 2016 International Conference on Science and Technology for Sustainable Development, Kuala Lumpur, Modernization of Data Warehousing through Open Source Softwares Preeti P. Deshmukh Department of Master in Computer Application, Prof. Ram Meghe Institute of Technology & Research, Bandera Amravati Abstract: This paper aims to investigate the utilization of different open source softwares to make the traditional data warehousing techniques modern at big data scale. Data warehouse stores current and historical data which are used for creating analytical reports for knowledge workers throughout the enterprise. The data warehouse plays a critical role in storing, managing, and processing information at big data scale. Now a days we have many applications as replacements for the data warehouse for example Hadoop, Cassandra, MongoDB, and other NoSQL and so the modernization is important for data warehousing. Enterprises are demanding business value from Big Data. As a result, the ability to blend petabytes and exabytes of data from historical and streaming sources becomes a necessity and data warehouse modernization becomes a top priority. Because the data warehouse is often thought of as the heart of an enterprise s Big Data and analytics strategies, modernizing it has a potentially powerful and very positive effect on the bottom-line impact of new technologies, platforms, tools, and practices. No matter what modernization strategy is in play, all data warehouses require significant adjustments to the logical and systems architectures of the extended data warehouse environment. expansive variety of data types, and the real-time processing rapidity of how data is being used to grow and operate the business. These changes are so profound that Gartner reports, Data warehousing has reached the most significant tipping point since its inception. The biggest, possibly most elaborate data management system in IT is changing. The modern enterprise needs a logical architecture that can smoothly scale to meet these volume demands with real-time processing power and the ability to manage any data type to rapidly connect the business to valuable insights. This means that the traditional data warehouse needs to evolve into a modern data warehouse. Real-Time Data: The traditional data warehouse was designed to store and analyze historical information on the assumption that data would be captured now and analyzed later. System architectures focused on scaling relational data up with larger hardware and processing to an operations schedule based on sanitized data. Keywords: Open source, DWE, DW, MPP Introduction: Definition: A data warehouse is a database designed to enable business intelligence activities: it exists to help users understand and enhance their organization's performance. It is designed for query and analysis rather than for transaction processing, and usually contains historical data derived from transaction data, but can include data from other sources. Data warehouses separate analysis workload from transaction workload and enable an organization to consolidate data from several sources. Fig-1: General Architecture of Traditional Data warehousing This helps in: Maintaining historical records Analyzing the data to gain a better understanding of the business and to improve the business So far the speed of how data is captured, processed, and used is increasing. Companies are using real-time data to change, build, or optimize their businesses as well as to sell, transact, and engage in dynamic, event-driven processes like market trading. The traditional data warehouse simply was not architected to support near real-time The traditional data warehouses we find are under pressure of the increasing heaviness of unpredictable volumes of data, the ISSN: Page 119

2 Traditional Data Warehouse Problems: The traditional data warehouse suffered from a number of problems: 1. High licensing and storage costs meant that only large companies could afford to implement data warehouse solutions. 2. Large and expensive database administrators and support staff required to tune and manage the data warehouse so that it can supply the constantly changing BI information 3. Recent requirements for predictive analytics have resulted in very slow query performance. This has caused business users frustration as they seek to quickly test their business models. 4. Small and medium size companies often could not afford to implement BI and data warehouse solutions and were therefore at a considerable competitive disadvantage. What was required was a new type of data warehouse that was both very fast for performing advanced data analytical queries, and, at the same time had a low Total Cost of Ownership (TCO) that ensured that it could be used by both Small Medium sized Enterprise (SME) and large companies. The data warehousing and business intelligence market can be described by the curve, with different component technologies at different points along with the curve. expensive proprietary appliances, such as large-scale extract, transform, and load (ETL) workloads. With the speed of business today, practitioners require more agile, more powerful approaches to data warehousing and analytics that are simultaneously cost-effective to scale. Only open source data warehousing can meet those requirements. While relating open source data warehousing to the larger big data technology stack, which is based on open source technology Data warehousing is an important part of the big data stack, and open source data warehousing in particular is a perfect complement to the other open source technologies in that stack, such as Hadoop. Open source data warehousing is also complementary to other big data technologies from a workload perspective, providing flexible, high-performance analytics and reporting capabilities that support other important workloads such as streaming and unstructured data analysis. Need of Data warehouse modernization Data warehouses support mission-critical workloads, so it is important to select an open source data warehouse with an active, growing community that is continuously developing the code base. It is also critical to pick an open source data warehouse that has at least one but preferably several trusted vendors backing it up with world-class support. Figure-2: Data warehousing technology markets graph Open Source Data warehousing Definition: An open source data warehouse is a specialized database built entirely on open source software code that supports enterprise, production-grade data analytics and reporting. An open source data warehouse should also support large-scale exploratory analytics and data science workloads including machine learning. Open source data warehousing significantly reduces Total Cost of Ownership. With open source data warehousing, there are no software licensing costs and no expensive proprietary hardware to purchase. The code is free and generally runs on inexpensive commodity hardware. Open source data warehouses are also an ideal environment for important but less complex workloads many enterprises currently run on DW modernization assumes many forms, from server upgrades and tweaks for data models, to adding new platforms into the extended data warehouse environment (DWE), to replacing the primary DW platform. Modernization may involve using features previously untapped, such as in-memory databases, in-database analytics, real-time functions, and data federation or virtualization. Systems integrated with the DW need attention, too. Analytics, reporting, and data integration are also modernizing, and the DW is under pressure to provision data in ways that enable modern end-user practices such as visualization, advanced analytics, data prep, and self-service data access. The arrival of big data has made such provisioning more business critical and more difficult. In recent surveys by TDWI Research, roughly half of respondents report that they will replace their primary data warehouse (DW) platform and/or analytic tools within three years. Ripping out and replacing a DW or analytics platform is expensive for IT budgets and intrusive for business users. This raises the question: What circumstances would lead so many people down such a dramatic path? It's because many organizations need a more modern DW platform to address a number of new and future business and technology requirements. In a nutshell, organizations that seek to modernize their data warehouse environment do so to ISSN: Page 120

3 improve advanced analytics, scale, speed, productivity, or economics. Each of the five reasons listed here has multiple meanings, they are all interrelated, and users sort the five into varying priority orders, based on their needs. Even so, in general, the list constitutes the top five reasons for data warehouse modernization, and they can provide some guidance for users facing modernization. 1. Advanced analytics. Many organizations have invested heavily in reporting and OLAP, but now they need to invest in advanced forms of analytics to leverage big data, find new customer segments, and stay competitive. 2. Speed. Organizations likewise need the data warehouse and related systems to operate faster because speed contributes to scale, supports agile development and discovery analytics, and brings analytics closer to real-time business operations. 3. Scale. This continues to be an issue with big data and other burgeoning enterprise datasets as well as with growing numbers of concurrent users, reports, analyses, and data structures. 4. Productivity. Traditional requirements gathering, prototyping, and development takes months, which is too long for a modern business. That's why agile development methods are now the norm in DW/BI and analytics. Likewise, users are adopting agile tool types, including those for data exploration and discovery, data profiling, and data visualization. 5. Costs. The good news that modernization is not only a chance to increase speeds and feeds in your data warehouse environment, but it is also a golden opportunity to rethink DW overall costs, as users seek to save money in some areas (storage, CPUs, upgrades, admin) so they can invest in others (new data platforms, analytic tools, and developing new solutions). Modernizing Data Warehouse In the world of revolutionary, game-changing big data developments, data warehouse modernization may sound like an evolutionary development. But it is something that can be executed today, with existing data warehouse skills, and represents a simple first step toward gleaning immediate business value and organizational agility from big data technologies. It is observed that the move to agile development methods is one of the strongest trends in data warehousing. Similar trends involve lean, logical, and virtual methods. Modernization also affects users skills, staffing, and team structure. Data Warehouse modernization is truly confronting. The TDWI report s survey says that 76% of Data Warehouses are evolving briskly; 89% of respondents say modernization is an opportunity for innovation. Warehouse with new business goals, increasing Data Warehouses scale for big data, enabling new analytics applications, and embracing new tools or data types and their attendant practices. The chief beneficiaries of modernization include analytics, business management, and real-time operations. The leading barriers involve problems with governance, staffing, funding, designs, and platforms. The rise of the multiplatform DWE is an evolution of the DW system architecture. Hence, changes at the system architecture level are the most common form of DW modernization (53% of users surveyed). At one end, this involves simple upgrades and patches for hardware and software servers or tools. Introduction to MPP DATA WAREHOUSES The data warehouse plays a critical role in storing, managing, and processing information at big data scale. This might sound counterintuitive, especially now that Hadoop, Cassandra, MongoDB, and other NoSQL platforms are marketed as replacements for the data warehouse. True, one or more SQL query engines exist for all of these platforms, but a SQL query engine does not a data warehouse make. One or more SQL query engines exist for all of these platforms, but a SQL query engine does not a data warehouse make. And that s why, If we want to do is take some flat files and execute SQL across them, that doesn t actually require a database. It requires a translator from SQL to execution. In order to design and build a massively parallel processing [MPP] database, some of the most difficult problems to solve are maintaining consistency across a huge database that runs on a distributed cluster, where you have concurrent access to that data, notes Ivan Novick, a product manager with Pivotal Software, Inc., which markets Greenplum, an open source MPP database. MPP (Massively Parallel Processing)-based databases, especially software-only options, provide a cost effective, scale-out data warehouse environment that allows companies to leverage Moore s Law on performance-to-cost ratio improvements in x86 processors. MPP databases provide a non-intrusive analytical platform/data warehouse for data discovery and exploratory work over massive amounts of data. Built on inexpensive commodity clusters, MPP databases can extend, complement, or even replace parts of your existing data warehouse, managing massive volumes of detailed data, while providing agile query, reporting, dashboards, and analytics. According to the survey, the leading drivers behind Data Warehouse modernization include realigning the Data ISSN: Page 121

4 Here s another example: the open source R statistical programming environment. Disciplines don t get much more specialized than statistics and data mining, which were dominated by proprietary vendors such as SAS Institute Inc. and the former SPSS Inc. (now IBM SPSS) for decades. R hasn t just mounted a challenge to the dominance of SAS and SPSS, it s arguably already won. Most graduates of college business, engineering, social scientific, and, of course, statistical programs learned their craft on R, not on proprietary platforms. Figure -3: MPP (Massively Parallel Processing) Architecture The number of ACID-compliant (atomic, consistent, isolated, durable), MPP analytical data warehouses is remarkably small. This doesn t mean MPP is a prohibitively expensive proposition, however. But the commodification of MPP software and server hardware and a little assist from the world of open source software made MPP performance surprisingly affordable and cost-effective, too. In the era of big data, MPP database systems are ideally suited for many if not most analytical workloads. They re able to support high concurrency rates and, in some cases, new kinds of advanced, NoSQL analytics. For example, some MPP platforms can parallelize and run different types of algorithms in the context of the database engine. The MPP data warehouses of today are utilizing a cluster of servers to store and process data. You can run a machinelearning algorithm that leverages the CPUs of all of the servers in that cluster to do the analysis in parallel. Hadoop and other NoSQL platforms have positive, distinctive roles to play in the big data architectures of today and tomorrow. NoSQL platforms are well suited for storing and managing multistructured data (e.g., text files, multimedia content, and binary objects), as well as for storing relational data at truly massive scale. The MPP data warehouse, however, is ideal for dynamic, mixed workloads, as well as for storing, managing, and processing information at big data volumes. Pivotal s Greenplum database is an open source MPP data warehouse. Greenplum itself is based on the open source PostgreSQL database, which has a rich open source pedigree. Open Source MPP Data Warehouse Open source software has lowered the proverbial bar with respect to cost of entry, cost of maintenance, and total cost of ownership (TCO) in many once-specialized markets. Take the open source GNU-Linux operating system, which turns 23 this year. A quarter of a century ago, the dominant UNIX operating systems were proprietary and ran on costly RISC hardware. GNU-Linux isn t technically UNIX, but it s UNIX-like and its market share now trounces that of its proprietary UNIX rivals. An MPP database can scale up (within a single SMP node) across all of the available cores in a server node. However, an MPP database also scales horizontally in the sense that it s distributed across multiple SMP nodes in a cluster. When an MPP database processes a query, each of the nodes in the cluster independently processes a piece of the query so instead of, say, 24 cores, an MPP database can muster 192 cores, 384 cores, 768 cores, and more. Open Source OpenStack: OpenStack is a massively scalable, open cloud computing platform designed for deploying and managing public, private, and hybrid cloud solutions through a single control plane. This open-source project has evolved quickly and many early adopters, including Intel, are using it to orchestrate large pools of compute, storage, and networking resources and to provide IT as a service to end users. The OpenStack community includes more than 18,000 members and 1,300 active contributors. Intel is a platinum member and a top contributor and is committed to making OpenStack easier to deploy and more capable through code contributions, blueprints, reference architectures, and other technical resources. Column-based Open Source Data Warehouse Business managers are increasingly aware that there is an urgent requirement for business intelligence (BI) that will spot trends about customers, products, markets and suppliers to identify new business opportunities, manage risk or formulate strategic initiatives that can lead to increased profitable revenues. Furthermore in the current recession companies should consider upgrading their BI capabilities to enable them to explore the how, why and when and not just the what of the data that is stored within their organisations. What is needed is advanced data analytics discover useful patterns within rapidly changing customer buying preferences to create cross-marketing opportunities and leading to increased customer revenues. At the heart of advanced analytics is the Data Warehouse (DW). However, most data warehouse implementations were not designed for advanced data analytics. Advanced data analytics requires fast access to data so that the data mining and statistical queries can be conducted in minutes rather than hours. ISSN: Page 122

5 Conclusion: There are in the sense that for most organizations, the reporting and analytics that data warehousing supports are mission-critical to the business, so it is important that they select a hardened, battle-tested, and reliable open source data warehouse. In the world of revolutionary, game-changing big data developments, data warehouse modernization may sound like an evolutionary development. But it is something that can be executed today, with existing data warehouse skills, and represents a simple first step toward gleaning immediate business value and organizational agility from big data technologies and open source technology for the continuous development and modifications. References: [1] Jiawei Han, Micheline Kamber, Data Mining: Concepts & Techniques 3rd ed. p. cm.... Chapter 4 Data Warehousing and Online Analytical Processing [2] TDWI Best Practices Report March-2016, Data Warehouse Modernization in the Age of Big Data Analytics [3] By TDWI e-book April-2016, Shaping the Future of Data Warehousing through Open Source Software [4] The_microsoft_modern_data_warehouse_white_paper ISSN: Page 123