Spotlight. Big data and the mainframe

Transcription

1 Spotlight Big data and the mainframe A Spotlight Paper by Bloor Research Author : Philip Howard Publish date : March 2014

2 there needs to be an infrastructure in place to manage the inter-relationship between the new big data world on the one hand, and traditional environments, such as the mainframe, on the other. Philip Howard

3 Introduction More or less every major organisation in the world has a mainframe at the heart of its enterprise and it is critical that big data deployments are viewed from that perspective The discussion about big data to date has been primarily focused on the volume, variety or velocity of data. We are making the assumption that readers are familiar with big data and, indeed, with the concept of the Internet of Things (IoT) so we will not be discussing the three Vs. However, now that big data deployments are beyond the trial stage and into deployment it is time to consider some of the issues that arise when big data implementations transition beyond skunk works and into general-purpose use and, in particular, the issues that arise when organisations are integrating their mainframe system of record alongside big data implementations. Note that by mainframe we mean an IBM z/os environment, for which we are using mainframe as shorthand. More or less every major organisation in the world has a mainframe at the heart of its enterprise and it is critical that big data deployments are viewed from that perspective, rather than treated as isolated efforts that are distinct from the mainframe environment. Big Data must ultimately (if not sooner) link back to the system of record, the mainframe. What we will discuss in this paper are some of the issues surrounding these big data deployments and how they might be resolved within the context of a mainframe environment. A Bloor Spotlight Paper Bloor Research

4 Issues around big data deployments It is all too clear that many companies are making the same mistakes with big data as they have historically with little data There are at least six issues around big data deployments. Not all of these are specific to mainframe environments and some of them are more concerned with providing data analytics rather than the infrastructure required to support analytics, which is the subject of this paper. Nevertheless, we include all of these issues here for the sake of completeness but also to highlight the fact that big data deployments need careful consideration and planning. It is all too clear that many companies are making the same mistakes with big data as they have historically with little data. That is, they are building piecemeal solutions to resolve particular short-term requirements. This will lead to the same sort of siloed mess that bedevils conventional environments and which requires seemingly endless amounts of IT plumbing to join the whole thing up. According to the latest research from The Gartner Group, over 90% of existing IT budgets are taken up with maintenance of existing applications and their supporting infrastructure. A large part of the reason for this is because of the siloed, duplicated and complex nature of the existing environments. Doing the same with big data implementations will only exacerbate the situation and it is therefore really important to consider big data from a strategic rather than a merely tactical perspective. The issues are: 1. Many big data and IoT deployments are already business critical and this is likely to be even more the case as time progresses. This means that you need enterprise class products to support them, with high (or preferably continuous) availability and no single point of failure. 2. Big data environments are potentially complex and this complexity manifests itself in a variety of different ways: a. There are often multiple sources of data from which information needs to be combined, often in a variety of different formats (relational data, social media, clickstream data, machine-generated data, spreadsheets et al). In this paper we are specifically concerned with environments where at least some of that data, customer data for example, is held on the mainframe. b. The query environment itself is often complex, involving conventional data warehouses, data marts, stream processing products and platforms for data discovery such as Hadoop, which need to be combined from an architectural point-of-view. c. There may well be multiple big data or IoT initiatives within the same organisation, each of which is likely to involve multiple platforms, some of which may overlap with one another and some of which may not. d. There are multiple deployment options: some analytic results may need to be embedded within business processes (this is especially true of the Internet of Things) while in other cases users will want direct, self-service access to the data along with advanced visualisation techniques. Moreover, presentation of the data will typically be required across a range of different device types. e. Deployment options are further complicated by the fact that some platforms may be running on premise while others may be in public or private clouds. f. Many big data deployments require real-time or near real-time performance and, even when this is not the case, timely performance is still a requirement for enterprise class implementations. While there are certainly some NoSQL Bloor Research A Bloor Spotlight Paper

5 Issues around big data deployments products that are geared towards high performance, Hadoop, for example, is essentially a batch-based environment. This is fine where timeliness is not critical and, to be fair to the Hadoop community, it has been trying to improve performance, but it is not there yet. However, it is not just the performance of the database whether it may be warehouse, mart or discovery platform that is at issue: it is the performance of the whole environment, including the ingestion of data in the first place and its presentation once analysed. The short answer to how you get performance across the environment is that you not only need high performing analytics but also appropriate and fast pipes to get the data into, out of and through the analytic environment. 3. Big data deployments typically require expensive skills that are in short supply. The most obvious example is the need to program using the MapReduce framework. In the case of the former, of course, there are SQL initiatives in place but these are limited not only by the sophistication (or lack thereof) of the SQL being implemented but also by the absence of a database optimiser which is, in our view, essential to good SQL performance. Again, the community is working on this: for the latest version of Impala, for example, Cloudera has implemented join optimisations, but this is not yet a fully developed database optimiser. The other problem area related to skills is with respect to data scientists. In theory, data scientists are highly trained to ask the right questions of the data, given the particular industry they are working in and the organisation s marketing and business strategy. In practice, the required skill sets are simply too broad: not many people have extensive knowledge of their business, and statistical capabilities, and know about both data and IT, so a business model based on a presumption of the availability of data scientists is not going to be sustainable. Moreover, data scientists are expensive: average salaries are around $350, We have already mentioned, when discussing complexity, that a lot of the insight that is derived from big data will be embedded into automated business processes and applications. This raises another issue, which is that the big data environment needs to integrate with business process management and business rules management software in order to enable this process of embedding. If we think of the need to present information to users in various different forms as BYOD (bring your own device) we can extend that concept to BYOI (bring your own interface) if we also incorporate this sort of insight automation. 5. It is a fallacy, which all too many people fall into, to think that big data does not need the governance, compliance and security that is thought appropriate for structured data stored in relational and other databases. You need to know (and do something about it, for regulatory reasons around storing personal data), for example, if someone has mistakenly embedded his or her social security number in a tweet that you are analysing. Similarly, a great deal of machine-generated data requires de-duplication. There is also a reverse factor when it comes to social media and other personal information: customers need to know that they can trust you to safeguard their data. As mentioned, we are not going to discuss a number of these issues: what you choose to use for a discovery platform (Hadoop or some other NoSQL database or something else entirely) is outside these discussions, as are (big) data governance and the sort of analytical capabilities you might need. What we are concerned with is the interplay between the mainframe and the analytic environment and how that can be implemented in an enterprise-class manner. A Bloor Spotlight Paper Bloor Research

6 Mainframe big data issues The best way to discuss this is probably by example. Suppose that you are doing an analysis based on social media data and clickstream data stored in a Hadoop cluster. Moreover, this is not a one-time exercise but something that you are going to be doing on an ongoing basis to support, for example, one-to-one marketing. And further suppose that you want to enrich this social media data with information about customers that is normally stored on your mainframe. This immediately brings in various issues as in Figure 1. Get the data from the mainframe to Hadoop so that you can enrich the data you are analysing Get the results back to the mainframe from Hadoop Implement repeatable data movement processes, in both directions Ensure that the information coming back to the mainframe is appropriately formatted and of appropriate quality Figure 1: Mainframe Big Data some typical issues Bloor Research A Bloor Spotlight Paper

7 Mainframe big data issues Another issue is the avoidance of irrevocable assumptions that may become outdated The situation can actually get more complex than this, because you may have analytic capability within your mainframe environment (for example, you might have an Analytics Accelerator [Netezza]). Now you have options about where you perform queries: you could ship the data to Hadoop and do all the query processing there, or you could do initial analysis on Hadoop and ship the results to the mainframe where the final analysis is performed, or you could do some query processing on each platform and then join the results (which, again, could be performed in either location). Any one of these approaches might make sense for either cost or performance reasons and which approach might be most suitable will be dependent on the analytics in question and how they are going to be used. Another issue is the avoidance of irrevocable assumptions that may become outdated: for example, workloads should run on the most appropriate platform, and what that is may change as technology changes. For instance, extract, transform and load (ETL, possibly to a non-mainframe platform) may be seen as giving assurance that big data analytics will run cheaply and can t impact production performance. However, a mainframe can run different workloads entirely independently, at the same time, with welldefined priorities, so, it may become feasible to run some near real-time analytics directly on the production data instead of on a copy. The availability of specialist processors (essentially, free computing power for specialised tasks, including the running of Java Virtual Machines) may, in future, make the mainframe an unexpectedly cost-effective host for some big data analytics or associated workloads. You might also want to consider possible cultural clashes. Mainframe DBAs often see themselves as the custodians, not the owners, of the data in their databases. They are concerned with its provenance, its safety and its quality but leave the business to decide what to do with it. In the non-relational world, the custodian of the data, the big data administrator may also be (or have been) the business owner of the data and may want to use it to support a particular point-of-view or career opportunity. In terms of governance, mixing these two points of view may be an issue. For many companies it is still early days for big data deployments. That has resulted in many such implementations being cobbled together, often with manual processes for moving data around or, where tools are used, there tend to be a plethora of them. This is not satisfactory. A Bloor Spotlight Paper Bloor Research

8 Resolving mainframe big data issues How do we square all of these circles? In practice, many of the issues we have raised do not form part of the mainframe equation specifically. For example, if you are using Hadoop or some other NoSQL database then you need to ensure that the distribution you use is suitably robust for deployment in an enterprise environment. That is not related directly to the mainframe environment. Similarly, you probably want to move away from MapReduce because Java developers are expensive and deploy using SQL where that is feasible but, again, that is a big data issue rather than a mainframe one. Leaving those issues aside, therefore, what should you look for in terms of managing the interaction between the mainframe and the big data environment? The first thing is that there needs to be a repeatable process for moving data between the two environments, as opposed to something that is done in an ad-hoc manner. This means that you will need some sort of data integration capability that probably needs to include both ETL data virtualisation or federation capability, where the former supports the physical movement of data between the different environments and the latter allows you to create queries that span both environments without physically moving the data. Both of these techniques are likely to be useful for different types of analytics. The ETL approach, in particular, will be required for data discovery purposes (the sort of work that data scientists do). In addition, data integration may require replication (moving copies of the data in real-time) if real-time or near real-time analytics needs to be performed. However, being able to move the data backwards and forwards is one thing but it is not everything. You will need a scheduler or, more likely, you will want data integration processes to work with your existing mainframe scheduler. It is, of course, true that data integration products are typically delivered with their own capabilities in this regard but why have two sets of functionality to do the same thing when one will do? Having two just adds to the management overhead and you must make sure that any data moving back into the mainframe is clean and synchronised with the original database schemas. The semantics of customer, for example, in the big data environment may be very different to those in the relational environment, so if you move data back, you may find that you have extra customers, which may or may not be valid in the relational context. Speaking of management overheads, the other thing that you are really going to want is to be able to manage the whole relationship between the mainframe and the big data environments. This starts with having a graphical representation of the whole environment so that you can see where there are links, where data movement is taking place or is scheduled to take place, where there are failures and how you remediate them, and so on. In other words, you need a full management console that spans the two environments. Further, this management console needs to include collaborative capabilities since it is likely to be used both by database administrators in the mainframe world as well as the administrators looking after Hadoop and other platforms within the big data environment. While this may not prevent any cultural clashes between the individuals in these two areas, collaborative capabilities should at least help to reduce tensions Bloor Research A Bloor Spotlight Paper

9 Mainframe and big data futures Finally, we should address the point we made earlier about changing conditions. Processing that currently takes place on the mainframe may, at some point in the future, move to the big data environment. It is equally true that some of the processing that we now think will be Hadoop-based may be moved back to the mainframe (especially as new specialist processors are introduced) at some future point. The implication of this is that the management framework, put in place for co-ordinating the mainframe and big data environments, needs to be flexible enough to cater for these changing conditions. In particular, as this paper makes clear, it is necessary today to place the mainframe as a first-class player in any enterprise Big Data strategy. However, future repositioning of the mainframe as just another enterprise server, albeit one with particular qualities of resilience, performance and availability, providing high quality cloud services in a box, could take this a step further. Bloor envisages a future where smart schedulers place workloads dynamically on the most appropriate platform, in accordance with predefined policies for security, availability, service level and so on, together with feedback from customer and end-user experience (using big data analytics). This is only feasible in practice for enterprises with a fully formulated cloud strategy but could start to give market leaders a competitive edge in the short to medium term. This vision is probably a little way out for most enterprises today, but they should start thinking about the possibilities now, in order to avoid making tactical decisions that limit their future choices. Bloor envisages a future where smart schedulers place workloads dynamically on the most appropriate platform, in accordance with predefined policies for security, availability, service level and so on, together with feedback from customer and end-user experience (using big data analytics) A Bloor Spotlight Paper Bloor Research

10 Conclusion Big data and the Internet of Things present huge opportunities for companies to become more efficient in a variety of ways for example, to improve customer intimacy and to increase operational efficiency that will ultimately improve financial results. However, according to Forrester Research, only 1% of the world s data is currently analysed so there is enormous scope for growth in this analysis. Unfortunately, this scope threatens uncontrolled growth in the deployment of big data analyses and that poses a significant risk that projects will get out of hand and management and oversight will fall by the wayside. In our view, big data initiatives need careful planning, not just in their own right, but also within the context of future big data projects that can already be foreseen. In addition, there needs to be an infrastructure in place to manage the inter-relationship between the new big data world on the one hand, and traditional environments, such as the mainframe, on the other. Without such a management framework administrative costs will increase, efficiency will be reduced and it is all too likely that siloed applications will cause as much pain and suffering in the future as they have in the past. Further Information Further information is available from Bloor Research A Bloor Spotlight Paper

11 Bloor Research overview Bloor Research is one of Europe s leading IT research, analysis and consultancy organisations. We explain how to bring greater Agility to corporate IT systems through the effective governance, management and leverage of Information. We have built a reputation for telling the right story with independent, intelligent, well-articulated communications content and publications on all aspects of the ICT industry. We believe the objective of telling the right story is to: Describe the technology in context to its business value and the other systems and processes it interacts with. Understand how new and innovative technologies fit in with existing ICT investments. Look at the whole market and explain all the solutions available and how they can be more effectively evaluated. Filter noise and make it easier to find the additional information or news that supports both investment and implementation. Ensure all our content is available through the most appropriate channel. Founded in 1989, we have spent over two decades distributing research and analysis to IT user and vendor organisations throughout the world via online subscriptions, tailored research services, events and consultancy projects. We are committed to turning our knowledge into business value for you. About the author Philip Howard Research Director - Data Management Philip started in the computer industry way back in 1973 and has variously worked as a systems analyst, programmer and salesperson, as well as in marketing and product management, for a variety of companies including GEC Marconi, GPT, Philips Data Systems, Raytheon and NCR. After a quarter of a century of not being his own boss Philip set up his own company in 1992 and his first client was Bloor Research (then ButlerBloor), with Philip working for the company as an associate analyst. His relationship with Bloor Research has continued since that time and he is now Research Director focused on Data Management. Data management refers to the management, movement, governance and storage of data and involves diverse technologies that include (but are not limited to) databases and data warehousing, data integration (including ETL, data migration and data federation), data quality, master data management, metadata management and log and event management. Philip also tracks spreadsheet management and complex event processing. In addition to the numerous reports Philip has written on behalf of Bloor Research, Philip also contributes regularly to IT-Director.com and IT-Analysis.com and was previously editor of both Application Development News and Operating System News on behalf of Cambridge Market Intelligence (CMI). He has also contributed to various magazines and written a number of reports published by companies such as CMI and The Financial Times. Philip speaks regularly at conferences and other events throughout Europe and North America. Away from work, Philip s primary leisure activities are canal boats, skiing, playing Bridge (at which he is a Life Master), dining out and walking Benji the dog.

12 Copyright & disclaimer This document is copyright 2014 Bloor Research. No part of this publication may be reproduced by any method whatsoever without the prior consent of Bloor Research. Due to the nature of this material, numerous hardware and software products have been mentioned by name. In the majority, if not all, of the cases, these product names are claimed as trademarks by the companies that manufacture the products. It is not Bloor Research s intent to claim these names or trademarks as our own. Likewise, company logos, graphics or screen shots have been reproduced with the consent of the owner and are subject to that owner s copyright. Whilst every care has been taken in the preparation of this document to ensure that the information is correct, the publishers cannot accept responsibility for any errors or omissions.

13 2nd Floor, St John Street LONDON, EC1V 4PY, United Kingdom Tel: +44 (0) Fax: +44 (0) Web: