Integrate Big Data into Business Processes and Enterprise Systems. solution white paper

Transcription

1 Integrate Big Data into Business Processes and Enterprise Systems solution white paper

2 THOUGHT LEADERSHIP FROM BMC TO HELP YOU: Understand what Big Data means Effectively implement your company s Big Data strategy Get business value out of Big Data without having to build everything from scratch With the rise of Big Data, a data-driven approach to business is transforming the enterprise. Companies today are thinking about and using data in myriad new ways to drive business value, from reducing risk and fraud in the financial sector to bringing new pharmaceuticals to market more quickly at a higher level of efficacy. Retailers can track purchase patterns and consumer preferences more accurately to guide product and marketing strategies. Media companies can offer more accurate recommendations and create specialized promotions. Businesses of all kinds can identify new revenue opportunities and operational efficiencies. Big Data can mean different things to different organizations, but one theme remains constant: Big Data calls for a new way of thinking about technologyone focused squarely on business outcomes. To enable the creation of a data-driven organization, IT needs to implement Big Data technologies into the enterprise environment. As companies scramble to come up to speed on both the technology and practice of Big Data, Apache Hadoop has emerged as a lynchpin technology, providing a framework for storing and processing the vast data sets involved in Big Data. Hadoop has quickly grown into a huge market, reflecting a fundamental shift in the way businesses use data. As support for Big Data has evolved and matured, though, it has become clear that Hadoop isn t the whole story. While unquestionably vital, Hadoop interacts with and depends on other essential enterprise components. IT can t afford to think of Hadoop as an island or silo, as many solution vendors initially did. From the earliest stages of Big Data initiatives, it s critical to understand where Hadoop instances are, what they look like, and the business services they supporteven if they haven t been fully operationalized yet. KEY BENEFITS OF AN ENTERPRISE APPROACH Reduce the time and effort to deliver Big Data business services with enterprise workload automation Maximize application developer and system administrator productivity with self-service provisioning when building dev, test, and production environments Leverage your existing infrastructure effectively with intelligent use of cloud and virtualization Achieve high levels of compliance and governance with application discovery and dependency mapping that ensures an up-to-date inventory of your Big Data assets To be able to manage and leverage Big Data effectively, in context, IT needs solutions that accommodate the building of holistic business processes that span additional technologies, platforms, and applications. Fortunately, many of the tools needed for this enterprise approach to Big Data are already available. The most successful initiatives will seek to leverage existing capabilities and expertise as much as possible. This white paper discusses a holistic approach to Big Data in which traditional, mature enterprise management solutions are extended to help automate, accelerate, and integrate Hadoop into the environment. UNDERSTAND BIG DATA IN AN ENTERPRISE CONTEXT It s important to ensure that your Big Data initiatives have a clear business need and purpose that they re business initiatives powered by technology, not technology initiatives for their own sake. To avoid expending resources on projects that won t drive value, begin by identifying key business challenges in the enterprise, then evaluate whether Big Data could help you solve them. 1

3 The key is to think of Big Data in terms of the business services it can power. A foundational concept for today s enterprise IT, business services are capabilities that IT delivers to the business and its customers to accomplish specific goals. By aligning technology with business operations, business services make IT services more easily consumable to address important business cases. Customer-facing business services range from online banking and e-commerce to flight and shipment tracking. Within the enterprise, business services include automated inventory replenishment, fleet management, automated couponing, and many other tasks. Business services also help organizations reduce risk by identifying potential fraud or non- compliance, especially in the financial services market. A Big Data initiative must include a clear understanding of the relationship between each Big Data service and the IT elements that support it. This helps you understand how to optimize the performance of critical services according to their role in various business processes, as well as understanding the impact of specific problems or outages in the IT environment on the business so you can prioritize their resolution accordingly. As your technology environment continues to evolve, you can better foresee how changes and new development will impact the business and its customers. While this insight is key for any business service implementation, the nature of Big Data projects makes it especially important, as they typically involve a more complex application architecture. This results in part from the way mobile devices and apps are changing the way people use data and services. Instead of relying on an all-purpose application or navigating a portal full of menus, enterprise and consumer users want more a personalized experience that addresses their specific needs. Businesses have responded by introducing a plethora of narrowly focused micro- apps, overturning the traditional one-to-one relationships among user sets, applications, and databases. IT must now maintain a back end that supports multiple apps with overlapping data sets, where any given app may be susceptible to event-driven usage spikes, such as heavy traffic on ecommerce sites during the holiday season or high demand on flight status and booking apps during a major storm. To ensure consistent service levels while optimizing utilization and controlling costs, IT must move beyond traditional siloed architectures and implement a scalable, flexible data cloud that includes not just Big Data, but also transactional systems and relational database management systems (RDBMS). The central role of business services in today s enterprises, and the more complex architecture through which they are delivered, make it essential to manage Big Data solutions from a business perspective. IT needs to manage components and services according to the business services they support, focusing on business objectives and benefit, and prioritize resources and activities according to the needs of the business. In this way, IT can ensure optimal service for internal and external customers, more effective support for business goals, and greater efficiency in the allocation of resources. AN ENTERPRISE APPROACH TO BIG DATA PROJECTS The perception of Big Data as a fast-evolving area with potentially transformative impact can lead IT organizations to approach it as a skunkworks project where people spurn established technologies and invent things from scratch. This is neither sustainable nor effective. Big Data is a new way of thinking about enterprise data and how it can drive business valuenot a wholesale replacement of systems and processes. It has to fit into your existing IT environment and meet all the usual requirements around standardization, compliance, efficiency, and manageability. Each Big Data initiative must address three fundamental challenges: Data acquisition Is there data that you could mine or analyze to drive business value? Is it in-house or elsewhere? Are there untapped enterprise stores you can leverage? Operations How will you get the data in question into a data store, and how will you access it to power the business services that depend on it? Data freshness How will you incorporate new data or change data sources over time to ensure that your analytics remain relevant to your business needs? 2

4 Your approach to data acquisition will depend on your specific business priorities. It s no less important to address your approach to data freshness up front. You should have a strategy in place to update your analytics to reflect changes in your business, customer behavior, data environment, and other factors. Otherwise, you may find that the value and relevance of the initiative diminishes over time. The operational challenge posed by Big Data initiatives often proves particularly difficult for companies to solve. Big Data operations revolve around two major components. First, you need to be able to store data reliably and resiliently across a number of nodes using the Hadoop Distributed File System (HDFS). Once your data has been distributed, you need a way to get it out again. The Hadoop MapReduce model provides a way to determine which data is needed, assemble it, and analyze it, leveraging multiple CPUs and IO across the HDFS cluster. While the technologies behind Big Data are new, its operational processes and methodologies are similar to many technology trends that have come before, and it is neither necessary nor desirable for IT to introduce entirely new methods for managing them. Too often, organizations overlook the need to integrate Big Data operations into not only their existing IT environment, but their established approach to workload management as well. Hadoop solutions involve tasks such as data visualization, ETL, repetitive analysis, and dashboards, often based on data drawn from multiple stores and systems throughout the enterprise, which must be loaded, processed, and extracted efficiently to feed into Big Data- powered business services. The operation of these services thus revolves around a large number of separate but interdependent workloadsa complex scenario with large processing clusters, a high number of data streams, multiple analytics to be performed, and often demanding requirements for timing and synchronization. Fortunately, these are well-known problems that people have been dealing with since the beginning of commercial computing. You don t need to reinvent the wheel or train people to do new thingsexisting management methods and readily available solutions can be applied to Big Data and Hadoop applications as well. Enterprises today typically manage large-scale processing through a batch approach: groups of similar workloads are scheduled to be run on a mainframe simultaneously. Batch processing is now used to manage 70 percent of all business processingand it will play a crucial role in Hadoop processing as well. First, though, companies must make sure their batch processing capabilities are up to the challenge. In the more complex IT architectures of the cloud and mobile era, effective job scheduling requires the orchestration of independent real-time and batch processes across multiple, disparate IT platforms. This has driven the rise of a new generation of workload automation solutions that replace traditional, siloed job schedulers and teams with a more unified, holistic approach across the enterprise. This ensures consistent service levels, improves business integration, and reduces cost and risk. GET THE MOST OUT OF YOUR BIG DATA INITIATIVE Accelerate the building of new Big Data business applications Reduce the time spent provisioning infrastructure in support of Big Data projects Ensure your Big Data projects adhere to corporate governance and meet all regulatory compliance requirements Enable IT to easily integrate Big Data services into your existing technology landscape Easily perform day-to-day management such as patching of your Big Data environments 3

5 This enterprise approach is particularly critical for batch processing in a Hadoop context because Big Data relies so heavily on interaction with the rest of the IT landscape. Enterprise workload automation helps you interact with all these problems natively, in a holistic manner, so you can build enterprise-level workflows that include Hadoop the same way as any other service, and incorporate Big Data services into any of your existing enterprise workflows. This greatly simplifies Big Data operationsyou can more easily see when you should run a particular workload, when it will finish, and how to deal with any problems that may arise. CONCLUSION The consensus is clear: Big Data technologies can have a major impact on business, helping companies serve customers and achieve their goals more effectively to drive growth and competitive advantage. In light of this potential, IT organizations face intense pressure to derive business benefits from Big Data initiatives. Although it may be tempting to throw effort and resources into rapidly deployed projects that lie outside the established environment, an enterprise approachone based on established methods and processesis far more effective. While Big Data represents a new approach for extracting value from data, it is not a fundamental shift in architecture, technology, or best practices. Mature IT management solutions such as enterprise workload automation can accelerate Big Data project delivery and create sustainable, operational environments that are much more likely to deliver ongoing business value. To learn more about this approach and how to implement it, please visit BMC delivers software solutions that help IT transform digital enterprises for the ultimate competitive business advantage. We have worked with thousands of leading companies to create and deliver powerful IT management services. From mainframe to cloud to mobile, we pair high-speed digital innovation with robust IT industrializationallowing our customers to provide amazing user experiences with optimized IT performance, cost, compliance, and productivity. We believe that technology is the heart of every business, and that IT drives business to the digital age. BMC Bring IT to Life. BMC, BMC Software, and the BMC Software logo are the exclusive properties of BMC Software, Inc., are registered with the U.S. Patent and Trademark Office, and may be registered or pending registration in other countries. All other BMC trademarks, service marks, and logos may be registered or pending registration in the U.S. or in other countries. UNIX is the registered trademark of The Open Group in the US and other countries. Tivoli and IBM are trademarks or registered trademarks of International Business Machines Corporation in the United States, other countries, or both. IT Infrastructure Library is a registered trademark of the Office of Government Commerce and is used here by BMC Software, Inc., under license from and with the permission of OGC. ITIL is a registered trademark, and a registered community trademark of the Office of Government Commerce, and is registered in the U.S. Patent and Trademark Office, and is used here by BMC Software, Inc., under license from and with the permission of OGC. All other trademarks or registered trademarks are the property of their respective owners BMC Software, Inc. All rights reserved. Origin date: 12/14 * *