PERFORMANCE MANAGEMENT FOR BIG DATA APPLICATIONS

Transcription

1 MEETS PERFORMANCE MANAGEMENT FOR BIG DATA APPLICATIONS Solution Brief 1

2 Introduction Cascading has emerged as a proven open source application development platform for building big data applications on Hadoop with more than 10,000 production deployments and growing. It is considered the de facto standard in open source application infrastructure technology. Acting as an abstraction layer, Cascading does the heavy lifting and converts applications built on Cascading into MapReduce jobs that run effectively on top of Hadoop. It delivers a computation engine, systems integration, data processing and scheduling capabilities through common interfaces. However, people who build and run big data applications using Cascading or other development platforms face a growing challenge. Their companies increasingly turning to big data processing for everything from product recommendations to fraud protection are demanding standards of quality and reliability they ve come to expect from traditional software. However, fragmented big data technologies, a dearth of monitoring tools, and sheer complexity have made it difficult to comply. While Cascading helps develop big data applications, more is needed to ensure performance and availability of these applications. Application performance management monitors and manages performance and availability of applications to detect and diagnose application performance issues so that applications deliver an expected level of service. APM [Application Performance Management] has been sorely missing from the Hadoop ecosystem at a level in which developers, IT ops, and even end users can quickly get to the bottom of any issues. Mike Matchett Technology Analyst Tanej Group 2

3 Cascading: A solid foundation for Big Data Applications Cascading has emerged as a proven open source application development platform for building big data applications on Hadoop with more than 10,000 production deployments and growing. It is considered the de facto standard in open source application infrastructure technology. Acting as an abstraction layer, Cascading does the heavy lifting and converts applications built on Cascading into MapReduce jobs that run effectively on top of Hadoop. It delivers a computation engine, systems integration, data processing and scheduling capabilities through common interfaces. Benefits of Cascading Build Robust Applications that Connect Business and Data Easily Mine, Model, and Analyze Big Data Quickly Develop Testable and Reusable Applications Leverage Java to Develop Complex Applications Minimize Application Failures Integrate Applications with Hadoop and Critical Systems 3

4 People who use Cascading or other development platforms have a great start on their journey to deliver big data applications. Their companies are increasingly turning to big data processing for everything from product recommendations to fraud protection and are now demanding the same standards of quality and reliability they ve come to expect from traditional software. Application performance management helps companies reach these goals. APM monitors and manages performance and availability of applications to detect and diagnose application performance issues so that big applications deliver an expected level of service. APM [Application Performance Management] has been sorely missing from the Hadoop ecosystem at a level in which developers, IT ops, and even end users can quickly get to the bottom of any issues. Mike Matchett Technology Analyst Taneja Group Filling the Gap: Driven and the Benefits of APM Application Performance Management is fundamental to the performance and availability of big data applications. Without a way to provide insights into your data pipeline, runtime behavior and resource consumption, IT is left responding to frequent pager calls and helpdesk tickets without the tools they need to resolve issues, data governance is limited by poor visibility and uptime isn t what it should be due to poor quality and reliability. Companies can realize a significant improvement in business continuity and in business impact by using application performance management to optimize their big data operations. To fill the void in application performance monitoring for big data applications, Concurrent, the team behind Cascading, designed Driven to help organizations address enterprise application and deployment challenges for business-critical applications. 4

5 Driven is a commercial application performance management tool that delivers unprecedented visibility into what big data applications actually do. Developers and operations teams collaborate through Driven to build better big data applications, run them more reliably, and manage big data resources more effectively. In addition to Cascading, Driven works with most major big data development technologies, including Hive, native Hadoop MapReduce and Spark. Visualize Your Entire Data Pipeline Driven is the only application performance management product designed to help developers and operations accelerate enterprise grade application development and management. A plugin seamlessly collects internal run-time and execution metadata directly from any Cascading application and visualizes complex applications and end-to-end data processing workflow. This deeper visualization allows users to troubleshoot errors, manage, monitor and search applications more effectively and in real time. Users get the unified view they need to support a proactive approach to application performance management and greater collaboration among various teams. 5

6 Given that applications are about to explode on Hadoop, Concurrent should do very well. Svetlana Sicular Research Director Gartner Track Runtime Behavior and Pinpoint Bottlenecks Driven also focuses on improving the reliability of big data applications. Driven s dashboard shows users exactly how their applications execute in real-time so that they can identify when they fail or are taking too long to complete. Driven pinpoints bottlenecks and identifies causes to quickly determine if code, data, hardware, network or cluster capacity issues were responsible for failure or poor performance. The dashboard also shows historical performance of application execution, SLA compliance and more to spot processing trends, identify governance risks and to promote continuous performance optimization. Understand Resource Consumption and Usage Driven makes it possible for developers and operations to manage big data applications more effectively by showing how all big data applications are consuming resources as they run. Users can organize and measure performance, resource consumption and other metrics for greater insight across teams, organizations or any user-defined segment. With Driven, organizations can intelligently account for resource costs and deliver transparent chargeback. 6

7 Build High Quality Big Data Applications Faster Run Big Data Applications More Reliably Manage Big Data Applications More Effectively Reduce Cluster Utilization Costs The coolest part about Driven is being able to visualize data pipelines and inspect components in real time for easy troubleshooting and optimization. I don t know of any other tool that s close in functionality. Neville Li Software Engineer Spotify Get Started With Driven Interested in seeing how Driven can supercharge your Cascading deployment and help you help improve the quality, reliability and manageability of all your big data applications? Sign up for a free trial today. 7