WHITE PAPER Redefining Monitoring for Today s Modern IT Infrastructures Modern technologies in Zenoss Service Dynamics v5 enable IT organizations to scale out monitoring and scale back costs, avoid service disruptions, and respond quickly to business demands.
Table of Contents INTRODUCTION... 1 SCALE OUT MONITORING, SCALE BACK COSTS... 1 Simplifying Unified Monitoring Operations...2 Deploy Monitoring More Quickly and Easily...2 Rapidly Move or Update Services...2 Administer Monitoring Operations More Efficiently...3 AVOID SERVICE DISRUPTIONS... 4 Improving Real-time Performance Data Collection...4 Leverage Big Data Storage and Analysis...4 Gain Real-Time Visibility Into Performance Data...4 Make Data Collection Cheaper and More Realiable...5 RESPOND QUICKLY TO BUSINESS DEMANDS... 5 CONCLUSION... 6
Introduction Modern IT infrastructures have evolved significantly in recent years, but monitoring solutions haven t kept pace. Most organizations still rely on either a monolithic legacy framework that can t extend to today s technologies OR dozens of siloed specialty monitoring tools that present a fragmented view of operations. Both of these approaches slow down IT to the point where service issues can t be addressed before users are impacted. The new service-oriented architecture in Zenoss Service Dynamics allows administrators to implement monitoring of new services more quickly, streamline updates and upgrades, and easily scale to meet business demand. Underpinning this architecture is a host of new technologies that simplify unified monitoring operations and improve the collection of real-time performance data. Learn how these technologies provide greater flexibility, scalability, and control for comprehensive monitoring of your IT environment. Scale Out Monitoring, Scale Back Costs Simplifying Unified Monitoring Operations The new Control Center in Zenoss Service Dynamics dramatically simplifies operations. Control Center is an application service orchestrator that provides out-of-band management services. It allows you to manage collections or pools of compute, network, and storage resources. It maps services to these resource pools, rather than to individual hosts, and provides a distributed file system for application data. If a host in a resource pool fails, Control Center automatically starts a new instance of the service on a different host in the pool. With control center, monitoring becomes as dynamic as any virtualized or cloud infrastructure you want to monitor. 1
The new technologies that support Control Center s service-oriented monitoring infrastructure include: Docker Btrfs and NFS Apache ZooKeeper logstash ElasticSearch Kibana Apache HBase OpenTSDB Redis Through the use of these technologies, administrators using Zenoss Service Dynamics are able to: Deploy monitoring more quickly and easily Rapidly move or update services Administer monitoring operations more efficiently Deploy Monitoring More Quickly and Easily The foundation technology in Control Center is Docker, an open platform that developers and system administrators use to build, ship, and run distributed applications. Docker facilitates the quick assembly of applications from components, which makes application deployment easier and more efficient. With Docker, components run inside of containers that contain any and all application dependencies. This removes any possibility of software conflicts within a package, since each container is isolated from everything else running on that machine. In addition to improving efficiency, Docker containers also create a consistent running environment. Consistency reduces operational overhead, which is crucial when monitoring cloud-scale service infrastructures. In Zenoss Service Dynamics, administrators package up all services in Docker containers that include the necessary prerequisites, such as the appropriate versions of RabbitMQ, MariaDB, and Python. Once containers are created, Control Center can deploy, upgrade, or move these containerized services as needed. For example, if one host in a resource pool goes down, Control Center can automatically move the impacted services to a new host, ensuring monitoring is uninterrupted. Rapidly Move or Update Services Zenoss Service Dynamics can move services dynamically between hosts by leveraging new file system and configuration technologies. 2
Docker Docker supports union file systems. A union file system consists of a set of layers, making it easier to coordinate updates and upgrades with a single, unified process. Each update builds on the previous layers of code, so when you apply a patch or update, you are only modifying one layer. Once a change is committed, you can push that change as a new layer out to all of your hosts, which look for and install only that layer. If any issues occur during installation, you simply uninstall that layer to roll back the application to its previous state. This makes updates very lightweight and fast. Btrfs and NFS In order to share and manage files across multiple services, Control Center uses the B-tree file system (Btrfs) and Network File System (NFS). Btrfs allows Zenoss Service Dynamics to take snapshots of the file system. NFS allows all services to share files across any number of servers. Apache ZooKeeper To coordinate configuration of Zenoss Service Dynamics services across multiple hosts in a resource pool, Control Center also takes advantage of Apache ZooKeeper. ZooKeeper is a centralized service for maintaining configuration information, naming, and providing distributed synchronization and group services. Administer Monitoring Operations More Efficiently Zenoss Service Dynamics includes new technologies to help make administration and management of your IT operations easier by providing better tools for collecting, searching, and interpreting log data. logstash Centralized logging is implemented in Control Center using logstash, a common tool that collects, parses, and stores log information. The tool takes logs from all Zenoss Service Dynamics components and centralizes them in a logstash master. That means administrators no longer need to open a command prompt to track down a specific log file that they need. All Zenoss logs are in a single, searchable location. ElasticSearch To make it easier for administrators to search all those logs, as well as service definitions, Zenoss Service Dynamics uses ElasticSearch. ElasticSearch is a search server based on Lucene. It provides a distributed, multitenant-capable full-text search engine with a RESTful web interface and schema-free JSON documents. By using ElasticSearch, Control Center has a system to store, index, and powerfully search very large sets of data, making it easier for administrators to find the logs they need and zero in on the source of any performance or availability issues. Kibana To make searching and viewing logs even easier, Control Center also incorporates Kibana as a front-end visual library for all log information. As a data visualization engine, Kibana allows you to natively interact with all of the data in ElasticSearch via custom dashboards. These panels can be saved, shared, or exported according to your business requirements and will automatically update based on real-time search queries. Kibana is also used inside of Resource Manager to view logs from all of your deployed collectors. 3
Avoid Service Disruptions Improving Real-time Performance Data Collection In addition to simplifying monitoring operations, the Zenoss Service Dynamics service-oriented architecture also includes technologies that help improve performance data collection and reporting. In earlier releases, Zenoss Service Dynamics stored performance monitoring data in Round Robin Databases (RRDs) on each collector server. These RRD files offered a limited window of real-time performance data, because as soon as the RRD file filled up, it started overwriting the oldest data. This required expensive, high performance storage hardware at each collector. In addition, if the collector failed, you would also lose all of the performance data in that RRD. To avoid this potential data loss, you needed to deploy duplicate servers for failover purposes. The latest version of Zenoss Service Dynamics implements three new storage technologies, Apache HBase, OpenTSDB, and Redis, that increase the performance, availability, and scale of performance data storage. These new technologies support the requirements of a service-oriented architecture by centralizing data collection in a way that gives IT the ability to: Leverage Big Data storage and analysis Gain real-time visibility into performance data Make data collection cheaper and more reliable Leverage Big Data Storage and Analysis The latest version of Zenoss Service Dynamics uses Apache HBase, a subproject associated with Hadooop, as the backend data store. Apache HBase is an open-source, distributed, versioned, non-relational database. It runs on a multinode cluster, allowing it to have infinite scale. With this new Big Data backend, it is possible to store a virtually unlimited amount of performance and availability data at the resolution it was collected at no rollup required. HBase gives you the flexibility and power of a Big Data solution for reporting at the most granular levels in any window of time you choose to evaluate historical trends. It can also provide long-term storage for compliance purposes. Gain Real-Time Visibility Into Performance Data Real-time performance data is read from and written to the HBase backend with an OpenTSDB (Open Time Series Database). This is important because HBase on its own isn t equipped with any knowledge about time series data a requirement when you are dealing with performance data. As its name suggests, OpenTSDB can easily handle the high transaction volume of time series data required in a real-time monitoring environment which is critical in understanding the real-time state of your service infrastructure. In addition to being able to handle real-time data collection, OpenTSDB also scales very efficiently. OpenTSDB consists of a Time Series Daemon (TSD) as well as set of command line utilities. Interaction with OpenTSDB is primarily achieved by running one or more of the TSDs. Each TSD is independent. There is no master and no shared state, so you can run as many TSDs as required to handle any load you throw at it. 4
Make Data Collection Cheaper and More Realiable While centralized storage of all data happens in HBase, the latest release of Zenoss Service Dynamics also caches data locally on the collector via a Redis (REmote DIctionary Server). This gives Zenoss the capability to retain performance data even if the collector loses connection to the HBase server. The data will stay cached locally until the collector reconnects to the HBase backend, ensuring that no data is lost. In terms of cost efficiency, Redis is a big improvement over the RRD files in previous versions of Zenoss Service Dynamics. It persists performance data by writing it continually to the OpenTSDB, so there is no longer a need for expensive local storage to support each Redis instance. Also, because the Redis never gets full, no data is ever overwritten. Respond Quickly to Business Demands The new service-oriented architecture in Zenoss Service Dynamics gives you even more flexible, scalable, and efficient unified monitoring so you can respond quickly to business demands. Leveraging a series of new technologies, the latest version of Zenoss Service Dynamics is not only more powerful it is easier to manage and can better scale to meet the needs of any size enterprise. IT Operations teams can quickly adopt new datacenter and cloud technologies and make changes and adjustments in real-time. This reduces friction in delivering on strategic business initiatives, helping elevate the value of IT Operations with both business leaders and DevOps teams. The use of modern technologies in Zenoss is strengthened by an open architecture with APIs that support quick, easy integration and automation. You can easily tie together orchestration/provisioning systems with Zenoss so that when new systems/resources are provisioned they are automatically monitored. And then when thresholds are crossed or critical events occur, you can automatically generate events in your ITSM (IT Service Management system, e.g. ServiceNow) to follow existing remediation workflows. 5
Conclusion IT Operations needs end-to-end infrastructure visibility and control with flexible, unified monitoring that adapts easily to any environment, identifies service issues more quickly, and reduces infrastructure and productivity costs. Zenoss Service Dynamics meets these requirements, monitoring today s highly dynamic enterprise IT infrastructures with a single, unified platform. Its service-centric view into the availability and performance of the physical, virtual, and cloudbased infrastructure supporting your IT services. Enabled with modern technologies, the solution allows IT operations to scale out monitoring and scale back costs, avoid service disruptions, and respond quickly to business demands. For additional information on the Zenoss Service Dynamics Architecture - Read More 11305 Four Points Drive Building 1, Suite 300 Austin, TX 78726, USA Phone: +1-512-687-6854 Toll-free: +1-888-936-6770 www.zenoss.com Zenoss and the Zenoss logo are trademarks, or service marks of Zenoss, Inc. All other trademarks listed in this document are the property of their respective owners.