On-Device Network Management Systems An Enterprise Management Associates White Paper October 2007
Table of Contents Introduction...1 New Services Increase Network Complexity...1 Enterprise and Carrier Requirements are Converging...1 Network Management Design Challenges...1 High Availability...1 Scalability and Performance...2 Security...2 Fine-grained Control and Logging...3 Data Consistency and Integrity...3 Change Automation...4 Rapid Development...5 Conclusion...5
Introduction The key issues involved in developing on-device network management systems for carrier-grade enterprise and operator networks are the same for any other networked resource, only with more complexity and built-in restrictions due to the limited space and processing capabilities inherent to these types of devices. The requirements for on-device network management systems will be reviewed in the context of the business and technical expectations of network operators that drive these requirements and their ever changing network environmental demands. Increasing levels of complexity and converging industry requirements are the key factors that are continuing to shape this unique industry segment. New Services Increase Network Complexity Modern networks have become increasingly sophisticated platforms for delivering data, voice, video, and wireless services. Network management systems must adapt to meet new challenges. The old paradigm of humans administering networks with a set and forget approach is long gone. Customer expectations for network performance have never been higher. Users want applications and services available in their homes, offices, and everywhere between without interruption or limitations. Network operators and equipment vendors are working hard to make this a reality. Competitive and economic forces are also affecting the networking industry. While the cost for the network equipment itself has been squeezed over many years, the cost for deploying new services and maintaining the networks has increased dramatically. Service providers and enterprises are constantly looking to reduce the operating cost of their networks as a means to become more competitive. Enterprise and Carrier Requirements are Converging Interestingly, the challenges facing CIOs of enterprises often mirror those of commercial service providers. Larger enterprise networks have many similarities with service provider networks. For example, the five-nines expectation for telco equipment has now crossed into the enterprise as Voice over Internet Protocol (VoIP) penetration explodes. Users rightly expect a dial tone every time they pick up the phone whether it comes over a Time-Division Multiplexing (TDM) or an IP network. Network Management Design Challenges The developers of networking equipment and their network management systems must meet or exceed the expectations of network operators in an environment that is increasingly complicated. The number of network devices and their form factors are proliferating. In addition to appliances and Advanced Telecommunications Computing Architecture (ATCA) compliant chassis products, blade servers are being increasingly adopted for networking applications. The growth in the number of network devices requires multiple devices to be configured and supported for flexible high availability frameworks. The number and complexity of required northbound management interfaces is also increasing. In addition to simple network management protocol (SNMP) and command line interface (CLI) agents, web and NETCONF/ Extensible Markup Language (XML) interfaces are increasingly expected on most network devices. All these management interfaces must be synchronized to ensure consistency. Flexible provisioning and frequent re-configuration is needed to support multi-service networks. As if this was not enough, developers of networking equipment must bring complete solutions to market in less time, with constrained resources, and often using distributed teams responsible for modular components of the overall system. High Availability Up until recently, network management was not high on a network operator s punch list for ensuring continuous availability. However, as networks become dependent on frequent software updates and dynamic service provisioning, this is changing. Building a highly available network management application requires a powerful software fabric with capabilities including database replication, support for masterslave heart beat, full redundancy, hot failover, multiple levels of security access controls, complete analytical drill-down functionality and of course a full line of reporting capabilities.
It is common for chassis-based network devices to have a separate management card managing individual line cards as shown in Figure 1. This capability requires an application programming interface (API) to make it possible for the management cards to manage resources on other hosts. Figure 1 Dual management cards should have the capability to be configured to ensure high availability, master-slave heart beat, and full redundancy. All configuration changes are always written to both management cards and when re-configuring the master all data is also replicated to the slave. Upon a failure of the master, the system will switch over to the slave and the slave will become active in the system. Some form of alerting, or warning, mechanism is necessary to notify network operations staff of the event and subsequent change in operational status of the device. Additionally, drill-down analytics is a necessity for the network operations staff enabling them to dive into the heart of any network event and ascertain the true cause and nature of the event. With this type of architecture it is also possible to upgrade the system without bringing down the entire service or device. To deliver redundancy in a scalable fashion, data replication should be possible from a single master to N number of slaves where N is not arbitrarily limited. Slave units or redundant devices could be physically located anywhere on the network. Control of the master, slaves, and redundant devices should be from a single console window. The same console window will also be the central alert / notification display mechanism for this type of network management architecture. Scalability and Performance Carrier-grade network management applications must have the capability to monitor, provision, and configure very large networks without impacting service delivery. The first problem is storing large numbers of configuration parameters. In a large complex network the number of configuration parameters can be in the millions. This implies the need for a large, robust, real-time data store that can scale linearly without performance degradation. The problem is much more difficult at the network device level where memory and processing resources are limited and not easily expanded without additional hardware additions. The best solution is to use a data store that is optimized for configuration and operational data as well as the types and database transactions specific to network management. The second issue is the performance and scalability of the transaction engine that manages the communication stream to end devices. To implement a configuration change in a large network implies the need for a transaction engine that can scale to tens of thousands of operations per minute. The growing use of XML data for configuration management requires the data streaming process in the transaction engine be highly tuned to these performance needs. In order to address growing need for scalable performance, some networking products are based on blade servers and stackable appliances as a means of delivering scalable performance. Both these approaches share the benefit of having the capability to add capacity without disrupting the network. The challenge is for the configuration management system to operate without any dependency on the hardware configuration. For example, each blade in a cluster should be unaware of the fact that it is running in a clustered environment. One of the blades must be dynamically assigned a management role and network administrators must have visibility to all network devices as they are added on the fly. This later capability is sometimes referred to as a cluster join. Security Simple password protection is not enough for a carrier-grade network management system. With a high incidence of severe threats and attacks on information assets, security has become a priority at the highest levels inside an organization. In addition to mitigating threats to mission-critical network systems, network operators and enterprises must also comply with a wide range of regula-
tions that require them to implement and verify the effectiveness of security information management controls. The first step in providing appropriate security is user authentication. A user must present credentials, such as a password or a public key, in order to gain access. The ability to support security solutions based on protocols such as Remote Authentication Dial-In User Service (RADIUS) and Lightweight Directory Access Protocol (LDAP) is an important option for any good authentication system. A remote authentication server will typically store both the users login credentials and their group information that can be applied to authorizing their access rights. The second step in the security model is for authorization to be performed. Even though a set of credentials are presented and confirmed as being acceptable, there s still the question of what actions, tasks, changes, etc. are to be allowed under the scope of those credentials. Once a user is properly and successfully authenticated, all operations performed by that user must be authorized by the appropriate access control source inside the organization. The internal access control source must confirm that the credentials presented are in fact authorized to perform the operation intended. If they are, then the intended operation is allowed to be performed. If those credentials don t have the necessary permissions or access control rights to perform the requested operation, then the request is denied. Accounting and auditing is the third major leg of the traditional AAA (authentication, authorization and accounting) services needed for robust security. When logins fail, access controls are denied, or unauthorized changes are attempted, those events must be recorded and reported to the appropriate authorities. There in lies the need for strong, robust, reporting and auditing capabilities that can take on many forms, i.e. compliance reports (Sarbanes-Oxley, HIPPA), forensic analysis, and billing. Since many network managements systems include a web interface, it is also important that the embedded web server does not expose the system to security vulnerabilities. Considerations here include using HTTPS for all secure communications between the client and the server plus avoiding cross-site scripting and caching configuration values. Fine-grained Control and Logging The ability to allow identified users or user groups to perform specific tasks is important in any large network and is the corner stone of a role-based system of administration. To provide true fine-grained control over which users or groups can execute particular actions requires authorization control of commands, data access and device access. Figure 2 As illustrated in Figure 2, role-based access control allows users to change while roles remain constant. When responsibilities for network administration are organized by seniority, geography, or line of business, role-based access control is very beneficial. Providers of managed services also need such granular control. Here the need is for multi-tenanted access to the network management application. Individual clients of a managed security gateway, for example, must be able to view their virtualized security policies and only make changes to rules and data that only affect their organization. Data Consistency and Integrity It is highly advantageous to have a single view, or window, of all sessions in all the northbound interfaces. As a result, access rules and audit trails can be uniformly applied by the CLI and web interfaces. To ensure data consistency, a carrier-grade network management system must guarantee that configuration changes made in any of the management interfaces are either entirely
completed or aborted and properly rolled back thereby ensuring that the integrity of the configuration, and ultimately the network, is maintained. Figure 3 Traditionally, network management applications have used stovepipe architecture as depicted in Figure 3. Here each management interface talks to managed objects using its own software adaptors and APIs. This cumbersome approach not only strains development resources, but also risks compromising configuration integrity. Ideally, backplane software should set up a transaction for every committed configuration change from northbound interfaces. It then should talk to all affected managed objects in a specific order, waiting for them to acknowledge that the change has been accepted and duly processed and successfully completed. If any of the managed objects returns an error (e.g., failed to set a specific configuration parameter in the routing stack), the backplane ensures that any changes made up to that point are completely and successfully rolled back. Figure 4 illustrates an architecture based on a single unified backplane or transaction engine. Figure 4 The use of a synchronous subscription API plays a useful role in ensuring data integrity. This allows managed objects to receive a notification over a subscription socket when configuration data changes. Each subscription should have a priority and the subscribers are notified in that priority order. A Dynamic Host Configuration Protocol (DHCP) server configuration change is an example of where ordered notifications are of significant value to the integrity of the network as a whole. If both the IP address of the network device, as well as some additional changes to the DHCP server configuration are performed in a single configuration change, it is imperative that the DHCP configuration code receives the configuration change after the IP interface manager code has acted on the change and changed address on the network device. Otherwise, there is a potential conflict in the consistency and overall integrity of the configuration of network devices versus the information contained in any particular server or configuration source. Change Automation Managing operating expenses is critical to the success of both service providers and IT organizations responsible to enterprise networks. Network administration is one of the many functions that make up the planning, deployment, management, and maintenance of networks. Network administration costs are significant both as direct expenses and as a by-product of configuration errors that can cause service outages or severe performance impacts. Service disruptions carry a significant toll in terms of organizational productivity, negative PR, and potential customer churn. Until recently the alternatives for configuration management such as manual configuration, CLI scripting, and SNMP faced multiple limitations. For a complete discussion of these issues read Tail-f Systems whitepaper Next Generation Network Management with NETCONF. The NETCONF standard for automated configuration was finalized by the Internet Engineering Task Force (IETF) in December of 2006 as Requests for Comment (RFC) 4741-4744. This Remote Procedure Call (RPC)- based protocol uses XML data encoding for configuration data and protocol messages that are exchanged between a manager and an agent. NETCONF includes robust features to ensure that configuration changes are made consistently across all network devices. For example, a configuration change will only be committed if no errors occur. If errors
do occur the changes will be automatically rolled back. This is illustrated in Figure 5. NETCONF frees network management applications and data stores from the overhead and complexity of dealing with data consistency safeguards at the protocol level. Equipment vendors and network operators are adopting NETCONF to facilitate scalable deployments of networks without the risks of disruptive configuration errors. The need for NETCONF is also being driven by the inherent complexity of networks and the explosion in the number and variety of network devices. Today many services require network devices to be changed in one single transaction. Rapid Development Network operators are under pressure to quickly bring innovative services to their customers and users. Network equipment vendors therefore require enabling technologies to shorten development time and minimize sustaining engineering overhead. There are several time-to-market bottlenecks affecting network management software. Where northbound interfaces are not tightly coupled, additions and changes to applications and managed objects are multiplied several fold as each management interface must be addressed independently. The need for increased code integration and testing among management interfaces is another negative byproduct of a stovepipe architecture. Interfaces between software modules usually have higher than average bug rates. In addition, an uncoupled architecture undermines the benefits of using distributed development teams by creating interdependencies at every stage of the development process. Ideally, developers share the same data store of managed objects and that data store allows individual developers to add or delete items in the configuration without impacting other developers. But at the same time, there s a need to monitor and control items that are changed in the configuration. The time to develop new network management functionality can also be accelerated by integrating new functions with existing applications and data stores. Reusing legacy software modules has trade-offs, but existence of well-defined APIs provides a practical option to pursue this approach vs. reinventing everything from scratch. Conclusion Carrier-grade networks are larger and more complex than ever before, and growing more complex all the time. The software that monitors, configures, and controls these networks must be designed for high performance, continuous service, comprehensive security, low cost and reliable operations. This is best achieved with a unified transaction based architecture that ensures consistent execution, fine-grained control, and rapid application development. This unified architecture will also need to be flexible to absorb the constant change brought about in today s networking environments due to changes in mission scope, mergers and acquisitions, as well as the Figure 5
constant application and network performance improvements required to keep pace with competition. That flexibility will also need to extended to those areas where the applications, or network devices, interact or interface with other platform and tool architectures already present in the network environment. The ability to interact and play with other applications and devices in today s market is imperative. Executive management wants everything to work in a homogenous manner so that past, present, and future investments can be utilized together without significant effort or loss of capability or service delivery. Software applications and network devices will increasing need to route their AAA activities and operational events through a highly reliable, robust, intelligent, and centralized data repository. A centralized data repository allows for a consolidated and improved analytic functionality to be provided for enhanced true cause troubleshooting along with incorporating a robust auditing and reporting feature set to aid in current and future compliance adoption. A centralized data repository also provides the enterprise or telco staff with a complete and cohesive view into their network environment that allows everyone to see the same information at the same time. Not only across the organization, but also up and down the various organizational levels as well.
About Enterprise Management Associates, Inc. Enterprise Management Associates is an advisory and research firm providing market insight to solution providers and technology guidance to Fortune 1000 companies. The EMA team is composed of industry respected analysts who deliver strategic awareness about computing and communications infrastructure. Coupling this team of experts with an ever-expanding knowledge repository gives EMA clients an unparalleled advantage against their competition. The firm has published hundreds of articles and books on technology management topics and is frequently requested to share their observations at management forums worldwide. This report in whole or in part may not be duplicated, reproduced, stored in a retrieval system or retransmitted without prior written permission of Enterprise Management Associates, Inc. All opinions and estimates herein constitute our judgement as of this date and are subject to change without notice. Product names mentioned herein may be trademarks and/or registered trademarks of their respective companies. Corporate Headquarters: 5777 Central Avenue, Suite 105. Boulder, CO 80301. Phone: +1 303.543.9500. Fax: +1 303.543.7687. www.enterprisemanagement.com 1460.102207