Network and Facility Management: Needs, Challenges and Solutions

Network and Facility Management: Needs, Challenges and Solutions Graham Jones Harris Broadcast Communications gjones04@harris.com New tools are needed to enable broadcasters to efficiently manage the modern broadcasting station and network environment. Broadcasters are required to introduce increasingly sophisticated systems and equipment, at the same time making more efficient use of highly trained staff, and reducing system downtime. Many station groups and state networks are planning to consolidate control and monitoring of multiple stations in centralized locations. This paper looks at the requirements for sophisticated system-wide and integrated network management solutions for broadcast applications, identifies some of the challenges, and indicates some of the factors to consider in choosing and implementing a network management system. What is Network Management? Network management may be defined as the management of network resources in order to meet service-level objectives effectively and efficiently. A network management system (NMS) typically includes tools for gathering data on network elements, tools for data storage, analysis and prediction, tools for configuration and control of network elements, and tools for performance and system planning management. LAN / WAN Management NMS tools first evolved for the management of large local area computer networks and today are used for every type of network, from local area networks (LAN) to wide area networks (WAN) and the Internet. They enable network element configuration, for example Ethernet router set-up, monitoring of faults and alarms, measurement of network performance, and status monitoring such as computer disk space, or an application status. (simple network management protocol) is a major component of computer network management. Telecom Network Management Telecommunications networks incorporate a wide range of equipment and a variety of services for consumer and business customers. Network management is widely used for such systems and, using a variety of protocols, typically provides monitoring and configuration of all elements. These may include everything from telephone switches to microwave links, and satellite ground stations to fire alarms. Other management capabilities include network optimization, service activation, automatic testing, trouble ticketing and billing mediation. Rather than simply telling the operator that there is a problem, the NMS improves the service being provided. A full-featured telecom NMS will comprise a whole range of software modules, working together as an integrated Operational Support System (OSS). International Circuits Calling Center Central Office Environmentals HNM Fiber Optic Switching Cable Microwave Satellite Loop Test System Metallic Cable Wireless Local Loop Telecommunications Network Local Exchange Shelterized Local Exchange Network and Facility Management: Needs, Challenges and Solutions Page 1

Broadcast Networks Television, radio and data broadcast networks comprise major sub-subsystems for production, contribution, distribution and transmission in one form or another. Today they virtually all use computer LAN/WAN technology for some parts of their operations or support infrastructure, and they also have some of the same characteristics as telecom networks. At the most basic level this means they include facilities, equipment, communication links and delivery of services to an end-user customer. This suggests they may be candidates for network management. Some sectors, such as satellite television networks, the new satellite radio networks and the emerging data broadcast networks, are already using NMS, in some cases for management of program chain equipment as well as for their LANs and WANs. Others the national networks, program originators, state networks, station group owners and individual stations are just awakening to the possibilities. For convenience, throughout this paper the term network management or NMS is used, but the systems described are intended to cover the management of complete end to end systems including all aspects of the facility. This paper presents the perspective of the over-the-air broadcaster, but much of it is equally applicable to satellite and cable networks. Why do we Need Network Management? Changing Environment in Broadcasting The world of broadcasting is changing with the introduction of digital television and with competition from alternative means of distribution. All broadcast stations will soon have an additional output channel for DTV, possibly with HD (high definition) or multiple channels of SD (standard definition), they may already have more than one analog channel, for direct cable access or LMAs (local marketing agreements). Data broadcasting offers a whole new range of services and may bring additional complexities of operation. As this is going on, there is a consolidation in local station ownership, with more stations in the hands of larger groups. They are looking at ways to improve efficiency, including more centralized operations, both for programming operations and engineering support. Many public broadcasting state networks have the same ends in mind. As well as the drive to reduce costs, another factor in this trend is the difficulty of finding suitably qualified engineering staff. Changing Technology An increasing amount of equipment is software-based, utilizing computer technology, with GUIs (graphical user interface), serial ports and LAN connections and (simple network management protocol) built-in. Stations and networks rely on LANs and WANs, both as a means of program material distribution and as a support infrastructure. Traditional methods of fault diagnosis are frequently not applicable, and tracing the root cause of service failure may be a difficult task. Current Monitoring and Control Stations today typically have multiple independent monitoring and control systems, including those for: transmitter site, satellite receiving and up-link, ENG receive site, building power, HVAC, fire alarms, security, and for the important on-air operations automation system. Each of these will usually have a dedicated status and configuration terminal. In addition, many individual items of equipment now require a separate PC (personal computer) for monitoring and configuration. The result of this is an explosion in the number of PCs in the master control room and equipment area. What Do These Changes Mean for the Broadcaster? There are major implications for the broadcaster: More equipment to be monitored and controlled by the same or fewer staff More remote operational procedures More complex operational situations Network and Facility Management: Needs, Challenges and Solutions Page 2

More problems in diagnosing faults and taking corrective action Push for greater efficiency All of which mean more stress for the operators and engineers, and implies a need for network management tools. User Needs This section identifies some of the needs and desires of broadcasters that a network management system can help to fulfil, based on feedback received from senior level broadcast engineers over the past 12 months. Of course the specifics will vary somewhat for different users depending on individual circumstances. High Level At the highest level the main desires are to: Functionality Improve system availability Improve quality of service Decrease trouble response time Reduce maintenance costs Reduce staffing requirements and training levels Reduce operating expenses Increase operating efficiency The functionality of the NMS system should: Provide control, configuration, status, alarm management and fault diagnostics for the majority of the technical plant Allow multiple sites and multiple stations to be monitored and controlled from a single or multiple points Consolidate current monitoring and control activities into a single system, with more capability Provide intelligent support for systems operations and maintenance Allow for additional system management functions including: - Automation confidence monitoring and log reconciliation - Resource management - Bandwidth allocation / Scheduling / Traffic / PSIP - Data Broadcasting Needs in Detail The NMS should provide: Status and alarms in near real-time - Events viewer and log - Graphical displays with custom screens - Screens with geographic, physical, flow and logical views of the system and equipment Escalating alarm functions - Notification systems screen display, audible, page, telephone, e-mail Configuration and control (not expected to be frame accurate) Ability to work with new and existing equipment Ability to work with local and remote equipment Ability to work with existing element managers Ability to integrate with automation system Network and Facility Management: Needs, Challenges and Solutions Page 3

Data analysis and reporting - Trending - Statistics Smart, rules-based, fault analysis and executive action Automatic equipment configuration Auto-discovery Industry standard open system architecture Software hooks for third-party applications Users and Operators The NMS should provide secure access, with individual profiles, over internal networks, intranet, internet, and dialup lines. Users or user-groups should have personalized system views and control privileges dependent on their authority and operational needs. Operator positions may typically be required as follows: Within the plant Master Control Engineering Operations Director of Engineering Building Manager External to the plant Director of Engineering at home or on the road Group or Network NOC Manufacturer s technical support center Third party NOC Challenges In designing a network management system for broadcast use there are many challenges that must be addressed, the more important ones are set out below. One Size Does Not Fit All Flexibility The network management system should be flexible enough to cope with the many variations of size and complexity of broadcast stations and networks. Hardware and software options should allow a system to be sized appropriately for the intended purpose. The system architecture should be scaleable and expandable to allow the management system to grow as the user needs grow. Preferably it should allow for multi-level systems, with a hierarchical architecture, and with the option of either a centralized or distributed configuration. There should be provision for back-up capabilities with full redundancy when required. Similarly with the feature set: the software should be modular in design to allow the use to select only the functions that are required. Hierarchical Architecture for groups and networks For station groups, state networks, or national networks, there is a great advantage in having a hierarchical system architecture. This allows the use of stand-alone systems at local broadcast stations that provide all the benefits of network management to that station itself. Multiple stations can then be overseen by a supervisory system at a centralized location that can ultimately reduce or even eliminate the need for some staff at the local station perhaps at certain times of the day. Network and Facility Management: Needs, Challenges and Solutions Page 4

Equipment Interfaces There is a considerable challenge with introducing network management into the broadcast world due to the large number of different device types, manufacturers and models for equipment that are in use by broadcasters in the US and around the world. These have many different interfaces for control and monitoring, including: network interface Serial interfaces - Multiple electrical standards - RS-232, RS-422, RS-485 - Multiple protocols IEEE-488 GPI - Contact closures - TTL Analog The network management system should have tools for developing and configuring all device drivers, including those for proprietary protocols, in a straightforward, timely and cost efficient manner. Legacy Equipment The issue of interfacing to devices is significant in the case of legacy equipment, in this context defined as that without interface capability. While it would be desirable for stations to replace all old equipment as they transition into the DTV age, business reality means that legacy equipment will be around for many years to come. To be useful to a broadcaster it may be essential for the network manager to communicate with the following types of equipment that may not have capability: Master control switchers Video tape machines and carts Video servers Character generators and graphics systems Test and monitoring Studio-transmitter links Transmitters Satellite receive and up-link systems Studio production and post-production Weather radar system Facility power, generator, UPS, HVAC Facility fire, smoke detection, elevators, security The installed base of different types of such equipment means that many hundreds, if not thousands, of variations of interface drivers may be required, ranging from the most simple two-state device to extremely complex equipment. In addition to the device driver, for systems using a graphical interface there may be significant effort involved in producing custom screens related to the device, depending on the complexity of the device and the desired user interface for showing the equipment. Element Managers The challenge of interfacing to existing element managers is not as severe as to legacy equipment direct since there are far fewer types of element managers currently deployed. However, to achieve maximum benefits from a new network management system it may be necessary to integrate it with an existing element manager that was not designed to work with a supervisory management system. This may require additional systems engineering effort, use of such interfaces as do exist (for example a printer port), or possibly modifications to the third party system itself. Element managers that may need to be integrated include: Transmitter site remote control systems Network and Facility Management: Needs, Challenges and Solutions Page 5

Satellite receive and uplink control systems ENG microwave receiver site control systems Encoder control Routing switcher control Video server control Remote test equipment HVAC control systems Fire and Security systems Web Browser Element Managers Where an item of equipment has a web-browser interface available for remote control and monitoring, then it may be possible to use the network manager as a gateway to the browser screens. This eliminates the need to develop custom screens for that item of equipment. In this case, however, there is the challenge of ensuring that an alarm or status change in the equipment is brought to the notice of the network management system for display and action; and also of transferring any data that may be required to be logged from the device to the NMS. Equipment Any item of equipment with an interface can be readily integrated into a network management system. Depending on the complexity of the device and the desired user interface for showing the equipment, there still may be significant effort involved in producing custom screens related to the device. There is an increasing amount of equipment now available with, including: Computer-based systems (depending on operating system) Network elements - Routers, bridges, gateways, etc. Newer broadcast equipment - DTV encoders - PSIP generators - Data encapsulators - Data servers - Video servers - Format converters - Test equipment - Smart frames - Automation systems Any item of equipment with a provided proxy agent For those who are new to this topic, this section provides a very brief overview of, the simple network management protocol. Simple evidently because it provides a limited set of management commands and responses. It was developed in the 1980s to monitor and maintain large Ethernet computer networks. runs on top of TCP/IP and enables communication between an Manager and an Agent to determine and report status of network elements. is the Agent Device Devices Manager Storage Network and Facility Management: Needs, Challenges and Solutions Page 6

de-facto standard operations and maintenance protocol for the Internet, and is probably the key technology that enabled the growth of the Internet. It is rapidly becoming the broadcast industry standard interface for remote monitoring and configuration on new equipment. One of the advantages of is that it places a low stress on the network on which it is running. One disadvantage is that it is not suitable for commands and controls requiring frame-accurate time implementation. The Manager is part of the network management system; it provides the user interface and holds the database of device information. The Agent for the network element is the device s point of contact to the Manager and determines what information can be relayed between the device and the manager. If the agent is not resident in the device, then it is known as a Agent. The manager and the agent work with a Management Information Base (MIB). The MIB acts as the translator between and the device-specific data being managed, and documents the set of available hierarchical information. The Agent populates the MIB with attributes for the managed device, which are identified with unique Object IDs (OIDs), it also sets up a polling cycle (solicited information) for the Manager to query the agent and perform a MIB walk to get variables and build a network tree. The Agent maintains the current value of the variables and sends an unsolicited alert message when an attribute changes in the MIB. exchanges data through PDU (protocol data unit) messages for reading and setting data and monitoring network events, v.1 defines five basic commands (primitives) for status monitoring and control of the device as shown below. versions 2 and 3 have slightly different sets: Get GetNext Set Get Response Trap retrieves an object instance from the agent retrieves the next object instance from the agent sets values for object instances within an agent response from the agent after receiving a get command unsolicited alert message from the agent to the manager Integration with Automation One challenge that requires special mention is what happens when we add network management to a station with an existing automation system that is controlling legacy equipment. In this case we will assume that the automation system interface to the manager is provided through an proxy agent running on a proxy client computer (Link 3 in the figure below). Network and Facility Management: Needs, Challenges and Solutions Page 7

The proxy client gathers information on the status of the automation system, communicating with the automation system device server through the automation system LAN (Link 2), also communicating with the file server and client work stations. The control interface from the automation system device server to the majority of automationcontrolled legacy equipment (Link 1) is typically implemented through RS-422 serial ports. In order to independently gather detailed information on the status of the controlled equipment it would be preferable for the NMS to separately communicate with the station devices as shown in Link 4 on the drawing. The problem issue is that the majority of legacy devices have only one serial port. If that is in use by the automation system, then it is not available for the network manager. One solution to this dilemma requires enhancement of the automation interface to the device to allow a full set of status information to be gathered by the automation device server and placed in the automation system MIB. In that case, care must be taken that the critical frame-accurate performance of the automation system is not degraded by the extra workload placed upon it. An alternative solution would require modification of the legacy device to provide more than one simultaneously available remote control and monitoring port. This is not in general an attractive option, but may be possible in some cases, for example with a master control switcher where tally contacts may be available to provide remote indications of some device statuses. Solutions Design Factors to Consider It is outside the scope of this paper to discuss specific network management products that are available, but in making a choice of system and vendor there are various factors and design features that should be considered. Apart from the obvious issues of functionality discussed in the User Needs section above, other things to look for include: System Architecture For use in the broadcasting environment the following system architecture is desirable: Highly reliable distributed client-server architecture Provision for full redundancy with automatic database synchronization Multi-user, multi-tasking system Works in multi-protocol, multi-vendor environment Scaleable and expandable Software Design There are advantages in having a modular software architecture, where you only pay for what you need. Modules may include: Core network management Advanced data analysis and reporting Automatic rules-based executive action Performance management Trouble ticketing Provisioning and service activation APIs should be provided to allow integration with third party applications. The use of tools such as CORBA (Common Object Request Broker Architecture) provides efficient management of links between software modules and of system objects in a distributed environment. Network and Facility Management: Needs, Challenges and Solutions Page 8

Software Architecture Software design should comply with Open System Standards, using a layered architecture such as the one set out in TMN Telecommunications Management Network Model as shown here. Business Management Service Management Network Management Element Management Solutions Implementation Network Elements Typical Network Diagram LOCAL STATION TRANSMITTER SITE CORPORATE NETWORK OPERATIONS CENTER Test Equipment Remote Terminal Unit RS-232 RS-232 HBM Server Network Supervisor HBM Client Corporate User GPI and Analog Interfaces Other Serial Interface Devices Transmitter Devices Terminal Server TCP/IP Ethernet Local Area Network TCP/IP Ethernet Local Area Network Network Router Modem STL Corporate WAN OTHER LOCAL STATIONS Public Telephone Network LOCAL STATION STUDIO LOCATION Subsystem Element Manager HBM Server Master Control HBM Client Chief Engineer HBM Client News Operations STL Network Router Modem Network Router TCP/IP Ethernet Local Area Network Terminal Server Remote Terminal Unit RS-232 RS-232 RS-232 Other Devices GPI and Analog Interfaces Other Serial Interface Devices Format Converter ARX-H200 Receiver FlexiCoder PSIPplus DataPlus Data Server Automation System Video Server Routing Switcher Smart Frame Test Equipment TYPICAL HBM SYSTEM DIAGRAM Examples of Graphics Screens Geographic Geographic Region Equipment Rack Card Frame Event Viewer Network and Facility Management: Needs, Challenges and Solutions Page 9

Network Operations Center Conclusion Network management systems can help broadcasters provide more complex services efficiently, with fewer staff and at lower cost. In selecting a network management system, careful analysis of user needs and customization of device interfaces and graphics screens is required to achieve maximum benefits. Suitable management products are now becoming available and will most likely become a commonly used tool in the broadcast station, group and network. Network and Facility Management: Needs, Challenges and Solutions Page 10