WHITE PAPER Improving Operational Readiness for P25 Systems through Advanced Network Monitoring
With the introduction of trunked radios systems over 20 years ago, public safety communications technology took a giant step into the future. The leap from the more simple analog conventional systems to trunked radio systems added capacity, enabled separate talk groups and enhanced private communications. Today public safety agencies are making yet another leap, migrating away from proprietary-based communications systems to open standards-based digital networks. This digital migration is in direct response to industry challenges, such as: Increasing radio spectrum congestion Demand for high performance data transmission Interoperability between systems Access to increased functionality Increased need for secure communications Improved digital voice quality over expanded coverage areas Cost effective, gradual migration to digital Digital networks enable increased functionality, as well as expanded coverage, bandwidth and flexibility. However, with advanced technology comes increased complexity. Not only are the features and capabilities of the new digital technology dramatically different, the platforms that support these features have increased in capability as well. These differences directly impact how agencies must manage their networks to enhance operational readiness. With rapid response even more critical than ever and system complexity on the rise, a reactive approach to network monitoring is no longer enough. Operational readiness depends on the ability for a technician to take corrective action before it can impact the network and before first responders even know there is a problem. This white paper will help you identify what changes should be made to put in place an effective, efficient network monitoring strategy that will ensure the highest level of operational readiness which first responders expect and require. The impact of P25 From dramatic singular events such as the 9-11 terrorist attacks to the everyday need for first responders from multiple districts to communicate and cooperate when responding to an incident, interoperability and collaboration are critical. An interoperable network enables seamless communications for vital public safety and federal agencies, as well as utilities, allowing those organizations to seamlessly collaborate for improved communications during every day operations and disaster response. APCO Project 25 (P25) was established to address the need for common digital public safety radio communications standards for first responders and homeland security/emergency response professionals. A basic requirement for new digital radio equipment is backward compatibility with standard analog FM radios to support an orderly migration into mixed analog and digital systems, enabling users to gradually trade out radios and infrastructure equipment. Selecting products and systems that comply with P25 recommended standards reassure agencies that the investment they have made in the latest technology pave the way for a clear migration path for the future. IP-based communications networks designed to meet APCO P25 standards of interoperability are based on an open architecture that utilizes standard operating systems (such as UNIX and Microsoft), as well as standard transport protocol (such as IP/ TCP), to allow adding, upgrading, and intermixing components from multiple vendors. With an open architecture system, basic network facilities and Services will permit all users of the network to interconnect and interface with each other. Open vs. closed architecture In the past, the lack of IT interoperability presented a hurdle in expanding and modernizing networks. Many communication networks only worked with proprietary system designs, making enhancements both financially and technically problematic. In support of the new APCO 25 standards, manufacturers are moving to more open architectures to accommodate interoperability and other critical new features, such as integrated voice and data, text messaging, over the air reprogramming, etc., for improved emergency response and coordinated communications. Today s open architecture utilizes standard operating systems, such as UNIX and Microsoft, and standard transport protocol, such as IP/TCP. Network monitoring best practices for P25 systems Network monitoring has become a tool for managers of communications systems to maintain appropriate levels of system availability. The tool could range from a Simple Network Management Protocol (SNMP) event monitor that reports problems with hardware, to a heat sensor that monitors HVAC equipment, to intrusion alarms that report an open door. When network monitoring tools detect an event, the technician is paged and sent to the site to address the issue. 2 white paper
Investment required self-manage network monitoring Staffing: 24x7x365 Training: Ongoing training to stay up to date on latest technology Tools: Diagnostics and restoration hardware and software Knowledge Database: History of known issues for rapid resolution Network Operating Center: TL9000 and Federal NISPOM Chapter 5 compliant The process is highly reactive and with the much different dynamic required for network monitoring today, that level of urgency is no longer enough. IP-capable, interoperable, P25 networks are held to a much higher standard to ensure network availability and network operators require a unique business plan that maintains tight control and enables rapid response. With increased systems complexity, software vs. hardware-based communications system and an IPenabled open architecture all increase the potential for network events, which can result in tens of thousands of alerts. For this reason, system managers should rethink whether the organization s existing network monitoring process provides the appropriate level of alarm filtering and control monitoring. Motorola recommends the following Advanced Network Monitoring business model for reliable operational readiness. The business model encompasses an endto-end strategy consisting of: Network monitoring tools to capture and report network events Filtering software with rules that provide efficient and effective network visibility Staffing and resources that are trained on the new P25 technologies (both RF and IP) Standard, repeatable, documented processes Reporting that enables informed technology decisions Network monitoring tools An IP open architecture network substantially increases the number of elements monitored and number of potential alarm events reported. Most communications networks are designed with a standard network manager that collects and displays raw network data, such as event traps and hardware status indicators. In order to be useful, however, the data must be managed so that potential problems can be identified and addressed. Network monitoring tools monitor, analyze, and display the status of zone controllers, servers and consoles for a higher level view of network performance. Numerous network monitoring tools are on the market today, but regardless of which you choose, a plan should be developed to incorporate multiple tools with clearly defined network alarm filtering rules that turns the raw data into information that will increase productivity, reduce cost, and enhance system availability and operational readiness. Alarm filtering The most critical network monitoring element is the ability to filter the thousands of events or alarms that are generated from the raw data reported by the system. For example, alarms can be caused by simple power fluctuations that do not need immediate response. However, a single power fluctuation can generate more than 80 different event alarms throughout the system with no easy way to identify the source of those alarms and whether they are critical or non-critical alarms. Alarms can also be generated by link failures which sometimes cause intermittent connectivity, triggering an alarm each time the link goes down. Appropriate filtering rules can instruct the network monitor tool to generate an alarm only if the link goes down an established number of times within a specified period, which can be a potential sign of trouble. Filtering establishes rules that separate and report only those alarms that indicate a potentially serious issue. For example, Motorola s System Support Center (SSC) monitors numerous large statewide networks for its customers, as well as smaller private systems. More than 1,000 alarms per month can be generated by a closed system and up to 10,000 alarms per month can be generated by a digital ASTRO network. Filtering those alarms to report only what the customer has identified as a priority sometimes reduces the number of alarms reported by 80 to 90 percent. If the appropriate filtering is not applied, a technician looking at the unfiltered data would see all of these events and investigate each one to see which need immediate attention, which can wait, and which do not need to be addressed at all. Alarm filtering and event correlation enables you to set rules for what you want to see and do not want to see on each system you monitor, and then can assign severity levels for the appropriate response. white paper
Resources Once the monitoring tool has been installed and the filtering rules have been defined and applied, the alarms that are reported are important and require someone to take action. Staffing and coverage: On critical communications systems, where lives are often at stake, network monitoring coverage must be available on a 24x7 basis to ensure around the clock eyes on the system. Depending on the size of the network, monitoring on a 24x7 basis requires a minimum of four to ten individuals for full time coverage. These individuals should be trained, certified, experienced technologists dedicated to monitoring, diagnosing and quickly addressing issues. Training: Just as technicians had to make the technology leap from conventional to trunked technology, now they have to make the leap from analog trunking to digital technology and how that impacts the network. They not only need basic RF training, they now also need to understand the new IP dependencies i.e. what is connected to the IP network and how it impacts the communications system. In addition, training may be needed on the network monitoring tool and the processes that are required for immediate system restoral. Knowledge database: Taking advantage of known alarms and issues and how to react to each significantly increases the technician s productivity and reduces the time it takes to address issues. Building a knowledge database that can capture thousands of technical solutions enables faster identification, diagnosis, and resolution of system problems when issues re-occur. Standardized, proactive processes Network monitoring can only be successful if there is a standard process in place to characterize, prioritize, and manage events through pre-defined procedures. At Motorola s SSC, ISO9000 certified processes are followed, as shown: Step 1: Once the advanced filtering software determines that an event has occurred that requires attention, the certified technologists at the SSC connect remotely to the network and perform diagnostic testing. This additional step in the characterization process determines whether the event requires a local technician dispatched to the site. Step 2: The technologist interrogates the knowledge database to help identify the problem more quickly and accurately. Step 3: If a remote fix is not possible, the technologist opens a Services case file that identifies the nature of the problem and the planned resolution, and then sends notification to the field technical resources. Step 4: Dispatch Operations checks the Customer Support Plan to identify the local field technician and dispatches the technician to the site with the information required to replace the faulty component or resolve the problem. Dispatch updates the case file. Step 5: The field technician arrives and reports site arrival. Step 6: Once the component is replaced, the SSC technologist logs into the network to program the new component and check that full network availability has been restored. Step 7: The field technician updates the case file with the appropriate information and makes arrangements to send the faulty component to the Motorola depot for repair. This update automatically advises the depot of the incoming part to schedule the repair. Step 8: Technicians report the issue has been resolved and the customer verifies that the system is fully operational before the case is formally closed. Step 9: When the repair is complete, the component is returned to the field to be used as spare equipment. The field technician updates the case file and the system automatically informs the customer of the replacement. Network Monitoring Support Path 4 white paper
Reporting Availability reports generated from network activity and performance data make it easier to document the effect of both recurring and one-time communications system problems. These reports can be used for internal briefings and to help make critical decisions on migration strategies or technology refresh timelines. Network availability and performance reports should provide information on outages, down time, types of problems, and whether there might be an underlying problem with the system. This information provides the ability to take a proactive stance to prevent future problems, increase system reliability, and reduce cost. The bottom line A well monitored network can more easily sustain network operational readiness levels of 99.999 percent. Leveraging Motorola s Network Monitoring best practices will help you support your network operational readiness strategy using a continuous improvement model of early detection, corrective action plans and prevention. As a result, you will maximize your investment and total cost of ownership and provide your first responders with consistent availability, high quality communications, and reliable network operational readiness. 5 white paper
Motorola, Inc. www.motorola.com The information presented herein is to the best of our knowledge true and accurate. No warranty or guarantee expressed or implied is made regarding the capacity, performance or suitability of any product. MOTOROLA and the Stylized M Logo are registered in the U.S. Patent and Trademark Office. All other product or service names are the property of their respective owners. Motorola, Inc. 2007