Chapter 02. Diagnostics & Troubleshooting

Chapter 02 Diagnostics & Troubleshooting

Chapter 02 Module 01 Using Diagnostic Tools

Module Introduction This module provides information about Avaya provided diagnostic tools. After completing this module you will be able to: Summarize the use of the diagnostic tools provided by Avaya for IP Office Identify where to acquire the Avaya provided diagnostic tools Explain the capabilities of Avaya provided diagnostic tools Capture traces using Avaya provided diagnostic tools Avaya provides three diagnostic tools for use specifically with IP Office: Debug View, System Status Application and System Monitor.

Avaya Diagnostic Tools Depending on the nature of an issue, there are several tools that may be useful for troubleshooting problems. For example, Ping, Trace Route and Netstat are useful for troubleshooting network connectivity issues. Additionally, Avaya provides three diagnostic tools for use specifically with IP Office: Debug View used with Voicemail Pro System Status Application - a diagnostic tool for system managers and administrators, in order to monitor and check the status of IP Office systems System Monitor used by system managers and administrators in order to capture system traces for submission to developers and support staff

Avaya Diagnostic Tools installation Installation The System Monitor and System Status Application are provided with the IP Office Admin Suite.

IP Office Debug View The voicemail console mode (Interact with Desktop) is no longer used and Avaya recommends using the Debug Viewer to view voicemail server activity. The DbgView application can be found in the attachments of this course DVD or can be downloaded on: http://marketingtools.avaya.com/knowledgebase/businesspartner/ipoffice/ general/debug/index.htm It can be used to view server activity real time or logged to file Reporting detail level can be configured Used for tracing various functions including: Voicemail Connections Outcalling Message Processing Debug View must be run on the VoiceMail Pro server PC

Debug View Logging Options By default events are shown in the DbgView window. The output can also be logged to a specified file. Select File Log Preferences from the DbgView menu Specify which Log Mode to use Enter the name of the file to store the information The log file is best viewed with WordPad word wrap disabled

Debug View Filtering The level of filtering can be set from 0 None to 9 High Trace to control the level of detail. Specific items can be selected or deselected to target specific output message types (e.g. TTS Support or MAPI Support)

Starting System Status Application (SSA) SSA can be launched independently or via Manager. For IP500v2 systems, it can also be launched by browsing to the IP Office IP address without needing to install the application. Click the Windows Start icon and select Programs IP Office System Status From within the IP Office Manager application, select File Advanced System Status Using a browser, enter the IP address of the IP500v2 system, for example 192.168.42.1 The Offline tab of the SSA Login provides the option to open previously saved files such as a Snapshot or Continuous log.

Services Base TCP Port The port(s) on which the IP Office listens for requests to access services on the LAN1 IP address. Each service uses a port offset from the base port value. If this value is changed from its default, the Manager application must be set to the same Base TCP Port through its Services Base TCP Port setting (File Preferences). The port is set on the System Details screen within the IP Office Security Settings (File Advanced Security Settings). Service Method Port Used Default IP Office Configuration Unsecured Base TCP Port 50804 3.2+ Secured Base TCP Port plus 1 50805 4.1+ System Status Interface Unsecured Base TCP Port plus 4 50808 4.0+ Security Administration Unsecured Base TCP Port plus 8 50812 3.2+ Secured Base TCP Port plus 9 50813 4.1+ Enhanced TSPI Unsecured Base TCP Port plus 10 50814 5.0+

Assigning Security Rights The default IP Office service users Administrator, Manager and Operator have SSA access rights. There can be up to two SSA clients connected to an IP Office unit at one time. The CCR application uses the same interface to the IP Office control unit as SSA.

System Status Application (SSA) SSA provides information on the following: Alarms SSA displays all alarms which are recorded within IP Office for each device in error. The number of occurrences and the date and time of the last occurrence are recorded. Call Details Information on incoming and outgoing calls; including call length, call reference and routing information. Extensions SSA details all extensions (including device type and port location) on the IP Office system. Information on the current status of a device is also displayed. Trunks IP Office trunks and connections (VoIP, analog and digital) and their current status are displayed.

System Status Application (Continued) Resources IP Office includes central resources that are utilized to perform various functions. Diagnosing these resources is often critical to the successful operation of the system. Those resources include: Voicemail Channels Conference Channels Data Channels VCM Channels Modem Channels Voicemail SSA provides real time information on the number of licenses available and details of the voicemail mailboxes including the number of messages and the current status of basic voicemail settings for the associated mailbox user or hunt group. IP Networking SSA displays the details of the IP Office Routing Table, including static and RIP routes as well as information on configured IPSec or L2TP VPN tunnels. A Ping can be executed from the IP Routes screen choosing LAN1, LAN2 (WAN) or RemoteManager (DialUP) as the source interface to aid in troubleshooting connectivity issues.

System Status Application (Continued) System The System screen within SSA provides information on the type and state of hardware installed in the IP Office to include: Hardware Summary Memory Cards Control Unit Ports Extension Ports Expansion Modules H.323 Extensions VoIP Trunks

SSA Snapshot Feature Include switch configuration: embeds a copy of the IP Office configuration file when Snapshot only or Continuous log is selected. Snapshot only captures the IP Office system status to a file with the.ssh extension which can be viewed offline at a later time. Continuous log: records all IP Office activity to a file or series of files with the.slo extension until logging is stopped. If logging is enabled for an extended period, new files are created based on the settings Daily at midnight or After the log reaches a configurable size.

Reviewing Files Offline Using the Offline tab of the SSA Login screen provides the option to review previously saved files with the.ssh or.slo file extensions. When using SSA to open a Continuous log file, the Replay options are available to control the log file playback. The Manager application can be used to review the IP Office configuration embedded into the SSA log files with either the.ssh or.slo file exensions.

SSA Tracing Options The System Status Application provides the ability to generate traces for calls, extensions and lines / trunks: Calls: From the Call Details screen the trace will show the changes of state for that call and events relating to both ends of the call. Extensions: From the screen for a specific extension, the trace will show events relating to the extension including button presses. Trace information for a call from a specific extension will provide the same detail as the Call Details screen for the same call. Trunks: The Trunk screen will show events relating to the selected channels, such as protocol messages, plus traces of all calls associated with these channels, for as long as the calls are connected.

SSA Tracing Options (Continued) Trace information is presented at the bottom of the screen. The Pause button and scroll bar enable viewing the information while the application continues to record new trace events. The Resume button displays all the events recorded when a trace is paused, as well as further new events as they occur. When a trace is displayed, the option to Print and/or Save As are available. A trace can be saved to file either as a.txt or.csv file. If the trace is paused, only the information currently displayed will be saved and/or printed. To diagnose problems with a call, it is generally best to trace the source of the call; e.g. trace the trunk for an incoming call or the extension for an outgoing call.

Using System Monitor The Password used for System Monitor is set in the IP Office system Security Settings. If the IP Office doesn t have a Monitor Password set, the System Password is used. By default, the System Password is password (case sensitive).

Starting System Monitor System Monitor can be run on a PC on the same local subnet as the IP Office or from a PC on a remote network with established bi-directional routing. Click on Start Programs IP Office System Monitor Enter the IP address of the IP Office control unit Enter the Password and click OK

System Monitor Interface The IP Office System Monitor application is used to assist in the diagnosis of problems. Through configuration of its settings it is able to display information on a specific area of an IP Office's operation. It can capture that information to log files for later analysis.

System Monitor Filters / Trace Options The System Monitor can be set to display and log a wide range of traffic types on the IP Office. In order to avoid missing significant events due to the monitor operation it will be helpful to use the filter options. The Filters Trace Options menu allows for customization of the events to trace. All IP Office installers and maintainers need to understand how to run Monitor when necessary as Avaya may request copies of Monitor traces to resolve support issues.

All Settings Options On the bottom of the All Settings screen there are buttons to aid in the selection of options. Additionally, if a combination of settings is configured to troubleshoot a specific issue, the settings can be exported using the Save File button for later use. On occasion Avaya support staff may provide specific settings which can be imported using the Load File button.

System Monitor File Logging Logging is enabled for the System Monitor application from the menu File Log Preferences. Select the options for logging and click OK.

Log File Naming Convention The naming convention is as follows: NAME DATE TIME (N) In the above example, the NAME of the trace is CallTrace1 The DATE format is: YYYYMMDD where Y represents the Year, M represents the Month and D represents the Day of the Month The TIME format is: HHMMSS where H represents the Hour, M represents the Minute and S represents the Seconds The (N) represents the Number of times the log file of the same name has rolled over either manually or based on the Log Mode settings enabled.

Event Notification The IP Office supports a number of methods by which events occurring on the system can be reported. These are in addition to the real-time and historical reports available through the IP Office System Status Application (SSA). SNMP Reporting: The IP Office SNMP client allows the IP Office system to respond to SNMP polling and to send alarm information to SNMP servers. SMTP Email Reporting: IP Office can send alarms using an SMTP email server. Syslog Reporting: IP Office can also send alarms and audit trail events to a Syslog server (RFC 3164) without needing to configure an SNMP server.

SNMP Reporting Support for SNMP is enabled by entering the required information on the System Events Configuration tab. Community (Read-only) and SNMP Port Enabling SNMP is not required if using SMTP or Syslog. The QOS Parameters are used if Enable RTCP Monitor on Port 5005 is selected (System LAN1 VoIP). If the thresholds are exceeded an alarm is sent to SSA and to a destination configured on the Alarms tab.

SNMP Settings Enter the IP address of the SNMP server (Trap Receiver) Verify the Port setting (162 is the default transmit port) Enter the Community [string] configured in the SNMP server

SMTP Settings (Continued) In order to send alarms to email, the IP Office Manager System SMTP tab must be configured with the senders information. Depending on the authentication requirements of the SMTP server, this may need to be a valid email address hosted by that server. Otherwise the SMTP email server may need to be configured to support SMTP relay.

Syslog Reporting Enter the IP address of the Syslog server Verify the port setting (514 is the default port used by Syslog)

02-01 Using Diagnostic Tools Explore System Status Application and System Monitor Task 1: 1. Start the System Monitor Application 2. Place test calls and observe the real time output of each application Answer call Answer call and transfer to another extension Allow call to go to voicemail Call into an auto attendant

02-01 Using Diagnostic Tools (Continued) Task 2: 1. Start the System Status Application 2. Place test calls and observe the real time output of each application Answer call Answer call and transfer to another extension Allow call to go to voicemail Call into an auto attendant

02-01 Using Diagnostic Tools (Continued) Task 3: 1. Enable file logging in System Monitor and review the output of the log files 2. Place test calls and observe the real time output of each application Answer call Answer call and transfer to another extension Allow call to go to voicemail Call into an auto attendant

02-01 Using Diagnostic Tools (Continued) Task 4: 1. Enable continuous logging in System Status Application and review the output of the log files 2. Place test calls and observe the real time output of each application Answer call Answer call and transfer to another extension Allow call to go to voicemail Call into an auto attendant

Checkpoint/Questions?

Summary This module provided information about Avaya provided diagnostic tools. After completing this module you should be able to: Summarize the use of the diagnostic tools provided by Avaya for IP Office Identify where to acquire the Avaya provided diagnostic tools Explain the capabilities of Avaya provided diagnostic tools Capture traces using Avaya provided diagnostic tools Avaya provides three diagnostic tools for use specifically with IP Office: Debug View, System Status Application and System Monitor.

Click the forward arrow to continue. Summary (Continued) Links to the documents used in this Learning Module: IP Office System Status Application IP Office System Monitor

Chapter 02 Diagnostics & Troubleshooting

Chapter 02 Module 02 Best Practices

Module Introduction This module provides information about using a structured approach for troubleshooting and collecting information for escalating IP Office issues. After completing this module you will be able to: Identify data collection requirements for escalating IP Office issues Summarize the use of the OSI Model for troubleshooting Describe the purpose of a network assessment Using a structured approach when troubleshooting IP Office issues will help obtain the information required for resolution more quickly.

Best Practices Overview Using a structured approach when troubleshooting IP Office issues will help obtain the information required for resolution more quickly. Identify and quantify symptoms of the issue: Who: individual, group, everyone What: hardware, application, feature, performance When: time, day, frequency, repeatable, call volume Where: local, remote How: sequence of events, inbound / outbound calls, number dialed

Additional Discovery Determine if there is additional documentation to assist in resolving the issue: Network assessment / baseline Network diagrams / schematics / IP address schemes Wiring diagrams IP Office Audit trail System backup procedures IP Office backup configurations VoiceMail Pro Other application servers

IP Office Knowledgebase For troubleshooting IP Office issues Avaya provides a number of online resources that can be used for assistance. The IP Office Knowledgebase is a central repository for information IP Office including: - Technical Tips, Bulletins and Caveats (known issues) - Product Documentation - Documentation corrections - Application Notes for integrating Third Party products

Avaya Support Web Site In addition to links to IP Office Documentation and software downloads, the Avaya Support web site hosts a searchable knowledgebase called the InSite Knowledgebase. InSite contains articles, job aides and other information useful for troubleshooting IP office issues. Registered users can access on-line services and additional content. (http://support.avaya.com)

Using OSI Model for Troubleshooting The OSI Model (Open System Interconnection) provides a common framework for dividing communications system into logical segments. Methods of using the OSI model: Bottom up troubleshooting by going from the physical layer (layer 1) up to the application layer (layer 7) Top down troubleshooting by going from the application layer (layer 7) down to the physical layer (layer 1) Divide and Conquer in this method, you start with whatever layer you feel is most likely the cause of the problem, then move in whatever direction you feel is the more likely cause of the issue (either up or down the OSI model)

OSI Fundamentals The segment division can be used to isolate problems. For example, testing each layer, starting from the bottom up makes it possible to isolate problems to a specific layer. Each layer of the OSI Model relies on the layers below it in order to operate. OSI Layer Layer Number Verify Represented by Application 7 IP Softphone, WireShark Presentation 6 CODECS (G.729 / G.711) Session 5 H.323 SIP Applications Transport 4 Access Control TCP / UDP RTP / RTCP Ports Network 3 ICMP Trace Route IP Address, Subnet Mask, Default Gateway Data Link 2 NIC Physical 1 Link Lights Cat 5, Fiber

Network Assessment A network assessment should include a determination of the following: A network audit to review existing equipment and evaluate its capabilities, including its ability to meet both current and planned voice and data needs. A determination of network objectives, including the dominant traffic type, choice of technologies and setting voice quality objectives. The assessment should leave you confident that the network will have the capacity for the foreseen data and voice traffic, and can support SIP, H323, DHCP, TFTP and jitter buffers in VoIP applications.

Network Assessment Targets Latency: Less than 180ms for good quality. Less than 80ms for toll quality. Note that the different audio codecs used each impose a fixed delay caused by the codec conversion as follows: G711: 20ms, G723a: 80ms, G729: 40ms Packet Loss: Less than 3% for good quality. Less than 1% for toll quality. Jitter: Less than 20ms Round Trip Delay: Less than 160ms is high quality. Less than 350ms is good quality. Duration: Monitor statistics once every minute for a full week.

Quality of Service When transporting voice over low speed links it is possible for normal data packets (1500 byte packets) to prevent or delay voice packets (typically 67 or 31 bytes) from getting across the link. It is important that all routers and switches in a network to have some form of Quality of Service mechanism. IP Office supports the DiffServ (RFC2474) QoS mechanism. This uses a Type of Service (ToS) field in the IP packet header. The IP Office uses this field to prioritize voice and voice signaling packets on its WAN interfaces. Note The system does not perform QoS for its Ethernet ports including the WAN Ethernet port on the Small Office Edition.

Escalating IP Office Issues It is essential that all relevant information is documented and collected for escalation. This will save valuable time as the problem is escalated to the appropriate level of support. Items required for escalation of application issues vary based on the application and the specific issue. However, some items are required for ALL escalations, including: A detailed description of the customer issues including what services are being affected, which user actions, user numbers and user types. A copy of the IP Office(s) configuration file(s) The version / release of IP Office software log files generated by the IP Office System Monitor application Notes regarding the troubleshooting methods used and results

Escalating IP Office Application Issues Items required for ALL application escalations, include: Computer information: CPU Type / speed Amount of installed memory Operating System & Service Pack level Hard Drive capacity and space available Other applications on the computer IP Office Applications (one-x Portal, ContactStore, etc.) Anti-virus, anti-spam, firewall

Escalating Voicemail Issues In addition to the IP Office and server PC information, items required for escalating voicemail issues include: Voicemail type and version: Embedded VoiceMail Pro Copy of the VoiceMail Pro database (.mdb file) Debug View trace(s) Note While not generally required for escalation purposes, System Status Application Snapshots and Continuous Logs can be very useful in isolating IP Office issues.

Escalating VoIP / IPO Network Issues Items required for escalating network related issues include: Network assessment / network diagram(s) Detailed network connectivity information including connection types and bandwidth IP address scheme of main switches and routers in the network QoS Settings VLAN information Make / model and software / firmware versions of IP phones Number of IP phones registered to the IP Office Verify the VCM hardware / licensing A trace using packet capture software (see Technical Tip 195)

Checkpoint/Questions?

Summary This module provided information about using a structured approach for troubleshooting and collecting information for escalating IP Office issues. After completing this module you should be able to: Identify data collection requirements for escalating IP Office issues Summarize the use of the OSI Model for troubleshooting Describe the purpose of a network assessment Using a structured approach when troubleshooting IP Office issues will help obtain the information required for resolution more quickly.

Click the forward arrow to continue. Summary (Continued) Links to the documents used in this Learning Module: Global Technical Tip 195 IP Office Caveats (internet connection is required) System H.323 Gatekeeper (http://marketingtools.avaya.com/knowledgebase/businesspartner/ipoffice/mergedprojects/manager/index.ht m?gatekeeper.htm)

Chapter 02 Diagnostics & Troubleshooting

Chapter 02 Module 03 Global Assistance Calls

Module Introduction This module provides information about common causes of degradation in the quality of VoIP calls. After completing this module you will be able to: Summarize the supported voice compression standards supported by IP Office Identify when voice compression channels are required Describe IP Office settings that should be configured to improve IP voice quality

Overview It is likely that any fault on a network, regardless of the cause, will be reported as a degradation in the quality of VoIP calls. This module provides some items to consider: End-to-End Matching Standards: VoIP depends upon the support and selection of the same voice compression, header compression and QoS standards throughout all stages of the calls routing. The start and end points must be using the same compression methods. All intermediate points must support DiffServ QoS. Avoid Hubs: Hubs introduce echo and congestion points. If the customer network requires LAN connections beyond the capacity of the IP Office Unit itself, Ethernet switches should be used. Ethernet switches are recommended as they allow traffic prioritization to be implemented for VoIP devices and for other device such as the Voicemail Server PC.

Voice Compression Calls to and from IP devices often require conversion to the audio codec format being used by the IP device. For IP Office systems this conversion is done by voice compression channels on a Voice Compression Module (VCM). IP Office VCMs support the common IP audio codecs: G711 G723 G729a For IP400 control units channels can be added by fitting IP400 Voice Compression Modules (VCMs). For the IP500 control units, channels can be added using IP500 VCM cards and licenses or IP400 Voice Compression Modules.

When VCM Channels are Used In order to minimize quality issues with IP audio due to VCM utilization it is important to understand when VCM channels are used. IP device to non-ip device Also includes MOH, voicemail access, conference resources, etc. IP device to IP device Short code confirmation, ARS camp on and account code entry tones require a voice compression channel If the IP devices use differing audio codecs Note T3 IP devices must be configured to 20ms packet size for the above conditions to apply. If left configured for 10ms packet size, a voice compression channel is needed for all tones and for non-direct media calls.

When VCM Channels are Used (Continued) SIP Line Call to/from Non-IP Devices Incoming SIP Line Call to IP Device (Voice compression channel reserved until call connected.) T38 Fax Calls IP Office 5.0+ supports T38 fax on SIP trunks and SIP extensions. Each T38 fax call uses a VCM channel. Within a Small Community Network, an T38 fax call can be converted to a call across an H323 SCN lines using the IP Office Fax Transport Support protocol. This conversion uses 2 VCM channels. In order use T38 Fax connection, the Equipment Classification of an analog extension connected to a fax machine can be set Fax Machine. Additionally, a new short code feature Dial Fax is available.

VCM Configuration Options Echo is typically generated by impedance mismatches when a signal is converted from one circuit type to another, most notably from analog to IP. To resolve this issue, an estimated echo signal can be created from one output and then subtracted from the input to hopefully remove any echo of the output. The System VCM form in the Manager application allows adjustment of the operation of VCMs installed in the IP Office control unit.

Automatic Codec Preference The setting in Manager on the System Telephony tab is used by all IP lines and extensions where the Compression Mode has been set to Automatic Select. The codec selected here will be the first codec the line or extension uses for codec negotiation. Note that the G.711 codecs are treated as a pair, so if one is selected as the first preference, the other will be second in the list. Setting Preference Order G.729 G729(a) 8k CS-ACELP / G711 U-Law 64K / G711 A-Law 64K / G723.1 6K3 MP-MLQ G723 G723.1 6K3 MP-MLQ / G729(a) 8K CS-ACELP / G711 U-Law 64K / G711 A-Law 64K G.711 U-Law G711 U- Law 64K / G711 A-Law 64K / G729(a) 8K CS-ACELP / G723.1 6K3 MP-MLQ G711 A-Law G711 U- Law 64K / G711 U-Law 64K / G729(a) 8K CS-ACELP / G723.1 6K3 MP-MLQ

VoIP Settings for IP Endpoints / IP Lines There are common VoIP settings on all IP extensions (H.323 / SIP) and IP Lines (H.323 / SIP / SES) for: Compression Mode VoIP Silence Suppression Allow Direct Media Path

Compression Mode This field defines the compression mode (codec) or modes offered during call setup. Automatic Select For pre-ip Office 5.0 systems, Automatic Select uses the codecs in the order of preference G729a, G711 ALAW, G711 ULAW and G.723.1. If required, a specific codec can be selected. If negotiation for a specific codec fails during call setup, the IP Office will fallback to using Automatic Select.

VoIP Silence Suppression The VoIP Silence Suppression option will detect periods of silence on a call and will not send any data during those silent periods. This feature is not used on IP lines using G.711 between IP400 and IP500 systems. A typical voice conversation consists of 40 to 50 percent silence. Since there is not any voice going through the network for 40 percent of a voice call, some bandwidth can be saved by deploying Silence Suppression. There is a trade-off. There is a small amount of time before the codecs detect speech activity following a period of silence which may result in the clipping of received audio.

Allow Direct Media Path This settings controls whether H323 calls must be routed via the H323 gatekeeper (the IP Office) or can be routed alternately if possible within the network structure. If enabled, H323 calls can take routes other than through the IP Office and removes the need for a voice compression channel. Both ends of the calls must support Direct Media. If Disabled (not selected) On pre-4.0 calls between IP devices using the same audio codec require a voice compression channel. On IP Office 4.0 and higher with RTP relay support, calls between devices using the same audio codec do not require a voice compression channel.

Line Analog Options IP Office has configuration options to address echo generated by impedance mismatches on the Line Analogue Options form. The Impedance field is only available for certain system locales; Brazil, Korea and the United States. The Quiet Line setting may be required to compensate for signal loss on long lines. (North American locales only) The Automatic Balance Impedance Match option can be used to test the impedance of a line and display the best match resulting from the test. (North American locales only)

Checkpoint/Questions?

Summary This module provided information about common causes of degradation in the quality of VoIP calls. After completing this module you should be able to: Summarize the supported voice compression standards supported by IP Office Identify when voice compression channels are required Describe IP Office settings that should be configured to improve IP voice quality

Click the forward arrow to continue. Summary (Continued) Links to the documents used in this Learning Module: Voice Compression Module settings VoIP Settings Analog Options