Avid inews Command Best Practices Failover and Redundancy This document outlines the Command Server failover and Command hotswap failover functionality of inews Command. It includes the performance expectations, recommended setup, and failure/recovery scenarios. 1 Performance Expectations 1.1 Failover inews Command 1.0.x functionality is expected to return within 30 seconds. inews Command 1.1.0 (Improved performance) functionality is expected to return within 15 seconds. inews Command 1.1.1 (Improved performance) functionality expected to return within 10 seconds. 1.2 Hotswap inews Command 1.0.x and Command 1.1.x In the event of a network or power failure to an Airspeed device, playout functionality is expected to return within 10 seconds. 2 Recommended Setup for Command Server Failover 2.1 Configuration It is recommended to have all primary Command services on one designated primary server with the backup services running on others. Command determines primary and backup roles based on the length of time the service has been running. The server that has been running longer is the primary. Therefore, in order to get all Primary services on one server, it is necessary to start the primary server first and when all services have started, start the backup servers. This is the only way to ensure all services on the Primary server take on the Primary role. This allows for more consistent behavior during times when redundancy is needed and helps simplify any necessary troubleshooting efforts. All Command Servers should be assigned static IP addresses. Windows Media Sense should be disabled. Microsoft QArticle 239924. If the Command server has dual network cards, the unused connection must be disabled. 2.2 Redundant Services It is best to dedicate a machine to be a Command server. We do not recommend running the main Command services on a workstation, Deko, or Thunder. The following are redundant Command services: o AirSpeed device service o Playlist Controller service o Playlist Manager service o Playlist Store service
o o Inventory Manager service Avid MOS Communication Service: This service is server specific, meaning that services on different computers act independently. In a failover scenario, the Command playlist will become disconnected, so users can continue playout functions. To get an updated playlist, the user must perform a MOS redirection in ENPS to the Command Server running the now primary Avid MOS and then continue as before. 2.3 Non-redundant services There are some services that are not redundant: o Lookup Service This service acts independently. If a second one is running, it is doing the same work as the first Lookup service they basically work as 2 primary services. In a failover scenario, the second Lookup service is already working, and services would continue seamlessly. o inews Communication Service This service is server specific, meaning that services on different machines act independently. In a failover scenario, the Command playlist will become disconnected, users can continue playout functions. To get an updated playlist, the user can reload the inews rundown to the other Command server running the Avid inews Communication ;service and continue. o Deko & Thunder Device Services These services need to run on the Deko and Thunder themselves, and you cannot install a second service on a machine. o Service Starter Service, Machine Monitor Service, Time Synchronization These services are machine specific redundancy does not apply. 2.4 Primary and Backup Roles for Command Services Command Services can have two roles, primary and backup. There can be only one primary service, but there can be several backup services. When a primary service goes down, Command conducts an election, and changes the backup service to primary. The new primary service will remain primary until it is stopped or restarted. When the old primary service comes back up, it becomes the backup service. If the Command Server running the primary services were to incur a failure or network disconnect, all of the backup services running on the redundant Command Server would become primary. 3 Failure/Recovery Scenarios This section provides various failover scenarios, the symptoms seen at a Command Workstation that would indicate such problems, and the recommended procedures for recovery. Failover Scenario Primary device channel failure. Mirrored device channel failure Status bar indicator for that device channel will show disconnected and the playability icon will change to. Status bar indicator for that device channel will show disconnected and the playability icon will change to. The person running the switcher will need to take the mirrored channel to air. No recovery needed
Failover Scenario If the Hotswap Channel is needed due to device channel failure Power Outage or computer restart on the server with the a primary role. Power Outage or computer restart on the Command server running the backup services. inews loses its connection to the Avid inews where inews monitor is currently loaded. Stop individual primary Command service (one of Playlist Controller, Manager, or a Device service). The channel status shows this icon,, when the channel fails, and when Hotswap takes over, an Alert appears and the channel status changes to this icon,, to indicate that it has been replaced by the Hotswap channel. The Count Up and Count Down clocks will stop immediately. There will be multiple alert messages sent to the workstation saying that it can t find the Playlist Store and other services. Note that the currently playing video will continue to play, but there will be no transport control. Nothing seen The inews client will show monitor in an ERROR state. Command will show the playlist as disconnected (red bar across top of playlist view). Any updates to the inews rundown while in this state will NOT be reflected in Command. The user may see an alert in the Alerts panel and there may be a short playout interruption while the backup service takes over the primary role. The person running the switcher will need to take the hotswap channel to air. Within 10 seconds, you will receive an alert message indicating that the Playlist Controller is available, with the details stating that items can be played again. If a newsroom playlist was being controlled at the time of the failure, the disconnected rundown can be played before the rundown is reloaded to the redundant inews Command server (now running primary services). No recovery needed. inews monitor will need to be reloaded to the Command server running the secondary Avid inews. After it s been reloaded to Command, unlock the channel groups in the disconnected playlist (when convenient) and lock them for the reloaded playlist and continue playing to air. Wait until the workstation returns to a normal state and then continue.
Failover Scenario Loss of network connectivity on the server with the a primary role. Loss of network connectivity on the server with the a backup role. ENPS loses connectivity with the primary Avid MOS ENPS loses connectivity with the Avid MOS running in the backup role. The Count Up and Count Down clocks will stop immediately. There will be multiple alert messages sent to the workstation saying that it can t find the Playlist Store and other services. Note that the currently playing video will continue to play, but there will be no transport control. Important information pertaining to network restoration: There may be a scenario where another 10 second loss of playout control may be experienced when network connectivity is restored. No effect when network connectivity is lost. Command will show the playlist as disconnected (red bar across top of playlist view). Any updates to the ENPS running order while in this state will NOT be reflected in Command. Nothing seen The services running in a backup role will take over as primary and restore functionality within 10 seconds. If a newsroom playlist was being controlled at the time of the failure, the playlist will need to be reloaded to the redundant server (now running primary services) from the newsroom system. Important information pertaining to network restoration: If possible, have someone restart that server before network connectivity is restored. If not, Command will go through another election and the Command workstation user will incur another 10 second period where they don t have playout control when the network is reconnected. Once the show is over, the system administrator may want to restart services so the primary server is once again the primary server. No effect when network connectivity is lost. Let Command automatically recover if the network connectivity is restored. In ENPS, perform MOS redirection to the Command server running the redundant Avid MOS Communication service. No recovery needed. 4 Troubleshooting Scenarios This section provides miscellaneous troubleshooting scenarios and the recommended procedures for resolution. Troubleshooting Scenario Recommended Resolution
Troubleshooting Scenario Failed communication between Interplay Framework and Command services. Dual network card enabled. Recommended Resolution Ensure that the DNS entries are accurate and have reverse lookup enabled. Also verify that your TCP/IP settings have the correct primary and alternate DNS addresses. Avoid using host names. You must disable the unused connection.