Avid Redundancy and Failover in Avid News Management Solutions
Table of Contents Introduction............................................................1 Newsroom Computer System (NRCS)....................................2 Redundancy and Failover................................................2 Local Redundancy.....................................................3 Wire Caching.........................................................3 Disaster Recovery......................................................4 Database Backups.....................................................4 Automation Assist System..................................4 Running Without Connection to......................................5 Flexibility............................................................5 Backup................................................5 Rundown Mirroring....................................................5 MOS (Media Object ) Gateway..........................................6 Backup MOS Gateway...................................................6 Conclusion.............................................................6
Introduction Newsroom technology has matured along with IT systems design and network architecture. While broadcasters are realizing greater productivity from increasing reliance on networked news management systems, they are also seeing the necessity for a robust architecture that provides high availability and reliability while getting the news on the air. Whether running remote newsrooms, airing multiple newscasts, ensuring protection during downtime or even during network-disabling disasters, Avid news management products provide very high availability and reliability across the system, through a redundant architecture and the capability for failover to a standby system. Redundancy in a newsroom system is one of the most important technical design elements for high reliability. By reducing single points of failure, enabling failover to standby production devices, servers, or network paths, and building in non-interruptive regular maintenance capability, newscast production can continue during a malfunction. Avid news management systems incorporate all these techniques in a carefully designed infrastructure, purpose-built for breaking news. 1
Newsroom Computer System (NRCS) The newsroom computer system automates ingest of news wires and other stories, sorting them into categories and sending notifications to groups of users automatically, all based on customizable rules. It collects news data from diverse sources, including serial wire feeds, telnet, email, and network shares. Journalists use to browse wire feeds and the internet from their desktops, dragging and dropping pertinent information into assignment sheets and news stories. increases efficiency by allowing multiple users to collaborate on the creation and production of a show. The latest content is instantly presented to all users working in the rundown. When the rundown is ready to air, it can be sent to the system or the MOS Gateway, or it can be extracted by a third party automation system. Any changes made to the rundown in are reflected in the attached production devices. Redundancy and Failover Up to three servers can be configured in a mirrored cluster, connected over a special LAN backbone (dedicated 100-base-T network recommended). All data, including directory structure, stories, and user information is constantly mirrored across each server in the system. User sessions and processes are load-balanced automatically across the servers. In the event that one of the servers becomes unavailable, the users connected to that system would be logged off with a message. When they attempt to log back into the system, they will automatically be connected to an available server. Loss of mirroring network service between servers causes the servers to disconnect and run independently. At this point, one server is designated as the master, and once any hardware faults have been rectified, the system administrator restores the failed server. This process also involves running the diskclear utility on the failed server in order to erase the residual database information on that machine. A diskclear operation typically takes 45 minutes or less, depending on hard drive configurations. Once diskclear has completed, the failed server is reconnected with the active server(s). This can be done without any system downtime. Once reconnected, a procedure called diskcopy is run on a reconnected server. This involves block-by-block re-mirroring with the active server at the same time that new data is mirrored and users are serviced. The diskcopy operation runs in the background and typically completes in 8 hours or less, depending on hard drive configurations, network configuration, and system activity. 2
Local Redundancy provides a local backup mechanism that allows the client to save a backup copy of a story as it is being edited. The backup location and interval are customizable. When a user is working on an unsaved story and becomes disconnected from the servers, he is prompted to restore the most recently backed-up copy as soon as reconnection occurs. supports a local database, which can be used alone or in combination with the online system database. Story forms and print styles can be synchronized with the online database at any time while the user is online. A directory structure can be created in the local database, and forms can be assigned to different groups or functions so that stories will always be written in the proper format. Stories can be moved or copied between the online and local databases, allowing users to view or edit stories offline, then move them back into the online system when they reconnect. Wire Caching The Data Receiver built for the system is designed to ensure that newsrooms reliably receive their wire content under demanding conditions. It ensures that no data is lost if the system is taken offline. The system captures status and error events in Data Receiver logs, and sends system notifications to specified users in email form. If the newsroom system becomes unavailable, the Data Receiver can cache incoming stories up to a configurable volume of disk space, transferring them into the newsroom system when it becomes available. This is a useful feature in the case of either planned maintenance downtime or unexpected network failures. A B Mirroring Switch C Fast Text Search (Indexing) Data Receiver (Wire Ingest) Data Receiver (Wire Ingest) LAN/WAN (Video) Control Room 1 (Backup Video) (CG) Figure 1 A newsroom network designed with mirrored servers, a hot standby Data Receiver, and backup servers ensures availability to all users and processes. Console Modem Console LEGEND 100 Base-T Ethernet Gigabit Ethernet Serial Control Connection Clients (Video) (Backup Video) Control Room 2 (CG) 3
PAPI Disaster Recovery For disaster recovery, customers have the option to set up a separate server system in a location other than the primary news production location. This system would be licensed identically to the primary production system. On a weekly basis, the database from the primary system would be backed up to the disaster recovery system, with possible incremental backups performed nightly. Some important, customer-defined areas of the database could be mirrored regularly throughout the course of the day via FTP transfers. This remote system would be configured with appropriate hardware and licenses to accommodate users in the event that one of the primary systems was to become unavailable. Wires could be ingested into the remote systems directly, ensuring that the latest wire content would be available in any event. Fairly static areas of the database, such as archives, user files, and rolodex, would be backed up from the primary system and restored to the remote system on a daily or weekly basis. Rundowns and assignment desk areas could be mirrored to the remote system throughout the course of each day. Database Backups All areas of the system, including the directory structure, stories, and user information, can be backed up and restored using any method supported by the server operating system. Automation Assist System server works with the system to assist in playout automation of news production devices including character generators, video servers, and still stores. Rundowns created in the system are sent to the server, which coordinates the playback from numerous devices. Changes made to the NRCS rundown are reflected in the playlist in real time. Even if the server becomes disconnected from the system, it maintains its playlist, allowing playback to continue without interruption. Figure 2 Avid assists automation of on-air production devices. One server can control multiple channels, as separate channels, on different video servers Rundown System PAPI Primary AirSPACE Monitor 4 Primary AirSPACE
Running without connection to Should the servers become unavailable, playback control by the system can continue using the last copy of the rundown that was available. However, the system would not receive playlist updates during the failure; once the system became available again, a user would simply reload the show in the NRCS and reserve the new playlist in the. Flexibility Separate servers can be installed for each control room, with the capability of controlling broadcast devices at either their local site (for CG, still stores, and video servers) or in a remote location (specified video servers only). Any workstation with network access can connect to any of the s, enabling them to control playout from the server s attached devices. Many larger Avid customers use the system to playout machine events from multiple geographic locations. The system can be configured to control channels on separate video servers. In the event that one of the servers becomes unavailable, playback can continue by assigning all the remaining video events to alternate channels. Backup In the event that the primary server becomes unavailable, a backup server can be loaded.this allows playback to continue with only a brief interruption. Rundown Mirroring The system supports action servers which can create mirrored copies of the working rundown in another location. The mirrored rundown can be loaded to a backup server, which would be available in the event of a failure of the primary server. The backup server can even be connected to a separate video server and workstation, providing complete redundancy with a hot standby for every component in the system. Working Rundown Rundown Mirroring Monitor Primary System PAPI Primary AirSPACE Figure 3 rundown mirroring can be utilized to provide a hot-standby system. Here, the rundown is mirrored to another location on the same system, with servers linked to both the main and the mirrored rundowns.this setup is in hot standby status at all times. Monitor PAPI 5 Mirrored Rundown Backup System Primary AirSPACE
MOS (Media Object ) Gateway Media Object (MOS) is a communication standard developed to facilitate the transfer of media object information between broadcast devices and newsroom systems. The standard has been implemented by a large number of vendors, including Avid. Avid has representation on the MOS Committee, and has been instrumental in moving the standard forward. The MOS Gateway offers an additional method for the system to communicate with third-party devices. Backup MOS Gateway In the event that the primary MOS Gateway becomes unavailable, a backup MOS Gateway can be loaded. This allows playback to continue with only a brief interruption. Conclusion The widespread user and technology base of worldwide news organizations and station groups, including ad hoc newsgathering and transmitting locales during significant events around the world, makes complete confidence in a system s reliability one of the foremost considerations in building the newsroom infrastructure. For local or independent stations, having fewer newscasts, users, locations and devices may seem to mean fewer points of failure. However, making sure to consider both typical and worst-case scenarios, from normal database backups or system monitoring to a remote system malfunction or disaster recovery, is the best way to ensure the technology in the newsroom keeps running smoothly. 6
7
Corporate Headquarters tel 800 949 AVID 978 640 6789 fax 978 640 1366 web www.avid.com Asian Headquarters tel +65 6476 7666 Avid China tel +8610 6708 1744 Avid Japan - Tokyo tel +81 3 3505 7937 Avid Japan - Osaka tel +81 6314 3132 Avid Korea tel +822 782 4210 European Headquarters tel +44 1753 655999 Avid Central Europe Munich tel +49 811 5520 0 Avid Broadcast Competence Center Berlin tel +49 30 5900993 0 Avid France tel +33 1 41 49 40 00 Avid Spain tel +34 91 715 66 81 Avid United Kingdom tel +44 207 534 2800 The following international operations are independent distributors operating under separate license from Avid. Avid Australia tel +61 2 9629 2633 Avid Italy tel +39 02 57 78 97 1 Avid Nordic AB Headquarters tel +46 8 442 55 80 Avid Austria AG tel +43 1 525 27 0 Avid Belgium tel +32 2 254 88 80 Avid Denmark tel +45 39 55 99 99 Avid Finland tel +358 9 2290160 Avid Netherlands tel +31 35 6250 080 Avid Norway NUF tel +47 22 02 12 99 Avid Sweden tel +46 8 442 55 70 2004 Avid Technology, Inc. All rights reserved. Product features, specifications, system requirements and availability are subject to change without notice. AirSPACE, Avid and make manage move media are either registered trademarks or trademarks of Avid Technology, Inc. in the United States and/or other countries. and are trademarks of inews, LLC. All other trademarks contained herein are the property of their respective companies. AIWP0204