How can I. implement a high-availability system? Develop your project. System Technical Guide High Availability solutions

Transcription

1 How can I implement a high-availability system? System Technical Guide High Availability solutions Develop your project

2

3 Disclaimer This document is not comprehensive for any systems using the given architecture and does not absolve users of their duty to uphold the safety requirements for the equipment used in their systems or compliance with both national or international safety laws and regulations. Readers are considered to already know how to use the products described in this document. This document does not replace any specific product documentation. 3

4 The STG Collection System Technical Guides (STG) are designed to help project engineers and Alliance System Integrators during the development of a project. The STGs support users during the architecture selection and the project execution (design, configuration, implementation and operation) phases, with an introduction to the system operating modes. Each STG is a starter kit that provides users with: Technical documentation Application examples Object libraries Each STG addresses one or several customer challenges within the proposed solution using the offer from Schneider Electric. All explanations and applications have been developed by both Schneider Electric experts and system integrators in our solution labs. The contributions from the system integrators help the kit s content address the needs of our users. All STGs are illustrated with industry-specific applications to give more concrete examples of the methodology. The STGs are not intended to be used as substitutes for the technical documentation related to the individual components, but rather to complement these materials and training. Development Environment Each STG has been developed in one of our solution platform labs using a typical PlantStruxure architecture. PlantStruxure, the process automation system from Schneider Electric, is a collaborative system that allows industrial and infrastructure companies to assess their automation needs while at the same time addressing their growing energy management requirements. In a single environment, measured energy and process data can be analyzed to yield a holistically optimized plant. 4

5 Table of Contents 1. Introduction Purpose Customer Challenges Prerequisites Project Methodology Selection Redundancy Basics Operational Principles Selection Criteria Selected Architecture Conclusion Design Introduction SCADA System Premium Hot-Standby System Quantum Hot-Standby System Configuration SCADA Control and Field Network Premium Hot-Standby PAC Station Quantum Hot-Standby PAC Station Advantys STB Ethernet I/Os Implementation Premium PAC

6 5.2. Quantum PAC Conclusion Performance Performance test protocols Premium PAC Architecture Quantum PAC Architecture

7 1-Introduction 1. Introduction 1.1. Purpose The purpose of this document is to provide recommendations, guidelines and examples to help implement a high availability automation system. A System Technical Note: How can I increase the Availability of a system detailing the theoretical basics of high-availability has already been issued. This guide focuses on the different ways to increase availability through redundancy at various layers of the automation system and proposes a methodology to implement an efficient high-availability automation system. Each step of the implementation of the solution, from the selection to the operation, is broadly described and illustrated with examples. The solutions described in this STG are fully part of the PlantStruxure control system. The High availability System Technical Guide is structured in two parts: Part 1: Redundant architecture with single attachment. Part 2: Complex architecture including dual network attachment and dual ring Customer Challenges For customers in industries where availability and reliability are major concerns, the challenges are: Provide a high level of availability. Attain a high level of reliability. Short recovery following system unavailability. Maintainability facilitated by redundancy. Switchover time adapted to critical processes. This STG suggests approaches to these challenges and highlights specific areas using SoCollaborative Engineering products in line with PlantStruxure Control System. 7

8 1-Introduction 1.3. Prerequisites We recommend the reader have knowledge of the following SoCollaborative software: Unity Pro Vijeo Citect We also recommend the reader become familiar with the System Technical Note: How can I increase the Availability of a system and to have knowledge of the Premium/Quantum/M340 Schneider PACs Project Methodology This STG describes the project methodology and includes the following phases: Selection, Design, Configuration, Implementation and Performance. This guide is illustrated using 2 architectures (Premium and Quantum). Their features are described in the Selection phase. Each architecture shows a specific feature necessary of Hot-Standby application. Beginning with process analysis and user requirements, we identify and develop common functionalities for all the architectures. These key functions are explained in the Design, Configuration and Implementation phases. Finally, the Performance phase summarizes the results of different tests performed on the 2 architectures. Here are the phases described in this document: I. Selection: In this phase, the selection procedure to define a redundant architecture is presented: Basic of redundancy Operational Principles Description of architectures II. Design: This phase covers the operational principles of the different components of high availability architecture: SCADA system Network PAC Station Quantum and Premium 8

9 1-Introduction Specifications and constraints I/O system DFB library III. Configuration: In this phase, the configuration information of the different components of the architectures is detailed. IV. Implementation: Using the same architectures and components, information about final customization to address the project requirements is provided. V. Performance: A summary of the architectures performance in response to simulated events is presented in this phase. 9

10 2-Selection 10

11 2-Selection 2. Selection During the selection phase, an optimal architecture is chosen as well as the most appropriate components of the project, according to your specific requirements. Several architectures and systems are presented in this chapter in order to address a wide range of functions and needs. Also, the way to select among these architectures given project needs is presented. The following illustration summarizes our project development approach: 11

12 2-Selection 2.1. Redundancy Basics This chapter describes redundancy general principles and its application in an automation system. The following PlantStruxure architecture is a representative example to illustrate the different layers where redundancy can be implemented. SCADA Clients Ethernet Data Servers Ethernet Control Network PACs Station PAC PAC PAC Field Network Ethernet Profibus DP Ethernet Field Devices The diagram also represents a wide range of hardware setups demonstrating the ability to achieve various redundancy levels Redundancy Layers Availability can be increased in an automation system at different layers: SCADA system: The SCADA system has to handle data acquisition, graphics, events, alarms, trends, and reports. SCADA server redundancy enhances the likelihood these services will continue to operate without loss of data in case of system interruption. Different software and hardware configurations allow different levels of availability. Control Network: A well defined topology and management of the control network increase network availability and reliability. Thus, in turn, makes communication between the SCADA system and the PAC stations more reliable. Several network topologies and network 12

13 2-Selection protocols are available to achieve the optimal level of availability and to fit the whole system needs. PAC station: According to your needs in terms of I/O number and topology, you can choose among a Quantum or a Premium Hot-Standby PAC system. The field network type is also an element to consider before choosing the PAC station, as well as the I/O system (local or distributed). Field Network: Redundancy can also be applied to the field network. As was the case for the control network, a well defined topology and management of the network increase field network availability. Device redundancy is also implemented to increase availability of the field equipment Redundancy level We can differentiate several levels of redundancy according to their performances in term of availability. A summary of these levels is presented in the following table: Redundancy Level State of the standby system Switchover performance No redundancy No standby system Not applicable Cold Standby Warm Standby Hot-Standby The standby system is only powered up if the default system becomes inoperative. The standby system switches from normal to backup mode. The Standby system runs together with the default system. Several minutes Large amount of lost data Several seconds Small amount of lost data Several milliseconds No lost data 13

14 2-Selection 2.2. Operational Principles Depending on the level of availability required, redundancy is applied in various ways. This chapter discusses Hot-Standby applications and describes the basics of redundancy in each layer of an automation system. Moreover the chapter describes the various options available in terms of redundancy and availability. The selected architectures used in the following parts of this STG are described in the Chapter SCADA The main operating principles of the different SCADA servers are described in the following paragraphs. General Principles The different servers of the SCADA system (Alarm, Report, Trends, and I/O servers) can either be installed on the same computer or on different computers allowing for more reliability. For a redundant configuration, each server (Primary) is associated with its redundant server (Standby) installed on a different computer. For example, the picture below describes servers installed on redundant computers (Primary and Standby) 14

15 2-Selection I/O Server redundancy In a redundant SCADA system, an I/O device is associated with the Primary and Standby servers. The Primary server accesses periodically the I/O device to read and write tags. The Standby server only checks the communication with the I/O device. At startup, if the Primary I/O server can not establish a connection with the I/O device, the SCADA system switches to the Standby I/O server. During operation, if the Primary I/O server stops communicating with the I/O devices, the system then switches to the Standby I/O server. The following diagrams illustrate 2 cases: a broken network cable and a server that has stopped communicating. When the I/O server defined as Primary returns to operational state, the SCADA system returns control back to the Primary server. Alarms / Trends / Reports Servers (ATR) redundancy The management of the Alarms, Trends and Report servers (ATR servers) by the SCADA system follows the steps listed below: If the Primary ATR server stops operating, the system switches to the Standby ATR server. When the ATR server defined as Primary returns to operational state, any clients connected to the Standby ATR server remain connected to the Standby server. 15

16 2-Selection For example, the following picture describes a server reconfiguration initiated by a switchover: I/O, Trends and Reports servers are working on the Primary SCADA server, and the Alarms server is working on Standby SCADA server. 16

17 2-Selection Network Various topologies and protocols are used to increase the availability of the network (control or field network). The principle is to create different paths to access devices. In case of a network element on the main path stops functioning, another path is used. The following table illustrates the main network topologies. Architecture Limitations Advantages Disadvantages Bus The traffic must flow serially, Cost-effective solution If a switch becomes therefore the bandwidth is not inoperative, used efficiently. communication is lost. Star Efficient use of the If the main switch bandwidth, as the traffic is becomes inoperative, spread across the star. communication is lost. Tree Cable ways and distances Preferred topology when there is no need for redundancy. Ring Auto-configuration if used The availability of auto- with self-healing protocol. configuration depends Possible to couple other on the protocol used. Dual Ring Behavior similar to Bus. rings for increasing redundancy. Ring topologies are mainly used to increase the level of network availability. Network redundancy management protocols, such as Hiper-ring or MRP, are used for network recovery in case of part of the network cease to function. 17

18 2-Selection PAC Station Hot-Standby Definition A Hot-Standby system is used when downtime cannot be tolerated. It delivers high availability through redundancy and always consists of two units with identical configurations. One of the two units acts as the Primary CPU controller, and the other acts as the Standby CPU controller. One controller must be set in the Primary CPU state and the other must be in the Standby CPU state or offline. The redundant unit takes the control when the main one encounters an anomaly. The Primary PAC updates inputs, manages Hot-Standby, runs the program while transferring data to the Standby PAC and updates outputs. Thus, the switchover between the Primary and the Standby PAC occurs without any loss of data. As described on the diagrams above, for each execution cycle, the outputs update only takes place when the data transfer AND the program execution are completed. Therefore, it is important to properly define the amount of data to be transferred from the Primary to the Standby PAC to minimize the wait time induced by a data transfer longer than the execution time of the program execution. On the diagram on the left, the cycle execution is optimized: the data transfer is performed faster than the program execution. On the diagram on the right, the longer data transfer induces a wait time that slows down the cycle execution. 18

19 2-Selection Primary and Standby PACs Assuming that the configuration of the system is correct, the first PAC to be powered up is automatically recognized as the Primary one. Therefore, you can define the PACs role by controlling the sequence order in which they are powered up. When two redundant CPU PACs are switched on simultaneously, the firmware automatically affects the Primary status according to the MAC address. The PAC with the lower MAC address is defined as the PAC A, that is the Primary at the powering up of the system. 19

20 2-Selection Hot-Standby System Programming Elements This paragraph describes programming basics, useful to know when implementing a Hot-Standby system. System Words %SW60: Command Register The command register defines the operating parameters of a Hot-Standby application for both the Primary and Standby CPU. The System Word %SW60 can be used to read and write the command register of Hot-Standby System. The diagram below illustrates the Quantum System Word %SW60: Disables LCD Invalidate Keypad - bit0 = 0 Enables LCD Invalidate Keypad - bit0 = 1 Sets Controller A to OFFLINE mode - bit1 = 0 Sets Controller A to RUN mode - bit1 = 1 Sets Controller B to OFFLINE mode - bit2 = 0 Sets Controller B to RUN mode - bit2 = 1 Forces Standby offline if there is a logic mismatch - bit3 = 0 Does not force Standby offline if there is a logic mismatch - bit3 = 1 Allows exec upgrade only after application stops - bit4 = 0 Allows exec upgrade without stopping application - bit4 = 1 MSB LSB bit5 = 0 - No application program transfer bit5 = 1 - Application program transfer requested bit8 = 0 - Swaps Modbus port 1 adress during switchover bit8 = 1 - Does not swap Modbus port 1 adress on a switchover bit9 = 0 - Swaps Modbus port 2 adress during switchover bit9 = 1 - Does not swap Modbus port 2 adress on a switchover bit10 = 0 - Swaps Modbus port 3 adress during switchover bit10 = 1 - Does not swap Modbus port 3 adress on a switchover The diagram below illustrates the Premium System Word %SW60: Sets Controller A to OFFLINE mode - bit1 = 0 Sets Controller A to RUN mode - bit1 = 1 Sets Controller B to OFFLINE mode - bit2 = 0 Sets Controller B to RUN mode - bit2 = 1 OS versions Mismatch (this bit can be used to permit temporary differences between the firmware versions on the respective Hot-Standby PACs) MSB LSB 20

21 2-Selection %SW61: Status Register The Hot-Standby Status Register is a readable register located at system word %SW61 and is used to monitor the current status of the Primary CPU and Standby CPU. The following diagram illustrates the Quantum System Word %SW61: This PAC in OFFLINE mode - bit1= 0. bit0= 1 This PAC running in primary CPU mode - bit1= 1. bit0= 0 This PAC running in standby CPU mode - bit1= 1. bit0= 1 Other PAC in OFFLINE mode - bit3= 0. bit2= 1 Other PAC running in primary CPU mode - bit3= 1. bit2= 0 Other PAC running in standby CPU mode - bit3= 1. bit2= 1 The remote PAC is not accessible - bit3= 0. bit2= 0 PACs have matching logic - bit4 = 0 PACs do not have matching logic - bit4 = 1 This PAC's switch set to A - bit5 = 0 This PAC's switch set to B - bit5 = 1 MSB LSB bit7 = 0 - Same PAC OS version bit7 = 1 - Different PAC OS version bit8 = 0 - Same copro OS version bit8 = 1 - Different copro OS version bit12 = 0 - Information given by bit13 is not relevant bit12 = 1 - Information given by bit13 is valid bit13 = 0 - NOE address set to IP bit13 = 1 - NOE address set to IP+1 bit15 = 0 - The hot standby has not been actived bit15 = 1 - The hot standby is active 21

22 2-Selection The following diagram illustrates the Premium System Word %SW61: This PAC in OFFLINE mode - bit1= 0. bit0= 1 This PAC running in primary CPU mode - bit1= 1. bit0= 0 This PAC running in standby CPU mode - bit1= 1. bit0= 1 Other PAC in OFFLINE mode - bit3= 0. bit2= 1 Other PAC running in primary CPU mode - bit3= 1. bit2= 0 Other PAC running in standby CPU mode - bit3= 1. bit2= 1 The remote PAC is not accessible - bit3= 0. bit2= 0 No application Program or Unity Pro configuration Checksum mismatch beetween Remote PAC - bit4 = 0 Application Program or Unity Pro configuration Checksum mismatch beetween Remote PAC - bit4 = 1 This PAC set as Unit A - bit5 = 0 This PAC set as Unit B - bit5 = 1 CPU-sync link OK - bit6 = 0 CPU-sync link NOK - bit6 = 1 No processor OS version mismatch - bit7 = 0 Main processor OS version mismatch - bit7 = 1 No Copro OS version mismatch - bit8 = 0 Copro OS version mismatch - bit8 = 1 MSB LSB bit10 = 0 - No monitored ETY OS version mismatch bit10 = 1 - Monitored ETY OS version mismatch bit13 = 0 - Configured IP or Modbus Adress bit13 = 1 - Configured IP or Modbus Adress +1 bit15 = 0 - The hot standby has not been actived bit15 = 1 - The hot standby is active bit9 = 0 - All in-rack (Monitored and no-monitored) ETY modules have the minimun version bit9 = 1 - At least one ETY does not have the minimun version 22

23 2-Selection %SW62 65: Reverse Register: System Words %SW62/63/64/65 are reverse registers reserved for the reverse transfer process. The reverse registers can be written in the application program (first section) of the Standby CPU controller and are transferred at each scan to the Primary CPU controller. Non-Transfer Area The Non Transfer Area is a defined memory zone which is not transferred during the update of the Standby CPU controller. The Premium PAC has a 100 words predefined zone (%MW0 to %MW99) whereas the Quantum PAC the size of the zone is defined by the user (%MW1 to %MWx). First Section (section 0) In a PAC redundancy system, the execution of the application program is different according to the PAC in which the execution takes place. The main difference is that the whole application program is executed in the Primary PAC whereas the Standby PAC only executes the first section (section 0). This point is very important as many settings of the system are defined in the section 0. Hot-Standby Management Switchover Conditions A Hot-Standby system is designed to provide uninterrupted service. This feature requires continuous monitoring of different equipment. Note: Concerning both Premium and Quantum, the switchover is performed only if the Standby PAC is operational and ready to take over control from the Primary PAC. A Hot-Standby system continuously monitors the key components in order to detect any stoppage in operation. Additional monitoring is performed by the application for more specific requirements. The monitoring by the system initiates a switchover on the following occurrences: Premium system Fault on power supply Fault CPU (firmware, hardware) Halt, Stop, Offline CPU Fault Monitored ETY module (firmware, hardware) 23

24 2-Selection Quantum System Fault power supply Fault CPU (firmware, hardware) Halt, Stop, Offline CPU Fault CRP module For the Premium and Quantum Hot-Standby architectures presented in this guide, additional equipment (Network Controller and Device Network) are monitored by the application in order to increase the availability. To perform this specific monitoring, we need to develop Derived Function Blocks (DFBs) that monitor the system, control and process the anomalies, and handle the switchover. DFBs Libraries The Unity Pro Quantum system library offers EFBs to manage a Hot-Standby system. These EFBs allow the handling of command (%SW60), status (%SW61) and reverse (%SW62 to 65) registers. The Unity Pro Premium library does not include pre-designed EFBs. Consequently, we have developed a user-defined Hot-Standby DFBs library. This library is described in the next chapter. Network monitoring is integrated in our architectures. This functionality is handled neither by the Premium PAC nor by the Quantum PAC (except for the Monitored ETY module on a Premium system). Therefore, we have developed a specific Hot-Standby DFBs library for each configuration: Premium ETY_Monitor (Ethernet) Quantum NOE_Monitor (Ethernet) PTQ_Monitor (Profibus) We have developed an events synthesis block that processes the output of these DFBs, while offering the possibility to mask some defaults, and a switchover management block which controls the availability of the Standby PAC before initiating a switchover. 24

25 2-Selection 2.3. Selection Criteria Various levels of performance can be attained with different architectures and using different components. It is crucial to select the right configuration that most closely fits your needs in terms of availability, cost and maintenance. Before implementing a high-availability system, consider the following points: What is the general availability level to reach for the whole system? How many devices can stop functioning, yet have the system remain operational? What is the maximum allowed downtime for the entire system? Are there any constraints (existing system, topology) that imply the use of specific tools and equipment? What is the size of the system and are additional extensions planned? What is the topology? Centralized? Distributed? Are there some areas of the process with priority needs? 2.4. Selected Architecture We have selected two representative architectures to illustrate in this guide, one for Quantum PAC and one for Premium PAC. All the other layers are common for the two architectures. These architectures are intended to represent a medium range automation system with high availability needs in terms of process control and medium availability in terms of SCADA and network control. Each layers in the system can tolerate one non-operating device and still remain operational. The implementation of the different layers is described in the following chapters SCADA The resources used by our application are moderate, so all the servers (Alarms, Trends, Reports and I/O) can be installed on one computer. In order to withstand one non-functioning device, the redundant servers are installed on a second computer. Two clients connected to the network allow the control and monitoring of the process. This configuration is adapted to our needs in terms of performance and redundancy level. 25

26 2-Selection Control Network The ring architecture is chosen for its redundancy capability. Four ConneXium switches handle the ring architecture. One switch or cable segment can cease operating with no impact on the communication through the network. The MRP redundancy management protocol is chosen for its performance in recovery time. 26

27 2-Selection PAC station The Hot-Standby architecture allows that one PAC stops operating without loss of data. As was the case with the control network, if the Primary PAC ceases to operate, the Standby PAC takes over from the Primary. The following diagram sums up the selection of the PAC station according to specific requirements of the application. Time Critical Application? N Y In-Rack I/O Stations only! Application requiring multiple and/or scattered I/O Stations? N In-Rack and/or distributed I/O system Y Remote I/O Stations Local I/O Station Redundant I/O Modules required? Y N Premium or Quantum HSBY Quantum HSBY Premium or Quantum HSBY Premium HSBY The selected architectures, Premium and Quantum, are detailed in the following paragraphs. Note: The number of In-Rack I/Os in the process is decisive for dimensioning the system. As a Premium Hot-Standby system does not handle extension racks, the use of In-Rack I/Os is limited. This means that, beyond a given number of In-Rack I/Os, a Quantum PAC that handles Remote I/Os will be used instead of a Premium PAC. Note: Only the Quantum PAC station provides redundancy solutions for a Profibus network. 27

28 2-Selection Premium The chosen redundant PAC station is a Premium Hot-Standby architecture with redundant analog and digital inputs and outputs. The 2 units are synchronized via an Ethernet link. From Control Network PAC A IP: MASK: PAC B IP: MASK: Sync-link IP: MASK: IP: MASK: JM Concept Modules Analog Output Analog Input ABE7 Connection Blocks ABE7 ABE7 Digital Output Digital Input To Ethernet Field Network 28

29 2-Selection Quantum The chosen redundant PAC station is a Quantum Hot-Standby architecture with a shared Remote I/Os module. The 2 units are synchronized via an optical fiber link (Sync link) Field Network Profibus DP The first part of the field network is composed of a Profibus DP daisy chain managed by 2 redundant Profibus Master modules. Each extremity of the daisy chain is connected to a redundant Quantum PAC. The control of the device on the chain is possible even if one of the PACS ceases to operate. The Standby Profibus master PAC then handles the control of the chain. 29

30 2-Selection Ethernet The second part of the field network is built around an Ethernet ring to bring redundancy to the field devices. The Ethernet ring is built in the same manner than for the control network. 3 Connexium switches run MRP on the ring and connect different devices connected on Ethernet. 30

31 2-Selection 2.5. Conclusion The 2 following diagrams present the whole Quantum and Premium architectures from the SCADA to the field network. These architectures will be used subsequently in the document to illustrate redundancy runtime principles and performance reviews. The second part of this document will use the same architectures, but including dual ring structures and dual attachment. 31

32 2-Selection Redundant Premium Architecture SERVER 1 SERVER 2 IP: Client 1 IP: IP: SW1 SW2 Manager IP: Client 2 SW3 SW4 PAC A IP: MASK: PAC B IP: MASK: Sync-link IP: MASK: IP: MASK: JM Concept Modules Analog Output Analog Input ABE7 Connection Blocks ABE7 ABE7 Digital Output Digital Input IP: IP: SW10 Manager SW12 SW11 IP: IP: IP: IP: IP:

33 2-Selection Redundant Quantum Architecture SERVER 1 SERVER 2 IP: IP: IP: IP: Client 1 SW1 SW2 Manager Client 2 SW3 SW4 PAC A IP: MASK: PAC B IP: MASK: Remote I/O IP: MASK: IP: MASK: IP: SW10 Manager SW12 SW11 IP: IP: IP: IP: IP: IP:

34 2-Selection 34

35 3-Design 3. Design 3.1. Introduction The design part of the STG covers the operational principles of the different components of high availability architecture. After a short review of the SCADA set up, the following points concerning the PAC station will be detailed: What is a Hot-Standby System (Premium and Quantum) Parts and tools of a Hot-Standby System (Premium and Quantum) Specifications and constraints of a Hot-Standby System (Premium and Quantum) Distributed and In-Rack I/Os (Premium and Quantum) We will also describe the DFBs used in our Hot-Standby library and why they have been developed. The SCADA and Network parts are more detailed in the Configuration chapter 3.2. SCADA System Architecture Presentation The architecture is composed of 2 redundant Vijeo Citect servers, 2 clients and 2 Hot- Standby PACs (Quantum or Premium). The communication between these components is achieved through an Ethernet ring. Each Vijeo Citect server handles I/O, Alarms, Trends and report server functionalities. In our hardware configuration, we choose to install the IO and ATR (Alarm, Trends, and Reports) servers on one computer. The performance of these computers is sufficient in terms of CPU and disk space to handle our application. Only one cluster is configured to manage all servers. 35

36 3-Design 3.3. Premium Hot-Standby System This chapter describes the different features and specifications of a redundant Premium system Premium PAC Specifications Primary and Standby PACs The Primary PAC executes the application program, controls Ethernet network and In-Rack I/Os and synchronizes the Standby PAC at the beginning of each program cycle. The Standby PAC does not run the whole program but only the first section (section 0). Moreover, it does not handle the redundant In-Rack and Ethernet I/Os but just checks the state of the Primary PAC. In case of an anomaly, the Standby PAC takes over the control from the Primary PAC (see switchover time measurements in Performance chapter). Primary and Standby PACs permanently exchange data in order to check the system integrity via the synchronization link. A Premium Hot-Standby system necessarily comprises Monitored ETY modules (one in each rack). These modules handle the diagnosis of Premium CPU redundancy configuration status. This diagnosis is achieved through Sync ETY link. Note: Sync ETY and Synchronization link are different and are not used for the same purpose. 36

37 3-Design Monitored ETY modules As for the CPUs, the position in the rack and the firmware version of the Ethernet modules must be identical. Note: A firmware version 4.0 or earlier is required. The monitored ETY module allows the swap of the Ethernet services as well as the automatic permutation of the IP addresses between Primary and Standby TSX ETY. ETY modules are linked with Ethernet switches (one switch per ETY) or via an Ethernet crossover cable. An optical connection is also possible in the case of a long distance communication. Sync ETY link also allows handling of Ethernet I/O devices with the proper Ethernet I/O Scanning service configuration. In order to initiate a switchover when a Sync ETY link stops operating on the Primary PAC, Ethernet I/O Scanning service must be configured on the monitored ETY module. In addition to the service activation, an I/O Scanning line must also be declared. If the service is not configured in the monitored ETY module, a switchover will not occur if a Sync ETY ceases to operate. In case a monitored ETY module ceases to function, the CPU sends a status modification command to all the configured ETY modules populating the X-Bus and the monitored ETY module populating the Standby PAC to switch their IP addresses. 37

38 3-Design Hardware Constraints The following table lists the only modules that can be used in a Premium Hot-Standby configuration: Power Supply Rack Ethernet Communication All available power supply modules Non-expendable Racks only TSX ETY4103 or TSX ETY5103 (firmware version v4.0 or earlier) - Modbus communication module TSX SCY21601 (firmware version 2.3 or earlier) equipped with multiprotocol communication board TSX SCP114 (firmware version 1.7 or earlier) (slave or master) Modbus Communication Digital I/Os Analog I/Os - Modbus communication module TSX SCY21601 (firmware version 1.1 or earlier) (Master Modbus only) Note: The TSX SCY associated with multiprotocol communication board TSX SCP114 allows the redundant Premium PAC systems to run as Modbus Slave or Master. This configuration allows using Modbus Masters from other suppliers. TSX SCY module can only be used in Modbus Master. No restrictions apply No restrictions apply 38

39 3-Design Software Constraints The following constraints apply at the application level The use of event tasks is not recommended. An event might be lost if it occurs just before or during the switchover. The use of FAST tasks handling dedicated outputs is not recommended as output status modifications might be lost during the switchover. The use of counting modules is not recommended. Following the frequency, some pulses might be lost during the switchover. The use of fronts is not recommended. They might not be accounted during the switchover. The use of the SAVE_PARAM function is not recommended in a CPU redundancy application. This function erases the initial value of a module parameter saved in the program code. This code is not transferred from the Primary PAC to the Standby. More generally, explicit instructions like WRITE_CMD and WRITE_PARAM must be well defined before use. Initial values declared with a recorded attribute (for example DFB variables) can not be replaced with actual values: Do not use%s94 bit. Following inherited functions blocks can not be used: PL7_COUNTER PL7_DRUM PL7_MONOSTABLE PL7_REGISTER_32 PL7_REGISTER_255 PL7_TOF, PL7_TON, PL7_TP PL7_3_TIMER The use of TON, TOFF and TP blocks is not allowed in the first section 39

40 3-Design Premium Hot-Standby DFBs Library The following table summarizes the different DFBs created for our application. DFB FUNCTION HSBY_RD Reading Command word (%SW60) hot-standby system SYSTEM HSBY_WR Writing Command word (%SW60) hot-standby system HSBY_ST Reading Status word (%SW61) hot-standby system ETHERNET ETY_MONITOR Monitoring ETY Ethernet Module SYNTH_FAULT Synthesis Fault monitored elements SYNTHESIS SYNTH_OR_ETY Synthesis Fault ETY module (Logic OR) SYNTH_AND_ETY Synthesis Fault ETY module (Logic AND) SWITCHOVER SWITCH_MANG Switchover Managment System DFBs In order to manage the different registers of a Premium Hot-Standby system, we have created blocks that allow reading and writing registers %SW60 and %SW61. HSBY_RD_P: Read the command register %SW60 HSBY_RD_P PLCA_RUN PLCB_RUN Offline_if_OS_Mismatch BOOL BOOL BOOL Run Mode Controller A Run Mode Controller B OS Versions Mismatch HSBY_WR_P: Write the command register %SW60 Manual Control BOOL Manual_Control_Enable Command Run Mode Controller A BOOL PLCA_RUN Command Run Mode Controller B BOOL PLCB_RUN HSBY_WR_P Forced Command OS no Mismatch BOOL Offline_if_OS_Mismatch This block allows sending switch commands from the program (PLCA_RUN, PLCB_RUN), also, in order to be able to update the CPU OS, the ETY module or the coprocessor, it allows to set the OS mismatch bit to 1 to avoid switching in offline mode. A dedicated input allows sending switch orders for example, during maintenance activities. 40

41 3-Design HSBY_ST_P: Hot-Standby system, status check This block allows to process data from register %SW61. It gives information about each PAC role (Primary, Standby, and Offline), OS version, and so on. HSBY_ST_P HSBY_Active THIS_ISA THIS_ISB THIS_OFF THIS_PRI THIS_Sby REMT_UNDEF REMT_OFF REMT_PRI REMT_SBY LOGIC_OK CPU_SyncLink_OK CPU_OS_OK Copro_OS_OK ETY_minVersion Mon_ETY_OS_OK BOOL BOOL BOOL BOOL BOOL BOOL BOOL BOOL BOOL BOOL BOOL BOOL BOOL BOOL BOOL BOOL Hot-Standby System active This Pac is PAC A This Pac is PAC B This Pac is Offline This Pac is Primary This Pac is Standby Remote state Pac undefined Remote Pac is Offiline Remote Pac is Primary Remote Pac is Standby Identical Logic Pac A et Pac B CPUs synchronized Same CPUs OS Same Copro OS ETY version ok Monitored ETY OS Mismatch 41

42 3-Design Ethernet link monitoring DFBs ETY_Monitor: Ethernet module monitoring ETY_Monitor External default, Ethernet cable unplugged BOOL BLK Fault BOOL Module Fault Module Error BOOL MOD_ERROR Command Run Mode Controller A (T_COM_X103) IODDT COM_ETY5103 COM_ETY5103 IODDT Monitoring Rate value INT Monitoring_Rate Enable BOOL Reading Pulse READ_STS function Pulse computer in the Standby Section BOOL Pulse Monitoring Rate current value INT RateEt RateEt INT The ETY_Monitor DFB monitors the status of the Ethernet link provided by the TSX ETY 5104 (or TSX ETY 4103). We use as inputs the BLK and MOD_ERROR information from IODDT T_GEN_MOD. BLK: external default, Ethernet cable unplugged MOD_ERROR: Module error The IODDT T_GEN_MOD is updated by the READ_STS function. This function reads the status word of a ETY module. The execution rate is controlled by the Monitoring Rate parameter configured by the user (see Chapter 5: Implementation). ETY_Monitor BOOL BLK Fault BOOL BOOL MOD_ERROR IODDT COM_ETY5103 COM_ETY5103 IODDT READ_STS INT Monitoring_Rate Enable BOOL Reading Pulse READ_STS function EN ENO BOOL Pulse %CHx.X.MOD CH INT RateEt RateEt INT The structure of the IODDT T_GEN_MOD is detailed in the table on the next page: 42

43 3-Design ETY3_State T_GEN_MOD + MOD_ERROR BOOL Module error EXCH_STS INT Exchange status STS_IN_PROGR BOOL Status parameter read in progress EXCH_RPT INT Channel report STS_ERR BOOL Error while reading module status MOD_FLT INT Module Faults MOD_FAIL BOOL Internal fault: Module failure CH_FLT BOOL Faulty channel(s) BLK BOOL External fault: Terminal Block CONF_FLT BOOL Hardware or software configuration fault NO_MOD BOOL Module absent or power down EXT_MOD_FLT BOOL FIPIO extension module fault MOD_FAIL_EXT BOOL Internal fault: Module failure (only FIPIO extension) CH_FLT_EXT BOOL Faulty channel(s) (only FIPIO extension) BLK_EXT BOOL External fault: Terminal Block (only FIPIO extension) CONF_FLT_EXT BOOL Hardware or software configuration fault (only FIPIO extension) NO_MOD_EXT BOOL Module absent or power down (only FIPIO extension) The MOD_ERROR bit is set to 1 when an ETY module ceases operation. One frequent cause is the cessation of communication of a device on the I/O Scanning which, in our case, should not initiate a switchover. Therefore, in order to filter this occurrence, we use the T_COM_X103 function to monitor the I/O Scanning status and validate the MOD_ERROR value. When implementing a Hot-Standby system, this block is used once for each ETY module in the configuration. 43

44 3-Design Switchover Management SYNTH_FAULT: Performs the defaults synthesis SYNTH_FAULT Synthesis Fault ETY Module BOOL Faulty_ETY Synthesis Fault SCY Module BOOL Faulty_SCY Synthesis Fault Scada BOOL Faulty_SCADA Fault Mask word WORD Fault_Mask Fault_Synth INT Synthesis Fault Word Fault BOOL OS Versions Mismatch This block aims at processing the faults that would lead to a switchover. We find in input the results of the ETY and SCY modules failure detection. Faulty_SCADA is an input pin in the case of the communication between the SCADA and the PAC is monitored. This DFB also processes: Battery faults %S67 = application memory card battery %S68 = processor battery %S75 = data storage memory card battery CPU fault %S12 = CPU running General In-Rack I/O fault %S119 = fault of one or several I/O modules in the rack Slots 3 to 10 fault %SW160 = operating status of Premium modules installed on station 1 The faults processing is performed using the mask value set on the input pin Fault_Mask. This mask allows to select which fault to take into account according the configuration and to the user s settings. 44

45 3-Design Each fault corresponds to one bit of the Fault_Synthesis word: BIT Element monitored Bit 0 Battery Fault Bit 1 Fault CPU Bit 2 General In-Rack I/O fault Bit 3 Fault on Slot 3 Bit 4 Fault on Slot 4 Bit 5 Fault on Slot 5 Bit 6 Fault on Slot 6 Bit 7 Fault on Slot 7 Bit 8 Fault on Slot 8 Bit 9 Fault on Slot 9 Bit 10 Fault on Slot 10 Bit 11 Ethernet Adapter(s) ETY Fault Bit 12 MODBUS Adapter(s) SCY Fault Bit 13 SCADA Fault The result of this synthesis is saved in a word and set as an output on the Fault_Synth_Plc pin. If there is at least one fault, the output pin Fault is set to 1. During the implementation of the system, this block is used twice: once for the Primary PAC and once for the Standby PAC. In order to be able to compute the status of several ETY modules, logical OR and AND processing DFBs have been created: SYNTH_AND_ETY BOOL FLT_ETY_1 FAULT_ETY BOOL BOOL FLT_ETY_2 BOOL FLT_ETY_3 SYNTH_OR_ETY BOOL FLT_ETY_1 FAULT_ETY BOOL BOOL FLT_ETY_2 BOOL FLT_ETY_3 45

46 3-Design SWITCH_MANAG: Approve or deny a switchover Synthesis Fault word Primary INT PRIM_DIAG Synthesis Fault word Standby INT STBY_DIAG SWITCH_MANAG Switchover Number Reset BOOL SWITCH_NB_Reset SWITCH_NB UNIT Switchover request Manual Switchover BOOL FORCE FORCE BOOL Manual Switchover The Switch_Manag DFB manages and counts switchover queries. The switchover approval is computed from the Primary and Standby PACs diagnosis coming from the Fault_Synthesis DFBs as seen above. A switchover is allowed if: The Standby PAC diagnosis is OK. More than 30s have elapsed since the previous switchover. Note: The time delay before the switchover takes place can be adjusted using variables of the DFB (Delay_Time_Before_Switchover). This delay is set to 1s by default. The switchover counter can be reset using the input pin Switch_N_Reset. For maintenance reasons, the input pin FORCE allows a manual switchover of the system. During the implementation, this block is used only once. Switchover Time Switch_Over_Time Remote Pac is Primary BOOL Remote_is_Primary Sw_Timer TIME Switchover Time This Pac is Primary BOOL This_is_Prima The time gap during the switchover is a very important feature of the Hot-Standby system. A DFB has been defined to measure this time. The principle is based on the measurement of the time when the Primary PAC loses its Primary status and when the Standby turns Primary. This block, placed in the section 0, processes the system word %SW61 information and uses the ITCNTRL block function which allows event time measurements. The accuracy of the switchover time depends on the PAC scan time, for more accuracy, other measurement can be performed as described in the performance chapter. 46

47 3-Design In-Rack I/O System This paragraph describes the management of the I/Os populating the main rack. Inputs acquisition is performed locally by both Primary and Standby PACs, whereas the Primary PAC outputs are mirrored on the Standby PAC (provided that there is no specific action programmed in the section 0). Redundant Digital I/Os Implementation Digital input and output signals are connected to the PAC through an ABE7 connection block. These signals are multiplexed/de-multiplexed by a Telefast connection device as seen on the above diagram (ABE7 ACC11 for the inputs and ABE7 ACC10 for the outputs). Exceptions detected on Digital inputs cannot initiate a switchover. The digital I/Os implementation is illustrated on the diagram below. Digital Outputs in the section 0 As the Standby PAC executes the first section (section 0) of the application program and then applies the object image %Q received from the Primary PAC, it is important not to modify the redundant output status in this section. A modification of the output bits in the section 0 can lead to an inconsistent status of the outputs as they are modified twice in the same MAST task. 47

48 3-Design Digital Outputs fall-back mode In general, the outputs fall-back mode must be similar to their current mode in order to avoid an operation discrepancy during the switchover. Pulse Triggered Actuators Digital output redundancy implies a distortion of the command signal. Output modules are connected in parallel of the physical output, via a connection block. The result of a command is based on the length of the pulse and the delay after which the pulse is applied on the Standby PAC. These mechanisms have to be taken into account in order to handle In-Rack digital output redundancy. Positive pulse trigger As shown on the above diagram, the length of the output is longer than the Pulse Time. This does not have any impact on the device behavior. In the case where the delay is greater than the pulse length and using an actuator with a low response time, the signal received by the actuator might be composed of 2 commands, as shown on the diagram below: 48

49 3-Design Negative pulse trigger As shown on the above diagram, the length of the output is shorter than the Pulse Time. This does not have any impact on the device behavior unless it cannot handle a shorter command. The following diagram presents the case where the delay is greater than the pulse time of the output signal: Because the delay is greater than the Pulse Time, the device will not receive any command. 49

50 3-Design Analog Inputs implementation Analog signals are connected to the PACs through a signal duplicator. For our application, we use a JMConcept TELIS9000U2 module (which replaces reference JK3000N2) The table below describes the signal range handled by the TELIS9000U2: INPUTS Current (continuous) Voltage (continuous) Probe Thermocouple Potentiometer Resistance Sensor Power Supply Standard scales 0/1mA; 0/10mA; 4/20mA; +/-1mA; +/-10mA; +/-20mA User defined scales from -22mA to 22mA Standard scales 0/100mV; 0/1V; 0/5V; 1/5V; 0/10V; 2/10V; 2/10V; 0/50V 0/100V; 0/200V User defined scales from -110mV to 110mV; from 2V to 11V; from -200V to 220V PT100; PT1000 Ni100; Ni1000 J, K, R, S, T, E, B, N, W3, W5, NiMo from 100Ω to 100kΩ 0/200Ω; 0/1kΩ; 0/10kΩ; 2 or 3 wires, 24V - 29mA max OUTPUTS Output 1 Current Output 1 Voltage Output 2 Current Output 2 Voltage Digital Output Relay Output 0/20mA; 4/20mA; from 0 to 20mA 0/10V; +/-10V - from 0 to 10V 0/20mA; 4/20mA; from 0 to 20mA 0/10V; +/-10V - from 0 to 10V USB connector in front panel RS485 Modbus Jbus isolated from input and output 1 Relay: 1RT; 2RT; 3RT; 4T; 1RT & 1T As seen on the next figure the signal from the process is duplicated and wired on both PACs thanks to the JMConcept module. 50

51 3-Design Analog Input Analog Outputs implementation The implementation is handled by a low-level commutation interface, in our case a JMConcept GK3000D1 module. The principle is to select the output coming from the Primary PAC. This selection is performed by 2 relays controlled by a PAC digital output. The management of the relays can be performed either by 1 non-redundant output or by 2 redundant outputs to increase reliability. In our architecture, we choose to manage the relays with 2 redundant digital outputs. The following table describes the GK3000D1 interface relays logical operation: Relay A Relay B Output Channel Analog Input 1 Analog Input 2 last correct channel Analog Input 1 Analog Input 2 4/20 ma 4/20 ma INPUTS Digital Input 1 Digital Input 2 Analog Output Digital Output on optocoupler 30V max on optocoupler 30V max OUTPUTS 4/20 ma RS 485 isolated from input Modbus, Jbus Digital link allows programing of the module Digital link allows acquisition of the measurements 51

52 3-Design Analog Outputs controlled by non-redundant Digital Outputs The 2 PACs analog outputs are connected to a GK3000 D1 module. This module, controlled by digital signals, routes to its output, using 2 relays, one of its 2 inputs. In our case, the digital signals that control the GK3000D1 module are 2 digital outputs of the PACs. The main benefit of this solution is that it only uses 1 digital output to route analog outputs. A wiring example is presented in the diagram below: Analog Output Digital Output The Primary PAC must set to 1 the digital output that controls the relay in order to route its own analog output to the output channel of the commutation interface. The Standby PAC then sets to 0 the digital output that controls the other relay. Note: This logical operating principle must be coded in section 0. The fall-back mode of the digital output module must be set to 0. For example, according to the diagram above, if PAC_A is the Primary PAC, the digital output connected on the relay A is set to 1 while PAC_B digital output connected to relay B is set to 0. This leads to route the analog signal A on the output of the communication interface. Note: This solution is not used in our architecture. 52

53 3-Design Analog Outputs controlled by redundant Digital Outputs Analog Output Digital Output An analog output redundancy is performed, thanks to a GK3000D1 communication interface and redundant digital outputs, using 2 digital outputs per PAC. Each relay of the GK3000D1 interface is connected to a digital output. If PAC A is Primary PAC B is Primary Then Digital output number 0 is set to 1 (relay A) Digital output number 1 is set to 0 (relay B) Digital output number 0 is set to 0 (relay A) Digital output number 1 is set to 1 (relay B) Note: This logical operating principle must not be coded in the first section (section 0). For example, according to the diagram above, if PAC_A is the Primary PAC, the digital output No 0 (relay A) is set to 1 and the digital output No 1 (relay B) is set to 0. Thus, the analog signal ANA_ A is routed to the output of the communication interface. Note: This solution is used in our architecture. 53