OM650 - HMI part of the T2000 system (Course module 100) T2000 System Configuration and Diagnostic tools (Course module 200)

Transcription

1 Course structure This course is starting with brief explanation of the principles of the system for each component and sub-component and later it explains in great detail how to maintain and repair given component should it be required. The components addressed at this course are listed below OM650 - HMI part of the T2000 system (Course module 100) PU - Processing Units SU - Server Units OT - Operator Terminal HMI Server OPC - OPC servers T2000 System Configuration and Diagnostic tools (Course module 200) ES Engineering Station DS Diagnostic System T2000 System Communication infrastructure and Synchronisation (Course module 300) GPS Clock - synchronisation of DCS system components Networking Plant Bus Terminal Bus Bridge between power plant units Other Networks Profibus network and special components like IM 153 module, Y-link etc. TCP/IP networks Automation and Field Instrumentation layer (Course module 400) AP - Automation Processor I/O interfaces like SIM modules Communication interfaces like profibus master or CP1430 cards Diagnostic and other essential hardware 1

2 2 AP-T - Turbine Governor (Course module 400) I/O interfaces Communication interfaces Diagnostic and other essential hardware CM - Communication Module 95F - Failsafe protection I/O interfaces Communication interfaces Diagnostic and other essential hardware S Siemens S7 PLC used at boiler protection Auxiliary systems (Course module 500) WIN_TS - Windows Turbine Stress Monitoring System TDY - Argus - Gas Turbine humming monitoring system PAS servers VM turbines, pump and generators vibration monitoring system Fight recorder Tec4system...? Service and maintenance systems (Course module 600) PG - Siemens computer system used for advanced diagnostic and configuration Laptops - Other computers used for various purposes

3 Contents 1 Course planning Day 1 - OM Day 2 - Engineering and Diagnostic Day 3 - Automation layer Day 4 - Dedicated controllers Day 5 - Auxiliary systems Day 6 - Hands on, Questions and Answers I Siemens T2000 system principles 9 2 General T2000 architecture Overview OM 650 process control and information system AS 620 Automation System ES 680 engineering system Network communication DS 670 diagnostic system II Component Based Training 15 3 OM650 - T2000 HMI subsystem OM System Infrastructure InfOmk.proj InfFb.proj InfDevInst.proj InfObm.inst License files - licensing OM 650 files on ES C101 - PU - Processing Unit Functional principles and component basic configuration Component Diagnostic and Routine maintenance Routine maintenance Diagnostic files OmProj.Check rdb Component Backup and Recovery principles

4 CONTENTS Backup Procedure Restore Procedure Component Installation and Commissioning References C102 - SU - Server Unit Functional principles and component basic configuration Component Diagnostic and Routine maintenance Routine maintenance Diagnostic files Resetting and analyzing arc subsystem Component Backup and Recovery principles Backup Procedure Restore Procedure Component Installation and Commissioning References C103 - HMI Server - web4txp Functional principles and component basic configuration Component Diagnostic and Routine maintenance Routine maintenance Diagnostic files Component Backup and Recovery principles Backup Procedure Restore Procedure Web4TXP system licensing Component Installation and Commissioning References References C201 - ES Engineering Station Functional principles and component basic configuration ES 680 User interface ES 680 component file structure Component Diagnostic and Routine maintenance Routine maintenance Component Backup and Recovery principles Disk Backup Procedure Database restore procedure Component Installation and Commissioning References C202 - DS Diagnostic station Functional principles and component basic configuration Starting and stopping DS Daily use of DS Component Diagnostic and Routine maintenance Routine maintenance Component Backup and Recovery principles Backup Backup Procedure Restore

5 CONTENTS Restore Procedure DS 670 system licensing Component Installation and Commissioning References C300 - OSM - Managed network switch References C400 - AP Automation Processor Functional principles and component basic configuration Maintenance Component Backup and Recovery principles Components Installation and Commissioning - Case study Burn New IM308 Flash Card General Info: When IM608 card should be programmed What You Need: Get IM308 Data From ES680: Burn The New Flash Card: Problem/Solutions: References C450 - CM - Communication module Functional principles and component basic configuration Maintenance Component Backup and Recovery principles References C490-95F - protection system On-the-Job training Protection system and TXP communication Control and protection systems brief description Gas Turbine governor (SIMADYN) DCS control system (TXP680) Protection system (95F) Startup Frequency Controller and Static excitation unit (SFC) Unit protection OM system (OM650) Systems interconnection Distribution of the systems over cabinets Protection system description in detail Hardware based over-speed protection Over-speed system diagnostic O/S system REAL test Software based over-speed protection Flame monitoring Manual turbine trip Fire protection trips Surge protection

6 CONTENTS Unit protection Protection circuit (protection signals coming from TXP system) Protection system redundancy and reliability Non-Coincidence alarms System auto-diagnostic Over-speed protection diagnostic sub-system Trip circuit diagnostic ET200 subsystem health monitoring Units synchronization check F self-diagnostic sub-system CPU (95F) is designed as two independent high-reliability redundant unit Trouble shooting Start-up problems Problems during running machine Installing 95F software on the PG How to install 95F software on new PG F testing mode Download code to 95F in testing mode F overall reset F faults rectifying Rectify MYB00EU111B fault Diagnostic mode after 95F restart References III Cybersecurity 133 IV Questions to be answered 135

7 Chapter 1 Course planning The course duration is five and half days long. Below is described daily planning of the course. 1.1 Day 1 - OM650 The course is starting with the HMI system (course module 100) - the most visible part of the system - everyone sees it. First principles of the TXP / T2000 DCS system are explained followed with the details about daily maintenance tasks, disaster recovery planning and finally detailed explanation of the system. 1.2 Day 2 - Engineering and Diagnostic Day number two will start with summarizing of functions of the OM650 and will continue to explain engineering tool ES 680 (course module 200) in relation to the OM650 system. This will be followed by the explanation of the networking (module 300) it s configuration and by explanation of the time synchronisation. The details about daily maintenance tasks of the ES 680 will be followed by disaster recovery planning. On the end of day DS 670 diagnostic system (course module 350) and it s daily use will be addressed. Since disaster recovery plan for DS 670 is identical to the OM650 components no additional material is required. 1.3 Day 3 - Automation layer Day number three will be dedicated to automation layer specially to AP (Automation Processor - module 400) it s structure, configuration and maintenance followed by configuration tasks like adding signals to the I/O system. This day training will be broken into three sections which will be held on different locations as well. Location: Training center (case study adding analog output signal - engineering and all related tasks) and Networking 7

8 1.4. DAY 4 - DEDICATED CONTROLLERS 8 Location: PCC (Physical implementation and examples, PCC hardware explanation, CM module backup) Location: MCR - Main Control Room (Diagnostics and practical implementation of case study) 1.4 Day 4 - Dedicated controllers On day four detailed explanation of the CM module (Module 450), principles of protection systems like overspeed system, vibration monitoring system, Boiler S7 protection system, 95F (course module 490) will be addressed. The functional principles and maintenance manuals are being discussed here. This day training will be broken into three sections which will be held on different locations as well. Location: Training center (Theory of protection systems) Location: PCC (CM module backup, hardware explanation) 1.5 Day 5 - Auxiliary systems After the initial test references will be discussed and reviewed. Analysis and discussion of disaster recovery planning will be addressed. Auxiliary systems like WIN_TS, Argus, VM 600 are being explained (course module 500). The maintenance point of view is being in focus on the documentation, manuals and disaster recovery planning. The functional blocks explained: Analog and binary inputs for physical hardware and signals sent over communication Controllers and setpoint calculations Generic blocks as a timers, switches etc. Brief explanation of the YOR diagrams The core I&C system explained as per reference Core I&C sagunto OPC servers configuration and hands on practice of the update Remaining AP-T - Turbine Governors and Excitation system to be studied and will be explained throught the future visit. 1.6 Day 6 - Hands on, Questions and Answers The half of the day number six is being dedicated to the training on the ES 680 station (module 200) and training on the OM components backup (module number 100). The hands on will be carried in CCR divided into two groups so both teams can try both tasks.

9 Part I Siemens T2000 system principles 9

10 10 Objective of this section is to introduce components and subcomponents of the Siemens T2000 system, briefly describe their function and establish basis for further detailed training.

11 Chapter 2 General T2000 architecture 2.1 Overview The Siemens T2000 DCS control system provides all I&C facilities that are necessary for automating, handling, monitoring, and archiving processes specifically for power plants. The tasks of the T2000 DCS control system are distributed to different subsystems as shown below. OM 650 Operating and Monitoring system. This is the process control and information system for operator-process communication and visualization. This is the system used for process automa- AS 620 Automation System. tion. ES 680 Engineering System. This system is employed for configuration and commissioning. This is a place where configuration database is located Network Communication. Plant bus - individual AP s (Automation Processors) communicater with each other via this bus and with OM system Terminal Bus - The HMI system is connected here DS 670 Diagnostic System. diagnostics. Optional component for detailed system 11

12 2.1. OVERVIEW 12 Figure 2.1.1: Main T2000 components are shown on this figure OM 650 process control and information system The OM 650 process control and information system is the interface between the system and the operator in the control room. This is also called HMI - Human Machine Interface. This system enables the process to be centrally monitored and controlled. In addition, the system provides all functions that are required for logging the process and for archiving the data. The HMI system is consisted of the following computers: PU Unit - connection with AP s, process calculations like a running hours SU Unit - long term archiving, description database, reports processing HMI server - server which generate Human Machine Interface TC - Thin Client - browser based client to display HMI interface For propper function each computer in HMI system requires unique name not only within one unit but in complete poweplant. Figure 2.1.2: Sugen computer naming concept

13 2.1. OVERVIEW AS 620 Automation System. The AS 620 subsystem performs the automation tasks of the plant processes. The AS 620 acquires measured values and states from the process, performs open and closed-loop control functions, and transfers the resulting manipulated variable values, correction values, and commands to the process. The other subsystems employ the AS 620 subsystem as the interface to the process. The AS 620 transfers the commands from the OM 650 operator communication and visualization system to the process. There are vatious versions of AS 620 depending on process needs AS 620B for basic automation tasks which could be onfigured as FUM or SIM variant AS 620 T - Turbine controller, high speed control ES 680 engineering system The ES 680 engineering system is the central configuration system of TXP. ES 680 is used for configuring: the AS 620 automation system, the OM 650 process control and information system, the SINEC H1 FO bus system, and the necessary hardware. ES 680 provides a configuration package for each target system. ES 680 centrally administers all configuration data, which means data is entered only once. The configuration of the AS functions and processing functions in OM 650 are based on control system flow charts. A control system flowchart editor (known as FUP editor) in the ES 680 permits interactive entry of these control system flow charts. The configuration principle of the ES 680 is based on consistent forward configuration. Initial configuration and modifications are exclusively performed through the configuration system with subsequent automatic code generation. This guarantees real time documentation of the system hardware and all AS, OM, and SINEC functions, and permits modifications to be centrally controlled.

14 2.1. OVERVIEW Network communication There are two principan networks at the T2000 system. Plant bus The network structure of the SINEC H1 bus system enables communication between the individual sub-systems of TXP. It mainly allows communication between AP s (Automation Processors) and OM 650 system. Special hardware (networking cards ) is required to allow this kind of communication. Terminal bus It s a standard TCP/IP network where each node has a unique IP address and node name. Other network communication If there is a need to establish communication with other systems there are possibilities of using OPC protocol running on PU (Processing Unit) Various RTU, Modbus, IEEE protocols running on CM (Comminication Module) The networks are designed to be one fault tolerant. This mean that failure of any of one component will not cause breakdown of entire network. The network is configured as a ring. Figure 2.1.3: The plant bus and terminal bus are configure as a ring DS 670 diagnostic system The optional DS 670 diagnostic system is the tool that is used for monitoring and detecting malfunctions in the I&C components of TXP.

15 Part II Component Based Training 15

16 Chapter 3 OM650 - T2000 HMI subsystem The OM 650 process control and information system is the interface between the system and the operator in the control room. This is also called HMI - Human Machine Interface. This system enables the process to be centrally monitored and controlled. In addition, the system provides all functions that are required for logging the process and for archiving the data. The HMI system is consisted of the following computers: PU Unit - connection with AP s, process calculations like a running hours SU Unit - long term archiving, description database, reports processing HMI server - server which generate Human Machine Interface TC - Thin Client - browser based client to display HMI interface For proper function each computer in HMI system requires unique name not only within one unit but in complete power plant. Figure 3.0.1: Sugen computer naming concept 16

17 3.1. OM SYSTEM INFRASTRUCTURE OM System Infrastructure Every OM component is part of the concept called OM infrastructure. The infrastructure is defined via means of the files described in following text. The infrastructure can be seen as shown on the figure below. Figure 3.1.1: The infrastructure of the OM system distributed over several components For the configuration of the distributed OM system infrastructure (INF) needs to be defined. Infrastructure is defined in following files: InfOmk.proj InfFb.proj InfDevInst.proj InfObm.inst All these files are located in the directory $OmProjData/inf

18 3.1. OM SYSTEM INFRASTRUCTURE InfOmk.proj In the file InfOmk.proj all OM components are listed and an internal unique component number is assigned to each component. The LTK-No. must always be negative, the LTK-Inst is the same number but positive, and the OMK-Inst is equal to LTK-Inst plus 1. These numbers are defined at topology diagram on ES 680 engineering station. ES 680 is generating these files automatically. This file must be identical on all OM components Algorithm 3.1 Content of the InfOmk.proj file at Sugen PU unit # InfOmk.proj # ICC-No = negative number in steps of 100 # RedPrio = redundant priority. 1 after runup leading / 2 after runup standby # OMC-Inst = instance number of OM component # ICC-Inst = instance number of superset I&C component # Hostname.Domain ICC-No RedPrio OMC-Inst ICC-Inst s30ot1.txp.om650.scn s30ot2.txp.om650.scn s30p1a.txp.om650.scn s30p1b.txp.om650.scn s30s1a.txp.om650.scn s30s1b.txp.om650.scn

19 3.1. OM SYSTEM INFRASTRUCTURE InfFb.proj In the file InfFb.proj every function area from a plant is assigned to exactly one redundant PU computer set. In bellow displayed file can be seen for example that functional area 207 is being processed as follows: MAC - process calculations by subcomponent 300 which is defined in the file InfOmk.proj as a computer s30p1a and s30p1b ASR - communication towards to AP s by subcomponent 300 which is defined in the file InfOmk.proj as a computer s30p1a and s30p1b ARC - short term archive by subcomponent 300 which is defined in the file InfOmk.proj as a computer s30p1a and s30p1b BDM - database of descriptions by subcomponent 400 which is defined in the file InfOmk.proj as a computer s30s1a and s30s1b LZA - long term archive by subcomponent 400 which is defined in the file InfOmk.proj as a computer s30s1a and s30s1b PRT - printing and searching archives by subcomponent 400 which is defined in the file InfOmk.proj as a computer s30s1a and s30s1b NTB - notebook functions by subcomponent 400 which is defined in the file InfOmk.proj as a computer s30s1a and s30s1b This file must be identical on all OM components. Algorithm 3.2 Content of the InfFb file at Sugen PU unit 216 MAC=300 ARC=300 BDM=400 LZA=400 PRT=400 NTB= MAC=300 ASR=300 ARC=300 BDM=400 LZA=400 PRT=400 NTB= MAC=300 ASR=300 ARC=300 BDM=400 LZA=400 PRT=400 NTB= MAC=300 ASR=300 ARC=300 BDM=400 LZA=400 PRT=400 NTB= MAC=300 ASR=300 ARC=300 BDM=400 LZA=400 PRT=400 NTB= MAC=300 ASR=300 ARC=300 BDM=400 LZA=400 PRT=400 NTB= MAC=300 ASR=300 ARC=300 BDM=400 LZA=400 PRT=400 NTB=400

20 3.1. OM SYSTEM INFRASTRUCTURE InfDevInst.proj The file InfDevInst.proj is needed for device monitoring. File structure: one line for each device 1. Column: OMK-Instance number from the corresponding OM computer 2. Column: The instance number of the device (this number is freely selectable, but it needs to be unique within the OM FC) 3. Column: Reference for the device (20550 = printer, = MOD drive) 4. Column: Device name (like the name present in /dev) This file must be identical on all OM components. Algorithm 3.3 Content of the InfDevInst.proj file at Sugen PU unit UPS UPS UPS UPS UPS UPS MOD MOD HPLJ HPLJ s00pr s00pr s00pr s00pr s00pr0

21 3.1. OM SYSTEM INFRASTRUCTURE InfObm.inst The file InfObm.inst defines which OM subcomponents will run of given computer. For details refer to specific OM component License files - licensing This is single most problematic part of the T2000 system. It is very poorly described however errors or misplacement of the licensing files may cause significant problems with the system and are very difficult to find out. The OM related license file is called OPT.conf and has to be located at $OmProjData/inf. The license is issued by the OEM and is related to the MAC address of the main-board installed at the computer. Therefore if computer is being changed new license is required or specific work around has to be applied. Figure 3.1.2: Similar error might be found in error log files when there is an issue with the OM license

22 3.1. OM SYSTEM INFRASTRUCTURE OM 650 files on ES680 All discussed files are available on ES 680 from which they can be transferred to the OM components if needed to be so. Detailed manual can be seen in the reference directory called OM650 transfer and engineering. The following OM system parameterization files can be automatically generated from the topology diagram: InfOmk.proj InfFb.proj Arc.proj Lza.proj Asr.proj bpr_ltk.dat (without the entries for devices such as printers and MODs) The following files cannot be automatically generated in the current version and still require manual editing: InfDevInst.proj Ot.hrn Ot.amu The file for the OM I&C components bpr_ltk.dat has to be edited for the device entries. The log file is stored in the directory $TXP_HOME/data/$Proj/om/inf/log under the following name: OMSYS_GEN_<date>_<time> Figure 3.1.3: Interface for the OM configuration files generation and transfer

23 3.2. C101 - PU - PROCESSING UNIT C101 - PU - Processing Unit This chapter is addressing OM 650 component called Processing Unit shortly known under name PU. The pair of PUs in installed at each unit at power-plant. It is addressing it from maintenance point of view. Following is being addressed: Functional principles and component basic configuration Component Diagnostic and Routine maintenance Component Backup and Recovery principles Component Installation and Commissioning The processing unit is main interface between operator and automation level. It s main tasks are as follows: Communication with AP s (Automation Processors) - Object manager called ASR shown on figure below updating image of the process values sending commands given by the operator to the AP Providing short term archive - Object manager called ARC shown on figure below Providing various calculations like hours or energy counters - Object manager called MAC shown on figure below Figure 3.2.1: OM system status

24 3.2. C101 - PU - PROCESSING UNIT Functional principles and component basic configuration PU component file structure The installation of OM 650 system is located in directories described below / > this is Unix root of the file system /txpsys The executable programs of the OM software are located in this directory or file system There is defined system variable called $OmSys /txpsys/txpconf In this directory generic TXP configuration is located There is defined system variable called $OmConfData /txpproj/proj_std In the txpproj directory or file system the engineering data are stored (as for example the pictures for the MMI or the userspecific network configuration data from ASR) There is defined system variable called $OmProjData /txptest in the directory txptest the diagnostic files are located (error log files), which are created by every OM subcomponent There is defined system variable called $OmDiagData /txpproz The process data (hard copies, notes, calculations) are stored in the directory txpproz There is defined system variable called $OmProzData Each file system or directory mentioned above has a function-related organization which is depending on the object managers installed on given OM component. Figure 3.2.2: File structure of typical OM650 component The directory structure of PU computer is as follows.

25 3.2. C101 - PU - PROCESSING UNIT 25 /root /txpsys /inf /asr /arc /mac /swi /txpproj/proj_std /inf /asr /arc /mac /txptest /inf /asr /arc /mac /txpproz /inf /asr /arc /mac

26 3.2. C101 - PU - PROCESSING UNIT 26 PU component configuration files Infrastructure - INF For the configuration of the distributed OM system infrastructure (INF) needs to be defined. Infrastructure is defined in following files: InfOmk.proj InfFb.proj InfDevInst.proj InfObm.inst All these files are located in the directory $OmProjData/inf. Detailed description of infrastructure files can be found on page 17. InfObm.inst The file InfObm.inst defines which OM subcomponents will run of given computer. This file is specific for each OM component. Algorithm 3.4 Content of the InfObm.inst file at Sugen PU unit #ObjMgr DirName ASR asr ARC arc MAC mac

27 3.2. C101 - PU - PROCESSING UNIT 27 ASR ASR module configuration files defines configuration of the communication between OM and AP systems. This file is located in the directory $OmProjData/asr Asr.proj For the ASR module only one engineering file exists: the Asr.proj. This file is generated on the ES.

28 3.2. C101 - PU - PROCESSING UNIT 28 Algorithm 3.5 Content of the Asr.proj file at Sugen PU unit # Asr. proj. s30p1a # ASR mode i s normal operation IBS 0 # Information fo r AS 341 IKZ_GK 341 IKZ_FB 207 IKZ_IN 1 AS_ueberwacht 1 AS_FB 207 end # Information fo r AS 342 IKZ_GK 342 IKZ_FB 207 IKZ_IN 42 AS_ueberwacht 1 AS_FB 207 end # Information fo r AS 373 IKZ_GK 373 IKZ_FB 208 IKZ_IN 2 AS_ueberwacht 1 AS_FB 208 end # Information fo r AS 375 IKZ_GK 375 IKZ_FB 208 IKZ_IN 15 AS_ueberwacht 1 AS_FB 208 end # Information fo r AS 351 IKZ_GK 351 IKZ_FB 210 IKZ_IN 462 AS_ueberwacht 1 AS_FB 210 end # Information fo r AS 392 IKZ_GK 392 IKZ_FB 210 IKZ_IN 1 AS_ueberwacht 1 AS_FB 210 AS_FB 213 AS_FB 215 end # Information fo r AS 311 IKZ_GK 311 IKZ_FB 213 IKZ_IN 2 AS_ueberwacht 1 AS_FB 213 AS_FB 217 end # Information fo r AS 312 IKZ_GK 312 IKZ_FB 213 IKZ_IN 78 AS_ueberwacht 1 AS_FB 213 end # Information fo r AS 313 IKZ_GK 313 IKZ_FB 213 IKZ_IN 125 AS_ueberwacht 1 AS_FB 213 end # Information fo r AS 393 IKZ_GK 393 IKZ_FB 217 IKZ_IN 1 AS_ueberwacht 1 AS_FB 215 AS_FB 217 end

29 3.2. C101 - PU - PROCESSING UNIT Component Diagnostic and Routine maintenance When PU component starts failing proper diagnosis is required. Following some most commons methods employed to detect which part of the system is failing to decide on proper corrective measures Routine maintenance The routine maintenance of PU component is at Sugen described by the procedure F36-MI-02 [902] Evaluation of the computer is in detail described in reference [001] PL -t Check if all OM components are in operation All components must be if fue status or akt for hot standby Time must be synchronized Pool /etc/dfspace Figure 3.2.3: OM system status Check for available disk space on all OM components Figure 3.2.4: PU unit output of the /etc/dfspace command

30 3.2. C101 - PU - PROCESSING UNIT 30 Poll tail -7 /usr/lib/powerchute/*log Check for UPS / Battery failure - check for error messages OmProj.Check This command is not required for daily use - only after OM reconfigaration and after updates Verify OM configuration - check for any errors reported Evaluation of the file /txp/om650/txptest/diagmeld* and subsequent errors if any Diagnostic files are explained in detail below Rule of thumb say - if both of diagnostic files have very different dates and error messages their content are very seldom system seems to be in order. /var/adm This is a location system log files for which root access might be required verify system log files - syspages - see attachment No. 3 of the reference [001] SAMPLER If any problems detected run SAMPLER - it will gather all log files - MUST be done prior to any modifications!!!! It keeps history Transfer SAMPLER to the safe location

31 3.2. C101 - PU - PROCESSING UNIT Diagnostic files Each object manager writes error log files. In these files errors and messages are stored as a report. All diagnostic files are located in the path /txptest. The infrastructure module writes directly into directory /txptest. Every other object manager writes into its respective subdirectory. For example the ASR writes into /txptest/asr. The diagnostic files are realised via a cyclic buffer. First all diagnostic messages are written into file DiagMld.0. If this file reaches a size greater than 1 MB, then the file DiagMld.1 is created. If this file also becomes greater than 1 MB, then DiagMld.0 is deleted and created again. Thanks to this mechanism, the hard disk will never be full and the latest diagnostic data are always available. All diagnostic files are always built with a two line set. The date and time of the message are written in the first line, as well as the program name, process id, module name, module version, line number and function name. On the second line we can find the error message number and the error message description. Unfortunately these diagnostic files have been written by developers and for other developers. Therefore they are often very difficult to understand. To see content of the generic diagnostic file follow these steps. Login as txpom user Navigate to the location of the log files by commend cd $OmTestData Check with the directory listing which diagnostic file is newer ls -la Dia* Use tail command to see last, let s say 30 lines of the file DiagMld.0 tail -30 D*.0 Should find that some object manager is reporting many errors, lets say asr than check diagnostic files of asr manager as follows. Login as txpom user Navigate to the location of the log files by commend cd $OmTestData cd asr

32 3.2. C101 - PU - PROCESSING UNIT 32 Check with the directory listing which diagnostic file is newer ls -la Dia* Use tail command to see last, let s say 30 lines of the file DiagMld.1 tail -30 D*.1 All unix commands mentioned in this section are explaned at Bohemia Market knowledge base at OmProj.Check This is a tool to verify OM configuration files.

33 3.2. C101 - PU - PROCESSING UNIT rdb The RDB is a tool to diagnose connection between AP and PU computer. To verify communication type following commands on the PU computer. login as a tpom user type command rdb On menu select Function No. 3 Answer amount of the seconds youwant to see update - lets choose every 3 seconds Answer by pressing number 0 to the question if storage to file is required Quit by pressing DEL key This command is as usual undocumented feature of OM detailed explanation can be found at Bohemia Market knowledge base at

34 3.2. C101 - PU - PROCESSING UNIT Component Backup and Recovery principles The backup method used for processing computer is using OEM developed script called Txp.Backup for backup purposes and Txp.Restore for restoration of the component. This is fairly reliable procedure however should be noted that this is logical type of the backup. Moreover!! this tools do not backup process related data!! There is a procedure at Sugen MIG654 [954] addressing this task. Backup Equipment required: External DDS2 or DDS3 DAT tape drive VGA monitor PS2 Keyboard Ultra wide SCSI cable with connector capable of connecting to the Adaptec Ultra Wide SCSI controller installed in the Celsius... computers. A new DDS2 DAT tape. The 1.44Mb 3.5 boot and root floppy disks see how to create disk at... if you do not have one

35 3.2. C101 - PU - PROCESSING UNIT Backup Procedure 1. Verification (a) If a redundant component is to be backed up, ensure the redundant component is in error free operation. (b) If a web4txp server is being backed up, ensure all web users have been disconnected from that component. 2. Stopping OM system (a) Login as txpom (b) enter the Om.Stop command. (c) When the OM process has stopped, logout the txpom user. 3. Shutdown computer (a) Login as root user (b) Shut down the OM650 system by entering the command init 0. (c) When the system is completely shut down and is displaying the prompt It is safe to power off, then turn the power off to the system. 4. Preparing Tape Drive (a) With the power off on the DAT drive and the power off on the OM650 component, connect the DAT to the UWSCSI adapter using the UWSCSI cable. (b) Set the SCSI id to 2 on the DAT drive. (c) Power on the DAT drive. (d) Insert properly labeled DDS2 DAT tape in the DAT drive, ensure the write protect tab is closed and wait till the DAT tape drive light stops flashing.

36 3.2. C101 - PU - PROCESSING UNIT Booting the computer (a) Clean the floppy drive by blow of air through it - otherwise floppy disk may be damaged or not work. Over years there is lot of dust in floppy drive. (b) Insert the boot floppy disk in to the floppy drive on the OM650 component. (c) Turn on the OM650 component. (d) When the boot prompt appears on the screen, press enter key once. (e) When prompted, eject the boot floppy disk, insert the root floppy disk in the floppy drive and press the enter key. 6. Performing backup (a) At the # prompt, type the command Txp.Backup. (b) On completion of the restore procedure, ensure that there are no error messages on the screen indicating bad sectors, bad tape etc. The backup takes about 15 to 20 minutes (c) It is advisable to see the content of the tape to make sure data were written to it. Since tape archive is created by cpio command it is easy to see it s content by executing command cpio -ivt -I /dev/rstp0 - more on this command can be found in Bohemia Market knowledge base under this link php?title=cpio (d) Eject the DAT tape. 7. Powering computer back to normal operation (a) Type the command haltsys. (b) Turn the power off to the Celsius computer and the DAT tape drive. (c) Disconnect the UWSCSI cable from the computer. (d) Apply power to the OM650 component and observe its successful boot, OM start and synchronization.

37 3.2. C101 - PU - PROCESSING UNIT 37 Restore Following a system restore, it is necessary to transfer the latest generated code to the component before activating it within the operating infrastructure. In most cases, the OM process automatically starts, so a special procedure has to be put in place to ensure the newly restored component is isolated until it has the latest code transferred. Equipment required: External DDS2 or DDS3 DAT tape drive VGA monitor PS2 Keyboard Ultra wide SCSI cable with connector capable of connecting to the Adaptec Ultra Wide SCSI controller installed in the Celsius... computers. A new DDS2 DAT tape. The 1.44Mb 3.5 boot and root floppy disks see how to create disk at... if you do not have one

38 3.2. C101 - PU - PROCESSING UNIT Restore Procedure 1. Shutdown computer and power it off 2. Preparing Tape Drive (a) With the power off on the DAT drive and the power off on the OM650 component, connect the DAT to the UWSCSI adapter using the UWSCSI cable. (b) Set the SCSI id to 2 on the DAT drive. (c) Power on the DAT drive. (d) Insert latest backup tape, ensure the write protect tab is open to avoid accidental deletion of the backup tape and wait till the DAT tape drive light stops flashing. 3. Booting the computer (a) Clean the floppy drive by blow of air through it - otherwise floppy disk may be damaged or not work. Over years there is lot of dust in floppy drive. (b) Insert the boot floppy disk in to the floppy drive on the OM650 component. (c) Turn on the OM650 component. (d) When the boot prompt appears on the screen, press enter key once. (e) When prompted, eject the boot floppy disk, insert the root floppy disk in the floppy drive and press the enter key. 4. Performing restore (a) At the # prompt, type the command cd / (b) Type the command Txp.Restore. (c) On completion of the restore procedure, ensure that there are no error messages on the screen indicating bad sectors, bad tape etc. The restore takes about 15 to 20 minutes (d) Eject the DAT tape. (e) Type the command haltsys (f) Power down the computer

39 3.2. C101 - PU - PROCESSING UNIT Post restoration procedure (a) Disconnect any LAN and CS275 Local Bus cables that connect to the OM650 component. This includes Terminal Bus connections, Plant Bus connections, Web Bus connections and CS275 Local Bus Connections. (b) Stopping OM system if started i. login as a txpom user ii. During startup OM650 component will very likely start automatically Om.Start process iii. Check this with the PL command iv. Activate an Om.Stop (c) Restoration of directory structure important to OM component i. login as a root ii. go to following directory: cd /usr/txpom/install. iii. Now in this directory you can see the files Om.Install and Root.Install. iv. First run the Om.Install by./om.install command. v. After completion of the process run the Root.Install by./root.install command. vi. Now again restart the server by init 6 (d) In case there were modifications of the OM related functions since last backup it will be required to transfer of all required code to that component. i. Stopping OM system if started ii. login as a txpom user iii. During startup OM650 component will very likely start automatically Om.Start process iv. Check this with the PL command v. Activate an Om.Stop vi. An OT would require transfer of MMI, DynFup. vii. A PU would require transfer of LAN, ASR, and Processing Functions. viii. A PU-ME would require transfer of ASR from GetOM. ix. A SU would require transfer of Prt, BDM x. Under special circumstances other configurations may have to be transferred or edited manually. (e) When it is clear that the component is correctly configured, the system can be shut down, 6. Putting computer back into service (a) re-establish LAN connections (b) switch on the computer (c) Please check the server status on other server monitor by typing the command PL t (txpom login).

40 3.2. C101 - PU - PROCESSING UNIT Component Installation and Commissioning The installation of new or repaired component is described in restoration section of the backup see on page 37 and extensive documentation can be found in OEM documentation. Most notably [100] - OM 650 Installation and Interfaces and [101] - Device Manual System Components and Peripherals should be referred to References Admin Report Sugen - July OM 650 Installation and Interfaces Device Manual System Components and Peripherals F36-MI-02 - Sugen procedure for daily maintenance MIG654 - Sugen Procedure for backing up OM components

41 3.3. C102 - SU - SERVER UNIT C102 - SU - Server Unit This chapter is addressing OM 650 component called Server Unit shortly known under name SU. The pair of SUs in installed at each unit at power-plant. It is addressing it from maintenance point of view. Following is being addressed: Functional principles and component basic configuration Component Diagnostic and Routine maintenance Component Backup and Recovery principles Component Installation and Commissioning The processing unit is main interface between operator and automation level. It s main tasks are as follows: BDM - keep database of descriptions - translating internal system numbering called IKZ to human readable KKS and Descriptions LZA - Long term archive which is divided into two principal archives Circular archive keeping records for limited amount of the time on hardisk Permanent archive on external media like MOD PRT - access to archives, data searching NTB - notebook functionality Figure 3.3.1: OM system status

42 3.3. C102 - SU - SERVER UNIT Functional principles and component basic configuration SU component file structure The installation of OM 650 system is located in directories described below / > this is Unix root of the file system /txpsys The executable programs of the OM software are located in this directory or file system There is defined system variable called $OmSys /txpsys/txpconf In this directory generic TXP configuration is located There is defined system variable called $OmConfData /txpproj/proj_std In the txpproj directory or file system the engineering data are stored (as for example the pictures for the MMI or the userspecific network configuration data from ASR) There is defined system variable called $OmProjData /txptest in the directory txptest the diagnostic files are located (error log files), which are created by every OM subcomponent There is defined system variable called $OmDiagData /txpproz The process data (hard copies, notes, calculations) are stored in the directory txpproz There is defined system variable called $OmProzData Each file system or directory mentioned above has a function-related organization which is depending on the object managers installed on given OM component. Figure 3.3.2: File structure of typical OM650 component The directory structure of PU computer is as follows.

43 3.3. C102 - SU - SERVER UNIT 43 /root /txpsys /inf /asr /arc /mac /swi /txpproj/proj_std /inf /asr /arc /mac /txptest /inf /asr /arc /mac /txpproz /inf /asr /arc /mac

44 3.3. C102 - SU - SERVER UNIT 44 SU component configuration files Infrastructure - INF For the configuration of the distributed OM system infrastructure (INF) needs to be defined. Infrastructure is defined in following files: InfOmk.proj InfFb.proj InfDevInst.proj InfObm.inst All these files are located in the directory $OmProjData/inf. Detailed description of infrastructure files can be found on page 17. InfObm.inst The file InfObm.inst defines which OM subcomponents will run of given computer. This file is specific for each OM component. Algorithm 3.6 Content of the InfObm.inst file at Sugen SU unit #ObjMgr DirName BDM bdm LZA lza PRT prt NTB ntb

45 3.3. C102 - SU - SERVER UNIT 45 LZA In the engineering file Lza.proj only the function areas ids need to be listed. These can be found at the ES680 database or at [000]. Algorithm 3.7 Content of the Lza.proj file at Sugen PU unit # Lza. proj # global user data of TXP OM long time archive # # plant xx # SU xx # release xx # date nn. nn. nnnn # # # enumeration of a l l function areas : # #=============================================================================== # # AnzAikz : maximum number of user specified IKZ # AnzLikz : maximum number of type specified IKZ # # PDA can only process ( AnzAikz+AnzLikz ) IKZ s # # AnzEr : number of events a sub archive (PDA) can st or e # # ( AnzEr) parameter value should be much great er than ( AnzAikz+AnzLikz ) ( between 5 20 times ) # # maximum s i z e of a sub archive : # ( AnzAikz+AnzLikz ) 24 + AnzEr 44 Byte # # standard pr oje ct planning ( AnzAikz =30000, AnzLikz =70000, AnzEr= ) # re qu ire s disk storage of about 46 MB per sub archive # # disk storage requirement of process data archive : (AnzTa) in Lza. conf # (AnzTa + 1) s iz e of sub archive + jou rnaling # disk storage requirement fo r jour naling inc r e a se s with run time # and relocation rate ( process data and protocol / f i l e s ). # r e a l conditions w i ll hardly r equire disk storage of more than 50 MB. # # P L E A S E N O T I C E : # # I f you enlarge below l i s t e d parameters the PDA fi le syst e m may overflow. # In t h is case you should reduce parameter AnzTa in Lza. conf. # After that action not needed sub ar chives may be removed from f ile sys t e m. # Notice that you t he r e fo r e may loose events! # # I f you enlarge ( AnzAikz+AnzLikz ), parameter (ShmAnzEr) in LzaSp. conf # should be enlarged as well. # Enlargement of these parameters i nc r e a se s shared memory requirement of PDA! # #=============================================================================== AnzAikz # number of user s p e c if i e d events AnzLikz # number of type s p e c if i e d events AnzEr # s i z e of a changeable archive EreigTypen 0 x f f f f # event types to archive # end Lza. proj Component Diagnostic and Routine maintenance When SU component starts failing proper diagnosis is required. Following some most commons methods employed to detect which part of the system is failing to decide on proper corrective measures Routine maintenance The routine maintenance of PU component is at Sugen described by the procedure F36-MI-02 [902] Evaluation of the computer is in detail described in reference [001] PL -t Check if all OM components are in operation All components must be if fue status or akt for hot standby Time must be synchronized

46 3.3. C102 - SU - SERVER UNIT 46 Figure 3.3.3: OM system status Pool /etc/dfspace Check for available disk space on all OM components Compare to the other systems one additional filesystem called /txparc is present - this is a location of all process files - this place is NOT being backup by routine backup program Figure 3.3.4: SU unit output of the /etc/dfspace command Poll tail -7 /usr/lib/powerchute/*log Check for UPS / Battery failure - check for error messages OmProj.Check This command is not required for daily use - only after OM reconfigaration and after updates Verify OM configuration - check for any errors reported Evaluation of the file /txp/om650/txptest/diagmeld* and subsequent errors if any Diagnostic files are explained in detail below Rule of thumb say - if both of diagnostic files have very different dates and error messages their content are very seldom system seems to be in order. /var/adm This is a location system log files for which root access might be required

47 3.3. C102 - SU - SERVER UNIT 47 Figure 3.3.6: Screenshot from the SU-B server - system messages verify system log files - syspages - see attachment No. 3 of the reference [001] Specially on SU needs to be clear that there are no errors on the hardisk or MOD - see figure below for specific errors in system log files. [110] Figure 3.3.5: SU unit system log files indicates problems with storage media SAMPLER If any problems detected run SAMPLER - it will gather all log files - MUST be done prior to any modifications!!!! It keeps history Transfer SAMPLER to the safe location

48 3.3. C102 - SU - SERVER UNIT Diagnostic files Each object manager writes error log files. In these files errors and messages are stored as a report. All diagnostic files are located in the path /txptest. The infrastructure module writes directly into directory /txptest. Every other object manager writes into its respective subdirectory. For example the ASR writes into /txptest/asr. The diagnostic files are realised via a cyclic buffer. First all diagnostic messages are written into file DiagMld.0. If this file reaches a size greater than 1 MB, then the file DiagMld.1 is created. If this file also becomes greater than 1 MB, then DiagMld.0 is deleted and created again. Thanks to this mechanism, the hard disk will never be full and the latest diagnostic data are always available. All diagnostic files are always built with a two line set. The date and time of the message are written in the first line, as well as the program name, process id, module name, module version, line number and function name. On the second line we can find the error message number and the error message description. Unfortunately these diagnostic files have been written by developers and for other developers. Therefore they are often very difficult to understand. To see content of the generic diagnostic file follow these steps. Login as txpom user Navigate to the location of the log files by commend cd $OmTestData Check with the directory listing which diagnostic file is newer ls -la Dia* Use tail command to see last, let s say 30 lines of the file DiagMld.0 tail -30 D*.0 Should find that some object manager is reporting many errors, lets say asr than check diagnostic files of lza manager as follows. Login as txpom user Navigate to the location of the log files by commend cd $OmTestData cd lza

49 3.3. C102 - SU - SERVER UNIT 49 Check with the directory listing which diagnostic file is newer ls -la Dia* Use tail command to see last, let s say 30 lines of the file DiagMld.1 tail -30 D*.1 All unix commands mentioned in this section are explaned at Bohemia Market knowledge base at

50 3.3. C102 - SU - SERVER UNIT Resetting and analyzing arc subsystem Sometimes it happen that values are not shown on the display and log files. Usual suspect is ARC subsystem. The reset of ARC archive require to stop redundant computer and than to stop it on master as well. Computer a02p1a The functions on this computer are in AKT which means computer acts as a redundant computer. Step 1 - Stop OM (stopping this computer prevent synchronization) - Om.Stop Step 2 - Transfer ASR Find ARC process on master computer Kill ARC process on master computer Step 6 - Restart computer (as a root user) - init 6 Wait until a02p1a restarts Step 8 - Arc.Moni should show that ARC contain correct data Computer a02p1b The functions on this computer are in FUE which means computer acts as an master computer. No action until OM stops on redundant computer Step 3 - Transfer ASR Step 4 - ps -ef grep Arc Step 5 - kill Wait until a02p1a restarts (as a root user) - init 6 Step 7 - Restart computer Figure 3.3.7: Identify the process number - result of Step 4 commands

51 3.3. C102 - SU - SERVER UNIT Component Backup and Recovery principles The backup method used for processing computer is using OEM developed script called Txp.Backup for backup purposes and Txp.Restore for restoration of the component. This is fairly reliable procedure however should be noted that this is logical type of the backup. Moreover!! this tools do not backup process related data!! There is a procedure at Sugen MIG654 [954] addressing this task. Backup Equipment required: External DDS2 or DDS3 DAT tape drive VGA monitor PS2 Keyboard Ultra wide SCSI cable with connector capable of connecting to the Adaptec Ultra Wide SCSI controller installed in the Celsius... computers. A new DDS2 DAT tape. The 1.44Mb 3.5 boot and root floppy disks see how to create disk at... if you do not have one

52 3.3. C102 - SU - SERVER UNIT Backup Procedure 1. Verification (a) If a redundant component is to be backed up, ensure the redundant component is in error free operation. (b) If a web4txp server is being backed up, ensure all web users have been disconnected from that component. 2. Stopping OM system (a) Login as txpom (b) enter the Om.Stop command. (c) When the OM process has stopped, logout the txpom user. 3. Shutdown computer (a) Login as root user (b) Shut down the OM650 system by entering the command init 0. (c) When the system is completely shut down and is displaying the prompt It is safe to power off, then turn the power off to the system. 4. Preparing Tape Drive (a) With the power off on the DAT drive and the power off on the OM650 component, connect the DAT to the UWSCSI adapter using the UWSCSI cable. (b) Set the SCSI id to 2 on the DAT drive. (c) Power on the DAT drive. (d) Insert properly labeled DDS2 DAT tape in the DAT drive, ensure the write protect tab is closed and wait till the DAT tape drive light stops flashing.

53 3.3. C102 - SU - SERVER UNIT Booting the computer (a) Clean the floppy drive by blow of air through it - otherwise floppy disk may be damaged or not work. Over years there is lot of dust in floppy drive. (b) Insert the boot floppy disk in to the floppy drive on the OM650 component. (c) Turn on the OM650 component. (d) When the boot prompt appears on the screen, press enter key once. (e) When prompted, eject the boot floppy disk, insert the root floppy disk in the floppy drive and press the enter key. 6. Performing backup (a) At the # prompt, type the command Txp.Backup. (b) On completion of the restore procedure, ensure that there are no error messages on the screen indicating bad sectors, bad tape etc. The backup takes about 15 to 20 minutes (c) It is advisable to see the content of the tape to make sure data were written to it. Since tape archive is created by cpio command it is easy to see it s content by executing command cpio -ivt -I /dev/rstp0 - more on this command can be found in Bohemia Market knowledge base under this link php?title=cpio (d) Eject the DAT tape. 7. Powering computer back to normal operation (a) Type the command haltsys. (b) Turn the power off to the Celsius computer and the DAT tape drive. (c) Disconnect the UWSCSI cable from the computer. (d) Apply power to the OM650 component and observe its successful boot, OM start and synchronization.

54 3.3. C102 - SU - SERVER UNIT 54 Restore Following a system restore, it is necessary to transfer the latest generated code to the component before activating it within the operating infrastructure. In most cases, the OM process automatically starts, so a special procedure has to be put in place to ensure the newly restored component is isolated until it has the latest code transferred. Equipment required: External DDS2 or DDS3 DAT tape drive VGA monitor PS2 Keyboard Ultra wide SCSI cable with connector capable of connecting to the Adaptec Ultra Wide SCSI controller installed in the Celsius... computers. A new DDS2 DAT tape. The 1.44Mb 3.5 boot and root floppy disks see how to create disk at... if you do not have one

55 3.3. C102 - SU - SERVER UNIT Restore Procedure 1. Shutdown computer and power it off 2. Preparing Tape Drive (a) With the power off on the DAT drive and the power off on the OM650 component, connect the DAT to the UWSCSI adapter using the UWSCSI cable. (b) Set the SCSI id to 2 on the DAT drive. (c) Power on the DAT drive. (d) Insert latest backup tape, ensure the write protect tab is open to avoid accidental deletion of the backup tape and wait till the DAT tape drive light stops flashing. 3. Booting the computer (a) Clean the floppy drive by blow of air through it - otherwise floppy disk may be damaged or not work. Over years there is lot of dust in floppy drive. (b) Insert the boot floppy disk in to the floppy drive on the OM650 component. (c) Turn on the OM650 component. (d) When the boot prompt appears on the screen, press enter key once. (e) When prompted, eject the boot floppy disk, insert the root floppy disk in the floppy drive and press the enter key. 4. Performing restore (a) At the # prompt, type the command cd / (b) Type the command Txp.Restore. (c) On completion of the restore procedure, ensure that there are no error messages on the screen indicating bad sectors, bad tape etc. The restore takes about 15 to 20 minutes (d) Eject the DAT tape. (e) Type the command haltsys (f) Power down the computer

56 3.3. C102 - SU - SERVER UNIT Post restoration procedure (a) Disconnect any LAN and CS275 Local Bus cables that connect to the OM650 component. This includes Terminal Bus connections, Plant Bus connections, Web Bus connections and CS275 Local Bus Connections. (b) Stopping OM system if started i. login as a txpom user ii. During startup OM650 component will very likely start automatically Om.Start process iii. Check this with the PL command iv. Activate an Om.Stop (c) Restoration of directory structure important to OM component i. login as a root ii. go to following directory: cd /usr/txpom/install. iii. Now in this directory you can see the files Om.Install and Root.Install. iv. First run the Om.Install by./om.install command. v. After completion of the process run the Root.Install by./root.install command. vi. Now again restart the server by init 6 (d) In case there were modifications of the OM related functions since last backup it will be required to transfer of all required code to that component. i. Stopping OM system if started ii. login as a txpom user iii. During startup OM650 component will very likely start automatically Om.Start process iv. Check this with the PL command v. Activate an Om.Stop vi. A SU would require transfer of Prt, BDM vii. Under special circumstances other configurations may have to be transferred or edited manually. (e) When it is clear that the component is correctly configured, the system can be shut down, 6. Putting computer back into service (a) re-establish LAN connections - ONLY when all activities terminated (b) switch on the computer (c) Please check the server status on other server monitor by typing the command PL t (txpom login). (d) In case of SU unit synchronization can take many hours - be patient do EVER NOT INTERUPT process!!!!!

57 3.3. C102 - SU - SERVER UNIT Component Installation and Commissioning The installation of new or repaired component is described in restoration section of the backup see on page 54 and extensive documentation can be found in OEM documentation. Most notably [100] - OM 650 Installation and Interfaces and [101] - Device Manual System Components and Peripherals should be referred to References Cheat Sheet Admin Report Sugen - July OM 650 Installation and Interfaces Device Manual System Components and Peripherals SU problem - hardisk failure broken ARC subsystem F36-MI-02 - Sugen procedure for daily maintenance MIG654 - Sugen Procedure for backing up OM components

58 3.4. C103 - HMI SERVER - WEB4TXP C103 - HMI Server - web4txp This chapter is addressing OM 650 component called HMI Unit shortly known under name OT or HMI Server. The HMI servers are non-redundant computers. However any web4txp is giving access to complete HMI system. It is addressing it from maintenance point of view. Following is being addressed: Functional principles and component basic configuration Component Diagnostic and Routine maintenance Component Backup and Recovery principles Component Installation and Commissioning The HMI server is responsible to process operator related screens, accessing log files and archives and giving operator control over the plant. The Sugen configuration allows to access all functionality of the HMI sever via Internet Explorer from windows TC (Thin Client) computer.

59 3.4. C103 - HMI SERVER - WEB4TXP Functional principles and component basic configuration Web4TXP component file structure The installation of OM 650 system is located in directories described below. / > this is Unix root of the file system /txpsys The executable programs of the OM software are located in this directory or file system There is defined system variable called $OmSys /txpsys/txpconf In this directory generic TXP configuration is located There is defined system variable called $OmConfData /txpproj/proj_std In the txpproj directory or file system the engineering data are stored (as for example the pictures for the MMI or the userspecific network configuration data from ASR) There is defined system variable called $OmProjData /txptest in the directory txptest the diagnostic files are located (error log files), which are created by every OM subcomponent There is defined system variable called $OmDiagData /txpproz The process data (hard copies, notes, calculations) are stored in the directory txpproz There is defined system variable called $OmProzData /txp/web For web4txp system there is a directory where web server users, applications, access control etc. is configured Each file system or directory mentioned above has a function-related organization which is depending on the object managers installed on given OM component. Figure 3.4.1: File structure of typical OM650 component

60 3.4. C103 - HMI SERVER - WEB4TXP 60 The directory structure of HMI computer is as follows. /root /txpsys /inf /asr /arc /mac /swi /txpproj/proj_std /inf /asr /arc /mac /txptest /inf /asr /arc /mac /txpproz /inf /asr /arc /mac /txp/web

61 3.4. C103 - HMI SERVER - WEB4TXP 61 HMI component configuration files Infrastructure - INF For the configuration of the distributed OM system infrastructure (INF) needs to be defined. Infrastructure is defined in following files: InfOmk.proj InfFb.proj InfDevInst.proj InfObm.inst All these files are located in the directory $OmProjData/inf. Detailed description of infrastructure files can be found on page 17. InfObm.inst The file InfObm.inst defines which OM subcomponents will run of given computer. This file is specific for each OM component. Algorithm 3.8 Content of the InfObm.inst file at Sugen PU unit #ObjMgr DirName ASR asr ARC arc MAC mac w4.install.para This is a file where configuration of the web4txp system is performed. The most common task performed is adding new user to the web4txp system, It is described in next chapter.

62 Add new user to web4txp system Sagunto - May 14, 2007 Adding new user into web4txp system is straight forward. Here is how to achieve this task: 1. Login to the web4txp system as root user in order to get right to create new users 2. Add new user into unix system see algorithm 3.9 on the next page for more details. 3. Create password for newly created users 1 by command: (a) passwd office1 (b) choose option number 1 (c) type new password and confirm it 4. Edit file /txp/web/install/w4.install.para (a) add line for each additional user into the OM_USER section see figure on the following page. (b) add line for each additional thin client (computer) into the OM_CLIENT section see figure on the next page. (c) if desired than add similar information for the ES680 station - ES_USER section to add application name and user name see figure on page 64. (d) if desired than add similar information for the ES680 station - ES_APPL section to add IP address and login for given user see figure on page Run the configuration script./w4.config in directory /txp/web/install/ (a) choose option number 6 (set up web4txp server database) 1 example is for the user called office1 62

63 3.4. C103 - HMI SERVER - WEB4TXP 63 Algorithm 3.9 Adding new user into unix system useradd -u g G web4txp -d /txp/web/users/office1 -m -s /bin/csh office1 +---> system name of the added user +---> Name of the shell +---> create user directory +---> place of the home directory +---> name of user group (check the file /etc/groups ) +---> number of user group (check the file /etc/groups ) +---> userid (uid) system number -- for TXP system is used > 8000 (see /etc/passwd for next free number) Figure 3.4.2: Edit OM_USER section add for each user application name Figure 3.4.3: Edit OM_CLIENT section add for each computer the line with IP address and user name which will be used

64 3.4. C103 - HMI SERVER - WEB4TXP 64 Figure 3.4.4: Edit ES_USER section add for each user application name Figure 3.4.5: Edit ES_APPL section add for each computer the line with IP address and user name which will be used

65 3.4. C103 - HMI SERVER - WEB4TXP Component Diagnostic and Routine maintenance When HMI component starts failing proper diagnosis is required. Following some most commons methods employed to detect which part of the system is failing to decide on proper corrective measures Routine maintenance The routine maintenance of HMI component is at Sugen described by the procedure F36-MI-02 [902] Evaluation of the computer is in detail described in reference [001] PL -t Check if all OM components are in operation All components must be if fue status or akt for hot standby Time must be synchronized Pool /etc/dfspace Figure 3.4.6: OM system status Check for available disk space on all OM components Figure 3.4.7: PU unit output of the /etc/dfspace command

66 3.4. C103 - HMI SERVER - WEB4TXP 66 Poll tail -7 /usr/lib/powerchute/*log Check for UPS / Battery failure - check for error messages OmProj.Check This command is not required for daily use - only after OM reconfiguration and after updates Verify OM configuration - check for any errors reported Evaluation of the file /txp/om650/txptest/diagmeld* and subsequent errors if any Diagnostic files are explained in detail below Rule of thumb say - if both of diagnostic files have very different dates and error messages their content are very seldom system seems to be in order. /var/adm This is a location system log files for which root access might be required verify system log files - syspages - see attachment No. 3 of the reference [001] SAMPLER If any problems detected run SAMPLER - it will gather all log files - MUST be done prior to any modifications!!!! It keeps history Transfer SAMPLER to the safe location

67 3.4. C103 - HMI SERVER - WEB4TXP Diagnostic files Each object manager writes error log files. In these files errors and messages are stored as a report. All diagnostic files are located in the path /txptest. The infrastructure module writes directly into directory /txptest. The diagnostic files are realized via a cyclic buffer. First all diagnostic messages are written into file DiagMld.0. If this file reaches a size greater than 1 MB, then the file DiagMld.1 is created. If this file also becomes greater than 1 MB, then DiagMld.0 is deleted and created again. Thanks to this mechanism, the hard disk will never be full and the latest diagnostic data are always available. All diagnostic files are always built with a two line set. The date and time of the message are written in the first line, as well as the program name, process id, module name, module version, line number and function name. On the second line we can find the error message number and the error message description. Unfortunately these diagnostic files have been written by developers and for other developers. Therefore they are often very difficult to understand. To see content of the generic diagnostic file follow these steps. Login as txpom user Navigate to the location of the log files by commend cd $OmTestData Check with the directory listing which diagnostic file is newer ls -la Dia* Use tail command to see last, let s say 30 lines of the file DiagMld.0 tail -30 D*.0 Should find that some object manager is reporting many errors, lets say asr than check diagnostic files of asr manager as follows. Login as txpom user Navigate to the location of the log files by commend cd $OmTestData cd asr Check with the directory listing which diagnostic file is newer ls -la Dia*

68 3.4. C103 - HMI SERVER - WEB4TXP 68 Use tail command to see last, let s say 30 lines of the file DiagMld.1 tail -30 D*.1 All unix commands mentioned in this section are explained at Bohemia Market knowledge base at

69 3.4. C103 - HMI SERVER - WEB4TXP Component Backup and Recovery principles The backup method used for processing computer is using OEM developed script called Txp.Backup for backup purposes and Txp.Restore for restoration of the component. This is fairly reliable procedure however should be noted that this is logical type of the backup. There is a procedure at Sugen MIG654 [954] addressing this task. Backup Equipment required: External DDS2 or DDS3 DAT tape drive VGA monitor PS2 Keyboard SCSI cable with connector capable of connecting to the Adaptec Ultra Wide SCSI controller installed in the Celsius... computers. A new DDS2 DAT tape. The 1.44Mb 3.5 boot and root floppy disks see how to create disk at... if you do not have one

70 3.4. C103 - HMI SERVER - WEB4TXP Backup Procedure 1. Verification (a) If a redundant component is to be backed up, ensure the redundant component is in error free operation. (b) If a web4txp server is being backed up, ensure all web users have been disconnected from that component. 2. Stopping OM system (a) Login as txpom (b) enter the Om.Stop command. (c) When the OM process has stopped, logout the txpom user. 3. Shutdown computer (a) Login as root user (b) Shut down the OM650 system by entering the command init 0. (c) When the system is completely shut down and is displaying the prompt It is safe to power off, then turn the power off to the system. 4. Preparing Tape Drive (a) With the power off on the DAT drive and the power off on the OM650 component, connect the DAT to the UWSCSI adapter using the UWSCSI cable. (b) Set the SCSI id to 2 on the DAT drive. (c) Power on the DAT drive. (d) Insert properly labeled DDS2 DAT tape in the DAT drive, ensure the write protect tab is closed and wait till the DAT tape drive light stops flashing.

71 3.4. C103 - HMI SERVER - WEB4TXP Booting the computer (a) Clean the floppy drive by blow of air through it - otherwise floppy disk may be damaged or not work. Over years there is lot of dust in floppy drive. (b) Insert the boot floppy disk in to the floppy drive on the OM650 component. (c) Turn on the OM650 component. (d) When the boot prompt appears on the screen, press enter key once. (e) When prompted, eject the boot floppy disk, insert the root floppy disk in the floppy drive and press the enter key. 6. Performing backup (a) At the # prompt, type the command Txp.Backup. (b) On completion of the restore procedure, ensure that there are no error messages on the screen indicating bad sectors, bad tape etc. The backup takes about 15 to 20 minutes (c) It is advisable to see the content of the tape to make sure data were written to it. Since tape archive is created by cpio command it is easy to see it s content by executing command cpio -ivt -I /dev/rstp0 - more on this command can be found in Bohemia Market knowledge base under this link php?title=cpio (d) Eject the DAT tape. 7. Powering computer back to normal operation (a) Type the command haltsys. (b) Turn the power off to the Celsius computer and the DAT tape drive. (c) Disconnect the SCSI cable from the computer. (d) Apply power to the OM650 component and observe its successful boot, OM start and synchronization.

72 3.4. C103 - HMI SERVER - WEB4TXP 72 Restore Following a system restore, it is necessary to transfer the latest generated code to the component before activating it within the operating infrastructure. In most cases, the OM process automatically starts, so a special procedure has to be put in place to ensure the newly restored component is isolated until it has the latest code transferred. Equipment required: External DDS2 or DDS3 DAT tape drive VGA monitor PS2 Keyboard SCSI cable with connector capable of connecting to the Adaptec Ultra Wide SCSI controller installed in the Celsius... computers. A new DDS2 DAT tape. The 1.44Mb 3.5 boot and root floppy disks see how to create disk at... if you do not have one

73 3.4. C103 - HMI SERVER - WEB4TXP Restore Procedure 1. Shutdown computer and power it off 2. Preparing Tape Drive (a) With the power off on the DAT drive and the power off on the OM650 component, connect the DAT to the SCSI adapter using the UWSCSI cable. (b) Set the SCSI id to 2 on the DAT drive. (c) Power on the DAT drive. (d) Insert latest backup tape, ensure the write protect tab is open to avoid accidental deletion of the backup tape and wait till the DAT tape drive light stops flashing. 3. Booting the computer (a) Clean the floppy drive by blow of air through it - otherwise floppy disk may be damaged or not work. Over years there is lot of dust in floppy drive. (b) Insert the boot floppy disk in to the floppy drive on the OM650 component. (c) Turn on the OM650 component. (d) When the boot prompt appears on the screen, press enter key once. (e) When prompted, eject the boot floppy disk, insert the root floppy disk in the floppy drive and press the enter key. 4. Performing restore (a) At the # prompt, type the command cd / (b) Type the command Txp.Restore. (c) On completion of the restore procedure, ensure that there are no error messages on the screen indicating bad sectors, bad tape etc. The restore takes about 15 to 20 minutes (d) Eject the DAT tape. (e) Type the command haltsys (f) Power down the computer

74 3.4. C103 - HMI SERVER - WEB4TXP Post restoration procedure (a) Disconnect any LAN that connect to the OM650 component. This includes Terminal Bus connections, Plant Bus connections, Web Bus connections. (b) Stopping OM system if started i. login as a txpom user ii. During startup OM650 component will very likely start automatically Om.Start process iii. Check this with the PL command iv. Activate an Om.Stop (c) Restoration of directory structure important to OM component i. login as a root ii. go to following directory: cd /usr/txpom/install. iii. Now in this directory you can see the files Om.Install and Root.Install. iv. First run the Om.Install by./om.install command. v. After completion of the process run the Root.Install by./root.install command. vi. Now again restart the server by init 6 (d) In case there were modifications of the OM related functions since last backup it will be required to transfer of all required code to that component. i. Stopping OM system if started ii. login as a txpom user iii. During startup OM650 component will very likely start automatically Om.Start process iv. Check this with the PL command v. Activate an Om.Stop vi. An OT would require transfer of MMI, DynFup. vii. Check web4txp configuration viii. Under special circumstances other configurations may have to be transferred or edited manually. (e) When it is clear that the component is correctly configured, the system can be shut down, 6. Putting computer back into service (a) re-establish LAN connections (b) switch on the computer (c) Please check the server status on other server monitor by typing the command PL t (txpom login).

75 3.4. C103 - HMI SERVER - WEB4TXP Web4TXP system licensing Customer is having five web4txp licenses. However licensing did not worked properly. It has been found that there is a need to correct number of OT licenses in the file /txp/web/web4txp/etc/w4_applparam.cfg Also following conditions needs to be met: 1. correct licensing file WEB4TXP.LIC 2. Correct number of the license files in the directory /usr/dyxlic Component Installation and Commissioning The installation of new or repaired component is described in restoration section of the backup see on page 72 and extensive documentation can be found in OEM documentation. Most notably [100] - OM 650 Installation and Interfaces and [101] - Device Manual System Components and Peripherals should be referred to References Admin Report Sugen - July OM 650 Installation and Interfaces Device Manual System Components and Peripherals TXP manual describing web4txt installation Installing web4txp server F36-MI-02 - Sugen procedure for daily maintenance MIG654 - Sugen Procedure for backing up OM components References OM650 transfer and engineering

76 Chapter 4 C201 - ES Engineering Station This chapter is addressing Engineering Station known under the name ES 680. The Engineering Station contain central database for complete DCS system accompanied with set of maintenance tools. It Also contains graphical editor of MMI system called OM 650 editor alias Dynavis. There is one engineering station per power-plant unit. This chapter is addressing ES 680 from maintenance point of view. Following is being addressed: Principles and component basic configuration Component Diagnostic and Routine maintenance Component Backup and Recovery principles Component Installation and Commissioning 76

77 Functional principles and component basic configuration Engineering station principal function is to configure all components of DCS system called T2000 at one single place. There three principal functions of the ES Configure networking for Automation processors and create a programs on automation level 2. Create HMI interface like operator screen 3. Maintain generated code and configuration files and allows their transfers ES 680 User interface The user interface is well described in the reference from TXP manual located in the directory ES 680 User interface ES 680 component file structure The installation of ES 680 system is fairly complex and it s configuration require very deep knowledge of the Unix system, Ingres database system and knowledge of complete ES 680 software package. Such a knowledge is beyond possibility of this course (it would take several days to explain all). The installation TXP manual can be found at reference directory ES 680 installation instructions. Therefore only directories important to maintenance are being addressed here and proper disaster recovery planning is being put in the place. The most important directory from maintenance point of view are: $HOME directory of main database user /save directories where automated night backup is located

78 78 ES 680 component configuration files Component Diagnostic and Routine maintenance Routine maintenance Unit Tape cron_sicherung.prot dbrepair.prot dbrepair_result.prot nachtlauf.prot sperrdatei df -h 10 OK errors errors OK empty OK 20 OK errors errors OK empty OK 30 OK errors errors OK empty OK Check backup directories cd /save Database checks - Data consistency errors - should be minimum amount of them... normally fixed by maintenance scripts. Check errors Check errors Check for the errors in cron_sicherung.prot Check for lines starting with E_ > ingres problems. Solve immediately! sperdatei This file MUST be empty Check the need for code generation No generation needed Check the need for code transfer No download needed Check for the need of LAN transfer The LAN transfer required a01wt1 AP192 Available disk-space. Important is /tmp directory. If full more than 50% than problem with printer. df -h

79 Component Backup and Recovery principles Since ES680 is central depository of the project there are two different types of backup requirements: 1. Engineering data backup - Daily, Weekly or in any other regular period of time as per site needs 2. Computer file system (operating system and software installed on the computer) backup There is a procedure at Sugen MIG654 [954] addressing this task Disk Backup Procedure Equipment required: External DDS2 or DDS3 DAT tape drive VGA monitor PS2 Keyboard SCSI cable with connector capable of connecting to the ES680 computer A new DDS2 DAT tape. Shutdown of the ES Make sure all clients have been logged out of the ES680 (command: finger). If they are not, and they have an application that opens the database, you run the risk of corrupting the database when the ES680 is shutdown... deep shit man! 2. Log into HP Vue as root and open a terminal session. 3. Use command: (a) shutdown h y 0 4. To restart the HP Machine, substitute the h switch with r 5. The HP machine will then shutdown. It takes a bit of time so be patient.

80 80 Primary Disk Root Backup 1. Make sure that the DAT tape you have matches the DAT Streamer you are using. Normally they are backward, but not forward compatible. At minimum you should be using DDS2 Tapes. 2. DAT Streamer must be on channel 3 (OM650 uses channel 2). 3. Connect the DAT Streamer with the 50 pin SCSI cable 4. Power up the DAT Streamer then HP machine. 5. Boot into Single User mode by pressing the esc key during the Boot up sequence (best to press the esc key as soon as you turn the HP box on). 6. Press enter at the prompt (don t know why, just need to!). 7. Boot the system by: (a) boot fwscsi6.0 pri (b) bo pri, providing the primary boot path is fwscsi.6.0 (use command pa to find out). i. Answer Y to Interact With IPL. ii. You will get the prompt ISL> (ISL is Initial System Loader). iii. Load the HP Unix bootstrap: hpux is (-is means boot into single user mode). 8. Once booted in Single User mode (a) Change to the / (root) directory at the # prompt by command (If you don t do this, the entire root file system will not be backed up. Only the file system you are in will be backed up) i. cd / (b) Insert cleaner tape into DAT Streamer. i. Insert DDS tape into DAT Streamer (green LED stops flashing when DAT is fully loaded). (c) Start backup i. tar cv ls 1 A. tar cv means create a new archive (c), and verbose (v) the archived files. B. ls 1 means list the files in a single column format. (d) Check for error messages when completed (e) Check that the backup procedure worked by command i. tar t tail Shutdown the HP machine command: (a) shutdown h y 0 (b) Remove the DAT Streamer. 10. Let it Boot it into Multi User mode.

81 81 Primary Disk Restore From DAT Tape This is a last resort method, and all other fixes should be attempted before doing a full ES680 restore. There is a good chance that the code on the restored ES680 will differ from that in the AS system. You may require a full offline download if there is not reliable project backup. All files on the ES680 will be removed, and then restored. 1. Make sure that the DAT tape you have matches the DAT Streamer you are using. Normally they are backward, but not forward compatible. At minimum you should be using DDS2 Tapes. 2. DAT Streamer must be on channel 3 (OM650 uses channel 2). 3. Connect the DAT Streamer with the 50 pin SCSI cable 4. Power up the DAT Streamer then HP machine. 5. Boot into Single User mode by pressing the esc key during the Boot up sequence (best to press the esc key as soon as you turn the HP box on). 6. Press enter at the prompt (don t know why, just need to!). 7. Boot the system by using secondary booting option (remember your disk is damaged) 1 (a) bo sescsi.3.0 isl, or o bo sec, providing the secondary boot path is sescsi.3.0 (use command pa to find out). (b) Answer N to Interact With IPL. (c) A basic system installation will commence. (d) The system will reboot several times. This is normal. When complete the HPVue login screen will appear. (e) Login with root (no password will be required). (f) Change to the root directory command: i. cd / (g) Put the DAT Tape into the DAT Streamer. (h) Start the tape restoration by: i. tar xv (i) Check for error messages. (j) Shutdown the HP machine (command: i. shutdown h y 0 (k) Remove the DAT Streamer. (l) Let it Boot it into Multi User mode. 8. If project data changed since last backup there is a need to recover complete project. 1 This may differ on model number

82 82 ES680 Overnight Run Scope of the Overnight Saving Procedure The current versions of ES680 provide automatic saving of the plant data configured each night. Automatic overnight saving currently covers three areas: The ES680 project database MMI data (OM path) Generated code for AP s Configuration The overnight run requires some project-specific and machine-related data in order to properly carry out it s task. These data are provided via the configuration file. Any measure to adapt automatic overnight saving should be carried out by the administrator at the master machine exclusively, preferably keeping in mind the initial notes on administrative activities. Path for the configuration files: /install/txpes/data/<project>/dba Also, in this path those logs are stored which are generated by the scripts and programs being executed during the night. ES680 Overnight Run Call-up of the overnight saving procedure To carry out time-delayed call-up of the overnight run, Cron call-ups are set under UNIX. This is done by using the crontab command, which can be used to transfer such call-ups to the cron. The time response (starting time of the individual tools) is stored in a file, which is activated using crontab. File with the path: /install/txpes/data/<project>/txpes.cron This file is automatically regenerated or overwritten on each update to a new ES680 version. If additional sections for the overnight run should be stored there, an additional copy of the configuration file should be set up (e.g. txpes.cron.<project>). Verification Verify content of safe directories and daily logs.

83 83 Crontab modification In alg. 4.2 on the next page is described how to set-up automatic tape backup. 2 Algorithm 4.1 Crontab modification for night backup. 1. Make a copy of the existing crontab crontab -l > crontab.list The content of the file will look like this: 0 0 * * 1,3,5 /txp/es680/txpes/sw/sw7.4.sco/bin/cron_nachtlauf.sh asir09 /txp/es680/txpes /txp/es680 u 0 0 * * 0,2,4,6 /txp/es680/txpes/sw/sw7.4.sco/bin/cron_nachtlauf.sh asir09 /txp/es680/txpes /txp/es680 g Above mentioned are TXP routines installed from Germany. 2. To make tape backup add following to the end of the file crontab.list : 15 3 * * * /usr/bin/tar cvf /dev/rstp0 /save At 3:15 morning cron program will run tar command and copy data from the /save directory to the tape. Used command tar cf instead of tar cvf to eliminate amount of messages in mail. In Asir have to be modified. 3. Update cron program by the file crontab.list crontab < crontab.list 4. Verify cron program by command crontab -l The another possibility is to edit crontab by the command crontab -e. 2 This description is applicable to the SCO unix version. The HP version has some differences in tape device names.

84 84 Tape verification PLEASE VERIFY TAPE BEFORE YOU INSERT NEW ONE - see alg. 4.2 for more details. What can be said about the output? Watch the dates of the files located in /save/save1 and /save/save2 directories and their sub-directories which to be same as dates of directories on the hardisk. Algorithm 4.2 Tape verification 1. Start xterminal 2. Enter command tar tv grep save 3. Watch the command output rw-rw-r--3000/ May 09 00: /save/save1/bisha05/as/ag0011/mc5/ap/system/lz_614.mc5 rw-rw-r--3000/ May 09 00: /save/save1/bisha05/as/ag0011/mc5/ap/system/lz_apf.mc5 rw-rw-r--3000/ May 09 00: /save/save1/bisha05/as/ag0011/mc5/ap/system/meld.mc5 rw-rw-r--3000/ May 09 00: /save/save1/bisha05/as/ag0011/mc5/ap/system/meld_aus.mc5 rw-rw-r--3000/ May 09 00: /save/save1/bisha05/as/ag0011/mc5/ap/system/neustart.mc5 rw-rw-r--3000/ May 09 00: /save/save1/bisha05/as/ag0011/mc5/ap/system/ob001.mc5 rw-rw-r--3000/ May 09 00: /save/save1/bisha05/as/ag0011/mc5/ap/system/ob019.mc5 rw-rw-r--3000/ May 09 00: /save/save1/bisha05/as/ag0011/mc5/ap/system/ob020.mc5 rw-rw-r--3000/ May 09 00: /save/save1/bisha05/as/ag0011/mc5/ap/system/ob021.mc5 rw-rw-r--3000/ May 09 00: /save/save1/bisha05/as/ag0011/mc5/ap/system/ob022.mc5 rw-rw-r--3000/ May 09 00: /save/save1/bisha05/as/ag0011/mc5/ap/system/ob023.mc5 rw-rw-r--3000/ May 09 00: /save/save1/bisha05/as/ag0011/mc5/ap/system/ob024.mc5 rw-rw-r--3000/ Mar 23 20: /save/save2/tahad_10/om_asr_0.tah rw-rw-r--3000/ Mar 23 20: /save/save2/tahad_10/obj_link.tah rw-rw-r--3000/ Mar 23 20: /save/save2/tahad_10/ombez_cp.tah rw-rw-r--3000/ Mar 23 20: /save/save2/tahad_10/om_ver.tah rw-rw-r--3000/ Mar 23 20: /save/save2/tahad_10/om_asr_b.tah rw-rw-r--3000/ Mar 23 20: /save/save2/tahad_10/om_asr_1.tah rw-rw-r--3000/ Mar 23 20: /save/save2/tahad_10/pd_proj.tah rw-rw-r--3000/ Mar 23 20: /save/save2/tahad_10/pic_d.tah rw-rw-r--3000/ Mar 23 20: /save/save2/tahad_10/pic_w.tah rw-rw-r--3000/ Mar 23 20: /save/save2/tahad_10/pic_r.tah rw-rw-r--3000/ Mar 23 20: /save/save2/tahad_10/pic_p.tah rw-rw-r--3000/ Mar 23 20: /save/save2/tahad_10/pr_w.tah rw-rw-r--3000/ Mar 23 20: /save/save2/tahad_10/pr_m.tah rw-rw-r--3000/ Mar 23 20: /save/save2/tahad_10/users.tah rw-rw-r--3000/ Mar 23 20: /save/save2/tahad_10/s5_apf_s.tah rw-rw-r--3000/ Mar 23 20: /save/save2/tahad_10/regel.tah rw-rw-r--3000/ Mar 23 20: /save/save2/tahad_10/msk_g.tah rw-rw-r--3000/ Mar 23 20: /save/save2/tahad_10/pic_kana.tah

85 85 Rebuild TXP database from night run save This procedure should only be carried out following very deep thought and if there is no other way to fix it. Many factors should be taken into consideration and other options may be considered. The database backup must be available Database restore procedure 1. Logon to the system where the database resides as the project administrator 2. Ensure that no other users are logged into that project locally or remotely. 3. Stop the project related services by changing directory to (a) cd $HOME/config/esMonitor 4. and entering the command (a) esmonitor.sh stop 5. Destroy the current database by typing (a) destroydb <project name> (b) Create and empty database by typing i. createdb d<project name> <project name> (c) Change to the save directory that holds the required backup Type i. reload.ing (d) Reboot the system Component Installation and Commissioning The installation of new or repaired component is described in restoration section of the backup see on page 92 and extensive documentation can be found in OEM documentation. Most notably [100] - OM 650 Installation and Interfaces and [101] - Device Manual System Components and Peripherals should be referred to References Admin Report Sugen - July ES 680 User interface ES680 manuals from TXP manual - how to use ES 680 installation instructions ES680 creating users F36-MI-02 - Sugen procedure for daily maintenance MIG654 - Sugen Procedure for backing up OM components

86 Chapter 5 C202 - DS Diagnostic station This chapter is addressing DS 650 server shortly known under name Diagnostic Station. It is being addressed from maintenance point of view. Following is being addressed: Functional principles and component basic configuration Component Diagnostic and Routine maintenance Component Backup and Recovery principles Component Installation and Commissioning 86

87 5.1. FUNCTIONAL PRINCIPLES AND COMPONENT BASIC CONFIGURATION Functional principles and component basic configuration There is very good TXP manual page which we will be following in this section. See reference DS 670 interface for functional description of the DS Starting and stopping DS 670 There is very good TXP manual describing step by step how to start and stop diagnostic station functions. See reference Stopping and Starting DS670 and installation of the DS 670 is described in DS 670 installation. 5.3 Daily use of DS670 DS 670 is great help for I&C department. If checked daily and errors acknowledge daily as shown below than it could be detected in no time what is a current system of the complete T2000. Figure 5.3.1: Common acknowledge function The diagnostic of particular components can be see in the reference directory Components diagnostic

88 5.4. COMPONENT DIAGNOSTIC AND ROUTINE MAINTENANCE Component Diagnostic and Routine maintenance When DS 670 component starts failing proper diagnosis is required. Following some most commons methods employed to detect which part of the system is failing to decide on proper corrective measures Routine maintenance The routine maintenance of DS670 component is at Sugen described by the procedure F36-MI-02 [902] Evaluation of the computer is in detail described in reference [001] - however should be noted that DS 670 point out any error experiencing with itself. Poll tail -7 /usr/lib/powerchute/*log Check for UPS / Battery failure - check for error messages /var/adm This is a location system log files for which root access might be required verify system log files - syspages - see attachment No. 3 of the reference [001]

89 5.5. BACKUP Component Backup and Recovery principles The backup method used for processing computer is using OEM developed script called Txp.Backup for backup purposes and Txp.Restore for restoration of the component. This is fairly reliable procedure however should be noted that this is logical type of the backup. There is a procedure at Sugen MIG654 [954] addressing this task. 5.5 Backup Equipment required: External DDS2 or DDS3 DAT tape drive VGA monitor PS2 Keyboard SCSI cable with connector capable of connecting to the Adaptec Ultra Wide SCSI controller installed in the Celsius... computers. A new DDS2 DAT tape. The 1.44Mb 3.5 boot and root floppy disks see how to create disk at... if you do not have one

90 5.5. BACKUP Backup Procedure 1. Stopping DS670 (a) Follow the procedure described at Stopping and Starting DS Shutdown computer (a) Login as root user (b) Shut down the DS 670 system by entering the command init 0. (c) When the system is completely shut down and is displaying the prompt It is safe to power off, then turn the power off to the system. 3. Preparing Tape Drive (a) With the power off on the DAT drive and the power off on the DS 670 component, connect the DAT to the SCSI adapter using the SCSI cable. (b) Set the SCSI id to 2 on the DAT drive. (c) Power on the DAT drive. (d) Insert properly labeled DDS2 DAT tape in the DAT drive, ensure the write protect tab is closed and wait till the DAT tape drive light stops flashing.

91 5.5. BACKUP Booting the computer (a) Clean the floppy drive by blow of air through it - otherwise floppy disk may be damaged or not work. Over years there is lot of dust in floppy drive. (b) Insert the boot floppy disk in to the floppy drive on the DS 670 station. (c) Turn on the DS 670 station. (d) When the boot prompt appears on the screen, press enter key once. (e) When prompted, eject the boot floppy disk, insert the root floppy disk in the floppy drive and press the enter key. 5. Performing backup (a) At the # prompt, type the command Txp.Backup. (b) On completion of the restore procedure, ensure that there are no error messages on the screen indicating bad sectors, bad tape etc. The backup takes about 15 to 20 minutes (c) It is advisable to see the content of the tape to make sure data were written to it. Since tape archive is created by cpio command it is easy to see it s content by executing command cpio -ivt -I /dev/rstp0 - more on this command can be found in Bohemia Market knowledge base under this link php?title=cpio (d) Eject the DAT tape. 6. Powering computer back to normal operation (a) Type the command haltsys. (b) Turn the power off to the Celsius computer and the DAT tape drive. (c) Disconnect the SCSI cable from the computer. (d) Apply power to the DS 670 station and start it as described at Stopping and Starting DS670.

92 5.6. RESTORE Restore Following a system restore, it is necessary to transfer the latest generated code, topology diagram and OM infrastructure. Equipment required: External DDS2 or DDS3 DAT tape drive VGA monitor PS2 Keyboard SCSI cable with connector capable of connecting to the Adaptec Ultra Wide SCSI controller installed in the Celsius... computers. A new DDS2 DAT tape. The 1.44Mb 3.5 boot and root floppy disks see how to create disk at... if you do not have one

93 5.6. RESTORE Restore Procedure 1. Stopping DS670 (a) Follow the procedure described at Stopping and Starting DS Shutdown computer (a) Login as root user (b) Shut down the DS 670 system by entering the command init 0. (c) When the system is completely shut down and is displaying the prompt It is safe to power off, then turn the power off to the system. 3. Preparing Tape Drive (a) With the power off on the DAT drive and the power off on the DS 670 component, connect the DAT to the SCSI adapter using the SCSI cable. (b) Set the SCSI id to 2 on the DAT drive. (c) Power on the DAT drive. (d) Insert properly labeled DDS2 DAT tape in the DAT drive, ensure the write protect tab is closed and wait till the DAT tape drive light stops flashing. 4. Booting the computer (a) Clean the floppy drive by blow of air through it - otherwise floppy disk may be damaged or not work. Over years there is lot of dust in floppy drive. (b) Insert the boot floppy disk in to the floppy drive on the DS 670 component. (c) Turn on the DS 670 component. (d) When the boot prompt appears on the screen, press enter key once. (e) When prompted, eject the boot floppy disk, insert the root floppy disk in the floppy drive and press the enter key. 5. Performing restore (a) At the # prompt, type the command cd / (b) Type the command Txp.Restore. (c) On completion of the restore procedure, ensure that there are no error messages on the screen indicating bad sectors, bad tape etc. The restore takes about 15 to 20 minutes (d) Eject the DAT tape. (e) Type the command haltsys (f) Power down the computer

94 5.6. RESTORE Post restoration procedure (a) Disconnect any LAN that connect to the OM650 component. This includes Terminal Bus connections, Plant Bus connections, Web Bus connections. (b) Follow manuals DS 670 installation and Stopping and Starting DS670 for details what specific steps are required (c) When it is clear that the component is correctly configured, the system can be shut down, 7. Putting computer back into service (a) re-establish LAN connections (b) switch on the computer

95 5.7. DS 670 SYSTEM LICENSING DS 670 system licensing 5.8 Component Installation and Commissioning The installation of new or repaired component is described in restoration section of the backup and extensive documentation can be found in OEM documentation. Most notably reference from TXP manual located at DS 670 installation is describing in great detail how to configure and commission DS 670 computer References Admin Report Sugen - July OM 650 Installation and Interfaces Device Manual System Components and Peripherals DS 670 interface Stopping and Starting DS Components diagnostic DS 670 installation MIG654 - Sugen Procedure for backing up OM components

96 Chapter 6 C300 - OSM - Managed network switch OSM is managed network witch and it is central part of all networks present at the T2000 system. The brief description of the module can be seen at the reference Ethernet network ring structure - OSM module and OSM manual OSM module manual 6.1 References Ethernet network ring structure - OSM module OSM module manual 96

97 Chapter 7 C400 - AP Automation Processor This chapter is addressing Automation Processor shortly AP. The AP is part of T2000 hardware on which automation tasks are being executed. This chapter is addressing AP 620 from maintenance point of view and it includes case study of small hardware modification. This modification is addressing all aspects of such a task. Following is being addressed: Principles and component basic configuration Component Diagnostic and Routine maintenance Component Backup and Recovery principles Component Installation and Commissioning This section will be carried as follows: Location: Training center The explanation of the configuration of the T2000 hardware, I/O system Location: PCC There will be case study adding analog signal to the system and if time allow brief description of the binary signal will be discussed After HW configuration SW part of the signal adding will be explained And finaly code and pbp card will be generated Physical implementation and examples How to program field devices How to deal with physical hardware Explanation of various tyoes of hardware 97

98 98 Location: MCR - Main Control Room Practical implementation of case study by students Will be dedicated to the diagnostic such a Simulations Dynamisation DS 670 tool

99 7.1. FUNCTIONAL PRINCIPLES AND COMPONENT BASIC CONFIGURATION Functional principles and component basic configuration Hardware structure of AP and it s hardware is shown below. From the software point of view AP is PLC with preloaded standard libraries which are being centrally configured by means of the engineering station. Figure 7.1.1: Typical structure of the AP processor Figure 7.1.2: Physical AP rack

100 7.2. MAINTENANCE Maintenance There is an TXP maintenance manual Maintenance of AS620.

101 7.3. COMPONENT BACKUP AND RECOVERY PRINCIPLES Component Backup and Recovery principles All data in regards of the AS 620 are located on the ES 680 station. Therefore recovery involves good backup procedures of the ES 680 and changing hardware parts if neccessary. 7.4 Components Installation and Commissioning - Case study As always there is detailed manual available. However to make any sence of there is best to perform some real example. During case study error occured and IM 308 module did not started after programming memory card has been programmed. After studying manual AS 620 commissioninf and error description it has been found that card needs to be deleted prior programming.

102 7.5. BURN NEW IM308 FLASH CARD 102 Figure 7.4.1: Page from TXP manual describing the errors status LEDs of the IM 308 card 7.5 Burn New IM308 Flash Card General Info: This process can be done with the AP online providing: You are careful, You know what you re doing, There are REDUNDANT AP s, and That the Redundancy Fail-over works on the AP rack. Make sure the IM308 is in STOP and POWERED OFF when ever you INSERT or REMOVE a Flash card!!

103 7.5. BURN NEW IM308 FLASH CARD When IM608 card should be programmed If you: Change any of the ET200 I/O types (i.e. from 0-10V to 4-20mA etc. etc, etc). Add new ET200 stations etc, etc, etc. Delete any of the ET200 stations or I/O etc, etc, etc. Sometimes if you change a channel from T/C to RTD you also may have to burn a new Flash Card. If you re just adding/deleting a signal in TXP to an existing (configured) ET200 card, then this Flashing malarkey isn t required What You Need: PG with Flash Card slot in it. PG running ComProfibus (make sure you have the correct licences installed). Flash Card from IM308. USB memory stick to get data from ES680 (could also use a Floppy Disk) or do it over ftp Get IM308 Data From ES680: Make sure a full HW Generation has been done first!! On the ES680 menu: Generators => Select AP => Create ET200 Memory Card Files => Enter AP number. Check for generation errors etc. If there are errors and you continue with burning the flash card you run the risk of damaging plant and causing havoc!! The new ET200 file is located at: $HOME/listen/as/ag0<AP #>/transfer/... The file will be called A00<AP #>_<Bus #>.pbp (e.g. A0021_1.pbp) Copy the *.pbp file over to a HMI server (using FTP) so you can copy it onto your USB memory stick: Cmd: cd $HOME/listen/as/ag0<AP #>/transfer to the */transfer/... directory detailed above, Cmd: ftp <HMI server name> (use PL t to find the HMI server name) (e.g. ftp se_hmi1). Login as the necessary user (usually user txpom),

104 7.5. BURN NEW IM308 FLASH CARD 104 Cmd: binary (to make the transfer mode as binary, instead of ASCii), Cmd: put A00<AP #>_<Bus #>.pbp (this puts the *.pbp file onto the HMI server), Cmd: close (closes the connection to the HMI server), Cmd: bye (shuts down the FTP program). Retrieve the *.pbp file from the HMI server: Put your Memory Stick into the HMI server (it should be a Windows machine), Navigate to the txpom users home directory (or just do a search fro the *.pbp file), Send it to the USB stick, Burn The New Flash Card: Start ComProfibus on the PG, Import the *.pbp file using ComProfibus: File => Import => ASCII Data => select correct drive => select the *.pbp file The imported file should be displayed showing the ET200 chain (review and make sure it s correct for the installation). Burn the card: Put the Flash card into the slot in the PG, Click the Head Station so it s highlighted, Click the Flash Button (one with the lightening bolt on it), Memory card will be written to. Final Touches: With the IM308 in STOP and Powered OFF, insert the new Flash Card. Turn the power ON and put the IM308 into RUN Problem/Solutions: No communication with the ET200 station: Incorrect station address (DIP switches) on IM153. Bus cable not ok. Wires (Red/Green) are switched or broken. End terminating resistor is not installed. IM308 showing BF (Bus Fault) or IF (Interface Fault):

105 7.6. REFERENCES 105 Incorrect station address (DIP switches) on IM153. Bus cable not ok. Wires (Red/Green) are switched or broken. End terminating resistor is not installed. IM153 has no power supply. ET200 not mounted as per Topology Diagram. ET200 module may have faulted. File import errors on the PG: PG config files (gsd, gse, master etc) are missing from the PG740. File not transferred properly from the ES680. Check end of each section for incomplete lines of 0x0. Need to correct with Notepad to make them 0x References Maintenance of AS620 - excerpt from the TXP manual addressing errors AS maintenance training TXP maintenance - AS620 - complete TXP manual instruction profibus programming memory card AS 620 commissioninf and error description

106 Chapter 8 C450 - CM - Communication module This chapter is addressing CM communication module. The CM module is used for communication with foreign systems. This chapter is addressing AP 620 from maintenance point of view and it includes demonstration of the configuration of the CM module. Following is being addressed: Principles and component basic configuration Component Diagnostic and Routine maintenance Component Backup and Recovery principles Component Installation and Commissioning This section will be carried as follows: Location: Training center The explanation of the configuration of the CM module hardware and explanation Location: PCC Physical backup of the CM module configuration file 106

107 8.1. FUNCTIONAL PRINCIPLES AND COMPONENT BASIC CONFIGURATION Functional principles and component basic configuration CM module is a PC computer running DRDOS operating system and consist of two principal parts (1) TXP part which is responsible for communication with TXP system and (2) foreign system part which in implementation of the required protocol. The setup of the CM module is implemented via means of the CM.INI file. Figure 8.1.1: CM 104 drawing There is a procedure for transfering CM.ini file described at Communication to CM module - transfer CM.ini.

108 8.1. FUNCTIONAL PRINCIPLES AND COMPONENT BASIC CONFIGURATION 108 Algorithm 8.1 Example of CM.ini configuration file with modbus configuration ; ; ; CM104 Modbus Configuration File ; ; Hardware with 4 serial ports (Serial 1 to 4) ; ; ; Remote Terminal Interface (Serial 1) [OPERATE] Hardware=CM104 Baudrate=57600 (default) ;Baudrate= PortAdr=0x3f8 Irq=4 RtsCts=2 ; Dataframe is fix (8-N-1) ; ;Redundanzverbindung on (Serial 2) ;[SYNCHRONISATION] ;PortAdr=0x2f8 ;Irq=3 ;Baudrate= ; Dataframe is fix (8-N-1) ; [H1] SubUnit=Single ;Exclude= ;Mac= Mac = ApCmTimeout=20 ; ;Modbus Slave (Serial 2) ;U7: EP RAPPER SYTSEM RED. A [ModbusSlave_1] PortAdr=0x2f8 ; SERIAL 2 Irq=3 ; SERIAL 2 Baudrate=19200 Parity=None StopBits=1 DataBits=8 Slave=1 Timeout=5000 ;Offset for each Function Code ;Read Coil Status Code 1 ; base address = 1 ;RCS-Offset = -1 RCS-Offset = 0 ;Read Input Status Code 2 RIS-Offset = ;Read Holding Register Code 3 ; base address = ;RHR-Offset = RHR-Offset = 0 ;Read Input Register Code 4 RIR-Offset = ;Force Single Coil Code 5 ; base address = 1 FSC-Offset = -1 ;Preset Single Register Code 6 ; base address = PSR-Offset = ; ; Display an identification string [DISPLAY] Info=CM92_TO_SHOAIBAH2 ; ; Enable the CM104 log-files [ErrorLog] TestMode=0 Diagnostic=1 ; ;End of configuration file

109 ; FUNCTIONAL PRINCIPLES AND COMPONENT BASIC CONFIGURATION 109 Algorithm 8.2 Example of CM.ini configuration file with IEC protocol ; ; ; CM104 IEC on TCP/IP Configuration File ; ; Hardware with 4 serial ports (Serial 1 to 4) an 2 Ethernet ports ; Serial 1 is used as operating interface ; Serial 2 is reserved for redundancy link in case of redundant cm solution ; for non redundant CM this interface can be used to connect also a Modbus device ; Serial 3 not used ; Serial 4 not used ; ;Note: ;Unnecessary blocks, parameters and comments must be commented out by ; at the beginning of a line ; Serial operating terminal / remote terminal on SERIAL [OPERATE] Hardware =CM104 PortAdr =0x3F8 Irq =4 RtsCts =2 Baudrate =57600 ;the default Baudrate is baud unless otherwise specified - data frame is fixed (8-N-1) ; ; Redudancy Connection on SERIAL [SYNCHRONISATION] PortAdr =0x2F8 Irq =3 RtsCts =2 Baudrate = ;the default Baudrate is baud unless otherwise specified - data frame is fixed (8-N-1) ; ; Plant bus specific configuration [H1] SubUnit=B ;Specify whether the CM is non-redundant or part (A or B) of a ;redundant machine. Here non-redundant mode has been specified. ;SubUnit=Single ;SubUnit=A ;SubUnit=B Mac = ;The MAC address of the CM ( see: node setup) ;This is an example of an Ethernet address (MAC address) typical of TXP Exclude = ; Here the MAC address of the DS670 of TELEPERM XP conntected to the plant bus must be ; specified. This is an example of an address. ApCmTimeout =10 ;Specify the maximum time in seconds that may pass without data having been received ;and before the CM signals a failure of the AP-CM connection. ;If no time is specified twice the configured receive cycle time is used. ;After expiration of the ApCmTimeout the process image of the AP is deleted as well. ;SaveEsFiles =2 ;SaveEsFiles=2 causes seq-files received from the ES to be saved on the Flash. This ;setting has a major impact on the time response and therefore may be used for testing purposes only NoTimersStamp =1 ;if NoTimeStamp=1 all time tags are generated in the CM. No time tags ;will be taken over. If you change the setting you again need to ;transfer the configuration to the CM. ; Info String [DISPLAY] Info =CM104_UNIT_B_IEC60870_5_104_TCP/IP ;Display/info about the display of a string in HMI. ;(The first blank marks the end of the string) ; Error logging [ErrorLog] FileSize =40000 ;FileSize specifies the maximum size of the files ErrLog01.txt and ErrLog02.TXT in bytes TestMode =1 ;If TestMode=1 has been specified, no alarms will be written into the file Errlog01.txt and ErrLog02.txt ;Diagnostic =1 ;Errors detected during the runtime are saved in the file ;ErrLog01.txt and ErrLog02.txt on the Flash during error logging. ;Logging may only be activated during the commissioning phase since ;each write access to the Flash will have a major impact on the real-time ;response of the system. Active error logging may result in sporadic ;faults of the H1 communication! ; Subsystem connection Settings ; ; CM-PC Configuration File ; ; ;adress from cm104 [TCP/IP] ip = netmask = ;router = hops = 42 ; ; [IEC] RED_MODE = 3 ;

110 8.2. MAINTENANCE Maintenance 8.3 Component Backup and Recovery principles All engineering data in regards of the CM module are located on the ES 680 station. Therefore recovery involves good backup procedures of the ES 680 and changing hardware parts if neccessary. However there is a configuration file CM.ini which is essential for proper function of the CM module and therefore as such must be backed up. 8.4 References CM modules - TXP manual collection Communication to CM module - transfer CM.ini

111 Chapter 9 C490-95F - protection system This chapter is addressing 95F - protection system. It is not addressing 95F engineering nor exact configuration of this processor. It is addressing typical maintenance task as debuging during the startup. It is explaining function of particular trip circuits and adding some important information as documented on various sites. Finally practical use of the COM 95 software is being demonstrated. Following is being addressed: Turbine protection system principles Discussion about particular trip circuits like overspeed protection system TXP protection circuits *EZ* diagrams Practical use of the COM 95 software is being demonstrated This section will be carried as follows: Location: Training center The explanation of the protection system as described above Location: PCC Physical use of COM 95F software 111

112 9.1. ON-THE-JOB TRAINING PROTECTION SYSTEM AND TXP COMMUNICATION On-the-Job training Protection system and TXP communication Control and protection systems brief description The gas turbine control and protection system contain the following subsystems: Gas Turbine governor (SIMADYN) DCS control system (TXP680) Protection system (95F) Startup Frequency Controller and Static excitation unit (SFC) Unit protection OM system (OM650) Auxiliary diagnostic sub-systems WIN_TS (turbine stress monitoring) Flight recorder (fault recorder) Gas Turbine governor (SIMADYN) This subsystem is responsible for load and speed control of the turbine DCS control system (TXP680) This system is responsible for: Communication between subsystems SFC system 95F protection system Ball valves control system Communication to GE system over modbus Simadyn communication OM650 subsystem Open loop control Collecting I/O signals Motor logic

113 9.1. ON-THE-JOB TRAINING PROTECTION SYSTEM AND TXP COMMUNICATION Protection system (95F) This system is responsible for safe operation of the machine and its protection if critical values are reached: Hardware based over-speed protection Software based over-speed protection Flame monitoring Manual turbine trip 1 Fire protection trips 2 Surge protection Unit protection Protection circuit (protection signals coming from TXP system) If conditions are in boundaries the protection system allow: opening ESV 3 valves permission to open Ignition Valves It is necessary to remind you that Protection System never initiates any startup commands. All commands are given by TXP system and releases to manipulate outputs are given if process values are in safe boundaries Startup Frequency Controller and Static excitation unit (SFC) This sub-system is responsible for: machine startup generator excitation Unit protection This sub-system is responsible for electrical protection of generator OM system (OM650) This system is interface for humans (MMI - Man Machine Interface) and it is made of the following sub-systems: OT - Operation Terminal - this function is sometimes called MMI PU - Process unit - this unit is responsible for: 1 Red buttons 2 Blue buttons 3 Emergency Shut-off Valve

114 9.1. ON-THE-JOB TRAINING PROTECTION SYSTEM AND TXP COMMUNICATION 114 Communication to the AP s (Automation processor). This function is called ASR Short term data storage Long term data storage - LZA Process calculation - calculation of running hours, counters Description database - BDM - Assigning KKS and Description to all system tags ES - Engineering station - here is stored all configuration for AP s and only place where such configuration can be modified. This station is available via OM menu. When ES and PU are running on one computer then this computer is called CU (Compact Unit). Both computers are processing data is parallel. Only one computer is executing command and is called Master. In case of Master failure the computer called Slave become master and continue in operation - this make smooth operation possible. No data will be lost or operation interrupted.

115 9.2. SYSTEMS INTERCONNECTION Systems interconnection Figure 9.2.1: Schematic interconnection of the systems - Please note that this is ONLY schematic and plant configuration must not be exactly the same as shown in this diagram. Exact plant configuration is shown in topology diagram which can be found in appendix.

116 9.2. SYSTEMS INTERCONNECTION 116 Figure 9.2.2: Topology Diagram - Network Infrastructure

117 9.3. DISTRIBUTION OF THE SYSTEMS OVER CABINETS Distribution of the systems over cabinets Table 9.1: Distribution of the systems in cabinets Cabinet System 05CJR01 05CJP01 05CJP01 05CJP02 05CJP02 05CJP41 05CJQ01 05CJT01 05CHA01/02 05CPA01~03 Gas Turbine governor (SIMADYN), Flight recorder, Plant clock DCS control system (TXP680) AP11 DCS control system (TXP680) AP12 DCS control system (TXP680) AP13 Communication center, modbus interfaces Protection system (95F) Over-speed protection, flame detectors, vibration sensors interface, flow computers SFC and excitation unit Unit protection ET200 station, DCS I/O system 9.4 Protection system description in detail 1. Hardware based over-speed protection 2. Software based over-speed protection 3. Flame monitoring 4. Manual turbine trip 4 5. Fire protection trips 5 6. Surge protection 7. Unit protection 8. Protection circuit (protection signals coming from TXP system) Hardware based over-speed protection There are two sets of the O/S (over-speed) protection. The set on the top consists of three input conditioners Braun E1518 and three relays Braun E1553 together with external relay logic shown below. This set is called hardware based over-speed protection. After energization or trip it is necessary to press button T2 which will energize relays BA75.1 and BA83.1 and BA91.1. If there is no trip then relays BA75.3 and BA83.3 and BA91.3 get energized and over diodes will energize BA75.2 and BA83.2 and BA91.2 which will make sure that relays relays BA75.3 and BA83.3 and BA91.3 stay energized after button T2 is released. If any channel is tripped then appropriate relay Baxx.3 get de-energized and will generate signal XK6x over its contact. 4 Red buttons 5 Blue buttons

118 9.4. PROTECTION SYSTEM DESCRIPTION IN DETAIL 118 Example: Cause: MBA10CS102 XK61 trip Effect : Relay BA83.3 will lost supply and will de-energize Signal XK62 will be set to 0 The speed signals are controlling directly relays K2, K3, K6 and K7 in the 95F cabinet and together with the K1 and K5 are forming hardware based O/S protection of the turbine. If relays K2, K3, K6 and K7 get de-energize then power supply for the solenoid valves of the ESV valves lost energy and those as such close. The set on the button is performing software based O/S protection and its outputs are going directly to the 95F system where they are evaluated and processed accordingly. Table 9.2: List of sensors for hardware based over-speed protection KKS Note 05MBA10CS101 05MBA10CS102 05MBA10CS Over-speed system diagnostic The system is containing two frequency simulators for diagnostic. Each simulator is emulating the TRIP speed and NOT TRIP speed. All tests are managed by the 95F system and any discrepancy cause trip of the turbine and failure of O/S system test. If any sensor of three in each set is not working properly then test will not be invoked and test will not be completed O/S system REAL test During the commissioning over-speed test was conducted which is made by setting trip speed to 700rpm and machine is started up by SFC to the washing speed and on 700rpm must be tripped.

119 9.4. PROTECTION SYSTEM DESCRIPTION IN DETAIL 119 Figure 9.4.1: Schematic of hardware over-speed

120 9.4. PROTECTION SYSTEM DESCRIPTION IN DETAIL 120 Table 9.3: List of sensors for software based over-speed protection KKS Note 05MBA10CS104 05MBA10CS105 05MBA10CS Software based over-speed protection There are two sets of the O/S (over-speed) protection. The set on the bottom consists of three input conditioners Braun E1518 and three relays Braun E1553. Output of these signals is send directly to the 95F as binary inputs Flame monitoring Flame monitoring system make sure that there is not injected fuel into combustion chamber without being burned. If both flame detectors detect NO FLAME in either combustion chamber than ESV valves will be closed. Table 9.4: List of sensors for flame monitoring KKS Note 05MBM11CR101/102 05MBM21CR101/ Manual turbine trip Signal processing modules are located in cabinet 05CJQ01 Signal processing modules are located in cabinet 05CJQ01 The manual trip buttons yellow are located all around the plant and in central control room. There are slight differences of the system behavior if is tripped by yellow or blue button. Please refer to the operating manual for more details. All buttons also contain signal which indicate on TXP system the exact location of the button pressed. Table 9.5: List of buttons for manual turbine trip KKS Note 05MYBGS020A/B Fire protection trips System can be tripped by: All buttons are in serie, thus only two signals are generated; each button contain three signals - one is used for location identification. Fire alarm system in case of fire detection Manually by pressing blue button - buttons are located all around the plant and in main control room

121 9.5. PROTECTION SYSTEM REDUNDANCY AND RELIABILITY 121 Table 9.6: List of fire alarm trip signals KKS Note Blue buttons All contacts are in parallel and in each button are four contacts - one is location signal 05MBY00EY011 / 012 / 013 There are three signals from fire alarm panel Surge protection There are three differential pressure switches on the turbine compressor to make sure that the compressor does not surge. If two out of three signals detect compressor surge than ESV valves will be closed and turbine will stop immediately. Table 9.7: List of the surge protection signals KKS Note 05MBA11CP001 / 1 / 2 / Unit protection This signal is generated by unit protection sub-system in case of serious electrical fault in generator circuit. Please refer to the unit protection sub-system documentation for more details Protection circuit (protection signals coming from TXP system) There are many more trip signals which are processed by DCS system and which may trip turbine. Those signals are send over six binary signals. Those signals are tested automatically before each turbine start and may be tested manually as well by Test button located in protection system cabinet. 9.5 Protection system redundancy and reliability To ensure reliability of the system the following measures are taken: Input sensors are configured as 2 out of 3 logic 2 out of 2 logic CPU (95F) is designed as two independent high-reliability redundant unit Non-Coincidence alarms In case that there is more than one signal in the different to the others noncoincidence logic is implemented. What does non-coincidence alarm mean? Let s say we have three signals in surge protection. If one signal out of three showing trip and two others not than we get non-coincidence alarm. In other words non-coincidence alarm is generated when signals do not have the same value at the same time.

122 9.5. PROTECTION SYSTEM REDUNDANCY AND RELIABILITY System auto-diagnostic As mentioned earlier many sub-systems contain diagnostic signals over-speed protection diagnostic sub-system trip circuit diagnostic ET200 subsystem health monitoring units synchronization check 95F self-diagnostic sub-system Over-speed protection diagnostic sub-system The over speed hardware is tested every hour by testing frequency and prior to the start Trip circuit diagnostic Trip circuit is tested for functionality prior to every start of the turbine ET200 subsystem health monitoring System has implemented Health signals for various parts of the I/O subsystem to make sure that AP (automation processors) are alive and their I/O subsystems is working properly Units synchronization check The unit A and B exchange variety of hardware signals which monitor mainly outputs. This makes sure smooth start-up of every unit and health monitoring. In other words unit A is monitoring unit B and vice versa F self-diagnostic sub-system System performs complete self-diagnostic. All systems parts are checked out at least once an hour. Any problem result switching of the unit into the diagnostic mode. 6 In this mode there are various possible actions for outputs: outputs are passivated outputs are set to some pre-defined value. Passivation / Depassivation Passivation means that output are set inactive. If this happens on one unit ONLY then the turbine will continue running. A lot of error messages will be sent to the TXP system. While error is rectified then outputs have to be de-passivated. De-passivation can be made by pressing button located on the upper-front side of the cabinet. 6 Yellow LED is on

123 9.6. TROUBLE SHOOTING CPU (95F) is designed as two independent highreliability redundant unit Every unit is built from two CPU units which are synchronized over synchronization bus. Figure 9.5.1: Each PLC is build from two CPU units. 9.6 Trouble shooting Start-up problems There can be problems during start-up sequence. Usual problem is that ESV valves does not open. What could go wrong during startup: 1. Low pressure in feeder line 2. No flame detection 3. Not a correct speed There could be many other reasons why protection system does not release ESV valves. Error log messages have to be carefully examined and analyzed with the help of operating manual.

124 9.6. TROUBLE SHOOTING Problems during running machine Every error message from protection system has to be carefully examined and rectified as soon as possible in order to prevent unnecessary trip of the machine.

125 9.7. INSTALLING 95F SOFTWARE ON THE PG Installing 95F software on the PG How to install 95F software on new PG 1. Copy software to the PG (over network or over the floppy) 2. Now install it to the directory you like (for example d:\asir ) 3. Edit file Gt05abpx.ini and file gt05bpx.ini as shown on alg. 9.1 on the following page and 9.2 on page 127. You need to place correct paths to that file. If you do not wish to modify these files manually follow points 4 to 7 listed below. 4. Create new project Project ---> Set. By pressing F4 you can move around in the tabs. 5. Go to the tab number 5 (Options) and by pressing F3 choose right directory (d:\asir) 6. Set correct files and directories in ALL tabs - mainly Program file in tab number 2 and Symbols files and Assignment list in tab number 3 Please remember that there are slight differences between unit A (located on the top of cabinet rows A and B) and unit B (located on the top of cabinet rows D and E)!!! 7. Save a project by pressing ENTER and selecting File ---> Project ---> Save As The above is illustration how to do it in case you did not edit files as above. Because you prepared your files you can simply press F10 and open init file as you prepared above F testing mode Download code to 95F in testing mode Now we should make sure that file in PG is same to the content of the RAM (EPROM) in 95F system. 1. Go to project ---> set (tab number 1 ) PLC and by selecting mode and pressing F3 go online. Keep in mind if you are working with unit A or unit B!!!! 2. Now you can go to File -> Blocks -> Compare and compare PLC to the program file while selecting in Selection Block list?a?? means ALL blocks. There will be differences in following blocs: DB 100 Block does not exist in program file DB 200 Block does not exist in program file DB 252 Block does not exist in program file DB 253 Block does not exist in program file DB 254 Block does not exist in program file FB 230 Block does not exist in program file FB 231 Block does not exist in program file FB 232 Block does not exist in program file

126 F TESTING MODE 126 Algorithm 9.1 S5 configuration files to be edited - ORIGINAL of the file [Project] Version= [OnlineSettings] PathFile=C:\STEP5\S5_HOME\@@@@@@AP.INI PLCMode=0 PLCInterface=0 PLCInterfacePar=COM1: Standard ChangeOnline=4 AttrPLCMode=0 Pathoptions=0 PathName= PLCChangeProgFileUpdate=0 [BlocksSettings] ProgFile=D:\STEP5\S5_DATEN\ASIR_BIS\BISHA05\GT-PROT\02_GT05\GT05A0ST.S5D ProtProgFile=0 Represent=2 Address=0 Comments=1 Checksum=0 DocBlockMode=0 FBFXPreHeaderMode=0 [SymbSettings] SymbIniFile=D:\STEP5\S5_DATEN\ASIR_BIS\BISHA05\GT-PROT\02_GT05\GT05A0Z0.INI SymbSeqFile=D:\STEP5\S5_DATEN\ASIR_BIS\BISHA05\GT-PROT\02_GT05\GT05A0Z0.SEQ Symbols=1 SymbDisplay=0 SymbLength=16 SymbCommLength=40 ProtSymbFile=0 ProtSeqFile=0 [DocSettings] PrinterFile=C:\STEP5\S5_HOME\PT10Q8DR.INI OutFile=D:\STEP5\S5_DATEN\ASIR_BIS\BISHA05\GT-PROT\02_GT05\NONAMELS.INI FooterFile=D:\STEP5\S5_DATEN\ASIR_BIS\BISHA05\GT-PROT\02_GT05\GT05A0F2.INI DocCommFile=D:\STEP5\S5_DATEN\ASIR_BIS\BISHA05\GT-PROT\02_GT05\GT05A0SU.INI CharSetASCII=0 PrinterInterface=0 Footer=1 DocumOutFile=0 [EpromSettings] SysidFile=D:\STEP5\S5_DATEN\ASIR_BIS\BISHA05\GT-PROT\02_GT05\NONAMESD.INI Prommer=0 EpromMode=1 [OptionsSettings] QuestEnd=1 QuestProjSave=1 Warning6x=0 Sorting=1 SortOrder=0 ProjFilesChangeLock=0 [STLBatchSettings] STLSourceFile=D:\STEP5\S5_DATEN\ASIR_BIS\BISHA05\GT-PROT\02_GT05\NONAMEA0.SEQ

127 F TESTING MODE 127 Algorithm 9.2 S5 configuration files to be eddited - MODIFIED file [Project] Version= [OnlineSettings] PathFile=C:\STEP5\S5_HOME\@@@@@@AP.INI PLCMode=0 PLCInterface=0 PLCInterfacePar=COM1: Standard ChangeOnline=4 AttrPLCMode=0 Pathoptions=0 PathName= PLCChangeProgFileUpdate=0 [BlocksSettings] ProgFile=D:\asir\GT05A0ST.S5D ProtProgFile=0 Represent=2 Address=0 Comments=1 Checksum=0 DocBlockMode=0 FBFXPreHeaderMode=0 [SymbSettings] SymbIniFile=D:\asir\GT05A0Z0.INI SymbSeqFile=D:\asir\GT05A0Z0.SEQ Symbols=1 SymbDisplay=0 SymbLength=16 SymbCommLength=40 ProtSymbFile=0 ProtSeqFile=0 [DocSettings] PrinterFile=C:\STEP5\S5_HOME\PT10Q8DR.INI OutFile=D:\asir\NONAMELS.INI FooterFile=D:\asir\GT05A0F2.INI DocCommFile=D:\asir\GT05A0SU.INI CharSetASCII=0 PrinterInterface=0 Footer=1 DocumOutFile=0 [EpromSettings] SysidFile=D:\asir\NONAMESD.INI Prommer=0 EpromMode=1 [OptionsSettings] QuestEnd=1 QuestProjSave=1 Warning6x=0 Sorting=1 SortOrder=0 ProjFilesChangeLock=0 [STLBatchSettings] STLSourceFile=D:\asir\NONAMEA0.SEQ

128 F TESTING MODE 128 FB 233 Block does not exist in program file FB 234 Block does not exist in program file FB 235 Block does not exist in program file FB 236 Block does not exist in program file FB 237 Block does not exist in program file FB 238 Block does not exist in program file FB 240 Block does not exist in program file FB 241 Block does not exist in program file FB 242 Block does not exist in program file FB 243 Block does not exist in program file F overall reset Manual reset With manual reset all data in both units are deleted. 1. Set both switches RUN/STOP in STOP position 2. Switch OFF both units 3. Remove batteries from both units 4. Switch ON both units 5. Insert batteries 6. Switch ON both units If EPROMS are not inserted system is in TEST mode > download code into the PLC. 1. File -> Transfer -> Blocks and in selection write A (means all). Some blocks cannot be transfered: FB 255 OB 31 DB 1 2. Do same for BOTH PLC?s (unit A tier A,B ;and unit B tier D,E) 3. After loading is necessary to restart unit by cycling switch run to stop and back to the run.

129 F TESTING MODE 129 Figure 9.8.1: Description of the 95F testing mode from the manual

130 F FAULTS RECTIFYING 130 Table 9.8: Errors reported by 95F program Error description Byte Nr. Signal Group Nr. 48: Module wrongly configured in DB : Module wrongly configured in DB : Module wrongly configured in DB : Module wrongly configured in DB F faults rectifying Rectify MYB00EU111B fault The above mentioned error was detected on the 95F equipment. Cross reference list shown that this error is set by output Q64.6 which is set in PB210:21 by bit F5.5 coming out of the logic in PB210:19. In that logic it was found that bit is set by F 72.6 which means PASSIVATED SG14. The same error could be seen in I/O externals errors which shown that the slots 10, 14, 22 were passivated. On the figure on the following page can be seen slot numbering for bisha06 project. The problem was wrongly located switch on the card located on slot Diagnostic mode after 95F restart The 95F is entering into the diagnostic mode immediately after its restart. The overall reset has been performed and software newly loaded - problem prevail. The COM 95F diagnostic program is reporting errors mentioned in table 9.8. Solution All cards have been tested in second (properly working) 95F rack DB1 has been verified Software configuration has been reloaded after reset Finally it has been found that bus connector linking external I/O card is not properly connected. Connector has been reconnected and the system is working again without problems. 7 7 See 95F manuals link located in Sagunto PR64 directory or search for manuals with order number 4NEB and manual for the software COM 95F with order number 6ES MF23

131 F FAULTS RECTIFYING 131 Figure 9.9.1: Slot numbering for 95F - 06CJQ41

132 9.10. REFERENCES References Maintenance of 95F Simulations 95F COM 95F - programming software for 95F U - S5 Programmable Controller Manual For 95U

133 Part III Cybersecurity 133

134 134 The idea is to address questions what to do about networking safety. This section to be ommitted due lack of sufficient knowledge of the audience. This section require some networking knowledge and hands on experience.

135 Part IV Questions to be answered 135

136 136 Day one 1. How does the single fault tolerant for OSM ring structure works (make a sketch)? 2. What will happen if BDM manager of master SU fails? 3. What does SAMPLER command do? 4. What does rdb command do? 5. What is meaning of InfFb.proj file? 6. Where (on which path) diagnostic files are located? 7. What is the command to know the status of the OM system? 8. What should be the SCSI id of DAT tape? 9. What does Sig.Attach command do? 10. What is the configuration file for Web4TXP and where it is located? 11. Where is the location of OM licensing file and how it s named? Day two 1. Describe the hierarchy level where we do the engineering for HW & SW? 2. What does crontab l command do? 3. What information does semaphore directory reveals? 4. Where is the night backup file saved? And what does actually it saves? 5. What does the tar t, tar c tar x commands do? 6. How to identify and unlock FUP diagram? 7. What types of errors are monitored by DS? Day three 1. How to configure bridge OSM? What steps are needed? 2. Why do we disable time sync on Bridge OSM? 3. How to configure RM mode on OSM? 4. Describe S5 AP rack structure used in Sugen plant. 5. From where Thin Client takes time? 6. What s the use of batteries on AP rack? 7. How to extract PBP file from ES680?

137 What are the steps to do the engineering of Analog signal on ES680? 9. How to configure IM308 memory card? 10. What will happen if Hopf Clock fails? Day four 1. What is the purpose of failsafe protection system & how it s communicating with AP? (describe communication and physical hardware involved) 2. Describe general hardware structure of 95F. 3. What is the purpose of CM104 system & how to take its backup? 4. Describe how signals flow from vibration sensors to AP. 5. Describe general function of Argus system.

138 List of Figures Main T2000 components are shown on this figure Sugen computer naming concept The plant bus and terminal bus are configure as a ring Sugen computer naming concept The infrastructure of the OM system distributed over several components Similar error might be found in error log files when there is an issue with the OM license Interface for the OM configuration files generation and transfer OM system status File structure of typical OM650 component OM system status PU unit output of the /etc/dfspace command OM system status File structure of typical OM650 component OM system status SU unit output of the /etc/dfspace command Screenshot from the SU-B server - system messages SU unit system log files indicates problems with storage media Identify the process number - result of Step 4 commands File structure of typical OM650 component Edit OM_USER section add for each user application name Edit OM_CLIENT section add for each computer the line with IP address and user name which will be used Edit ES_USER section add for each user application name Edit ES_APPL section add for each computer the line with IP address and user name which will be used OM system status PU unit output of the /etc/dfspace command Common acknowledge function Typical structure of the AP processor Physical AP rack Page from TXP manual describing the errors status LEDs of the IM 308 card CM 104 drawing

139 LIST OF FIGURES Schematic interconnection of the systems - Please note that this is ONLY schematic and plant configuration must not be exactly the same as shown in this diagram. Exact plant configuration is shown in topology diagram which can be found in appendix Topology Diagram - Network Infrastructure Schematic of hardware over-speed Each PLC is build from two CPU units Description of the 95F testing mode from the manual Slot numbering for 95F - 06CJQ