SOP-RTMKTS.0060.0005 - Test and Approve Operations Software Applications Contents 1. Objective... 2 2. Background... 2 3. Responsibilities... 3 4. Controls... 4 5. Instructions... 5 5.1 Document Operations Software Application Problem... 5 5.2 Revise/Develop Operations Software Applications... 7 5.3 Develop Operations Software Application (Tool) Test Plan... 9 5.4 Test Operations Software Application (Tool)... 12 5.5 Review ISO Staff Testing Process... 14 6. Performance Measures... 15 7. References... 15 8. Revision History... 15 9. Attachments... 16 This document is controlled when viewed on the ISO New England Internet web site. When downloaded and printed, this document becomes UNCONTROLLED, and users should check the Internet web site to ensure that they have the latest version. In addition, a Controlled Copy is available in the Master Control Room procedure binder. Rev. 2 Page 1 of 16
1. Objective 2. Background The objective of this procedure is to outline minimum acceptable testing required prior to using Operations Software Applications (tools) (i.e., Excel spreadsheet used as calculator tools) developed by the ISO New England (ISO) Operations Support Services (OSS) staff. The Operations Software Applications (tools) tested by this procedure include the maximum voltage transfer limit calculators for the following interfaces: Connecticut Southwest Connecticut New England - New Brunswick. The Operations Software Applications (tools) are complex tools developed by OSS engineers to aid day ahead planning and real time operations staff in the daily planning and operations of the New England Transmission System by providing interface voltage limits for some of the major New England interfaces. These tools contain complex algorithms that must be fully tested before they become a production tool. These tools all must be designed with system operations and key operating procedures in mind, such as ISO New England Operating Procedure No. 19- Transmission Operations (OP-19) and Master/Local Control Center Procedure No. 15 - System Operating Methodology (M/LCC 15). Rev. 2 Page 2 of 16
3. Responsibilities Any North American Electric Reliability Corporation (NERC) Certified System Operator, certified at the Reliability Coordinator (RC) level, has the authority to take action(s) required to comply with NERC Reliability Standards. 1. The OSS Engineer developer is responsible for: Developing new and revising existing Operations Software Applications (tools) Developing a plan to test the limits produced by Operations Software Applications (tools) Updating the design requirements documentation required to adequately test any Operations Software Application (tools) in the EMS environment 2. The Power System Model Management (PSMM) personnel or designee is responsible for verifying: Data inputs and outputs between the Excel calculator and the EMS are in sync The Operations Software Application (tools) architecture meets both of the following: o Applicable internal and external IT standards o Any appropriate security requirements 3. The ISO Manager, Control Room Operations (or designee) and OSS staff (or designee) are responsible for developing an acceptable test plan that verifies the Operations Software Application (tool) functionality and, by comparing limit trends (positive or negative) assesses the algorithm that calculates the limit for various data input values. 4. To verify Operations Software Application (tool) consistency and accuracy the ISO Operations Control Room Operations staff, OSS staff and Power System Model Management (PSMM) personnel are responsible for comparing the limits between the Operations Software Application (tool) and the EMS environment set up to test the limits. Rev. 2 Page 3 of 16
4. Controls 1. The Director, OSS and the Director, Operations (or designees) designate ISO personnel to perform functions pertaining to the product test for any Operations Software Application (tool) employed in Day Ahead planning and Real-Time operations. 2. The assigned Control Room personnel and OSS staff communicate test results of any Operations Software Application (tool) problems found during testing to the Operations Software Application (tool) developer or Real Time Support oncall engineer by: Taking a screen shot of the input and output conditions Providing a written description of the conditions leading up to the discovered problem (e.g., load level, generation dispatch etc.). 3. When a problem is found after deployment of the Operations Software Application (tool), the Real Time Support on-call engineer is notified. 4. If the Real Time Support on-call engineer is not able to resolve the problem found in the Operations Application Software, the Real Time Support on-call engineer provides limits to the Day Ahead and Control Room Operations staff as required until the Operations Software Application problem is resolved. Rev. 2 Page 4 of 16
5. Instructions 5.1 Document Operations Software Application Problem Operations Software Applications (tools) developed by OSS staff are complex algorithms that: Cover combinations of operating conditions that include the following: o system load o area load o generation dispatch o reactive resource availability o interchange levels Are used by the Control Room System Operators, Forecasters, Outage Coordinators and Day-Ahead Market staff to efficiently make operating decisions related to reliable system operation in accordance with OP-19 and M/LCC 15. Operations Software Application development entails complex programming that captures many independent variables that are combined with mathematical modeling of the inter-relationships to develop both first and second contingency limits. Operations Software Application (tool) revisions may also be required when OSS staff observes unexpected software behavior that affects the functionality and accuracy of the existing Operations Software Application (tool). 1. When a discrepancy in an Operations Software Application (tool) is discovered by any member of the Control Room Operations staff, OSS group staff or Power System Model Management (PSMM) personnel during testing of a new or revised Operations Software Application or anytime during Real Time operation, that member contacts the Real Time Support on-call engineer and discusses the problem. 2. To assist the Real Time Support on-call engineer in the timely resolution of the issue, any member of the Control Room Operations staff, OSS group staff, or Power System Model Management (PSMM) personnel discovering the discrepancy provides detailed documentation of the problem, including any related screenshots and the system conditions. Rev. 2 Page 5 of 16
The Real Time Support on call engineer or OSS software application developer maintains the Change Request System that tracks the problem or issue until is resolved. 3. Following notification of a problem, the Real Time Support on-call engineer documents the problem in the Change Request System. 4. The Real Time Support on-call engineer assesses the problem and determines if the Control Room Operations staff can continue using the Operations Software Application (tool) or if the Control Room Operations staff ceases using the Operations Software Application (tool) until the problem is resolved. A. If the Real Time Support on-call engineer determines the Operations Software Application (tool) cannot be used, the Real Time Support on-call engineer provides limits to the applicable Day-Ahead staff and the Control Room Operations staff until the Operations Software Application (tool) problem is resolved. 5. If a new or a revised Operations Software Application (tool) is required, the Real Time Support engineer or software application developer documents the new Operations Software Application (tool) requirements consistent with the following applicable ISO New England procedures: INFTCH.0010.0020 - Develop Test Plan and Conduct Testing INFTCH.0030.0020 - Review and Monitor Change Requests INFTCH.0030.0030 - Manage Change Development and Resolution. INFTCH.0030.0040 - Manage Emergency Migrations or Outages INFTCH.0030.0050 - Manage Change Deployment MNGPJT.0007.0010 - Identify Business Requirements Rev. 2 Page 6 of 16
5.2 Revise/Develop Operations Software Applications Significant changes in the bulk power system network and load may require the OSS staff to develop a new Operations Software Application (tool) or revise an existing an Operations Software Application (tool). It may require many months of off-line studies by personnel internal and external to ISO to develop a new Operations Software Application and/or revise an existing Operations Software Application (tool). These off-line studies may involve personnel from the following groups: ISO Market Operations ISO Planning ISO Power System Modeling, ISO Control Room Operations staff Local Control Center affiliates. Any Operations Software Application (tool) development or revision is to be performed in a manner that verifies the resulting Operations Software Application (tool) conforms to all ISO New England Information Technology policies related to: Change Management (CM) review and approval of the software changes Documentation of software development and testing that is in accordance with IT procedures 1. The Real Time Support on-call engineer obtains and verifies that the Operations Software Application (tool) business requirements are documented and approved by the business owners prior to any software development or revision. A. In order to avoid potential testing failures due to omissions or gaps in the design requirements that can potentially result, the Real Time Support oncall engineer determines if there are any omissions or gaps in the design requirements by: (1) Comparing the test plan against the design requirements before developing the Operations Software Application (tool) test plan. (2) The Real Time Support engineer reviews the test plan with the developer and business owner and verifies all the omissions or gaps in the design requirements are closed prior to starting the Operations Software Application (tool) test. Rev. 2 Page 7 of 16
2. When a new Operations Software Application (tool) or a revision to an existing Operations Software Application (tool) is required, the Real Time Support engineer or software developer performs the following: Software application (tool) development entails complex programming that captures many independent variables that are combined with mathematical modeling of the inter-relationships to produce both first and second contingency limits. A. Develop the new Operations Software Application (tool) or revise the existing Operations Software Application (tool). Even when an unplanned emergency arises that requires either a revision of an existing Operations Software Application (tool) or development of a new Operations Software Application (tool), coincident with a compressed time frame, an Operations Software Application (tool) test plan is still required. B. Go to Section 5.3 and develop an Operations Software Application (tool) test plan that validates the business and technical requirements as documented in the business requirements document. Rev. 2 Page 8 of 16
5.3 Develop Operations Software Application (Tool) Test Plan 1. When revision to or development of an Operations Software Application (tool) is required, the OSS engineer develops an Operations Software Application (tool) Test Plan that validates existing business and technical design requirements. Even when an unplanned emergency arises that requires either a revision of an existing Operations Software Application (tool) or development of a new Operations Software Application (tool), coincident with a compressed time frame, an Operations Software Application (tool) test plan is still required. The Operations Software Application (tool) test plan is a roadmap or prescriptive sequence of conditions that the test employs to determine if the software tool will perform as designed 2. When a new Operations Software Application (tool) or a revision to an existing Operations Software Application (tool) is required, the OSS engineer or software developer performs the following concurrently: A. Develops a test plan that validates the technical requirements based upon the approved business requirements are: (1) Archived in accordance with IT policies (2) Located in the ISO Change Request Database B. If the functional requirements are not available in the Change Request Database, the OSS engineer obtains documented and approved functional or design requirements from the business owner(s). 3. The OSS engineer verifies the new Operations Software Application (tool) or revised Operations Software Application (tool) conforms to its functional or design requirements by designing a test plan based on the functional or design requirements of the new or revised Operations Software Application (tool). A. The test plan simply verifies specific actions or functions that the business owner expects to see when parameters are entered into the Operations Software Application (tool). Rev. 2 Page 9 of 16
4. To avoid testing errors due to omissions or gaps in the functional or design requirements that can result in testing failures, the OSS engineer performs the following: A. Determines if there are any omissions or gaps in the functional or design requirements before developing the test plan by performing the following: (1) Review the functional or design requirements with the Operations Software Application (tool) developer and verify there are no omissions or gaps in the functional or design requirements prior to starting the test. 5. The Real Time Support on-call engineer or software developer uses the following sources when developing the test plan: A. Study results B. Knowledge of existing software issues (Change Request Database) that require resolution C. All available business requirements. 6. The Real Time Support on-call engineer or software developer verifies the test plan: A. Tests each input parameters independently and over each input parameter range of acceptable input values: (e.g., between the New England load level minimum and maximum) B. Includes boundary parameter testing to confirm output and compare against expected outcomes. Unlike a static test which is done manually, a dynamic test allows for more extensive testing by employing a program (automated test script) that runs through all defined scenarios automatically and records the results. These automated custom scripts are maintained by the OSS engineers and are archived with the test results in the Change Request database. 7. Whenever possible, the OSS staff tester employs dynamic testing to conduct the assessment of the Operations Software Application (tool) functionality. A. Standard software application test scripts maintained by OSS staff can be used as supplemental data to fulfill a Product Test within the Change Management cycle. Rev. 2 Page 10 of 16
B. Standard software application test scripts allows consistent testing of software applications from cycle to cycle to confirm testing consistency with: (1) Each new/revised Operations Software Application (tool) released (2) Archival software application performance testing C. The Real Time Support on-call engineer verifies the new/revised Operations Software Application (tool) test plan requirements are established and the test plan is ready for use. Rev. 2 Page 11 of 16
5.4 Test Operations Software Application (Tool) A new or revised Operations Software Application (tool) requires thorough testing before its use as either of the following Operations Software Applications (tools): A stand-alone Operations Software Application (tool): e.g., as in the case of an Excel spreadsheet. A fully integrated Operations Software Application (tool) within a larger Operations Software Application (tool), e.g., a form, as in the case of voltage calculator spreadsheets residing within the DOUBLC application of the EMS software. 1. A complete product test of the revised Operations Software Application (tool) is performed in accordance with an approved Operations Software Application Test Plan. A. The approved Operations Software Application Test Plan verifies that the revised Operations Software Application (tool): (1) Works as expected (2) Meets all design specifications (3) Satisfies the Operations Software Application (tool) test requirements. 2. Once the test plan has been developed, the Real Time Support on-call engineer verifies that an OSS staff member other than the Operations Software Application (tool) developer/reviser and a member of the Control Room Operations staff are assigned to test the new or revised Operations Software Application (tool). A. The assigned Control Room Operations personnel and the OSS staff conduct their tests independently and compare their findings. (1) The tester enters each condition described on the test plan into the Operations Software Application (tool) and records the outcome. (2) Since the Operations Software Application (tool) is also employed in the real time environment, the tester compares the outcome of the Operations Software Application (tool) against the results observed from the EMS. Rev. 2 Page 12 of 16
To facilitate efficient product testing of a new/revised or existing Operations Software Application (tool), automated test scripts are used to verify a larger number of scenarios when compared to a manual test script. Standard software application test scripts maintained by OSS staff can be used as supplemental data to fulfill a Product Test within the Change Management cycle. Standard software application test scripts allow consistent testing of software applications from cycle to cycle to confirm testing consistency with: o Each new/revised Operations Software Application (tool) released o Archival software application performance testing B. When appropriate and to allow more extensive testing, the OSS staff tester employs custom automated scripts to test all of the Operations Software Application (tool) functionality. C. During testing, if a discrepancy is discovered by any tester (i.e., the Control Room staff, OSS staff or Power System Model Management (PSMM) personnel, the tester contacts the Real Time Support on-call engineer to discuss the discrepancy. (1) To assist the Real Time Support on-call engineer in the timely resolution of the discrepancy, the tester develops detailed documentation of the problem including any related screenshots of the system conditions when the discrepancy was found. D. If the new/revised Operations Software Application (tool) test results in one or more testing requirement failures, the Operations Software Application (tool) is not ready for deployment and the Real Time Support -on-call engineer verifies the following actions are taken: (1) CR staff, OSS staff or Power System Model Management (PSMM) personnel immediately contacts the Real Time Support on-call engineer to discuss the problem (2) The new/revised Operations Software Application (tool) is revised as necessary to eliminate the testing requirement failure(s) (3) The new/revised Operations Software Application (tool) is retested. E. When the new/revised Operations Software Application (tool) testing requirements are satisfied, the Real Time Support on-call engineer emails notification to the applicable Day Ahead staff and Real Time Control Room Operations staff that the Operations Software Application is ready for use. Rev. 2 Page 13 of 16
5.5 Review ISO Staff Testing Process To comply with INFTCH.0030.0030 - Manage Change Development and Resolution requirements, the principal Operations Software Application (tool) developer documentation is subject to an independent peer review by another qualified member of OSS staff. 1. The OSS staff member assigned as the independent peer reviewer participates in both of the following: A. Code Review of the Operations Software Application (tool) algorithmic structure and design. 1) The Real Time Support engineer participating in the Operations Software Application (tool) code review performs the following: a. Verifies code has been properly documented showing the intent of each major component. b. Verifies Assumptions have been properly documented c. Limits the code review to small and manageable pieces of the code d. Reviews the data flow ( inputs and outputs) and identifies any potential data flow issues between the various parts of the application e. Identifies key characteristics that appear repeatedly and verifies they are carried out consistently throughout the application B. Testing of the finished Operations Software Application (tool) including all test documentation required to meet the Change Management cycle requirements. 2. Since the Operations Software Application (tool) integrates into the EMS, a qualified IT representative tests the Operations Application Software (tool) in its final resident environment and validates that the input and output variables align properly. 3. Since the ISO Control Room Operations staff and OSS staff are the ultimate end users of the Operations Software Application (tool), the Control Room Operations and OSS staff or designees participate in final testing prior to full acceptance of the Operations Software Application (tool). If any discrepancies are discovered, the Control Room Operations staff and OSS staff provides written notification to the software developer and Real Time Support on-call engineer for remediation. Rev. 2 Page 14 of 16
6. Performance Measures 1. This procedure is deemed to be properly followed as evidenced by the following: Code Review Results Unit Test Results Operations Software Application (tool) Test Results Documentation of any issue during production and notification to on-call engineer. 7. References ISO New England Operating Procedure No. 19, Transmission Operations (OP-19) Master/Local Control Center Procedure No. 15 - System Operating Methodology (M/LCC 15) INFTCH.0010.0020 - Develop Test Plan and Conduct Testing INFTCH.0030.0020 - Review and Monitor Change Requests INFTCH.0030.0030 - Manage Change Development and Resolution. INFTCH.0030.0040 - Manage Emergency Migrations or Outages INFTCH.0030.0050 - Manage Change Deployment MNGPJT.0007.0010 - Identify Business Requirements 8. Revision History Rev. No. Date Reason Contact 0 11/29/10 Initial draft procedure Dean LaForest 1 11/26/12 Biennial review by procedure owner; Headers, updated copyright date & added Process Title; 1 st page Footer, deleted disclaimer 2 nd paragraph; Step 3.3, modified; Step 4.1, use acronym OSS and corrected punctuation; Globally minor grammar and format changes; Step 5.3.6.A, deleted the e.g., Min and Max Load MW values ; Dean LaForest Rev. 2 Page 15 of 16
Rev. No. Date Reason Contact 2 11/05/14 Biennial review by procedure owner; Globally removed the use of the terms shall and ensure per a directive from ROC and made the required editorial and grammar changes; Section 2, added M/LCC 15; Globally updated and made editorial changes required to be consistent with current practices and management expectations; Section 5.1, 1 st, 2 nd bullet added M/LCC 15; Section 7, added M/LCC 15 Dean LaForest 9. Attachments None Rev. 2 Page 16 of 16