Light Water Reactor Sustainability Program. Computer-Based Procedures for Field Workers: Results From Three Evaluation Studies

Transcription

1 INL/EXT Light Water Reactor Sustainability Program Computer-Based Procedures for Field Workers: Results From Three Evaluation Studies Johanna Oxstrand Katya Le Blanc Aaron Bly September 2013 The INL is a U.S. Department of Energy National Laboratory operated by Battelle Energy Alliance

2 INL/EXT Light Water Reactor Sustainability Program Computer-Based Procedures for Field Workers: Results From Three Evaluation Studies Johanna Oxstrand Katya Le Blanc Aaron Bly September 2013 Idaho National Laboratory Idaho Falls, Idaho Prepared for the U.S. Department of Energy Office of Nuclear Energy Under DOE Idaho Operations Office Contract DE-AC07-05ID14517

3 DISCLAIMER This information was prepared as an account of work sponsored by an agency of the U.S. Government. Neither the U.S. Government nor any agency thereof, nor any of their employees, makes any warranty, expressed or implied, or assumes any legal liability or responsibility for the accuracy, completeness, or usefulness, of any information, apparatus, product, or process disclosed, or represents that its use would not infringe privately owned rights. References herein to any specific commercial product, process, or service by trade name, trade mark, manufacturer, or otherwise, does not necessarily constitute or imply its endorsement, recommendation, or favoring by the U.S. Government or any agency thereof. The views and opinions of authors expressed herein do not necessarily state or reflect those of the U.S. Government or any agency thereof.

4 EXECUTIVE SUMMARY The Computer-Based Procedure (CBP) research effort is a part of the Light- Water Reactor Sustainability (LWRS) Program The LWRS program is a research and development (R&D) program sponsored by Department of Energy (DOE) and performed in close collaboration with industry R&D programs that provides the technical foundations for licensing and managing the long-term, safe, and economical operation of current nuclear power plants. One of the primary missions of the LWRS program is to help the U.S. nuclear industry adopt new technologies and engineering solutions that facilitate the continued safe operation of the plants and extension of the current operating licenses. The nuclear power industry is highly proceduralized, i.e. almost all activities that take place at a nuclear power plant are conducted by following procedures. The paper-based procedures (PBPs) currently used by industry have a demonstrated history of ensuring safety, however there is room for improvement in the use of procedures. The industry can increase efficiency and safety by taking advantage of technological advancements, such as replacing the PBPs with Computer-Based Procedures (CBPs), which may increase usability and allow human performance aspects to be integrated into the procedure. CBPs offer the option to move towards more dynamic procedure presentation. For example, the CBP system can display only the relevant steps based on operating mode, plant status, and task at hand. Additionally, the incorporation of advanced technology, such as CBP systems, may help to manage the effects of aging systems, structures, and components. Additionally, the introduction of advanced technology may also make the existing light-water reactor fleet more attractive to the future workforce. However if the user interface of a CBP is designed poorly, it may create new problems that do not exist with PBPs. This may eliminate the advantages of CBPs, or worse, make CBPs less effective than PBPs. In order to ensure that advantages of technology are achieved without introducing unintended consequences, research needs to be conducted to understand how technology changes the work process. Additionally, design concepts for CBPs need to be tested in realistic settings in order to determine which aspects of the interaction are undesirable and which ones actually improve performance. The long-term goal for the present research effort is to develop guidance the nuclear industry can use in their discussions with potential CBP system vendors. The guidance will primarily focus on how to best design the graphical user interface. Additionally, the guidance will define the necessary structure and content of procedures as they are transferred from paper-based documents to data that can be used by an electronic CBP system. This research effort is an iterative process where the human factors issues related to CBP usage are systematically addressed and evaluated. This research effort is focused on how to improve the efficiency, productivity, and safety of procedure usage by employing CBPs instead of PBPs. This report describes the research activities conducted to meet the DOE milestone M2LW-13IN Complete evaluation of final LWRS II&C Computer-Based Procedure prototype for field workers. ii

5 A model of procedure usage and the initial set of design requirements were used as the basis for the development of the first CBP prototype, which is used for demonstration and evaluation of the design concepts. The concepts demonstrated in this research effort are intended to be platform-independent, which allows the nuclear utilities to decide what they want to use for their CBP system. So far, versions of the CBP prototype have been developed for Apple ios and Android systems. Four studies to evaluate the design concepts and the effect the CBP system has on human performance have been planned and three of these evaluation studies have been conducted to date. The first study was conducted in an electrical training facility at a collaboration partner s utility. Thirteen technicians ran through a procedural scenario with both the CBP prototype and the traditional PBP. The researchers compared performance when operators were using the CBP to performance using the PBP. They also collected quantitative and qualitative usability data. The results from the study indicate that mobile devices with CBP software can be used to execute procedures in the field. The study also indicated that context sensitivity and simplified step logic are desirable features in a CBP system (100% of the participants thought so). The second study was conducted at a flow loop training facility at a collaboration partner s utility. Nine field operators and one trainer participated in the study. The procedure selected for the study was an existing procedure used to train field operators. The scenario used in the study involved initiating the cold water system and the control loop system. The scenario used in this study was more complex and included the presence of multiple conditional statements and branching to other procedures. The results indicate that CBPs may be effective in enhancing human performance. There were fewer overall deviations when the procedure was executed using the CBP than with the PBP and a greater number of non-recovered deviations when the procedure was executed using the PBP. CBPs may help operators to catch potential errors and prevent them. Results also indicate that operators are likely to readily accept CBPs. The majority of participants reported that they preferred the CBP over PBPs. They rated the CBP as highly usable. They unanimously preferred the dynamic context-sensitivity of CBPs to static PBPs. The third study had a slightly different focus and was therefore carried out a bit differently. The objective was to conduct a more focused evaluation of the user interface and usability of the CBP system rather than comparing the CBP with a PBP version of the same procedure. The third study was conducted during a utility working group meeting held at Idaho National Laboratory in August Due to conducting the study in a meeting environment rather than in a training facility at a utility, the researchers made a mock-up of the plant system used in the scenario. The mock-up consisted of a physical mock-up of pipes and valves, digital mock-ups of controllers and a tank, and a large photo of the actual system in the plant. The researchers observed each participant and made notes about design features that seem to cause issues for the participant. The results were consistent with previous evaluation studies, indicating that users prefer context sensitivity and simplified step logic; however some items need further investigation. Examples of such items are; showing versus hiding information iii

6 that is not applicable to the current conditions, displaying step numbers, and which steps require correct component verification. The results from the three evaluation studies indicate that the CBP prototype supports operator performance of procedural tasks. In all three studies, users were able to use the CBP to successfully complete a procedural task. In the second study, the CBP supported performance better than the PBP for the same task (as indicated by a fewer number of overall deviations with the CBP). The results of the studies also indicate that the human-system interface for the CBP prototype is intuitive and usable. In all of the studies operators were able to use the prototype to execute a procedure with minimal training (at most, 30 minutes). Additionally, users generally rated the CBP as highly usable in the three studies (and the usability increased with each iteration of the prototype). In general, the results indicate that the CBPs should always present contextsensitive information if it is available. Based on feedback from the participants in the three studies it is acceptable to ask the user to provide the context (e.g., what is the task? what are the conditions?) if it cannot be retrieved automatically from a data base. Additionally, simplifying the step logic of conditional statements is a desirable function. There are still some items that need further investigation and validation. One of these items are how to balance automatically guiding the operator through the task (i.e., not burden the operator with complex logic and branching) while still allowing him/her to be engaged in the procedure. Eliminating the appearance of branching, i.e., seamless transition to other procedures, can potentially enhance the efficiency of procedural tasks but it may also reduce the amount of contextual cues related to the task at hand. The researchers will continue to investigate how to best design the CBP to support the balance between automatic guidance through the task and operator engagement. The researchers have planned a fourth evaluation study that will be conducted at a collaboration partner s utility. The remaining issues will be addressed in a final prototype and investigated in a final evaluation study to prepare for the field evaluations. Moving forward, the research team aims to validate design of the CBP system in field evaluations. The underlying information architecture, i.e. the technology used to transfer the information the paper-based procedures to the format used by the CBP system will be developed. The team will use the data structure to transfer one or multiple nuclear power plant field procedures into the CBP format. The validation study will be conducted by having the field operators use the CBP system in the plant for the selected procedures for an extended amount of time. The procedures selected for the field evaluations will be low risk and high frequency data points such as frequency of use, time to execute procedure, path through procedure, etc. will be collected automatically in the background while the procedure is performed by the field worker. The result of the validation study will be incorporated in the design guidance for CBP system for field procedures as well as being important input to the work with CBPs for control room procedures. The design concepts identified when studying CBPs for field procedures are thought to be generalizable to a large extent. Hence, it is the concepts will be useful when transferring other procedures used in other organizations at a nuclear iv

7 power plant to a CBP system. However, there might be some differences in design and interaction with the procedure that need further investigation to ensure that the CBP design adequately supports the users. The researchers aim to investigate which of the design concepts for CBPs are transferrable to the design of CBPs for control room operators as well as identifying areas where the design either has to be altered or new design concepts need to be developed. These potential alterations or new design concepts will be developed and evaluated. The evaluation of design concepts will be both conducted in smaller scale studies and in the DOE Human Systems Simulator Laboratory at INL. v

8 ACKNOWLEDGEMENTS The authors would like to express special gratitude to the following people for their collaboration and support of this research effort. Michael Grigsby, Palo Verde Nuclear Generating Station, for being the Palo Verde Management Sponsor for the research effort. Carlos Williams, Regina (Gina) Cunningham, and John Triolo, Palo Verde Nuclear Generating Station, for all their support in the process of identifying requirements and procedures as well as providing valuable feedback in the prototype development process. Thomas Waicosky, Catawba Nuclear Station, for his support in the process of identifying requirements, prototype development, and for hosting the evaluation study conducted in November The research team also wants to extend its gratitude to all the participants and others involved in the successful execution of the November 2012 evaluation study. vi

9 CONTENTS EXECUTIVE SUMMARY... ii ACKNOWLEDGEMENTS... vi ACRONYMS... ix 1. INTRODUCTION General LWRS and CBP background Background Project Overview Purpose and Goals Collaboration Partners Field Procedures versus Main Control Room Procedures The Process Model of Procedure Usage Initial Requirements General Design Concepts Dynamic Context-Sensitivity Maintain Focus on the Task Reduce Operator Burden Underlying Data Structure TASK FLOW CHARACTERISTICS Action Step Action Verb Conditional Step Time Dependent Steps Continuously Applicable Steps Multiple Action Steps Concurrent Verification, Independent Verification, and Peer-Checking Placekeeping Correct Component Verification Notes, Cautions, and Warnings Supplemental Information and Attachments Hierarchical Step Structure Bulleted Steps Hold Points Branching Steps Procedure Specific Information EVALUATION OF CONCEPTS Prototyping of the Computer-Based Procedure System Iterations of Design Decisions Evaluation Studies Evaluation Study vii

10 3.2.2 Evaluation Study Evaluation Study Evaluation Study 4 To be Conducted General Discussion of Results of Evaluation Studies PATH FORWARD REFERENCES FIGURES Figure 1. Research Activities Conducted To Date... 3 Figure 2. Examples of Embedded Job Aids and Computational Aids Figure 3. Examples of the Initial Version's Overview and Single Step View Modes Figure 4. Example of Simplified Step Logic Figure 5. Example of Dynamic Context Sensitivity When Decisions Are Made Before Initiating Procedure Figure 6. Examples of Contingencies Figure 7. Example of Multiple Action Step Figure 8. Examples of Placekeeping Figure 9. Correct Component Verification Using Barcodes Figure 10. Example of Failed Correct Component Verification Using Manual Input Figure 11. Example of a Note Embedded in a Step Figure 12. Example of supplemental material Figure 13. Example of Bulleted Step Presentation Figure 14. Example of Presentation of Procedure-Specific Information Figure 15. The Use of the Prototype in the First Study Figure 16. A demonstration of the Use of the Second Version of the Prototype Figure 17. The Physical Mock-Up Used in Study Figure 18. Performer Conducting Valve Line-Up Figure 19. Correct Component Verification and Displaying of Tank Level Calculation Figure 20. Relationship Between the Evaluation Studies and Design Guidance TABLES Table 1. Summary of How The CBP System address Error-Prone Aspects of Procedure Usage Table 2. Task Flow Characteristics Addressed in the Prototype Development Table 3. Usability Sub Scores From Study viii

11 ACRONYMS CBP CCV DOE HTML INL LWR LWRS N/A NASA OCR PBP R&D RDBMS SD TLX XML U.S. Computer-Based Procedures Correct Component Verification Department of Energy Hypertext Markup Language Idaho National Laboratory Light-Water Reactor Light-Water Reactor Sustainability Not Applicable National Aeronautics and Space Administration Optical Character Recognition Paper-Based Procedures Research and Development Relational Database Management System Standard Deviation Task Load Index Extensible Markup Language United States ix

12 Computer-Based Procedures for Field Workers: Results From Three Evaluation Studies 1. INTRODUCTION 1.1 General LWRS and CBP background The Computer-Based Procedure (CBP) research effort is a part of the Department of Energy (DOE) sponsored Light-Water Reactor Sustainability (LWRS) Program conducted at Idaho National Laboratory (INL). The LWRS program is performed in close collaboration with industry research and development (R&D) programs to provide the technical foundations for licensing and managing the long-term, safe, and economical operation of current nuclear power plants. One of the primary missions of the LWRS program is to help the U.S. nuclear industry adopt new technologies and engineering solutions that facilitate the continued safe operation of the plants and extension of the current operating licenses. One way that introducing advanced technology can enhance competitiveness and efficiency in the existing light water reactor (LWR) fleet is to transition from paper-based procedures (PBPs) to computerbased procedures. The commercial nuclear power industry is highly procedure driven and nearly all activities conducted in a nuclear power plant are executed using written procedures. In the existing LWR fleet, those procedures are executed using a paper-based process. Though the paper-based process has an established history of ensuring safe operation, advances in digital technology offer an opportunity to further enhance safety through migration of the current process to Computer-Based Procedures. CBPs have the potential to enhance performance in a variety of ways including: Enforcing procedure adherence expectations Automatic place keeping Providing dynamic context-sensitive information Embedding relevant process information (Fink et al. 2009) Although many researchers agree that CBPs may enhance performance, the impacts of introducing advanced technology into a historically paper-based process are not well understood. The introduction of digital systems has often changed operator tasks in unanticipated ways, resulting in new types of errors. A successful migration to CBPs will include the aspects of the current paper-based process that are necessary to provide sufficient support to the operator, while removing aspects of the current process that have negative effects on efficiency and safe operation. To make the migration successful, the process also has to be adapted so that the benefits that come with using technology are fully realized. For example CBPs could integrate plant information (including plant mode, equipment status, and procedure-relevant plant parameters) and make it readily available to plant personnel in real time. CBPs with this enhanced functionality will be dynamic and context-sensitive, affording the opportunity to mitigate many of the issues associated with static paper-based procedures (PBPs). However if the user interface of a CBP is designed poorly, it may create new problems that do not exist with PBPs. This may eliminate the advantages of CBPs, or worse, make CBPs less effective than PBPs. In order to ensure that advantages of technology are achieved without introducing unintended consequences, research needs to be conducted to understand how technology changes the work process. Additionally, design concepts for CBPs need to be tested in realistic settings in order to determine which aspects of the interaction are undesirable and which ones actually improve performance. This report describes the concepts used in the CBP prototype designed and developed by the research team, how the concepts are implemented in the CBP prototype, and the execution and results from three evaluation studies. The report also describes future work, such as a fourth evaluation study, validation 1

13 studies to be conducted in actual nuclear power plant (not in a training facility), and the investigation of how to best implement the CBP design concepts for hybrid main control rooms Purpose and Goals 1.2 Background Project Overview The purpose of the Computer-Based Procedure research effort is to define the human factors and human performance requirements related to the design and use of CBPs. The long-term goal is to develop design guidance that nuclear power plants can use in their discussions with potential CBP system vendors. The two primary areas for the guidance are: The design of the interaction between the human user and the CBP system, including the graphical user interface of the CBP system, and The design of the underlying data structure, which will be needed when PBPs are converted to an electronic medium. The goal of the effort is to summarize all lessons learned and insights gained based on the results of experimental research investigating several iterations of CBP prototypes into one design guidance document. The guidance developed will have a strong empirical basis that utilities can use in their discussion with vendors of CBP systems. This current report describes the research activities conducted to meet the DOE milestone M2LW- 13IN Complete evaluation of final LWRS II&C Computer-Based Procedure prototype for field workers Collaboration Partners There is an apparent disconnect between existing CBP research and the potential end users. This may explain why a large amount of research has been conducted on CBPs, yet CBPs have not been implemented in U.S. nuclear power plants. In order to ensure that this CBP research effort meets the needs of the nuclear power industry, the research team has worked in close collaboration with the U.S. commercial nuclear power industry. During the course of the current CBP research effort the INL team has worked mainly with the South Texas Project, Duke Energy, and Arizona Public Service. Furthermore, most U.S. nuclear utilities have been involved in one way or another in the current research effort, e.g., the user needs assessment and in the LWRS Computer-Based Procedure Special Interest Group. The researchers have chosen to collaborate with multiple utilities to ensure that the research benefits the entire industry rather than a single utility. Another benefit of collaborating with multiple utilities is that it enables the researchers to develop guidance that addresses the standardization of content and quality of procedures, which will serve to enhance quality and efficiency Field Procedures versus Main Control Room Procedures Most existing research in the area of CBPs has focused on CBP systems for operations in the main control room, and more specifically on emergency operating procedures. There has been very little research that focuses on CBPs for field workers and how to these field organizations can increase their efficiency and improve human performance. The term field workers is used to describe field operations, chemistry operations, maintenance, and any other organization that might exist in the nuclear power plant that conducts procedure-driven tasks out in the plant. Utilities have expressed a need for research tailored to the specific needs of field workers, because the limitations of PBPs are particularly relevant for the activities conducted out in the field. For example, if a field worker encounters unexpected conditions in the field, he may need to contact a supervisor to resolve the issue. In the current process, that may mean several trips to and from the field, and many hours of waiting. The time it takes to resolve the issue could be drastically reduced with real-time communication. 2

14 The terms field worker, field operator, and operator are used interchangeably throughout this report. 1.3 The Process Five activities have been conducted previously in the CBP research effort; a qualitative study, the development of a model of procedure usage, the identification of requirements, the development and evaluation of an initial CBP prototype, and the development and evaluation of a revised CBP prototype. The first three activities are described and discussed in detail in Computer-Based Procedures for Field Workers in Nuclear Power Plants: Development of a Model of Procedure Usage and Identification of Requirements (Oxstrand & Le Blanc, 2012). The development and evaluation of the initial prototype are documented in Evaluation of Computer-Based Procedure System Prototype (Oxstrand, Le Blanc, & Hays, 2012). The development and evaluation of the revised prototype are described in Evaluation of Revised Computer-Based Procedure System Prototype (Oxstrand, Le Blanc, & Fikstad, 2013). A brief discussion of the main conclusions from each activity is provided below. A third iteration of prototype development and evaluation has been conducted since the publication of the Oxstrand et al. (2013) report. This third study is described in section Evaluation Study. Characterization of Procedure Usage Model of Procedure Usage Identification of Requirements Development and Evaluation of Initial Prototype Development and Evaluation of Revised Prototype Qualitative study Identification of actions involved A general set of requirements for CBPs for field operators Simple scenario (one procedure and one component) More complex scenario (multiple procedures and components) Mapping of information and task flow Identification of factors affecting the risk of human errors Execution of step from Single Step Mode Simplified Graphical User Interface User needs assessment Overview Mode Execution of step from Overview Mode Decision points before entering the procedure Figure 1. Research Activities Conducted To Date Model of Procedure Usage The researchers visited a nuclear power plant to gather the information needed to construct a model of procedure usage. The model was constructed based on on-the-job observations of field workers, structured interviews, and focus groups. The model describes what an operator does in the process of executing a single procedure step. The model also describes how the operator accomplishes his goal of successfully completing the procedure driven activity, and the cognitive aspects that affect success. The model was used to define the error-prone aspects of procedure following. The researchers then developed requirements based on inferences about how the CBP prototype could mitigate the error prone-aspects of paper-based procedure following. A summary of the operator actions describe in the model, the factors that affect the error likelihood, and how the issues are addressed in the CBP system is provided in Table 1. 3

15 Table 1. Summary of How The CBP System address Error-Prone Aspects of Procedure Usage. Action taken by operator (from Model) Read procedure step. Am I on the right step? How the operator accomplishes action Locate placekeeping function on page in the procedure. Factors that affect error likelihood Visibility of placekeeping function. Solution in the CBP system Automatic placekeeping and active step is highlighted saliently. Do I understand step? Do other steps depend on this step? Develop action plan/strategy. Locate component. Are the initial conditions as expected? Right location, label, and name? Predict expected system response. Execute step. Evaluate the situation when the expected response was not achieved due to incorrect execution of the step. Identify actions to be taken and compare actions to mental model. Identification of cues that signal dependency (i.e., warnings and cautions). Compare expected conditions to actual conditions. Compare location, label, and equipment description in procedure to those on the actual equipment. Compare expected conditions to actual conditions. Understanding of constrained language, complexity of step logic, and the accuracy of mental model. Existence and visibility of cue that signal dependency. Accuracy of mental model that dictates expected conditions and detection of cues that indicates departure of expected conditions. Ability to correctly read both the procedure and the equipment information and the ability to integrate the values/text read and compare them accurately. Accuracy of mental model that dictates expected conditions and detection of cues that indicate departure of expected conditions. Only presenting steps that are relevant for the current conditions, reduce step logic complexity, and provide easy access to supplemental information. Provide a visual cue for dependency and provide the ability to look ahead. Provide a visual illustration of expected conditions, partially automate assessment of initial conditions, and provide just-in-timetraining. Automate correct component verification with barcode scanning or optical character recognition. Provide a visual illustration of expected initial conditions and provide just-in-timetraining. 4

16 Evaluate the situation when the expected response was not achieved even though the step was executed correctly. Check for indications that there are more parts to the step (e.g., was the current action a substep?). Existence and visibility of indications that signal the step is not complete. Automatic placekeeping that warns the field worker when she/he attempts to proceed without completing a step. Is the step complete? Check for indications that there are more parts to the step (e.g., was the current action a substep?). Existence and visibility of indications that signal the step is not complete. Automatic placekeeping that warns the field worker when she/he attempts to proceed without completing a step. Proceed to the next procedure step Initial Requirements The model of procedure usage, especially the conditions for success or failure, was used to identify an initial set of minimum requirements needed to be fulfilled when designing computer-based procedures for field workers. Examples of these requirements are; guide field worker through the logical sequence of the procedure, ease the burden of placekeeping for the field worker, make the action steps distinguishable from information gathering steps, alert the field worker to dependencies between steps, and ease the burden of correct component verification for the field worker. 1.4 General Design Concepts The model of procedure usage helped researchers to identify several high-level design concepts that need to be incorporated into all of the CBP prototypes Dynamic Context-Sensitivity One of the main benefits of using digital technology to present the procedure is the dynamic nature of the technology, especially compared to procedures printed on paper. The digital technology provides the capability to display the parts of the procedure that is relevant to the operator for the specific task at hand and the current state of the equipment and system. This dynamic context sensitivity makes it possible to only present relevant steps based on user input, plant state, operation mode, etc. The PBPs currently used in the industry are written in a manner that encompasses every foreseen situation, which at times makes the procedure quite large. When the operator is assigned to perform a task in the plant, it is most likely only a small part of the procedure that is needed to conduct the task. Before the operator heads to the work site he/she and a supervisor have to decide which parts of the procedure that are relevant for the task at hand. Important information regarding the task at hand is communicated during the pre-job brief. Items such as what equipment to take action on (e.g., Start Pump A, not Pump B), what plant conditions might influence the task (e.g., other procedures in progress), and the current operation mode are communicated to the operator. This information usually affects the path the operator have to take through the procedure and which other procedures that potentially have to be used to complete the task. By utilizing dynamic context sensitivity, the CBP system will guide the operator thought the correct path in the procedure based on the information given during the pre-job brief. The CBP system will ask for information such as the task is to start pump A (hence, no actions will be taken on equipment related to starting Pump B) before the instruction part of the procedure is initiated. Based on the user input, the CBP system would automatically disable the steps related to start Pump B. 5

17 Some conditions are not feasible to be captured in the pre-job brief. These are conditions based on what the operator sees in the field while conducting the task, e.g., the reading of a tank level. The CBP system will update the path through the procedure based on the value the operator inputs after reading the tank level. The CBP system presents the procedure as a list of steps. If a transfer to another procedure or to another section in the current procedure is needed based on user input or other input to the CBP system, the system will simply add the steps needed in to the initial list of steps. Compared to the use of PBPs the operator does not have to get the next procedure out from his or hers back pocket and find the right section to transfer to. Hence, the CBP system will help the operator stay focused on the task at hand rather than finding the correct steps in multiple procedure documents. One question related to dynamic context sensitivity that has been investigated by the researchers is if it is possible and/or feasible to incorporate real-time plant status into the procedure. Digital technology allows for the CBP system to communicate with databases that keep track of other procedures in use, current plant status, and operation mode, all of which have potential to affect the path thought the procedure. Having access to this type of information would allow for continuous updates of the procedure in real time based. This capability would also provide the opportunity to track the operators status on the specific task in real time. This type of real time task tracking would be to great benefit during outages where every minute is important. However, this kind of real time updates requires wireless network in the nuclear plants. Currently, there are very few utilities that have wireless networks in their plants and it seems as the implementation of wireless networks in the remaining plants is a low priority activity. Something that is more in reach for the current fleet of LWRs is near time information. This can, for example, be achieved by using wireless hot spots or connecting stations throughout the plant Maintain Focus on the Task Another important general design concept is to keep the operator focused on the task at hand at all times rather than focusing on the CBP system itself. In general, this means keeping the operator interaction with CBP device to a minimum, so he/she can focus on successfully executing the tasks. Examples of how this is accomplished include simple navigation schemes, requiring a minimum number of actions to execute the step, and easy access to supplemental information when needed Reduce Operator Burden The reduction of operator burden is a question of how to balance automation and decision support with operator engagement and the procedure execution process. The high-level solution to the question is to always provide a mean to easily get information to the operator about completed steps, steps marked not applicable (i.e., N/A d steps), future steps, decisions made that influenced path through the procedure, etc Underlying Data Structure A significant challenge to shifting from PBPs to CBPs is the need to convert existing paper-based procedure documents into data that can be used by CBP system. One important issue is how to structure that data so that design concepts can be realized, and particularly so that it can be used to produce context-sensitive procedures. Another issue is the need to ensure that the procedure data can be used by a variety of systems. The advent of the relational database management system, commonly referred to as RDBMS, revolutionized data storage. And then another revolution evolved with the Internet and the hypertext markup language or HTML. Documents could be sent around the world and read on almost any type of machine imaginable as long as it had a screen and a web browser. HTML had its roots in a more general markup language called standard generalized markup language but gave rise to the idea of a more extensible language, dubbed the extensible markup language or XML. 6

18 In the beginning of XML many thought the new standard might replace RDBMS altogether. That has proven to be incorrect and the two standards have co-existed for some time. Why is this when they seem to solve the same problem of storing textual data for retrieval from a computer system? The answer is simple; they don t solve the same problem. The key difference between RDBMS and XML is the relationship structure. A RDBMS stores data in a flat hierarchical model using tables and unique keys to for relationships between data stores. This structure is very flexible in its ability to graph textual data and tables together. XML on the other hand is by nature uses a hierarchical model resembling a tree. There are no inherent keys. Rather each textual node in XML may have multiple children nodes that in turn have their own children nodes. Systems that require high volume and fast transactions do well with a RDBMS backend. Systems that require defined objects benefit from the nature of XML structures. Using either of these systems for CBPs has tradeoffs, and ultimately only one can be used. In any given procedure the operator is required to perform some operation. On paper these actions are presented as a series of steps. The current step may define which of the following set of steps to complete and which to ignore. The current step may also require the operator to repeat a series of previously performed steps. If the operator were to take a pen and draw a line between steps that relate to each other the result may look very much like a relationship graph with lines moving both forward and backward on the paper. A RDBMS would fit nicely in solving this hierarchical structure. But the analysis of a step is not complete. If we dig a little deeper the step is not as simple as text. A step may require input from the operator while reading an instrument. A choice may need to be made that would further define the current step. A visual verification may be required before proceeding with the step. A step may also have sub steps that must be completed before the entire step can be considered complete. In other words a step may have one or several attributes and sub steps. An XML Document could neatly represent steps for such a structure. Originally the choice seemed simple. Using the Rho Mobile framework the backend was provided for us in the form of a RDBMS. The choice was made and off we started like a sprint. Not too far into the project the problem became apparent. The user interface was more of a tree structure. Though our backend stored data with keys and values, the data had to be converted to a XML front end. It was no less complex than a two headed monster. It may have been a problem that could be accepted if we did not run into problems with multiple event loops from the native device and the Rho Mobile framework. A single user input was unable to register correctly on the device. A change had to be made. With the experience of developing on the Rho Mobile framework converting the program to a native application began with a solid understanding of what was needed in a strong backend. This time XML would be the obvious choice and one that would prove to be the right choice for us. A single step could be easily mapped with multiple attributes to an XML document that would then dynamically create the application. The main component of the XML document was the type attribute. Using a textual name to define types and combining that with a predefined user interface template the map from text to program was complete and operated with proper functionality on any single step. Since a step may influence the state of a following step other attributes could be used to define this functionality. Adding a serialize attribute and de-serialize element to the XML provided the means by which dynamic data could be stored on the device and retrieved by requiring steps when a choice caused the procedure state to change. However XML does not lend itself inherently to relationships outside of the parent child model. The extensible part of XML is more than adequate to make up for this shortcoming. Each step has an ID attribute that mimics the Key in a RDBMS. Any step could refer to a previous step by its ID grabbing serialized data or even resetting the procedure to a previous step. 7

19 When a step refers to a step other than itself it will always refer to a previous step rather than a following step. A computerized step can never say change the state of the following steps. A step simple changes its own data and then signals the system for an update. All steps receive the update request and search for data changes in any previous steps they may require data from. This form of interaction is often referred to as a PULL relationship as opposed to a PUSH relations ship. Data is pulled from the source rather than pushed to the destination. Modern web browsers work the same way requesting data from a server rather than being fed data directly. The CBP system uses the PULL relationship for a reason different then a web browser. As a procedure is being created it is logically created from the first step down to the last. As a step is entered into the computer it has no information regarding steps that may follow and as such cannot properly for a relationship with an unknown step. If the procedure does not contain evidence of a following step it then cannot create the PUSH logic that would be needed. However using a PULL relationship on previously created steps makes this a non-issue. With these elements in place the XML document is able to meet any requirements a CBP system could need. The structure is proper for handling string data and converting it into a computer program for dynamic procedure handling The XML schema Mobile devices with limited screen space are best approached with an organized table if the data is large and usage requires visually scanning through lists. In some instances a single step requires multiple actions, such as referring to an addendum as well as selecting from a list of choices. The XML schema must be generic enough to allow for a template approach to a step while being specific enough to manage the details a step. The XML must also be able to map to a user interface element with relative cohesiveness. At the top level of the CBP system is the document element. The document element contains sections as its children. Each section contains steps that map one to one with a procedure step. Each step contains views that allow for customization of a step s appearance. The final structure looks something like the following. <document> <section> <step> </step> <step> </step> </section> </document> Attributes are used to further define the elements. Each step element is required to have an ID attribute that consists of a single integer value. Each step element may also have a DependentStepID attributed. The DependentStepID attribute is used by the programs state manager to determine if the step should be visible or not. The value for the DependentStepID attribute is the step ID attribute of some previously defined step. The steps can display pre-determined views. Which views the step displays is dependent on the attributes the user chooses. The types of views that a step can have are: 8

20 StepNoteView StepTextView ScanView DecisionView InformationView InputView MultiInputView CalculationView MarkCompleteView The views will appear in the step in the order shown. The MarkCompleteView is a special case and the user will not interact much with it. The only view that would have a sub ID is the MutliInputView as this allows the user to have a combination of more than one input or decision in a single step. The ID for each input/decision is kept in order with the step ID + an underscore and then another integer. These IDs are auto-assigned but to use the data collected the user needs to be able to understand the ID if they want to retrieve the data in a subsequent step. A working example of the schema is shown here: <document> <section> > <step id= 1 StepText= Is the lab energized? DecisionText1 = "Yes" DecisionText2 = "No" DecisionValue1 = "Yes" DecisionValue2 = "No" HasDecisionView = true </step> <step id = 2 StepText = "Energize the lab by pushing \"AC MAINS\" on the Patch."; Filename = "a2"; InformationViewText = "REFER TO the image of the Patch Panel"; HasInformationView = true; DependentStepID = "1"; DependentSkipOutcome = "Yes"; > 9

21 </step> </section> </document> This gives two simple steps. Step 1 is a step asking for a decision to be made. Then Step 2 is hidden if the dependent step id, in this case step 1, is found and the outcome is the outcome specified. So if the user answers Yes then step 2 is hidden. Step 2 also shows the InformationView Template which allows the application to display help files or help text in a pop up. With the use of post definition references to create relationships between steps and a set of templates provided by an extensible schema the document author is able to quickly create a responsive application with enough detail to provide a computerized procedure for a wide range of operator duties Data structure - future work Of all the benefits to using an extensible framework like XML there are a few drawbacks that come into play when authoring a new procedure. An XML document requires the author to be familiar with a syntax they may not have much experience with. In addition to a new syntax, XML requires an author to manually type a lengthy document. Even if the author is familiar with XML mistakes such as out of order sequences and typos creates a need for testing the document and searching for errors. Steps should be made to mitigate the time requirements for learning this syntax and checking for document errors. Designing a tool for writing a procedure that removes the author from the XML document can avoid afore mentioned obstacles in XML creation. An example of a tool that is commonly used to create a behind the scenes XML document is Microsoft Word. Word doc files are nothing more than compressed XML files. The XML schema for word files fits a different purpose then the XML schema needed for computer based procedures and requires a custom tool to be developed for its specific needs. By creating a procedure writing tool an author is presented with an interface that ensures all the proper attributes and elements are presented in the XML document. While the user is presented with a clear input screen behind the scenes the tool can ensure that steps are identified and sequenced in their proper order. This is critical because missing elements will cause a step to be rendered with errors and an out of order sequence will cause the user interface to fail. The tool could not be as simple as a word processor without burdening the author with other syntactic requirements. More syntactic requirements would defeat the purpose of trying to remove the author from the xml document. While each step could be authored individually other input methods are needed to choose an interface template for the step and to define relationships between differing steps in the procedure. Relationships that need to be managed are repeated processes or a choice that requires moving forward in one branch of the procedure while ignoring the other. Most computer users are familiar with the concept of drop down boxes, input boxes, lists, and options. Using these elements in a tool to create an XML document will militate against errors that will otherwise be introduced into the document. It will also allow a user to create a document in less time without the need to verify the syntax is correct and the sequence is in order. 10

22 2. TASK FLOW CHARACTERISTICS The overarching design philosophies mentioned in the previous sections describe principles that, when followed for the CBP prototype, should enhance procedure use compared to PBPs. Table 1 illustrate specific ways in which the CBP prototype mitigates the error-prone aspects of procedure following. Though both the overarching design philosophies and the specific design characteristics described are intended to enhance performance compared to PBPs, CBPs also need to be designed to aid the operator in accomplishing the same tasks that a PBP does. To meet this need, the researchers identified the task-flow characteristics of procedures. The task flow characteristics define the elements of procedures that need to be present, regardless of what medium (paper or electronic) the procedures are presented on. This section describes how the task flow characteristics are incorporated into the CBP prototypes. The advanced features of the CBP prototypes are also illustrated in this section. 2.1 Action Step An action step is an instruction written in active voice that directs the operator to perform an action and contains an action verb and an object. An action step should be written in a manner that highlights the important information needed to correctly carry out the step. Some steps include calculations and/or reference to supplemental information. When using a CBP system computational aids and validation of results may be incorporated in the step. Job aids and other supplemental information may also be incorporated and hence be easily accessible when needed. Figure 2 shows one example of an embedded job aid and one example of a computational aid. Figure 2. Examples of Embedded Job Aids and Computational Aids. The initial prototype had two different presentation modes; overview mode and single step mode, see Figure 3. The overview mode listed all steps in the procedure. This mode provided a sense of where the operator was in the procedure, i.e., it provided the opportunity to look back at completed steps and look ahead to read future steps. The single step view presented one step at the time. As the default the current step was presented in this mode. However, the operator could step back or forward one step at the time in 11

23 the procedure from this mode. A One-click option was provided to easily get back to the current step. All actions in the procedure were conducted from the single step mode. Figure 3. Examples of the Initial Version's Overview and Single Step View Modes. The first evaluation study indicated that the operator were concerned that the single step mode might reduce the overall picture of the task at hand, i.e., that the single step mode potentially could lead to tunnel vision. The researchers therefore decided to investigate ways to make the overview mode more interactive and remove the single step mode. Both the second and third iterations of the prototype one have one presentation mode the overview mode. Each step in the procedure is conducted from this view. Context sensitivity for action steps was added in the following ways: Step presentation is based on task, conditions, mode, and other contextual factors, Decisions or calculations made in previous steps that affect subsequent steps are carried though, and All interface elements (buttons, menus, etc.) have context-sensitive information so that the operator. 2.2 Action Verb An action verb is a verb that directs the action within a step to be taken by the performer. According to the model of procedure usage, one of the questions the field operator has to ask is if he/she understands the step. This includes understanding of constrained language, complexity of step logic, and the accuracy of mental model. While conducting the qualitative study that led to the development of the model the researchers learned that there are two types of action verbs. The first two iterations of the prototype differentiated between action verbs that were aimed toward carrying out a specific action and action verbs that described a monitoring action. For example, verbs such as Perform, Ensure, Pull, etc. are action verbs where the operator is required to carry out a physical action. Verbs such as Check and Verify do not require an action to be taken. The action verbs that required action were clearly marked in purple. However, none of the two first evaluation studies were designed to explicitly test this feature. There was no indication that the users like this more or less than what they were used to from the current paperbased procedures, i.e., where there is no difference in how the two types of verbs are presented. 12

24 Therefore, the topic of how action verbs should be represented, and if differentiating between verbs that require action on plant equipment and verbs that require verification reduces error, needs further investigation. 2.3 Conditional Step A conditional step is an action step based on plant condition or combination of conditions to be satisfied prior to the performance of an action. One of the main design principles investigated by the researchers is the principle of simplified step logic. In other words, remove complexity from step descriptions by presenting IF/THEN, WHEN/THEN, AND, OR, etc. statements as simple questions to the greatest extent possible. For example, statements such as IF starting pump A THEN perform the following would be presented a What pump do you want to start; Pump A or Pump B? Depending on the answer the procedure will take the operator to either a step with the actions needed to start the pump A or the step with the actions needed to start pump B. All three iterations of the prototype use this simplified step logic. Figure 4 demonstrates an example of how a conditional statement is presented in the CBP system. Note how the next step in the list change based on the answer to the question. Figure 4. Example of Simplified Step Logic. Incorporated into the second iteration of the prototype is the option to make some decisions before initiating the procedure. Before a field worker goes out in the field and conducts a procedural task, he or she will have a pre-job brief with a supervisor. During the pre-job brief the task will be discussed and the procedure will be reviewed. Currently, during the pre-job brief the operator will mark any steps that are not applicable to the task at hand. The second version of the CBP prototype asks the operator a set of questions when the operator reviews the procedure needed for the task at hand. Based on the answers, the CBP system will automatically mark steps N/A d and take the operator down the path of relevant steps when the procedure is initiated, see Figure 5. Some of these decisions can be made by the supervisors or planner before the pre-job brief. Additionally, if the CBP system had access to current conditions the logic of these steps could be resolved before executing the procedure (and updated during procedure execution). 13

25 Figure 5. Example of Dynamic Context Sensitivity When Decisions Are Made Before Initiating Procedure. 2.4 Time Dependent Steps A time dependent step is a step to be completed within a specified time frame. None of the scenarios used for the evaluation studies conducted to date required time dependent steps. However, the researchers have developed concepts for how this would be handled. Time-dependent steps would be handled in much the same way a multiple action steps (see section 2.6). The operator would be provided with an indication that the following steps need to be performed within a certain amount of time and to verify equipment before they begin the actions in the step. 2.5 Continuously Applicable Steps A continuously applicable step is a step that is applicable over a period of time and requires periodic monitoring until a specific condition is met. The scenarios used in the two first evaluation studies did not require continuously applicable steps. Therefore, this specific characteristic was not evaluated in those studies. The procedure used in the third evaluation scenario had a couple of contingencies that were applicable throughout the whole scenario, see Figure 6 for examples. However, these contingencies were not incorporated in the scenario used in the third evaluation study. Hence, the design concepts for continuously applicable steps need further investigation and evaluation. 14

26 Figure 6. Examples of Contingencies. 2.6 Multiple Action Steps A multiple action step contains actions that are functionally related and have to be performed simultaneously to obtain a single result. This type of step was incorporated into the third version of the prototype. Before initiating a step with multiple actions the operator was informed about the order the steps had to be carried out and the fact that they have to be performed at the same time. Figure 7 provides an example of a multiple action step and the note which is embedded in the step. Figure 7. Example of Multiple Action Step. 15

27 2.7 Concurrent Verification, Independent Verification, and Peer- Checking Concurrent verification is a series of actions by two individuals working together at the same time and place to separately confirm the condition of a component before, during, and after an action, when the consequences of an incorrect action would lead to immediate and possibly irreversible harm to the plant or personnel. Independent verification is a series of actions by two individuals working independently to confirm the condition of a component after the original act that placed it in that condition. Peer-checking allows another individual to observe or check the work of a performer to ensure correct performance of a specific set of actions. When using the CBP system, the verifier or peer-checker may log in to the active procedure and sign off on the specific step to conduct a concurrent verification or peer-checking. The CBP system will notify the operator when an independent verification is needed. The system will also assign and notify personnel to conduct the independent verification. A new concept remote verification was introduced in the DOE LWRS Human Performance Pilot Project. When performing a remote verification, the verifier will review video or photos of the actual task being performed at the work site. The proof of concept of these characteristics was conducted in the DOE LWRS Human Performance Pilot Project (Farris & Medema, 2012). The concepts have not been evaluated further in the current research effort. 2.8 Placekeeping Placekeeping is the process used to help users track performance of steps within a procedure by physically marking steps in a procedure that have been completed or are not applicable. It is well known in the nuclear industries that conducting steps out of order or omitting a step are two common types of deviations. The current practice used for placekeeping in the nuclear industry is often referred to as the circleslash method. By following the circle-slash process the operator reads the step, circles the step number, conducts the step, and then marks the step complete by drawing a slash through the circle. However, the current practice of circle-slash where the operator is required to circle the step before reading it is a quite unnatural behavior for a human which may unnecessarily increase the risk of deviations. Regardless, the researchers adapted a placekeeping strategy that closely aligns with the circle-slash approach for the first prototype. The first prototype required the operator to start the step to make it active. Before actively starting the step, the operator could review the step but not carry out any of the actions. The operator indicated that a step was completed by clicking Mark as completed. 16

28 Figure 8. Examples of Placekeeping. The result from the first evaluation study indicated that having to both take actions to start and complete the step required too many interactions with the procedure, i.e., too many clicks. One important consequence of requiring excessive interaction with the CBP is that it takes focus away from the procedural task. The circle slash method is meant to ensure that an operator does not omit a step, lose his place, or move on without completing a step. Figure 8 depicts the evolution of placekeeping from the circle-slash method on PBPs, the start step -mark complete method in the first CBP prototype, to the current version of only marking a step as completed which automatically initiate the next step. The CBP prototype inherently prevents those undesirable behaviors; therefore, the second and third iterations of the prototype no longer require the operator to explicitly start the step. As soon as the previous step is completed the next relevant step will be marked active. The operator can only take actions related to the currently active step. Completed steps are clearly marked with a check-mark. The operator can read completed steps, future steps, and N/A d steps, but no actions can be taken on these steps without first overriding the system (and getting the supervisor s approval). In all three iterations of the prototype, the active step has always been clearly indicated in the overview mode. The current step is easily identified in a long list of procedure steps. The second and third iterations of the prototype also clearly indicated steps that are marked as not applicable due to decisions made by the operator earlier in the procedure. These N/A d steps are grayed out, i.e., no step-related action can be taken. However, the operator can get additional information regarding what decision rendered the non applicable marking of the step. 2.9 Correct Component Verification Before taking an action on a component or piece of equipment, the operator is required to verify that he/she is at the right component. This is called correct component verification (CCV). Currently, this is carried out by looking at the procedure and reading the component id out loud. Then, the operator will touch the component s label and read the component id out loud. If there is a match, the operator is at the right component. However, incidents where the operator manipulates the incorrect component still occur. 17

29 Figure 9. Correct Component Verification Using Barcodes. The researchers investigated different ways to provide support to the operator via the CBP system. Correct component verification via barcodes, optical character recognition (OCR), and manual input has been evaluated. Figure 9 shows an example of CCV using barcode. The system will match the input with a component database. If the correct component is verified the operator will be able to continue on with the step. If the correct component is not verified the operator will have to find the correct component before being able to proceed through the procedure. Figure 10 is both an example of CCV using manual input and an example of how the CBP system responds if the CCV fails. In this case the CCV failed due to incorrect component name or id. Figure 10. Example of Failed Correct Component Verification Using Manual Input Notes, Cautions, and Warnings Notes are statements that provide explanatory information to support a procedure step or series of steps. A caution is a statement placed immediately before applicable step(s) that informs users of undesirable equipment results such as potential for equipment damage, plant transients, or conditions that may adversely affect plant operation. A warning is a statement placed immediately before applicable step(s) to warn users of potential for personnel injury, loss of life, or health hazards. 18

30 In the initial prototype notes, cautions, and warnings were presented in a similar fashion as the procedure steps. The operator had to activate the statement by clicking start. In order to acknowledge that the statement had been read, the operator marked the step as completed. In the second iteration of the prototype the statements were presented differently than the procedure steps. In order to acknowledge the statement and move on to the next relevant step the operator had to click the Next button. Steps associated with the note, caution, or warnings were marked with a triangular symbol. By clicking on the symbol/icon the operator would get additional information related to the relevant statement associated with the step. In the third iteration of the prototype the notes, cautions, and warnings are incorporated in the step(s) where they apply. Figure 11 depicts this type of presentation. In this case a note is embedded in the step. Figure 11. Example of a Note Embedded in a Step Supplemental Information and Attachments Supplemental information refers to procedure content that supports a procedure step or series of steps and provides explanatory information. Attachments are information separated from the main body of the procedure used in the performance or understanding of a procedure such as graphs, figures, tables, sketches, and forms. Appendices and enclosures are equivalent terms. The researchers have had as a goal to make both supplemental information and attachments easily accessible to the operator. Initially, supplemental information and attachments were accessible through hyperlinks in the step description. In the second iteration, the information was embedded in the particular procedure step where it applied. An example that was evaluated was an embedded table used for a valve line-up. The researchers wanted to further investigate the embedded information. Hence, in the third iteration of the prototype even more information was embedded. However, images and other information can take up quite a bit of screen real-estate. To make the most of the available screen real-estate the option to see the supplemental information is presented to the operator in the form of a More information button. Figure 12 is an example of supplemental material accessible to the operator. The image is scalable using a zoom function, which provides easy access to more detailed information even though the image is displayed on a small screen. 19

31 Figure 12. Example of supplemental material Hierarchical Step Structure The step numbering schemes should differentiate between steps and substeps of the procedure by providing identifiable differences from one level or step level to the next. Step numbering is an interesting issue when presenting a procedure in a dynamic manner. When the procedure system takes the operator down the correct path of the procedure based on decision made by the operator, e.g., the N/A d steps are less relevant to the user, the step numbering scheme might become less relevant. The researchers wanted to investigate the value of step numbers compared to other means of communicating a hierarchal step structure. The first stab at addressing the hierarchical step structure made use of the two different presentation modes. In the single step mode the steps where presented in a flat manner, i.e., no hierarchy was indicated to the operator. No step numbers were showed. Steps and substeps were formulated in a way that there was no need for the hierarchy. In the overview mode, the original step structure (same as in the paperbased version of the procedure) and the step numbers were displayed to the user. However, it was concluded that the effort needed to rephrase and restructure steps to reduce the need for the step hierarchy was not justifiable when attempting to convert more than one short procedure into the computer-based procedure format. Therefore, the researchers investigated other presentation techniques to communicate step hierarchy without using numbers. The results from the evaluation studies do not indicate that the participants are missing the numbers nor are confused by the presentation of step hierarchies. The second iteration of the prototype incorporated indention of substeps. In other words, the operator gets a visual cue similar to what is commonly used in lists with different levels. In the second version of the prototype the parent step was also indicated by making the step description bold. The bold text in the parent description was removed in the third version. However, there are still concerns related to not displaying step numbers that need further investigation. These concerns include navigation between procedures and communication between field operator and control room operator or supervisor Bulleted Steps Bulleted steps within a single step may be performed in any order and shall be completed prior to proceeding to next step. The researchers wanted to find ways to communicate that steps can be conducted 20

32 in any order without using bullets. The procedure used in the first scenario evaluated did not contain any bulleted steps. However, the scenarios used in both the second and third studies did. In both the second and third version of the prototype bulleted steps were presented as a list where each bulleted step was listed on one row. All actions needed to complete the bulleted step were presented within the same row. The only main difference between the two versions is how the table is graphically represented. Figure 13 provides an example of how the bulleted steps are presented in the third version of the CBP prototype. Figure 13. Example of Bulleted Step Presentation Hold Points A hold point is a pre-selected step in a procedure that identifies a point beyond which work may not proceed until the required action is performed. Hold points have not been evaluated yet in the research effort. However, the conceptual design of hold points has been considered. In conditions where wireless communication is available, steps beyond a hold point would not be allowed to be activated until the CBP system receives notification that the action has been performed. If wireless is not available, then the operator would input that the action has been performed into the CBP when he receives notification Branching Steps A branching step is a step that directs the operator to other steps or sections in the same or another procedure and the operator does not return to the original step. Branching steps are so far being handled in the same manner as conditional steps, i.e., the branching depends on input from the user. The procedure system prompts the users to input the conditions that are the basis for branching, and the CBP system takes the operator to the appropriate step, section or procedure. In the future branching could potentially happen automatically based on operational data, e.g., data points related to the system s status Procedure Specific Information The procedure specific information is also sometimes called the front matter of the procedure, i.e., all the information that is of importance to the procedure but are not procedure instructions per se. Example of procedure specific information are; procedure title, procedure number, revision number, level of use, purpose and scope, precautions and limitations, definitions, and precautions and initial conditions. The procedure specific information can be divided into two groups; the information on the title page (title, number, revision, etc.) and the detailed information needed to correctly prepare to perform the task at 21

33 hand (purpose and scope, precautions and limitations, etc.). Figure 14 is an example of how the detailed information is presented in the CBP system. The operator is required to review all the procedure specific information before carrying out the task at hand. All this information is therefore presented to the operator and has to be acknowledged before entering the instruction part of the procedure. Figure 14. Example of Presentation of Procedure-Specific Information. 22

34 3. EVALUATION OF CONCEPTS To date, four studies to evaluate the design concepts have been planned and three of these have been conducted. This section will briefly describe the three conducted studies and their results. The intent of the fourth evaluation study is also described. 3.1 Prototyping of the Computer-Based Procedure System The model of procedure usage and the initial set of design requirements were used as the basis for the development of the first CBP prototype. The researchers also constructed a visual specification in the form of a mock-up displaying all design concepts to be incorporated in to the prototype. The researchers constructed visual specifications for each iteration of the prototype development. The concepts and designs in this research effort are intended to be platform-independent. That is, the results do not depend on specific software, platforms, or hardware. This approach allows the nuclear utilities to decide what they want to use for their CBP system. Versions of the prototype have been developed for ios and Android systems. The design of the CBP prototype was optimized for field operators (also known as Auxiliary Operators or Nuclear Equipment operators). Field operators often need to crawl through tight spaces, climb ladders, and require that they have their hands free to manipulate equipment. Therefore the researchers concluded that the device used to present the procedures needs to be small enough to fit in a typical pocket. The researches decided to use ipod Touch and Nexus 4 for the evaluation studies. It was assumed that a smaller screen real-estate would pose the largest impact on design decisions Iterations of Design Decisions The table below provides a summary of how the different task flow characteristics were addressed in each of the iterations of the prototype design iteration. Only the first three iterations have been evaluated to date. The planning of the fourth evaluation study is not yet finalized, hence no dates have yet been determined. The design changes made are based on the results from evaluating the concepts in the previous design. Some task flow characteristics have not yet been addressed or evaluated. These are grayed out in the table. For more detailed discussion of each task flow characteristic (Section 2) and the mapping to the model of procedure usage (Section 1, Table 1), please refer to the previous sections. Table 2. Task Flow Characteristics Addressed in the Prototype Development Task Flow Characteristics Action step 1 st iteration 2 nd iteration 3 rd iteration 4 th iteration Single step view and overview modes. Actions taken from the single step view. All actions are performed from the overview mode. The single step view is removed. All actions are performed from the overview mode. Calculation capability is incorporated into the relevant steps. More context specific information is incorporated in each step. Enhanced focus on task at hand in each step. All actions are performed from the overview mode. 23

35 Action verb Conditional step Time dependant steps Continuously applicable steps Multiple Action Steps Concurrent verification, Independent verification, and Peer checks Placekeeping Action verbs that required a physical action (e.g., Ensure, Start, Stop, etc.) are presented in purple color. Simplified step logic, i.e. all the conditional statements are presented as questions. Operator has to actively start the step and when the step is complete the operator have to mark the step as completed. Action verbs that required a physical action (e.g., Ensure, Start, Stop, etc.) are presented in purple color. Simplified step logic, i.e. all the conditional statements are presented as questions. Some decision can be done before starting the procedure. When the previous step is marked as completed by the operator the next relevant step becomes active. Indications of what decisions lead the operator down a particular path are added. Action verbs are not differentiated from other words in the step description. Simplified step logic, i.e. all the conditional statements are presented as questions. Continuously applicable steps are clearly indicated. Before initiating the step the operator is informed that the steps have to be performed at the same time. When the previous step is marked as completed by the operator the next relevant step becomes active. Action verbs are not differentiated from other words in the step description. Simplified step logic, i.e. all the conditional statements are presented as questions. Continuously applicable steps are clearly indicated. Before initiating the step the operator is informed that the steps have to be performed at the same time. The substeps will be presented in the order they have to be conducted. When the operator make a decision, e.g., answer a question or providing other input, the system will automatically move to the next relevant step. 24

36 Notes, Cautions, and Warnings Supplemental information and Attachments Hierarchical Step Structure Bulleted steps Hold points Branching steps Presented as a procedure step. The operator had to actively mark that they start reading as well as mark the statement as read. Presented as hyperlinks in the step description. The overview mode presents the full procedure including step numbers. The single step view has no step numbers. If it is a condition based branching the step is formulated as a question. If no conditional statement, then the CBP system will automatically take the operator to the next relevant step. Statements are presented differently than procedure steps. Operator has to click Next to acknowledge having read the statement. An icon on the step indicates that a statement is associated with the step. Embedded in the step. The overview mode does not show step numbers. Hierarchies are indicated by labels and indentions. Parent step s description is bolded. Presented as a table. Each bulleted step and the actions related to that step are presented on one row in the table. If it is a condition based branching the step is formulated as a question. If no conditional statement, then the CBP system will automatically take the operator to the next relevant step. Statements are incorporated into steps they apply to. It is clearly indicated if the statement is applicable to multiple steps. The option to see relevant information is indicated by the show information button. Hierarchies are indicated by indentions. Presented as a table. Each bulleted step and the actions related to that step are presented on one row in the table. If it is a condition based branching the step is formulated as a question. If no conditional statement, then the CBP system will automatically take the operator to the next relevant step. Statements are incorporated into steps they apply to. It is clearly indicated if the statement is applicable to multiple steps. The option to see relevant information is indicated by the show information button. Hierarchies are indicated by indentions. Presented as a table. Each bulleted step and the actions related to that step are presented on one row in the table. If it is a condition based branching the step is formulated as a question. If no conditional statement, then the CBP system will automatically take the operator to the next relevant step. 25

37 Correct Component Verification CCV is conducted by scanning components Quick Response (QR) codes. CCV is conducted by scanning components barcodes. CCV can either be conducted by scanning the component s barcode or via manual input. CCV can either be conducted by scanning the component s barcode or via manual input Evaluation Study Evaluation Studies The objective of the first evaluation study was to demonstrate the prototype in as simple context as possible to get focused feedback on the design of the user interface. A simple procedure and scenario were selected, i.e. racking out a breaker. The study was conducted in an electrical training facility at a collaboration partner s utility. Figure 15 depicts the training facility and a demonstration of the CBP prototype used in the study. Figure 15. The Use of the Prototype in the First Study Method Participants The participants in the evaluation study included 13 technicians at an operating NPP. The technicians came from varied disciplines within the plant including two electricians, two mechanics, three I&C technicians, one chemistry technician, two procedure writers, one IT expert and two others. All of the participants were male. The average age of participants was 48 years (SD = years). Procedure Selection The researchers worked with plant personnel to identify a procedure that would ensure that the functionality of the prototype could be demonstrated during the evaluation scenario. The procedure selected was a plant procedure for racking out a breaker adapted for use in the plant s electrical training lab. The use of a training lab in the evaluation study allowed for a realistic setting, in which technicians could take actions on the equipment without affecting the plant. The researchers prepared a paper-based version of the procedure that conformed to the plant s procedure-writing guide, and a computer-based version using the prototype CBP software. Based on previous research (LeBlanc & Oxstrand 2012), the researchers determined that field operators need a 26

38 device small enough to put in their pockets (due to the fact that they frequently climb ladders and work in cramped spaces). Therefore, the computer-based version was presented on an Apple ipod touch. Surveys To assess the workload associated with using the CBP prototype compared with the traditional PBPs, researchers used the NASA Task Load Index (TLX) (Hart, 2006). The NASA TLX was administered using paper and pencil. Researchers developed their own usability survey to assess the interface of the CBP prototype. The 8-item survey targeted the availability of information, ease of navigation, and ease of use of the CBP interface. Researchers also developed a 6-item usability survey to assess the usability of the device. Researchers also developed a debrief questionnaire to gain more detailed feedback on the design of the user interface and the overall experience using the CBP. The questions on this survey were open-ended. Design The researchers used a 2-factor within-subjects design. The independent variable was procedure presentation type (i.e., CBP or PBP). Participants were assigned to either PBP-first or CBP-first order. The order was counterbalanced across participants. Experimental Protocol When participants arrived, they filled out an informed consent form. They then completed a pre-job brief that included a review of the procedure, a discussion of the conditions that would be encountered in the scenario, as well as a discussion of the potential safety issues associated with the scenario. This prejob brief served as the pre-job brief for both conditions (PBP and CBP) and was executed with the PBP. If the assigned order was PBP-first, the participant completed a two-minute drill using the paper-based procedure; the two-minute drill occurred at the job-site and included a brief overview of the expected initial conditions and the potential safety hazards Participants were instructed to complete the procedure scenario to the best of their ability and at their own pace. A researcher and a qualified electrician observed each scenario. The researchers were trained to recognize deviations from the optimal procedure path. They followed the scenario closely and recorded any deviations. Additionally, the qualified electrician observed the scenario and was instructed to note any deviations and share them with the researchers after each participant completed the scenario. The researcher started a stopwatch at the initiation of the first step and stopped the stopwatch once the final step was completed. When the scenario was complete, the participant was given the first NASA TLX and then he completed the scenario with the CBP. Before the CBP scenario, the participant was given a 5-minute training session on how to use the CBP interface. The training provided was minimal, but was expected to be sufficient to allow the participants execute the procedure. The rest of the scenario occurred exactly as the PBP scenario, except that the procedure was executed using the CBP prototype. Once the scenario was completed, the participant was given another NASA TLX form and the usability survey. If the participant was assigned to the CBP-first scenario, the sequence of events was the same except that the participant completed the drill using the computer-based procedure and executed the scenarios in reverse order. Once both scenarios were complete, the participant was given the debrief questionnaire Results and Discussion Deviation from optimal procedure path The researchers recorded no deviations in the optimal procedure path across all participants and conditions. Because there were no verified deviations in the study, a comparison of performance between the paper and computer-based versions of the procedures was not possible. The lack of deviations is likely due to the fact that the procedure and the scenario were relatively simple to execute. 27

39 Completion time The completion time was defined as the time elapsed between the start of the first step of the procedure and the successful execution of the last step in the procedure. The researchers used a stopwatch to measure the time in seconds. The completion time data was subjected to a 2 X 2 (Order X presentation style) mixed analysis of variance. The within-subjects factor was procedure style (CBP or PBP) and the between-subjects factor was order (PBP-first or CBP-first). The results indicated that it took longer to execute the scenario with the CBP (M =447, SD = 103) than with the PBP (M = 332, SD = 111), the main effect of presentation style was significant F(1, 11) = , p <.001. There was a significant interaction between procedure style and order F(1,11) = , p <.01; if the CBP scenario was completed first, then the difference between CBP and PBP was greater than if the PBP was completed first. Essentially, the effect of practicing the actual procedure had a greater effect on the CBP than the PBP. There was not a significant main effect of order. There are several possible explanations for why it took longer to execute the scenario with the CBP than with the PBP. The first is that the participants were using the CBP for the very first time with minimal training, while most the participants were used to using the PBPs on a daily basis. Another potential explanation is that the participants would often stop and comment on aspects of the CBP during the scenario execution even though they were instructed not to, increasing the overall completion time of the scenario. Subjective Workload Subjective workload, as measured by the NASA TLX scores, was compared between the scenario execution with the CBP and scenario execution with the PBP. The raw TLX (Hart, 2006) score was computed for each participant and compared between procedure styles. A paired samples t-test revealed that there was not a significant difference between subjective workload scores for the CBP (M = 2.23, SD = 2.04) and the PBP (M = 1.73, SD = 1.63). The workload scores were relatively low across conditions, indicating that participants found the scenario to be relatively easy to execute. A more difficult task may have yielded larger differences in workload. Importantly, the workload was similar for both the procedure execution with the CBP and the PBP. This indicates that managing the CBP interface did not add significant workload for the participants. In a task in which the overall workload is higher, it might be possible to detect an advantage in workload for CBPs. Usability Survey The usability survey was designed to assess the overall usability of CBP interface by inquiring about the ease of navigation, the availability of information, and other common usability dimensions. The overall usability score was computed by averaging the scores across all of the questions. Participants rated the CBP interface as moderately usable; the mean overall usability score was 3.5 on a 6-point scale. The lowest scores were reported for items related to the navigation of the user interface. Debrief Questionnaire Before scoring the debrief questionnaire, researchers developed a coding scheme for the open-ended responses. Any responses that did not fit into the a priori coding scheme were marked as other during coding. Procedure style preference More participants preferred the CBP to the PBP. Seven participants indicated that they preferred the CBP; two said they preferred the PBP, and four indicated that they had no preference. The most common reason for favoring the PBP (or having no preference) was that the CBP provided no way of looking ahead. This is an interesting result, because the CBP prototype had multiple ways of looking ahead, including an overview and the capability to preview the next steps. These features were pointed out 28

40 during training. This indicates that either the participants were not trained extensively enough on these features, that they did not remember the look-ahead functions, or that they did not find them useful. Other reasons for preferring the PBP (or having no preference) were the fact that the procedure pages took too long to load, and that the ipod touch was too small. The most common reason for preferring the CBP was that it reduced the opportunity for errors compared with PBP. Context-sensitivity All participants reported that only seeing the relevant steps in the CBP was an improvement over the paper-based procedure. This result indicates that participants did not find it confusing or disorienting to have portions of the procedure hidden because they were irrelevant to the current task. Additionally, participants unanimously preferred the simplified step logic of the conditional statements. This indicates that context-sensitivity and simplification of step logic are highly desirable features of CBPs. In addition to the results presented here, the researchers elicited specific feedback on many of the features of the user interface. They incorporated that feedback into a revised prototype evaluated in the second study Evaluation Study 2 The objective was to evaluate whether the modification made to the design of the prototype improved its usability and to provide quantitative performance data that could be used to compare CBP and PBP usage. The study was conducted at a flow loop facility at a collaboration partner s utility. Figure 16 demonstrates the use of the CBP prototype in the flow loop facility. Figure 16. A demonstration of the Use of the Second Version of the Prototype. 29

41 Method Participants Ten employees at an operating nuclear power plant (NPP) participated in this study. Nine were Nuclear Equipment Operators (NEOs) and one was a training manager. The mean age of the participants was 40 years (SD = 11 years). The participants had an average of 10 years of experience (SD = 8 years) in their current role. All participants were male. Procedure Selection Results from evaluation study 1 indicated that it may be difficult to detect performance advantages with CBPs if the procedure and scenario are too simple. Therefore, the researchers worked with plant personnel to select a more complex procedure and scenario for this evaluation study. The procedure utilized was an existing procedure used to train field operators in a functioning flow loop training facility; the specific scenario involved initiating the cold water system and the control loop system. The features that contributed to the increased complexity in the scenario chosen for this study include the presence of multiple conditional statements and branching to other procedures. These features of the procedure and scenario allowed the research team to better assess the impact of context sensitivity in the CBP prototype. Identical versions of the procedure were prepared for paper and the CBP prototype (which was presented on an Apple ipod touch). Surveys The researchers used the same surveys described in evaluation study 1 with a one exception. The debrief survey was modified to reflect changes made to the CBP prototype. Design This evaluation study was designed to determine whether the CBP prototype offers performance advantages over the PBP. Therefore, the main factor in this study was the procedure presentation style (PBP or CBP) and it was manipulated within participants. The main dependent variables in the study were completion time and deviations in the optimal procedure path. In order to investigate the CBP effect on performance time, researchers measured the completion time of each scenario via a stopwatch. The timer was started when the participant indicated that they were entering the first section of the procedure, and stopped when the participant completed the final step in the procedure. The researchers also recorded the number of deviations from the optimal procedure path. The researchers worked with plant personnel to identify possible deviations prior to conducting the study. Deviations were broken down into two categories: recovered deviations and non-recovered deviations. Recovered deviations were defined as situations in which operators were not following the optimal path, but ultimately recovered. For example, when an operator walked to the wrong location or attempted to verify the wrong component (but ultimately found the right component or location), it was recorded as a recovered deviation. Non-recovered deviations were defined as deviations in which the operators failed to take an action specified in the procedure or took the wrong action. The researchers chose to classify the deviations separately because they wanted to capture deviations that indicate confusion or misunderstanding (especially with respect to the CBP prototype, but that would not typically be considered deviations in the procedure). The non-recovered deviations would have a greater impact on system performance and safety than the recovered ones. Experimental Protocol Participation in the evaluation study was conducted in two sessions. During the first session, the participants were familiarized with the task and trained on how to use the CBP prototype. During the second session, participants executed the scenario with the CBP and with the PBP. Upon arrival at the first session, participants signed an informed consent form. Participants were then randomly assigned to complete the scenario with the CBP first or with the PBP first (order was counterbalanced). Participants 30

42 then filled out a brief demographics survey. Following the completion of the demographics survey, participants were instructed on how to use the CBP. The researchers trained the participants to navigate through the CBP, and then completed a simulated walk-through of the procedure to demonstrate how to use features such as barcode scanning. The training took approximately 30 minutes. The participants returned the following day to execute the procedure using both the PBP and the CBP. Participants were given a pre-job brief which included a discussion of the task to be completed, the potential hazards, and other important safety concerns. If the participant was assigned to the PBP-first condition, then he completed the procedure with the PBP. When the participant indicated that he was starting the first section of the procedure (i.e., reviewing the prerequisites), the researcher started the stopwatch. The researcher watched the scenario closely and recorded any deviations. Once the final step was completed, the researchers stopped the stopwatch and recorded the time. The participant then filled out the NASA-TLX for the PBP condition. The participant then completed the same scenario using the CBP in the same manner. Following the execution of the scenario with the CBP, the participant completed the NASA-TLX for the CBP task, the CBP usability survey, and the debrief survey. If the participant was assigned to execute the task with the CBP first, the experiment proceeded in the same manner except they executed the scenario with the CBP before the PBP. The researcher had to make one potentially important modification to the above protocol because equipment in the flow loop facility was malfunctioning. When the first participant executed a portion of the scenario, it was discovered that an air operated valve was malfunctioning. Maintenance personnel were not able to repair the valve during the visit, so the researchers instructed the participants to simulate the portion of the procedure that relied on that valve. For that portion of the procedure, the participants ran through the procedure as though they were conducting it, but did not actually manipulate the equipment as they did for the remainder of the procedure. This modification was the same for both the CBP and PBP conditions Results and Discussion Scenario completion time The time to complete the scenario was measured in seconds using a stopwatch. A 2 X 2 (order by presentation style) mixed analysis of variance indicated that there was a significant main effect of presentation style (CBP or PBP) on the scenario completion time F(1,7) = 7.12, p =.032. It took longer to complete the scenario with the CBP (M = 1724 seconds, SD = 358) than the PBP (M = 1231 seconds, SD = 274). There was no significant main effect of order and there were no significant interactions with order. The finding that it took longer to execute the scenario with the CBP than the PBP is consistent with study 1. Though the researchers provided more training in study 2, the difference in completion times may still be related to learning to use the new interface. The operators will need more time to become familiar with the CBP interface in order to eliminate familiarity as an explanation for differences in completion time. Deviations Recorded deviations were reviewed by the research team and a trainer from the plant. Any error that did not fit straightforwardly into the predetermined coding scheme was discussed until the team came to a consensus as to how (and if) the error should be coded. Five recorded errors were eliminated by this process (four in the PBP condition and one in the CBP condition). Analysis of the recorded deviations was conducted using non-parametric statistical tests because the data violated the assumption of normality. A Friedman test revealed a marginally significant effect of presentation style based on type of deviation (χf2 = 7.73, df = 3, p =.052). There were more non-recovered errors with the PBP than the CBP. These results indicate that the CBP may be effective in preventing non-recovered errors. The CBP helped operators catch potential errors (e.g., recovered errors such as scanning the wrong barcode), and correct them before them became unrecoverable. The most common deviation in this study was a failure 31

43 to correctly verify a tank level. The CBP ensured that operators verified the tank level, because it forced them to scan the barcode, which directed the operator s attention to the tank. There were other deviations that occurred with the PBP, but not the CBP (e.g., taking unnecessary action), but due to the limited scope of the study and the limited number of participants, it is less clear how the CBP prevented these errors. Usability Participants generally found both the device and the interface to be usable. The mean usability rating for the device was 3.9 (SD =.68) on a six-point scale. One of the specific factors that concerned operators related to the ipod was the fact that it was likely not rugged enough for work in some areas of the plant (e.g., radiation areas). The mean usability rating of the CBP interface was 4.7 (SD =.69) on a six-point scale, indicating that the operators felt the interface was usable. Workload Scores For the workload scores, the simple averages (or raw TLX scores) were computed for each condition. There were no significant differences in subjective workload between the PBP condition (M = 2.15, SD = 1), and the CBP condition (M = 1.67, SD = 1). This indicates that the CBP did not increase or decrease workload compared with the PBP for this specific task. The fact that the CBP was new to the participants compared with the PBP, yet did not increase workload may indicate that as users become more familiar with the CBP, it will reduce workload. Additionally, the task used in this study was simpler than many real-world task, so it may not have been ideal to show a potential reduction in workload with the CBP. Open-Ended Questions on the Debrief Questionnaire. Before scoring the debrief questionnaire, researchers developed a coding scheme for the open-ended responses. Any responses that did not fit into the a priori coding scheme were marked as other during coding. Procedure Style preference Eighty percent of the participants preferred the CBP over the PBP. Most participants indicated that they preferred the CBP because it eliminated irrelevant steps and information and because it provided a more reliable means of correct component verification through barcode scanning. The participants that indicated they preferred the PBP, cited familiarity with the PBP process as the reason for preferring the CBP. This indicates that given a usable interface, even those performers who are resistant to switching to CBPs may shift their preference as the using the CBP becomes more familiar. These results highlight the fact that context-sensitivity is one of the more desirable advancements that can be achieved with CBPs. Context-Sensitivity Participants unanimously preferred the context-sensitive CBP presentation compared to the static PBP presentation. Most operators indicated that only being presented the steps relevant to the current task and conditions greatly streamlined the process, and prevented them from spending time and effort evaluating which conditions were relevant while they were out in the field Evaluation Study 3 The objective of the third study was to conduct a more focused evaluation of the user interface and usability of the CBP system. The study was conducted during a utility working group meeting held at Idaho National Laboratory in August Method Participants Ten members of the LWRS utility working group participated in this evaluation study. The participants came from varied disciplines within nuclear power. There were two from operations, one manager, one training professional, two instrumentation and control (I&C) technicians, and three human 32

44 factors researchers. Fifty-percent of the participants had previous experience in nuclear operations. The mean age of participants was 53 years. There were nine males and one female. Forty-percent of the participants use procedures as part of their everyday job responsibilities, 20% had in the past, but not in their current role, and 40% have never used procedures as part of the their daily job responsibilities. Process Control Simulation Mockup Due to conducting the study in a meeting environment rather than in a training facility at a utility, the researchers constructed a mock-up of the plant system used in the scenario. The mock-up consisted of a physical mock-up of pipes and valves, digital mock-ups of controllers as well as a tank, and a large photo of the actual system in the plant, see Figure 17. The participants were asked to only evaluate the CBP system not the mock-up of the plant. Figure 17. The Physical Mock-Up Used in Study 3 The scenario selected for the study required the participant to record the current level in the tank, calculate how much additional water to be added in order to fill the tank, conduct a valve line-up to prepare the system for filling the tank, and then use the batch controller, a pump, and a motorized valve to fill the tank the desired amount. Figures 18 and 19 both demonstrates the use of the CBP system and the physical mock-up used in the third evaluation study. Protocol All participants received initial training on the CBP system in a group setting where one of the researchers showed the main functionality of the system. Each participant was offered additional training on the system before they began the scenario. All participants declined the additional training. Each participant had a pre-job brief regarding the task at hand before starting the scenario. The focus of this evaluation study was to get feedback on how to improve the final prototype for field evaluation; therefore no performance data was collected). However, the researchers did observe each participant and made notes about design features that seem to cause issues for the participant. When the scenario was 33

45 completed, the participant was asked to answer a survey. The survey contained both demographic questions and questions focused on evaluating the usability of the CBP system. The usability survey contained aimed at collecting both quantitative and qualitative data. The qualitative data focused on eliciting suggestions on how to improve the user interface. Additionally, there were two questions regarding the simplified step logic and context sensitivity to enable elaborative feedback on how each was handled in the CBP prototype. The quantitative data focused on usability of the CBP. The usability questions were designed to parallel the model of procedure usage. The questions were aimed at identifying how well the CBP prototype handled the issues and error-likely situations that were identified in the model. Figure 18. Performer Conducting Valve Line-Up. Figure 19. Correct Component Verification and Displaying of Tank Level Calculation. 34

46 Results and discussion Usability scores The mean usability rating was 16.15/20. Overall usability scores were divided into four sub scores: Navigation, Situation awareness, Error Prevention, and Overall Ease of use, see Table 3. Table 3. Usability Sub Scores From Study 3. Navigation Situation Awareness Error Prevention Ease of Use There were no systematic effects of the demographic variables (e.g., age, role, or experience in operations) on the usability dimensions. Though there were no systematic differences in the sub-scores, one specific item (How would you rate the ability to undo or correct an error when using the CBP ) was scored considerably lower than the others (12.62 versus an overall mean of 16.15). This indicates that the researchers should pay particular attention to the usability of the undo functions in future revisions of the CBP prototype. These usability ratings indicate that the CBP prototype was effective in mitigating some of the errorlikely situations identified in the model of procedure usage. Open-ended Questions The researchers developed two open-ended questions to target specific aspects of the CBP. These two questions assessed the context-sensitivity and reduced complexity of the step logic in the CBP. Researchers coded the response to these questions as positive neutral and negative. 60 % of participants made positive comments regarding the context-sensitivity. The remaining 40% were coded as neutral because they were positive with some suggestions for additional improvement. There were no negative comments about the context sensitivity of the CBP. 80% of the participants reported positive comments about the simplified step logic. The remaining 20% reported positive comments with some suggestions for improvement. This suggests that context sensitivity and simplified step logic are desirable features and that the implementation of each was acceptable in the CBP prototype. The suggestions for improvements will be considered in the next revision of the prototype. Suggested Improvements to the Graphical User Interface The participants provided numerous suggestions to further improve to the user interface. Those suggestions were coded based on a predetermined coding scheme and divided in to the following categories: Add information that is not currently presented in the CBP. The majority of suggestions were related to providing the participant with information that was not currently presented in the CBP. Many participants noted that it was difficult to go back and review what actions had been taken because some of that information was hidden. Barcode Scanning. Several of the comments were related to barcode scanning. One participant had particular difficulty using the barcode scanning function on the device and another mentioned that the barcode scanning was slow. Another common comment was related to the number of steps that require the barcodes to be scanned. However, there was not a consensus on whether the CBP prototype required too much or too little barcode scanning. Some participants commented that operators do not need to re-verify equipment that has already been verified. While other commented that operator s should have to scan every piece of equipment in every step. This 35

47 highlights the importance of further research on this topic to fully understand how barcode scanning can help to prevent errors when using field procedures. Procedure Flow. Several Participants commented that the presentation and flow of steps that needed to be conducted at the same time was awkward. Others commented that the CBP should make the flow of actions in each step more explicit. For example, on steps that require equipment verification using the barcode scanner, the CBP should prompt the participant to scan the barcode, and then present the instructions Discussion Many of the participants in this study were from disciplines other than field operations. Because the design of the CBP was intended for field operators, the responses might not necessarily reflect those we would receive form the intended users. While this may limit the validity of the results, it may also provide additional perspectives that would not be captured if only polling the intended users of this particular CBP prototype. For example, having trainers and managers evaluate the prototype provides insight into administrative requirements that we would not necessarily receive from users who simply want a tool to do their jobs better. While many of the recommendations and suggestions provided by participants as part of the three evaluations studies have been consistent across individuals and across versions of the CBP prototype, there are a few design decisions that remain controversial. That is, for a few issues there does not seem to be any consensus as to how these situations should be handled. Showing versus hiding information that is not applicable to the current conditions. One of the main design philosophies of the CBP is to only present task and condition-relevant information to the operator. However, the researchers recognize that it is important to provide access to all the information in case the operator needs to assess whether the CBP took him or her down the wrong path. Showing versus hiding procedure step numbers. Although none of the participants in this study mentioned step numbers, there was an intense discussion regarding the lack of step numbers in the group training session. The researchers decided not to present step numbers in the initial prototype because the CBP eliminated irrelevant steps. This would result in step numbers being skipped. The researchers concluded that it would be confusing if the procedure jumped from step 1 to step 5 because steps 2-4 were deemed irrelevant. Another alternative that the researchers considered was to renumber the steps so that they reflect the actual sequence taken through the procedure. This was deemed unacceptable because it essentially eliminated the meaning of the step number as a unique identifier for that particular action, which would be particularly bad in the event of a need to use a paper-based backup. The researchers decided that eliminating step numbers in the presentation would avoid the potential for confusing jumps from one step number to another. The researchers decided to maintain a unique identifier for each step in the background. This identifier can be used to communicate the current step to other procedure performers as well as identifying the current place in case of the need to transition to a backup procedure. Participants in the last evaluation study suggested that they should be able to view step numbers on demand (i.e., toggle them on/off). Further research will determine of this is necessary. When to require scanning of barcodes. In all three evaluation studies, there has been disagreement among participants (and the collaboration partners) about which steps should require barcode scanning. Some insist that they should be used only on critical steps, others 36

48 insist that they should be used any time equipment is mentioned in a step, and still others insist that they should only be used when the step requires the operators to take action on equipment. Clearly, this issue requires further research to define the circumstances in which scanning barcodes prevents errors, and when it simply becomes a nuisance. Conclusions This evaluation study demonstrated that the CBP prototype functionality might improve performance compared to paper-based systems; however there are a few opportunities to further the interaction. The specific suggestions provided by the participants of this study will be incorporated into the field evaluation prototype which will undergo initial testing in study Evaluation Study 4 To be Conducted The objective of the fourth study is to evaluate the user interface of the fourth iteration of the CBP prototype for field procedures. The evaluation study will be designed to answer the following questions. 1. Have the modifications to the prototype resulted in a more usable CBP? 2. What magnitude of performance improvement, if any, did the CBP prototype have over PBPs? 3. Are there any concerns that still need to be addressed and can are there any reasonable solutions to these issues that can be used in the guidance? The first question will be answered by comparing usability across all three evaluation studies focusing on comparison between the studies conducted at the same utility (the second and fourth evaluation studies). The usability scores and the preference for CBP versus PBP will likely be the most valuable in this analysis given that the initial study failed to detect and performance deviations in any condition. The second question will be addressed by comparing performance times and deviations between the CBP and PBP execution of the scenario. The third question will be answered by asking openended questions once the performers have had an opportunity to use the prototype in a realistic and complex task. In addition to answering the questions above, the results of all four evaluation studies will be synthesized to provide guidance on how to design the user interface of a CBP for field operators. This guidance will also address what implications the human factors specifications have for the underlying data structure and the procedure content Method The evaluation study will be conducted over the course of four days in a training facility on site at one of the collaborating partner s nuclear utility. The study will consist of performers executing the same procedural scenario with a CBP version of the procedure and a PBP version of the procedure, hence the study follow the same protocol as the two first evaluation studies (Study 1 and 2). Participants The researchers will recruit 20 auxiliary operators at one of our collaborations partner s facilities to participate in this study. CBP prototype The researchers will make modifications to the prototype to address usability concerns that were identified in the previous evaluation studies as well as incorporate additional functionality (if the need is identified). They will then test those modifications (along with all the previous design characteristics to validate whether they contribute to enhanced performance or cause unintended errors and problems. For example, the modifications will include addressing situations where step numbers might be of importance and improvement of step presentation to enhance the contextual cues relevant to the task. 37

49 Evaluation Study Protocol 1. Obtain participant informed consent 2. Assign participants to either perform the scenario using the PBP first or the CBP first (to control for order effects) 3. Train the operator on how to use the CBP interface (using a dummy procedure) 4. If the performer is assign to start with the CBP 5. Provide a pre-job brief using the CBP 6. Instruct the performer to execute the scenario 7. Researchers will either record video of the scenario execution to be coded for completion time and deviations by an independent coder or they will train an independent coder to record those values in real-time. Deviations will include: Scanning wrong barcode Skipping a step Performing step out of sequence Performing a step incorrectly Taking the wrong action Failing to take an action 8. The performer will fill out surveys that assess workload and usability 9. The performer will then execute the scenario with the PBP in the same manner 10. The performer will fill out surveys assessing workload and final impressions regarding the prototype 11. If the performer is assigned to the complete the PBP first, steps 4-10 will occur with the PBP first and then the CBP. 3.3 General Discussion of Results of Evaluation Studies The results from the three evaluation studies indicate that the CBP prototype supports operator performance of procedural tasks. In all three studies, users were able to use the CBP to complete a procedural task. In the second study, the CBP supported performance better than the PBP for the same task (as indicated by a fewer number of overall deviations with the CBP). The results of the study also indicate that the human-system interface for the CBP prototype is intuitive and usable. In all of the studies operators were able to use the prototype to execute a procedure with minimal training (at most, 30 minutes). Additionally, users generally rated the CBP as highly usable in the three studies (and the usability increased with each iteration of the prototype). Simplifying the step logic of conditional statements is desirable even though it adds a question that the operator must respond to. During early stages of this project, operators stated that it might be annoying if all conditional statements were broken into two parts as they are in the CBP prototype. However, in each implementation of the CBP prototype, operators strongly preferred the simplified presentation over the original IF/THEN logic. The researchers plan to test whether this simplified approach can be applied to make constrained language more explicit (e.g., rather than stating to ensure the valve is closed, ask if the valve is closed and then instruct the operator to close it if the answer is no). The results also indicate that the CBPs should always present context-sensitive information if it is available. The results also indicate that it is acceptable to ask the operator to provide the context (e.g., what is the task, what are the conditions) if it cannot be retrieved automatically from a data base. Additionally, context-sensitive information should be presented on all elements of the user interface (i.e., buttons and links and menus should always tell the operator what he should be doing, or where he will be going from that link). There are still some items that need further investigation and validation. One of these items is how to balance automatically guiding the operator through his task (i.e., not burden him with complex logic and 38

50 branching) while still allowing him to be engaged in the procedure. Eliminating the appearance of branching, i.e., seamless transition to other procedures, can potentially reduce the amount of contextual cues related to the task at hand. The researchers will continue to investigate how to best design the CBP to support the balance between automatic guidance through the task and operator engagement. Another item that needs further validation is the method for allowing the operator to review his actions and the path through the procedure. The validation of the review function requires minor changes to the CBP user interface, such as presenting decisions made at previous steps and an improved mechanism to go back and edit the previous step. 39

51 4. PATH FORWARD The four evaluation studies described previously in the report were/will be conducted to address and evaluate different design concepts related to the interaction with a computer-based procedure. As described in Figure 20 the results from the studies will be the basis for a design guidance document, which will be developed by the research team. Figure 20. Relationship Between the Evaluation Studies and Design Guidance. The research team aims to validate the design of the CBP system. The underlying data structure, i.e. the technology used to transfer the information in the currently used paper-based procedures to the format used by the CBP system will be developed. The underlying data structure is an important piece needed to implement context-sensitivity, simplified step logic, seamless transitions between procedure sections, correct component verifications, and other functions the CBP system should support. The team will use the data structure to transfer one or multiple nuclear power plant field procedures into the CBP format. The utility/utilities participating in the validation study will support the INL team by identifying procedures suitable for the purpose of the validation study. Procedures used on non-safety related systems, in non-radiological areas, and that are used frequently are of interest. The validation study will be conducted by having the field operators using the CBP system in the actual plant for the selected procedures in an extended amount of time. Data points such as frequency of use, time, path through procedure, etc. will be collected automatically in the background while the procedure is performed by the field worker. The research team will provide appropriate training to the field workers who will be using the prototype in their everyday work before the data collection effort is initiated. A debrief session including gathering of qualitative input from the field workers will conclude the study. The result of the validation study will be incorporated in the design guidance for CBP system for field procedures as well as being important input to the work with CBPs for control room procedures, which is described below. Another outcome of the validation study/studies will be a better description of what ground work in needed in the industry in order to be better prepared for the migration to a computer-based procedure system. Items such as standardization of procedure content across the different organizations that use procedures at the nuclear plant, the transfer or encoding of existing PBPs to a data structure, and how to best create new computer-based procedures will be investigated. 40

52 The design concepts identified when studying CBPs for field procedures are thought to be generalizable to a large extent. Hence, it is thought that the concepts will be useful when transferring other procedures used in other organizations at a nuclear power plant to a CBP system. However, there might be some differences in design and interaction with the procedure that need further investigation to ensure that the CBP design adequately supports the users. One area where there is great potential to use CBPs are in the main control room. The physical environment in the control room and the work processes used by the control room operators are similar to field workers in some respects and quite different in others. The researchers aim to investigate which of the design concepts for CBPs are transferrable to the design of CBPs for control room operators as well as identifying areas where the design either has to be altered or new design concepts need to be developed. These potential alterations or new design concepts will be developed and evaluated. Due to the fact that most of the current LWR fleet, if not all, has hybrid control rooms, i.e., their control rooms consist of a mixture of analogue and digital controls, the researcher will investigate potential for integrating the CBP system into the hybrid control room. The evaluation of design concepts will be both conducted in smaller scale studies and in the DOE Human Systems Simulator Laboratory at INL. 41

53 5. REFERENCES Farris, R.K. and Medema, H. (2012). Guidance for Deployment of Mobile Technologies for Nuclear Power Plant Field Workers. Idaho National Laboratory External Report. INL/EXT Le Blanc, K. and Oxstrand, J. (in press). Computer-Based Procedures for Nuclear Power Plant Field Workers: Preliminary Results from Two Evaluation Studies. Proceedings of the International Annual Meeting of the Human Factors and Ergonomics Society. San Diego, September 30 October 4, Le Blanc, K. and Oxstrand, J. (2012). Requirements for Computer-Based Procedures for Nuclear Power Plant Field Operators - Results from a Qualitative Study. Proceedings of 3rd International Conference on NPP Life Management for Long Term Operations (IAEA PLiM 2012). Salt Lake City, USA. Oxstrand, J., Le Blanc, K., and Fikstad, C. (2013). Evaluation of Revised Computer-Based Procedure System Prototype. Idaho National Laboratory External Report. INL/EXT , Rev 0. Oxstrand, J.H, & Le Blanc, K. L. (2012). Computer-Based Procedures for Field Workers in Nuclear Power Plants: Development of a Model of Procedure Usage and Identification of Requirements. Idaho National Laboratory External Report. INL/EXT , Rev. 0. Oxstrand, J. H, Le Blanc, K. L., and Hays, S. M. (2012). Evaluation of Computer-Based Procedure System Prototype. Idaho National Laboratory External Report.INL/EXT , Rev