Air Traffic Controller Performance and Workload Under Mature Free Flight: Conflict Detection and Resolution of Aircraft Self-Separation

Transcription

1 THE INTERNATIONAL JOURNAL OF AVIATION PSYCHOLOGY, 11(1), Copyright 2001, Lawrence Erlbaum Associates, Inc. Air Traffic Controller Performance and Workload Under Mature Free Flight: Conflict Detection and Resolution of Aircraft Self-Separation Scott M. Galster, Jacqueline A. Duley, Anthony J. Masalonis, and Raja Parasuraman Cognitive Science Laboratory The Catholic University of America Washington, DC The effects of conflict detection and self-separating aircraft resolution on the mental workload and performance of en-route air traffic controllers were examined. An air traffic control simulator was used to manipulate traffic loads and traffic complexity. A mature stage of free flight was simulated by having controllers monitor self-separating aircraft. Four 30-min scenarios were created to combine moderate (11 aircraft) and heavy traffic loads (17 aircraft) in a 50-mile radius sector with the presence or absence of self-separating and conflicting aircraft. Conflicts (defined as a loss of separation of 5 nm laterally and 1,000 ft vertically) were indicated to the controller by the appearance of a red circle around each of the aircraft involved. A self-separation event was defined as an evasive maneuver (either altitude or speed change) made by 1 aircraft to avoid a potential conflict with another aircraft. Performance and workload measurements indicated that controllers had difficulty both in detecting conflicts and in recognizing self-separating events in a timely manner in saturated airspace. Controller mental workload also increased, as indexed both by subjective and secondary task measures. Implications for the design of automated tools to support controllers under free flight environments are discussed. The rapid growth in worldwide air travel projected for the next decade will dramatically increase the demand for air traffic services. This demand will increase loading Requests for reprints should be sent to Raja Parasuraman, Cognitive Science Laboratory, The Catholic University of America, Washington, DC parasuraman@cua.edu

2 72 GALSTER, DULEY, MASALONIS, PARASURAMAN on already burdened air traffic control (ATC) systems that are at or near their designated maximum handling capacity (Nordwall, 1998; McAlindon & Gupta, 1993; Perry, 1997). Several new strategies for more efficient air traffic management (ATM) have been proposed to address the airspace congestion and system delays that are likely in the near future. One proposed solution currently under consideration in the United States is free flight (FF), which would allow user-preferred routing and free maneuvering, among other changes aimed at minimizing ATC restrictions (RTCA, 1995). An alternative is to extend the current system of ground-based ATC but to use automation to support air traffic controllers (ATCos) in the management of an increasingly dense airspace (Parasuraman, Duley, & Smoker, 1998; Wickens, Mavor, & McGee, 1997; Wickens, Mavor, Parasuraman, & McGee, 1998). The Federal Aviation Administration (FAA; 1997) FF Phase I plan includes elements from both of these proposals. European proposals have similarly ranged from full-fledged aircraft self-separation (Duong, 1996) to ground-based control with enhanced systems for communications, navigation, and surveillance (Eurocontrol, 1998). FF represents a radical philosophical shift from current ATM policy, although a much more restricted form involving direct routing was proposed many years ago (e.g., FAA, 1981). The modern goal of FF is to allow aircraft under instrument flight rules the ability to choose (in real time) optimum routes, speeds, and altitudes in a manner similar to the flexibility now given only to aircraft operating under visual flight rules. In theory, FF will result in the utilization of more fuel-efficient routes and a reduction in the delays imposed by ATC (cf. Ball, DeArmon, & Pyburn, 1995). To achieve this goal, traditional strategic-based separation (flight path based) will be replaced by tactical separation based on flight position and speed. The responsibility for separation will gradually shift from the current ground-based system to a more cooperative mix between the air and ground. Both airborne and ground-based technologies will need to be added before significant benefits of FF can be realized. Use of the global positioning system (GPS) or the Russian counterpart, global navigation satellite system, or both, will allow aircraft to pinpoint their locations in much greater detail than what is available today using standard radar tools. Equipped aircraft will be able to communicate this and other information using Mode S datalink and automatic dependent surveillance (ADS/ADS B). Voice and data communications via an electronic datalink capability will also enhance communications between the air and the ground. Ground-based tools to help ATCos in sequencing and landing aircraft, such as the center TRACON automation system (CTAS), and predictive conflict detection tools will ease the burden of increased traffic loads for the ATCos. Decision support tools covered under FF Phase I will allow benefits of FF to be realized as soon as they are implemented. These tools, primarily for ATCos, include the surface movement advisor, user request evaluation tool (URET), traffic management ad-

3 AIR TRAFFIC CONTROLLER PERFORMANCE 73 visor (TMA), and collaborative decision making, which is a communications system designed to provide up-to-date information to interested participating service providers. FF can be viewed as a series of stages on a continuum, along which new technologies and procedures will be added as they prove useful and effective in the attainment of less restrictive airspace. The endpoint on the continuum, representing the utilization of the most advanced technologies and procedures available, is termed mature FF. Characteristics of mature FF include the ability of aircraft to enjoy unfettered access to preferred routes and maintenance of free maneuvering with little or no intervention from ATCos. Although relinquishing active responsibility for separation, the ATCo will become more of a monitor as FF moves toward a mature level. Two recent reports by a National Research Council (NRC) panel on future ATM raised some concerns regarding this development (Wickens et al., 1997; Wickens et al., 1998). One concern is how effective an ATCo can be in a purely monitoring role (Parasuraman & Riley, 1997). Human monitoring of automated systems can be poor, especially if the operator has little active control over the automated process and is engaged in other tasks (Parasuraman, Molloy, & Singh, 1993). Failure recovery is also a concern. At low levels of automation, throughput is low, and the time available for failure recovery is generally high. The time required for recovery from a failure is low due to the immersion of the ATCo in the task of managing the airspace. Increased levels of automation, however, particularly of decision-making and control functions, may increase the time required for recovery because of loss of traffic awareness by the ATCo. The second report by the NRC panel (Wickens et al., 1998) recommended that extensive human-in-the-loop simulations be conducted to evaluate different FF concepts for their impact on safety and efficiency. Moreover, no matter what level of FF is eventually implemented, ATCos will remain ultimately responsible for the maintenance of safety (ensure separation), even under mature FF (RTCA, 1995). As a result, there is an urgent need to investigate the effects of different proposed levels of FF on ATCo workload and performance. Two recent studies looked at ATCo performance and workload under various simulated FF conditions. Endsley, Mogford, Allendoerfer, Snyder, and Stein(1997) conducted a study with full performance level (FPL) en-route civilian ATCos under four different procedural conditions: current procedures (baseline), direct aircraft routing, direct aircraft routing allowing pilot deviations with conveyed intent to the ATCo, and direct aircraft routing allowing pilot deviations without conveyed intent to the ATCo. ATCos reported significantly higher subjective workload when they were not advised of pilot intentions in advance of a deviation from the flight plan. They also showed a trend toward committing more operational errors in the direct routing without shared intent condition compared to the baseline condition. This trend, although not statistically significant, suggested that ATCos found it difficult to

4 74 GALSTER, DULEY, MASALONIS, PARASURAMAN maintain separation standards when they did not have a clear picture of the pilot s intentions, as might be the case in mature FF. Hilburn, Bakker, Pekela, and Parasuraman (1997) also examined the impact of FF on the performance of military ATCos in England. Conventional control in structured airspace was compared to FF conditions in which aircraft did or did not share intent information with ATCos. These conditions were crossed with low and high traffic levels. ATCos reported higher subjective mental workload for high traffic loads; however, there were no overall differences between the control conditions. ATCos also reported higher mental workload in high traffic scenarios under conventional control conditions than they did under uninformed FF. Physiological measures of workload, including blink rates and pupil diameter, revealed a trend toward lower mental workload in the FF conditions. The discrepancy between these two studies may be attributed to several factors, including the different populations tested, that is, U.S. civilian ATCos by Endsley et al. (1997) and English military ATCos by Hilburn et al. (1997). Differences between these groups have recently been noted (Duley, Galster, Masalonis, Hilburn, & Parasuraman, 1997; Hilburn & Parasuraman, 1997). For example, military controllers in England are accustomed to working without flight progress strips, which may acclimate them to a certain level of uncertainty regarding flight paths, a by-product of increased levels of FF. Neither of the previously mentioned two studies examined the end stage along the FF continuum or mature FF. As noted earlier, mature FF will involve the deployment of several technologies such as GPS, datalink, and ADS B. There will also be several procedural changes concerning responsibility for separation. A recent simulation study (van Gent, Hoekstra, & Ruigrok, 1998) found that pilots can use a cockpit display of traffic information (CDTI) to maintain airborne self-separation effectively, without any ATC involvement. An early study by Kreifeldt (1980) also found that use of a CDTI did not increase ATCo communications workload, although pilot workload was impacted (see also Morphew & Wickens, 1998). If successfully implemented, these technological and procedural changes will result in an airspace that will be considerably more dense and complex than it is today. Note, however, that even under the RTCA vision of mature FF, ATCo intervention will be mandated under the following four circumstances: (a) to ensure separation, (b) to preclude exceeding airport capacity, (c) to prohibit unauthorized flight through special use airspace, and (d) to ensure safety (RTCA, 1995). Thus, because ATCos will be involved in management by exception (Smith et al., 1999), they will still be required to monitor the airspace to intervene effectively in these four cases. Our investigation addressed the following issues concerning mature FF. First, will ATCos still be able to effectively monitor the saturated airspace that mature FF will bring when they are no longer actively involved in separation? Second, if so, how long will it take ATCos to recognize the development of a potential threat-

5 AIR TRAFFIC CONTROLLER PERFORMANCE 75 ening situation? Third, what effect will mature FF have on ATCo mental workload? We anticipated that the results would not only be informative with respect to the effects of mature FF on ATCo performance and workload, but also provide relevant performance goals for the design of automated tools (e.g., conflict probe) that will be needed to support ATCos under FF. We carried out a simulation study of mature FF using experienced civilian en-route ATCos. The characteristics of the airspace included a high traffic load (simulating saturated airspace), aircraft initiating and executing evasive maneuvers to avoid conflicts without ATCo guidance, and separation violations. The ATCos in our study operated primarily as monitors of the airspace, as would be the case under mature FF. Participants METHOD Ten currently active FPL en-route ATCos, most from the Washington Air Route Traffic Control Center, served as paid participants. Their ages ranged from 32 to 42 years (M = 35.6 years), and their ATC experience ranged from 8 to 15 years (M = 12.1 years). ATC Simulation We used an ATC simulator specifically designed to allow for close experimenter control over display features and task performance, while providing a moderate degree of realism (Masalonis et al., 1997). The simulator included a primary visual display (PVD) of traffic including data blocks, display control tools, waypoints, and high-altitude jet routes. An adjacent monitor contained a datalink interface and display windows devoted to electronic flight progress strips. Black and white schematic renditions of the two monitors are shown in Figures 1 and 2, respectively. A trackball allowed the ATCo to traverse between the two screens seamlessly. The ATC simulator had the capability of presenting traffic scenarios in any airspace in North America. However, in this study, a generic airspace was developed to avoid any possible interactions between degree of airspace familiarity by ATCos and the experimental conditions. PVD. The PVD was presented on a 20-in. color monitor (75 Hz, resolution). The PVD shown in Figure 1 contained eight aircraft at the time of the screen capture. Each data block shows the call sign of the aircraft on the first line and the current altitude of flight on the second, followed by either a C representing a constant Mode C transponder emitted altitude or a plus or minus (not shown)

6 FIGURE 1 Primary visual display of traffic at range equal to 75 miles. FIGURE 2 Datalink interface as shown on second adjacent monitor. 76

7 AIR TRAFFIC CONTROLLER PERFORMANCE 77 indicating an ascending or descending status, respectively. The third line of the data block contains the computer identification number, traditionally assigned by the host ATM system, and the current airspeed of the aircraft. The PVD also contains various waypoints designated by three or five letter combinations. Some of these waypoints are connected with visible lines indicating established high-altitude jet routes in which aircraft normally travel, although they are not required to. Two concentric circles and a polygon are also visible on the PVD the outer circle represents a 50-mile radius projecting from the CMH waypoint, whereas the polygon represents the actual sector boundary used in the study. The inner circle marks the boundary of the hand-off zone that indicates the earliest time the ATCo can hand off outbound aircraft. The PVD incorporated a number of display controls in the form of function keys, shown in Figure 1 in the upper and lower right-hand corners. There was also a clock (upper left-hand corner) indicating elapsed time since the onset of the scenario. The history function key controlled the history tool, which placed hash marks behind each aircraft and updated with each screen refresh (every 5 sec in this study). ATCos could turn this tool on or off and use the plus and minus keys to set how many hash marks to display behind each aircraft. This tool allowed the ATCo to display a maximum of five hash marks for all aircraft and was used primarily as a perceptual aid for judging heading and speed. Another optional tool was the J-ring, which drew a circle of specified radius around each aircraft. Both the history and J-ring tools were global, being either on or off for all aircraft. The vector tool, by contrast, could be used with individually selected aircraft. ATCos could set the range (time into the future at current speed) and select one or more aircraft on the PVD. This resulted in the appearance of a directional purple vector of the specified range for each aircraft selected. The bottom right-hand function buttons were used to control the functionality of the PVD. The Fore and Back buttons controlled the contrast brightness of the foreground and background respectively. The range function allowed the ATCo to zoom in or out from a minimum of 25 miles to a maximum range of 400 miles. Datalink and electronic flight progress strips. A second monitor (17 in., 60Hz, resolution) contained the datalink interface and the electronic flight progress strips. The datalink screen consisted of four functional areas: flight strip area, flights area, datalink message bay, and an area devoted to operation of the datalink interface. In addition, there was a small function button at the bottom right hand of the screen (the function of this will be explained next). The flights area listed all of the aircraft that had gone through, were currently in, or were scheduled to go through the sector. The scheduled flights appeared in this area when they were 100 miles outside of the sector. This allowed the ATCo to have access to the electronic flight strip at about the time ATCos have access to the paper strips currently.

8 78 GALSTER, DULEY, MASALONIS, PARASURAMAN The ATCo was required to select a flight in the flight list to bring up the associated aircraft electronic flight strip, which appeared in the flight strip area. The flight strip area was scrollable, like the flights list area, and contained the call sign, computer identification number, and filed flight path for the aircraft selected. It also displayed the altitude of the aircraft over each scheduled waypoint crossing. The waypoints listed in the flight strip were functioning buttons that needed to be pressed by the ATCos as the aircraft passed the designated waypoint. An alt dev button was also present, although nonfunctioning for this study, that ATCos could have pressed if the aircraft was not at the assigned altitude over the designated waypoint. The ATCo could open or close the flight strips at their discretion. The datalink message bay listed all of the incoming messages from each aircraft. A message was automatically docked when an aircraft was about to enter the sector, mimicking an automated hand off from the adjacent sector. The ATCo read the message and then accepted the aircraft into the sector by pressing the accept button located in the datalink function section. Once the accept button was pressed, the message disappeared from the message bay area. A hand-off action was issued by the ATCo by selecting the appropriate aircraft in the flights list and pressing the auto hand button at the bottom of the datalink screen. A message subsequently appeared in the message bay indicating that the aircraft would comply with the orders sent by the ATCo. A nonautomated hand-off procedure was also available for use by the ATCo. If they used this option, they were required to specify the aircraft; choose the appropriate outbound sector radio frequency; send that information to the aircraft; and, finally, hand off the same aircraft. Not surprisingly, all ATCos chose to use the auto hand-off method. Traffic Scenarios Four 30-min scenarios were created that combined two traffic loads (in a 50-mile radius sector), moderate (11 aircraft) and heavy (17 aircraft), with either the presence or absence of self-separating and conflicting aircraft. Each scenario started without traffic and a 10-min, ramp-up period was used to reach the desired number of aircraft in the sector. The traffic level then remained relatively constant during the last 20 min of each scenario. In the two scenarios in which they occurred, conflicts (defined as a loss of separation of 5 nm laterally and 1,000 ft vertically) were indicated to the ATCo by the appearance of a red circle around each of the aircraft involved. A self-separation event was defined as an evasive maneuver (either altitude or speed change) made by one aircraft to avoid a potential conflict with another aircraft. There were two conflicts and four self-separating events in each of the conflict-present scenarios. Two of the self-separating events were resolved early, whereas the other two were resolved closer to the potential conflict (or late). Early resolutions occurred more than 250 sec prior to the loss of separation the pair of air-

9 AIR TRAFFIC CONTROLLER PERFORMANCE 79 craft would have had if an evasive maneuver had not been made. A late resolution occurred when the evasive maneuver started less than 150 sec prior to the impending loss of separation. Figure 1 shows a conflict that is about to occur in the left side of the sector between NWA 936 traveling eastbound from TONNY and SWA 388 traveling on a northeastern heading from LENNI. These two aircraft, both flying at FL290, are on course for a separation violation or conflict approximately at the intersection of where the two jet routes intersect. A descent by one of the aircraft, an evasive maneuver of more than 1,000 ft, would result in a self-separation event involving the same pair of airplanes. ATCo Tasks ATCos were required to accept all incoming aircraft into the sector, monitor each aircraft s progress over the waypoints listed in its flight strip, and hand off each aircraft to the adjacent outbound sector. In the conflict-present scenarios, they were also required to indicate verbally any potential conflicts that they judged would occur by stating which two aircraft were involved in the potential conflict and approximately over which waypoint the conflict would occur. The number of conflicts and self-separating events was kept low in the interest of ecological validity. As a result, a traditional statistical approach to the evaluation of detection performance based on many trials could not be used. To augment the sensitivity in performance differences between the experimental conditions, the secondary task of monitoring aircraft progress was added and evaluated using conventional statistical procedures, as described later. Procedure The functionality of the simulator was demonstrated by the experimenter to each ATCo. After this, ATCos participated in a practice trial to familiarize themselves with the displays and controls. They continued until they indicated that they were ready to begin the study. Each ATCo completed three 30-min scenarios and was offered a break between successive scenarios after they filled out the NASA Task Load Index (NASA TLX) subjective mental workload questionnaire. All 10 ATCos completed the moderate- and heavy-traffic load scenarios without conflicts and self-separating events. This allowed ATCos to have the maximum amount of exposure to the system without requiring them to perform the additional task of calling out potential conflicts. Five ATCos then completed the moderate traffic load scenario containing conflicts and self-separating events, whereas the other 5 completed a similar scenario under a high traffic load. A completely crossed design could not be used because of the limited time availability of the ATCos (4 hr) and

10 80 GALSTER, DULEY, MASALONIS, PARASURAMAN the need for scenarios that were sufficiently long (30 min) to allow traffic patterns to emerge and to allow the ATCo to develop a coherent picture of the traffic. Dependent Measures Accepting and handing off aircraft were considered primary tasks, as were conflict and self-separation detection. Tracking and monitoring aircraft by updating the electronic flight strips provided a realistic secondary task measure of ATCo workload. For each aircraft and for all required ATCo tasks, response time (RT) for correctly executed actions and frequencies of missed actions was tabulated. RT for acceptance of an aircraft consisted of the time elapsed between the ATCo reading the message on the datalink interface and the time the ATCo accepted the aircraft. The monitoring task RT consisted of the time elapsed after an aircraft passed a waypoint to the time the ATCo pressed the corresponding waypoint button on that aircraft s flight progress strip. A missed action was recorded if the ATCo never selected the corresponding waypoint in the flight strip after the aircraft passed the waypoint. A miss was not recorded if the ATCo completed a hand off, either correctly or incorrectly, prior to the aircraft reaching the remaining waypoint (or waypoints) on its flight plan within the sector. Hand-off RTs were calculated as the elapsed time the ATCo used to complete the hand-off procedure once the aircraft had entered the hand-off zone shown on the PVD. Simulation studies that manipulate and analyze multiple discrete events often do not capture performance trends seen over time. These trends can be used by researchers to explore shifts in tactical strategies, illustrate time accuracy trade-offs present in the performance data, and provide useful information about planning future studies. Thus, to capture such variations in performance over time, we calculated cumulative efficiency ratings for each task by dividing the number of hits obtained by the number of possible events for each 10-sec period of the 30-min scenario. The resulting time series was computed for each task and for each ATCo. In addition, a total efficiency rating was computed by adding up the combined hits across all tasks and dividing by the total number of possible events. Hence, each ATCo had efficiency ratings for accepting aircraft, monitoring performance, handing off aircraft, and a total score for each 10-sec block of time in each scenario. In the conflict scenarios, an additional measure, termed the advanced notification time, was also computed. In the case of a conflict, the advanced notification time was defined as the time the separation violation occurred subtracted from the time the ATCo gave a verbal indication of the conflict. Accordingly, the greater the duration, the more in advance the ATCo recognized that a conflict was about to occur, and the better the performance of the ATCo. If the ATCo did not detect the conflict prior to its occurrence, a miss was registered. The advanced notification time for a self-separation event consisted of the reported time prior to when the

11 AIR TRAFFIC CONTROLLER PERFORMANCE 81 conflict would have occurred had the aircraft not made an evasive maneuver. A miss in this case was recorded when the ATCo did not recognize the potential conflict prior to the aircraft initiating the evasive maneuver. Thus, the difference between conflicts and self-separating events was that ATCos could respond to conflicts up until the separation minimum was violated, whereas the self-separations required the ATCos to respond before the evasive maneuver was initiated. The restriction on self-separations was instituted because it was deemed that a change in aircraft progress (altitude or speed change) would draw attention to the potential conflict that was being avoided. Subjective measures of workload were derived by taking the average of the six subscales of the NASA TLX: mental demand, physical demand, temporal demand, performance, effort, and frustration. This method has been found to be psychometrically equivalent to the weighted subscore averages that account for individual differences normally computed in the NASA TLX (Nygren, 1991). RESULTS Subjective and Secondary Task Measures of ATCo Mental Workload A one-way analysis of variance (ANOVA), or repeated measures ANOVA if appropriate, was used in the following statistical analyses. Table 1 shows the mean NASA TLX subjective workload estimates for each experimental condition. Using the no-conflict scenarios, the effect of traffic load was significant, F(1, 9) = 24.14, p <.001, indicating that a higher level of subjective workload was reported under the high traffic condition compared to the moderate traffic condition. A similar traffic load effect, F(1, 8) = 8.28, p <.05, was found for the conflict-present scenarios. However, the main effect of conflict presence absence was nonsignificant. As Table 1 shows, subjective workload was affected by traffic load in the expected direction but was relatively unaffected by the addition of conflicts. TABLE 1 Mean NASA-Task Load Index Scores Scenario M SE Without conflicts Moderate traffic High traffic With conflicts Moderate traffic High traffic

12 82 GALSTER, DULEY, MASALONIS, PARASURAMAN The secondary task measure of ATCo mental workload (monitoring and tracking flight progress) gave similar results. Table 2 gives the mean percentage of waypoints that were omitted in each condition the higher this percentage, the higher the presumed mental workload of the primary task. When conflicts were not present, ATCos omitted checking off significantly more waypoints, F(1, 9) = 14.37, p <.01, in the high compared to the moderate traffic condition. Also, under moderate traffic, ATCos missed checking off significantly more events, F(1, 13) = 8.77, p <.05, with conflicts present than with them absent. These results suggest that a possible strategy employed by ATCos was to partially shed the secondary monitoring task as traffic increased and also when conflicts were present under moderate traffic. However, under high traffic, the proportion of missed events with conflicts present did not differ significantly from that with conflicts absent. Thus, the majority of task shedding, if performed, would have been due to increased traffic rather than conflict presence. Table 2 also shows the mean RTs for the monitoring task. Unlike the percentage of events missed, the mean RTs for the monitoring task events that were detected did not differ significantly with traffic load or conflict presence. Primary Task Performance: Aircraft Acceptances and Hand Offs Mean RTs for aircraft acceptances and hand offs that were considered hits are also shown in Table 2. There were no significant effects of traffic load or conflict presence on mean RTs for accepting aircraft into the sector. In accordance with its importance as a primary task, ATCos, in general, promptly accepted all aircraft enter- TABLE 2 Mean Response Times (Seconds) and Percentage of Missed Events for Acceptance, Monitoring, and Hand-Off Tasks for Each Scenario Type Acceptance Task Monitoring Task Hand-Off Task Response Time % Missed Response Time % Missed Response Time % Missed Scenario M SD M SD M SD M SD M SD M SD Without conflicts Moderate traffic High traffic With conflicts Moderate traffic High traffic

13 AIR TRAFFIC CONTROLLER PERFORMANCE 83 ing their sector irrespective of traffic load and the type of scenario. For the hand-off task, when conflicts were absent, RTs were significantly longer, F(1, 9) = 13.17, p <.01, under high than under moderate traffic loads, but the corresponding RTs in the conflict-present scenarios were not significantly different. An analysis of the percentage of missed events for both of these tasks was not done due to possible floor effects. ATCos diligently accepted and handed off aircraft under all traffic conditions (see Table 2). Primary Task Performance: Detection of Conflicts and Aircraft Self-Separation An important primary task for each ATCo was the detection of conflicts and self-separation events. Performance was assessed in the two scenarios (moderate and high traffic) in which these events occurred. Figure 3 shows the mean detection rates for each type of event. In the moderate traffic condition, ATCos detected 90% of conflicts prior to the loss of separation. The detection rate dropped to 50% under high traffic. As noted earlier, conventional parametric statistical analyses of these data were not possible due to the low number of these critical events in the scenario. Nevertheless, application of a nonparametric test (Mann Whitney U test) revealed that the reduction in the detection rate of conflicts between traffic loads approached significance (U = 4.50, p <.07). The conflicts that were recognized (14 of 20 across all participants) prior to their occurrence had mean advanced notification times of 196 sec in the moderate traffic condition and 145 sec under high traffic, as shown in Figure 4. These results indicate that under a high traffic load, ATCos were slower to report impending conflicts and committed more operational errors than they did under a moderate traffic load. ATCos also had great difficulty in recognizing self-separation maneuvers initiated by aircraft, also shown in Figure 3. ATCos detected 30% of the potential conflicts (that were resolved by self-separation) in the moderate traffic condition but only 15% under the high traffic condition. Although this difference failed to reach significance using the Mann Whitney U test, the low detection rates in both conditions is noteworthy. For self-separation events that were recognized (9 of 40 across all participants), the mean advanced notification time provided by ATCos was markedly reduced under high traffic (157 sec) compared to the moderate traffic (337 sec) condition (see Figure 4). We also carried out a nonparametric statistical test (Wilcoxon Signed Rank test for repeated measures) to compare the detection rate of conflicts and self-separations. This revealed that significantly more conflicts (70%) were detected than self-separations (22.5%) across both traffic loads (p <.02). If, however, the window of time allowed for the ATCo to respond to self-separating events was lengthened by 90 sec, an additional 10 self-separating events would have registered as

14 FIGURE 3 Percentage of detected conflict and self-separation events under moderate and high traffic loads. 84 FIGURE 4 Mean advanced notification times given by air traffic controllers for correctly identified conflicts and self-separations by traffic load.

15 AIR TRAFFIC CONTROLLER PERFORMANCE 85 hits and the difference would not have been statistically significant. This served as a confirmation of our prediction that the initiation of the evasive maneuver would probably direct the attention of the ATCos to the potential conflict. Efficiency Ratings As reported earlier, efficiency ratings over time were calculated to examine fluctuations in ATCo performance that might be masked by the overall workload and detection scores analyzed previously. Figure 5 shows an example of the efficiency ratings in a time series for the secondary task of flight progress monitoring. The example shows the efficiency ratings for all 5 ATCos in the high traffic condition with conflicts and self-separations present. It is apparent that there was considerable variability in monitoring efficiency between the 5 ATCos in this scenario. An example of how this information can be of use to the researcher is demonstrated by the efficiency rating of the fourth ATCo (shown in bold in Figure 5). Although this ATCo s efficiency ratings were generally below the average for the group, this participant recognized four out of the possible six self-separating and conflict events the highest percentage in this group of ATCos. This ATCo most likely followed a strategy of shedding the secondary flight progress task in favor of conflict and resolution detection. FIGURE 5 Efficiency ratings (percentage) for monitoring aircraft task in the high traffic condition with self-separations and conflicts present.

16 86 GALSTER, DULEY, MASALONIS, PARASURAMAN We analyzed the efficiency ratings to see whether there was any systematic variation before and after the occurrence of a self-separation or conflict event. One possibility was that the appearance of red circles around aircraft that violated separation minimums might have caused the ATCos to alter their task strategy. However, this was not the case. There were no appreciable differences in the before-conflict and after-conflict efficiency ratings, regardless of traffic load. The largest difference occurred in the hand-off task, in which there was a decrease in efficiency in the high traffic condition surrounding the second self-separation event. This event occurred 790 sec into the scenario, which also coincided with an increase in outbound traffic brought about by the end of the ramp-up period. A detailed inspection revealed that this difference diminished as more aircraft departed the sector and the ATCos became more adept at handing off aircraft. Although this result is not of great theoretical or practical interest, it does illustrate the sensitivity of this method of analysis to transient changes in task complexity. We also examined whether there were differences in efficiency ratings between those ATCos who recognized the self-separating or conflict event and those who failed to do so. Because traffic load has already been shown to be a major contributing factor to performance and workload, only the high traffic scenario data are presented here. Also, the task of accepting aircraft into the sector did not show differences and is excluded from further discussion. The temporal order of the events in the high traffic scenario was as follows: Self-separation 1 (355 sec), Self-separation 2 (449 sec), Self-separation 3 (796 sec), Conflict 1 (909 sec), Self-separation 4 (1,242 sec), and Conflict 2 (1,533 sec). No participants in the high traffic scenario recognized the third and fourth self-separation events, so no analysis could be conducted for these events, and they were omitted. What remains are the valueslistedintable3.notethatnohand-offvaluesforthefirstandsecondself-separations are given the reason being that no aircraft had entered the hand-off zone when these events occurred. The data shows that the efficiency rating for the monitoring and hand-off task, as well as the total efficiency rating, were lower for the TABLE 3 Mean Efficiency Ratings for Detected (Hit) and Undetected (Miss) Self-Separation and Conflict Events Efficiency Rating Monitoring Task Hand-Off Task Total Event Hit Miss Hit Miss Hit Miss Self-separation n/a n/a Self-separation n/a n/a Conflict Conflict

17 AIR TRAFFIC CONTROLLER PERFORMANCE 87 group of ATCos who missed the event and higher for those who recognized the event. This pattern persisted for all events except the second conflict in which the pattern was reversed. A Traffic Load (moderate or high) Time Period (2-min blocks) ANOVA of the monitoring efficiency rating data was also carried out. There was a significant effect, F(14, 112) = 4.82, p <.01, for time period as well as a significant interaction, F(14, 112) = 2.92, p <.01, between traffic load and time period (see Figure 6). There was an initial rise in efficiency in the early portion of the combined scenarios followed by a decline as the scenarios progressed. The interaction indicates a general decline in the efficiency rating over time for the high traffic condition but an inconsistent pattern for the moderate traffic condition. DISCUSSION This study posed three main questions regarding the influence of a mature FF regime on ATCo performance and workload; namely (a) whether ATCos could effectively detect aircraft conflicts and self-separation events, (b) how timely such detection was, and (c) the impact on ATCo mental workload. We used a medium-fidelity ATC simulator (Masalonis et al., 1997) with experienced FPL ATCos who worked primarily as monitors of the airspace, as would be the case under mature FF. An increase in traffic load led to significant increases in both subjective mental workload, as measured by the NASA TLX, and in objective workload as indexed by the secondary task of flight progress monitoring. These results, in addition to the relatively high and stable performance levels on the aircraft acceptance and hand-off tasks, served to validate the sensitivity of the simulation to the typical sources of task loading in ATC. The ATCos Under Mature FF Mature FF will require ATCos to monitor aircraft that will be free to maneuver to minimize fuel burn and to avoid conflicts with other aircraft. The results showed that ATCos had difficulty both in detecting conflicts and in recognizing self-separating events in a simulation of such conditions. Although conflict detection performance was near 100% under moderate traffic, it dropped markedly to only 50% under high traffic. Moreover, detection of self-separation events was poor under both traffic loads. In addition, when ATCos did identify conflicts, the advanced notification time was markedly reduced under high traffic. Advanced notification times were also reduced by high traffic load for detection of self-separation events. These results indicate that ATCo performance can be vulnerable under an advanced FF regime in which all authority for separation decision is ceded to air-

18 88 GALSTER, DULEY, MASALONIS, PARASURAMAN FIGURE 6 Monitoring efficiency ratings (percentage) in high and moderate traffic. borne systems, but ATCos are still required to intervene. Airborne self-separation has been shown to be feasible in both modeling studies (Duong, 1996) and simulations (van Gent et al., 1998). However, as noted previously, an advanced FF system that implemented airborne self-separation would nevertheless require ATCo intervention under four general cases identified by the RTCA (1995). These are to (a) ensure separation, (b) preclude exceeding airport capacity, (c) prohibit unauthorized flight through special use airspace, and (d) to ensure safety. Two of these exceptions specifically involve cases in which the ATCo might be called on to resolve a conflict. Our results suggest that to the extent that ground-based intervention is required to detect conflicts not detected by pilots or to resolve conflicts not resolvable by airborne systems, unaided ATCo performance may not be sufficient to meet the challenge. These findings extend those of previous studies examining ATCo performance under FF (Endsley et al., 1997; Hilburn et al., 1997). However, in contrast to these previous studies, we examined the impact of mature FF on ATCo workload and conflict-detection performance. ATCos, in the study reported here, were required to monitor under very high traffic loads, did not have intent information, and were not provided any automation support tools. The use of these conditions was deliberate: System safety may be best evaluated under such worst-case scenarios. Our findings for conflict detection are also generally consistent with those of Endsley et al., who found that ATCos showed a trend toward missing more conflicts when they were not provided with intent information regarding pilot-initiated deviations.

19 AIR TRAFFIC CONTROLLER PERFORMANCE 89 Automation Tools to Support ATCos Under FF As noted previously, an alternative to the advanced FF regime simulated in this study is to modify the existing ATC system with additional ground-based automation tools for ATCos (Wickens et al., 1998). The FAA FF Phase 1 Plan (FAA, 1997) also calls for the development of both ground and airborne automation in future ATM. This raises the question of how such automation tools should be designed. The results from this study have some implications for the development of ATC automation. Historically, aviation automation has been implemented to enhance performance of the overall system and optimize pilot workload (Billings, 1996). Although clearly providing benefits, automation has also been associated with potential costs, including the introduction of new error forms (Sarter & Woods, 1995), peripheralization of the operator (Billings & Woods, 1994), increased monitoring demands placed on the operator (Parasuraman, Mouloua, Molloy, & Hilburn, 1996), reduced awareness of the airspace and of aircraft systems (Endsley, 1996), and imbalances in operator workload (Wiener, 1988; see also Parasuraman & Riley, 1997, for a recent review). Automation in the ATC environment has not proceeded with the same pace as cockpit automation and to date has involved relatively low-level functions such as aircraft data integration and replacement of verbal procedures for hand offs. However, automation of higher level cognitive functions are currently underway in ATC. Examples include datalink, CTAS, and conflict probe. Human factors evaluation studies of some of these systems have been reported (e.g., Hilburn, Jorna, & Parasuraman, 1995; Kerns, 1991; Small, Hammer, & Rouse, 1997) with mixed results. What characteristics should ATC automation tools have to minimize some of the human performance costs that have been noted previously with cockpit automation? Wickens et al. (1998) suggested that high levels of automation should be pursued for information acquisition and analysis functions. For decision-making functions, however, they suggested that full automation should not be implemented for high-risk functions, such as issuing clearances in congested airspace, but that the ATCo should retain some responsibility. This recommendation was based on the previously mentioned findings of problems of new error forms, reduced situation awareness, complacency, and manual skill degradation that have been associated with high levels of automation (Parasuraman & Riley, 1997). Wickens et al. (1998) suggested that failure recovery might be particularly compromised under high levels of automation in which all decision-making responsibility is ceded to computers. High-level decision automation and advanced FF will reduce the time available to recover from an emergency situation because of reduced separation and a denser airspace. At the same time, the out-of-the-loop problems of complacency, reduced traffic awareness, and skill loss will result in

20 90 GALSTER, DULEY, MASALONIS, PARASURAMAN ATCos requiring greater time to respond to such emergency situations. The results of our study of mature FF show that ATCos may not identify conflicts within this time period, if at all, and increased levels of traffic produced shorter advanced notification times. Figure 7 shows the trade-off between the time available and the time required to respond to an emergency. When the latter exceeds the former (e.g., the right-hand side of Figure 7), failure recovery may be severely compromised. The results support previously expressed concerns about the ability of ATCos to monitor airspace under mature FF (Parasuraman et al., 1998; Wickens et al., 1998). At the same time, however, the results also provide a benchmark against which performance improvements can be measured through the use of automation tools particularly information automation that supports but does not replace the decision-making responsibilities of the ATCo. The results reported here provide a baseline assessment of mature FF against which the benefits of such ground-based automation support tools for the ATCo can be evaluated. One example of such a tool is the conflict probe, which could improve conflict detection and allow the ATCo more time to consider available options. Our results, if replicated and extended to other airspace and traffic scenarios, could be used to set performance parameters for these and other automation support tools. For example, the advanced notification time of 157 sec that we observed for self-separation maneuvers, detected in the high traffic condition, suggests a minimum notification time that a ground-based automated conflict probe should provide under mature FF. This corresponds well to the minimum time required by the system operator for failure recovery as outlined earlier. Of course, development of automated tools will also need to consider conflict resolution (in addition to conflict detection) times, as well as the integration of detection and resolution decisions between the air and the FIGURE 7 Failure recovery time required and available under differing levels of automation.

21 AIR TRAFFIC CONTROLLER PERFORMANCE 91 ground. Corker, Pisanich, and Bunzo (1997) recently used a computational human performance model, Man Machine Integrated Design and Analysis System, to predict flight crew times to initiate evasive maneuvers to maintain separation under different conflict scenarios. For a scenario requiring ATC intervention (because one aircraft was not equipped for conflict detection), they obtained a mean time of 134 sec for resolution of a 90º encounter geometry but indicated that longer times might occur for shallower encounter angles. Human performance data such as these, as wellasthoseofthisstudy,canbeusedtosetdesignparametersforbothair-basedand ground-based conflict detection tools. CONCLUSIONS The results of this study have a number of implications for advanced ATM concepts, including FF. It is widely accepted that the ATC system cannot remain as it is today but must change in response to the projected worldwide increase in air traffic. FF technologies and procedural changes will increase the density of airspace and allow for flexibility in flight paths. However, these results suggest that system safety might be compromised under an advanced FF regime in which all authority for separation decisions is ceded to airborne systems but in which ATCos are nevertheless required to intervene when airborne self-separation fails or is not possible for other reasons, such as weather. To the extent that ATCo intervention is required to resolve conflicts that cannot be handled by airborne systems, our results indicate that unaided ATCo performance may not be sufficient to meet the challenge, particularly in dense airspace. Accordingly, automation tools must be developed to support ATCo performance under FF. Several such tools are currently in various stages of development and field testing, including CTAS, URET, ADS B, and TMA. These tools must be designed at a level of automation that supports rather than replaces the decision-making functions of the ATCo. High levels of decision automation coupled with increased traffic density reduce the time available for ATCo intervention, which, as our results indicate, can be relatively sluggish when ATCos are managing the airspace by exception. The results on intervention times in this study, as well as those of other empirical and computational analyses of ATCo RTs, can be used to set design parameters for both air-based and ground-based automation tools. ACKNOWLEDGMENTS This work was supported by Grant NAG from the NASA Ames Research Center, Moffett Field, CA (with Kevin Corker as technical monitor). We thank all of the controllers who participated in this study, as well as Mike and Kimberly Connor from the National Aviation Research Institute for their assistance in recruiting controllers.