Tracker Quality Monitoring by Non-Dedicated Calibration Flights

Tracker Quality Monitoring by Non-Dedicated Calibration Flights Matthias Hess, Ralf Heidger TM/SP Deutsche Flugsicherung GmbH (DFS) Langen, GERMANY {matthias.hess ralf.heidger} @ dfs.de Jochen Bredemeyer FCS Flight Calibration Service GmbH Braunschweig, GERMANY brd @ flightcalibration.de Abstract Sensor and tracking quality analysis is a key factor in the quality assurance in surveillance for Air Traffic Control (ATC) at an Air Navigation Service Provider (ANSP) like the DFS. In this paper we present a collaboration infrastructure that allows automatic tracker quality analysis by using results of regular ferry and mission flights conducted for inspection of navigation aids. The goal of this infrastructure is to allow the two companies Flight Calibration Service GmbH (FCS) and Deutsche Flugsicherung GmbH (DFS) to combine their respective expertise in an efficient manner. Flight Calibration, Tracker Evaluation I. INTRODUCTION The quality of a tracking system can be estimated in several ways which should be combined for a thorough analysis: Evaluation against synthetically created scenarios Evaluation against reconstructed trajectories Evaluation against calibrations flights When employing synthetic scenarios, one generates artificial sensor data on which the tracker operates. Usually the true trajectory is perfectly known, but the simulated sensor data do not necessarily reflect real behavior. The tracker is evaluated against this simulated trajectory. At DFS the AFS simulator [1], [12] is such a tool to create tracker testing scenarios. In order to evaluate the tracker against traffic of opportunity, an offline approach is quite common. A trajectory is reconstructed from recorded real world sensor data with algorithms, that are potentially more precise but take more time than online tracking algorithms ([5] or [6], for instance), which prohibits their use in a tracking system. This approach is taken by the well-established SASS-C program developed by EUROCONTROL [3]. Derived from the second method is the so called quasionline quality control used in a program suite developed by DFS [10], [2]. Here a less time consuming algorithm is used for the reconstruction that allows tracker quality analysis in a timely fashion rather than the offline method of SASS-C. A third and more expensive method is using calibration flights. These give a reference trajectory of unparalleled precision in an opportunity traffic scenario, but have spatial and temporal limits. All these approaches are deployed at DFS but the focus is put on the third approach. Radar flight inspection for civil facilities is nowadays usually not performed periodically, there may be only a single flight inspection after deployment. In contrast to dedicated radar flight trials as depicted in ICAO DOC 8071 (Part III) [1], this new approach makes use of the position data collected during ferry and mission flights performed for flight inspection of terrestrial navigation aids. This is a task to be performed on a regular basis according to ICAO DOC 8071 (Part I) [1]. II. OVERVIEW In order to accomplish a flight calibration data based tracker evaluation, several steps are required: Conducting a calibration flight Preprocessing the collected raw data Collecting corresponding tracker data Analyzing tracker data with respect to the flight path data The key for the kind of tracker evaluations that we will describe in this paper is bringing together the calibration flight position data and the tracker data. The former is provided by FCS as a by-product of their calibration flights and tracker data is recorded at DFS. As DFS is doing the tracker evaluation and has the respective data already ready at their hands, a data exchange server between FCS and DFS was setup to provide DFS with the flight path data. The different stages are schematically presented in figure 1. Proceedings of ESAV'11 - September 12-14 Capri, Italy 141

FCS Calibration Flight Flight Position Data Data Exchange Server Track Data dedicated targets having a high precision position vector with timestamps against the GPS second-of-week. The two FCS measurement aircraft with callsigns D-CFMD and D-CFME can be easily identified through their Mode S technical addresses which is delivered from a Mode S sensor monoradar service message (ASTERIX CAT034). A continuous validation of the multi-sensor tracker results allows a long-term quality assurance with no additional flight costs incurred. DFS Tracker Analysis Figure 1. Principal data flow in a flight calibration data based tracker analysis. In section III we focus on the calibration flights and the processing of the resulting data, i.e. steps 1 and 2. In section IV we describe in a more detailed fashion the tracker evaluation. Finally, to show how we employ the presented infrastructure, we present an evaluation based on opportunity traffic with corresponding calibration flight data in section V. III. CALIBRATION FLIGHTS Under normal operations, a single FCS flight inspection aircraft spends more than 25 hours a week on ferry and mission flights. This mission requires that the flight inspection system (FIS) has a reference position estimate significantly more accurate than that of the facility under inspection. The implemented position estimation technique integrates differential global navigation satellite systems (DGNSS, spacebased, ground-based correction) and an inertial navigation system (INS). A total uncertainty in positioning of < 1 m in the horizontal plane is normally achieved. Furthermore, the attitude vector (roll, pitch, bank angle) is available which helps to identify critical turning maneuvers. These potentially block line-of-sight transmissions from the to the ground sensor, resulting in a degraded slant range or monopulse estimation of a certain beam dwell. Position data is generated and recorded always when airborne, so there is a huge dataset of 25 flight hours available by the end of the week when one crew returns to their home base. Flight inspection missions of the two FCS aircraft cover, among others, dense areas as Frankfurt, Munich, Vienna and Zurich terminal manoeuvering area (TMA) within coverage of at least one Airport Surveillance Radar (ASR) and two midrange radars (SREM). The mission traffic serves as a valuable source to obtain the necessary data basis to check the radar tracker results against a reliable, known target. There is no ordinary traffic used but two IV. TRACKER QUALITY ANALYSIS As mentioned above there are several ways to measure the quality of a tracking system - once one has defined the respective measures. For the purposes of this paper tracking quality can roughly be divided into two main areas, track detection and tracking accuracy. A track detection analysis (TDA) tries to measure how well the tracker finds real targets and can distinguish them from false ones. It also answers questions about the ability of the tracker to follow the target throughout its flight. On the other hand, a track accuracy analysis (TAA) is concerned with the precision of the tracker output, i.e. how closely the tracker follows a target. We use calibration flights for tracker quality analysis for the purposes of this paper. Here the reference trajectory is given by high-precision FIS data collected aboard a calibration aircraft. These data are usually much more precise than ordinary sensor data. Hence, the basis for tracker evaluation is much more sound. The drawbacks of this method are the costs for calibration flights and their restricted data set, spatially as well as temporally. These flights can cover a certain area of interest and for a limited period of time only. But for very exact tracker evaluation there is no other reference data being more accurate: These data are obtained in a real-world scenario (rather than a synthetic scenario) and have unique precision. A. Tracking Quality Tracking quality can be roughly divided into two domains, TDA and TAA. The former indicates the ability of a tracking system to reliably detect any real target and to distinguish it from erroneous sensor signals. The latter measures the accuracy with which a system follows real targets. Commonly used measures for TAA are deviations of the tracking signal from the true trajectory across and along its path. B. Accuracy Analysis Two of the most commonly used quality measures for tracking systems are the across trajectory and along trajectory deviations. Since it is of interest to have as few quality measures as necessary, one combines the deviations during the time of the trajectory into a single root mean square (rms) value. 142 Proceedings of ESAV'11 - September 12-14 Capri, Italy

If we have a quantity that depends on time then the rms value of that quantity is defined as follows: where and are the start and end time over which is defined and denotes the duration of. For our analysis we use the across and along deviations, and, respectively. Trajectories can be modeled as mappings from time into state space that is spanned by position, velocity and acceleration. Here we neglect the acceleration and consider the velocity to be equal to the first time derivative of the position. So we can reduce a trajectory to a mapping from time to position: and we can set Figure 2. Data of an actual calibration flight shown in the AWP that is used for analysis purposes. We show in figure 3 a detailed comparison of the calibration flight data and the actual tracking output. Most of the time the tracker does not deviate significantly from the reference trajectory. So we obtain for and the distances across and along two trajectories and the following expressions: and where denotes the Euclidean norm of a quantity. And finally, we define and to be the rms values of and respectively. For data in the WGS84 frame [13], the Euclidean distance must be replaced by a numeric approximation of the true distance on the spheroid. V. LIVE EXAMPLE This section finally shows how the flight calibration and the tracker evaluation work together. We present how the chain links from collecting flight position data to an actual tracker evaluation work together. For the tracker evaluation we use the Analysis Working Position (AWP) developed at DFS [10] together with an extension of the Batch Estimator (BE), an analysis module for the AWP that was presented in an earlier paper [2]. Figure 2 shows the course of one of the calibration flights that was used in our analysis within the AWP. The data of that flight were collected for a different purpose but as a by-product they could be used for a tracker evaluation. Figure 3. Comparison of the tracker output (green dots) and the reference trajectory obtained from the calibration flight. The indicated deviation is about 0.13 nm. As described in section II there are two different data sources that are combined together for the final evaluation: Data recorded during a calibration flight and sensor and tracking data of an ATC system that covers the time and location frame spanned by the calibration flight, i.e that contains data for the whole calibration flight. In order to demonstrate the potential of our analysis system, we have chosen not to take data from an operational tracking system, but from a system that is not tuned optimally. Additionally we do not use all sensor data that is available. This forces the tracker to produce errors that can be detected by our automated evaluation. A. Data Collection The calibration flight data is collected on-board during the flight. The data is processed and put on a server. At DFS this Proceedings of ESAV'11 - September 12-14 Capri, Italy 143

server is checked regularly for data updates and newly arrived data are downloaded and processed. If a new data set is available, the corresponding sensor and tracking data are retrieved from an archive at DFS. These data are then merged together to form the basis of the tracker evaluation. B. Data Association The time bases of these data will be different. The calibration flight is based on GPS time and the ATC system is fixed to UTC. So these time bases have to be adjusted as a first step in merging them together. The next step is data association. Data of the calibration flight must be correlated with the corresponding tracker output. Only then is an evaluation meaningful. So far we use only a few correlation steps: Firstly, we identify the 24 bit Mode S address of the calibration aircraft. This data is unique and stored in the position data file. From the tracker recording we extract the corresponding track numbers, if sensors with Mode S capabilities are available and cover the calibration flight. Then data from regions without Mode S coverage are correlated based on these track numbers. There can be several track numbers for the calibration flight if the tracker fails to detect a continuous track. In a final step we correlate tracking data to the reference trajectory by statistical and geometric measures. The calibration data contain, amongst other, the standard deviation of the position and velocity. So it is possible to use statistical distance measures if the tracker output contains the specific data, too. This is not guaranteed because EUROCONTROLs ASTERIX standard does not require these data to be present in tracker output. If these data are not available we use simple geometric distances for correlation. C. Automated Analysis The associated data obtained in the association step is used to calculate several measures that provide indication of the quality of the tracker. Ideally the quality would be indicated by one or few numbers only, like the rms of across and along distances. For a thorough understanding of the tracking behavior this is usually not sufficient, so we calculate more values in our automated analysis. Our TAA analysis for this calibration flight gives an overall rms value for the deviations of 333.6 m across and 315.9 m along, respectively. Although not very good for operational systems these values are acceptable. Surprisingly the value for the along deviation is about the same as the value for the across deviation. One would expect this to be much smaller as large along deviations usually indicate a problem related to time. So further investigations into the reasons for this behavior is required. There is another automatically calculated quantity at our disposal, the histogram of deviations across and along. Figure 4 shows those histograms. As expected there are many small deviations, indicating that the tracker usually performs well, but there are too many large deviations which add to the rms distance values. Figure 4. Non-normalized histogram of deviations across and along. We have chosen a logarithmic scale in order to see the small frequencies properly. When we plot the deviations over the trajectory time (figure 5) it is easily seen that there are few spots with very large deviations. This gives hints for manual inspection. Figure 5. Deviation plotted over the time of the trajectory. Remarkably, there are a few spots with very large deviations. These are caused by track drops and the fact that the tracks are interpolated in the calculation of rms values. Manual Analysis At first glance an automated analysis is a good indicator for the tracker working properly. So if, for instance, the rms values are always well below a certain level, no further analysis would have been required. But in case of unexpected peaks, an automated analysis can give valuable hints but cannot in general obtain the reasons for erroneous behaviour of the tracker. So those cases require a manual intervention. The automated analysis already identified hot spots in the deviation. An indication of why the tracker shows these large deviations can be obtained through figure 6. Here the correlation between flight altitude and deviation is shown. For low altitudes the tracker shows large deviations. 144 Proceedings of ESAV'11 - September 12-14 Capri, Italy

Figure 6. Correlation between altitude and devaition. For low altitudes the deviation becomes excessively large. Through the time of day one can identify the situation that caused these deviations: Figure 7. Tracker output vs reference trajectory at the time of day where there are the hottest spots. This figure shows a part of the calibration flight close to an airport. The largest deviations are found near the airport. Obviously the tracker drops the track and continues it at another location (figure 7). This is usually caused by an insufficient sensor plot supply. So for certain parts there are no tracks. Note again, that these errors have been induced deliberately and do not reflect the operational performance of the tracker. The lack of sensor plots for that area is confirmed by inspection of the corresponding data set. So we are able to explain the tracking behavior. The airport is not covered adequately by the chosen sensor set. VI. CONCLUSION AND OUTLOOK In this paper we have presented a method that automatically combines data from calibration flights with data from ATC sensors and tracking systems in order to conduct a tracker evaluation. We have shown the feasibility of the presented approach with real data from both sources, the calibration flight data and sensor data from opportunity traffic. In order to force the tracker to show erroneous behavior, we have inhibited some sensors and mistuned the tracker. We then showed a detailed analysis that identified some of the problems that lead to the erroneous tracker behavior. From that analysis the limitations of automated tracker tests become obvious. The results of such tests should not be taken for granted, even if the quality numbers indicate proper work. To find the cause of erroneous behavior, automated tests can be helpful in providing hints on where to look more thoroughly. A combinatorial analysis of different diagrams from an automated test, e.g. the deviation and altitude along the time of trajectory, is very valuable in identifying poor performance. The next version of SASS-C (version 7) is supposed to handle comparisons between different tracking sources, too. We installed that version and converted the calibration flight data to ASTERIX CAT062, as the exchange format for tracking output. Effectively we created a second tracker from the calibration flight data and used that in SASS-C as reference trajectory. We then tried to analyze the above situation with SASS-C. Unfortunately, we were not able to use the whole data set of one and a half hour of German sensor data to obtain meaningful results. The reason is probably the still-beta status of that SASS-C version. Our future development aims at providing and implementing more statistical measures, that furthers facilitates a complete analysis of tracking behavior. We have seen that correlations of basic measures (like the aforementioned deviation and altitude over trajectory time) play an important role in such an analysis. We will transfer these ideas to the batch estimator where the calibration flight reference trajectory is replaced by reconstructed trajectories. We will also focus on the TDA which has been left out in this paper completely. We also hope that further manual analysis of our tracker will result in more experience to improve the algorithms which then lead to a higher degree of automation in tracker evaluations. REFERENCES [1] ICAO Document 8071 - Manual on Testing of Radio Navigation Aids, Volume III (Testing of Surveillance Radar Systems), First Edition 1998. [2] Matthias Heß, Ralf Heidger, "Trajectory Reconstruction for OTQC in the Phoenix Analysis Working Position", in Enhanced Surveillance of Aircraft and Vehicles (ESAV) Proceedings (2010), Berlin, Germany. [3] - SASS-C-UM-MAN-30, ed. 1.90, Eurocontrol, Brussels, 2010 [4] Jesús García, Juan A. Besada, Andrés Soto and Gonzalo de Miguel: Opportunity Trajectory Reconstruction Techniques for Evaluation of ATC Systems, International Journal of Microwave and Wireless Technologies (2009), 1 : 231-238. [5] Jesús García, Andres Soto, Gonzalo de Miguel, Juan Besada, Paula Tarrio: Trajectory reconstruction techniques for evaluation of ATC systems, in Enhanced Surveillance of Aircraft and Vehicles (ESAV) Proceedings (2008), Island of CAPRI, Italy, 198-203. [6] Juan Besada, Gonzalo de Miguel, Andrés Soto, Ana Bernardos: Algorithms for Opportunity Target Reconstruction, in Enhanced Surveillance of Aircraft and Vehicles (ESAV) Proceedings (2008), Island of CAPRI, Italy, 212-217. Proceedings of ESAV'11 - September 12-14 Capri, Italy 145

[7] Radoslav Natchev, Ralf Heidger: Trajectory computation for tracker evaluation and linkage processing, in Enhanced Surveillance of Aircraft and Vehicles (ESAV) Proceedings (2008), Island of CAPRI, Italy, 192-197. [8] Ralf Heidger, Kai Engels: An Infrastructure for Online Tracking Quality Control, in Enhanced Surveillance of Aircraft and Vehicles (ESAV) Proceedings (2008), Island of CAPRI, Italy, 218-224 [9] Ralf Heidger: The PHOENIX White Paper. V. 3.0. DFS Langen 2011. [10] Ralf Heidger, Ha Son Nguyen: An analysis working position for radar data processing quality control in Enhanced Surveillance of Aircraft and Vehicles (ESAV) Proceedings (2007), Bonn, Germany. [11] Ralf Heidger, Thomas Klenner, Roland Mallwitz: The PHOENIX Multi-Radar Tracker System for Air Traffic Control Applications, in: Air Traffic Control Quarterly. Vol. 12, Number 3, 2004, pp. 193-222. [12] Roland Mallwitz: DFS Approach on Tracking System Performance Analysis to determine ATC separation minima in International Radar Symposium (IRS 2005), Conference Proceedings, DGON, Bonn, Germany. [13] Department of Defense World Geodetic System 1984, Its Definition and Relationships With Local Geodetic Systems, NIMA Technical Report TR8350.2, Third Edition, 4 July 1997 146 Proceedings of ESAV'11 - September 12-14 Capri, Italy