FULLY AUTOMATIC AND OPERATOR-LESS ANOMALY DETECTING GROUND SUPPORT SYSTEM FOR MARS PROBE "NOZOMI"

Proceeding of the 6 th International Symposium on Artificial Intelligence and Robotics & Automation in Space: i-sairas 2001, Canadian Space Agency, St-Hubert, Quebec, Canada, June 18-22, 2001. FULLY AUTOMATIC AND OPERATOR-LESS ANOMALY DETECTING GROUND SUPPORT SYSTEM FOR MARS PROBE "NOZOMI" NAOMI NISHIGORI Fujitsu Limited 9-3, Nakase 1-chome, Mihama-ku, Chiba-shi, Chiba 261-8588, Japan Phone: 81-43-299-3248 Email: RHE00337@nifty.com MASASHI HASHIMOTO Institute of Space and Astronautical Science (ISAS) 3-1-1 Yoshinodai, Sagamihara, Kanagawa 229-8510, Japan Phone: 81-42-759-8352 Email: masashi@nnl.isas.ac.jp AKINARI CHOKI Institute of Space and Astronautical Science (ISAS) 3-1-1 Yoshinodai, Sagamihara, Kanagawa 229-8510, Japan Phone: 81-42-759-8386 Fax: 81-42-759-8387 Email: akinari@nnl.isas.ac.jp MITSUE MIZUTANI Fujitsu Advanced Solutions Limited 1-2-4, Shinkoyasu, Kanagawa-ku, Yokohama-shi, Kanagawa 221-0013, Japan Phone: 81-45-438-2082 Email: MLA10503@nifty.com Keywords: expert system, GEOTAIL, Mars probe, monitoring and diagnostic expert system, NOZOMI, satellite abnormalities, scientific satellite, spacecraft abnormalities, rule-based reasoning, heuristic classification problem. Abstract This paper describes a fully automatic and operatorless anomaly detecting ground support system named ISACS-DOC for Japan s first Mars probe NOZOMI. This system is the second application of expert technique to daily operation of scientific spacecraft at the Institute of Space and Astronautical Science (ISAS) in Japan. Many improvements are achieved based on the experiences of the first expert application to monitoring and diagnosis system for geomagnetic observation satellite GEOTAIL. ISACS-DOC is now used to keep the safer operation of NOZOMI as well as the first system for GEOTAIL at ISAS Sagamihara Spacecraft Operation Center (SSOC). ISACS-DOC has caught many abnormalities including serious problems which actually happened to NOZOMI and warned to NOZOMI operators about them immediately at SSOC. ISACS-DOC for NOZOMI will be continuously used until the end of the NOZOMI mission to keep the operation safe under strict limitation of the operation budget at ISAS. In this paper, the process of ISACS-DOC development, the improvement points that were performed, and some actual examples of abnormalities that were detected by ISACS-DOC will be reported as well as the outline of ISACS-DOC itself. 1. Introduction The study of diagnostic expert system has been becoming active since the second half of 1980s and also been applied to the space field. The first stage of diagnostic expert system treated problems as heuristic classification problem, namely, the diagnosis was conducted by forward or backward

inference engines referring the knowledge database that was previously defined using specialists knowledge or experiences. ISAS also decided to apply rule-based reasoning diagnostic expert system to perform the safe operation of the spacecraft control on the ground with small number of operators. This system was named Intelligent SAtellite Control Software -DOCtor (ISACS-DOC) and its first version was developed and has been used to geomagnetic observation satellite GEOTAIL for almost nine years. Although model-based reasoning diagnostic expert system has been very active and many reports have been able to find nowadays, ISAS decided to apply the same technique to the second version of ISACS-DOC and successfully developed the fully automatic and operator-less anomaly detecting ground support system for Japan s first Mars probe NOZOMI. Two ISACS-DOCs for GEOTAIL and NOZOMI are both backing up daily spacecraft operations powerfully in ISAS. This paper reports on many trials and ideas on developing ISACS-DOC and also the latest status of ISACS-DOC for NOZOMI. 2. Process of ISACS-DOC Development 2.1 Basic Concept of ISACS-DOC Following is the basic concept in designing ISACS- DOC. Although these ideas were discussed at the beginning of the first version of ISACS-DOC for GEOTAIL launched in 1992, they were not changed throughout the second version of system development for NOZOMI launched in 1998. (1) Operators can easily recognize abnormalities of satellites/spacecraft by being provided appropriate monitoring and abnormality detection capability even if they do not have enough knowledge. (2) Workload on operators to use ISACS-DOC should be minimized. (3) Knowledge database can be easily modified to correspond to an unexpected situation after launch. (4) A fatal damage of the satellites/spacecraft can be avoid even if the operator fails to find a complete solution to the trouble by taking first aid action. (5) The necessary information for diagnosing satellites/spacecraft can be obtained not only from the telemetry data but also from the ground tracking stations. 2.2 Four Steps on GEOTAIL System Development It was the first trial for ISAS to develop diagnostic system for the actual scientific spacecraft, and we developed the system in following four steps: (1) Prototype system for the limited subsystem : We developed a prototype system to evaluate the effectiveness of the rule-based reasoning diagnostic system targeting to the spacecraft commutation system only. We decided to use a commercially available application software to reduce the development cost. This diagnostic expert tools package running on the OS/2 environment had three standard functions of the knowledge database editor, the forward inference engine, and the rule-based knowledge database. (2) Diagnostic system operated in manual mode : We developed the second-step prototype system targeting the whole subsystems of the spacecraft. The necessary data for diagnosis was manually input to the system as we did not have much confidence that the system was useful to the spacecraft operation. (3) Diagnostic system operated in on-line data feeding mode : It took 30-40 minutes to input necessary data to the system in manual mode, though it was recognized that the system could diagnose the spacecraft status almost correctly. The system was revised to the on-line data feeding system as almost all necessary data was input on a real time basis from telemetry data. (4) Diagnostic system cycled automatically : It was strongly required to watch whole spacecraft status all the time automatically during the real time operation and to warn operators of the spacecraft abnormality when some trouble happened. The system upgraded to repeat the diagnosis every three minutes during the real time operation. In a spacecraft operation, a significant amount of data is produced and several pages of quick look screens are necessary to display them. The operator would be flustered without the appropriate knowledge on how to deal with the identified trouble. We noticed through the actual spacecraft operation that ISACS- DOC was very useful to avoid overlooking the spacecraft contingencies as the system automatically checking 570 items of spacecraft health condition every few minutes. ISACS-DOC for GEOTAIL found several spacecraft contingencies in the Page 2

Figure 1 Monitor Screen of Solar Particle Monitor (SPM) Including Reproduced Data communication subsystem, control subsystem and scientific instruments and helped the spacecraft operators to cope with them. 2.3 New Ideas on NOZOMI System Development Learned from GEOTAIL experiences It is possible to accumulate statistical data of the diagnostic knowledge for cars, electric appliances for home use because there are plenty of exactly the same or almost the same products on the market. It is very easy to obtain the information about what kind of troubles happens at what rate. Using this information, it is possible to analyze the troubles statistically. However, there is no exactly the same scientific spacecraft. So the knowledge database for a scientific spacecraft mainly depends on the design knowledge. For this reason, a wide variety of design knowledge for developing spacecraft has to be accumulated in the diagnostic expert system. The knowledge database of the ISACS-DOC for GEOTAIL was made using a lot of design knowledge obtained by interviewing 30 or more experts. We also decided to apply the rule-based reasoning diagnostic system for NOZOMI from the following reasons, though the model-based reasoning technique becoming active: It would have been very hard and difficult to verify the diagnoses model of NOZOMI before spacecraft launch with low cost, if the model-based reasoning technique was applied to the system. Many experiences and knowledge obtained from the system development of ISACS-DOC for GEOTAIL seemed to be useful and applicable to other future satellite operations by relatively little efforts. ISACS-DOC for NOZOMI is developed according to the following ideas. Some of them are introduced based on lessons learned from ISACS-COD for GEOTAIL". (1) Wide range of information can be gathered through an on-line network to realize extensive and reliable diagnosis, and all data used for diagnosis are saved in the system and can be used for checking a long-term trend of the data. (2) Only reliable results should be shown in order to avoid giving operators unnecessary concerns. Only related facts should be shown if the Page 3

coordinate the differences between actual status of NOZOMI after launch and the preset-knowledge database before launch, to add insufficient knowledge and tune the data collection function. It took another 5 months to finely adjust the knowledge database through daily operations of ISACS-DOC. After that, operation of ISACS-DOC was placed under the control of the NOZOMI operation team on June of 1999. Furthermore, following new functions were added to ISACS-DOC for NOZOMI according to the actual spacecraft operation until now. Figure 2 Window of Long-Term Power Supply Monitor diagnostic result is uncertain. To perform this idea, parallel formed decision tree is more appropriate for the spacecraft diagnosis than serially formed decision tree. (3) Since NOZOMI is the deep space mission, communication links should be carefully watched by comparing the real receiving levels of downlinks and up-links with estimated valued which are calculated from antennas patterns, attitude of NOZOMI, distance between NOZOMI and ground tracking antenna, performance of ground station, etc. (4) Transmission time between NOZOMI and ground station should be taken into account in designing ISACS-DOC since it reaches up to 30 minutes of round-trip time. This is very important when NOZOMI data and ground station data are used together. (5) Important items for diagnosing should be reflected on spacecraft designing. (1) Diagnosis for Data from Onboard Data Recorder NOZOMI is operated using only one tracking antenna at Usuda Deep Space Center (UDSC) of ISAS usually. SSOC cannot make contact with NOZOMI during the invisible time for long hours. In addition to that, NOZOMI lost its S-band telemetry capability after launch because of the TMS malfunction that happened on July 5, 1999. SSOC can not see the real-time data while the remaining X- band transmitter is sending reproduced data from the onboard data recorder. Therefore, it is very important for ISACS-DOC to be able to diagnose the data sent from the onboard data recorder as well as the real-time data. The NOZOMI operation team can quickly see what happened during the invisible time by this capability. Figure 1 is an example of the Solar Particle Monitor (SPM) including reproduced data from the data recorder onboard. 2.4 Many Improvements after NOZOMI Launch Five months of test running were allocated to Figure 3 Screen Display of Range-Rate Measurement Page 4

Figure 4 System Configuration of ISACS-DOC for NOZOMI (2) Trend-Graph Drawing of Long Period Data A long-term variation of some data is essential to investigate the cause of degradation, malfunction and so on. For examples, solar-cell efficiency, battery capacity, A/E of thermal control material, etc. change slowly with time. Figure 2 is an example screen window of the long-term power supply monitor shown by ISACS-DOC for NOZOMI. (3) Diagnosis of Range-Rate Measurement System The trajectory of NOZOMI is determined from the data obtained by range and range-rate measurement systems. ISACS-DOC is watching the health of the range-rate measurement system by monitoring the difference between the measured value and expected value that is calculated from the latest trajectory elements. This function is also helpful to monitor the performance of trajectory change operation roughly though its major role is to watch the health condition of the range-rate measurement system. Figure 3 shows a screen display of the range-rate measurement monitor. 3. Running Status of Latest ISACS-DOC According to the above mentioned implements, the fully automatic and operator-less anomaly detecting ground support system are powerfully backing up daily NOZOMI operation The outline of ISACS- DOC for NOZOMI including system configuration, diagnosis flow, and also some actual examples of diagnosing results will be introduced in this section. 3.1 System Configuration ISACS-DOC consists of two computers as shown in figure 4. They have the following functions. (1) Data collecting function This function collects the data to be monitored and diagnosed from each data server of various ground operation systems through ISAS ground network in real-time, edits the files of these data, and transfers these files to another computer that has the diagnosis function. Below are the data to be collected. Telemetry data sent from the spacecraft. Status data of ground tracking station. Orbit data. Page 5

Figure 5 Diagnosis Screen of TMS Temperature High Aspect data of NOZOMI. This function is repeatedly practiced while the computer is working. DOC. About 610 of questions are automatically issued and approximately 460 of results are prepared in the present ISACS-DOC. (2) Diagnosis function All actual diagnoses are practiced on a personal computer (PC). This function is realized with the commercially available application software named Manadeshi-kun running on Microsoft Windows NT environment. The diagnostic information files made by the data collecting function are transferred to the PC automatically and used for diagnosis. The rule-base knowledge database of NOZOMI is stored on the PC. Diagnosis is executed on the basis of this database. The database can be easily modified according to the actual-status change of NOZOMI. The data flow in the data collecting function are also shown in Figure 4. The scale of knowledge database in ISACS-DOC is often changed according to status change of NOZOMI and newly obtained information that is useful to improve the diagnostic capability by ISACS- 3.2 Diagnosis Flow ISACS-DOC for NOZOMI is always waiting for the diagnosis before the start of NOZOMI operation at SSOC. This system automatically starts diagnosing when the necessary data are prepared in the WS in Figure 4 during real-time operation. The diagnosis is cycled automatically about every 5 minutes. The diagnosis pauses when the data is not prepared for some reason, for example, in case that the ground-tracking antenna is not ready. The diagnosis is executed in all of the areas of NOZOMI. ISACS-DOC warns the operators when it finds something abnormal. The operators can get the information about the details of the abnormality including first aid action to rescue NOZOMI from fatal damage that may be caused by the abnormality. Five standard windows on the screen of ISACS-DOC for NOZOMI are Page 6

Figure 6 Diagnosis Screen of Helium Gas-Tank Pressure Low prepared to display the diagnosed information of abnormal items, explanation of each abnormality, normal and actual status/values causing the abnormality, related figures/graphic data, urgent levels, contact information such as telephone numbers of senior engineers/scientists who can supervise the further contingency operation for the abnormality, urgent commands to save the probe from catastrophe (first-aid commands), and some common data like distance between the probe and the earth, etc. as shown in Figure 1. 3.3 Operation Results ISACS-DOC has issued a lot of warnings including some serious unexpected problems. The screen in Figure 5 warns the temperature of the S-band transmitter (TMS) onboard NOZOMI is rather high, though it is still within the allowable range. However, the temperature change should be carefully monitored after this caution. The related data like the temperature history of TMS and the first aid action, if necessary, automatically appears on the screen. Figure 6 is another and more hazardous example. A major malfunction of helium gas leakage in the propulsion system of NOZOMI suddenly occurred at beginning of May in 1999. ISACS-DOC showed the primary pressure of the helium gas-tank at a dangerously low level and still decreasing. The leakage was stopped by sending the proper commands. Fortunately, there was no serious impact on the Mars observation plan by NOZOMI. Monitoring and diagnostic items related to the serious problems are carefully refined and reinforced. Communication-link monitor is one of the examples. The difference between actual and estimated receiving levels of S-band up-link and X-band down-ink is reduced by reevaluating the performance of the communication system at UDSC, antenna patterns and temperature dependence of AGC calibration data of the receivers onboard NOZOMI. Figure 7 is a window screen of the communication links monitor. 4. Conclusion ISAS has been studying to develop the rule-base reasoning diagnostic expert system to perform the safe Page 7

Figure 7 Screen Display of Communication Links Monitor of ISACS-DOC spacecraft operation under strong demand of cost reduction since 1990. After many years trials and efforts, ISAS successfully developed a fully automatic and operator-less anomaly detecting ground support system called ISACS-DOC for Mars probe NOZOMI. ISACS-DOC for NOZOMI has been used to keep the safer operation of NOZOMI on dairy basis at ISAS from February, 1999. The effectiveness of this system has been already shown by finding some abnormalities of NOZOMI. NOZOMI will be injected into Mars orbit on January 2004 and collect scientific data to study the structure and dynamics of the Martian atmosphere on its interaction with solar wind approximately for 2 years after the injection. ISACS-DOC for NOZOMI will also support this operation powerfully. The discussion of next ISACS-DOC for sample-return mission of ISAS named MUSES-C has already stated. A lot of lessons learned through the development of ISACS-DOC for NOZOMI will be introduced into the next ISACS-DOC. The authors would like to thank all of the people for giving us precious information of each onboard instruments of NOZOMI and ground support systems. These information are essential to construct the knowledge database of ISACS-DOC. The authors also would like to acknowledge Professors Ichiro Nakatani, Toshifumi Mukai, and Hajime Hayakawa of ISAS for their great supports of this development. References [1] Hashimoto, M. Nishigori, N., and Mizutani, M.: Running Status of Monitoring and Diagnostic Expert System for Mars Observer NOZOMI, Proceedings of the 22 nd International Symposium on Space Technology and Science, ISTS 2000-f- 08, 2000. [2] Yamada, T.: Cost Effective Development of Communication Systems for Space Operations, Proceedings of the 2 nd International Symposium on Reducing the Cost of Spacecraft Ground Systems and Operation, RAL.GS2.12, pp.12.1-12.7, 1997. [3] Hashimoto, M. Nishigori, N., and Mizutani, M.: Operating Status of Monitoring and Diagnostic Expert System for Geomagnetic Satellite GEOTAIL, Proceedings of the 2 nd International Symposium on Reducing the Cost of Spacecraft Ground Systems and Operation, RAL.GS2.73,pp.73.1-73.8,1997. Page 8