NuSTAR Operations Implementation A New Approach from Mission Development to On-orbit Operations

Transcription

1 SpaceOps 2010 Conference<br><b><i>Delivering on the Dream</b></i><br><i>Hosted by NASA Mars April 2010, Huntsville, Alabama AIAA NuSTAR Operations Implementation A New Approach from Mission Development to On-orbit Operations Mark Lewis *, Bryce Roberts, Jeremy Thorsness, Martha Eckert, Renee Dumlao **, William Marchant, Thomas Clemons, and Samuel Johnson Space Sciences Laboratory, University of California, Berkeley, CA Gregory Greer *** the Hammers Company, 7474 Greenway Center Drive, Suite 710, Greenbelt, MD and Manfred Bester Space Sciences Laboratory, University of California, Berkeley, CA The University of California, Berkeley has conducted flight operations for eight NASAfunded satellites from its highly automated Multi-mission Operations Center, located at Space Sciences Laboratory. To implement operations for a new mission, namely the Nuclear Spectroscopic Telescope Array, the Berkeley operations team took a different, proactive approach towards supporting all phases of spacecraft bus and instrument development. Transportable clones of the already operational, integrated ground systems, including spacecraft command and control systems, telemetry data processing and trending analysis systems, and other software tools are pre-configured at Berkeley and deployed at various integration and test facilities. Members of the operations team work directly with the spacecraft contractor and the instrument developers to create telemetry and command dictionaries, telemetry display pages and scripts, and participate in all test activities as console operators. During subsystems and systems testing, telemetry data are streamed from remote locations into a central database at Berkeley to allow for real-time and post-test trending analysis. Tying the operations center into all spacecraft development and test phases reverses the conventional flow of activities and ensures a smooth transition to launch and on-orbit operations. This approach also provides excellent training opportunities for the entire flight operations team, beginning more than two years prior to launch. We describe details of the operations implementation, as well as lessons learned. T I. Introduction HE Nuclear Spectroscopic Telescope Array (NuSTAR) is a new NASA Small Explorer mission, scheduled for launch in early Its prime mission goal is to image high-energy X-ray sources such as black holes with high-resolution, focusing X-ray optics. 1,2 The Project lead institution is the California Institute of Technology (Caltech). Project management and systems engineering functions are provided by the Jet Propulsion Laboratory * Mission Operations Manager, UCB/SSL, 7 Gauss Way, Berkeley, CA Ground Systems Engineer, UCB/SSL, 7 Gauss Way, Berkeley, CA Lead Flight Controller, UCB/SSL, 7 Gauss Way, Berkeley, CA Flight Controller, UCB/SSL, 7 Gauss Way, Berkeley, CA ** Flight Controller, UCB/SSL, 7 Gauss Way, Berkeley, CA Senior Programmer, UCB/SSL, 7 Gauss Way, Berkeley, CA Ground Systems Technician, UCB/SSL, 7 Gauss Way, Berkeley, CA Database Administrator, UCB/SSL, 7 Gauss Way, Berkeley, CA *** ITOS Product Lead, the Hammers Company, 7474 Greenway Center Dr., Suite 710, Greenbelt, MD Director of Operations, UCB/SSL, 7 Gauss Way, Berkeley, CA , AIAA Senior Member. 1 Copyright 2010 by the, Inc. The U.S. Government has a royalty-free license to exercise all rights under the copyright claimed herein for Governmental purposes. All other rights are reserved by the copyright owner.

2 (JPL) at Caltech. Mission operations are conducted at Space Sciences Laboratory (SSL) at the University of California, Berkeley (UCB), and science operations at Caltech. All mission control functions for this new mission are seamlessly integrated into the existing Multi-mission Operations Center (MOC) at UCB/SSL that has already supported eight NASA spacecraft in a highly automated ground systems environment.3 A. Observatory and Concept of Operations The NuSTAR observatory, shown in Fig. 1, consists of two identical, co-aligned grazing incidence hard X-ray telescopes. A focal length of 10 m is achieved by mounting the focusing mirrors at the end of a long mast that is deployed on orbit. A combination of a simple laser metrology system that removes mast flexure along with a star tracker camera head mounted on top of the focusing optics provides fine aspect information.4 The three-axis stabilized spacecraft is based on an Orbital Sciences Corporation (OSC) LEOStar-2 bus, and the total observatory mass is less than 400 kg. The concept of operations includes a launch by a Pegasus XL from Kwajalein Atoll into a 600 x 550 km orbit at an inclination of 6 deg. Once deployed and calibrated on orbit, the observatory carries out long pointed observations of survey fields and of specific science targets. In addition, a number of targets of opportunity may be observed during the 25.5-month long primary mission. Science telemetry data are acquired and downloaded in store-and-forward mode. The Malindi, Kenya ground station, operated by the Italian Space Agency (ASI), provides primary telemetry and command support with four passes per day on average during normal science operations.5 NASA s Space Network (SN) supports special operations, such Figure 1. Deployed NuSTAR observatory. Credit: Orbital Sciences as real-time monitoring of the mast Corporation. deployment operations. B. Novel Approach to Support Mission Development and Integration Ground support equipment for spacecraft bus and science instrument development and integration often utilizes a different architecture and different software tools than those eventually used for on-orbit operations. However, based on lessons learned from previous missions, the NuSTAR operations team at UCB proposed to the Project a reverse, proactive approach for supporting all phases of spacecraft bus and science instrument development, integration and testing from their earliest stages, beginning almost three years prior to launch. This unusual approach takes advantage of the fully integrated multi-mission operations center, and involves providing a portable clone of the Berkeley MOC, called a mini-moc that includes spacecraft command and control systems and other integrated software tools, to the flight systems developers. The operations team has historically been involved in all phases of mission life cycles, particularly during the mission integration and test phase, when flight controllers acted as console operators, providing early training opportunities in preparation for on-orbit operations. However, for NuSTAR, this approach has been taken a significant step further. In addition to delivering integrated hardware and software systems, the operations team also works closely with the spacecraft bus contractor and the instrument science team to support development of telemetry and command databases, scripts and telemetry display pages. The overall approach is expected to allow for a very smooth transition from pre-launch testing to on-orbit operations, reduces the risks typically associated with such a transition, and also reduces development costs dramatically. II. Ground Data Systems Architecture Hardware and software systems supporting on-orbit mission operations often differ from one mission to the next, and historically also differed from those systems used during the development, integration and test phases. The Berkeley flight operations team has steadily worked towards implementing a consistent set of multi-mission ground data systems that can be utilized across missions and mission phases. Once implemented, such a multi-mission 2

3 environment can then be expanded to accommodate new missions at substantial cost savings, as well as lower risk to pre-launch development and integration, and transition to on-orbit operations. As a result, the ground data systems architecture for the NuSTAR mission is a direct expansion of the integrated MOC at UCB/SSL. 3,6 A. Spacecraft Command and Control System The heart of the NuSTAR ground data system, shown in Fig. 2, is the spacecraft command and control system. NuSTAR employs the Integrated Test and Operations System (ITOS) that was initially developed at NASA Goddard Space Flight Center (GSFC), and was later commercialized by the Hammers Company. 7,8 ITOS is used by numerous NASA missions and has been employed at the Berkeley MOC since FAST operations transitioned to SSL in 1999, and currently supports FAST, RHESSI and THEMIS operations. ITOS decommutates Consultative Committee for Space Data Systems (CCSDS) telemetry frames and source packets in real-time, and allows flight controllers to monitor spacecraft state-of-health and to send commands. 9 ITOS also performs limit checking on telemetry parameters and decodes event messages. Scripts are written in the Spacecraft Test and Operations Language (STOL). In the integration and test environment, ITOS interfaces to both spacecraft and instrument test equipment via Transmission Control Protocol / Internet Protocol (TCP/IP) network socket connections. At the Berkeley MOC, ITOS runs on a variety of Sun Microsystems workstations and servers under SunOS 9 or This combination of hardware and software has been very reliable, so that command consoles typically are not restarted for months at a time, and usually only to update SunOS security patches. B. Integrated Software Tools 1. Mission Operations The backbone of the control center automation is provided by the Berkeley Flight Dynamics System (BFDS) that is based on the SatTrack Suite. 11 SatTrack also includes flight dynamics product generation and pass scheduling functions, and manages network connectivity for telemetry (TLM) and command (CMD) flows via the FrameRouter and FrameRelay applications. 12,13 Major software tools developed in-house at SSL include the following: 1) BTAPS the Berkeley Trending Analysis and Plotting System for telemetry archiving, plotting and checking of limit violations within specified time intervals, 14 2) BMPS the Berkeley Mission Planning System to generate Absolute Time Sequence (ATS) tables for onboard execution, 6 and 3) BEARS the Berkeley Emergency and Anomaly Response System for detection of spacecraft and ground systems anomalies and corresponding paging of operations personnel. 15 Auxiliary software tools in the mission operations area are used to configure on-board flight parameter and Relative Time Sequence (RTS) tables, and also include various interface programs. The MOC interacts with multiple institutions to schedule passes with external network assets, such as the Malindi Ground Station, operated by the Italian Space Agency (ASI), 5 the Universal Space Network (USN), and the White Sands Ground Terminal (WSGT) for communications via NASA/SN. The MOC is also responsible for retrieving all engineering and science telemetry data from the spacecraft and performs Level Zero processing functions that include quality checking of recovered telemetry data, time ordering and removal of duplicate packets, and other pre-processing functions. 2. Science Operations Science operations for NuSTAR are carried out at Caltech. The Science Operations Center (SOC) receives the Level Zero products from the MOC on a daily basis and performs all subsequent processing functions. Details are described elsewhere. 3. Flight Dynamics Flight dynamics functions for NuSTAR are based on two-line element (TLE) sets that are downloaded twice per day from the United States Strategic Command (USSTRATCOM) via their Space-Track.org web site. The next step is to perform quality assurance (QA) functions by employing a state vector overlap compare technique to detect sudden changes or discontinuities in the orbital elements. TLE sets are then archived and committed for generation of all ephemeris products, such as view periods for pass scheduling, crossing times of the South Atlantic Anomaly (SAA), and acquisition data for all supporting networks. The observing schedule for astronomical targets is provided by the SOC. At the MOC, times of observation and target coordinates are checked against a number of constraints, such as Earth occultation, Sun and Moon avoidance 3

4 and star tracker fields-of-view. Once required attitude maneuvers are verified and validated, the coordinates are included in an ATS table for upload to the spacecraft. Figure 2. NuSTAR ground data system overview (see text for explanation of acronyms). 4

5 III. Taking the Mission Operations Center to the Spacecraft In line with Berkeley s test-like-you-fly philosophy, and following through with the proposed approach to take the MOC to the spacecraft, the operations team at SSL designed and assembled two transportable ground data systems to support spacecraft bus flight software development and spacecraft integration at OSC. A. Hardware and Software The transportable ground data systems, shown in Fig. 3, are referred to as mini-mocs or cubes. Each system includes one Sun Microsystems V215 server, three V215 workstations, a SonicWall firewall, a data back-up system, an uninterruptible power supply, and cabling harnesses all mounted in a small rack, nearly cubical in shape (hence the term cube). Each cube was configured and pre-loaded with identical versions ITOS, BTAPS, and other software tools to allow the mini-moc to function very much like the Berkeley MOC, but in a stand-alone configuration. Three monitors, keyboards and mice were included with each cube to allow multiple test conductors and subsystems engineers to simultaneously view real-time telemetry on ITOS, and trending data via BTAPS. The first cube was delivered to OSC in late 2008 while the second system was delivered in early April One cube is dedicated to NuSTAR spacecraft testing, moving with it as it travels between integration and environmental test sites. The second cube is an integral component of the high-fidelity spacecraft and instrument simulator, known as the FlatSat, laid out on a bench like a flat satellite, and is shown in Fig. 4. Figure 3. Portable mini-moc, also termed cube due to its shape. B. ITOS Configuration OSC typically uses its own proprietary systems for subsystem and spacecraft integration and testing. However, two previous NASA Small Explorer missions that shared a strong spacecraft bus heritage with NuSTAR were developed using the customer s command and control system. The Solar Radiation and Climate Experiment (SORCE) and the Aeronomy of Ice in the Mesosphere (AIM) missions, both of which are also based on a LEOStar-2 spacecraft bus, are operated by the Laboratory for Atmospheric and Space Physics (LASP) at the University of Colorado, Boulder, and employed their Operations and Science Instrument Support (OASIS) command and control system for spacecraft integration and testing, as well as for on-orbit operations16. As a starting point for NuSTAR, UCB s operations team members took advantage of the similarity between OASIS and ITOS, and converted the existing SORCE/AIM command and telemetry databases, as well as page display definitions and test scripts from OASIS to ITOS format. The operations team also supported the initial creation of new database definitions, telemetry pages, and scripts specific to NuSTAR. OSC flight software and subsystems engineers further modify these items with assistance by Figure 4. Spacecraft flight simulator the Berkeley team. (FlatSat). All ITOS telemetry and command dictionaries, telemetry page configuration files, and STOL scripts plus operational software tools are stored in a central repository at UCB/SSL, using the Apache Subversion (SVN) configuration management system.17 Scripts and page definition files that are modified during a test session are checked back into the main repository and are subsequently available to all NuSTAR users. Because the cubes are configured like the Berkeley MOC, STOL 5

6 scripts, shell scripts for automation support, and other software tools run on the cubes in the same was as in the MOC. IV. Support of Instrument Development and Testing In parallel with support of the early stages of flight software development for the spacecraft bus at OSC, ITOS is also used to support various stages of instrument development and testing at Caltech. However, unlike the spacecraft bus, all versions of the instrument hardware and software generate CCSDS packets that are not directly compatible with ITOS in the flight-like configuration used at the MOC. Once integrated with the spacecraft, packets generated by the instrument are then encapsulated into Channel Access Data Units (CADUs) by the spacecraft bus prior to transmission to the ground. Since the spacecraft bus is not available during instrument development and testing, a Minimum Instrument Simulator and wrapper software called Packetizer are developed as a temporary stand-in to allow testing the instrument in a flight-like configuration in the absence of the spacecraft bus. A. Minimum Instrument Simulator and Packetizer The Minimum Instrument Simulator, shown in Fig. 5, consists of a Field Programmable Gate Array (FPGA) that generates basic instrument packets in CCSDS format and delivers these via a RS-422 interface. A Sealevel Systems Model 4203 unit converts these data to TCP/IP format. The Packetizer code then reads the instrument packets and wraps these in standardized CCSDS compatible CADUs, consisting of a commonly used, 10-byte long telemetry data delivery header, the standard Attached Synchronization Marker (ASM), the Virtual Channel Data Unit (VCDU), also known as telemetry transfer frame, and the Reed-Solomon codeblock for processing by ITOS. This simple hardware and software solution allows usage of all operational software tools in both the minimoc and the MOC environment for all stages of instrument development, integration and testing. B. Instrument Test Configurations With the availability of the Minimum Instrument Simulator and the Packetizer code, a wide variety of flight-like test configurations, shown in Fig. 6, are feasible. Configuration A is Figure 5. Minimal Instrument Simulator interface the simplest configuration for instrument configuration. interface testing and verification at the earliest development stage. This configuration allows entire end-to-end data flow testing and early interface verification and validation to occur at the most basic level between the instrument and the ITOS command and control system. Configuration B allows testing of one of the Focal Plane Modules (FPM) or the Optics Bench Electronics Box (OBEB). Configuration C represents the test scenario for the completely integrated instrument suite that consists of Focal Plane Modules A and B (FPMA, FPMB), the OBEB, and the Central Electronics Box (CEB). These configurations are used to test parts of the instrument flight hardware during development and environmental testing. Configuration D shows how the instrument simulator is integrated with the FlatSat environment, and Configuration E represents the telemetry flow from the instrument to the spacecraft bus as the integrated observatory during on-orbit operations. The strategy is to emulate this configuration as early as possible in the development flow. C. Integrated Operations Network Telemetry streams are not only ingested by ITOS for real-time decommutation and display, but are also processed by BTAPS, as indicated in Fig BTAPS is a database system that stores all received telemetry packets in a MySQL database on a server at UCB/SSL, and then allows subsequent retrieval and decommutation for graphing and trending analyses. This software, developed in-house at SSL to support the five-spacecraft THEMIS constellation, already became an essential tool to support all phases of instrument development, integration, and testing. 6

7 A more detailed diagram of the telemetry data flow paths during integration and testing, utilizing the already integrated operations network at the MOC is shown in Fig A sample screen shot with thermal test data of the fully assembled Focal Plane Module A, as processed by the integrated, operational software systems and networks at the MOC is shown in Fig. 8. Figure 6. Various instrument development and test configurations (see text for explanation of acronyms). Figure 7. Utilization of integrated operational network data flows for spacecraft and instrument testing. Various instances of ITOS connect to the FrameRouter system within the secure MOC that facilitates telemetry and command data flows between external ground stations and the ITOS command and control workstations. Real-time telemetry streams are also relayed to another instance of FrameRouter on the Open SSL Local Area Network (LAN) where additional instances of ITOS display telemetry to allow monitoring of spacecraft state-of-health by subsystems engineers. In parallel to ITOS, the real-time telemetry streams are also fed into the BTAPS telemetry database (TLM DB) to allow for subsequent retrieval, plotting and trending analyses. Real-time telemetry feeds are also accepted on the Open SSL LAN from external sources, such as instrument integration and test facilities. 7

8 Figure 8. Sample of telemetry data recorded during instrument environment testing. Top panel: Focal Plane Module A temperatures [C]. Bottom panel: Heater duty cycles [% 2]. V. Operations Team Roles and Responsibilities The NuSTAR Flight Operations Team (FOT) is a part of the larger Berkeley operations team, with team members cross-trained on multiple missions and tasks. Such cross-training results in a team that is well prepared to handle a wide spectrum of anomalies that may arise, that is able to adapt quickly to new situations, and versatile to take on new missions. A. Team Roles with Previous Missions UCB/SSL flight controllers participated as console operators in the integration and test phases of previous missions, namely RHESSI, CHIPS and THEMIS. With all three missions, but most recently for the five-spacecraft THEMIS constellation, the spacecraft buses were transported to SSL for instrument integration prior to environmental testing and launch site integration. Only certified flight controllers were allowed to send commands to the spacecraft to enforce a level of discipline that is essential in preparation for on-orbit operations. Flight controllers also assisted systems and subsystems engineers with writing display pages and STOL scripts. B. Roles and Responsibilities during NuSTAR Integration For NuSTAR, the FOT has been involved from an earlier stage yet, working closely with spacecraft engineers at OSC and instrument engineers at Caltech. OSC has over 1000 command scripts that are re-used for NuSTAR with little modification, aside from the OASIS-to-ITOS STOL conversion. Flight controllers convert these scripts and assist OSC engineers with testing these on the NuSTAR FlatSat. Flight controllers also convert telemetry display pages to ITOS format. These activities help to prepare the flight controllers for spacecraft integration, which is scheduled to begin in May UCB/SSL provides OSC with two flight controllers throughout the entire spacecraft integration and test phase. A pool of six experienced flight controllers already working on the RHESSI and THEMIS missions rotate to participate in NuSTAR activities, providing the necessary ramp-up in staffing toward supporting the launch and early orbit campaign. In the process assisting OSC personnel, the flight controllers also become very familiar with all spacecraft systems and experience their characteristics first hand. By the time the 8

9 spacecraft is ready for launch, the FOT will be very well prepared to respond to any situations and anomalies that may occur on orbit. C. Roles and Responsibilities during On-orbit Operations 1. Launch, Early Orbit Checkout and Commissioning After NuSTAR launches, the roles of the flight controllers shift, as their primary responsibility becomes the health and safety of the spacecraft. Once on orbit, the NuSTAR observatory is commanded exclusively by trained and certified flight controllers who staff all ground station and SN contacts during the launch and early-orbit phase, assess spacecraft health using real-time telemetry, and also analyze short and long-term trend plots produced by BTAPS. Under the direct supervision of the NuSTAR Mission Operations Manager and relevant subsystem engineers, the flight controllers execute on-orbit check-out procedures for the spacecraft bus and the instrument, and deploy the mast. Flight controllers are eventually authorized to perform routine commanding without direct supervision, such as loading Absolute Time Sequence (ATS) command tables, or sending transmitter on/off commands associated with a pass support. Non-routine commanding is performed by following procedures under the direct supervision of the Mission Operations Manager. 2. Routine Operations Once on-orbit checkout is complete and spacecraft operations normalize, staffing shifts from 24/7 to a single shift, 5-day-per-week scheme. Passes are fully automated, including configuring all necessary ground and space systems (FrameRouter, ITOS, BEARS, BTAPS, and the NuSTAR spacecraft). It is not be necessary for a flight controller to be present on console during most passes, as all real-time and playback telemetry data are automatically checked for limit violations by ITOS and BEARS with redundant paging systems. BEARS employs persistent paging to ensure any anomalies are recognized and promptly attended to. Once flight controllers are freed up from sitting at the console for every pass, they are able to work on other projects such as scheduling, data trending, updating procedures or writing and updating shell scripts to further automate ITOS, plus other operations tasks. Allowing flight controllers to work on diverse projects keeps them interested in their jobs and reduces costs. However, spacecraft safety always remains top priority. Flight controllers are responsible for generating ATS table loads for NuSTAR, using BMPS. These table loads include commands to configure the spacecraft for pass supports, to configure instruments for observations, and to slew the spacecraft to observing targets per instructions by the science team. BMPS ingests corresponding schedule and orbit products generated by SatTrack to configure the spacecraft. BMPS also ingests and merges instrument command sequences provided by the science team. Each ATS table load is checked by software, and is inspected and signed off by two flight controllers prior to upload to the spacecraft. Special attitude maneuvers are verified and validated on FlatSat prior to on-orbit execution. 3. Anomaly Response and Contingency Operations In the event of an anomaly, flight controllers assess the situation and immediately notify the Mission Operations Manager. Anomaly response is then coordinated by the Mission Operations Manager, following either already established anomaly procedures, or by developing new procedures for unanticipated anomalies. While the FOT is adequately trained to handle most anomalies without calling for outside assistance, cognizant systems and subsystems engineers are called upon when necessary. In the event the spacecraft or instrument are in immediate jeopardy and the Mission Operations Manager is not available, flight controllers are authorized to act as first responders to safe the spacecraft. Flight controllers also use BTAPS to quickly generate telemetry trend or event plots for review by systems engineers supporting the anomaly recovery process. VI. Experiences and Lessons Learned A. General Experiences and Lessons Learned The most important lesson we have learned from operating multiple NASA missions is that a well-trained team using cleanly and efficiently integrated flight and ground systems can provide high-quality operations at a reasonable cost. Allowing the flight controllers to work as console operators during mission integration and testing builds up a deep knowledge base along with a strong sense of respect and care for the flight hardware. Overall, this approach results in a flight operations team that truly understands the spacecraft they are going to operate on orbit with a high level of confidence, and prepares the team to handle anomalous situations when they arise. Using experienced flight controllers during integration and testing also imposes a certain level of commanding discipline, even while running tests on the ground. 9

10 B. Ground Systems Utilization during Mission Development and Integration 1. Historical Use of Diverse Ground Systems It has been common practice for instrument teams, spacecraft bus developers and operations teams to develop their own ground systems. This approach leads to significant efforts to convert telemetry and commands databases, display pages and command procedures, first when instruments are integrated with the spacecraft bus, and again after integration and testing, when the spacecraft is handed off to the operations team for on-orbit operations. To make matters worse, these transitions have to be made during times when mission schedules are typically very tight. 2. Support of Spacecraft Bus and Instrument Development For the RHESSI and THEMIS missions, the spacecraft bus providers agreed to use the ITOS command and control system already in use at the Berkeley MOC. However, the instruments were developed with unique ground support systems. A considerable effort was later required to convert these ground systems over to ITOS, while under tremendous schedule pressure, to avoid delays with testing the instrument after integration with the spacecraft, or with environmental testing. With the NuSTAR mission, all parties agreed to use ITOS for all of their spacecraft bus and instrument development work from the beginning. Moreover, the ITOS systems were provided and pre-configured at SSL. This approach placed additional requirements on the flight operations team very early in the mission cycle, but the results are well worth the effort. By using ITOS from the earliest development stages, both instrument and spacecraft teams are already able to benefit from the integrated software tools at the MOC, such as BTAPS for plotting and stripcharting, or FrameRouters for distributing telemetry data from ongoing tests in real-time to multiple locations, including the Berkeley MOC. 3. Lessons Learned from Previous Mission Integration On some past missions, even when ITOS was used during the mission integration and test phase, ground systems equipment, software, and configuration were often provided and administered by the spacecraft manufacturer. This led to some problems when software was not configured identically compared with the setup at the MOC. Often there were differences in the directory structure that would cause ITOS setup or automation scripts developed at SSL to need modification before these would work on the systems used at the integration and test sites. This resulted in more complex ITOS scripts or multiple versions of the same script, complicating configuration control. For NuSTAR, significant portions of the ground system were provided by SSL, pre-configured with the necessary software, and configured identical to the MOC environment. Any questions by OSC engineers regarding ground systems functions could be answered by virtually anyone on the FOT, since all were familiar with the systems hardware, software, directory structure, and configuration. This streamlined approach moves integration and testing very close to the way we intend to fly the mission. 4. Configuration Management The configuration management software and workspace structure at the MOC have evolved from mission to mission, but the system used for NuSTAR is directly modeled after the THEMIS implementation. The major difference is that all SVN repositories (master copies) reside at SSL throughout all mission phases. Files in the workspace can be checked out, modified, and checked back into the repository by authorized users at SSL, Caltech, OSC, or other remote sites. When the spacecraft moves from an integration site to an environmental test site, the associated NuSTAR cube travels with it. Once the cube arrives at the new site, a simple SVN update ensures that the latest versions of ITOS databases, telemetry pages and scripts are installed. Even though the central repository is located at UCB, configuration management authority resides with OSC from the time spacecraft bus flight hardware is connected to the ground system, until transition to on-orbit operations occurs. At this time, UCB takes over authority for configuration management. 5. Benefits from Reusing Previously Integrated Tools The integrated ground systems that were developed to streamline and automate operations for FAST, RHESSI and THEMIS can be directly adapted to support new missions, such as NuSTAR. There are also advantages for cross-training, as the operations team is already familiar with the entire ground system. As an example, flight controllers did not require any special training to begin writing and modifying NuSTAR display pages and command scripts, since they were already familiar with the display formats and the STOL language. Existing workstations in the MOC can also be used to test the NuSTAR operations environment, and serve as backup in the event a NuSTAR workstation fails. In fact, any of the workstations in the MOC can be quickly re-configured to support any of the active missions. The only changes necessary are to login as the appropriate user and to change certain configuration files to allow a socket connection for sending spacecraft commands to be established from a different workstation. 10

11 VII. Conclusion The Berkeley operations team extended the architecture of the highly integrated Multi-mission Operations Center to also support NuSTAR spacecraft bus and instrument development efforts, beginning almost three years prior to launch. With this approach, an appreciable up-front investment is made by the operations team to provide essentially a complete command and control system in almost turn-key fashion. Early involvement of the operations team and the utilization of already integrated, operational software tools avoid potential problems with incompatibilities that may otherwise surface later in the flow. This approach already paid off, supporting both spacecraft bus and instrument development in the early stages at different locations, and is expected to reduce overall risk of schedule and cost impacts, and essentially guarantees a smooth transition from mission integration and testing to launch and on-orbit operations. Close collaboration between members of the operations team, the spacecraft contractor and the instrument developers allows the operations team to gain a detailed understanding and in-depth training that is invaluable in preparation for on-orbit operations. Acknowledgments The authors wish to thank Dr. Fiona Harrison, NuSTAR Principal Investigator, for the opportunity to participate in this exciting new mission. We also wish to thank all of our team members at UCB, Caltech, JPL, and Orbital Sciences Corporation for their excellent collaboration with planning and implementing the ground systems, and for supporting early data flows during instrument development, integration and testing. NuSTAR work at UCB is conducted under California Institute of Technology subcontract CIT-44a to NASA contract NNG08FD60C. References 1 Harrison, F. A., Christensen, F. E., Craig, W., Hailey, C., Baumgartner, W., Chen, C. M. H., Chonko, J., Cook, W. R., Koglin, J., Madsen, K.-K., Pivavoroff, M., Boggs, S., and Smith, D., Development of the HEFT and NuSTAR Focusing Telescopes, Experimental Astronomy, Vol. 20, Springer, Dordrecht, 2005, pp Harrison, F., et al., The Nuclear Spectroscopic Telescope Array (NuSTAR), Bulletin of the American Astronomical Society, Vol. 213, 2009, p Bester, M., Lewis, M., Roberts, B., Thorsness, J., McDonald, J., Pease, D., Frey, S., and Cosgrove, D., Multi-mission Flight Operations at UC Berkeley Experiences and Lessons Learned, AIAA 2010 SpaceOps Conference Papers on Disk [CD-ROM], Huntsville, AL, April 25-30, 2010, Paper AIAA Liebe, C. C., Burnham, J., Cook, R., Craig, B., Decker, T., Harp, D. I., Kecman, B., Meras, P., Raffanti, M., Scholz, C., Smith, C., Waldman, J., and Wu, J., Metrology System for Measuring Mast Motions on the NuSTAR Mission, 2010 IEEE Aerospace Conference Papers on Disk [CD-ROM], ISSN X, Ed Bryan (ed.), Big Sky, MT, 2010, Paper Salotti, L., An Italian Ground Station for the NASA-led Swift High Energy Astrophysics Mission, AIAA 2006 SpaceOps Conference Papers on Disk [CD-ROM], Rome, Italy, 2006, Paper AIAA Bester, M., Lewis, M., Roberts, B., McDonald, J., Pease, D., Thorsness, J., Frey, S., Cosgrove, D., and Rummel, D., THEMIS Operations, Space Science Reviews, Vol. 141, Springer, Dordrecht, 2008, pp Pfarr, B., Donohue, J., Lui, B., Greer, G., and Green, T., Proven and Robust Ground Support Systems GSFC Success and Lessons Learned, 2008 IEEE Aerospace Conference Papers on Disk [CD-ROM], ISSN X, Ed Bryan (ed.), Big Sky, MT, 2008, Paper ITOS, Integrated Test and Operations System, Software Package, Ver. 703pl-6-4, the Hammers Company, Greenbelt, MD, Recommended Standards (Blue Books), Consultative Committee for Space Data Systems (CCSDS), Reston, VA, URL: [cited 24 April 2010]. 10 SunOS, Operating System, Ver. 9 and 10, Sun Microsystems, Oracle Corporation, Redwood Shores, CA, SatTrack, Satellite Tracking and Orbit Analysis Software Suite, Software Package, Ver , BTS, Richmond, CA, Bester, M., Automated Multi-Mission Scheduling and Control Center Operations at UC Berkeley, 2009 IEEE Aerospace Conference Papers on Disk [CD-ROM], ISSN X, Ed Bryan (ed.), Big Sky, MT, 2009, Paper Bester, M., and Stroozas, B., Telemetry and Command Frame Routing in a Multi-mission Environment, 43 rd International Telemetering Conference (ITC) Papers on Disk [CD-ROM], ISSN , Las Vegas, NV, 2007, Paper Roberts, B., Johnson, S., and Bester, M., The Berkeley Trending Analysis and Plotting System Revised and Improved, AIAA 2010 SpaceOps Conference Papers on Disk [CD-ROM], Huntsville, AL, April 25-30, 2010, Paper AIAA Roberts, B. A., BEARS A Multi-mission Anomaly Response System, Proceedings of the Space Exploration Technologies II Conference, W. Fink (ed.), Orlando, FL, April 13-17, 2009, Proc. SPIE, Vol. 7331, DOI: / McGrath, M. T., The Aeronomy of Ice in the Mesosphere Mission, 2009 IEEE Aerospace Conference Papers on Disk [CD-ROM], ISSN X, Ed Bryan (ed.), Big Sky, MT, 2009, Paper

12 17 Apache Subversion Version Control System, The Apache Software Foundation, URL: [cited 24 April 2010]. 18 Roberts, B., Lewis, M., Thorsness, J., Picard, G., Lemieux, G., Marchese, J., Cosgrove, D., Greer, G., and Bester, M., THEMIS Mission Networks Expansion Adding the Deep Space Network for the ARTEMIS Lunar Mission Phase, AIAA 2010 SpaceOps Conference Papers on Disk [CD-ROM], Huntsville, AL, April 25-30, 2010, Paper AIAA