MONITOR AND CONTROL SOFTWARE FOR GROUND SYSTEMS IN THE DEEP SPACE NETWORK

Transcription

1 MONITOR AND CONTRO OFTWARE FOR GROUND YTEM IN THE DEEP PACE NETWORK Pul Pechkm Network Control Tsk NAA/ Jet Propulsion bortory July 22,22

2 ABTRACT NAA s Deep pce Network (DN) is n interntionl network of ntenn trcking complexes used for two-wy communiction with deep spce nd Erth-orbiting spcecrft. A DN complex will utilize severl ntenns concurrently nd shre ground systems, which include trnsmitters, telemetry processors nd receivers, for exmples, from pool of resources. These complexes re mnned opertions centers, but they re lso networked to centrl opertions center for remote monitoring. This dynmic lloction of resources in ddition to providing worldwide visibility requires distributed nd dynmic monitor nd control system. Building relible nd open softwre infrstructure is the key to sclble nd distributed monitor & control nd ultimtely, utomtion t multiple levels. This hs been ccomplished by Network Monitor nd Control (NMC) softwre developed t Jet Propulsion bortory (JP). Developed initilly to replce legcy monitor nd control hrdwre nd softwre, the NMC ws designed to provide pltform for supporting legcy ground equipment, future equipment, nd utomtion of DN opertions. Typicl of monitor nd control softwre, NMC includes grphicl user interfces (GUI s) for humn opertors to input commnds to ground equipment nd displys for viewing equipment helth or trcking sttus. NMC lso introduces utomtion concepts for opertions, but the bsis of ny utomtion is the retrievl of monitor dt to build the world view. This pper will discuss how NMC chieves opertionl gols nd performnce required of monitor nd control of numerous ground equipment. The focus is on infrstructure nd detiled softwre concepts for providing sclble, relible nd performing softwre nd functionlity. A distributed client-server rchitecture for infrstructure llows mny clients to ccess or produce hundreds of monitor dt items or event messges ech. Optimiztion methods nd design ptterns re used in server nd client rchitecture to obtin work distribution. In ddition, softwre techniques for building servers include effective thred nd network progrmming in ddition to using existing protocols s TCP/IP. Another importnt fctor is client API lyer, which is key to supplying trnsprent connectivity between ground equipment nd monitor & control interfces, which ny form of utomtion relies on. ystem rchitecture for NMC is bsed on distributed servers tht perform different functions such s processing user commnds, trnsltion of legcy dt nd dt mngement. How these functions re distributed is key to providing sclbility nd mintinbility. Currently, the NMC is being utilized for 24/7 opertions t the three min DN complexes supporting ll of the mjor spcecrft supported nd mnged by JP. A portble version of NMC is being exercised in Comptibility Test Triler for off-site monitor nd control. Remote monitoring of the three complexes t JP is ccomplished s prt of the NMC infrstructure. 2

3 INTRODUCTION The Network Monitor nd Control group ws tsked to develop softwrehrdwre rchitecture to provide monitor nd control in the Deep pce Network (DN). One of the mjor tsks ws to build n infrstructure to replce legcy system. An infrstructure tht used Distributed Computing Environment (DCE) s its core ws first implemented. For vrious resons, this component ws determined too risky for the DN s future needs. This pper documents the softwre infrstructure tht ws put in plce s prt of wht is known s Delivery 1.3 of the Network Monitor nd Control (NMC) softwre. Though the NMC softwre covers vst rry of functionlity, the focus will be on softwre infrstructure. The Deep pce Network NAA s Deep pce Network (DN) is n interntionl network of ntenn trcking complexes used for two-wy communiction with deep spce nd Erth-orbiting spcecrft. A DN complex will utilize severl ntenns concurrently to trck different spcecrft or the sme spcecrft (vi rrying two or more ntenns) simultneously. To provide round-the-clock trcking of prticulr spcecrft, three complexes hve been positioned round the world: the Goldstone Deep pce Communiction Complex (GDCC) in Brstow, Cliforni; Mdrid Deep pce Communiction Complex (MDCC) in Mdrid, pin; nd Cnberr Deep pce Communiction Complex (CDCC) in Cnberr Austrli. These three complexes collect dt from the spcecrft, pckge it, nd trnsport the dt bck to control center t Jet Propulsion bortory in Psden Cliforni where it cn be distributed to the spcecrft s project liisons or scientists. Trcking spcecrft is globl ffir, nd thus, network of spcecrft trcking complexes provides the bckbone for these opertions. Ech trcking complex comprises of severl ntenns of different sizes (from 26 meter to 7 meter) nd types (including Bem-wve guide nd Highefficiency Frequency). Typicl services supplied by ground equipment include telemetry, trcking, rnging, nd commnding. Ech equipment type will report configurtion, sttus or mesurement dt vi monitor dt or event messges tht cn be viewed vi displys, or utomtion softwre cn use the dt to determine the next logicl step in opertions. A prticulr piece of equipment cn produce n unlimited number (in the thousnds) of monitor dt thus system is required tht cn hndle the dt flow efficiently. In ddition, new types of equipment re being developed such s rrying softwre or upgrdes to current up-linking or down-linking systems tht require n open system for esy integrtion. HIGH EVE REQUIREMENT The Deep pce Network requires n infrstructure tht: upports opertions of up to eight concurrent ntenns upports legcy equipment (requiring some form of trnsltion) upports publiction of thousnds of monitor dt upports event notifiction upports relible communiction to equipment in the form of commnds or directives upports trnsporttion of monitor dt between sites or to centrl loction over Wide Are Network (WAN) Optimizes for new equipment upgrdes upports utomtion plug-ins. HIGH EVE OPERATIONA CONCEPT FOR MONITOR AND CONTRO Complexes nd their equipment re dynmic environment, which mkes it more difficult to monitor nd control. A key opertionl concept strts with time before the beginning of trcking, in which set of required ground equipment, which could include set of trnsmitters, receivers, or telemetry processors, will be llocted from common pool of resources to connection ssocited with prticulr ntenn nd spcecrft. A humn opertor will configure the equipment nd then monitor the equipment for the durtion of the trck (i.e. while the spcecrft signl is locked nd flowing spcecrft dt). After the trck is complete, the llocted equipment will be relesed into 3

4 the common pool where it will be llocted for the next support on the schedule, which my not be on the sme ntenn. 24/7/365 performnce - NAA s Deep pce Network (DN) is responsible for round-the-clock trcking of mny deep-spce nd orbitl spcecrft. The infrstructure must be relible nd perform in its best cpcity t ll times. Tht is, the system should be ctive with the ltest dt nd be ble to communicte with equipment (vi opertor commnds or directives) t ny time. This requires system tht chieves 24 hours dy, 7 dys week, 365 dys yer level of opertion. Any clenup, routine mintennce, nd initiliztion of new clients must occur without effect on system tht is currently running. Automtion - As more spcecrft re being supported by the DN, cost-effective solutions such s utomtion re being pursued. Thus, system must be built to support utomtion plug-ins. By defult, utomtion requires the sme informtion tht humn opertor would require to perform specific ction. However, good utomtion needs ccess to more informtion so tht it cn build dtbse nd sense of history for trend nlysis nd to predict fults. ome ccess point must be vilble for utomtion to plug in nd issue directives nd receive monitor dt. This lso implies level of security in operting equipment. ERVICE ORIENTED APPROACH FOR THE INFRATRUCTURE In order to fulfill the functionl requirements, the pproch is to define services tht cn be contined nd distributed. A client will be user of one or more of these services. Nme services - clients will need to know how to tlk to servers on network level. Physicl host informtion for client is mpped to user-defined physicl nme. This is the bsis for ll network communictions. Functionl nme services - dt needs to be represented by proxies or functions for orgnizing dt. A client my register severl functionl nmes to distinguish sets of monitor dt. These functionl nmes will then be mpped to physicl nme provided by the Nme ervices. Monitor Dt services - provide mens of publishing nd/or subscribing to dt items represented by the Functionl nme services Messginp services - llows for clients to send dt to other clients. These re regrded s peer to peer trnsctions defined nd rrnged by the respective clients. Event notifiction services - llows for clients to send events tht cn be logged in centrl loction. Events re different from monitor dt in tht combined with previous events they cn provide history. Monitor nd Control services - provide protocols nd trnsctionl processes for client to direct nother client. For exmple, these services would be used to send commnd from disply to prticulr equipment controller. Timeouts nd exceptions re provided s prt of these services. Connection services - llocte nd mnge equipment for trcking session. oftwre is then developed to mke these services vilble to pplictions. 4

5 OFTWARE DECRIPTION The NMC softwre cn be described s set of network pplictions; they fit perfectly within vrious network configurtions so tht the customer cn tke dvntge of the ltest server hrdwre or network hrdwre nd technologies. An Internet-like strtegy is dopted; typicl of mny Internet pplictions, NMC pplictions fit on top of the trnsport lyer in the open system interconnection model (OI). TCP/IF', stndrd communictions protocol nd known for its relibility, is used, replcing legcy proprietry protocols. A client-server softwre model is used. ervices re distributed between servers nd client lyer (see Tble 1). A client will utilize one or more of the services provided by server nd mke trnsctions. Contining some of these services in client-side librry provides client-client communiction. Functionl Nme, Monitor Dt Event Nme ervices Messging, Control ervices Tble 1 - Component mpping to service A client-server nd client-client interction is shown in Figure 1. The Nme ervices, Functionl Nme ervices nd Control ervices re utilized to send commnd ( request for ction) from one client to nother client. A client will ddress the commnd to functionl nme. Through trnsctions, the functionl nme will eventully mp to host informtion for direct communictions between clients. (6) Commnd sent to host ) Register functionl nme (3) Functionl to physicl nd physicl nme nme mpping (cched) (1) Register host info nd physicl nme Figure 1 - Client-erver processing of commnd (Control) Another concept used is publisher-subscriber, in which client will subscribe to dt items tht cn be published by nother client. This ensures tht only interested dt is sent over the network to only the requesting destintions. The client-server interction for subscribing to dt nd receiving dt is shown in Figure 2. This ctivity utilizes the Functionl nme services nd Monitor Dt services. 5

6 (1) Register functionl nme nd physicl nme (3) subscription (2) subscribe to dt belonging to functionl nm (4) dt Figure 2 - Client-erver processing of monitor dt (Monitor) A key softwre component is the Appliction Progrmming Interfce (API) contined in the form of shred librry. This librry contins client-server nd client-client protocols for communiction nd ccess to the vrious infrstructure services. These re event-driven synchronously using threds to provide cllbck technology or triggers. This llows subscribing clients to rect to monitor dt updtes when they rrive. This rection could led to n updte to disply or it cn cuse utomtion softwre to rect with response. An interpreted scripting lnguge is lso vilble for creting simple forms of utomtion. Grphicl displys cn be developed using provided GUI API, nd web services cn lso be provided to produce HTM pges of monitor dt nd events, for exmple. Currently supported pltforms include olris nd olris 8. Infrstructure services re written in C++ llowing for future portbility onto vrious pltforms. Connection oriented services re written in C, C++, nd templte lnguge clled NAP. Displys re written in motif. For incresed portbility, stndrds re used including POIX where pplicble. Thred models nd queue processing provide synchronous processing to efficiently process both server trnsctions nd client receipt of dt. The NMC softwre is optimized for dt trnsmissions nd locl nd wide re networking rchitectures. A typicl NMC configurtion t complex will include ocl Are Network (AN) of t lest one worksttion for humn opertor, which he/she will use s n interfce to operte one or more ntenn shred file-system, nd the servers. This configurtion is networked with the vrious ground equipment controllers tht it hs to communicte with. 6

7 Gr pment NMC Worksttion NMC ervers Figure 3- An NMC t complex NMC Filesystem (NF) The NMC system t ech complex is connected to centrl loction: Goldstone Deep pce Communictions Complex Mdrid Deep pce Communictions Complex I Network Opertions Control Center (JP) Cnberr Deep pce Communictions Complex Figure 4 - NMC's connected for remote monitoring This configurtion llows services to be exchnged between NMC systems such tht n opertor t the Network Opertions Control Center t JP cn view monitor dt from the Mdrid complex. FEATURE Relibility - opertions is mintined if server process were to fil. erver pools cn be defined such tht clients cn connect to n vilble server without interruption. This is ccomplished vi client cching of ll subscriptions, mppings nd ctivities. 7

8 clbility - whether it s to support eight concurrent trcking ctivities t site or just one, NMC cn scle to the pproprite needs with full server complement or NMC in box configurtions, where ll Nh4C softwre components re run on single mchine. oftwre is designed so tht the constrints re reduced to hrdwre trits including CPU nd network performnce (YO crds, configurtion, etc.). More importntly, n upgrde pth is vilble by llowing the option for fster, more powerful mchines. Interoperbility - the client librry (currently supporting olris nd 8 but esily portble ) llows pplictions running on different operting systems or hrdwre pltforms to communicte with the NMC llowing new equipment to esily integrte itself. This lso llows utomtion softwre mens of interfcing nd providing monitor nd control. Trnsltors cn be built using the API to provide communictions between new equipment to tlk to legcy equipment. Performnce - to mintin performnce, server pplictions re threded per client nd dtbse locking is mnged so tht only the required dt sets within the servers re locked for ccess. This provides multi-client ccess to services. Queue processing lgorithms tht filter on stle monitor dt cn effectively keep processes from bcklogging nd ensuring timely delivery of the ltest dt. Mngebility - tools re provided for viewing the stbility nd configurtion of the vrious servers. An API is lso provided for further ccessibility nd control. EON EARNED ome lessons were lerned during the development nd testing of the infrstructure softwre. Develop with performnce mesuring nd nlysis in mind - in designing infrstructure, pln for performnce mesuring nd nlysis, which should be used in evluting performnce to lod conditions nd the robustness of the infrstructure. Develop softwre with hooks into viewing vilble resources. Mny vilble CPU ndor network mesuring tools cn be used to indicte inefficient resource usge or bottlenecks in dt flow. Protect servers from misbehving clients - it is difficult to determine how client will behve. An unchecked client cn conceivbly produce high ctivity nd hord resources from other clients. Incorporte defense mechnisms like timeouts nd disconnect lgorithms to protect servers - mintining ccessibility to ll its services. The network is limited resource - stle dt should be filtered before sending over network. If using TCP/IP, bewre sending smll pckets tht collect on overhed. In ddition, mny smll pckets hve wys of provoking other ill behviors like interpcket gp problems between networking hosts or cusing pcket collisions on hlf-duplex networks. Colesce dt into lrger chunks nd enble the Ngle lgorithm. Bewre of thred brodcsting - threds re ment to efficiently use CPU to hndle mny different tsks simultneously, but void unnecessry brodcsting clls tht cn ccumulte CPU cycles needlessly. Wide Are Networks (WAN) behve differently from ocl Are Networks (AN) - Wide Are Network (WAN) behvior is never s relible s ocl Are Networks (AN). Tht is, there re multiple points of filure in WAN tht cn cuse disruption in service. Thus, WAN communiction The client librry is only dependent on stndrd threds nd sockets, which re vilble on mny pltforms 8

9 needs to be developed differently thn AN communiction. Hert-bets nd timeouts with reconnect lgorithms should be employed to hndle lengthy outges or routing errors. Be wry of 3rd prty librry components - librries of constructs tht re intended to mke development life esier cn ultimtely be mintennce nd portbility problem if they become unsupported. Choose products wisely nd tht dhere to stndrds so tht they cn be replced esily if necessry. TATU Currently, the NMC is being utilized for 24/7 opertions t the three min DN complexes supporting most if not ll the mjor spcecrft mnged by JP. The Goldstone Deep pce Communictions Complex, the most populted with ntenns of the three complexes, will use NMC to monitor nd control six of its ntenns simultneously on dily bsis. An NMC system hs recently been instlled t the Network Opertions Control Center (NOCC) t JP in Psden Cliforni to provide remote monitoring of the three complexes. A scled-down version of NMC is being exercised in Comptibility Test Triler for monitor nd control on the move. The infrstructure is supporting current utomtion cpbilities t ll complexes, in ddition to supporting new utomtion technologies currently in development. Current NMC plns include providing support for new opertion concepts s trcking multiple spcecrft per ntenn. ACKNOWEDGMENT The uthor wishes to cknowledge nd thnk the contributors nd implementers of the softwre: John Angione, Erik Brkley, Robert Blkenhol, Ev Bokor, Geoffrey Cowrd, hirley A. Cizmr, Emily w, Dniel Mgllon, Ronld E. Normn, Mike Pester, teve tedry, Doug tones, D Arcy Tyrrell, Jonne Wisniewski, opertions nd engineering t Cnberr DCC, Goldstone DCC, Mdrid DCC, nd NOCC. This pper represents their incredible dediction nd expertise. REFERENCE Bokor, Ev Bryn Cmilli, Mnuel Gomez, Ptrick Olguin, Pul Pechkm, Ptrici ntos, Mrl Thornton, oftwre for Monitoring nd Controlling DN Opertions, NAA Tech Brief, Vol. 25, No. 6, 21. Buschmnn, Frnk, Regine Meunier, Hns Rohnert, Peter ommerld, Michel tl, Pttern- Oriented oftwre Architecture, Volume 1: A ystem of Ptterns, John Wiley & ons, Grhm, In, Object-Oriented Methods: Principles nd Prctice, Addison Wesley Pub. Co., December 2. Rosenberry, Wrd, Dvid Kenney, nd Gerry Fisher, Understnding DCE, -Reilly & Assocites, Inc. My Reference herein to ny specific commercil product, process or service by trde nme, trdemrk mnufcturer or otherwise, does not constitute or imply its endorsement by the United ttes Government or the Jet Propulsion bortory, Cliforni Institute of Technology. The work described here ws crried out t the Jet Propulsion bortory, Cliforni Institute of Technology under contrct with the Ntionl Aeronutics nd pce Administrtion. 9

10 pceops 22 - Ground egment Engineering nd Architecture Monitor nd Control oftwre for Ground ystems in the Deep pce Network Pul Pechkm July 25, 22

11 pceops 22 - Ground egment Engineering nd Architecture A w Objectives Describe the requirements for monitor nd control in NAA s Deep pce Network Introduce Network Monitor nd Control (NMC) softwre nd its components Focus on infrstructure design concepts used infrstructure components some lessons lerned Wht is infrstructure? Underlying softwre/hrdwre foundtion of monitor nd control system

12 pceops 22 - Ground egment Engineering nd Architecture Deep pce Network The Deep pce Network (DN) interntionl network of spcecrft trcking complexes H Ech complex is self contined site of opertions pool of ntenns; simultneous trcking (limited to number of ntenns) of multiple spcecrft shred ground equipment including trnsmitters, receivers, telemetry processors, etc. complex-to-complex hnd-over for continuous trcking of single spcecrft H Remote monitoring monitoring of complex ctivity vilble t the Network Opertions Control Center t Jet Propulsion bortory

13 pceops 22 - Ground egment Engineering nd Architecture Deep pce Network (cont.) A 1 Opertions - typicl duties (some re utomted) include: llocting equipment from pool of resources for trck configuring equipment monitoring equipment during trck vi grphicl displys or ccess to rw monitor dt 17 post-trck clenup (stowing the ntenn pckging spcecrft dt) relesing equipment for next support Opertions require n interfce to equipment with the bility to control, Opertions require ccess to monitor dt nd event messges produced by equipment for monitoring sttus nd helth

14 pceops 22 - Ground egment Engineering nd Architecture I nf rst ructu re Requirements I Why infrstructure is importnt Incresing number of spcecrft supports nd limited resources require relible opertions 24/7/365 New spcecrft nd/or new technologies promote new types of ground equipment tht need to be integrted Additionl lod equls more dt processed nd flowing Accessibility nd visibility to ll ground equipment is required for monitor nd control in ny form of utomtion. Dt exmples include: equipment helth signl helth s reported by telemetry or signl receivers Different trcking configurtions such s ntenn rrying nd/or multiple spcecrft per ntenn require incresed communiction between equipment nd feedbck bck to opertions nd utomtion

15 pceops 22 - Ground egment Engineering nd Architecture Building Infrstructure Network Monitor nd Control (NMC) softwre uses network rchitecture of servers nd user worksttions Good infrstructure will provide mngebility relibility performnce sclbility interoperbility security (if needed) off-the-shelf network hrdwre including switches nd routers to Hrdwre interconnect un servers nd worksttions nd estblish communiction with ground equipment

16 pceops 22 - Ground egment Engineering nd Architecture A ervice oriented Nme services - provide mens for clients to rech other clients Functionl nme services - llow for dt representtion vi formtted nmes Monitor dt services - provide for monitor dt ccessibility Messging services - provide mens for clients to sendheceive messges to/from other clients Event Notifiction services - provide event notifiction nd the bility to trigger responses ccording to specific events Control services - provide mens for sending nd receiving commnds with proper formt Connection services - llocting nd mnging resources for n ntenn

17 = n I c (> W. (> I cn I. >, cn.c-r cn c, U Q. C I Q, c, c U cn in c,.- cn cn - m c, 2 Y CI Q m - Y ul v) v) CI CI n Q k.- i, cn e cn 13) cv),.- tn m Q 3 cn *, Y cn O 2 cn 3 Q W G o cn.u -. > m.c-r n 8 Y E c, U 12._. v) 12 c 3 c,.- U c c, t 12.- c, m t5 v) 4 U 3 v) = $ % Y 13) T c, 12 U c, c Y ~~ c, v).-cn Y U - b =

18 pceops 22 - Ground egment Engineering nd Architecture A w w Bse softwre components Nmeewer Monitor Dt erver Client librry nd Appliction Progln ming Interf1 e (API) contin client-server nd client-client protocols hred file system - bsed on NF Additionl components for mnging ground equipment Connection engine/server(s) - server process representing n ntenn nd its configured/ssigned equipment. Trnsltor - server process used to trnslte legcy protocols User Interfce - suite of grphicl displys for rel-time monitoring nd control of ssigned equipment

19 pceops 22 - Ground egment Engineering nd Architecture NMC in Deep pce Communictions Complex t NMC Workst ion Ground Equipment e I n n n o I I I 7 1 / NMC ervers (Nme ewer, Monitor Dt ewer, Connection Engines, Trnsltor) NMC Filesystem (NF)

20 pceops 22 - Ground egment Engineering nd Architecture Distributed NMC s in the Deep pce Network Goldstone Deep pce Communictions Complex Mdrid Deep pce Communictions Complex I I I i Network Opertions Control Center (JP) Cnberr Deep pce Communictions Complex

21 cn.- c,.- o m Q'cr > c) U U c c, C E I. v).- v) c, > C cn -- v) 'cr m - U CI - Q c, m.- CI v) U v) c, ki c, d N + U v) > Y.- E.- CI Q cn Y 73 > X - v> > > cf) z cn 3 m

22 pceops 22 - Ground egment Engineering nd Architecture Performnce (cont.) Obtining sclbility server processes re network processes nd cn run on one host or distributed on mny hosts Obtining relibility fil-over cpbility client-side cching server pools

23 pceops 22 - Ground egment Engineering nd Architecture A essons erned H Pln for performnce mesuring nd nlysis Protect servers from non-nominl client ctivity w The network is limited resource H Avoid inefficient thred models A Wide Are Network (WAN) is not s relible s ocl Are Network (AN) H Avoid server dependencies on shred resources e.g. shred filesystem

24 pceops 22 - Ground egment Engineering nd Architecture Conclusion oftwre nd hrdwre infrstructure is n importnt prt of ground communictions (in the DN) 17 Allow opertions to worry bout the spcecrft only Redy for new equipment nd/or opertionl scenrios Automtion relies on dt vilbility nd feedbck from equipment 17 Automtion cn better predict the environment with incresed dt ccess nd evlute decisive ction Numerous pplictions beyond spcecrft trcking NMC is opertionl nd supporting live spcecrft trcking