Middeware Architecture for Federated Contro Systems Girish Baiga and P.R. Kumar University of Iinois at Urbana-Champaign A federated contro system (FCS) is composed of autonomous entities, such as cars, that cooperate to provide a goba behavior, such as traffic contro. A middeware is a convenient mechanism to separate software infrastructure, as a ayer, from other FCS components. Eement abstraction is a mode for designing, impementing, and operating software components on this infrastructure ayer. We focus on systems composed of entities operating with autonomous contro, such as automated production pants and traffic contro systems. Traditionay, the software controing such systems has been taiored to each system s requirements. However, creating custom, one-of-a-kind designs is costy and not conducive to proiferation. In recent years, technoogy has evoved to the point at which powerfu processors embedded in the environment can support generaized approaches. Such approaches use forma theory and we-defined design methodoogies, and can be made avaiabe as commercia technoogies. However, generaizations must evove from experience and be vaidated in practice. This artice introduces the notion of federated contro systems through a study invoving a compex traffic contro system under deveopment. We use the notion of eements to represent a service abstraction for software components in the FCS and present a concomitant middeware architecture to support this service abstraction. Prototype system Figure 1 iustrates the prototype system, which consists of a set of cars moving on a track. The cars are controed remotey using an RF transmitter, which is connected to the seria port of a dedicated aptop. A program contros the car s speed and direction by executing a contro agorithm on the dedicated aptop. Contros are sent to the seria port as a stream of aphabetic characters. The port is connected to a microcontroer that converts the character stream into a sequence of signas to operate the car. The contro program that runs the car performs cosed-oop contro and obtains the necessary feedback from an array of cameras (currenty two). At IEEE Distributed Systems Onine Pubished by the IEEE Computer Society 1541-4922/03/$17.00 @ 2003 IEEE
intersections, traffic ights iustrate traffic arbitration. Figure 1. Prototype federated contro system. A set of aptops operates the system, with each aptop controing one car. A set of desktops process vision feed from the cameras and send the feedback to the aptops. The desktops run on Windows, and the aptops on Linux. The computers are connected by Ethernet and an 802.11 wireess LAN. The prototype represents a cass of systems, each composed of sensors, actuators, computers, and communication networks, in which various contro agorithms operate on subsystems with strict rea-time constraints. We categorize such systems as the cass of federated contro systems fitting the foowing definition: A federated contro system (FCS) is a networked system of sensors, actuators, and computers that are composed into hierarchica sets of cooperating entities operating with autonomous contro. Sensors and actuators represent the interface between the system and its environment. Computers represent programmabe system components, which are networked by a topoogy of connections imposed by system design, technoogy, and convention. Compositions of components represent independent system entities, which operate with autonomous contro. In the prototype system, a car exempifies this type of composition. It consists of a controing aptop, an RF connection, and actuators on the car itsef. Usefu behavior is obtained when these autonomous entities cooperate to 2
achieve desirabe goas. We can compose a set of autonomous entities, such as cars, into a higher-eve autonomous entity, such as a feet of cars, and so on. This hierarchy of sets of autonomous entities represents an FCS. Simpe cient-server architecture Our study of the prototype FCS has paraeed our impementation of a simpe working system. The system design emerged as a bottom-up composition of code fragments that were written for simpe use cases 2 of various subsystems. Figure 2 shows the overa software architecture of the first working impementation. The vision sensor is a camera providing a vision feed to a dedicated desktop that executes an image-processing appication and determines the position and orientation of cars on the track using a oca coordinate system. A centra data server connects to the vision sensors (the programs on the desktops), and obtains the vision information for a cars in the system, which it then transforms to a goba coordinate system. A centra trajectory server computes and serves trajectories for a cars in the system. A trajectory is a sequence of car position coordinates in time. The trajectory server obtains ocation information about the cars from the data server. Figure 2. A simpe cient server architecture for the prototype system. Each aptop executes a contro agorithm that moves the car according to a trajectory provided by a trajectory server. The contro program obtains feedback from the data server. This impementation is working quite we. We have tested scenarios with up to eight cars operating simutaneousy on the track and cosey foowing prespecified trajectories. However, we encountered some fundamenta probems during system impementation. Location independence. Operating the system required considerabe preconfiguration. For exampe, a the cients needed to know the addresses and ports of the respective servers. Moving the servers to different machines entaied 3
much reconfiguration. _Bootstrap. Starting up the system was an eaborate process. The servers and cients needed to be started in the right order and on prespecified computers. Synchronization (impemented at the trajectory server) was needed to ensure that a cars were ready before any car started to move. Software update. We had to propagate a change in the car contro program to a the aptops, requiring a system-wide reboot. Software mode and architecture. Mutipe team members working on the different probems of image processing, car contro, car scheduing 3, and infrastructure deveoped the code concurrenty. Separating the infrastructure code as an independent body in this context was particuary chaenging. Maintaining and updating this code required changing and testing the other code using it. The next iteration of the system impementation addresses these and other probems. The foowing sections outine some of our ongoing research. Reated research and appicabiity Some of the probems mentioned earier, such as ocation independence and preconfiguration, have eegant soutions in commercia middeware technoogies such as CORBA 4, DCOM 5, and Jini 6. However, using these technoogies as the base infrastructure has some fundamenta probems. One key probem is that they are intended for transaction-based appications, where the focus is on providing a rich interface. FCSs require an event-based infrastructure, in which data or contro update can proceed asynchronousy. The Open Contro Patform (OCP) 7 extends CORBA to support event streams and other primitives. However, some unsupported features remain. For exampe, OCP doesn t address the software update probem, which must be engineered into the system. In addition, OCP is essentiay a transaction-based system with reevant functionaity imposed as a new ayer. Message-oriented middeware such as IBM s WebSphere Message Queuer 8, Sun s Java Message Service 9, and Microsoft s Message Queue Server 10 represent commercia event-based middeware technoogies. These middeware patforms focus on business appications, however, which vaue secure data deivery over timeiness and other FCS requirements. Other interesting approaches to providing software infrastructure for contro systems incude Cactus 11 and Giotto 12. Contro frameworks such as AOCS 13, Reatime Framework 14, and Nexus 15 have interesting approaches to specific casses of systems. Bonnie S. Heck, Linda M. Wis, and George J. Vachtsevanos 16 give a good overview of research and technoogy that has been deveoped for FCSs and reated domains. 4
The eement pattern As this brief survey shows, middeware is one of the predominant architectura modes that have been appied to FCSs. Middeware nicey separates infrastructure from the rest of the software. However, designing, impementing, and operating software components on the infrastructure ayer require a good service mode. Figure 3a presents one candidate mode, the eement abstraction, that modes software components in an FCS. Figure 3. Eement service mode: (a) eement abstraction in an FCS; (b) eements in an FCS; and (c) the eement pattern An eement encapsuates a contro agorithm operating on some subsystem in the FCS. Eements participate in contro hierarchies. For exampe, in Figure 2, if car contro were a representative eement, the trajectory server woud be a higher-eve eement, and the car actuator woud be a ower-eve eement. Eements aso participate in data fows. In Figure 2, the data server is an input eement to car contro. Not a eements need a the connections shown in Figure 3a, however. Figure 3b, for exampe, shows connections that woud exist for sensors, actuators, and controers in an FCS. The eement abstraction can be represented in many ways. We use a design pattern 17. As Figure 3c shows, the eement pattern is essentiay a composition of 5
design patterns 17 : A strategy represents the contro aw impemented by the eement. A memento corresponds to the concept of state in a contro aw. The proxy operates as the conduit for communicating with other system eements. The composite pattern denotes the idea that mutipe eements can be handed as a coection. Strategy and memento represent the service mode for eements in an FCS. Proxy is an eement s interface to the rest of the system and is anaogous to CORBA s skeetons and stubs 4. The composite pattern represents the infrastructure s view of the set of eements in the system. We mode connections to other eements as streams of asynchronous events. This represents the typica case, in which a eft eement or sequence of contro points from an upper eement periodicay updates an eement. Synchronous communication, such as a handshake, can occur as connections are estabished between eements. A pair of asynchronous event streams hidden by the proxy can represent this communication. Further detais of the eement abstraction and other aspects of eement behavior are avaiabe esewhere. 18 We concude by briefy considering how this architecture addresses some of the issues raised previousy. Because eements have addressabe profies that identify and address them, eements are ocation independent. They must aso have a we-defined state (communicated as mementos 17 ) that can be used to pause and restart them. Downoading code and initia state from one centra ocation permits bootstrapping and update of eements. The middeware architecture aso ceary separates appication code from the infrastructure. Eement-based architecture One way to provide infrastructure for FCS eements is through a simpe architecture that directy supports eements as primitive components. Figure 4 shows an architecture for eement-based systems. Each eement communicates with its she, which represents the eement to the system, and vice versa. The she aso provides a convenient mechanism for maintaining eement-specific data structures and modifications for additiona functionaity (for exampe, adaptation for synchronous communication). A manager maintains and operates on the eements as a coection (the composite pattern 17 ) of simiar entities. Because each eement wi potentiay operate in its own thread, the manager schedues the threads and supports various poicies. Simiary, it schedues the events fowing between eements. The events are maintained in an event board, which the eement shes and the rest of the system access uniformy. 6
Figure 4. An architecture for supporting the eement abstraction. In addition to the manager, the eement-based architecture provides three sets of utiities: Reactive utiities are passive components that are activated by appropriate events requesting their services. For exampe, a refector service consumes the refection 19 events generated by the eements and exposes the system infrastructure as metadata. Proactive utiities are active components that both generate and consume events. One such component is the goba event dispatcher, which forwards events generated by the oca eements to remote eements, and vice versa. Optimization utiities are components that do not participate directy in eventprocessing activities. Instead, they observe other components (through notification mechanisms, for exampe) and perform optimizations as appropriate. Previous work considers the rationae, functions, and responsibiities of the architectura components in more detai. 18 The notions of eements and eement architectures have been further deveoped to EtherArch a software architecture for FCSs. This work is under review for pubication. Acknowedgments This materia is based on work partiay supported by the US Air Force Office of Scientific 7
Research under contract number F49620-02-1-0217, DARPA under contract numbers F33615-01-C-1905 and N00014-01-1-0576, the US Army Research Office under contract numbers DAAD19-00-1-0466 and DAAD19-01010-465, DARPA/AFOSR under contract number F49620-02-1-0325, and the US Nationa Science Foundation under contract number NSF ANI 02-21357. Any opinions, findings, and concusions or recommendations expressed in this pubication are those of the authors and do not necessariy refect the views of the above agencies. References 1. J.S. Bayne, Automation and Contro in Large-Scae Interactive Systems, Proc. 5th IEEE Int Symp. Object-Oriented Rea-Time Distributed Computing (ISORC), IEEE CS Press, 2002, pp. 3 12. 2. R.S. Pressman, Software Engineering: A Practitioner s Approach, 5th ed., McGraw- Hi, 2001. 3. A. Giridhar, Scheduing Traffic on a Road Network, master s thesis, Dept. of Eectrica and Computer Eng, Univ. of Iinois at Urbana-Champaign, 2002. 4. Common Object Request Broker Architecture: Core Specification, Object Management Group, Nov. 2002; http://www.omg.org/cgi-bin/doc?forma/02-11-01. 5. DCOM Technica Overview, Microsoft Corp., Nov. 1996; http://msdn.microsoft.com/ibrary/defaut.asp?ur=/ibrary/enus/dndcom/htm/msdn_dcomtec.asp. 6. K.W. Edwards, Core JINI, Prentice Ha, 1999. 7. L. Wis et a., An Open Contro Patform for Reconfigurabe, Distributed, Hierarchica Contro Systems, Proc. Digita Avionics Systems Conference, vo. 1, IEEE Standards Office, 2000, pp. 4D2/1 4D2/8. 8. A. Banks, WebSphereMQ: Providing More Processing Power through Custered Queue Managers, white paper, IBM, Juy 2002, http://www- 3.ibm.com/software/integration/mqfamiy/ibrary/whitepapers/g325-2032-00.pdf. 9. K. Haase, Java Message Service API Tutoria, Sun Microsystems, 2002; http://java.sun.com/products/jms/tutoria/. 10. P. Houston, Buiding Distributed Appications with Message Queuing Middeware, Microsoft Corp., 1998, http://www.microsoft.com/ntserver/techresources/appserv/msmq/msmqdistributed.asp. 8
11. M. Hitunen and R. Schichting, The Cactus Approach to Buiding Configurabe Middeware Services, Proc. Workshop Dependabe System Middeware and Group Comm. (DSMGC), IEEE CS Press, 2000. 12. T. Henzinger, B. Horowitz, and C. Kirsch, Giotto: A Time-Triggered Language for Embedded Programming, Proc. 1st Int Workshop Embedded Software (EMSOFT), Springer-Verag, 2001, pp. 46 60. 13. T. Brown et a., A Reusabe and Patform-Independent Framework for Distributed Contro systems, Proc. IEEE/AIAA 20th Digita Avionics System Conf. (DASC), vo.2, IEEE Standards Office, 2001, pp. 6A1/1 6A1/11. 14. A. Traub and R. Schraft, An Object-Oriented Rea-Time Framework for Distributed Contro Systems, Proc. IEEE Conf. Robotics and Automation, vo. 4, IEEE CS Press, 1999, pp. 3115 3121. 15. J. Fernandez and J. Gonzaez, Nexus: A Fexibe, Efficient and Robust Framework for Integrating Software Components of a Robotic System, Proc. IEEE Conf. Robotics and Automation, vo. 1, IEEE CS Press, 1998, pp. 524 529. 16. B.S. Heck, L.M. Wis, and G.J. Vachtsevanos, Software Technoogy for Impementing Reusabe, Distributed Contro Systems, IEEE Contro Systems Magazine, vo. 23, no. 1, Feb. 2003, pp. 21 35. 17. E. Gamma et a., Design Patterns, Addison-Wesey, 1995. 18. G. Baiga, A Software Architecture for Federated Contro Systems, master s thesis, Dept. of Computer Science, Univ. of Iinois at Urbana-Champaign, 2002. 19. F. Kon et a., The Case for Refective Middeware, Comm. ACM, vo. 45, no. 6, June 2002, pp. 33 38. Girish Baiga is pursuing a PhD in the Department of Computer Science at the University of Iinois at Urbana- Champaign. His research interests incude software architecture and federated contro systems. Contact him at Coordinated Science Laboratory, Univ. of Iinois, Urbana-Champagne, 1308 W. Main St., Urbana, IL 61801; gibaiga@uiuc.edu; http://back.cs.uiuc.edu/%20~baiga/. P.R. Kumar is the Frankin W. Woetge Professor of Eectrica and Computer Engineering at the University of Iinois at Urbana-Champaign. He was the recipient of the Donad P. Eckman Award of the American Automatic Contro Counci, and is a Feow of the IEEE. Contact him at prkumar@uiuc.edu. 9