Amoeba made compatible with Unix: the ADE approach. School of Computing & Information Technology, Grith University

Transcription

1 Amoeba made compatible with Unix: the A approach CZ Sun P Keuning + G Dekker + J Schipper + M De Lange + + ACE Associated Computer Experts BV Van Eeghenstraat 100, 1071 GL Amsterdam, The Netherlands amoeba@acenl School of Computing & Information Technology, Grith University Nathan, Brisbane, Queensland 4111, Australia CSun@citgueduau Abstract This paper 1 reports a recent development on the Amoeba distributed operating system the A emulation system, which aims at making Amoeba fully compatible with the Unix (POSIX) operating system so that existing Unix applications can directly run on the Amoeba system and immediately benet from the use of the Amoeba multiple processors 1 Introduction Amoeba is a generalpurpose distributed operating system [1] It is intended for parallel and distributed applications, as well as traditional applications It takes a collection of machines and make them act together as a single integrated system In Amoeba, multiple processors can be dynamically allocated to a single job or program to gain speed In general, users are not aware of the number and location of the processors that run their commands, nor of the number and location of the le servers that store their les To the casual user, an Amoeba system looks like a single oldfashioned timesharing system Amoeba was originally designed and implemented by a group under the direction of Prof Andrew S Tanenbaum at the Vrije Universiteit (VU) in Amsterdam, The Nether 1 This paper was published in the 17th Australian Computer Scinece Conference held in Christchurch, New Zealand, Jan 19{ It is also available in the proceedings of ACSC17 on pp 249{258 lands, partly in cooperation with the Centrum voor Wiskunde en Informatica (CWI), also located in Amsterdam When Amoeba reached the point where it was beyond the laboratory stage and was ready for commercial use, VU and CWI decided to join forces with ACE, a leading Dutch R & D company specializing in advanced software, including operating systems, compilers, CASE tools, and networking While the research labs will continue to push the OS technology forward, ACE will provide commercial support for and conduct further R & D work on Amoeba This paper reports a recent development on Amoeba conducted at ACE This work aims at making Amoeba fully compatible with the Unix operating system so that existing Unix applications can directly run on the Amoeba system and immediately benet from the use of the Amoeba multiple processors The rest of this paper is organized as follows: First, the incompatible aspects between Amoeba and Unix are discussed in Section 2 Then, the basic idea of the A approach compatibility by integration is described in Section 3 Next, the design and implementation strategies of the A system are discussed in Section 4 Finally, some concluding remarks are given in Section 5 2 Incompatible aspects between Amoeba and Unix In contrast to other parallel/distributed operating systems [3, 4, 5], the Amoeba distributed 1

2 operating system was originally designed without regard for backward compatibility with systems like Unix From an operating system research point of view, having the freedom to selectively use ideas from existing systems is an advantage, which has resulted in one of the fastest distributed operating systems in the world [2] However, when Amoeba has gone beyond the laboratory stage and used in research institutes and companies all over the world, its compatibility with Unix has become very important to its success in supporting migration of existing software systems from sequential platforms to parallel and distributed platforms In many aspects (eg process management, signal handling, etc), Amoeba is incompatible with Unix Some Unix features are fundamentally dierent with that in Amoeba and are virtually impossible to be emulated on top of Amoeba, including: Protection mechanism: Since capabilities are used for protection of objects in Amoeba, it does not have the Unix equivalent of userids or groupids The whole concept of userids and groupids is very hard to get right in a capabilitybased system Directory service semantics: There are no symbolic links in the Amoeba Soap directory server [1] The only kind of links are hard links but they have a different semantics from those of Unix hard links There are no link counts anywhere in Amoeba so that deleting an object will break all links to it! Emulating Unix directory service (which is tightly coupled with the le service) on top of the Soap Server is virtually impossible File service semantics: The Amoeba Bullet le server implements immutable les [1] That is, once a Bullet le has been created it cannot be changed Combined with the incompatible semantics of the Soap directory server, the Bullet le service semantics makes the emulation of Unix le service on top of it virtually impossible Although the Ajax approach has been taken to emulate partial Unix functionality on top of the Amoeba kernel and standard servers (eg the Soap directory server and the Bullet le server) [1], 100 % compatibility has never been a goal and may never be achieved by the Ajax approach To make Amoeba fully compatible with Unix, an alternative approach is needed to deal with the above basic incompatible aspects 3 Compatibility by integration The basic idea of our approach is to achieve compatibility by integrating real Unix systems as specialized Unix servers into the Amoeba environment As shown in Fig 1, the Amoeba architecture consists of four principle components: First, each user has a workstation for running the user interface and X window System; Second, there exists a pool of processors (computing servers) that are dynamically allocated to users as required; Third, there are specialized servers, such as le servers, database servers, and Unix servers which provides transparent Unix system service to applications running in the Amoeba environment; Finally, there are gateway machines that allow multiple Amoeba systems that are far apart to be connected together transparently In the proposed approach, the real Unix system is used to provide the Unix le system service and protection based on Unix userids and groupids, avoiding the basic incompatible problems between Amoeba and Unix, 100% compatibility is achievable Furthermore, most of the Unix system functionalities (eg le system, process management and signal handling) can be directly obtained from the real Unix system The main task of the emulation software is to integrate the Unix functionalities into the Amoeba system by implementing proper RPCs (Remote Procedure Calls [7]) on the Unix system, which is signicantly simpler than implementing an emulator on top of the Amoeba kernel Since many Unix system calls, particularly the le system calls, will be served on the re 2

3 Processor Pool (Computing Servers) Workstations Gateway WAN File DB Unix Server Server Server Specialized Servers Figure 1: An Amoeba conguration with real Unix systems as Unix servers mote Unix system, the performance of this emulation will be slower than that of emulation on top of the Amoeba kernel and Amoeba standard servers However, the proposed approach is mainly intended as a bridge for a smooth migration of existing Unix applications from traditional sequential environment to the Amoeba parallel and distributed environment Unix applications with concurrent processes can not only migrate to the Amoeba system without modication, but also immediately benet from having these concurrent processes executed on multiple processors in Amoeba Furthermore, migrated applications can be gradually adapted to the new environment to take advantage of Amoeba's facilities for parallel and distributed computing (eg the highly ecient interprocess communication facilities) 4 A prototype implementation of the A system To realize the proposed approach, a prototype system, called the A (Amoeba Dedicated Engine) system, has been developed [8, 9] The design and implementation strategies of the A system are discussed in this section 41 System structure The A system consists of two multilayered subsystems distributed on two operating system environments: the Amoeba system and the Unix system, as shown in Fig 2 The AmoebaRPC layer and the FLIP (Fast Local Internet Protocol) layer connect the Amoeba system and the Unix system together and provide a fast remote procedure call interface for communication between the A components of the two sides On the Amoeba system side, the AXSI layer and ARPC layer are built on top of the AmoebaRPC layer, and the application program () is running on top of the AXSI layer On the Unix server system side, the (Unix Server) component is built on top of the AmoebaRPC layer Dynamically, the A system consists of multiple processes running on both the Amoeba system and the Unix system, as shown in Fig 3 There are two types of processes in the A system: one is the process, which executes the application program () linked with the library ( = AXSI + ARPC) program and runs on the Amoeba system; the other is the process, which executes the Unix Server program and runs on the Unix system There is an onetoone correspondence between processes and processes 3

4 Amoeba System AXSI ARPC AmoebaRPC AmoebaRPC FLIP FLIP : AXSI: ARPC: : AmoebaRPC: FLIP: Application Program A X/Open System Interface library A Remote Procedure Call stub library Unix Server Ethernet Amoeba Remote Procedure Call interface Fast Local Internet Protocol Figure 2: Statical structure of the A system 42 Functional specication of the A components The functionalities of the A components are briey specied below For descriptions of the AmoebaRPC layer and the FLIP layer, the reader is referred to [11, 6] The AXSI layer: The AXSI layer supports the functions and headers of XSI the X/Open System Interface dened in the X/Open Portability Guide of 1989, informally known as XPG3 [13] Since XPG3 is a superset of POSIX [12], most of the XSI interface routines are implemented as C library routines, without communicating with the process For these XSI interface routines whose implementations rely on the process, the main functionality of this layer is to prepare and format the parameters properly for the ARPC layer The ARPC layer: The ARPC layer is a remote procedure call interface to the process The main functionality of this layer is to marshal parameters, to call the AmoebaRPC primitives for communicating with the process, and to unmarshal the results from the process Every Unix system call whose implementation relies on the process has a corresponding ARPC routine 4

5 Amoeba System (Dedicated Engine) = AXSI + ARPC Figure 3: Dynamic structure of the A system with the same Unix call name prexed by "us " Amoeba System The process: The process runs on the Unix server system and provides remote Unix system services, including the le service, process and job control, and signal delivery As shown in Fig 4, when an application program () makes a Unix system call, an RPC is made to the process by the part of the process The process makes the corresponding Unix call on the Unix server system and returns the result to the application process 43 Implementation strategies Most of the Unix system calls can be implemented with the strategy illustrated in Fig 4 However, a few Unix system calls, such as execve(), fork(), exit(), and the Unix signal handling, need special arrangement on both the process side and the process side, which are discussed in following subsections Unix call(1) RPC(2) Unix call(3) Unix kernel Figure 4: Remote implementation of a Unix call 431 Program execution In Amoeba, a library routine exec f ile() is used to execute a new program To execute a Unix executable le in Amoeba, the format of the Unix executable le need to be changed into the format of the Amoeba executable le There is an Amoeba utility ainstall [11], which reformats and installs the Unix executable le into the Amoeba system To emulate the Unix execve() call, the following steps are executed (see Fig 5): When 5

6 Amoeba System exec le(4) execve(1) exitprocess(5) Amoeba kernel us install(2) Unix kernel ainstall(3) Figure 5: Protocols for executing a program the part of the process calls execve(), the part makes an RPC call to the process, which uses ainstall() to reformat and install the Unix executable le to the Amoeba system Then, a new process is created for the new program and connected to the old process Finally, the old process calls Amoeba library routine exitprocess() to quit itself The reason for creating a new process rst and then quitting the old one is that the Amoeba routine exec f ile() does not replace the existing program with the new one (like the Unix execve()) Instead, it always creates a new process for executing a new program 432 Process forking The emulation of the Unix fork() involves the forking of both a child process on the Unix system and a child process on the Amoeba system In Amoeba, a process server interface routine pro exec() is used to create a new process [11], whose behavior and status are determined by the process data structure provided as an argument As shown in Fig 6, the following steps are executed: When the part of the process calls fork(), the part makes an RPC call to the process Then, the process asks the Unix kernel to fork a child process on the Unix system The child will inherent all the attributes (eg userid/groupid, pid and ppid, etc) of the parent On the Amoeba side, if the process is able to make a copy of its own process data structure, then it would be farely easy to "fork" a child process by calling pro exec() However, the current version of Amoeba has no way for a process to get the process data structure of itself Therefore, an auxiliary owner process is used, which receives a checkpoint message from the parent process, makes a copy of the parent process data structure, and calls the pro exec() routine to start a child process 433 Process termination An process can be terminated abnormally by signals (to be discussed in Section 434) or normally by calling Unix exit() As shown in Fig 7, the normal termination of an process takes the following steps: rst, the part of the process calls 6

7 Amoeba System pro exec(6) checkpoint(5) OWNER 6 fork(1) pro stun(4) Amoeba kernel us fork(2) fork(3) Unix kernel Figure 6: Protocols for forking a new process exit() explicitly or the C language runtime system calls it implicitly Then, the part of the process makes an RPC call to the corresponding process, which terminates itself immediately after replying the process Finally, the process calls the Amoeba library routine exitprocess() to terminate itself exitprocess(4) Amoeba System exit(1) exit(3) Amoeba kernel us exit(2) Unix kernel Figure 7: Protocols for terminating a process normally 434 Signal and exception handling In Amoeba, signals are used for asynchronous communication between threads within a process A thread may specify a catcher function (via sig catch()) for a particular signal number; another thread may asynchronously cause a call to this catcher in the catching thread by generating a signal (via sig raise()) When a signal is generated, it is broadcast to all threads in the same process that have a catcher for it Threads that have no catcher for the signal are completely unaected In threads that catch the signal, system calls are completed or aborted; the catcher is called just before the system call returns If the 7 catcher returns, the thread continues where it was interrupted When a thread causes an exception (eg, it tries to divide by zero), and it has a catcher for the corresponding exception number, that catcher is called When a thread causes an exception for which it has no catcher, the process is stunned and its owner receives a checkpoint Exceptions are not broadcast to other threads, nor can they be ignored [11] To emulate the Unix signal handling, the following infrastructures are implemented in the A system: Signal server thread: In addition to the main thread which executes the appli

8 Amoeba System signal handler(7) 6 signal dispatcher(6) 6 signal propagator(4) signal server kill(1) sig raise(5) Amoeba kernel us kill(2) kill(3) Unix kernel Figure 8: Protocols for propagating signals cation program, a signal server thread exists in the process to receive messages (containing signal numbers) from the signal propagator (see below) in the process and raise the corresponding signals to the main thread Signal dispatcher: A signal dispatcher routine is used in the process to catch Amoeba signals and to dispatch the signal to a proper handler stored in the signal handler vector (see below) When the process is started, the dispatcher routine is installed in the Amoeba kernel for all Unix signal numbers and those Amoeba exception numbers which can be mapped into a Unix signal number Signal handler vector: There is a signal handler vector maintained by the process to store application signal handlers Initially, all signal handlers in the vector will be set to SIG DF L (Unix de ned default handler) When the application program installs a new handler for a signal (via sigaction()), this handler will be stored in the vector When the application program calls a Unix execve() to execute a new program, the contents of the vector will be updated according to the XSI denition of the semantics of the execve() (see [13]) Signal propagator: A signal propagator is used in the process to catch Unix signals and to propagate them (by sending messages) to the signal server thread in the process The signal propagator routine is basically a Unix signal handler in the process, and it is installed in the Unix kernel for a signal if this signal has a user dened routine as its signal handler, or this signal has SIG DF L as its signal handler and the default action is to terminate the process Signal generation and delivery Signals can be generated on both the Unix system side and the Amoeba system side When the part of an process calls a Unix kill(), the part will make an RPC call to the process, which simply asks the Unix kernel to generate and deliver the Unix sig 8

9 nal to the proper process (see Fig 8) For Unix signals generated by terminal activities and timer expiration, the Unix kernel will deliver them to the proper processes When the part of an process generates an exception (eg illegal memory reference) on the Amoeba system, the Amoeba kernel will deliver it to the proper process For signals propagated as messages from the signal propagator in the process, the signal server thread of an process will rely on the the Amoeba kernel to deliver them to the main thread of this process Signal propagation When a signal is delivered to a process, if SIG IGN (Unix dened ignore handler) has been installed for this signal, it will be ignored automatically by the Unix kernel without bothering the process and the process However, if SIG DF L has been installed for this signal and the default action is not to terminate the process, the default action will be taken automatically by the Unix kernel Otherwise, the signal propagator routine must have been installed for this signal, and will be called at the moment just before returning from a Unix system call It will send the signal number to the signal server thread in the process On the Amoeba system side, when the signal server thread has received the signal number from the signal propagator, it will raise this signal to the main thread of this process by calling the Amoeba routine sig raise() (see Fig 8) Signal dispatching Since the signal dispatcher routine is installed for all the Unix signal numbers and those Amoeba exception numbers which can be mapped into a Unix signal number, once a Unix signal is propagated from the process or an Amoeba exception is delivered by the Amoeba kernel, the signal dispatcher routine will be called at the moment just before an Amoeba system call returns Depending on the nature of the handler for this signal in the signal handler vector, the signal dispatcher routine will ignore the signal, or terminate the process, or call the user dened routine to handle the signal (see Fig 8) 5 Conclusions We have proposed a new approach to making the Amoeba distributed operating system fully compatible with Unix (POSIX) at the source level This approach integrates existing Unix systems into the Amoeba distributed computing environment as specialized Unix servers, so that existing Unix source programs can migrate to the Amoeba distributed environment without modication, immediately benet from the multiple processors computation power in Amoeba, and gradually be adapted to the new environment to make full use of the parallel and distributed facilities in Amoeba A prototype implementation of the A system has been operational on an existing Amoeba system with a network of 3 MC68030 processors plus a Sun 3/60 station The A system is now being used to combine the Amoeba distributed operating system with the sharedmemory Multiprocessor Unix (MU) operating system [10] Work on the A system is continued to make it a standard component of Amoeba Acknowledgements The work described in this paper was partially supported by the CAMAS (ESPRITIII 6756) project References [1] Mullender, SJ, Rossum, G van, Tanenbaum, AS, Renesse, R van, Staveren, H van: \Amoeba { A Distributed Operating System for the 1990s," IEEE Computer Magazine, May 1990 [2] Renesse, R van, Staveren, H van, Tanenbaum, AS: \Performance of the World's Fastest Distributed Operating System," Operating System Review, ACM Press, vol 22 No 4, Octo

10 [3] Accetta, A etc: \Mach: A New Kernel Foundation for UNIX Development," Proceedings of the Summer Unenix Conference, Atlanta, GA, July 1986 [4] Rozier, M etc: \CHOR Distributed Operating Systems," Technical Report 8876, Chorus Systems, Nov 1988 [5] Ousterhout, J, Cherenson, A, Douglis, F, Nelson, M, and Welch, B: \The Sprite network operating system," IEEE Computer, 21(2):2336, Feb 1988 [6] Kaashoek, MF, van Renesse, R, van Staveren, H, and Tanenbaum, AS: \FLIP: an Internetwork Protocol for Supporting Distributed Systems," Vrije University, Amsterdam, The Netherlands, 1992 [7] Birrell, AD, and Nelson, BJ: \Implementing remote procedure calls," ACM Transactions on Computer Systems, 2(1):3959, Feb 1984 [8] Sun, CZ: \Design and implementation of the A system," ACE3002design, ACE Associated Computer Experts BV, Amsterdam, The Netherlands, 1993 [9] Sun, CZ: \Interface specication of the A system," ACE3003spec, ACE Associated Computer Experts BV, Amsterdam, The Netherlands, 1993 [10] Sun, CZ, Keunin, P, Dekker, G, Schipper, J, De Lange, M : \Towards the combination of distributedmemory operating systems with sharedmemory operating systems," Proceedings of Parallel Computing & Transputer Conference, pp 5865, Brisbane, Australia, Nov 1993 [11] The Amoeba Reference Manual, Version 50, 1992 [12] Partable Operating System Interface for Computer Environments, IEEE Standard [13] X/Open Portability Guide, XSI System Interface and Headers, X/Open Company Limited,