Fault Management. Control Subsystem. Management Information Repository

Transcription

1 A Management Information Repository for Distributed Applications Management y Patrick Martin & Wy Powley Dept. of Computing and Information Science Queen's University at Kingston (fmartin, Abstract The Management of Distributed Applications and Systems (MANDAS) project addresses problems arising in the management of distributed applications. Specically, we are studying the areas of conguration management, fault management, performance management, and application metrics and modeling. We are also investigating the tools, techniques, and services needed to support the above management applications. The MANDAS Management Information Repository (MIR) provides database support for the management applications and supports their integration into a single management environment. In this paper, we examine the problem of distributed applications management to extract the requirements for an MIR. We present an information model for distributed applications management and outline a prototype MIR developed for the MANDAS project. Keywords: distributed applications management, information modeling, repositories 1 Introduction Distributed computing systems typically consist of large numbers of heterogeneous computing devices connected by communication networks, various operating system resources and services, and user applications. These resources and applications are becoming indispensable to The authors thank IBM Canada Ltd. and NSERC for their support of this research. y The IBM contact for this research is Patrick Finnigan many enterprises, but as distributed systems become larger and more complex, more things can go wrong, potentially interrupting or crippling critical operations. Thus management support is often cited by -users as the single most important aspect of a distributed system. Management issues at the network and systems levels have received a great deal of attention but the same thing cannot be said for the application level. The objective of the Management of Distributed Applications and Systems (MANDAS) project is to address problems in the management of distributed applications. We take management to include conguration management, fault management, performance management, and application metrics and modeling. More specically, in applications management we include the following [12]: 1. The software tools and techniques, called management applications, that designers of distributed applications and system administrators use to ensure the ongoing and eective design and operation of the systems and applications. 2. The data and information about the applications and systems required by the management applications, and the means of collecting, storing, and maintaining the data. 3. The tools, techniques, and services needed to support the above. MANDAS considers distributed applications that rely on the support of middleware, such

2 as DCE [9] and CORBA [8], for communication between software components. Middleware makes it easier to build distributed applications by providing a bridge between heterogeneous systems in the form of common interfaces and communication paradigms. The middleware can hide many of the complexities of distributed systems through features such as location transparency and data format translation. Figure 1 shows the architecture we used to support distributed applications management [1]. The management applications access the available management services through a common management service interface. The management services interact the management agents through various protocols, for example SNMP or CMIP. Each management agent monitors and collects data about a set of managed objects. The Management Information Repository (MIR) provides services to the three component subsystems and provides a common interface to the set of management agents. The development of an MIR is the focus of our part of the MANDAS project and the subject of the paper. The remainder of the paper is structured as follows. Section 2 outlines data management requirements for distributed applications management. Section 3 describes the information model used by the MIR and Section 4 describes a prototype MIR developed for the MANDAS project. Section 5 discusses related work and Section 6 summarizes the paper. 2 Data Management Requirements To provide database support for distributed applications management it is necessary to understand the properties of the data that exists in distributed applications. We distinguish three types of data: observation data, management data, and control data. Haritsa et al. [2] provide a similar classication for network management data. 2.1 Observation Data The observation data is the raw data collected by the management agents, which includes event notications and a variety of variables that dep upon the managed object. For example, variables describing the execution of a process may include memory and CPU usage, process up time, and the number and type of each interaction another process. Observation data provides the primary input for the performance management, performance modeling, and fault management applications. It provides both the current state, and the execution history of an application in terms of resource usage and quality of service provided to users. During normal execution, observation data is usually collected by a management agent at a regular frequency, but under fault conditions data may begenerated at a higher rate than normal. Another situation is where observation data only arrives when an extraordinary event occurs or when it is requested by a management application. The accesses to the observation data consist mainly of updates to the data by the management agents and reads of the data by the management applications. The updates are typical database updates where the new value is indepent of the current value, but atypical because they happen in real-time. The real-time characteristic of updates means that the overhead incurred by an update must be minimized and so favours a design in which updates can be performed on databases local to the management agents. Read accesses do not have real-time requirements like updates, but they may involve data about more than one managed object or data gathered by more than one management agent. Thus, an important requirement for read accesses is a common interface through which a management application can gather and correlate data about multiple managed objects. 2.2 Management Data Management data is composed of \static" information, such as an application's conguration and the network topology. This data provides the primary input for the conguration management and is used by other management

3 Management Applications Configuration Management Performance Management Fault Management Modelling Management Services Configuration Subsystem Control Subsystem Management Service Interface Monitoring Subsystem Management Information Repository Management Agent Managed Objects Figure 1: MANDAS Management Architecture applications in locating, requesting, and interpreting observation data. Management data can be both read and written by the operator(s) or by MIR control processes. For example, changing the conguration of an application is usually initiated by an operator, but system migration of a process would trigger an update by a control process. Most of the data is stored at system initialization time and changed at a moderate rate consistent classical DBMS applications. Since it is possible that multiple transactions will access the same data at the same time, standard database transaction management would be used. Management data must represent the structures and relationships of a variety of management entities and concepts. Descriptions of resource- level entities, such as those provided by the SNMP and the CMIP SMIs (Structures of Management Information) [13], must be maintained, as well as descriptions of middleware services and application-level entities such as programs and processes. 2.3 Control Data Control data captures the current setting of application and system tuning parameters, such as the maximum number of requests to a server. The process for changing an existing set of control settings is either initiated by the operator or triggered automatically as a function of the information contained in the sensor data. In either case, no more than one process should be able to update a set of control variables at one time. Otherwise control setting would be at the mercy of the transaction processing order. As observation data, it may be necessary to maintain a history of the control data. However these changes will occur at a moderate rate

4 relative to the observation data. 3 Information Model The main role of the information model is to represent the dierenttypes of data objects and their relationships, relevanttodistributed applications management. Because a wide range of tasks is included in distributed applications management, a potentially large number of objects may be related to a distributed application. For example, consider a single process that is running as part of a distributed application. In order to manage the process we need to maintain information such as: attributes of the process such as process ID and its associated port number; the location and format of observation data being collected about the process; the number and types of interactions the process has other processes in the application; the host on which the process is running; the set of source code objects that make up the process. We can also note that some of the observation data collected about application level objects may be derived from observations of resource level objects. Distributed application management must therefore encompass some systems management and network management concepts. The information model must also capture the structure of the management model. It should describe managers, management agents and managed objects. The managed objects consist of the application components and resources used by those components. Management agents provide views of managed objects to the managers. Managers provide the interface to the management applications. Objects may undergo changes during operation. For example, an application process may be moved from one site to another. Performance management requires that a history of an object be maintained so that the eectiveness of the changes can be evaluated. 3.1 Modeling Concepts We describe a distributed application in terms of a set of concepts which include the engineering structures of the Reference Model for Open Distributed Processing (RM-ODP) [11]. This allows us to model the application in terms of its individual components such as objects, operating system processes and nodes. The structure provides enough abstraction to represent applications built using middleware such as DCE or OMG's CORBA. The engineering structures of the RM-ODP are dened as follows: Basic Engineering Objects (BEOs) are the building blocks of the application. They contain the procedures or methods that perform the processing of the system. Clusters are groupings of related objects. They are an abstraction used in reconguring an application. Capsules are the executables that make up the application and are instantiated on nodes as processes. Nodes are the stations in a distributed environment. In addition, we also dene the following concepts: Applications are the entities to be managed. They are described by the set of requests, or transactions, processed by the application and by the set of component clusters. Application Instances are run-time instances of applications. 3.2 Modeling Constructs We use a structurally object-oriented model, based on the Telos language [6], to dene the structures of our information model. We chose an object-oriented model for two main reasons: 1. the entities to be modeled can have complex structures and relationships;

5 2. there are a large number of similar entities, for example processes, which can be grouped into classes. The model uses the following constructs: attribute: An attribute is a particular property of an object. Each attribute has a type which maybeaprimitive type, such as string, integer or real, or a user-dened class. In the latter case, the value for the attribute is a reference to an instance of that class. Attributes maybesingle-valued or multi-valued. object: An object is an identiable collection of attribute values which represents an entity or component such as a process, management agent or resource. Every object has a unique object identier (OID) or name. class: A class is a collection of objects that share common properties. An object is created as an instance ofa particular class. Classes are related via the is-a relationship. metaclass: A metaclass is a collection of classes that share common categories of attributes. The metaclass denes the categories but not the attributes in the categories in the particular classes. The current class structure for the information model is shown in Figure 2. Metaclasses appear above the line and classes below it. The dashed lines represent the instance-of relationship and the solid lines represent the is-a, or inheritence, relationship. The metaclasses dene the groups of related classes in the class hierarchy and identify the categories of attributes shared by these related classes. We present several example denitions to illustrate the features of the model. The root of the metaclass hierarchy is the Repository- ObjectClass. It introduces three attribute categories, namely stateinfo (general properties of an object), repositoryinfo (repository management properties) and relationships (links to other objects), and is dened as: MetaClass RepositoryObjectClass in RepositoryMetaMetaClass descriptions stateinfo: Proposition; repositoryinfo: Proposition relations relationships: Proposition The children of RepositoryObjectClass identify the main groups of classes in the MIR and add attribute categories specic to each group. For example, the metaclass RuntimeObjectClass is dened as: MetaClass RuntimeObjectClass in RepositoryMetaMetaClass isa RepositoryObjectClass descriptions mgmtinfo: Proposition relations mgmtrelationships: Proposition where mgmtinfo represents properties related to the managing object and mgmtrelationships are links related to the managing object. The root of the class hierarchy is the class RepositoryObject. The two subtrees of the root, under ManagedObject and StoredObject describe, respectively, the objects managed at execution time and the software objects used to produce the execution time objects. The class RepositoryObject is dened as: class RepositoryObject in RepositoryObjectClass stateinfo OID: String; description: String It contains the single attribute category stateinfo and denes the two properties OID and description which are inherited by all objects. The class ManagedObject, dened as:

6 RepositoryObjectClass HardwareObjectClass RuntimeObjectClass SoftwareObjectClass VersionedObjectClass RepositoryObject StoredObject ManagedObject DataFile Method VersionedObject HardwareMO SoftwareMO Cluster Application Host Relay Network ApplInstance Process Capsule BasicEngineeringObject Figure 2: Information Model Class Hierarchy class ManagedObject in RuntimeObjectClass isa RepositoryObject mgmtinfo protocol: ManagementProtocol mgmtrelationships mgmtagent: ManagementAgent; mib: ManagementDatabase is a specialization of RepositoryObject and the ancestor class for all instances of managed objects. It introduces attributes which describe the observation data being collected for a particular managed object. The class SoftwareMO, dened as: class SoftwareMO isa ManagedObject groups the collection of managed objects that are components of a distributed application. We view an application as composed of a set of interacting processes (Process) and possibly instances of another application (ApplInstance). For example, a nancial application can use an indepent database application. We assume for now that applications do not wish to manage objects beyond the level of detail of processes. The denition of the Process class is as follows: class Process isa ManagedObject stateinfo PID: Integer; PPID: Integer; UID: Integer; GID: Integer; state: StateClass; relocatable: BooleanClass; starttime: TimeStamp; Time: TimeStamp relationships componentofappinstance: AppInstance; instantiatedfrom: Capsule; depson: Process; depedonby: Process; usesdatafile: DataFile;

7 resideson: Host 4 MIR Prototype The MIR prototype is shown in Figure 3. It is a basic client-server architecture. The MIR Server has two components: the Telos Repository which provides the back- database for the MIR and the MIR Server Interface, which takes requests from MIR Clients and translates them into requests to the Telos Repository. The MIR Client also has two components: an application that requires access to the MIR and the MIR Client Library, which is an API to the MIR. The Telos Repository used for the MIR prototype is the University oftoronto implementation of the Telos language [6]. This implementation uses ObjectStore 1 as the underlying storage mechanism. The Telos Repository currently runs on an IBM RS/ machine located at the University oftoronto. The MIR Server Interface communicates the repository via the Telos Message Bus (TMB). Although theoretically MIR Clients could use the TMB API directly to access the Telos Repository, we decided to build an intermediate layer (the MIR Server Interface) between the Telos Repository and the MIR Clients for the following reasons: Database Indepence - The MIR Server Interface buers the MIR clients from the specics of Telos. This will allow us to change the underlying storage mechanism minimal modications to the overall system. Only the back- of the MIR Server Interface would need to be modied in order to accommodate such a change. The MIR Client Interface would remain stable, thus requiring no change to the possibly numerous MIR Clients. Multiplexing - The TMB allows only a single client to access the Telos Repository at a time. The MIR Server uses threads 1 ObjectStore is a trademark of Object Design, Inc. 2 RS/6000 is a registered trandemark of International Business Machines Corporation to service and coordinate multiple clients, sing one request at a time to the Telos Repository. Exted Query Capabilities - The Telos Repository provides very limited query capabilities. The MIR Server Interface exts these capabilities to include conjunctive queries based on instance-of conditions, is-a relations and queries by attribute value. The MIR Server Interface generates a query which can be processed by the Telos Repository, ss the request to the repository, then lters the result to return only the objects requested by the MIR Client. MANDAS Project Requirement - One of the original design decisions of the MANDAS project was to interface the various components using DCE. The MIR Server Interface is implemented in C++ and runs on an IBM RS/6000 computer on top of OSF DCE. The back- of the interface communicates the Telos Repository via the TMB API. Strings which are parsed and understood by the Telos Repository, called s-expressions, are passed from the MIR Server Interface via the TMB to the Telos Repository. Requests are satised by the repository by returning s-expressions to the MIR Server Interface (via the TMB). The front- of the MIR Server Interface provides an API for use by clients of the MIR Server. A set of functions are provided for connecting to the repository, submitting and retrieving objects, and querying the MIR. Clients communicate the MIR Server via DCE RPCs. A common set of C++ data structures are used to pass information from client to server. The two components of the MIR Client include an application requiring the MIR services and the MIR Client Library. Typically in the MANDAS system, the client program will be a management application. The client library is a collection of functions that allow clients to interact the MIR. The management application uses these functions to store and/or retrieve information from the MIR. The MIR Client Library is implemented in C++ and uses the C++ API provided by the

8 MIR Clients Telos Repository MIR Server Interface TMB DCE MIR Server MIR Client Library Mgmt Appls Figure 3: MIR Prototype MIR Server Interface. The Client Library provides both a C and a C++ interface. The MIR Client Library presents management applications a view of the MIR which is consistent the MANDAS Information Model. There are three basic object types used in the MIR: Management Meta Objects (MMOs), Management Schema Objects (MSOs), and Management Data Objects (MDOs). MSOs are similar to classes in an object-oriented programming language. They dene the attributes that an MDO may contain. The attributes are further grouped into categories that are dened by an MMO. Each MSO is an instance of one or more MMOs (thus specifying the categories of the attributes), and each MDO is an instance of an MSO. The MIR Client library provides functions for the following tasks: connect to, and disconnect from the MIR create, modify, and retrieve MMOs create, modify, and retrieve MSOs create, modify, and retrieve MDOs dene query conditions and issue MDO queries The MIR is used to store the management data, the control data, and the descriptions of the observation data. Observation data is stored in a location determined by the management agent. For performance reasons, it is likely that the observation data would be kept local to the agent to ensure that the data is current. A management application which needs to access observation data rst accesses the MIR to obtain information on the data such as its location, how it is stored (e.g., le, relational database), and how it is structured. The client then interacts directly the observation data source. To date, the repository has been integrated two of the MANDAS tools - the monitoring agents which register each new process the repository [10], and the performance analyser tool which retrieves information about active processes for analysis [3]. Also several tools use the MIR; the MIR Browser which allows a user to examine the contents of the MIR, an administrator's tool which permits population and querying of the MIR via a graphical user interface, and a schema translation tool which extracts information from IBM's DADT tool [5] and stores it in the MIR. Other tools are currently under development and we aim to integrate other MANDAS distributed appli-

9 cations management tools the MIR in the near future. 5 Related Work While there has not been previous work on distributed applications management, the role of databases in network and systems management has been studied. Haritsa et al. describe MAN- DATE (Managing Networks using Database Technology), which is a proposed MIB (Management Information Base) to support network management [2]. Their approach is to have operators interact solely the MIB - the database embodies the network for the operator. The MIB, like the MIR, is the focal point for integration of the management applications. MANDATE also proposes an object-oriented model to support management. Implementation in MANDATE is client-server sophisticated client caching, while our implementation is based on distributed databases. The work of Wolfson et al. [7] has a dierent focus than either our work or MANDATE. They propose a model of network management actions as data manipulation operations and provide an SQL-like language for specifying network management functions. They also discuss the use of database triggers as a mechanism for automatically initiating management events. Finally, Hong et al. [4] investigate the use of an X.500 directory system as a repository for systems management information. The X.500 system has the advantages that it is a distributed database and it is a widely available standard. However, X.500 is inted as a white pages system and not as a general DBMS. There are questions about its general modeling capabilities and its performance. 6 Summary Distributed systems management is crucial to the success of future distributed systems and is an interesting application area for database systems research. We have introduced the MAN- DAS project in distributed applications management and explained the role played by the Management Information Repository in the integration and support of a collection of management applications. We outlined the requirements for the MIR, based on our view of distributed applications management, and discussed an information model and prototype implementation of the MIR. We are continuing to integrate the MIR other MANDAS components and we will soon be able to evaluate its performance and the suitablity of this approach. We also plan to explore a number of extensions to the MIR. We will examine how to incorporate versions and triggers into the model and the implementation. We will also study the relationships among the MANDAS information model and management models such as SNMP and CMIP [13] from network and system management. 7 Acknowledgements The authors would like to thank the members of the Database Systems Group at Queen's University who contributed to the development of the MIR: Dave Barrowman, Peter Zion, Yichun Ding, Daniel Boudreault, and J.P. LeBlanc. About the Authors Patrick Martin is an associate professor at Queen's University at Kingston and is a principal investigator in the MANDAS project. He can be reached at the Department of Computing and Information Science, Queen's University, Kingston, Ontario, K7L 3N6. His internet address is martin@qucis.queensu.ca. Wy Powley is a research associate working on the MANDAS project at Queen's University. She can be reached at the Department of Computing and Information Science, Queen's University, Kingston, Ontario, K7L 3N6. Her internet address is wy@qucis.queensu.ca. References [1] M. Bauer, P. Finnigan, J. Hong, J. Rolia, T.Teory, and G. Winters. Reference

10 architecture for integrated distributed systems management. IBM Systems Journal, 33(3):426{444, [2] J. Haritsa, M. Ball, N Roussopoulos, A. Datta, and J. Baras. MANDATE: MAnaging Networks using DAtabase TEchnology. IEEE Journal on Selected Areas of Communication, 11(9):1360{ 1372, [3] G. Hills, J. Rolia, and G. Serazzi. Performance engineering of distributed software process architectures. In Proc. of International Conference on Performance Tools'95, Heidelburg, September [4] J. Hong, M. Bauer, and J. Benett. Integration of the directory service in distributed systems management. In Proc. of 1992 Conference on Parallel and Distributed Systems, pages 142{149, Hsin Chu, Taiwan, December Integrated network Management, Ottawa, Ontario, October [11] K. Raymond. Reference model of open distributed processing: A tutorial. In J. De Meer, B. Mahr, and S. Storp, editors, Open Distributed Processing II (C- 20), pages 3{14, North-Holland, Elsevier Science B.V. [12] J. Rolia, C. Woodside, V. Vetland, R. Bunt, D. Eager, M. Bauer, J. Hong, H. Lutyya, J. Black, T. Kunz, D. Taylor, P. Martin, T. Teory, and P. Finnigan. Distributed application management, the mandas project. In Proc. of the 4th IFIP/IEEE International Symposium on Integrated Network Management, Ottawa, Ontario, October [13] M. Sloman. Network and Distributed Systems Management. Addison-Wesley Publishing Co. Wokingham England, [5] IBM. IBM VisualAge C++ Distributed Application Development Toolkit for OS/2. User's Guide (Second Edition), February [6] J. Mylopoulos, A. Borgida, M. Jarke, and K. Koubarakis. Telos: A language for representing knowledge about information systems (revised). Technical Report KRR- TR-89-1, Department of Computer Science, University of Toronto, August [7] S. Sengupta O. Wolfson and Y. Yemini. Managing communication networks by monitoring databases. IEEE Transactions on Software Engineering, 17(9), September [8] Object Management Group, Framingham MA. The Common Request Broker: Architecture and Specication, [9] Open Software Foundation Cambridge MA. The OSF Distributed Computing Environment Rationale, [10] G. Perrow, J. Hong, H. Lutyya, and M. Bauer. The abstraction and modelling of management agents. In Proc. of the 4th IFIP/IEEE International Symposium on