A framework for performance monitoring, load balancing, adaptive timeouts and quality of service in digital libraries

Transcription

1 Int J Digit Libr (2000) 3: 9 35 INTERNATIONAL JOURNAL ON Digital Libraries Springer-Verlag 2000 A fraework for perforance onitoring, load balancing, adaptive tieouts and quality of service in digital libraries Sarantos Kapidakis, Sotirios Terzis 2, Jakka Sairaesh 3 Institute of Coputer Science, FORTH, Heraklion, Crete, GR 70, Greece; E-ail: sarantos@ics.forth.gr 2 DSG, Coputer Science Departent, Trinity College, University of Dublin, Dublin 2, Ireland; E-ail: Sotirios.Terzis@cs.tcd.ie 3 IBM T.J. Watson Research Center, 30 Sawill, Hawthorne, New York, NY 0532, USA; E-ail: jraesh@watson.ib.co Received: 8 Deceber 998/Revised: June 999 Abstract. In this paper, we investigate the issues of perforance anageent in large scale, autonoous and federated digital library systes, perforing the tasks of indexing, searching and retrieval of inforation objects. We have defined a anageent architecture and perforance fraework for easuring and onitoring the behavior of digital libraries as they operate. Our architecture and echaniss are easily applicable to other digital library systes of siilar flavor and architecture. We ipleented this architecture over a testbed of Dienst servers using real data and workload fro the operational NCSTRL syste. We have defined the relevant paraeters for investigating the perforance of the servers and have developed visualization tools to onitor the paraeters. In addition, our perforance fraework provides echaniss for load-balancing search requests in a network of digital library servers. We have deonstrated this by building a testbed for investigating a few novel load balancing policies. Given that network delays and outages are unpredictable over the Internet, we have developed new adaptive echaniss to detect tieouts and provide quality of service. Key words: Perforance anageent Load balancing Adaptive tieouts Quality of service Dienst servers NCSTRL Introduction The rapid advances in coputer and networking technology in recent years has provided worldwide access to a huge volue of inforation and services for an increasingly larger nuber of people. Digital libraries [5] recently have eerged to offer a structured way of organizing, indexing, searching, and retrieving inforation. They are architectures for the provision of inforation access and anageent services for inforation repositories consisting of various inforation objects such as text, audio, video, and iage. Currently, there is research being conducted around the world on digital libraries and their associated issues, architectures, and echaniss [3, 5, 6, 9, 3, 4, 7, 24, 32, 39] or solutions to specific issues. Their wide-spread and increased popularity will influence the design of future inforation systes. Digital library systes are characterized by the huge volue of inforation they store (e.g., the Alexandria syste [] stores gigabytes of inforation as satellite iages and aps), by the wide distribution of their nodes (e.g., the NCSTRL-Dienst [8] syste has ore than 00 nodes in over 5 countries covering ost of the USA and Europe) and by the fact that they often integrate existing collections of inforation over the World Wide Web (e.g., the Infobus [25] that connects systes like Altavista, the Alexandra syste [] and the university of Michigan digital library syste [2]). These characteristics ake those systes especially changeable and have a profound influence on their perforance. As a result, echaniss for the dynaic adaptation of the syste to the constantly changing operation environent are required. In this article we investigate the issues of perforance anageent and onitoring in large scale, autonoous and federated digital library systes. In the developent of digital library systes various odels have been deployed. For exaple, Stanford University s [25] view of digital libraries is as a shared inforation bus that connects various inforation sources. On the other hand, Michigan University s [2] view is of a collection of collaborating agents. The coon denoinator in all these different views is that a digital library syste consists of a nuber of servers, spread over the

2 20 S. Kapidakis et al.: A fraework for perforance onitoring, load balancing, adaptive tieouts and QoS in DL s Internet, that interact with each in order to service user requests. In processing a request a server ight invoke a nuber of external progras. All the counication between the servers is done with the use of the World Wide Web s protocol, HTTP. This is the syste odel for our investigation. Our priary goal is to define an architecture for onitoring and easuring the perforance of digital libraries. This architecture should be based on the odel presented above and should allow us to onitor the syste s behavior as it operates, since it will be the base for the dynaic behavior of adaptation echaniss, such as load balancing, dynaic tieout adaptation and support for quality of service searching and retrieval [5, 33]. A secondary but equally iportant goal is that the onitoring process should be applicable to any digital library syste. This iplies that the architecture should (a) ipose inial overhead on the syste s perforance and (b) require inial changes to the syste s code. We have developed an architecture and fraework for the onitoring and easuring of digital libraries. We ipleented this architecture over a testbed of Dienst servers using NCSTRL data. We defined the relevant paraeters for investigating the perforance of the servers. We conducted a perforance study on our testbed using soe special visualization tools, which we ipleented and tested. The perforance study was based on the tracking of a search request by onitoring the syste and led to the proposal of soe odifications on the design and ipleentation of the Dienst syste. In order to deonstrate the perforance architecture for dynaic behavior adaptation, we designed and ipleented soe load balancing strategies for distributed searching over a network of servers. According to these strategies a server, using local observations, can forward a request to the least loaded server fro the pool of those servers where the requested inforation is available (replicated). We built a testbed to deonstrate the extensibility of our architecture, investigate load balancing policies and dynaic tieout adaptation during distributed searching. In Sect. 2 we present our architecture for perforance analysis and interesting application areas: perforance onitoring, load balancing, dynaic adaptation of tieouts and quality of service. We also discuss architectural ipleentation issues and the supporting visualization tools we have ade. In Sect. 3 we describe the Dienst syste, analyze its operation, present our extensions to it to take advantage of perforance onitoring, and explain how we carried out perforance onitoring on Dienst. In Sect. 4 we present the results fro using our ipleentation, with ephasis on the load balancing results, The least loaded server could be based on the average of the response ties for search requests sent to that server. The response tie takes into account the processing load at the server and network delays. More coplex functions can be chosen, but this is beyond the scope of this paper. and conclude in Sect. 5. Appendix A gives details, and a atheatical forulation, of the Dienst request processing perforance odel. 2 Perforance fraework for onitoring and load balancing Considerable work has been done in the area of perforance anageent and onitoring in distributed systes [0, 2, 6, 20, 23, 28, 29, 3, 34]. Besides this research work, a series of coercial products for distributed systes perforance anageent is also available [, 4, 22, 26, 27, 30]. Of particular interest in our case is the work in online perforance onitoring [34], since our goal is to design a perforance onitor that (a) will not interfere with the syste s operation (external onitor) and (b) will onitor the syste in operation. Additionally, perforance onitors deploy various seantic odels in their operation according to [2] and can be classified based on approaches like progra profiling (e.g., Parasight []), event based (e.g., Pablo [28, 29]), or inforation odeling based (e.g., the perforance onitor described by Snodgrass in [3]). In our case the ost appropriate approach sees to be the event based since according to our syste odel the actions of interest are coponent invocations and essage exchange. Although past research work has addressed ost of the ain issues in perforance anageent, the use of a current coercial product is forbidden because they (a) are proprietary and (b) are liited only to soe hardware and software platfors. 2 In this section, we discuss the perforance onitor architecture. The perforance onitor easures a nuber of perforance paraeters and provides access to the easureents. These paraeters are defined through a perforance analysis of the library syste. With the use of these easureents we can ipleent a series of policies that iprove the perforance of the systes both at the adinistrative and user level. These policies are load balancing, dynaic adaptation of tieouts, and quality of service. 2. Perforance onitoring in digital libraries We defined the iniu server functionality to support a anageent architecture for onitoring and easuring. We developed a siple client-server architecture odel for perforance anageent of the digital library. This odel is based on ideas fro SNMP-based perforance anageent in networks and distributed systes. 2 The developent of the Universal Measureent Architecture UMA [37], an X/Open standard (ipleentation available by Adahl [38]), although dealing with the proble of hardware and software incopatibilities, does not deal with the proble of proprietary technology.

3 S. Kapidakis et al.: A fraework for perforance onitoring, load balancing, adaptive tieouts and QoS in DL s 2 We developed echaniss for easuring the delays in the various coponents of the digital library syste. We have broken down the perforance onitoring architecture into five ain eleents, which are stated as follows: Perforance paraeters: we define and nae the perforance paraeters (of various software odules and coponents) for the various tasks perfored by the server when servicing a user request. Each perforance onitor keeps a list of well-defined paraeters, which it updates based on every request it processes. Measureent syste: we easure and store the perforance paraeters, by developing a easureent syste for updating and storage. We developed echaniss to easure the variables for every request generated. A easureent process (daeon) updates the perforance variables and stores their current values, averages and variances in a database. Protocol: we define a protocol to retrieve the perforance paraeters. The paraeter database is anaged by a process, called database anager process (DMP), which returns the variables and values in the database. Visualization tools: we use tools to visualize the perforance paraeters during the operation of the digital library syste. Messaging: we extended the digital library protocol in order to retrieve and report the perforance paraeters during searching. We developed a siple essage protocol to onitor and debug the server. Only the first and last coponents are digital library specific and ust be designed and ipleented differently for each syste. The rest are generic enough for ost digital library systes. A perforance variable is associated with each task (or function) that we wish to easure. For exaple, we easure the tie spent by a server while searching its local database. In order to define the perforance paraeters, we first analyze the digital library syste to find and clearly define all procedures of interest, which we call coponents. Although different coponents ay overlap, we ust specifically locate their starting and ending ties. Figure indicates a possible relation for soe coponents and their starting and ending ties. Module B Module A Module C Tie Axis Request Tie Staps start t_start MODULE MECHANISM Perforance Monitor end t_end Request Fig. 2. Messages passed by coponents, to the perforance onitor The perforance onitor captures the tie spent by the server while perforing the various tasks of a user request. The entry and exit tie of coponents are recorded, as illustrated in Fig. 2. Each server sends two essages to the perforance onitor: the first essage when the task begins, and the second just before it finishes. This is done for every task we easure. The essages are sent through a socket to the perforance onitor. These are the only changes that are needed on the server. This way, the changes to the code of the digital library are inial, and do not coproise its functionality and coplexity. Additionally, the digital library perfors as inial additional work as possible, and does not sacrifice its perforance, as only soe extra essages are sent, and separate processes take care of the rest of the procedure. Finally, the digital library does not depend on the existence or the operation of the perforance onitor. In our architecture, the perforance onitor can accept any connections, for retrieving perforance paraeters of the digital library syste. Two alternative interfaces are provided for this purpose. One for direct requests to the perforance onitor, and one for requests through the digital library syste, as seen in Fig. 3. This way, others, like syste adinistrators and users interested in the syste perforance, as well as the digital library itself, can ask for the perforance of the syste. Using the appropriate requests, we can retrieve specific or all perforance variables fro the perforance onitor. USER req / rsp Digital Library paraeters easureents Perforance Monitor TB TC TA Fig.. Exaple relation of coponents (T A >T B + T C ) ADMIN Fig. 3. Interaction with the perforance onitor

4 22 S. Kapidakis et al.: A fraework for perforance onitoring, load balancing, adaptive tieouts and QoS in DL s Using this architecture, we can also build a selfadapting digital library syste, a syste where the digital library, just like any other process, queries its own perforance, as seen in Fig. 3, and can take decisions that depend on past perforance. Now, the digital library can adapt to the perforance conditions (e.g., the tieouts), and can explore new alternatives (e.g., load balancing). 2.2 Load balancing In creating and aintaining a large distributed digital library syste with any servers on the Internet, the probles of replication of indexes and objects and load balancing need to be addressed. Especially when the digital libraries contain ultiedia objects, which, while being searched and retrieved by users, will use up any syste (local, reote and network) resources. A easure of the syste load is very useful to route requests to the idle (or least loaded) servers that contain the relevant objects. We expect that servers will replicate inforation objects or indexes of inforation objects in order to utilize network resources efficiently. By the ter load balancing we ean the dynaic routing of requests to the servers of the library syste. Since digital libraries are naturally dynaic systes static routing of requests is deeed to be inefficient. The perforance onitor with the easureents it produces allows the library systes to estiate the response tie of the various servers. So, it can use these estiates for dynaic routing of the requests. The goal is the iniization of the total response tie. There are a nuber of different echaniss for the retrieval of the easureents: Polling: the library syste periodically sends a essage to all the servers of the syste, easuring their response tie. These essages are either ping essages or pseudo requests. Polling is not widely used due to its high counication cost (a lot of ping essages). Probing: the library syste periodically sends probing requests to all the servers and they reply with their average processing tie and their current status. The disadvantage of this ethod is that all the servers have to be contacted which draatically increases the response tie of the syste. Nae services: every server advertises its response ties in special naing servers. In this case we have a cobination of the above approaches since the servers periodically send data to the nae servers and they retrieve these data whenever they need an update. The only difference is that the whole process is done through a nae server. This approach is particularly appropriate for large scale systes. In these systes the nae server could be distributed in order to iniize the cost for contacting the. Load balancing is possible only when replication is used. The replication can be based on political and/or technical reasons or, in the best case, on past perforance statistics fro server load and network conditions. Considerable work [7, 8, 35, 40] has been done in developing algoriths for load balancing jobs or transactions or queries in large coputer systes, but very few [2] have investigated issues in designing and ipleenting echaniss for onitoring perforance and load balancing jobs in distributed coputer systes spread across a vast network such as the Internet. Soe advantages of load balancing are that there is no need for a priori network knowledge, especially when nodes are setup for the first tie, and it can lead to better perforance, with dynaic syste reconfiguration, as the syste always adapts to the changing environent. Also, it leads to a sipler syste structure, where there is no need for explicit specialized roles such as Merged Index Servers and Backup Index Servers, and provides better reliability when servers are down, as others are autoatically the next choice. To avoid lengthy coputations on the digital library server, the digital library protocol has to be further extended to get precalculated results, like the ordered list of servers to query. Soe disadvantages and probles of load balancing is that the network conditions change continuously, and perforance data based on non current inforation are partially useful. In a few circustances, there ay be no previous history and the decision will be alost rando. Finally, there ay be unpredictable response tie arising fro factors that are not easily detected, such as fro a dependency on query coplexity. Our perforance onitoring syste can also be used for load balancing: since the perforance easureents are known to the perforance onitor, the digital library server can ask for these easureents and decide where to send its requests. The load balancing ay refer to requests that access indexes (ostly search queries) and/or to requests that access objects (objects retrieval). There can be advantages in both cases: the use of indexes is ore dense, while objects ay be bigger. The echaniss for load balancing are the sae for both cases. We provide the echaniss for load balancing, a good testbed for developing, debugging and evaluating policies. 2.3 Dynaic adaptation of tieouts A big proble in distributed systes is the synchronization of the distributed processes, due to the high diversity of syste heterogeneity and internet bandwidth. When a process waits for another process on a different host to finish, it never knows how long it should wait for, as the process ay still be working, or ay be unable to return its result. Unnecessary waiting results in degradation of perforance, and a good estiation of the aount of tie to wait, the tieout period, can iprove the response tie with inial (or the desired degree of) inforation

5 S. Kapidakis et al.: A fraework for perforance onitoring, load balancing, adaptive tieouts and QoS in DL s 23 lost. In ost systes, the tieout periods are predefined constants. We also propose novel tieout adaptation echaniss which set tieouts for distributed searching in a dynaic fashion by estiating the round-trip delays to various digital library servers and by defining appropriate waiting tie intervals. The estiations are based on the response tie history for each server that the perforance onitor keeps. The tieout period is set to be: in{to user,ax{t esti } + T SI },whereto user is the upper liit for the tieout period set by the user, T esti is the response tie estiate for server i,andt SI is a safety interval. The purpose of the safety interval is to aintain the probability that server i will respond in tie above a user-defined liit. So, if we know the ean response tie and the response tie variance for server i then the Tchebisev inequality (P [ x E(x) >c ] σ 2 (x)/c 2, where x is the current response tie, E(x) andσ(x) are the current ean and standard deviation of the response tie and c is a user defined constant) can provide us with the required safety interval. For the sooth operation of parallel and distributed searching echaniss, tieouts of search requests to various servers need to be addressed. These are crucial to the operation of the digital library syste under network or server failures. The values of the tieouts play an iportant role in the distributed search response tie. For exaple, short tieouts will ake the servicing of coplicated queries or the use of reote servers ipossible. On the other hand, too long tieouts will deteriorate the response tie of the syste and its utilization (the syste will spend a lot of tie waiting for servers that are unreachable). 2.4 Quality of service provisioning The data included in digital libraries are in various forats and every forat usually has a different quality of presentation (e.g., higher analysis iages, different docuent forats). Also, for the retrieval of the library objects various sets of servers could be asked (e.g., only the closest servers) and various search ethods could be used (e.g., keyword search or full-text search). In any case the retrieval is a tradeoff between speed and quality. It would be good for the user to be able to specify the level of quality he/she wants for his/her requests. The use of the perforance onitor can provide the user with estiates on consuption of resources for the various levels of quality. So, the user can specify the level of quality he/she wants and the systes could find out if it can guarantee that level of quality based on the perforance estiates it has. The library syste could even ipleent a negotiation echanis for the discovery of a set of servers that could provide the requested quality of service. Finally, since the syste onitors resource consuption it could support charging echaniss too. Quality of service guarantees can only be given when we have an estiate of the current network perforance. Thus, our architecture is a prerequisite for that. As a first step, we only show the expected perforance to the users by estiating the expected perforance of the available operations (like the transfer tie of files) fro past history. For this functionality, we also used an external tool that onitors all TCP/IP packets fro the local network to all destinations, keeps statistics and provides available bandwidth inforation, on request. Our echaniss can be used for full quality of service support, according to the desired levels of service. To support any levels of quality of service, the past and current perforance is used and any possible scenarios can be evaluated. 2.5 Ipleentation issues and supporting tools The way that the perforance onitor counicates with the rest of the world, does not specify its internal structure and its evolution. In its sipler for, it could be a single-threaded process. A syste with two anager processes, which, technically, can be different processes or just threads, is ore functional:the first process captures the essages, tie-staps the and passes the to the second process through an open pipe; the second process coputes the difference between the tie-staps and updates the corresponding variables. It also coputes the ean and variance. This way the first process is always unloaded and can process requests instantly, so that the added tie-staps are accurate. Of course, if the digital library syste has the ability to satisfactorily provide accurate tie-staps with negligible perforance cost, the first process can be eliinated. As the second anager process ay have to ake heavy coputations, and during this tie is unable to process requests, a third anager process, called Database Manager Processes (DMP), that anages the variable database, ensures that perforance responses are given instantly, as seen in Fig. 4. The second anager process still akes all its heavy calculations, and when new results are available, it sends Digital Library Outside World signals socket Request Perforance Paraeters socket Response tie-staping st MP (DMP) Database Manager Process tie-staps pipe Transfer Variables socket Perf Variables Perforance Monitor coputation 2nd MP Transfer Variables socket Fig. 4. Internal structure of the perforance onitor Outside World

6 24 S. Kapidakis et al.: A fraework for perforance onitoring, load balancing, adaptive tieouts and QoS in DL s the updates to all processes connected it, such as the DMP, which is then responsible for answering requests fro the outside world. Perforance tools can be easily connected now: tools that poll for new values can connect to the DMP, and tools that passively wait for updates when they are available can connect to the second anager process Visualization of perforance paraeters The perforance paraeters are useful to adinistrators, to onitor the perforance of the network and their systes, to detect probles and to ake appropriate decisions to iprove the perforance. Also, the perforance paraeters are useful to users, to see the perforance of their syste and adjust their actions or expectations. In any case, visualization tools are needed to present the perforance behavior to huans. There are any different ways that users can see the perforance paraeters, such as: Using the perforance log files directly. Our perforance onitor keeps logs, if configured to do so, in any files in htl forat with links between the (like fro one digital library coponent to another, following the execution path), so that users can use the offline to process the paraeters and see the flow of inforation and to follow process or paraeters relations, using the links. Although this is very helpful for statistics and post-orte debugging, this was not the ain goal of our onitor. Using a WWW browser and perforance onitoring requests, users can see current values of the perforance paraeters. The perforance onitor accepts http requests for perforance and forats and sends the reply in htl forat, so that the user can use his/her failiar WWW browser to utilize the onitor. The replies have a strict structure, so that they are easily parsable by progras, too. Using a graphics tool users can see the paraeters as they change. We built a graphical visualization tool, as shown in Fig. 5, that can connect at the perforance onitor and obtain the current syste perforance and display it. In this tool, every request creates a new set of points. The tool has a Java-based user interface, can selectively display soe of the perforance variables as they change in real-tie (i.e., new requests are coing), and can also read the perforance onitor log files and display the past values of the perforance variables, while appending the new requests to the picture. Fig. 5. Our perforance visualization tool

7 S. Kapidakis et al.: A fraework for perforance onitoring, load balancing, adaptive tieouts and QoS in DL s 25 Fig. 6. Load balancing visualization tool Using other (ost probably interactive ) custo tools or agents. Since our syste is open, any such tools can be ade and connect to the perforance onitor (possibly at the sae tie, too). In order to visualize the load balancing status and effectiveness, we also ade a load balancing onitor tool, as shown in Fig Using the perforance onitor on Dienst In order to test our ideas, we need to ipleent the on a real, working syste. We applied our ideas to the NCSTRL [9] based digital library syste, which consists of Dienst servers distributed across the Internet. Dienst was our syste of choice because it is used in NCSTRL, 3 and connects sites all over the world, providing a good natural testbed for distributed testing on digital libraries. Each Dienst server anages a collection of coputer science technical reports (docuents) owned by organizations such as coputer science departents and coputer science research institutions. 3. Description of Dienst syste We applied our perforance anageent architecture for easuring and onitoring the operation of the NCSTRL-based digital library syste. We conducted our experients over a testbed of Dienst [9] servers; Dienst 3 NCSTRL stands for Networked Coputer Science Technical Report Library. uses the WWW protocols (ainly HTTP) for searching and presentation. These servers anage three basic library services: (a) repositories of ulti-forat technical reports; (b) indexes of the technical reports collection and search engines for these indexes; (c) distributed search and retrieval. Dienst is a digital library syste that provides transparent distributed search and docuent access and each node of the syste consists of a database that contains the available objects (reports), a WWW server that handles all incoing Dienst requests, Dienst CGI stubs that are called by the WWW server, and a Dienst server that is called by the CGI stubs. The operation of a Dienst server is as follows: a user subits a keyword search query 4 (request) to one of the Dienst servers. The Dienst server initiates a search request to the other Dienst servers, responses are collected, and the user is presented with the search results. We defined the perforance variables which capture the delays experienced by each user request as it propagates through the coponents (odules) of the digital library syste. The coponents, for exaple, include the delays in WWW interface to the digital library (DL) syste, local digital library server processing, network delays and reote digital library server processing of the user requests. To overcoe bad network connectivity and delays, Dienst divides the servers into regions and replicates the indexes for reliability. It uses Backup Index Servers, when a Dienst server fails to respond in tie and additionally 4 In this paper, request, search request, query, all ean the sae.

8 26 S. Kapidakis et al.: A fraework for perforance onitoring, load balancing, adaptive tieouts and QoS in DL s it uses (statically assigned) Regional Meta Servers and Merged Index Servers (Regional Index Servers), which keep a replica of the indexes of the other regions. A query within a region is first directed to the Dienst servers in the region and the MIS (Merged Index Server). The cobined results are collected and subitted to the user. The user then chooses docuents, and obtains the via the URLs supplied by the search results. There are several factors that affect the overall response tie of the search request, like the processing capacity of each Dienst server, the load (current nuber of active user requests) generated at the Dienst servers, the network delays between the Dienst servers, the (local and/or reote) processing tie of a search request as a function of its type (there are any types of queries: siple keyword to ore coplex ultiple keyword based searching), and local searching tie and the size of the index file at each Dienst server and the MIS. Each MIS acts a front-end index server to the rest of the regions in the world. Due to the replication of indexes by the MIS of each region, user queries could be subitted by the Dienst servers of a region to other MISs as well. To iprove scalability and perforance, an entirely new architecture is needed, without such strong statically assigned roles, based on perforance facts and not speculations. 3.2 Analysis of Dienst operation As seen in Fig. 7, the user counicates directly with one Dienst server, and when he/she issues a Dienst request, this server decides where to forward the split subqueries. Each server is responsible for searching its own local Database. Each Dienst request goes through any stages, as seen in Appendix A, where ost of the are trivial coponents, but ay introduce significant overhead. According to our perforance architecture, our odel for perforance onitoring on Dienst is siple as illustrated in Fig. 8. The perforance requests also use the native Dienst odel: they can be sent to a Dienst server, Dienst Server st MP MMS DMP 2nd MP Fig. 8. The Dienst perforance anageent syste and this server decides where to forward the split subqueries. Each server is also responsible for counicating with its own perforance onitor to retrieve perforance paraeters. By using our perforance onitoring syste on Dienst, we studied the Dienst protocol and progra and located inefficiencies. We also studied the log files, to better explore the flow of inforation. We propose iproveents, such as a new protocol for server counication and tieouts. We extended Dienst architecture to perfor load balancing and proceeded to an ipleentation, which was used for experientation and to obtain exaple results. In order to have a user-friendly way to access the perforance requests and to be able to ask the Dienst server itself for the, we extended the Dienst protocol by adding new Dienst perforance requests: Dienst/Stat/2./Print-Local-Paraeters for retrieval of all paraeters local to this host only and Dienst/Stat/2./Print-Paraeters for retrieval of all paraeters of all hosts. These requests just call the appropriate request fro the perforance onitor and redirect their output. The perforance requests can be asked for, like all other requests, directly on the WWW or fro links in the Dienst interface. Users can continue using original Dienst requests only, and the perforance onitor is invisible to the. The requests return siple htl tables, as in Fig., that can be shown directly to the user, or parsed by progras. More coplex requests can always be answered directly by the perforance onitor. 3.3 Distributed search tieout adaptation User ADMIN Keyword Perf DS Perforance Paraeters Statistics Perforance Paraeters DS2 MP Fig. 7. The user view of the route of a query The variance of the response ties, as seen in Fig. 9, indicates that tieout selection affects perforance. Figure 9 shows the distribution of the response ties for 00 requests to 00 distinct servers. Soe servers did not respond and have no response tie in the picture. The horizontal lines indicate possible tieout settings. The higher the tieout setting is, the longer the user has to wait for getting the final answer to his/her query. For these tieout setting, we can see the nuber of servers that would not have been able to respond in tie. The Dienst syste in the current ipleentation uses TO reote (the local or reote database search tieout),

9 S. Kapidakis et al.: A fraework for perforance onitoring, load balancing, adaptive tieouts and QoS in DL s Response Tie (secs) Request Nuber Fig. 9. Response tie distribution and tieout cut-off TO search (the total search tieout) and TO backup search (see Fig. 0). The values of all three are set at configuration tie. The use of static values for the tieouts coplicates the tieout proble because of the dynaic nature of the syste. As a solution to the tieout proble, we propose dynaic adaptation of tieouts, based on the history the perforance onitor keeps for the response tie of each server as was described in Sect By abolishing the use of the tieout for the search of the local index database (TO reote ), this tieout will be ignored and replaced by the total search tieout (TO search )in the reote request essage. The reote server will estiate the probability of processing the request locally before the tieout expires, using perforance statistics. If the estiated probability is low then the server can notify for its inability to service the request. The above extensions to the protocol result in soe changes in the use of tieouts. Since the local server knows quite early if the reote server is alive or not and the goal of the total search tieout is to avoid blocking because the reote server is down or unreachable, then we have ore inforation for the reote server s condition and we can adjust the tieout value. Thus, we can give a bigger safety interval T ESI (Extended Safety Interval), for the servers that follow the extended protocol because quite soon either it will not respond and we could proceed with forwarding the request to another server, or it will tell us it is alive and give it ore tie to process the request. A new tieout for the alive essages is needed. The above extensions abolish the notion of Backup Index Server. Thus, the procedure alive essage tieout and total search tieout procedures can be expanded Start Local Dienst Server Reote Dienst Server Start "alive" Tieout st Backup Start "alive" Tieout Local Dienst Server Reote Dienst Server Search Tieout Backup Start Backup Index Server Search Tieout Reote Dienst Server 2 Backup Search Tieout Backup Search Tieout 2nd Backup Start "alive" Tieout Reote Dienst Server 3 Backup Search Tieout Tie Axis Tie Axis Fig. 0. Current and new tieout echanis

10 28 S. Kapidakis et al.: A fraework for perforance onitoring, load balancing, adaptive tieouts and QoS in DL s to all the appropriate servers, until either we get an answer to the request or we exhaust the tie of TO user.on the other hand the notion of the backup search tieout (TO backupsearch ) could be aintained, set as: in[to user,t st est + T BSI ]. () T st est = ax[t est i] of forula and T BSI is the Backup Safety Interval. For exaple, the Local Dienst Server sends a request to a Reote Dienst Server. The reote server ties out, and the request is sent to the Backup Index Server and a reply coes back. For exaple (see Fig. 0, the local Dienst server sends a request to reote Dienst server. The reote server replies I alive and can t service the request. The local server forwards the request to reote Dienst server 2. The reote server 2 replies I alive and can service the request. The reote server 2 ties out and the local server forwards the request to reote Dienst server 3. The reote server 3 replies I alive and can service the request and after soe tie returns the reply to the request. Note that the request is forwarded to reote server 3 before the search tieout expires and the absence of specialized Backup Index Servers. 3.4 Load balancing We have neither developed efficient policies nor built a specific policy into our syste. We did try an indicative policy that gave acceptable results. We did this in order to deonstrate the functionality of our syste. Good load balancing policies ay depend on geographical data, connectivity, reliability requireents, data distribution and replication, user requireents and even on legal issues and individual cooperation and deals. We have provided the testbed to experient with such algoriths. We have built the echaniss, but in order to test the we also need policies. Load balancing can refer to retrieval of either indexes or data. In our policy ipleentation, we only perfored load balancing on indexes. The algoriths for dynaic routing are based on estiates of the response tie of alternative server choices that the perforance onitor provides. Their goal is the iniization of the total response tie. In particular, suppose that we want to route a request for the publishers a i where i =,...,N and that ER a,...,a Nk jk is the estiated response tie for server S k for a request fro server S j for the publishers a i, i =,...,N k and M the nuber of servers. Then the proble of choosing the appropriate servers for the requests to all the publisher is as follows: Choose:S a i k,k=,...,m For all ites a i,i=,...,n Where in (S,...,S M ) S k (S,...,S K )ax k=,...,m ER a,...,a Nk jk Choose a nuber of servers and a distribution of publishers to those servers that iniize the total estiated response tie. The estiated total response tie is the axiu estiated response tie of all the chosen servers. This algorith gives the optial server choice but it requires coputing the estiate for all the cobinations of servers and publisher distributions. This eans that its coplexity grows exponentially with the nuber of available choices. We chose to ipleent a sub-optial algorith but with significantly better coplexity as follows: Choose:S a i k,k=,...,m For all ites a i,i=,...,n Where:in Sk ER a i jk For each publisher choose the server with the iniu estiated response tie. 4 Experiental results Here we describe a session where we experiented with onitoring the perforance of three Dienst servers (DS, DS2 and DS3), and interpret the results obtained. We defined a few variables and onitored the. Dienst requests (keyword based) were sent to one server (DS) and the perforance variables were onitored for each request and the statistics coputed. Figure, displays an instance of the output of the perforance onitor, as will be explained later: The perforance variables capture delays in the following coponents: (a) local search response tie; (b) local index database processing tie; (c) reote processing tie of the index database; (d) reote search response tie (includes network delay+reote processing tie). For each perforance variable, the following are observed:calls: nuber of requests (easureent points); RSP: current value of the variables (last request response tie); Min: iniu value of the variable; Max: axiu values of the variable; Total: aggregate of the easureents; Mean: total divided by the nuber of requests; Std Dev: standard deviation of variable values. In Fig., 5 requests were generated at Dienst server. The requests were routed to the reote Dienst servers 2 and 3 for a search in their index files. The request is also routed locally for a search in the local index file (of Dienst server ). The paraeters for Dienst server 2 and 3 are siply the reote search response ties for both servers. There are two variables per reote server. T DS2 is the tie taken to perfor a search in the index file for the user request. The average tie is seconds. T nds is the tie taken for the overall operation before the reply is sent back to Dienst server, which initiated the request (for an explanation of what the paraeters represent see Appendix A).

11 S. Kapidakis et al.: A fraework for perforance onitoring, load balancing, adaptive tieouts and QoS in DL s 29 Print Paraeters Perforance Paraeters of Dienst Server 3 Paraeter Calls RSP Min Max Total Mean Std Dev T DS T nds local Perforance Paraeters of Dienst Server 2 Paraeter Calls RSP Min Max Total Mean Std Dev T DS T nds local Perforance Paraeters of Dienst Server Paraeter Calls RSP Min Max Total Mean Std Dev T DS T DS T Server ->Server 3 isserver 3 T Server ->Server 2 isserver 2 T Server ->Server isserver T local nds T total nds T isserver T isserver T isserver NCSTRL-ERCIM This server operates at Institute of Coputer Science Technical Report Library at Server. Send eail to terzis@csi.forth.gr Fig.. Web-based user interface to onitor the perforance variables Network delay between two Dienst servers can be coputed by siply subtracting the reote request response tie between two servers (stored in the reote database) fro the reote request processing tie (stored in the local database). 4. Locating perforance inefficiencies via onitoring Perforance onitoring was used in profiling Dienst search queries and for studying the effect of paraeters on their response tie. The paraeters studied are divided into two categories: (a) those that depend on the configuration of the Dienst server (e.g., index database size); (b) those that depend on the achine that hosts the server (e.g., eory size, cpu speed, etc) and the network delay (latency) to reach the server. The testbed consists of four Dienst servers:. a Sun4c workstation with 48 MB eory and SunOS an Alpha workstation with 32 MB eory and Digital Unix V an Alpha workstation with 64 MB eory and Digital Unix V a Sun4c workstation with 6 MB eory and SunOS The two Sun achines have the sae index database (4644 words). The first Alpha achine indexes 0 979

12 30 S. Kapidakis et al.: A fraework for perforance onitoring, load balancing, adaptive tieouts and QoS in DL s words and the second words. The load was heavy and artificially created. The load is generated by a perl script which subits keyword queries to the one of the Dienst servers. Even fro our testing easureents, it becae apparent that the perforance of the Dienst version 4..4, that we tested, is seriously affected by: Machine load (and other achine properties). The first observation was that the high resource consuption of the current Dienst ipleentation iposes a liit on the rate of requests it can service. This request rate liit depends on the achine configuration (ostly the eory), and thus, factors that affect the, like the index database size and the coplexity of the queries. We can see the difference of request processing tie over achine eory size: Sun4c, Meory Size in MB 6 48 Request Processing Tie s s Also, a request that searches only one database needed.6 seconds when the database was on a different achine that was easily accessed (on the sae local network) and 2.95 seconds when the call was to the sae achine (to the local database), even though the databases had the sae content and there was no network latency involved: the overhead that was introduced by creating a new process on the non idle achine is high. The continuous forks of Dienst (one for every request to the Dienst server) and the fact that it is written in a scripting language, perl, significantly affects the syste overhead. Data Base size. It sees that the decision of Dienst to hold the whole index database in eory, creating huge processes and swapping the out of the eory seriously affects its perforance. In the following table we see the size of the Dienst process and a typical request processing tie as a function of the size of the index: Words in Index DB MB of Dienst process Request processing tie 2, 327 5, 326 0, , s 2.2s 4.9s 6.4s Request coplexity. Different types of queries need different processing ties. Dienst sees to optiize queries up to 2 keywords, sacrificing perforance on the other, ore rare cases. Network delays. Typically, the ost significant part of the request s tie is the network latency, and is non flexible. The exact percentage of the Network Latency to the total request servicing tie (as shown in the table below) depends on other factors, and ainly the index database size. It takes an index database alost seven ties bigger in order to ake the network latency part non-doinant in request servicing tie. Words indexed 0, , 05 Network Latency Total execution 70% 35% The percentage of the Network Latency depends, also, on the network distance of the server initiating the request and the server servicing it. For exaple the use of a reote server instead of a local, added about 2 ore seconds of the Network Latency tie, as easured at that tie. Tieout setup. The values of the different tieouts on the Dienst protocol also affect its perforance. A long tieout value not only increases delays, but also ay let Dienst processes copute results that nobody is waiting for. 4.2 Load balancing Using this siple perforance architecture in our testbed, we easured the delays in various coponents of the digital library syste, such as local processing of the user request, network round-trip delays when the request is sent to the reote sites for processing, and reote site processing delays. Each server keeps statistics such as averages and variances of each perforance variable that is defined. For the sake of siplicity, these variables are coon for all kinds of queries. In general, siple queries need very sall coputational tie when copared to the ore coplex ultiple-keyword or for-based query. Keeping track of variables for each kind of request is coputationally intensive. Here we describe soe ipleentation details in our load balancing experients. We set up 8 servers, divided into 4 sets, each set having 2 servers that have identical content. Also, there is partial overlap between the content of the different sets of servers. The server sets and locations 5 were: UoCrete-Greece UoColubia-USA FORTH-Greece_3 GMD-Gerany FORTH-Greece_ FORTH-Greece_2 FORTH-Greece_4 CNR-Italy Here are a few siplifications that were used by our policy: When a digital library node holds data for any 5 The four FORTH servers are pseudo-reote. This eans that they add a rando delay to each response so that their response tie is siilar to the really reote servers.

13 S. Kapidakis et al.: A fraework for perforance onitoring, load balancing, adaptive tieouts and QoS in DL s 3 publishers, we assue that all questions to the publishers are independent. When a server can provide ultiple publishers the perforance of the server depends on all the publishers in question and the total load assigned to the node. For ore detailed results, see [36]. In Fig. 2 we present two different experients. In both experients we show the distribution of the sae set of requests 6 testbed. It is iportant to notice that changes in network and achine load are depicted in the distribution of the requests. For exaple, looking at the pair of the Italy and Gerany servers, which are identical, we see that in the first experient the network latency and thus the response ties of both servers were alost the sae, as the requests were distributed alost evenly between the two. On the contrary, in the second experient the network latency to the server in Italy was significantly higher than that for Gerany. Thus, the response ties of the Gerany server were constantly lower, so the requests were alost all routed to the server in Gerany. A siilar case is with the pair of the CSI 3 and CSI 4 servers, but in this case the difference is not due to network latency but because of other achine load that was introduced. The exact point where the change in the environent (network and achine) and thus to the response tie takes place can be better seen by analyzing the request distribution between the pairs of identical servers: the plateau in Fig. 3 represents the periods where the respective server was not preferred. So, by looking at the pair CSI3 CSI4, we see that in the beginning both servers are equally preferred, and at soe point in the iddle the CSI 4 server was ostly preferred, due to bursts of load that were introduced to CSI 3. When this external (to the digital library syste) activity resued both servers becae equally preferred again. On the other hand, the concurrent plateau in the Gerany- Italy pair show a period during which both servers were unavailable. Concluding, our experients confir that ost requests are sent to the achine that is faster to access. Also, on achines with siilar access tie, a possible external load on a achine can change the target of ost requests. 5 Conclusions In this paper, we have provided a anageent architecture and perforance fraework for easuring and onitoring digital library systes. For the perforance fraework, we defined relevant perforance paraeters which captured the tie spent in the various phases of a search request as it propagates through the syste. We used these paraeters to perfor load balancing so as to iprove the overall request response tie. In addition, we designed algoriths for estiating and adapting to tieouts in distributed searches. Our echanis design for the perforance enhanceent is fairly general and can be applied to siilar digital library systes where indexing and searching is distributed. Gerany Italy CSI 3 CSI 4 USA CSI 2 CSD CSI 00% 80% 60% 40% 20% 0% A A2 A3 A4 A5 A6 A7 A8 A9 A0 A A2 A3 A4 A5 A6 A7 A8 A9 A0 Fig. 2. Load balancing experient results 6 The set used consisted of 50 requests and was subitted twice. The requests were selected such that they siulate the Cornell University Dienst server load. More details about the experiental procedure can be found in [36].

14 32 S. Kapidakis et al.: A fraework for perforance onitoring, load balancing, adaptive tieouts and QoS in DL s Request Distribution 40 Gerany Italy CSI 3 CSI Fig. 3. Analysis of request distribution between pairs of identical servers Our perforance onitoring fraework can be used in digital library systes to iprove on the design, dynaic behavior, scaling, adaptive echaniss and quality of service provisioning. Our architecture and echaniss capture the tie spent by the various tasks in a digital library server and helps in harnessing this inforation for load balancing user queries to servers. The ephasis in this paper is on echaniss that can apply to a wide variety of digital library systes. We used an operational Dienst-based distributed digital library syste as our testbed for perforance onitoring. We gathered load data and other inforation fro this syste for our load balancing and adaptive tieout anageent. Our future work is to design echaniss for userlevel analysis (session, account, access patterns), to exaine copatibility with the Universal Measureent Architecture and to perfor ore experients and find better policies for load balancing and tieouts. Finally, we would like to experient with ore coplex perforance enhancing policies that take into account epirical observations and can do soe forecasting of load based on tie of day. Our goal is also to explore ore realistic data distributions for load balancing. Using these echaniss, sophisticated policies can be exercised. Appendix : Perforance odel for Dienst request processing Each Dienst request goes through any stages, as seen in Fig. 4, where ost of the are trivial coponents, but ay introduce significant overhead. The following is a atheatical odel of the above operation of the Dienst servers in serving a query. The odel is necessary to understand what we would like to easure and where, and what are the relationships between the odules that we easure. We odel the tie spent in iportant coponents of the Dienst server, as seen in Fig. 5. Each odule has a tie tag associated, which is nothing but the variable. The odel is illustrated below, in a step by step exaple. For ore inforation on the atheatical odel, see [36]. Subit request odules Our atheatical odel also takes into consideration the axiu tieout periods involved. With this odel we can better understand the procedures and the relationships between the coponents, prove properties about coponents and their execution, express perforance results precisely, and trace probles. Please note that the variable TX Y iplies that T is the tie spent in a odule, where X is the subscript to denote a odule and Y is the superscript to denote an interaction or a function. WWW Client HTTP WWW Server CGI CGI Stub Dienst Reote WWW/dienst Servers HTTP Dienst Server FILE I/O Docuent Database Fig. 4. The syste view of the stages of a query

15 S. Kapidakis et al.: A fraework for perforance onitoring, load balancing, adaptive tieouts and QoS in DL s 33 T nds : The tie spent by the nph-dienst-stub (nds) to process the Dienst request sent by the browser. This is the tie spent in subitting the request to DS 0 and present the response in htl forat to the browser. Note that the Web server and the Dienst server are on different achines. T nds = Tnds >DS socket 0 + T fork DS 0 + TDS first (ortds last ) Tnds >DS socket 0 : Tie spent in the socket when nphdienst-stub passes the request to DS 0. T fork DS 0 :TieDS 0 needs to fork DS that will service the request. TDS first :TietakenbyDS to subit the search and send the first results to the nds odule. Note that DS sends the results to nds as they arrive fro any of the is processes. TDS last :TietakenbyDS to subit and coplete the parallel search and send the whole result page (final result, or URL). Begin parallel search odules TDS first T E >F is 3 }} = T fork is DS +in{to search, ax{t A >B is,t C >D is 2, T fork is DS : The tie DS needs to fork all indexer_ stubs: is, is 2 and is 3. TO search : The total search tieout. T A >B is : The tie taken by is to send the query to DS 0, receive the results and forward the to DS. Siilarly for the rest of the variables. According to the probability of tieout, we have three cases: If none of the index servers respond, then the variables T A >B is, T C >D is 2, T E >F is 3 becoe infinity, and then TDS first = T fork is DS + TO search. IfoneofT X >Y is j >TO search for j =, 2, 3, then the backup server is contacted, and the analysis continues (see [36]). If all of the servers respond before the tie-out, then TDS first = T fork is +ax{t A >B is,t C >D is 2,T E >F is 3 }. DS Processing request in local site T A >B is =in{to search, (T is + T socket is >DS )} Tis socket >DS : The socket tie to transfer results fro is to DS. T is = T Http + Tnds local Tnds local :Thetiends (local) takes to send the query to DS 0 and get the response in htl forat. T Http = T req Http + T rsp Http : The HTTP request and response tie. Tnds local = T nds >DS socket 0 + T fork 2 DS 0 + T DS2 + TDS socket 2 >nds T DS2 =in{t DS2,TO reote } T DS2 : The tie the second forked Dienst server needs to process a local query. TO reote : The local or reote database search tieout. nds 5.5 Results DS 2 Web Browser Htl Docuent Dienst Request Response to Dienst 5.6 Request Local Site 5.3 Local Site 5.4 fork 2 httpd DS 0 Dienst Request 2 nds URL Request Results 3 9 at all sites fork 4 Reote Response Http Req/Resp 5. 8 B Response at A local site DS E F 00 is 5.7 C at reote D00 site Reote Response 7 at reote site 2 is is Http Req/Resp httpd Http Req/Resp httpd Reote Site Dienst Request nds Response to Dienst Request Results Local Site DS 2 DS 2 0 fork Dienst Request Fig. 5. The processes involved in a query processing nds Response to Dienst Request Local Site Results DS 30 DS 3 fork Reote Site 2

16 34 S. Kapidakis et al.: A fraework for perforance onitoring, load balancing, adaptive tieouts and QoS in DL s The analysis of T C >D is 2 and T E >F is 3 siilar to the analysis of T A >B is Processing request at reote site T C >D is 2 that follows, is above. =in{to search, (T is2 + T socket is 2 >DS )} T is2 = T Http + T reote nds T Http = T req Http + T rsp Http Tnds reote T DS2 =in{t DS2,TO reote } = T socket nds >DS2 0 + T fork DS2 0 + T DS2 + T socket DS2 >nds Processing request at reote site 2 T E >F is 3 =in{to search, (T is3 + T socket is 3 >DS )} T is3 = T Http + T reote2 nds T Http = T req Http + T rsp Http Tnds reote2 T DS3 =in{t DS3,TO reote } = T socket nds >DS3 0 + T fork DS3 0 + T DS3 + T socket DS3 >nds The sae notation is used in the user interface shown in Fig.. For exaple, the paraeters for Dienst server represent the following: Response tie of request fro Dienst server forwarded to Dienst server 3 (Tis Server >Server3 server3 )isan averageof seconds for the 5 requests generated. This tie is the su of the round-trip network delay and the reote processing tie of the request. Siilarly for reote Dienst server 2 and the local search response tie. Reote request response tie for Dienst server 3 is represented by T isserver3.theeanis6.48 seconds. This includes the reote request processing tie and the operating syste overheads. Siilarly for Dienst server 2 and (local). Reote request processing tie for Dienst server 3 is shown in the first table of the figure (Tnds local ). This is the tie taken to search the index database at the reote server T DS2 and soe overhead. Total request response tie, T nds, is the su of all delays in collecting the responses. This is 6.96 seconds for the 5 requests that were generated. References. Aral, Z., Gertner, I.: Non-intrusive and interactive profiling in Parasight. Proc. ACM/SIGPLAN PPEALS 2 30, Atkins, D.E.: The University of Michigan Digital Library Project: The Testbed. D-Lib Magazine 2(7), 996; available at 3. Biringha, W.: An agent-based architecture for digital libraries. D-Lib Magazine (), 995; available at Continuous Profiling Infrastructure.: Product Description, Digital Equipent Corp., Digital Libraries. Special issue, Counications ACM 38(4): 23 96, Fenske, D.E., Dunn, J.W.: The VARIATIONS Project at Indiana University s Music Library. D-Lib Magazine 2(6), 996; available at 7. Ferguson, D., Georgiadis, L., Nikolaou, C.: Satisfying response tie goals in transaction processing. Proceedings of the 2nd International Conf. on Parallel and Distributed Inforation Systes, IEEE Coputer Society Press, Los Alaitos, CA, USA, pp Ferguson, D., Sairaesh, J., Cieslak, R.: Black holes, sacrificial labs, and a robust approach to transaction routing. Proc. Int. Conf. on Parallel and Distributed Coputing, Fox, E.A., Eaton, J.L., McMilan, G., Kipp, N.A., Weiss, L., Arce, E., Guyer, S.: National Digital Library of theses and dissertations: a scalable and sustainable approach to unlock university resources; D-Lib Magazine 2(8), 996; available at 0. Gu, W., Eisenhauer, G., Kraeer, E., Schwan, K., Stasko, J., Vetter, J.: Falcon: On-line onitoring and steering of largescale parallel progras. GIT-CC-94-2, College of Coputing, Georgia Institute of Technology. Frew, J., Freeston, M., Kep, R.B., Sipson, J., Sith, T., Wells, A., Zeng, Q.: The Alexandria Digital Library Testbed. D-Lib Magazine 2(7), 996; available at dlib/july96/alexandria/07frew.htl 2. Gu, W., Vetter, J., Schwan, K.: An annotated bibliography of interactive progra steering. SIGPLAN Notices (ACM Special interest group on prograing languages) 29(9):40 48, Haru, S.L., Mischo, W.H., Schatz, B.R.: Federating repositories of scientific literature. D-Lib Magazine 2(7), 996; available at 4. Hearst, M.A.: Research in support of digital libraries at Xerox PARC. D-Lib Magazine 2(5), 996; available at Kapidakis, S., Sairaesh, J., Terzis, S.: A anageent architecture for easuring and onitoring the behavior of digital libraries. Proc. 2nd European Conf. on Research and Advanced Technology for Digital Libraries, Crete, Greece, Septeber 998, pp Kilpatrick, C., Schwan, K.: ChaosMON-application-specific onitoring and display of perforance inforation for parallel and distributed systes. Proc. ACM/ONR Workshop on Parallel and Distributed Debugging, May 99, pp Klavans, J.L.: New center at Colubia University for digital library research. D-Lib Magazine 2(3), 996; available at / 8. Lagoze, C., Davis, J.: Dienst: an architecture for distributed docuents libriaries. Counications ACM 38(4):47, Lagoze, C., Shaw, E., Davis, J.R., Krafft, D.B.: Dienst: Ipleentation Reference Manual, TR Cornell University, May 5th, Lange, F., Kroeger, R., Gergeleit, M.: JEWEL: design and ipleentation of a distributed easureent syste. IEEE Transactions on Parallel and Distributed Systes 3(6):657 67, Litzkow, M., Livny, M., Mutka, M.W.: Condor - a hunter of idle workstations. Proc. 8th Int. Conf. of Distributed Coputing Systes, June 988, pp Managing Application Perforance with TME0. Technical White Paper, Tivoli Systes, Miller, B.P., Callaghan, M.D., Cargille, J.M., Hollingsworth, J.H., Irvin, R.B., Karavanic, K.L., Kunchithapada, K., Newhall, T.: The paradyn parallel perforance easureent tools. IEEE Coputer 28():37 46, Nikolaou, C., Kapidakis, S., Georgianakis, G.: Towards a paneuropean scientific digital library. TR96-067, Institute of Coputer Science FORTH, May Paepcke, A.: Suary of Stanford s Digital Library Testbed Design and Status. D-Lib Magazine 2(7), 996; available at Perfstat.: Product description, Instruental Inc., POLYCENTER Perforance Advisor for Unix.: Product Description, Coputer Associates, Digital Equipent Corp., 996

17 S. Kapidakis et al.: A fraework for perforance onitoring, load balancing, adaptive tieouts and QoS in DL s Reed, D.A., Olson, R.D., Aydt, R.A., Madhyastha, T., Birkett, T., Jensen, D.W., Nazief, B.A.A., Totty, B.K.: Scalable perforance environents for parallel systes. In: Proc. 6th Distributed Meory Coputing Conference, IEEE Coputer Society Press, 99, pp Reed, D.A., Noe, R.J., Shields, K.A., Schwartz, B.W.: An overview of the Pablo Perforance Analysis Environent. Dept. Coputer Science, University of Illinois, Urbana, IL, Noveber Resource Manageent Facility.: IBM Corp., June Snodgrass, R.: A relational approach to onitoring coplex systes. ACM Transaction on Coputer Systes 6(2):57 95, Rusbridge, C.: The UK Electronic Libraries Prograe. D- Lib Magazine (6), 995; available at dlib/deceber95/briefings/2uk.htl 33. Sairaesh, J., Kapidakis, S., Nikolaou, C.: Architectures for QoS based retrieval in digital libraries. Workshop on Networked Inforation Retrieval, SIGIR Schroeder, B.A.: On-Line onitoring: a tutorial, IEEE Coputer 28(6):72 78, Tantawi, A.N., Towsley, D.: Optial static load balancing in distributed coputer systes. J ACM 32(2): , Terzis, S.: Perforance onitoring in digital library systes. Master thesis, TR97-020, Institute of Coputer Science FORTH, October UMA Technical Inforation.: Perforance Manageent Working Group, March A+UMA Perforance Data Manager Technical Overview, Adahl Corp., October Witten, I.H., Cunningha, S.J., Apperley, M.D.: The New Zealand Digital Library Project. D-Lib Magazine 2(0), 996; available at newzealand/witten.htl 40. Yu, P.S., Leff, A., Lee, Y.H.: On robust transaction routing and load sharing. ACM Transactions on Database Systes 6(3):476 52, 99