GRIDB: A SCALABLE DISTRIBUTED DATABASE SHARING SYSTEM FOR GRID ENVIRONMENTS *

GRIDB: A SCALABLE DISTRIBUTED DATABASE SHARING SYSTEM FOR GRID ENVIRONMENTS * Maha Abdallah Lynda Temal LIP6, Paris 6 University 8, rue du Capitaine Scott 75015 Paris, France [abdallah, temal]@poleia.lip6.fr ABSTRACT Grid computing technologies provide mechanisms that allow the seamless sharing of distributed and heterogeneous resources, such as supercomputers, storage systems, and data sources, in a large-scale and dynamic environment. Research works in data grids have generally focused on applications where data resources are stored in files. However, many scientific applications are highly dependent on database management systems for organizing and storing their data. This stresses the need for developing techniques that enable the sharing of and access to massively distributed, heterogeneous, and autonomous databases created and managed independently by several research institutions. In this paper, we address this issue and propose GriDB, a database sharing technique that has the following desirable characteristics: (1) it supports data indexing at a finer granularity than the file level, enabling contentbased search and access; (2) it allows data sharing without a globally shared schema; (3) it is distinguished by its peer-to-peer flavor, making it highly scalable, autoconfigurable, and fully decentralized with no need to centralized or dedicated servers. KEY WORDS Distributed Databases, Distributed Indexing, Peer-to-Peer and Grid Computing 1. Introduction Grid computing allows the seamless sharing of heterogeneous resources, such as computing power, data sources, and other on-line tools, securely in a large-scale and dynamic setting [1, 2]. Data grids focus on scientific data sharing among research communities willing to collaborate across institutional and geographic boundaries by leveraging knowledge, research results and data generated by multiple research groups. A wide range of This work has been partially funded by the SemWeb project of the ACI Masses de Données program. scientific applications, such as life and environmental sciences, are highly dependent on database management systems for organizing and storing their data. Surprisingly, however, research works in data grids have generally focused on applications where data are stored in files [2], and provide solutions for only file-level sharing, thus preventing content-based search central to database applications. A major difficulty in database sharing, as compared to file sharing, stems from the inherent semantic and structural heterogeneity between databases created and managed by a set of different researchers and organizations. Although these issues have been the subject of much attention by the database research community over the past decades, the proposed solutions do not consider the scale and dynamism properties intrinsic to grid environments. Indeed, distributed database technologies [3] rely on a predetermined global schema maintained in a centralized manner. While this might indeed make sense in a static environment with a limited number of participating nodes (the largest database systems scale up to at most a few hundred nodes), maintaining a global view in a large-scale distributed system with frequent dynamic change is simply infeasible. Enabling the sharing of databases without a globally shared schema is thus one of the major issues and challenging tasks towards the intended goal. In the light of the above observations, we believe that devising scalable solutions that enable the seamless sharing of massively distributed, heterogeneous, and autonomous databases created and managed by several independent research institutions arises as a critical issue for the advancement of next generation scientific applications. This argument is further strengthened by the fact that, like distributed databases, today's grids rely on centralized or hierarchical approaches to data (resource) discovery, and should be reconsidered in order to meet the ever-growing scalability requirement of complex grid applications. Peer-to-peer (P2P) computing is emerging as a key paradigm for structuring large-scale distributed systems. P2P systems make it possible to harness resources distributed world-wide in a cost-effective manner by

having all nodes participating in the system (called peers) play an equal role and act as both clients and servers. The key advantages of P2P systems are scalability, faulttolerance, absence of central control, and selforganization, allowing peers to join and leave the system at any time. P2P systems have been popularized by file-sharing applications, such as Napster [4], Gnutella [5], and KaZaA [6]. A fundamental issue in these systems is locating resources (files), and systems have been classified into unstructured and structured systems with respect to the resource location technique they employ. In unstructured systems, such as Napster and Gnutella, nodes have no knowledge about resources that other nodes maintain. In Napster for instance, a global index of all resources available in the system is maintained at a central server. To locate a resource, a node sends a request to the index server and gets the IP address of the node storing the requested resource. Subsequent downloads are performed directly between nodes. This centralized approach to locating resources introduces a scalability problem in the index server and makes it a single point of failure. To avoid these problems, another extreme approach is followed by Gnutella in which no indexing mechanism is supported. Resource location is then achieved by flooding requests to all neighboring nodes within a certain scope. Flooding on every node's request clearly does not scale in terms of communication cost and worse, might not find a resource that is actually in the system due to scope limits. These problems have motivated the research community to focus on the design of structured P2P systems, such as CAN [7], Chord [8], D2B [9], Tapestry [10], and Pastry [11]. In structured systems, a global index of all available resources is maintained and carefully distributed among all participating nodes: every resource is associated with a unique key, and every node in the system is responsible of a set of keys; based on its key, a resource index entry is routed to (and stored at) the node in the system responsible of that key; key-based queries are then directly routed to that same node. Simply stated, structured P2P systems allow nodes to publish and locate resources based on their keys, thereby providing a hash-table-like functionality over multiple distributed nodes; this is why they are commonly referred to as Distributed Hash Tables (DHTs). The high scalability of DHT systems is due to the scalability of the mechanism by which a key-based request is routed to the node in possession of the requested key, and that without global knowledge of the mapping between keys and nodes. This is achieved by having every node keep information about only a set of neighboring nodes. Routing is then performed by forwarding the request to the neighbor who is "closest" to the destination node. In most DHT systems, this routing is achieved in only O(log N) overlay hops in a system with N participating nodes. Although DHT systems meet high scalability requirements necessary to grid applications, they however only provide file-level sharing of data, assuming a requested file's key is known in advance. This raises an important question on whether they can be effectively exploited in a database sharing, where content-based data access is a crucial requirement. In this paper, we answer this question positively and propose GriDB, a flexible database sharing mechanism that extends DHT technology to provide high scalability, while enabling fine-grain data indexing to support SQLlike content-based queries, and that without a globally shared database schema. The rest of the paper is organized as follows. Section 2 discusses related work. Section 3 details the underlying system model, and gives some background on sharing databases without a shared schema. Section 4 describes our DHT-based indexing mechanism, as well as its use to search and access database through SQL-like queries. Section 5 concludes this work and points out some of our future perspectives. 2. Related Work The problem of large-scale database sharing has not been studied until very recently. With the emergence of P2P systems as a powerful paradigm for large-scale data sharing, P2P databases are becoming an important area of investigation. [12] addresses data placement issues in a P2P setting. [13, 14] present a rather orthogonal study to the present work and study semantic data integration between heterogeneous and autonomous database management systems. In [15], the authors discuss a P2Pbased scheme for caching OLAP queries. [16] proposes a DHT-based caching scheme to store range query results. However, they assume a unique database located at known node. More relevant to our work are the studies presented in [17, 18]. In [17], the authors propose a DHT-based relational query processor over widely distributed relational data stores. However, the underlying assumption is a common standard schema adopted by all data stores. Furthermore, indexing is achieved at the tuple level. While this assumption makes indeed sense in applications with very few sharable data items, such as resource discovery services where a resource is described by a single data item, distributed indexing of every tuple in data-intensive scientific applications with TeraBytes of sharable data will have a prohibitive communication and index maintenance cost, and thus does not scale in our context. [18] is probably the first reported work on a database sharing system that facilitates data sharing without predetermined shared schema, and combines mobile agents and P2P technologies for large-scale resource sharing of raw data as well as processed information. The assumption underlying PeerDB is an unstructured overlay

network with no indexing mechanism. Thus, data discovery is achieved by flooding on requests to all nodes, and thus inherits all the disadvantages of Gnutella-like resource discovery systems. Our research builds on PeerDB by extending it with a DHT-based indexing mechanism. This work can be seen as a first step towards efficient, highly scalable, and flexible data sharing in massively distributed heterogeneous databases, and constitutes a candidate approach to data discovery for collaborative dataintensive grid applications. 3. System Model As an example application, we consider the case of biomedical research institutions distributed world-wide, where each has independently created, managed, and populated a local database with data generated by local research experiments. There is no doubt that the greatest future advances in Science will be made by combining knowledge, research results and data from all these independent distributed databases to allow collaborative work across institutional boundaries [19]. Our goal is to enable users to query and access all these data as if they were stored in a single database. We consider relational database management systems (RDBMS) in which data are stored in relations, where each row (called tuple) represents a data item, and each column is an attribute describing those items. Following the model proposed in [18], we also assume that the different databases have been created without following a predetermined common schema. This suggests that, even though users belong to the same research community, they might have followed different design approaches to representing their data. For instance, different scientists might employ different relation names to store semantically similar data. Also, attributes describing the same semantic property of data items might have been named differently. As an example, a Gene Therapy researcher can call the relation storing data on the studied sicknesses Sickness, while another can call it. Similarly, the attribute identifying a sickness tuple may be called sickid by the former and simply ID by the latter. To solve this discrepancy in data definition and representation, we exploit traditional information retrieval (IR) techniques for text search based on keyword lists describing shared documents [20]. In database management systems, data is traditionally associated with metadata. Metadata is information that describes data, and allows to record information about the creation, ownership, transformation, and semantic meaning of data. Based on this facility, and for each relation, we maintain metadata information that consists of a set of keywords describing the relation name/attributes names, and serving as synonyms. Given that collaborating scientists usually belong to the same or a close research community, they PEERS Peer1 Peer2 Peer3 Peer4 Relation Schema Sickness - sickid - description - treatment HumanSick - ID - desc - therapy - symptoms - therapy Disease - sickid - sickname - sicksymp etc., Keywords - ID, identifier, key - description - therapy Sickness, Human - key, identifier - description, symptoms - treatment, identifier, ID, key - symptoms - treatment, Sickness - ID, identifier, key - symptoms Figure 1. Keywords for relation names/attribute names. are very likely to give common keywords to describe similar data. Figure 1 illustrates this approach for a relation holding data about the different sicknesses studied by four biomedical research labs (i.e., four peers). We see that although relation or attribute names vary from one peer to another, they are very likely to give at least one common keyword to describe the relation content. The same holds for the relation's attributes. Note that Keywords need to be defined only for the relations that a peer is willing to share. We call these sharable data. Now, consider the following SQL query initiated by Peer1 asking for all the suggested treatments for Cancer. SELECT FROM WHERE treatment Sickness name = Cancer ; The list of keywords for the relation Sickness at Peer1 includes, which is also a keyword for Peer3's relation and Peer4's Disease relation. Similarly, Peer2's HumanSick relation has a corresponding Sickness Keyword. Thus, all these relation are good candidates for answering Peer1's query. Regarding the query's attributes, we note that both (or one of their associated keywords) appear at Peer2 and Peer 3. However, Peer4 does not have either the treatment attribute or its associated therapy keyword. Therefore, the query should not be forwarded to Peer4 for processing,

given that it does not possess the required treatment information. This constitutes a major advantage of our approach over the one presented in [18] in which a query is flooded to all peers in the network so as to determine interesting peers matching the required information. Indeed, through an elegant combination of distributed indexing and metadata, GriDB is able to achieve query routing directly to the interesting peers as described in the next section. 4. Fine-grain P2P Indexing As stated before, GriDB is a DHT-based indexing mechanism that allows fast and direct access to the requested information in a highly scalable manner. Furthermore, the system is very dynamic in the sense that nodes can join and leave the system at any time. 4.1 Index Construction In our database context, the distributed index is constructed based on the keywords provided for the sharable relations and attributes in the following way. For every relation a peer is willing to share, the peer generates an index entry for every keyword associated with that relation conforming to the following schema: (Keyword, IPaddress, RelationSchema), where IPaddress designates the physical location of the publishing peer, and RelationSchema specifies the original relation name and attributes that keyword is associated with. The Keyword field is then used as the index key for the DHT. This means that the generated index entry will routed and stored at the peer responsible of that key, which ensures that index entries having the same keyword (i.e., published by different peers) will be stored at the node. Examples of relation index entries are shown in Figure 2. Similarly to the relation keywords, an index entry is created for every keyword associated with the relation's attributes. However, unlike the relation case, each attribute keyword is concatenated with each keyword associated with the relation. This is indeed important in order to timestamp an attribute with its origin relation, thus preventing ambiguity in attributes belonging to two or more different relations. The generated index entry conforms to the following schema: (Rel_Keyword Att_Keyword, IPaddress), where the Rel_Keyword Att_Keyword field is used as the index key for the DHT. Again, this ensures that the index entries of attributes described by a common keyword and belonging to relations having also common keywords are hosted by the same node. Figure 3 shows examples of attribute index entries. 4.2 Data Search and Query Execution Before describing query execution, we first show the design of a GriDB node in Figure 4. GriDB is introduced keyword Sickness IPaddress, RelationSchema @Peer1, Sickness(sickID, name, description, Treatment) @Peer2, HumanSick(ID, name, desc, therapy) @Peer4, Sickness(sickID, name, description) keyword IPaddress, RelationSchema @Peer1, Sickness(sickID, name, description, Treatment) @Peer3, (name, symptoms, therapy) @Peer4, Sickness(sickID, name, description) Figure 2. Index entries for relation keywords Sickness and. keyword Sickness name Peers @Peer1 @Peer2 @Peer4 keyword Peers Sickness treatment @Peer1 @Peer2 @Peer3 Figure 3. Index entries for attribute keywords name and treatment. as a software layer on top of DBMSs for two major purposes. First, it preserves node autonomy by avoiding any change to the DBMS kernel. Second, it maps users' queries to local and remote queries by exploiting the metadata information and performing search on the DHT using the provided keywords, thus relieving the user from dealing with these issues and providing global data access transparency. When a user submits a query, the GriDB layer intercepts the user's query, constructs the keyword lists for all the query's relations and attributes using the relation's metadata, and then submits the original query to the local DBMS. At the same time, the GriDB layer starts the distributed search process to identify peers in the system susceptible to answer the query and provide relevant information. This is achieved by hashing each of the identified keywords and collecting its associated index entries previously published by peers having that keyword in their database. As stated in the previous section, the search for a single keyword results in a direct access to the node hosting that keyword's index entries, thus identifying at once all peers in possession of the keyword. When the index entries of all the query keywords are collected, peers in possession of all the relations and attributes needed for the execution of the query are identified. Once this is done, the GriDB layer reformulates the query by mapping relation names and attribute names with the corresponding names employed by the destination peer. Such information is available in the relation index entries showing the original relation schema at each peer, and that independently from the keyword used to publish and locate the relation.

Database Metadata Data Query GriDB Layer Local Query DBMS Remote Queries Figure 4. A GriDB node. 5. Conclusions and Future Work DHT In this paper, we have presented GriDB, a data indexing and search mechanism that allows the sharing of massively distributed, heterogeneous, and autonomous databases for next-generation scientific applications. GriDB can be seen as a first step towards database globalization, thus enabling databases to become an important part of large-scale collaborative environments. GriDB has several desirable features, making it a good candidate for data discovery in data grid applications. By an elegant combination of several technologies, GriDB provides fine-grain indexing, search, and access thus enabling expressive SQL-like queries over databases. It allows database sharing with no need for a standard shared schema, making it very flexible for the integration/removal of databases into/from the system. Finally, by exploiting recent P2P technological advances, it is provides a fully distributed indexing mechanism based on distributed hashing techniques (DHTs), making extremely efficient, highly scalable and fault-tolerant. Extensive performance evaluation of our prototype implementation is currently underway. This will allow studying the system behavior with respect to the network characteristics as well as the number of nodes participating in the system. We finally plan to study the use of Globus Grid Security Infrastructure (GSI) [2] for performing database connections. References [1] M. Baker, R. Buyya, and D. Laforenza. Grid and Grid technologies for wide-area distributed computing. International Journal of Software: Practice and Experience (SPE), Volume 32, Issue 15, Wiley Press, USA, December 2002. [2] I. Foster and C. Kesselman Eds. The Grid: Blueprint for a new computing infrastructure. Morgan Kaufman, 1999. [3] M. T. Ozsu and P. Valduriez. Principles of Distributed Database Systems, Second Edition. Prentice Hall, 1999. [4] Napster. http://www.napster.com/. [5] Gnutella. http://gnutella.wego.com/. [6] KaZaA. http://www.kazaa.com/. [7] S. Ratnasamy, P. Francis, M. Handley, R. Karp, and S. Shenker. A Scalable Content-Addressable Network. In Proc. of the ACM SIGCOMM 2001 Conference, August 2001. [8] I. Stoica, R. Morris, D. Karger, F. Kaashoek, and H. Balakrishnan. Chord: A scalable peer-to-peer lookup service for internet applications. In Proc. of the ACM SIGCOMM 2001Conference, August 2001. [9] P. Fraigniaud and P. Gauron. An Overview of the Content-Addressable Network D2B. In Proc. of the Twenty-Second ACM Symposium on Principles of Distributed Computing (PODC), July 2003. [10] Y. B. Zhao, J. Kubiatowicz, and A. Joseph. Tapestry: An infrastructure for fault-tolerant wide-area location and routing. Technical Report UCB/CSD-01-1141, University of California at Berkeley, 2001. [11] P. Druschel and A. Rowstron. Pastry: Scalable, distributed object location and routing for large-scale peer-to-peer systems. In Proceedings of the 18th IFIP/ACM International Conference on Distributed Systems Platforms (Middleware), November 2001. [12] S. D. Gribble, A. Y. Halevy, Z. G. Ives, M. Rodrig, and D. Suciu. What can database do for peer-to-peer? In Proc. of the fourth International Workshop on the Web and Databases (WebDB), May 2001. [13] A. Y. Halevy, Z. G. Ives, D. Suciu, and I. Tatarinov. Schema mediation in peer data management systems. In Proc. of the 19th International Conference on Data Engineering (ICDE), March 2003. [14] P. Bernstein, F. Giunchiglia, A. Kementsietsidis, J. Mylopoulos, L. Serafini, and I. Zaihrayeu. Data management for peer-to-peer computing: A vision. In Proc. of the fifth International Workshop on the Web and Databases (WebDB), June 2002. [15] P. Kalnis, W. S. Ng, B. C. Ooi, D. Papadias, and K. L. Tan. An adaptive peer-to-peer network for distributed caching of OLAP results. In Proc. of the ACM SIGMOD International Conference on Management of Data, June 2002. [16] A. Gupta, D. Agrawal, and A. El-Abbadi. Approximate Range Selection Queries in Peer-to- Peer Systems. In Proc. of the Biennial Conference on Innovative Data Systems Research (CIDR), January 2003. [17] R. Huebsch, J.M. Hellerstein, N. Lanham, B. T. Loo, S. Shenker, and I. Stoica. Querying the Internet with PIER. In Proceedings of the 29th International

Conference on Very Large Data Bases (VLDB), September 2003. [18] W. Ng, B. Chin Ooi, K. Tan, and A. Zhou. PeerDB: A P2P-based System for Distributed Data Sharing. In Proc. of the 19th International Conference on Data Engineering (ICDE), March 2003. [19] R. Buyya, K. Branson, J. Giddy, and D. Abramson. The Virtual Laboratory: Enabling Molecular Modeling for Drug Design on the World Wide Grid. Journal of Concurrency and Computation: Practice and Experience (CCPE), Volume 15, Issue 1, Wiley Press, USA, January 2003. [20] M. Berry, Z. Drmac, and E. Jessup. Matrices, Vector Spaces, and Information Retrieval. SIAM Review, Volume 41, Issue 2, 1999.