irods and Metadata survey Version 0.1 Date March Abhijeet Kodgire 25th

Transcription

1 irods and Metadata survey Version 0.1 Date 25th March Purpose Survey of Status Complete Author Abhijeet Kodgire

2 Table of Contents 1 Abstract Categories and Subject Descriptors Keywords Introduction Architecture ) Virtualization ) Metadata Catalouge ) Rule Engine ) Execution Engine ) Scheduler ) Messaging System Rule Oriented Programming ) State Information ) Rule System ) The Rule Engine Micro Services Future Work References... 8

3 1 Abstract In this paper, we present short summary of how irods are implemented and used for big data science purpose. 2 Categories and Subject Descriptors [Storage Systems, IRODS, Distributed Systems] 3 Keywords irods, Metadata and Storage Systems. 4 Introduction irods, which stands for I Rule Oriented Data System, is a project for building next generation data management cyber infrastructure. One of the main ideas behind irods is to provide a system that enables a flexible, adaptive, customizable data management architecture. The irods is software infrastructure that can be used to implement any desired data management application ranging from data grids to a digital library. Also it can be thought of as middleware software which controlled collection of distributed object across multiple storage locations. irods project was funded by NSF and developed on lesson learned from first generation of data management cyber infrastructure SRB. irods belongs to class of adaptive middleware. The adaptive middleware architecture (AMA for short) provides a means for adapting the middleware to meet the needs of the end user community without making any programming changes. One can view the AMA middleware as a glass box where users can see how the system works and can tweak the controls to meet their demands. The irods system expresses management policies as computer actionable rules and management procedures as remotely executable micro services. The rules control the execution of micro services and the state information generated by the services is stored in a metadata catalog. The rules and micro services are written to meet data management needed for a variety of data management services. The irods framework is extensible and it allows users to add rules and micro services dynamically

4 5 Architecture Figure 1 irods Architecture irods software belongs to a class of middleware which we term adaptive middleware. The adaptive middleware architecture provides a means for adapting the middleware to meet the needs of the end user community without making any (if at all) programming changes. There are multiple ways for achieving adaptive middleware architecture. irods uses a particular methodology named Rule Oriented Programming or ROP for short. Using ROP data management functions can be customized in an easy and declarative fashion. This can be done by creating the processes that are being performed in irods as rules which control the operations that are done by invoked tasks. These small operations are called micro services which are implemented as C-functions in irods system, which are called while executing the rule body. Major features of the irods architecture include the following: 1) Data grid architecture based on a client/server model and distributed storage and compute resources. 2) A database system for maintaining the attributes and states of data and operations. 3) A rule system for enforcing and executing adaptive rules

5 The irods Server software is installed on each storage system where data will be stored. The remote storage system is normally defined by an IP (Internet Protocol) network addresss. The irods Server translates operations into the protocol understood by the remote storage system. In addition, a Rule Engine is also sitting at each storage location. The Rule Engine controls operations performed at that site. Above figure illustrates the components of the irods system, including a Client for accessing the Data Grid, Data Grid Servers installed at each storage system, a Rule Engine installed at each storage location, the icat Metadata Catalog that stores the persistent state information, and a Rule Base that holds the Rules. Figure 3 Components of irod System

6 5.1) Virtualization Virtualizing the data is the underlying idea in the irods data grid system. Instead of a physical naming, irods adopts a virtual for every entity that interface with user or application. The mapping from the logical name to physical name is stored in the Metadata Catalog and the mapping is resolved at run time by the Virtualization sub-system. 5.2) Metadata Catalouge The icat catalog provides persistent storage for the state of the irods system. This catalogue also includes mapping related to virtualization and metadata of both system and users. 5.3) Rule Engine The Rule Engine executes rules and enforcing policies of the data being managed by irods. Many irods APIs automatically call the rule engine so that any required policies can be automatically checked and enforced. 5.4) Execution Engine The execution engine executes the micro-services. Every irods node (server) is capable of executing micro-services. The rule syntax provides means for remotely executing micro-services. Hence one can have a chain of micro-services performed at different sites so as to cover entire distributed storage connected together. 5.5) Scheduler The delayed execution and periodic execution are controlled by a built-in scheduler in the irods system. The icat stores information about the schedule of activities and executes them through the execution engine as and when they need to be performed. 5.6) Messaging System In a future release, irod team plan to incorporate a messaging system as part of irods. This would be helpful for inter-micro-service communication. This will also help when parallel processing is supported allowing them to communicate among them. The messaging system will provide a true distributed workflow environment for data processing. 6 Rule Oriented Programming In ROP, controlling the functionality is more with the users than with system and application developers. Hence, any change to a particular process or policy can be easily constructed by the user, and then tested and deployed without the aid of system and application developers. The first three components of a rule are an action name, a testable condition, and a chained workflow. Workflow consisting of actions and micro services and each rule has a priority associated with it. The fourth component of a rule is a set of recovery micro services.

7 If a micro service fails, these are the chain of micro services to be executed in order to restore the system to a consistent state. 6.1) State Information A session is defined as a single irods server execution. A session starts when a client connects to a server and ends when the client disconnects. During the session the actions and the micro services act on two distinct state information spaces. 1) Persistent State Information (denoted by the symbol #) is defined by the attributes and schema that are stored in the icat catalog 2) Session State Information (denoted by the symbol $) is temporary information that is stored in the memory as a C-structure temporary when the Actions are being performed. The persistent state information is the information available across sessions and is present in the metadata catalog once the action is performed. The session state information since its temporary is not available across sessions. Micro services uses session state information to communicate with each other without going to the database to retrieve every attribute value. The persistent variables have a logical name space defined in the icat catalog and these names are mapped to columns in the database. 6.2) Rule System Two types of irods rules. 1) System level rules: These rules are invoked on the servers internally to enforce data management policies. Examples include enforcement of authenticity, integrity etc. 2) User level rules: The rule engine can be invoked externally through a client through the irule command. These rules are generally workflow type rules which allow users to request the server to perform a sequence of operations Delayed Execution Service allows Rules/Micro-services to be queued and executed at a later time by the Rule Execution Server. Examples of Micro-services that are suitable for delayed execution are postprocessing operations such as checksumming, replication, and metadata extraction. 6.3) The Rule Engine At the centre of the irods system is the rule engine which acts as interpreter for the rules. The rule engine reads the rule base and decides which rules to apply. The rule engine can be invoked using any server side function call using the C API. First the rule selects all rules that have the same action name as the inaction string. The rule engine tries to execute the rules in the order they were added to the rule base and goes from one rule to another in order. While executing the chain of Micro-services/Actions, if any one of them fails, then the Rule Engine starts recovery procedure.

8 7 Micro Services As name suggest these services are small. Micro Services are well defined functions that perform simple tasks. Micro services are developed and are made available by compiling them into the irods server code. Users can use these micro services to get the control they need on performing a macro level function or an Action. Using priorities and validation conditions at run-time, the system chooses the 'best' Micro-service chain to be executed. Micro Services can be categorized as follows. 1) Core Micro Services: Functions for Rule Engine control, Workflow creation, Low-level Data Object Low-level manipulation. 2) icat Services: Functions for manipulating system metadata and for interacting with the icat metadata Catalog. 3) Framework Services: Functions for Rule-oriented remote database access, supporting Userdefined services. 4) Module Micro Services: Sets of functions developed for a certain domain users like XML manipulation. Each category can have multiple micro services defined in it. Input arguments to micro services can take two forms Literal and Variable. If the input argument doesn t start with a special symbol (#,$,*), it is a character string input. If an argument starts with the * character, it is considered as a variable argument which can be used as both input and output arguments. 8 Future Work As future work, irod team is planning to incorporate a messaging system as part of irods. Also irod team intend to look at irods and try to make it more generic and make it faster. 9 References [1] Integrated Rule-Oriented Data-management System, a community-driven, open source, data grid software solution DOI =