glibrary: Digital Asset Management System for the Grid

Transcription

1 glibrary: Digital Asset Management System for the Grid Antonio Calanducci INFN Catania EGEE User Forum Manchester, 09 th -11 th May EGEE and glite are registered trademarks

2 Outline Motivations and objectives Features Usage scenarios Architecture and implementation Current status Conclusions and references 2

3 Motivations Grid infrastructures provide huge amount of distributed storage resources to save user s data But finding a specific file in this enormous storage space could be like looking for a needle in a haystack weakness of additional information describing file contents Existing grid services to approach the problem: File Catalogues: organize files in hierarchical structures, but no file description Metadata Catalogues: useful to add metadata to files, but quite complex to use, not suitable for final unexperienced users 3

4 What is glibrary Enabling Grids for E-sciencE glibrary challenge is to offer a multiplatform, flexible, secure and intuitive system to handle digital assets on a Grid Infrastructure. By Digital Asset, we mean any kind of content and/or media represented as a computer file. Examples: Images Videos Presentations Office documents s, web pages Newsletters, brochures, bulletins, sheets, templates Receipts, e-books... (only the imagination can make a limit) It allows to store, organize, search and retrieve those assets on a Grid environment. 4

5 Store assets on the Grid User s local assets are uploaded to one or more (as replicas) Storage Systems the user is authorized on Uploads are managed through Java Applets: a direct GSIFTP copy is done from the local file to the chosen Storage Element File already on the Grid can be managed by glibrary too a LFC File Catalogue browser is integrated to select existing grid files. 5

6 Abstract - glibrary challenge is to offer an extensible, robust, secure and easy-to-use system to handle digital assets stored as files in a Grid infrastructure. It offers a intuitive web interface to browse the available entries, and thanks to its powerful set of attribute Enabling filters, Grids finding for an E-sciencE asset is just a matter of seconds. The assets are hierarchically organized by types. Each type has its own set of attributes, allowing the user to customize his/her search applying different filters. Once a user has found the asset, he/she will be able to retrieve its file replica from one of the Grid Storage Elements into his/her own desktop with just a mouse click. All entries are organized according to their type: a list of A fine-grained specific authorization attributes mechanism is implemented to describe each kind of inside the system: each asset has a set of ACLs that restrict its usage. assets to be managed by the system; glibrary is built on top of the grid services of the EGEE glite middleware. hierarchical (child type shares parent s attributes) by the Grid infrastructure. I. INTRODUCTION defined by the glibrary administrators In every Grid Infrastructure a huge amount of queried distributed by storage users resources is available to save user's looking for a needle in a haystack, because of the Assets weakness can of be additional organized information describing also file by category: data. However, finding a specific file is usually like contents. File Catalogs offer virtual file systems distributed among Grid Storage Servers, that can help to organize data Group in hierarchical together structures, related grouping them assets according to of different types; some user's criteria, but they do not provide a way to describe file contents. On the other hand, Grid Metadata Useful Services also can be to used define to attach additional subset information of to assets belonging to the files, but they are usually quite complex to use, and then same not type suitable for novice users. glibrary challenge is to offer an extensible, robust, secure and easy-to-use system to handle digital assets Multiple category assignment per asset TABLE I widespread stored on a distributed Grid infrastructure. This goal can be achieved exploiting a serie of Grid services together with a proper business logic, and providing an intuitive front-end, accessible from everywhere and anytime. Users do not have to care about the complexity of the underlying systems and the geographical location of their data and they can consider the available Grid storage as a huge virtual disk. The remainder of this paper is organized as follows. Section II describes features and benefits offered by the system. The background of Grid technologies adopted is discussed in Section III in order to understand the architecture. Implementation details are presented in sections IV and V. Conclusions are drawn in Section VI. II. FEATURES glibrary can be used to store, organize, search and retrieve any kind of digital assets represented as files in a Grid environment. Consequently, it can be useful for different users that need a secure way to save and share their assets. For instance, home users could store all their songs, photos, movies, documents and any type of files and organize them defining for each type which fields are relevant for their future searches. Users belonging to a company or organization can also store and share with other authorized users their specific assets, such as invoices, presentations, layouts, medical images, etc. Both communities can define permissions on their assets inside glibrary in order to grant or deny access to given users, groups or even whole organizations, exploiting the underlying authentication and authorization system offered Assets are saved on the grid and can be encrypted and replicated on several storage servers, assuring maximum security and high availability to the users' data. All entries in glibrary are organized according to their type: a list of specific attributes to describe each kind of asset to be managed by the system. These are the same attributes that can be queried by users. Table I presents an example of types and their attributes that can be used for a digital library of multimedia contents. Each type can have multiple subtypes with additional attributes and all types share a common attribute list (root type), that is fixed by design. Before users can start uploading assets, a hierarchy of types has to be defined by the library administrator. Figure 1 shows an example of hierarchical type structure. EXAMPLE OF TYPES AND ATTRIBUTES LIST Type Audio Music Presentation Training (Root) Organize assets Attributes list Format, Bitrate, Samplerate, Time (Format, Bitrate, Samplerate, Time), Name, Artist, Album, Genre, Tracknumber, Year, Artwork, Lyric, Rating Format, NumOfPages (Format, NumOfPages), Title, Runtime, Speaker, Author, Subject, Event, Date, Type FileName, SubmissionDate, Description, Keywords, LastModificationDate, Size 6

7 Search assets Enabling Grids for E-sciencE Assets are browsed selecting a type (or category) and selecting one or more filters: defined list of type attributes used to narrow the result set Filter application is context-sensitive: the selection of a filter value dynamically influences subsequent filter values ( à la itunes browser) Classic search available too 7

8 User is presented with a list of replicas Download from the chosen storage element is matter of a mouse click Transfer handled over GridFTP with a Java Applet Retrieve assets from the Grid 8

9 Features Implemented as Web 2.0 application AJAX and Javascript are strongly used to offer a desktop like user experience 9

10 Security As a grid application, glibrary inherits all the security features coming from the underlying technology: Authentication and authorization is done through VOMS enabled X.509 proxies A login applet is in charge of the proxy creation starting from the personal X.509 certificate The generated proxy is moved over HTTPS to the web application server. (Next release will make use of the pure Java VOMS APIs) Physical transfer are all authorized with the saved proxy 10

11 User management VOMS attributes are used to distinguish users and assign the right privileges to them. 3 kinds of users: glibrarymanager: (s)he can define new content type and allows a generic VO user to become glibrarysubmitter glibrarysubmitters: they can add new entries and define access rights on the entries they create. Fine-grained permission (reading, writing, listing) settings on each entry: whole VO members, VO groups, list of DNs generic VO users: browse and make queries (on entries they have access to) 11

12 Usage scenarios Useful for different users that need to share different kind of assets. Example: home users: songs, photos, movies, office documents, etc.. enterprise/company users: presentations, layouts, invoices, images, etc Both communities can define permissions on their assets inside glibrary in order to grant or deny access to given users, groups and even whole organizations, exploiting the underlying authorization system offered by the grid infrastructure. glibrary is envisaged as document repository by several projects submitted to the Seventh Framework Program (FP7) calls of European Commission 12

13 Enabling Grids for E-sciencE Architecture Built on top of glite Data Management Services: any SRM Storage Element LFC File Catalog AMGA Metadata Services The AMGA Metadata Catalogue is used to archive and organize assets metadata and to answer users queries built on-the-fly by browsing interface PHP APIs (fully developed by our team in Catania) are used to interact with AMGA. Authentication is done through the on-the-client created user proxy AMGA groups and ACLs are used to restrict access on metadata, allowing fine-grained authorization capabilities LFC File Catalogue interactions: a PHP-LFC C APIs wrapper has been employed to browse LFC virtual filesystem and retrieve replicas (SURL) from Logical File Names (LFN) 13

14 Implementation details Java Applets use Java COGKit to manage proxy and GridFTP transfers: data move only between the client and the destination Storage Element (without using the application web server as a temporary buffer) Interactions with the Storage Elements are done through SRM interfaces (using GFAL libraries): the Information System is contacted to generate a new SURL a TURLfromSURL call is done to the SE s SRM interface when the TURL is ready, set_xfer_running is called the physical transfer starts over GSIFTP a set_trasfer_done request is done to finalize the transfer a PHP-GFAL wrapper has been developed to accomplish the previous operations 14

15 Current status and roadmap Deployed on the GILDA t-infrastructure Web front-end available at Application server runs Apache and PHP5 on a Fedora Core 5 machine. Successfully set up with several glite client libraries and tools: LFC, GFAL APIs, VOMS, lcg_utils, and lfc clients To Do: Category management Permission handling on types and categories Administration (glibrarymanager) panel Improving error handling Integration with Secure Storage to support Asset Encryption/ Decryption 15

16 Browsing screenshot 16

17 Entry detail screenshot 17

18 Upload screenshot 18

19 Conclusions glibrary challenge is to offer an flexible, multiplatform, secure and easy-to-use system to handle digital assets stored as files on a distributed Grid infrastructure flexible: it allows to handle any kind of file types, user defined multiplatform: implemented as a web application with Java Applets can be used on any platforms secure: fine grained permissions can be set for assets and their attributes easy-to-use: its intuitive web interface, with à la itunes browser, allows to find the right asset with a just a few mouse clicks Totally built on top of glite with a proper business logic, glibrary makes user assets accessible from everywhere and at anytime, hiding the complexity of the underlying system and the geographical location of data 19

20 Acknoledgements Claudio Cherubino (AMGA PHP APIs, PHP business logic, AJAX & Javascript web interface) Diego Scardaci & Marco Fargetta (Java applets developing and debugging) Leandro Neuman Ciuffo (web designer and printmaker) 20