Data management in EGEE
|
|
- Claud Fox
- 8 years ago
- Views:
Transcription
1 Home Search Collections Journals About Contact us My IOPscience Data management in EGEE This content has been downloaded from IOPscience. Please scroll down to see the full text J. Phys.: Conf. Ser ( View the table of contents for this issue, or go to the journal homepage for more Download details: IP Address: This content was downloaded on 13/11/2015 at 17:32 Please note that terms and conditions apply.
2 Data Management in EGEE Ákos Frohner 1, Jean-Philippe Baud 1, Rosa Maria Garcia Rioja 1, Gilbert Grosdidier 2, Rémi Mollon 1, David Smith 1 and Paolo Tedesco 1 1 CERN, Switzerland 2 LAL/IN2P3/CNRS, France Abstract. Data management is one of the cornerstones in the distributed production computing environment that the EGEE project aims to provide for a e-science infrastructure. We have designed and implemented a set of services and client components, addressing the diverse requirements of all user communities. LHC experiments as main users will generate and distribute approximately 15 PB of data per year worldwide using this infrastructure. Another key user community, biomedical projects, have strict security requirements with less emphasis on the volume of data. We maintain three service groups for grid data management: The Disk Pool Manager (DPM) Storage Element (with more than 100 instances deployed world-wide), the LCG File Catalogue (LFC) and the File Transfer Service (FTS) which sustains an aggregated transfer rate of 1.5GB/sec. They are complemented by individual client components and also tools which help coordinating more complex uses cases with multiple services (GFAL-client, lcg util, eds-cli). In this paper we show how these services, keeping clean and standard interfaces among each other, can work together to cover the data flow and how they can be used as individual components to cover diverse requirements. We will also describe areas that we consider for further improvements, both for performance and functionality. 1. Introduction Data management is one of the cornerstones in the distributed production computing environment that the EGEE project aims to provide for a e-science infrastructure [1]. This infrastructure currently includes 267 sites over 54 countries with about 114,000 CPU computing resource and more than 20 PB storage in 291 storage elements (SE). The main customer of this infrastructure is High-Energy Physics community via the LHC Computing Grid project. They share this infrastructure with more than 15 application domains, like Life Sciences, Computational Chemistry and Earth Sciences, grouping their users into 150 virtual organizations (VO). The EGEE grid infrastructure consists of a set of middleware services deployed on the worldwide infrastructure. Most of these software components are provided by the project s own development efforts under the name of glite middleware[2]. In Section 2 we introduce the data management software stack of the EGEE project and in Sections 3-9 we describe the components provided by our team. Section 10 describes in more detail the typical use cases from a user s perspective and how they are implemented by the data management components. c 2010 IOP Publishing Ltd 1
3 2. EGEE Data Management The EGEE data management components can be classified in three main categories: Storage Elements, as the foundation of the infrastructure Higher level services to cover use cases over the basic file access and Client tools, which interact with these services and provide simpler interfaces to the users. Figure 1. EGEE Software Stack Storage Elements are the most fundamental data management services as they are storing the files on disk and tape. They provide access to these files via a wide variety of protocols, such as: gridftp, rfio, dcap, xrootd, HTTP(S) and NFS. They also provide a management interface for the storage resources via specialized protocols and the standardized SRM[3] interface. In the EGEE infrastructure there are a number of SEs used: DPM, Castor, dcache, StoRM and BestMan. Higher level services provide functionality over the basic file access use cases: File catalogs enable the users to find their files among the storage elements. They manage the SE independent namespace of logical file names (LFN), which are mapped to a number of replicas, files in the storage elements. Reliable file transfer services take care of file replication among storage elements by hiding the details of resource negotiations and error handling from the users. Keystore service provides encryption keys for encrypted files in the storage elements. Client tools provide application programming and command line interfaces for the above mentioned services with a layer of convenience libraries. One of these abstractions is the POSIX style I/O API over the file access protocols provided by the storage elements. The goal of such a library is that application programmers do not need to be able to handle all protocols, but write their code against a single API, which will handle the differences. There are also complex operations, such as uploading a file to a storage element and registering it with a file catalog, which are typical to many applications. The implementation of these typical use cases is provided as a convenience library. 2
4 3. Overview We have designed and implemented a set of services and client components, addressing the diverse requirements of all user communities. Figure 2. glite data management components We maintain three service groups for grid data management: The Disk Pool Manager Storage Element (DPM, see Section 4), the LCG File Catalogue (LFC, see Section 5), and the File Transfer Service (FTS, see Section 6). They are complemented by individual client components and also tools which help coordinating more complex uses cases with multiple services for average clients (GFAL-client, see Section 7 and lcg util, see Section 8) and for clients requiring encrypted files (eds-cli, see Section 9). Most of these clients are installed on all the worker nodes of the EGEE, OSG and NorduGrid infrastructures. There are other services, which are not maintained by our team, but used by our services and tools, such as the BDII Information System or other Storage Elements (Castor, dcache, StoRM, BeStMan) Grid Security The glite data management components follow the model of other grid services for authentication and authorization. They use X509 certificates to mutually authenticate clients and services to each other based on trusted Certificate Authorities, which are managed by the International Grid Trust Federation (IGTF). The authorization decisions are based on authenticated individuals and grouping of these entities. We use the Virtual Organization Membership Service (VOMS)[4] to provide the grouping information of X509 entities by the form of Fully Qualified Attribute Names (FQAN). Otherwise the access control information about a managed object (e.g. file, directory, channel, transfer job) is stored inside a service s database Information System The EGEE glite services publish information about themselves to the information system, BDII, so that the client tools can locate them. The information is currently published according to the GLUE 1.3 schema, however it is planned to be extended to the new GLUE 2.0 schema as well. 3
5 The published information, such as service endpoint, are retrieved from the BDII using the LDAP protocol. The FTS clients and services could use other information systems as well via the Service Discovery layer Storage Protocols If it is otherwise not stated the clients and services use the Storage Resource Manager (SRM)[3] protocol to manage the storage elements (SE) and the gridftp (or gsi-ftp) protocol to access the content of files. 4. DPM The light-weight Disk Pool Manager (DPM) offers a simple solution for a disk-only Storage Element. It is easy to install and configure and requires very low maintenance effort. It is deployed in about 190 sites within EGEE. At one site the DPM is used to manage up to 360TB of data. DPM consists of a set of services with their own client interface: the Disk Pool Name Server (DPNS) to keep the hierarchical namespace and the access authorizations; the Disk Pool Manager (DPM) to manage the disk space and process the user requests; the Remote File Input Output (RFIO), GridFTP, HTTP(s) and xrootd services to provide access data and the Storage Resource Management (SRM) as standard Web Service interface [3]. Storage management features of this service include pool and space protection; garbage collection of unused replicas and replication of hot files (triggered by the administrator). For the client DPM provides a client API library in C, Perl and Python (proprietary socket interface to DPNS/DPM/RFIO), a command line interface, and the standard SRM interfaces (v1.1, v2.1 and v2.2). All the interfaces work only in secure mode using X509 certificates or Kerberos5 tokens. DPM implements POSIX style file and directory authorization, where a user name is an X509 certificate DN or Kerberos principal and groups are taken from VOMS FQANs. The authorization is implemented independent of the underlying operating system by using an internal user and group database. DPM has a portable codebase, which has been built on a variety of Linux distributions (RedHat Enterprise Linux, Debian), Mac OS X and Solaris. DPM does not need to contact any external service but static and dynamic information concerning the Storage Element is published in the Information System. As any other SE, DPM is used by applications like GFAL, lcg util and FTS. 5. LFC The LCG File catalogue (LFC) offers a hierarchical view of files to users, with a UNIX-like client interface. The LFC is deployed at most EGEE European and Asian sites, at some of the OSG sites, in addition to CERN. The LFC catalogue provides mappings between a Logical File Name (LFN) and Storage URLs (SURL) with POSIX style authorization (see DPM). It supports session based connections to minimize the authentication overhead and transactions for complex modifications. The LFC relies on a client-server model, using a proprietary socket interface. The LFC server communicates with a database (Oracle, MySQL or PostgreSQL), where all the data is stored. Read-only replicated file catalog can be deployed by using Oracle streams or a fail-over solution with DataGuard. The LFC server is accessible from the client side by API libraries (POSIX style namespace operations and non-posix bulk methods) in C, Perl and Python and by a command line interface. 4
6 Working together with the user communities the basic set of namespace operations have been extended over the past years to provide better performance via more complex operations for bulk deletion, querying attributes of a set of entries and registering files in a single aggregated operation. 6. FTS The glite File Transfer Service (FTS) is a data movement service for transferring files between Storage Elements. It was designed to balance site resource usage, prevent network or storage overload, enforce job prioritization, retrying failed transfers and facilitate administration and monitoring of transfers. The FTS exposes an interface to submit asynchronous bulk requests and performs the transfers using either third-party GridFTP or SRM Copy. These third-party transfers enable FTS to drive transfers in parallel among many disk servers, thus being able to scale up to the limits of the underlying network and reach an aggregated transfer rate of 1.5GB/sec. The FTS servers are typically deployed at (large) sites where there are large amounts of data to be transferred. FTS manages the transfers in an unidirectional queue, called channel. A channel is typically defined between two sites and describes the transfer protocol, parameters and resource restrictions. Figure 3. FTS channels 6.1. FTS architecture The FTS front-end is a secure Web Service that provides three different port types for submitting requests and monitoring their status, administering and monitoring the channels, and retrieving the FTS usage statistics. The main FTS functionality is provided by a set of daemons, File Transfer Agents, responsible for triggering the third-party transfers (Channel Agent) and applying the VO-specific policies for retrying failed transfers (VO Agents). The WS front-end communicates with the Agents by storing requests into an Oracle database. Users can access FTS services either via the Web Service API or using the provided client. FTS makes extensive use of the Service Discovery API, for discovering the endpoint and the properties of external services. From the security point of view, the interactions with these external services (mainly SRMs and Storage Elements) always use the client proxy credentials either retrieved from MyProxy or delegated by the delegation components and renewed using the proxy-renewal APIs. 5
7 7. GFAL The Grid File Access Library (GFAL) is a library that offers to the user a POSIX style API to access data on various flavours of Storage Elements offering (de-facto) standard interfaces. lcg util includes a set of command line tools and libraries that provide higher level functionality on top of the GFAL functionalities. Here POSIX style means that function prototypes follow the signature of their POSIX equivalents, however with a custom prefix: int gfal open (const char *, int, mode t). With pre-processing macros or libc-preload wrappers it is possible to replace the normal POSIX calls by these functions. GFAL is currently interfaced to SRM-compliant back-ends (both v1.1 and v2.2) or de-facto standard facades to the massive storage systems such as Castor, dcache or DPM. It provides a common abstraction over these interfaces by using relevant protocols transparently behind the scenes. Using information published in the information system, it resolves relevant abstract domain data/file names so that the physical data access as well as the end-points of services are achievable transparently. It allows and unifies access to various types of items such as: LFN, GUID, SURL, SRM and TURL or local path. In addition, some of the crucial, yet common, backend calls are exposed through the library so that users are not limited to POSIX mapping to do specific calls e.g. to reserve space or pin a file. The pluggable architecture of the library permits optional loading and dynamic change of the versions of some of the supported protocols (i.e. rfio, dcap) without need of redeployment. 8. lcg util In the lcg util a mixture of GFAL and Globus functionality was used to create Grid equivalents of UNIX copy commands and registration that spans storage systems and file catalogues. Other functions have been implemented, such as getting the list of file replicas from many sources, performing lookups on LFNs or GUIDs, changing the status of the file or finally removing it from the Grid. The choice of commands implemented was user driven and is often on the first frontier for the Grid user or Grid testers. lcg util provides the C library, command-line programs based on this library and Python bindings for easier integration. 9. Encrypted Data Storage The Encrypted Data Storage system is made of a set of components on top of the previously described data management infrastructure Encrypted Data Storage Client The Encrypted Data Storage provides a client-side C library to encrypt and decrypt block level data on the fly. It uses the OpenSSL cryptographic library for the symmetric cryptography routines, thus it can utilize any of the available cipher algorithms, such as the AES cipher. The encryption/decryption keys are stored in the Hydra key store. The component also provides command line utilities (eds-cli) for managing the keys in a Hydra key store. Other command-line utilities integrate the library with GFAL, thus one can retrieve and decrypt or encrypt and store files transparently Hydra Keystore The symmetric encryption keys for encrypted data (files) storage are stored in a specific set of servers called Hydra. Hydra provides controlled access to these keys (through certificate DN and VOMS attributes based ACLs) and secured communication to the requester. The Hydra service is a Java Web Service, which can be deployed in a J2EE container, such as Tomcat. It requires a database back-end, and communicates via the HTTPS protocol with its clients. 6
8 In addition, Hydra exploits the Shamir secret-sharing scheme to improve security and reliability of this service. Shamir s scheme consists of splitting keys into N fragments stored in different places. Only M < N fragments are needed to reconstruct a complete key. However, owning less than M key fragments, does not give any information on the complete key. Thus, the system is both resistant to attacks (at least M key stores need to be compromised for an attacker to be able to reconstruct a key) and reliable (the disconnection of a limited number of servers does not prevent the key reconstruction) DPM/DICOM Interface The Digital Imaging and Communications in Medicine (DICOM) is a standard medical image storage system used by Biomedical researchers, an important user community in EGEE. glite includes an interface between DICOM and the DPM storage element. This DPM/DICOM interface is a plug-in for DPM that takes anonymized medical images from a DICOM system, enters the image metadata into a metadata storage system such as AMGA, uses EDS and Hydra to encrypt the image, stores the resulting file in DPM and stores storage metadata in LFC. These encrypted files may then be analyzed or studied in a Grid environment. See more of the details at [5]. 10. Use Cases This section describes the typical data management use cases of the grid middleware Uploading a file When a client creates a new file on the User Interface or on the Worker Node, then it needs to be uploaded to a storage element to make it accessible across the grid. Figure 4. Uploading a file 1. Lookup of the storage element endpoint and VO directory using the destination site or SE name in BDII, for example myse.cern.ch 2. Generating a storage URL (SURL) locally based on the SE endpoint information, for example srm://myse.cern.ch:8443/srm/managerv2?sfn=/cern.ch/dteam/myfile 3. Acquiring a transfer URL for the SURL via the SRM interface of the storage element, for example gsiftp://disk101.cern.ch:2811/dteam/myfile Uploading the data to the TURL by gridftp 5. Registering the LFN-SURL pair in the LFC 7
9 The following lcg util command will do all these steps: lcg-cr -d myse.cern.ch -l lfn:/grid/dteam/myfile /path/to/my/local/file Downloading a file A client wants to download a file to the local file system, knowing its logical file name. Figure 5. Downloading a file 1. Get a SURL for the LFN by looking up the registered replicas in LFC. If there are more than one replicas, then the client chooses one, preferably from the local storage element. For example for the lfn:/grid/dteam/myfile LFN the LFC may return two SURLs: srm://grid.edu.tw/castor/grid.edu.tw/dteam/myfile srm://myse.cern.ch/srm/managerv2?sfn=/cern.ch/dteam/myfile 2. Lookup the SE endpoint for the SURL in BDII 3. Acquiring a transfer URL for the SURL via the SRM interface of the storage element, for example gsiftp://disk145.cern.ch:2811/dteam/myfile Downloading the file from the TURL by gridftp The following lcg util command will do all these steps: lcg-cp lfn:/grid/dteam/myfile /path/to/my/local/file The client tool will try to handle error conditions such as the file not being available at the first selected storage element. In this case it will iterate through the prioritized list of replicas and attempt to access them until it manages to download one of them. The client tool may choose other protocol than gridftp to access the file. For example a worker node may access a local storage element via rfio, dcap or even direct POSIX file open (file) Decrypting a file A variation of the previous use case, when the file in the storage element is encrypted, thus the client needs to get the encryption key and have to decrypt the file locally. 1. Get a SURL for the LFN by looking up the registered replicas in LFC 2. Lookup the SE endpoint for the SURL and the Hydra endpoints (keys are split among multiple services) for the LFN in BDII 8
10 Figure 6. Decrypting a file 3. Get the pieces of the en/decryption key from the Hydra services for the LFN and combine them into a single key 4. Acquiring a transfer URL for the SURL via the SRM interface of the storage element 5. Downloading the file from the TURL 6. Decrypting the downloaded file as it arrives block-by-block The following glite-data-eds-cli command will do all these steps: glite-eds-get lfn:/grid/dtem/myfile /path/to/my/local/file Transferring a file A client can initiate a file transfer between two storage elements directly or can delegate this job to the File Transfer Service, which will schedule and retry this transfer as needed. Figure 7. Transferring a file using FTS 1. Submitting the transfer job to the FTS. 1.1 FTS acquires transfer URL from the destination SE by its SRM interface 1.2 FTS acquires get the transfer URL from the source SE by its SRM interface 1.3 FTS prepares the transfer on the destination via gridftp 9
11 1.4 FTS initiate the transfer on the source SE via gridftp and follows its progress. If something goes wrong, it will clean up the partially transferred file and retry the transfer 2. Meanwhile the client can poll the FTS for the status of the transfer job glite-transfer-submit \ srm://myse.cern.ch:8443/srm/managerv2?sfn=/foo \ srm://myse.example.org:8443/srm/managerv2?sfn=/foo glite-transfer-status 49ce183b-3fb4-11de-b943-abe14bae4af8 11. Future Directions The main priorities in our team for the data management components are Stability focusing on quickly solving problems, providing bug fixes and maintaining backward compatibility in the application programming and command line interfaces. Reliability improving the error handling in our clients and services against internal failures and also against failures of other services used by the component. Maintainability providing portable code to be prepared for future platform changes and documenting it internally for anyone participating in the development. Besides these main objectives we also plan to improve the administrative tools, encapsulating routine procedures into single commands and providing web interfaces to ease the administration. We would like to provide real time monitoring information of services for easier problem determination and to enable proactive administrative changes. And we would also like to automate regular procedures, for example cleanup and archival of old and unused records. For resource protection we plan to implement a quota system for storage and network bandwidth limits for individual file accesses. For better integration with user data management frameworks we are looking into simplifying the client libraries and using using messaging instead of polling. The detailed plans in respect to each individual components are maintained by the LCG Savannah software development portal as bugs[6] and tasks[7]. References [1] Abadie L, Badino P, Baud J P, Casey J, Frohner A, Grosdidier G, Lemaitre S, Mccance G, Mollon R, Nienartowicz K, Smith D and Tedesco P 2007 Mass Storage Systems and Technologies, IEEE / NASA Goddard Conference on [2] EGEE glite middleware URL [3] Abadie L, Badino P, Baud J P, Corso E, Crawford M, Witt S D, Donno F, Forti A, Frohner A, Fuhrmann P, Grosdidier G, Gu J, Jensen J, Koblitz B, Lemaitre S, Litmaath M, Litvinsev D, Presti G L, Magnoni L, Mkrtchan T, Moibenko A, Mollon R, Natarajan V, Oleynik G, Perelmutov T, Petravick D, Shoshani A, Sim A, Smith D, Sponza M, Tedesco P and Zappi R 2007 Mass Storage Systems and Technologies, IEEE / NASA Goddard Conference on [4] Alfieri R, Cecchini R, Ciaschini V, Dell Agnello L, Gianoli A, Spataro F, Bonnassieux F, Broadfoot P, Lowe G, Cornwall L, Jensen J, Kelsey D, Frohner A, Groep D, De Cerff W S, Steenbakkers M, Venekamp G, Kouril D, McNab A I, Mulmo O, Silander M, Hahkala J and Lhorentey K 2003 Managing dynamic user communities in a grid of autonomous resources Tech. Rep. cs.dc/ [5] Montagnat J, Jouvenot D, Pera C, Frohner A, Kunszt P Z, Koblitz B, Santos N and Loomis C 2006 Bridging clinical information systems and grid middleware: a medical data manager Tech. Rep. EGEE-PUB [6] EGEE JRA1 bugs and feature requests URL [7] EGEE JRA1 workplan URL 10
Analisi di un servizio SRM: StoRM
27 November 2007 General Parallel File System (GPFS) The StoRM service Deployment configuration Authorization and ACLs Conclusions. Definition of terms Definition of terms 1/2 Distributed File System The
More informationThe glite File Transfer Service
Enabling Grids Enabling for E-sciencE Grids for E-sciencE The glite File Transfer Service Paolo Badino On behalf of the JRA1 Data Management team EGEE User Forum - CERN, 2 Mars 2006 www.eu-egee.org Outline
More informationThe glite File Transfer Service
The glite File Transfer Service Peter Kunszt Paolo Badino Ricardo Brito da Rocha James Casey Ákos Frohner Gavin McCance CERN, IT Department 1211 Geneva 23, Switzerland Abstract Transferring data reliably
More informationStorage Resource Managers: Recent International Experience on Requirements and Multiple Co-Operating Implementations
Storage Resource Managers: Recent International Experience on Requirements and Multiple Co-Operating Implementations Lana Abadie 1, Paolo Badino 1, Jean-Philippe Baud 1,Ezio Corso 2, Matt Crawford 3, Shaun
More informationThe dcache Storage Element
16. Juni 2008 Hamburg The dcache Storage Element and it's role in the LHC era for the dcache team Topics for today Storage elements (SEs) in the grid Introduction to the dcache SE Usage of dcache in LCG
More informationFile Transfer Software and Service SC3
File Transfer Software and Service SC3 Gavin McCance JRA1 Data Management Cluster Service Challenge Meeting April 26 2005, Taipei www.eu-egee.org Outline Overview of Components Tier-0 / Tier-1 / Tier-2
More informationglibrary: Digital Asset Management System for the Grid
glibrary: Digital Asset Management System for the Grid Antonio Calanducci INFN Catania EGEE User Forum Manchester, 09 th -11 th May 2007 www.eu-egee.org EGEE and glite are registered trademarks Outline
More information1967-35. Advanced School in High Performance and GRID Computing. 3-14 November 2008
1967-35 Advanced School in High Performance and GRID Computing 3-14 November 2008 Grid tutorial hands-on session on data management and advanced job submission TAFFONI Giuliano Osservatorio Astronomico
More informationA Secure Grid Medical Data Manager Interfaced to the glite Middleware
A Secure Grid Medical Data Manager Interfaced to the glite Middleware Johan Montagnat, Akos Frohner, Daniel Jouvenot, Christophe Pera, Peter Kunszt, Birger Koblitz, Nuno Santos, Charles Loomis, Romain
More informationChapter 12 Distributed Storage
Chapter 12 Distributed Storage 1 2 Files File location and addressing What is a file? Normally we collapse. Concepts: name; contents; gui. What about the backup of this file? How do we distinguish? File
More informationManaged Storage @ GRID or why NFSv4.1 is not enough. Tigran Mkrtchyan for dcache Team
Managed Storage @ GRID or why NFSv4.1 is not enough Tigran Mkrtchyan for dcache Team What the hell do physicists do? Physicist are hackers they just want to know how things works. In moder physics given
More informationPlateforme de Calcul pour les Sciences du Vivant. SRB & glite. V. Breton. http://clrpcsv.in2p3.fr
SRB & glite V. Breton http://clrpcsv.in2p3.fr Introduction Goal: evaluation of existing technologies for data and tools integration and deployment Data and tools integration should be addressed using web
More informationA Survey Study on Monitoring Service for Grid
A Survey Study on Monitoring Service for Grid Erkang You erkyou@indiana.edu ABSTRACT Grid is a distributed system that integrates heterogeneous systems into a single transparent computer, aiming to provide
More informationData Management in an International Data Grid Project. Timur Chabuk 04/09/2007
Data Management in an International Data Grid Project Timur Chabuk 04/09/2007 Intro LHC opened in 2005 several Petabytes of data per year data created at CERN distributed to Regional Centers all over the
More informationdcache, a managed storage in grid
dcache, a managed storage in grid support and funding by Patrick for the dcache Team Topics Project Topology Why do we need storage elements in the grid world? The idea behind the LCG (glite) storage element.
More informationThe EU DataGrid Data Management
The EU DataGrid Data Management The European DataGrid Project Team http://www.eu-datagrid.org DataGrid is a project funded by the European Union Grid Tutorial 4/3/2004 n 1 EDG Tutorial Overview Workload
More informationEDG Project: Database Management Services
EDG Project: Database Management Services Leanne Guy for the EDG Data Management Work Package EDG::WP2 Leanne.Guy@cern.ch http://cern.ch/leanne 17 April 2002 DAI Workshop Presentation 1 Information in
More informationXerox DocuShare Security Features. Security White Paper
Xerox DocuShare Security Features Security White Paper Xerox DocuShare Security Features Businesses are increasingly concerned with protecting the security of their networks. Any application added to a
More informationAMGA - Official Metadata Service for EGEE
Consorzio COMETA - Progetto PI2S2 FESR AMGA - Official Metadata Service for EGEE Salvatore Scifo INFN Catania Tutorial per utenti e sviluppo di applicazioni in Grid Catania, July 16 th - 20 th 2007 www.consorzio-cometa.it
More informationWP5 Progress Report. John Gordon EDG Conference Barcelona, May 2003. escience Centre
WP5 Progress Report EDG Conference Barcelona, May 2003 escience Centre Outline Status of release 2.0 Testing and support structures Priorities up to TB3 Open issues Status of release 2.0 Core SE Software
More informationTechnical. Overview. ~ a ~ irods version 4.x
Technical Overview ~ a ~ irods version 4.x The integrated Ru e-oriented DATA System irods is open-source, data management software that lets users: access, manage, and share data across any type or number
More informationGrid Data Management. Raj Kettimuthu
Grid Data Management Raj Kettimuthu Data Management Distributed community of users need to access and analyze large amounts of data Fusion community s International ITER project Requirement arises in both
More informationMass Storage at GridKa
Mass Storage at GridKa Forschungszentrum Karlsruhe GmbH Institute for Scientific Computing P.O. Box 3640 D-76021 Karlsruhe, Germany Dr. Doris Ressmann http://www.gridka.de 1 Overview What is dcache? Pool
More informationEMI Storage meets EMI security
EMI Storage meets EMI security Component/ Middleware glite (LFC,FTS,DPM,GFAL) ARC UNICORE StoRM dcache Staff With kind contributions by Oliver Keeble, Jean- Philippe Baud Jon Kerr Nilsen Ralph Müller-
More informationDatabase Services for Physics @ CERN
Database Services for Physics @ CERN Deployment and Monitoring Radovan Chytracek CERN IT Department Outline Database services for physics Status today How we do the services tomorrow? Performance tuning
More informationApigee Gateway Specifications
Apigee Gateway Specifications Logging and Auditing Data Selection Request/response messages HTTP headers Simple Object Access Protocol (SOAP) headers Custom fragment selection via XPath Data Handling Encryption
More informationImprovement Options for LHC Mass Storage and Data Management
Improvement Options for LHC Mass Storage and Data Management Dirk Düllmann HEPIX spring meeting @ CERN, 7 May 2008 Outline DM architecture discussions in IT Data Management group Medium to long term data
More informationDistributed Storage Management Service in UNICORE
Distributed Storage Management Service in UNICORE Tomasz Rękawek 1,2, Piotr Bała 1,2, Krzysztof Benedyczak 1,2 1 Interdisciplinary Center for Mathematical and Computational Modelling, University of Warsaw,
More informationCERN Cloud Storage Evaluation Geoffray Adde, Dirk Duellmann, Maitane Zotes CERN IT
SS Data & Storage CERN Cloud Storage Evaluation Geoffray Adde, Dirk Duellmann, Maitane Zotes CERN IT HEPiX Fall 2012 Workshop October 15-19, 2012 Institute of High Energy Physics, Beijing, China SS Outline
More informationCluster, Grid, Cloud Concepts
Cluster, Grid, Cloud Concepts Kalaiselvan.K Contents Section 1: Cluster Section 2: Grid Section 3: Cloud Cluster An Overview Need for a Cluster Cluster categorizations A computer cluster is a group of
More informationHow To Get To A Cloud Storage And Byod System
Maginatics Security Architecture What is the Maginatics Cloud Storage Platform? Enterprise IT organizations are constantly looking for ways to reduce costs and increase operational efficiency. Although
More informationForschungszentrum Karlsruhe in der Helmholtz-Gemeinschaft. dcache Introduction
dcache Introduction Forschungszentrum Karlsruhe GmbH Institute for Scientific Computing P.O. Box 3640 D-76021 Karlsruhe, Germany Dr. http://www.gridka.de What is dcache? Developed at DESY and FNAL Disk
More informationCERN local High Availability solutions and experiences. Thorsten Kleinwort CERN IT/FIO WLCG Tier 2 workshop CERN 16.06.2006
CERN local High Availability solutions and experiences Thorsten Kleinwort CERN IT/FIO WLCG Tier 2 workshop CERN 16.06.2006 1 Introduction Different h/w used for GRID services Various techniques & First
More informationOSG PUBLIC STORAGE. Tanya Levshina
PUBLIC STORAGE Tanya Levshina Motivations for Public Storage 2 data to use sites more easily LHC VOs have solved this problem (FTS, Phedex, LFC) Smaller VOs are still struggling with large data in a distributed
More informationResume. Wenjing. Date of birth: June 11th, 1982 Nationality: Chinese Phone number: 8610-88236012-608 Cell phone: 13366466802 wuwj@ihep.ac.
Resume Personal information First name: Wenjing surname: Wu Gender: Female Date of birth: June 11th, 1982 Nationality: Chinese Phone number: 8610-88236012-608 Cell phone: 13366466802 Email: wuwj@ihep.ac.cn
More informationManaging Credentials with
Managing Credentials with MyProxy Jim Basney National Center for Supercomputing Applications University of Illinois jbasney@ncsa.uiuc.edu http://myproxy.ncsa.uiuc.edu/ What is MyProxy? A service for managing
More informationIGI Portal architecture and interaction with a CA- online
IGI Portal architecture and interaction with a CA- online Abstract In the framework of the Italian Grid Infrastructure, we are designing a web portal for the grid and cloud services provisioning. In following
More informationAnalyses on functional capabilities of BizTalk Server, Oracle BPEL Process Manger and WebSphere Process Server for applications in Grid middleware
Analyses on functional capabilities of BizTalk Server, Oracle BPEL Process Manger and WebSphere Process Server for applications in Grid middleware R. Goranova University of Sofia St. Kliment Ohridski,
More informationDistributed Database Access in the LHC Computing Grid with CORAL
Distributed Database Access in the LHC Computing Grid with CORAL Dirk Duellmann, CERN IT on behalf of the CORAL team (R. Chytracek, D. Duellmann, G. Govi, I. Papadopoulos, Z. Xie) http://pool.cern.ch &
More informationGridFTP: A Data Transfer Protocol for the Grid
GridFTP: A Data Transfer Protocol for the Grid Grid Forum Data Working Group on GridFTP Bill Allcock, Lee Liming, Steven Tuecke ANL Ann Chervenak USC/ISI Introduction In Grid environments,
More informationowncloud Architecture Overview
owncloud Architecture Overview Time to get control back Employees are using cloud-based services to share sensitive company data with vendors, customers, partners and each other. They are syncing data
More informationIn this chapter, we will introduce works related to our research. First, we will
Chapter 2 Related Works In this chapter, we will introduce works related to our research. First, we will present the basic concept of directory service and Lightweight Directory Access Protocol (LDAP).
More informationData Collection and Analysis: Get End-to-End Security with Cisco Connected Analytics for Network Deployment
White Paper Data Collection and Analysis: Get End-to-End Security with Cisco Connected Analytics for Network Deployment Cisco Connected Analytics for Network Deployment (CAND) is Cisco hosted, subscription-based
More informationThe EDGeS project receives Community research funding
Desktop Grids EDGeS project Delegation for access to trusted resources The EDGeS project receives Community research funding 1 DG = Desktop Grid = Loose grid scavenging idle resources Unit of Work = Application
More informationWeb Service Robust GridFTP
Web Service Robust GridFTP Sang Lim, Geoffrey Fox, Shrideep Pallickara and Marlon Pierce Community Grid Labs, Indiana University 501 N. Morton St. Suite 224 Bloomington, IN 47404 {sblim, gcf, spallick,
More informationIntegrating a heterogeneous and shared Linux cluster into grids
Integrating a heterogeneous and shared Linux cluster into grids 1,2 1 1,2 1 V. Büge, U. Felzmann, C. Jung, U. Kerzel, 1 1 1 M. Kreps, G. Quast, A. Vest 1 2 DPG Frühjahrstagung March 28 31, 2006 Dortmund
More informationData Grids. Lidan Wang April 5, 2007
Data Grids Lidan Wang April 5, 2007 Outline Data-intensive applications Challenges in data access, integration and management in Grid setting Grid services for these data-intensive application Architectural
More informationOASIS: a data and software distribution service for Open Science Grid
OASIS: a data and software distribution service for Open Science Grid B. Bockelman 1, J. Caballero Bejar 2, J. De Stefano 2, J. Hover 2, R. Quick 3, S. Teige 3 1 University of Nebraska-Lincoln, Lincoln,
More informationBridging clinical information systems and grid middleware: a Medical Data Manager
Bridging clinical information systems and grid middleware: a Medical Data Manager Johan Montagnat 1, Daniel Jouvenot 2, Christophe Pera 3, Ákos Frohner 4, Peter Kunszt 4, Birger Koblitz 4, Nuno Santos
More informationEvaluating Metadata access
Evaluating Metadata access strategies with the GOME test suite André Gemünd Fraunhofer SCAI www.eu-egee.org EGEE-II INFSO-RI-031688 EGEE and glite are registered trademarks Motivation Testing the test
More informationEMC DATA DOMAIN ENCRYPTION A Detailed Review
White Paper EMC DATA DOMAIN ENCRYPTION A Detailed Review Abstract The proliferation of publicized data loss, coupled with new governance and compliance regulations, is driving the need for customers to
More informationdcache, Software for Big Data
dcache, Software for Big Data Innovation Day 2013, Berlin Patrick Fuhrmann dcache Innovation Day Berlin Patrick Fuhrmann 10 December 2013 1 About Technology and further roadmap Collaboration and partners
More informationCMS Dashboard of Grid Activity
Enabling Grids for E-sciencE CMS Dashboard of Grid Activity Julia Andreeva, Juha Herrala, CERN LCG ARDA Project, EGEE NA4 EGEE User Forum Geneva, Switzerland March 1-3, 2006 http://arda.cern.ch ARDA and
More informationThe Data Grid: Towards an Architecture for Distributed Management and Analysis of Large Scientific Datasets
The Data Grid: Towards an Architecture for Distributed Management and Analysis of Large Scientific Datasets!! Large data collections appear in many scientific domains like climate studies.!! Users and
More informationThe ENEA-EGEE site: Access to non-standard platforms
V INFNGrid Workshop Padova, Italy December 18-20 2006 The ENEA-EGEE site: Access to non-standard platforms C. Sciò**, G. Bracco, P. D'Angelo, L. Giammarino*, S.Migliori, A. Quintiliani, F. Simoni, S. Podda
More informationConfiguring Security Features of Session Recording
Configuring Security Features of Session Recording Summary This article provides information about the security features of Citrix Session Recording and outlines the process of configuring Session Recording
More informationDeploying a distributed data storage system on the UK National Grid Service using federated SRB
Deploying a distributed data storage system on the UK National Grid Service using federated SRB Manandhar A.S., Kleese K., Berrisford P., Brown G.D. CCLRC e-science Center Abstract As Grid enabled applications
More informationCA Performance Center
CA Performance Center Single Sign-On User Guide 2.4 This Documentation, which includes embedded help systems and electronically distributed materials, (hereinafter referred to as the Documentation ) is
More informationBlackBerry Enterprise Service 10. Secure Work Space for ios and Android Version: 10.1.1. Security Note
BlackBerry Enterprise Service 10 Secure Work Space for ios and Android Version: 10.1.1 Security Note Published: 2013-06-21 SWD-20130621110651069 Contents 1 About this guide...4 2 What is BlackBerry Enterprise
More informationE-mail: guido.negri@cern.ch, shank@bu.edu, dario.barberis@cern.ch, kors.bos@cern.ch, alexei.klimentov@cern.ch, massimo.lamanna@cern.
*a, J. Shank b, D. Barberis c, K. Bos d, A. Klimentov e and M. Lamanna a a CERN Switzerland b Boston University c Università & INFN Genova d NIKHEF Amsterdam e BNL Brookhaven National Laboratories E-mail:
More informationArchiving, Indexing and Accessing Web Materials: Solutions for large amounts of data
Archiving, Indexing and Accessing Web Materials: Solutions for large amounts of data David Minor 1, Reagan Moore 2, Bing Zhu, Charles Cowart 4 1. (88)4-104 minor@sdsc.edu San Diego Supercomputer Center
More informationIBM Tivoli Storage Manager Version 7.1.4. Introduction to Data Protection Solutions IBM
IBM Tivoli Storage Manager Version 7.1.4 Introduction to Data Protection Solutions IBM IBM Tivoli Storage Manager Version 7.1.4 Introduction to Data Protection Solutions IBM Note: Before you use this
More informationStatus and Evolution of ATLAS Workload Management System PanDA
Status and Evolution of ATLAS Workload Management System PanDA Univ. of Texas at Arlington GRID 2012, Dubna Outline Overview PanDA design PanDA performance Recent Improvements Future Plans Why PanDA The
More informationBatch and Cloud overview. Andrew McNab University of Manchester GridPP and LHCb
Batch and Cloud overview Andrew McNab University of Manchester GridPP and LHCb Overview Assumptions Batch systems The Grid Pilot Frameworks DIRAC Virtual Machines Vac Vcycle Tier-2 Evolution Containers
More informationEvolution of Database Replication Technologies for WLCG
Home Search Collections Journals About Contact us My IOPscience Evolution of Database Replication Technologies for WLCG This content has been downloaded from IOPscience. Please scroll down to see the full
More informationMichael Thomas, Dorian Kcira California Institute of Technology. CMS Offline & Computing Week
Michael Thomas, Dorian Kcira California Institute of Technology CMS Offline & Computing Week San Diego, April 20-24 th 2009 Map-Reduce plus the HDFS filesystem implemented in java Map-Reduce is a highly
More informationSecurity Policy Revision Date: 23 April 2009
Security Policy Revision Date: 23 April 2009 Remote Desktop Support Version 3.2.1 or later for Windows Version 3.1.2 or later for Linux and Mac 4 ISL Light Security Policy This section describes the procedure
More informationowncloud Architecture Overview
owncloud Architecture Overview owncloud, Inc. 57 Bedford Street, Suite 102 Lexington, MA 02420 United States phone: +1 (877) 394-2030 www.owncloud.com/contact owncloud GmbH Schloßäckerstraße 26a 90443
More informationSharePoint 2013 Logical Architecture
SharePoint 2013 Logical Architecture This document is provided "as-is". Information and views expressed in this document, including URL and other Internet Web site references, may change without notice.
More informationDiagram 1: Islands of storage across a digital broadcast workflow
XOR MEDIA CLOUD AQUA Big Data and Traditional Storage The era of big data imposes new challenges on the storage technology industry. As companies accumulate massive amounts of data from video, sound, database,
More informationCisco UCS Central Software
Data Sheet Cisco UCS Central Software Cisco UCS Manager provides a single point of management for an entire Cisco Unified Computing System (Cisco UCS) domain of up to 160 servers and associated infrastructure.
More informationService Level Agreement (SLA) Arcplace Backup Enterprise Service
(SLA) Arcplace Backup Enterprise Service 1. Introduction This Service Level Agreement ( SLA ) forms an integral part of the Agreement between Arcplace and Customer. This SLA describes the Backup Enterprise
More informationBlackBerry Enterprise Server for Microsoft Exchange Version: 5.0 Service Pack: 2. Feature and Technical Overview
BlackBerry Enterprise Server for Microsoft Exchange Version: 5.0 Service Pack: 2 Feature and Technical Overview Published: 2010-06-16 SWDT305802-1108946-0615123042-001 Contents 1 Overview: BlackBerry Enterprise
More informationAn approach to grid scheduling by using Condor-G Matchmaking mechanism
An approach to grid scheduling by using Condor-G Matchmaking mechanism E. Imamagic, B. Radic, D. Dobrenic University Computing Centre, University of Zagreb, Croatia {emir.imamagic, branimir.radic, dobrisa.dobrenic}@srce.hr
More informationSee-GRID Project and Business Model
Improvements of the grid infrastructure and services within SEE-GRID Anastas Misev MARNET/MARGI/UKIM Macedonia Introduction SEE-GRID Project series SEE-GRID establish infrastructure SEE-GRID-2 extend infrastructure,
More informationObjectivity Data Migration
Objectivity Data Migration M. Nowak, K. Nienartowicz, A. Valassi, M. Lübeck, D. Geppert CERN, CH-1211 Geneva 23, Switzerland In this article we describe the migration of event data collected by the COMPASS
More informationOnCommand Performance Manager 1.1
OnCommand Performance Manager 1.1 Installation and Setup Guide For Red Hat Enterprise Linux NetApp, Inc. 495 East Java Drive Sunnyvale, CA 94089 U.S. Telephone: +1 (408) 822-6000 Fax: +1 (408) 822-4501
More informationWindows Server 2003 default services
Windows Server 2003 default services To view a description for a particular service, hover the mouse pointer over the service in the Name column. The descriptions included here are based on Microsoft documentation.
More informationConfiguration Guide BES12. Version 12.3
Configuration Guide BES12 Version 12.3 Published: 2016-01-19 SWD-20160119132230232 Contents About this guide... 7 Getting started... 8 Configuring BES12 for the first time...8 Configuration tasks for managing
More informationGridICE: monitoring the user/application activities on the grid
GridICE: monitoring the user/application activities on the grid Cristina Aiftimiei 1, Sergio Andreozzi 2, Guido Cuscela 3, Stefano Dal Pra 1, Giacinto Donvito 3, Vihang Dudhalkar 3, Sergio Fantinel 4,
More informationAlfresco Enterprise on AWS: Reference Architecture
Alfresco Enterprise on AWS: Reference Architecture October 2013 (Please consult http://aws.amazon.com/whitepapers/ for the latest version of this paper) Page 1 of 13 Abstract Amazon Web Services (AWS)
More informationAutomated deployment of virtualization-based research models of distributed computer systems
Automated deployment of virtualization-based research models of distributed computer systems Andrey Zenzinov Mechanics and mathematics department, Moscow State University Institute of mechanics, Moscow
More informationCLOUD BASED SERVICE (CBS STORAGE)
CLOUD BASED SERVICE (CBS STORAGE) Defining next generation of cloud based grid Power All Networks Ltd. Technical Whitepaper September 2008, version 1.04 PAGE 1 Table of Content Cloud Based Services (CBS
More informationAlliance Key Manager A Solution Brief for Technical Implementers
KEY MANAGEMENT Alliance Key Manager A Solution Brief for Technical Implementers Abstract This paper is designed to help technical managers, product managers, and developers understand how Alliance Key
More informationStorage Virtualization. Andreas Joachim Peters CERN IT-DSS
Storage Virtualization Andreas Joachim Peters CERN IT-DSS Outline What is storage virtualization? Commercial and non-commercial tools/solutions Local and global storage virtualization Scope of this presentation
More informationComparing Microsoft SQL Server 2005 Replication and DataXtend Remote Edition for Mobile and Distributed Applications
Comparing Microsoft SQL Server 2005 Replication and DataXtend Remote Edition for Mobile and Distributed Applications White Paper Table of Contents Overview...3 Replication Types Supported...3 Set-up &
More informationPRIVACY, SECURITY AND THE VOLLY SERVICE
PRIVACY, SECURITY AND THE VOLLY SERVICE Delight Delivered by EXECUTIVE SUMMARY The Volly secure digital delivery service from Pitney Bowes is a closed, secure, end-to-end system that consolidates and delivers
More informationConfiguration Guide BES12. Version 12.2
Configuration Guide BES12 Version 12.2 Published: 2015-07-07 SWD-20150630131852557 Contents About this guide... 8 Getting started... 9 Administrator permissions you need to configure BES12... 9 Obtaining
More informationAcano solution. Security Considerations. August 2015 76-1026-01-E
Acano solution Security Considerations August 2015 76-1026-01-E Contents Contents 1 Introduction... 3 2 Acano Secure Development Lifecycle... 3 3 Acano Security Points... 4 Acano solution: Security Consideration
More informationWindows 2000 Security Architecture. Peter Brundrett Program Manager Windows 2000 Security Microsoft Corporation
Windows 2000 Security Architecture Peter Brundrett Program Manager Windows 2000 Security Microsoft Corporation Topics Single Sign-on Kerberos v5 integration Active Directory security Delegation of authentication
More informationActive Directory Compatibility with ExtremeZ-IP
Active Directory Compatibility with ExtremeZ-IP A Technical Best Practices White Paper Group Logic White Paper October 2010 About This Document The purpose of this technical paper is to discuss how ExtremeZ-IP
More informationActiveVOS Server Architecture. March 2009
ActiveVOS Server Architecture March 2009 Topics ActiveVOS Server Architecture Core Engine, Managers, Expression Languages BPEL4People People Activity WS HT Human Tasks Other Services JMS, REST, POJO,...
More informationConfiguration Guide BES12. Version 12.1
Configuration Guide BES12 Version 12.1 Published: 2015-04-22 SWD-20150422113638568 Contents Introduction... 7 About this guide...7 What is BES12?...7 Key features of BES12... 8 Product documentation...
More informationData Management System for grid and portal services
Data Management System for grid and portal services Piotr Grzybowski 1, Cezary Mazurek 1, Paweł Spychała 1, Marcin Wolski 1 1 Poznan Supercomputing and Networking Center, ul. Noskowskiego 10, 61-704 Poznan,
More informationMichał Jankowski Maciej Brzeźniak PSNC
National Data Storage - architecture and mechanisms Michał Jankowski Maciej Brzeźniak PSNC Introduction Assumptions Architecture Main components Deployment Use case Agenda Data storage: The problem needs
More informationOracle WebLogic Server 11g: Administration Essentials
Oracle University Contact Us: 1.800.529.0165 Oracle WebLogic Server 11g: Administration Essentials Duration: 5 Days What you will learn This Oracle WebLogic Server 11g: Administration Essentials training
More informationCHAPTER 1 - JAVA EE OVERVIEW FOR ADMINISTRATORS
CHAPTER 1 - JAVA EE OVERVIEW FOR ADMINISTRATORS Java EE Components Java EE Vendor Specifications Containers Java EE Blueprint Services JDBC Data Sources Java Naming and Directory Interface Java Message
More informationAngel Dichev RIG, SAP Labs
Enabling SSL and Client Certificates on the SAP J2EE Engine Angel Dichev RIG, SAP Labs Learning Objectives As a result of this session, you will be able to: Understand the different SAP J2EE Engine SSL
More informationTroubleshooting BlackBerry Enterprise Service 10 version 10.1.1 726-08745-123. Instructor Manual
Troubleshooting BlackBerry Enterprise Service 10 version 10.1.1 726-08745-123 Instructor Manual Published: 2013-07-02 SWD-20130702091645092 Contents Advance preparation...7 Required materials...7 Topics
More information