The DESY Big-Data Cloud Service

Size: px
Start display at page:

Download "The DESY Big-Data Cloud Service"

Transcription

1 The DESY Big-Data Cloud Service Peter van der Reest On behalf of the project team Slides by Patrick Fuhrmann The DESY BIG DATA Cloud Service PvdR HEPiX Spring

2 Content mo(va(on project goals suggested solu(on and components quick introduc(on of owncloud dcache the proposed hybrid System status and issues The DESY BIG DATA Cloud Service PvdR HEPiX Spring

3 about DESY science program Photon Science Petra III, Flash Center for FEL research: CFEL European X- Ray Free Electron Laser: EXFEL Center for structural system biology: CSSB Accelerator Research and Development High Energy Physics & Astropar(cle Physics Outphasing: HERA with Zeus, H1, Hermes, HERA- B WLCG (Atlas, CMS and LHCb), Belle I and II, ILC Cherenkov Telescope Array: CTA IceCube The DESY BIG DATA Cloud Service PvdR HEPiX Spring

4 Why suddenly Cloud? due to the well publicized poli(cal affairs, DESY is banning all non- local mail and storage providers: DESY data should be kept at DESY for mail we had a replacement right away no instant replacement for DropBox, GoogleDrive solu(on had to be available asap so we had to design and engineer a well featured cloud storage system for DESY within months The DESY BIG DATA Cloud Service PvdR HEPiX Spring

5 Project Goal currently maintained storage systems are focused on Scien(fic Big Data access with POSIX seman(cs sharing via ACLs customers, especially new/young communi(es (Photon Science), are reques(ng Cloud storage seman(cs project objec(ve: installa(on of a modern Cloud Storage System for scien(sts within 6 months integrated into the exis(ng AAI and storage infrastructure if possible: Reducing amount of exis(ng systems The DESY BIG DATA Cloud Service PvdR HEPiX Spring

6 We had to find out what Cloud means for our scien(fic customers. Big Data management support of scien(fic data lifecycle Web 2.0 feeling The DESY BIG DATA Cloud Service PvdR HEPiX Spring

7 Big Data management? unlimited storage space, pay per use quotas are a no go and/or pointless indestruc(ble data store, never loosing data Amazon S3 is designed to provide % durability of objects over a given year. For example, if you store 10,000 objects with Amazon S3, you can on average expect to incur a loss of a single object once every 10,000,000 years. offering Quali(es of Service (related to costs) Access Latency (how long do I have to wait) Reten(on Policy (how safe is my data, durability) extremely high availability of storage service No regular maintenance breaks beyond once a year, 4 days The DESY BIG DATA Cloud Service PvdR HEPiX Spring

8 Scien(fic Data Lifecycle High Speed Data Ingest Fast Analysis NFS 4.1/pNFS Visualiza(on & Sharing by WebDAV Wide Area Transfers (Globus Online, FTS) by GridFTP The DESY BIG DATA Cloud Service PvdR HEPiX Spring

9 The Web 2.0 experience? easy sharing with registered users and groups the public (publishing, anonymous sharing) synchronizing (bidirec(onally) with all relevant OS es access from mobile devices, preferably using upload/download integrated in OS web browser access and configura(on The DESY BIG DATA Cloud Service PvdR HEPiX Spring

10 The DESY Cloud What does that mean for DESY? Big Data Part Web 2.0? Here we need some help The DESY BIG DATA Cloud Service PvdR HEPiX Spring

11 Web 2.0 Cloud interface for the web 2.0 interface we need some experts reusing previous evalua(on (different context) going for the most popular solu(on reduce likelihood for product disappearing possibly building a user- community Germany: TU- Berlin, FZ- Jülich, TU- Dresden, interna(onally: CERN, HEPiX, United Na(ons The DESY BIG DATA Cloud Service PvdR HEPiX Spring

12 What exactly do we need from owncloud sync clients for all OS s upload/download clients for mobile devices sharing of data with individuals and groups (including public links) web browser based file access and configura(on The DESY BIG DATA Cloud Service PvdR HEPiX Spring

13 Now, what s a dcache? The DESY BIG DATA Cloud Service PvdR HEPiX Spring

14 dcache cheat - sheet dcache.org is an interna(onal Collabora(on, composed of developers and support people from DESY, Fermilab, NDGF and the HTW Berlin dcache is operated on about 70 sites around the world total space about 120 Petabytes we store 50 % of the en(re WLCG storage biggest dcache holds about 50 Petabytes larges dcache spans 4 countries The DESY BIG DATA Cloud Service PvdR HEPiX Spring

15 dcache spec for Dummies NFS/pNFS hpp/webdav gridftp xrootd, dcap unlimited hierarchical storage space Virtual File- system Layer dcache Automa(c and Manual Media transi(ons SSDs spinning disks tape, BlueRay The DESY BIG DATA Cloud Service PvdR HEPiX Spring

16 Star(ng with possibly the biggest 40 PBytes Tape 770 Write Pools 420 Read Pools 26 Stage Pools US- CMS Tier I 14 PBytes on Disk *** Total: 260 Doors 6 Head 280 Pool/Door Physical Hosts Informa(on provided by Catalin Dumitrescu and Dmitry Litvintsev The DESY BIG DATA Cloud Service PvdR HEPiX Spring

17 To certainly the most widespread 4 Countries HPC Center North One dcache Uni of Bergen PDC Uni of Oslo CSC dcache head node Nordu Net National Supercomputer Center Slide stolen from Mattias Wadenstein, NDGF The DESY BIG DATA Cloud Service PvdR HEPiX Spring

18 To very likely the smallest One Machine One Process NFS 4.1 Door WebDAV Door PoolManager Pool 1 TB gplazma 700 MHz ARM 512 MB Memory 2 * USB MB Ethernet The DESY BIG DATA Cloud Service PvdR HEPiX Spring

19 dcache cheat - sheet (cont) protocol support NFS 4.1 / pnfs (scalable NFS) WebDAV GridFTP (Grid transfers) xrootd dcap authen(ca(on and authoriza(on support Kerberos user / password X509 (Cer(ficates and Proxies) LDAP/NIS or other informa(on providers The DESY BIG DATA Cloud Service PvdR HEPiX Spring

20 What do we need from dcache massive scale out managed space (up(me, availability) migra(on between media and decommissioning of hardware w/o down(me. mul( protocol access (scien(fic use case) NFS, CDMI(Cloud), WebDAV, gridftp(globusonline) Service Classes with automa(c and manual transi(ons (Access Latency, Reten(on Policy) hot spot detec(on storage (ering: tape, spinning disk, SSD s The DESY BIG DATA Cloud Service PvdR HEPiX Spring

21 What does the integra(on look like? The DESY BIG DATA Cloud Service PvdR HEPiX Spring

22 WEB 2.0 sync & share dcache owncloud Integra(on unlimited hierarchical storage space NFS 4.1 GridFTP, WebDAV dcache SSDs spinning disks tape, BlueRay The DESY BIG DATA Cloud Service PvdR HEPiX Spring

23 dcache owncloud Scien(fic Data Lifecycle GridFTP Unlimited hierarchical Storage Space Globus Online NFS 4.1 / pnfs HPC, HTC The DESY BIG DATA Cloud Service PvdR HEPiX Spring

24 dcache owncloud What does it look like for the user My dcache XXL Home My owncloud Home Sync Share Web 2.0 NFS 4.1/pNFS GridFTP WebDAV SRM (some private Grid Protocols) dcap xrootd The DESY BIG DATA Cloud Service PvdR HEPiX Spring

25 dcache owncloud Scalability (NFS4.1/pNFS does it) NFS Client NFS Client NFS Client pnfs Door pnfs Door pnfs Door The DESY BIG DATA Cloud Service PvdR HEPiX Spring

26 dcache OwnCloud integra(on simply running owncloud on dcache was the easy bit and works nicely dcache provides an NFSv4.1/pNFS interface which lets it look like a regular file system this is exactly what owncloud needs the fact the dcache doesn t allow files to be modified doesn t really bother owncloud. The DESY BIG DATA Cloud Service PvdR HEPiX Spring

27 But how about ownership? Ownership files owned by patrick in OwnCloud are owned by apache/owncloud in dcache this prevents us from using the same data with NFS4.1, gridftp or CDMI from dcache Tigran Mkrtchyan solved that issue: files now owned by patrick, but fully accessible by apache/owncloud dcache ACL s versus OwnCloud Sharing files shared in OwnCloud should ideally have similar ACLs in dcache data shared in owncloud is not automa(cally shared in dcache The DESY BIG DATA Cloud Service PvdR HEPiX Spring

28 sharing & ACL issue Web 2.0 Sync Share DESY LDAP Kerberos NFS WebDAV, GridFTP, CDMI The DESY BIG DATA Cloud Service PvdR HEPiX Spring

29 More issues besides the permission one The DESY BIG DATA Cloud Service PvdR HEPiX Spring

30 Name Space Issue We have We need Patrick Patrick Helge Helge Sandy Sandy The DESY BIG DATA Cloud Service PvdR HEPiX Spring

31 What we need WebDAV redirec(on to our nodes WebDAV/hpp redirect The DESY BIG DATA Cloud Service PvdR HEPiX Spring

32 What actually would be good instead of requiring a mounted filesystem (POSIX) for owncloud primary space, a network API/protocol would be beper best would be a standard (e.g. Cloud Data Management Interface, CDMI) CDMI is provided by big vendors allows to handle meta data, user and ownership as well The DESY BIG DATA Cloud Service PvdR HEPiX Spring

33 What s done we already installed two systems one connected to the DESY LDAP for DESY account holders one with the dcache.org private cloud for HTW students (different user contract J ) self registra(on with any valid cer(ficate most features are already available ordering more hardware about 200 Terabytes on top of the 100 Terabytes which are already deployed in above two systems The DESY BIG DATA Cloud Service PvdR HEPiX Spring

34 What s s(ll missing? access to owncloud defined in DESY User Registry resul(ng in group membership in DESY LDAP the plauorm adapter needs to be completed full account lifecycle: create, expire, archive, change group membership, terminate service etc external users integra(on, e.g. light- weight accs customizing the owncloud name space to support our scheme evalua(on of a owncloud sync client working against dcache directly by HTW student The DESY BIG DATA Cloud Service PvdR HEPiX Spring

35 ToDo: Tes(ng and verifica(on defining a set of reproducible test, which we can run on about lab machines verify scalability check against future dcache or owncloud updates func(onal performance The DESY BIG DATA Cloud Service PvdR HEPiX Spring

36 Further (meline we expect to have a full pre- produc(on system ready in about 6-8 weeks DESY IT colleagues and HTW students will con(nue to be guinea pigs (or anyone managing to find the dcache.org cloud storage registra(on page) next report at HEPiX Fall 2014 The DESY BIG DATA Cloud Service PvdR HEPiX Spring

37 further reading The DESY BIG DATA Cloud Service PvdR HEPiX Spring

38 dcache Big Data Cloud LOFAR antenna Huge amounts of data X- FEL (Free Electron Lasers) Fast Ingest The DESY BIG DATA Cloud Service PvdR HEPiX Spring

The DESY Big-Data Cloud System

The DESY Big-Data Cloud System The DESY Big-Data Cloud System Patrick Fuhrmann On behave of the project team The DESY BIG DATA Cloud Service Berlin Cloud Event Patrick Fuhrmann 5 May 2014 1 Content (on a good day) About DESY Project

More information

The Big-Data Cloud. Patrick Fuhrmann. On behave of the project team. The BIG DATA Cloud 8 th dcache Workshop, DESY Patrick Fuhrmann 15 May 2014 1

The Big-Data Cloud. Patrick Fuhrmann. On behave of the project team. The BIG DATA Cloud 8 th dcache Workshop, DESY Patrick Fuhrmann 15 May 2014 1 The Big-Data Cloud Patrick Fuhrmann On behave of the project team The BIG DATA Cloud 8 th dcache Workshop, DESY Patrick Fuhrmann 15 May 2014 1 Content About DESY Project Goals Suggested Solution and components

More information

dcache, Sharing, Sync`ing Big Data and Cloud Strategies

dcache, Sharing, Sync`ing Big Data and Cloud Strategies dcache, Sharing, Sync`ing Big Data and Cloud Strategies LSDMA Topics Meeting on Sharing data 2014, Jűlich Patrick Fuhrmann dcache Sharing Strategy LSDMA Topic Meeting: Sharing Patrick Fuhrmann 03 Apr 2014

More information

dcache, list of topics

dcache, list of topics dcache, list of topics EGI Meeting on H2020 Patrick Fuhrmann dcache EIG Meeting Patrick Fuhrmann 22 October 2013 1 Content The project structure Project funding, customers and contacts Current work areas

More information

Patrick Fuhrmann. The DESY Storage Cloud

Patrick Fuhrmann. The DESY Storage Cloud The DESY Storage Cloud Patrick Fuhrmann The DESY Storage Cloud Hamburg, 2/3/2015 for the DESY CLOUD TEAM Content > Motivation > Preparation > Collaborations and publications > What do you get right now?

More information

dcache, Software for Big Data

dcache, Software for Big Data dcache, Software for Big Data Innovation Day 2013, Berlin Patrick Fuhrmann dcache Innovation Day Berlin Patrick Fuhrmann 10 December 2013 1 About Technology and further roadmap Collaboration and partners

More information

dcache, large scale-out affordable storage

dcache, large scale-out affordable storage dcache, large scale-out affordable storage On behave of the team 1 Content People and Funding Deployment Supported protocols Data access Storage control Managed storage Design facts 2 People Funding 3

More information

The dcache Storage Element

The dcache Storage Element 16. Juni 2008 Hamburg The dcache Storage Element and it's role in the LHC era for the dcache team Topics for today Storage elements (SEs) in the grid Introduction to the dcache SE Usage of dcache in LCG

More information

QoS and DLC in IaaS INDIGO-DataCloud

QoS and DLC in IaaS INDIGO-DataCloud QoS and DLC in IaaS INDIGO-DataCloud Presenter : Patrick Fuhrmann Contributions by: Giacinto Donvito, INFN Marcus Hardt, KIT Paul Millar, DESY Alvaro Garcia, CSIC Zdenek Sustr, CESNET And many more INDIGO

More information

Scientific Storage at FNAL. Gerard Bernabeu Altayo Dmitry Litvintsev Gene Oleynik 14/10/2015

Scientific Storage at FNAL. Gerard Bernabeu Altayo Dmitry Litvintsev Gene Oleynik 14/10/2015 Scientific Storage at FNAL Gerard Bernabeu Altayo Dmitry Litvintsev Gene Oleynik 14/10/2015 Index - Storage use cases - Bluearc - Lustre - EOS - dcache disk only - dcache+enstore Data distribution by solution

More information

DESYcloud: an owncloud & dcache update

DESYcloud: an owncloud & dcache update : an owncloud & dcache update Paul Millar (on behalf of team) : an owncloud & dcache update Cloud Services for Synchronisation and Sharing Zürich, Switzerland. 2016-01-18 2016-01-19 http://cs3.ethz.ch/

More information

Forschungszentrum Karlsruhe in der Helmholtz-Gemeinschaft. dcache Introduction

Forschungszentrum Karlsruhe in der Helmholtz-Gemeinschaft. dcache Introduction dcache Introduction Forschungszentrum Karlsruhe GmbH Institute for Scientific Computing P.O. Box 3640 D-76021 Karlsruhe, Germany Dr. http://www.gridka.de What is dcache? Developed at DESY and FNAL Disk

More information

dcache, a managed storage in grid

dcache, a managed storage in grid dcache, a managed storage in grid support and funding by Patrick for the dcache Team Topics Project Topology Why do we need storage elements in the grid world? The idea behind the LCG (glite) storage element.

More information

Managed Storage @ GRID or why NFSv4.1 is not enough. Tigran Mkrtchyan for dcache Team

Managed Storage @ GRID or why NFSv4.1 is not enough. Tigran Mkrtchyan for dcache Team Managed Storage @ GRID or why NFSv4.1 is not enough Tigran Mkrtchyan for dcache Team What the hell do physicists do? Physicist are hackers they just want to know how things works. In moder physics given

More information

Maurice Askinazi Ofer Rind Tony Wong. HEPIX @ Cornell Nov. 2, 2010 Storage at BNL

Maurice Askinazi Ofer Rind Tony Wong. HEPIX @ Cornell Nov. 2, 2010 Storage at BNL Maurice Askinazi Ofer Rind Tony Wong HEPIX @ Cornell Nov. 2, 2010 Storage at BNL Traditional Storage Dedicated compute nodes and NFS SAN storage Simple and effective, but SAN storage became very expensive

More information

Preview of a Novel Architecture for Large Scale Storage

Preview of a Novel Architecture for Large Scale Storage Preview of a Novel Architecture for Large Scale Storage Andreas Petzold, Christoph-Erdmann Pfeiler, Jos van Wezel Steinbuch Centre for Computing STEINBUCH CENTRE FOR COMPUTING - SCC KIT University of the

More information

Next Generation Tier 1 Storage

Next Generation Tier 1 Storage Next Generation Tier 1 Storage Shaun de Witt (STFC) With Contributions from: James Adams, Rob Appleyard, Ian Collier, Brian Davies, Matthew Viljoen HEPiX Beijing 16th October 2012 Why are we doing this?

More information

Untertitel durch Klicken bearbeiten. Big Data Management. DESY & HTW-Berlin Hamburg, Sept. 24th, 2015. Big Data Minds 2015 DESY-IT V.

Untertitel durch Klicken bearbeiten. Big Data Management. DESY & HTW-Berlin Hamburg, Sept. 24th, 2015. Big Data Minds 2015 DESY-IT V. Untertitel durch Klicken bearbeiten Big Data Management - Motor der Wissenschaft Volker Guelzow DESY & HTW-Berlin Hamburg, Sept. 24th, 2015 Big Data Minds 2015 DESY-IT V. Gülzow Seite 1 Introduction What

More information

SC09 Tutorial M06 Cluster Construc5on Tutorial

SC09 Tutorial M06 Cluster Construc5on Tutorial SC09 Tutorial M06 Cluster Construc5on Tutorial Paul Marshall, Michael Oberg Theron Voran, Ma=hew Woitaszek University of Colorado, Boulder Na5onal Center for Atmospheric Research Advanced topics Parallel

More information

CERN Cloud Storage Evaluation Geoffray Adde, Dirk Duellmann, Maitane Zotes CERN IT

CERN Cloud Storage Evaluation Geoffray Adde, Dirk Duellmann, Maitane Zotes CERN IT SS Data & Storage CERN Cloud Storage Evaluation Geoffray Adde, Dirk Duellmann, Maitane Zotes CERN IT HEPiX Fall 2012 Workshop October 15-19, 2012 Institute of High Energy Physics, Beijing, China SS Outline

More information

Data storage services at CC-IN2P3

Data storage services at CC-IN2P3 Centre de Calcul de l Institut National de Physique Nucléaire et de Physique des Particules Data storage services at CC-IN2P3 Jean-Yves Nief Agenda Hardware: Storage on disk. Storage on tape. Software:

More information

Leveraging the Hybrid Cloud For Complete Data Protec:on. Private Public Managed

Leveraging the Hybrid Cloud For Complete Data Protec:on. Private Public Managed Leveraging the Hybrid Cloud For Complete Data Protec:on Private Public Managed Agenda Introduc>ons What is the Hybrid Cloud Data Protec>on Key Concepts Backup vs. Disaster Recovery How can I achieve Complete

More information

IBM ELASTIC STORAGE SEAN LEE

IBM ELASTIC STORAGE SEAN LEE IBM ELASTIC STORAGE SEAN LEE Solution Architect Platform Computing Division IBM Greater China Group Agenda Challenges in Data Management What is IBM Elastic Storage Key Features Elastic Storage Server

More information

NERSC Archival Storage: Best Practices

NERSC Archival Storage: Best Practices NERSC Archival Storage: Best Practices Lisa Gerhardt! NERSC User Services! Nick Balthaser! NERSC Storage Systems! Joint Facilities User Forum on Data Intensive Computing! June 18, 2014 Agenda Introduc)on

More information

Mass Storage at GridKa

Mass Storage at GridKa Mass Storage at GridKa Forschungszentrum Karlsruhe GmbH Institute for Scientific Computing P.O. Box 3640 D-76021 Karlsruhe, Germany Dr. Doris Ressmann http://www.gridka.de 1 Overview What is dcache? Pool

More information

SURFsara Data Services

SURFsara Data Services SURFsara Data Services SUPPORTING DATA-INTENSIVE SCIENCES Mark van de Sanden The world of the many Many different users (well organised (international) user communities, research groups, universities,

More information

Scalus A)ribute Workshop. Paris, April 14th 15th

Scalus A)ribute Workshop. Paris, April 14th 15th Scalus A)ribute Workshop Paris, April 14th 15th Content Mo=va=on, objec=ves, and constraints Scalus strategy Scenario and architectural views How the architecture works Mo=va=on for this MCITN Storage

More information

VoIP Security How to prevent eavesdropping on VoIP conversa8ons. Dmitry Dessiatnikov

VoIP Security How to prevent eavesdropping on VoIP conversa8ons. Dmitry Dessiatnikov VoIP Security How to prevent eavesdropping on VoIP conversa8ons Dmitry Dessiatnikov DISCLAIMER All informa8on in this presenta8on is provided for informa8on purposes only and in no event shall Security

More information

dcache - Managed Storage - LCG Storage Element - HSM optimizer Patrick Fuhrmann, DESY for the dcache Team

dcache - Managed Storage - LCG Storage Element - HSM optimizer Patrick Fuhrmann, DESY for the dcache Team dcache - Managed Storage - LCG Storage Element - HSM optimizer, DESY for the dcache Team dcache is a joint effort between the Deutsches Elektronen Synchrotron (DESY) and the Fermi National Laboratory (FNAL)

More information

Secure Hybrid Cloud Infrastructure for Scien5fic Applica5ons

Secure Hybrid Cloud Infrastructure for Scien5fic Applica5ons Secure Hybrid Cloud Infrastructure for Scien5fic Applica5ons Project Members: Paula Eerola Miika Komu MaA Kortelainen Tomas Lindén Lirim Osmani Sasu Tarkoma Salman Toor (Presenter) salman.toor@helsinki.fi

More information

Clusters in the Cloud

Clusters in the Cloud Clusters in the Cloud Dr. Paul Coddington, Deputy Director Dr. Shunde Zhang, Compu:ng Specialist eresearch SA October 2014 Use Cases Make the cloud easier to use for compute jobs Par:cularly for users

More information

Mobility in the Modern Factory. Discussion of Mobile Adop7on for the Factories of the Future

Mobility in the Modern Factory. Discussion of Mobile Adop7on for the Factories of the Future Mobility in the Modern Factory Discussion of Mobile Adop7on for the Factories of the Future Talking Points History Lesson The Reasons for Going Mobile Mobile Infrastructure Mobile Device Security BYOD

More information

Direct NFS - Design considerations for next-gen NAS appliances optimized for database workloads Akshay Shah Gurmeet Goindi Oracle

Direct NFS - Design considerations for next-gen NAS appliances optimized for database workloads Akshay Shah Gurmeet Goindi Oracle Direct NFS - Design considerations for next-gen NAS appliances optimized for database workloads Akshay Shah Gurmeet Goindi Oracle Agenda Introduction Database Architecture Direct NFS Client NFS Server

More information

Panasas at the RCF. Fall 2005 Robert Petkus RHIC/USATLAS Computing Facility Brookhaven National Laboratory. Robert Petkus Panasas at the RCF

Panasas at the RCF. Fall 2005 Robert Petkus RHIC/USATLAS Computing Facility Brookhaven National Laboratory. Robert Petkus Panasas at the RCF Panasas at the RCF HEPiX at SLAC Fall 2005 Robert Petkus RHIC/USATLAS Computing Facility Brookhaven National Laboratory Centralized File Service Single, facility-wide namespace for files. Uniform, facility-wide

More information

Zadara Storage Cloud A whitepaper. @ZadaraStorage

Zadara Storage Cloud A whitepaper. @ZadaraStorage Zadara Storage Cloud A whitepaper @ZadaraStorage Zadara delivers two solutions to its customers: On- premises storage arrays Storage as a service from 31 locations globally (and counting) Some Zadara customers

More information

Analyzing Big Data with Splunk A Cost Effective Storage Architecture and Solution

Analyzing Big Data with Splunk A Cost Effective Storage Architecture and Solution Analyzing Big Data with Splunk A Cost Effective Storage Architecture and Solution Jonathan Halstuch, COO, RackTop Systems JHalstuch@racktopsystems.com Big Data Invasion We hear so much on Big Data and

More information

Balancing Usability and Security for Medical Devices

Balancing Usability and Security for Medical Devices Balancing Usability and Security for Medical Devices Ken Hoyme Adven&um Labs ken.hoyme@adven8umlabs.com Robert North, LLC bnorth@humancenteredstrategies.com March 17, 2014 3/17/2014 2014 Adven8um Labs

More information

DSS. High performance storage pools for LHC. Data & Storage Services. Łukasz Janyst. on behalf of the CERN IT-DSS group

DSS. High performance storage pools for LHC. Data & Storage Services. Łukasz Janyst. on behalf of the CERN IT-DSS group DSS High performance storage pools for LHC Łukasz Janyst on behalf of the CERN IT-DSS group CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/it Introduction The goal of EOS is to provide a

More information

owncloud Architecture Overview

owncloud Architecture Overview owncloud Architecture Overview Time to get control back Employees are using cloud-based services to share sensitive company data with vendors, customers, partners and each other. They are syncing data

More information

Understanding Object Storage and How to Use It

Understanding Object Storage and How to Use It SWIFTSTACK WHITEPAPER An IT Expert Guide: Understanding Object Storage and How to Use It November 2014 The explosion of unstructured data is creating a groundswell of interest in object storage, certainly

More information

Experience in integrating enduser cloud storage for CMS Analysis

Experience in integrating enduser cloud storage for CMS Analysis Experience in integrating enduser cloud storage for CMS Analysis Hassen Riahi CERN IT CS3 Workshop Zürich, 18 th January 2016 1/18/16 H. Riahi; CS3 workshop 2 Outline Overview Goals Integration architecture

More information

Cloudian The Storage Evolution to the Cloud.. Cloudian Inc. Pre Sales Engineering

Cloudian The Storage Evolution to the Cloud.. Cloudian Inc. Pre Sales Engineering Cloudian The Storage Evolution to the Cloud.. Cloudian Inc. Pre Sales Engineering Agenda Industry Trends Cloud Storage Evolu4on of Storage Architectures Storage Connec4vity redefined S3 Cloud Storage Use

More information

Update on the Cloud Demonstration Project

Update on the Cloud Demonstration Project Update on the Cloud Demonstration Project Khalil Yazdi and Steven Wallace Spring Member Meeting April 19, 2011 Project Par4cipants BACKGROUND Eleven Universi1es: Caltech, Carnegie Mellon, George Mason,

More information

Improvement Options for LHC Mass Storage and Data Management

Improvement Options for LHC Mass Storage and Data Management Improvement Options for LHC Mass Storage and Data Management Dirk Düllmann HEPIX spring meeting @ CERN, 7 May 2008 Outline DM architecture discussions in IT Data Management group Medium to long term data

More information

Running the scientific data archive

Running the scientific data archive Running the scientific data archive Costs, technologies, challenges Jos van Wezel STEINBUCH CENTRE FOR COMPUTING - SCC KIT University of the State of Baden-Württemberg and National Laboratory of the Helmholtz

More information

HyperQ Hybrid Flash Storage Made Easy White Paper

HyperQ Hybrid Flash Storage Made Easy White Paper HyperQ Hybrid Flash Storage Made Easy White Paper Parsec Labs, LLC. 7101 Northland Circle North, Suite 105 Brooklyn Park, MN 55428 USA 1-763-219-8811 www.parseclabs.com info@parseclabs.com sales@parseclabs.com

More information

Using Ac+ve Directory and LDAP for directory management kept in sync

Using Ac+ve Directory and LDAP for directory management kept in sync 1 2 Using Ac+ve Directory and LDAP for directory management kept in sync 3 3 different evalua+on a@empts star+ng in mid- 2000 s: First: Inconclusive reluctance to move off of homegrown system BlitzMail

More information

Cloud Computing Backgrounder

Cloud Computing Backgrounder Cloud Computing Backgrounder No surprise: information technology (IT) is huge. Huge costs, huge number of buzz words, huge amount of jargon, and a huge competitive advantage for those who can effectively

More information

CMS Tier-3 cluster at NISER. Dr. Tania Moulik

CMS Tier-3 cluster at NISER. Dr. Tania Moulik CMS Tier-3 cluster at NISER Dr. Tania Moulik What and why? Grid computing is a term referring to the combination of computer resources from multiple administrative domains to reach common goal. Grids tend

More information

Best Prac*ces for Deploying Oracle So6ware on Virtual Compute Appliance

Best Prac*ces for Deploying Oracle So6ware on Virtual Compute Appliance Best Prac*ces for Deploying Oracle So6ware on Virtual Compute Appliance CON7484 Jeff Savit Senior Technical Product Manager Oracle VM Product Management October 1, 2014 Safe Harbor Statement The following

More information

KIT Site Report. Andreas Petzold. www.kit.edu STEINBUCH CENTRE FOR COMPUTING - SCC

KIT Site Report. Andreas Petzold. www.kit.edu STEINBUCH CENTRE FOR COMPUTING - SCC KIT Site Report Andreas Petzold STEINBUCH CENTRE FOR COMPUTING - SCC KIT University of the State of Baden-Württemberg and National Laboratory of the Helmholtz Association www.kit.edu GridKa Tier 1 - Batch

More information

PROJECT PORTFOLIO SUITE

PROJECT PORTFOLIO SUITE ServiceNow So1ware Development manages Scrum or waterfall development efforts and defines the tasks required for developing and maintaining so[ware throughout the lifecycle, from incep4on to deployment.

More information

Integrating a heterogeneous and shared Linux cluster into grids

Integrating a heterogeneous and shared Linux cluster into grids Integrating a heterogeneous and shared Linux cluster into grids 1,2 1 1,2 1 V. Büge, U. Felzmann, C. Jung, U. Kerzel, 1 1 1 M. Kreps, G. Quast, A. Vest 1 2 DPG Frühjahrstagung March 28 31, 2006 Dortmund

More information

Analisi di un servizio SRM: StoRM

Analisi di un servizio SRM: StoRM 27 November 2007 General Parallel File System (GPFS) The StoRM service Deployment configuration Authorization and ACLs Conclusions. Definition of terms Definition of terms 1/2 Distributed File System The

More information

owncloud Architecture Overview

owncloud Architecture Overview owncloud Architecture Overview owncloud, Inc. 57 Bedford Street, Suite 102 Lexington, MA 02420 United States phone: +1 (877) 394-2030 www.owncloud.com/contact owncloud GmbH Schloßäckerstraße 26a 90443

More information

SwiftStack Filesystem Gateway Architecture

SwiftStack Filesystem Gateway Architecture WHITEPAPER SwiftStack Filesystem Gateway Architecture March 2015 by Amanda Plimpton Executive Summary SwiftStack s Filesystem Gateway expands the functionality of an organization s SwiftStack deployment

More information

MaxDeploy Ready. Hyper- Converged Virtualization Solution. With SanDisk Fusion iomemory products

MaxDeploy Ready. Hyper- Converged Virtualization Solution. With SanDisk Fusion iomemory products MaxDeploy Ready Hyper- Converged Virtualization Solution With SanDisk Fusion iomemory products MaxDeploy Ready products are configured and tested for support with Maxta software- defined storage and with

More information

HAMBURG ZEUTHEN. DESY Tier 2 and NAF. Peter Wegner, Birgit Lewendel for DESY-IT/DV. Tier 2: Status and News NAF: Status, Plans and Questions

HAMBURG ZEUTHEN. DESY Tier 2 and NAF. Peter Wegner, Birgit Lewendel for DESY-IT/DV. Tier 2: Status and News NAF: Status, Plans and Questions DESY Tier 2 and NAF Peter Wegner, Birgit Lewendel for DESY-IT/DV Tier 2: Status and News NAF: Status, Plans and Questions Basics T2: 1.5 average Tier 2 are requested by CMS-groups for Germany Desy commitment:

More information

Merit Member Conference 2015 Does Migra+ng to a Virtualized Data Center Make Sense in Higher Educa+on?

Merit Member Conference 2015 Does Migra+ng to a Virtualized Data Center Make Sense in Higher Educa+on? Merit Member Conference 2015 Does Migra+ng to a Virtualized Data Center Make Sense in Higher Educa+on? is underway with a pilot migra8on from a tradi8onal university data center to a scalable virtualized

More information

UNINETT Sigma2 AS: architecture and functionality of the future national data infrastructure

UNINETT Sigma2 AS: architecture and functionality of the future national data infrastructure UNINETT Sigma2 AS: architecture and functionality of the future national data infrastructure Authors: A O Jaunsen, G S Dahiya, H A Eide, E Midttun Date: Dec 15, 2015 Summary Uninett Sigma2 provides High

More information

File Services. File Services at a Glance

File Services. File Services at a Glance File Services High-performance workgroup and Internet file sharing for Mac, Windows, and Linux clients. Features Native file services for Mac, Windows, and Linux clients Comprehensive file services using

More information

High Performance Computing OpenStack Options. September 22, 2015

High Performance Computing OpenStack Options. September 22, 2015 High Performance Computing OpenStack PRESENTATION TITLE GOES HERE Options September 22, 2015 Today s Presenters Glyn Bowden, SNIA Cloud Storage Initiative Board HP Helion Professional Services Alex McDonald,

More information

Volunteer Computing, Grid Computing and Cloud Computing: Opportunities for Synergy. Derrick Kondo INRIA, France

Volunteer Computing, Grid Computing and Cloud Computing: Opportunities for Synergy. Derrick Kondo INRIA, France Volunteer Computing, Grid Computing and Cloud Computing: Opportunities for Synergy Derrick Kondo INRIA, France Outline Cloud Grid Volunteer Computing Cloud Background Vision Hide complexity of hardware

More information

DSS. The Data Storage Services (DSS) Strategy at CERN. Jakub T. Moscicki. (Input from J. Iven, M. Lamanna A. Pace, A. Peters and A.

DSS. The Data Storage Services (DSS) Strategy at CERN. Jakub T. Moscicki. (Input from J. Iven, M. Lamanna A. Pace, A. Peters and A. The Data Storage Services () Strategy at CERN Jakub T. Moscicki (Input from J. Iven, M. Lamanna A. Pace, A. Peters and A. Wiebalck) HEPiX Spring 2012 Workshop Prague, April 2012 The big picture Situation

More information

U"lizing the SDSC Cloud Storage Service

Ulizing the SDSC Cloud Storage Service U"lizing the SDSC Cloud Storage Service PASIG Conference January 13, 2012 Richard L. Moore rlm@sdsc.edu San Diego Supercomputer Center University of California San Diego SAN DIEGO SUPERCOMPUTER CENTER

More information

Return on Experience on Cloud Compu2ng Issues a stairway to clouds. Experts Workshop Nov. 21st, 2013

Return on Experience on Cloud Compu2ng Issues a stairway to clouds. Experts Workshop Nov. 21st, 2013 Return on Experience on Cloud Compu2ng Issues a stairway to clouds Experts Workshop Agenda InGeoCloudS SoCware Stack InGeoCloudS Elas2city and Scalability Elas2c File Server Elas2c Database Server Elas2c

More information

Storage Virtualization. Andreas Joachim Peters CERN IT-DSS

Storage Virtualization. Andreas Joachim Peters CERN IT-DSS Storage Virtualization Andreas Joachim Peters CERN IT-DSS Outline What is storage virtualization? Commercial and non-commercial tools/solutions Local and global storage virtualization Scope of this presentation

More information

The Development of Cloud Interoperability

The Development of Cloud Interoperability NSC- JST Workshop The Development of Cloud Interoperability Weicheng Huang Na7onal Center for High- performance Compu7ng Na7onal Applied Research Laboratories 1 Outline Where are we? Our experiences before

More information

Intel Solid- State Drive Data Center P3700 Series NVMe Hybrid Storage Performance

Intel Solid- State Drive Data Center P3700 Series NVMe Hybrid Storage Performance Intel Solid- State Drive Data Center P3700 Series NVMe Hybrid Storage Performance Hybrid Storage Performance Gains for IOPS and Bandwidth Utilizing Colfax Servers and Enmotus FuzeDrive Software NVMe Hybrid

More information

SOFTWARE-DEFINED STORAGE IN ACTION

SOFTWARE-DEFINED STORAGE IN ACTION SOFTWARE-DEFINED STORAGE IN ACTION Anne Tarantino VP Sales- Velocity Tech Solutions Copyright 2014 DataCore Software Corp. All Rights Reserved. Copyright 2014 DataCore Software Corp. All Rights Reserved.

More information

EMI Storage meets EMI security

EMI Storage meets EMI security EMI Storage meets EMI security Component/ Middleware glite (LFC,FTS,DPM,GFAL) ARC UNICORE StoRM dcache Staff With kind contributions by Oliver Keeble, Jean- Philippe Baud Jon Kerr Nilsen Ralph Müller-

More information

SURFsara HPC Cloud Workshop

SURFsara HPC Cloud Workshop SURFsara HPC Cloud Workshop www.cloud.sara.nl Tutorial 2014-06-11 UvA HPC and Big Data Course June 2014 Anatoli Danezi, Markus van Dijk cloud-support@surfsara.nl Agenda Introduction and Overview (current

More information

An Integrated Security System of Protec5ng Smart Grid against Cyber A9acks Authors: Dong Wei, Yan Lu, Mohsen Jafari, Paul Skare, and Kenneth Rhode

An Integrated Security System of Protec5ng Smart Grid against Cyber A9acks Authors: Dong Wei, Yan Lu, Mohsen Jafari, Paul Skare, and Kenneth Rhode An Integrated Security System of Protec5ng Smart Grid against Cyber A9acks Authors: Dong Wei, Yan Lu, Mohsen Jafari, Paul Skare, and Kenneth Rhode Presenter: Somo Peyechu Submi9ed in Par5al Fulfillment

More information

dcache Firewall Issue

dcache Firewall Issue dcache Firewall Issue Tigran Mkrtchyan for dcache Team Protocols and Ports 20000:25000 => gridftp in pools 33115:33145 => passive dcap & xrootd 8443 => SRM 22223 => ssh 22125 => dcap 22128 => gsidcap 1049

More information

Michał Jankowski Maciej Brzeźniak PSNC

Michał Jankowski Maciej Brzeźniak PSNC National Data Storage - architecture and mechanisms Michał Jankowski Maciej Brzeźniak PSNC Introduction Assumptions Architecture Main components Deployment Use case Agenda Data storage: The problem needs

More information

Large Scale Storage. Orlando Richards, Information Services orlando.richards@ed.ac.uk. LCFG Users Day, University of Edinburgh 18 th January 2013

Large Scale Storage. Orlando Richards, Information Services orlando.richards@ed.ac.uk. LCFG Users Day, University of Edinburgh 18 th January 2013 Large Scale Storage Orlando Richards, Information Services orlando.richards@ed.ac.uk LCFG Users Day, University of Edinburgh 18 th January 2013 Overview My history of storage services What is (and is not)

More information

Faster Cloud Backup: How It's Done

Faster Cloud Backup: How It's Done TECH BRIEF Faster Cloud Backup: How It's Done What You Need to Know One of the main concerns we hear from customers is around the ability to transmit large amounts of data over the Internet with their

More information

Enterprise Private Cloud Storage

Enterprise Private Cloud Storage Enterprise Private Cloud Storage The term cloud storage seems to have acquired many definitions. At Cloud Leverage, we define cloud storage as an enterprise-class file server located in multiple geographically

More information

Cloud Compu)ng in Educa)on and Research

Cloud Compu)ng in Educa)on and Research Cloud Compu)ng in Educa)on and Research Dr. Wajdi Loua) Sfax University, Tunisia ESPRIT - December 2014 04/12/14 1 Outline Challenges in Educa)on and Research SaaS, PaaS and IaaS for Educa)on and Research

More information

Splunk implementa-on. Our experiences throughout the 3 year journey

Splunk implementa-on. Our experiences throughout the 3 year journey Splunk implementa-on Our experiences throughout the 3 year journey About us Harvard University University Network Services Group Serving over 2500 faculty and more than 18,000 students Jim Donn Management

More information

INDIGO-DataCloud Wupi 4 (Resource Virtualization)

INDIGO-DataCloud Wupi 4 (Resource Virtualization) INDIGO-DataCloud Wupi 4 (Resource Virtualization) All stolen from Markus, Enol, Maciej, Giacionto and many others High level objective This work package is focusing on virtualizing local computing, storage

More information

Breaking the Storage Array Lifecycle with Cloud Storage

Breaking the Storage Array Lifecycle with Cloud Storage Breaking the Storage Array Lifecycle with Cloud Storage 2011 TwinStrata, Inc. The Storage Array Lifecycle Anyone who purchases storage arrays is familiar with the many advantages of modular storage systems

More information

OPTIMIZING PRIMARY STORAGE WHITE PAPER FILE ARCHIVING SOLUTIONS FROM QSTAR AND CLOUDIAN

OPTIMIZING PRIMARY STORAGE WHITE PAPER FILE ARCHIVING SOLUTIONS FROM QSTAR AND CLOUDIAN OPTIMIZING PRIMARY STORAGE WHITE PAPER FILE ARCHIVING SOLUTIONS FROM QSTAR AND CLOUDIAN CONTENTS EXECUTIVE SUMMARY The Challenges of Data Growth SOLUTION OVERVIEW 3 SOLUTION COMPONENTS 4 Cloudian HyperStore

More information

Amazon Cloud Storage Options

Amazon Cloud Storage Options Amazon Cloud Storage Options Table of Contents 1. Overview of AWS Storage Options 02 2. Why you should use the AWS Storage 02 3. How to get Data into the AWS.03 4. Types of AWS Storage Options.03 5. Object

More information

Introduction to NetApp Infinite Volume

Introduction to NetApp Infinite Volume Technical Report Introduction to NetApp Infinite Volume Sandra Moulton, Reena Gupta, NetApp April 2013 TR-4037 Summary This document provides an overview of NetApp Infinite Volume, a new innovation in

More information

Debunking Five Myths About Tape Storage. Tech Brief. an Internet.com Tech Brief. 2011, Internet.com, a division of QuinStreet, Inc.

Debunking Five Myths About Tape Storage. Tech Brief. an Internet.com Tech Brief. 2011, Internet.com, a division of QuinStreet, Inc. Debunking Five Myths About Tape Storage Tech Brief an Technological change is part of life, and that s true whether you re in your living room or in the data center at the office. Machines and devices

More information

Simple Storage Service (S3)

Simple Storage Service (S3) Simple Storage Service (S3) Amazon S3 is storage for the Internet. It is designed to make web-scale computing easier for developers. Amazon S3 provides a simple web services interface that can be used

More information

SUMMIT. November 2010

SUMMIT. November 2010 SUMMIT November 2010 Why Summit? Comprehensive Summit provides a unified approach to IT enterprise management following a prescriptive, ITIL based framework Rapid Deployment Summit is developed for and

More information

Tamanna Roy Rayat & Bahra Institute of Engineering & Technology, Punjab, India talk2tamanna@gmail.com

Tamanna Roy Rayat & Bahra Institute of Engineering & Technology, Punjab, India talk2tamanna@gmail.com IJCSIT, Volume 1, Issue 5 (October, 2014) e-issn: 1694-2329 p-issn: 1694-2345 A STUDY OF CLOUD COMPUTING MODELS AND ITS FUTURE Tamanna Roy Rayat & Bahra Institute of Engineering & Technology, Punjab, India

More information

Experience of Data Transfer to the Tier-1 from a DIRAC Perspective

Experience of Data Transfer to the Tier-1 from a DIRAC Perspective Experience of Data Transfer to the Tier-1 from a DIRAC Perspective Lydia Heck Institute for Computational Cosmology Manager of the DiRAC-2 Data Centric Facility COSMA 1 Talk layout Introduction to DiRAC?

More information

Data Management in the Cloud: Limitations and Opportunities. Annies Ductan

Data Management in the Cloud: Limitations and Opportunities. Annies Ductan Data Management in the Cloud: Limitations and Opportunities Annies Ductan Discussion Outline: Introduc)on Overview Vision of Cloud Compu8ng Managing Data in The Cloud Cloud Characteris8cs Data Management

More information

Quantum StorNext. Product Brief: Distributed LAN Client

Quantum StorNext. Product Brief: Distributed LAN Client Quantum StorNext Product Brief: Distributed LAN Client NOTICE This product brief may contain proprietary information protected by copyright. Information in this product brief is subject to change without

More information

Privacy- Preserving P2P Data Sharing with OneSwarm. Presented by. Adnan Malik

Privacy- Preserving P2P Data Sharing with OneSwarm. Presented by. Adnan Malik Privacy- Preserving P2P Data Sharing with OneSwarm Presented by Adnan Malik Privacy The protec?on of informa?on from unauthorized disclosure Centraliza?on and privacy threat Websites Facebook TwiFer Peer

More information

Nexenta Performance Scaling for Speed and Cost

Nexenta Performance Scaling for Speed and Cost Nexenta Performance Scaling for Speed and Cost Key Features Optimize Performance Optimize Performance NexentaStor improves performance for all workloads by adopting commodity components and leveraging

More information

Neil Stobart Cloudian Inc. CLOUDIAN HYPERSTORE Smart Data Storage

Neil Stobart Cloudian Inc. CLOUDIAN HYPERSTORE Smart Data Storage Neil Stobart Cloudian Inc. CLOUDIAN HYPERSTORE Smart Data Storage Storage is changing forever Scale Up / Terabytes Flash host/array Tradi/onal SAN/NAS Scalability / Big Data Object Storage Scale Out /

More information

StorPool Distributed Storage. Software-Defined. Business Overview

StorPool Distributed Storage. Software-Defined. Business Overview StorPool Distributed Storage. Software-Defined. Business Overview StorPool, 2015 Page 1 of 5 About StorPool StorPool is the leading vendor of distributed storage software. Our innovative solution eradicates

More information

Business Con*nuity with Docker

Business Con*nuity with Docker CloudOpen Japan 2015 Business Con*nuity with Docker 2015/06/04 Yoshitaka Kuwata Muroran Ins*tute of Technology Overview of Talk 1. Who is Talking 2. Mo*va*on of Disaster Recovery 3. Exis*ng Solu*ons 4.

More information

Red Hat Storage Server

Red Hat Storage Server Red Hat Storage Server Marcel Hergaarden Solution Architect, Red Hat marcel.hergaarden@redhat.com May 23, 2013 Unstoppable, OpenSource Software-based Storage Solution The Foundation for the Modern Hybrid

More information

Polish National Data Storage. Norbert Meyer, Maciej Brzeźniak, Maciej Stroiński PSNC

Polish National Data Storage. Norbert Meyer, Maciej Brzeźniak, Maciej Stroiński PSNC Polish National Data Storage Norbert Meyer, Maciej Brzeźniak, Maciej Stroiński PSNC Workshop on Big Data and Open Data, Brussels. May 7-8, 2014 Data = value => needs protection! Data is value:! Expensive

More information