Introduction to Grid computing

Size: px
Start display at page:

Download "Introduction to Grid computing"

Transcription

1 Introduction to Grid computing The INFNGrid Project Team

2 Introduction This tutorial has been implemented considering as starting point the DataGrid (EDG) tutorial Many thanks to the EDG tutorials team! Various slides and exercises come from the EDG tutorial Many changes to the above material have been applied E.g. information on Globus has been added by the INFNGrid Tutorials team

3 Overview What is a Grid? What is Grid computing? Why Grids? Grid projects world wide The European DataGrid project

4 What is a Grid? Dependable, consistent, pervasive access to resources Enable communities ( virtual organizations ) to share geographically distributed resources as they pursue common goals in the absence of central control, omniscience, trust relationships Make it easy to use diverse, geographically distributed, locally managed and controlled DB computing facilities as if they formed a coherent local cluster DB DB DB

5 What is Grid computing Coordinated resource sharing and problem solving in dynamic, multi-institutional virtual organizations [ I.Foster] A VO is a collection of users sharing similar needs and requirements in their access to processing, data and distributed resources and pursuing similar goals. Big challenge for computing with high sociological impact and a new way of collaborating

6 Researchers perform their activities regardless geographical location, interact with colleagues, share and access data The Grid Vision The Grid: networked data processing centres and middleware software as the glue of resources. Scientific instruments and experiments provide huge amount of data

7 The Grid distributed computing idea 1/2 Once upon a time.. mainframe Microcomputer Mini Computer Cluster (by Christophe Jacquet)

8 The Grid distributed computing idea 2/2 and today (by Christophe Jacquet) January, 2004 EGrid Project - Grid Tutorial, Trieste

9 Grid vs distributed computing Distributed computing Distributed applications tend to be specialised systems, intended for a single purpose or user group Resources are usually centrally managed and controlled, or at least they are managed using the same mechanisms, policies, tools Resources are usually homogeneous Static nature Grid Computing Resources locally managed and controlled The owner of the resource has the full control, and therefore different Grid resources can be managed using different policies and mechanisms, e.g. Computing resources can be managed by different local resource management systems Different storage systems (disk pools, mass storage systems, ) used in different Grid sites Different policies and priorities given to the same user in different Grid systems Different kind of resources Not always the same hardware and applications Dynamic nature Resources and users added/removed/changed frequently

10 Main entities of a Grid architecture Service providers Publish the availability of their services via information systems Such services may come-and-go or change dynamically E.g. a Grid site that offers x CPUs and y GB of storage Service brokers Register and categorize published services and provide search capabilities E.g. 1) Resource Broker selects the best resources for a job 2) Catalogues of data held at each Grid site Service requesters Single sign-on: log into the grid once Use brokering services to find a needed service and employ it E.g. scientist submits a simulation job that needs 12 CPUs for 6 hours and 15 GB which gets scheduled, via the Resource Broker, on a certain Grid site

11 Why Grids Distributed (world-wide) user collaborations Distributed computational and storage resources Scale of the problems Grids provide access to large data processing power and huge data storage possibilities As the Grid grows its usefulness increases (more resources available) Large communities of possible Grid users : High Energy Physics Environmental studies: Earthquakes forecast, geologic and climate changes, ozone monitoring Biology, Genetics, Earth Observation Astrophysics, New composite materials research Astronautics, etc.

12 High Energy Physics ATLAS The LHC Detectors CMS ~6-8 PetaBytes / year ~10 8 events/year ~10 3 batch and interactive users Federico.carminati, EU review presentation LHCb

13 Earth Observation ESA missions: about 100 Gbytes of data per day (ERS 1/2) 500 Gbytes, for the next ENVISAT mission (2002). Grid contribute to EO: enhance the ability to access high level products allow reprocessing of large historical archives improve Earth science complex applications (data fusion, data mining, modelling ) Federico.Carminati, EU review presentation, 1 March 2002 Source: L. Fusco, June 2001

14 Biology BioInformatics Bio-informatics Phylogenetics Search for primers Statistical genetics Bio-informatics web portal Parasitology Data-mining on DNA chips Geometrical protein comparison Medical imaging MR image simulation Medical data and metadata management Mammographies analysis Simulation platform for PET/SPECT 1. Query the medical image database and retrieve a patient image 3. Retrieve most similar cases Exam image patient key ACL... Metadata Medical images 2. Compute similarity measures over the database images Submit 1 job per image Similar images Low score images

15 Status (some years ago) People have been discussing about Grid for several years but till some years ago more or less only Globus toolkit available Globus toolkit: core services for Grid tools and applications (Security, Information service, Resource management, etc ) Good basis to build on but: No high level services Many problems (e.g. handling of high volumes of data) not addressed No production quality implementations Not possible to do real work with Grids yet Necessary to address the shortcomings and implement the missing (high level) services

16 Major existing Grid projects Europe-based projects: European DataGrid (EDG) : LHC Computing GRID (LCG): cern.ch/lcg CrossGrid : DataTAG : GridLab : EGEE : European National Projects: INFNGRID, UK-GridPP, NorduGrid(Nordic test bed for wide area computing )

17 Major existing Grid projects US projects: GriPhyN HEP PPDG HEP ivdgl ( joint GriPhyN, PPDG) TERAGRID (NSF) IBM, Intel Qwest,Myricom, Sun Microsystems, Oracle. National Middleware Initiative (NSF NMI) ESG NEESgrid virtual lab earthquake engineering BIRN biomedical informatics research network birn.ncrr.nih.gov/birn/ Asia-based projects: ApGRID TWGRID Many Grid projects in : Korea, Japan, China, Australia

18 Global Grid Forum (GGF) Mission To focus on the promotion and development of Grid technologies and applications via the development and documentation of "best practices," implementation guidelines, and standards with an emphasis on "rough consensus and running code" An Open Process for Development of Standards A Forum for Information Exchange A Regular Gathering to Encourage Shared Effort See

19 The European Data Grid (EDG) Project To build on the emerging Grid technology to develop a sustainable computing model for effective share of computing resources and data Applications/End Users Communities: HEP, Earth Observation, Biology Start : Jan 1, 2001 End : Dec 31, 2003 Specific project objectives: Middleware for fabric & Grid management Large scale testbed To collaborate with and complement other European and US projects To contribute to Open Standards and international bodies: DataGrid co-founder of Global Grid Forum (GGF)

20 DataGrid in Numbers Testbeds People >15 regular sites >350 registered users 12 Virtual Organisations >10 000s jobs submitted 16 Certificate Authorities >1000 CPUs >5 TeraBytes disk >500 people trained 3 Mass Storage Systems 278 man-years of effort 100 years funded Software 50 use cases 18 software releases Scientific applications 5 Earth Obs institutes 9 bio-informatics apps 6 HEP experiments >300K lines of code January, 2004 EGrid Project - Grid Tutorial, Trieste

21 EDG software architecture EDG architectural functional blocks: Basic Services (authentication, authorization, Replica Catalog, secure file transfer, Info Providers) rely on VDT (basically Globus + Condor) Higher level EDG middleware (developed within EDG) Applications (HEP,BIO,EO) Specific application layer VOs common application layer Grid middleware VDT ALICE ATLAS CMS LHCb Other apps LHC Other apps High level Grid middleware Basic Services OS & Net services

22 EDG middleware Grid architecture Local Computing Local Application Local Database APPLICATIONS Grid Grid Application Layer Job Management Data Management Metadata Management Collective Services Grid Scheduler Replica Manager Information & Monitoring Underlying Grid Services Grid SQL Database Services Computing Element Services Storage Element Services Replica Catalog Authorization Authentication and Accounting Service Index M / W Fabric Fabric services Resource Management Configuration Management Monitoring and Fault Tolerance Node Installation & Management Fabric Storage Management GLOBUS

23 Current EDG Status EDG currently provides a set of middleware services Job & Data Management Grid Information Services Grid & Network monitoring Security, Authentication & Authorization tools Fabric Management EDG release 2.1 currently deployed to the EDG-Testbeds GNU/Linux RedHat 7.3 on Intel PCs ~15 sites in application testbed actively used by application groups EDG software used also in other Grid infrastructures: Deployed in the LCG Grid infrastructure (for LHC experiments) EDG sw also deployed at total of ~40 sites via CrossGrid, DataTAG and national grid projects Deployed in the GRID.IT grid (used by different scientific applications) Many applications ported to EDG testbeds and actively being used Activities will continue in the framework of the EGEE project

24 Conclusions Grids enable distributed communities ( virtual organizations ) to share geographically distributed resources Many achievements in the last years Achievements in Grid middleware functionality and stability More and more applications using the Grid Also industry is joining the Grid but still a lot to do Going towards standardizations (OGSA, etc.) Going from Grid prototypes to Grid infrastructures to be used for real production activities

25 28/9/2003: Computational grids finally reached the same level of stability of electrical power grids!

Overview of the EU Data Grid Project

Overview of the EU Data Grid Project DataGrid is a project funded by the European Union Grid Tutorial 8/10/2003 n 1 Overview of the EU Data Grid Project The European DataGrid Project Team http://www.eu-datagrid.org The EU DataGrid (EDG) Project

More information

Analyses on functional capabilities of BizTalk Server, Oracle BPEL Process Manger and WebSphere Process Server for applications in Grid middleware

Analyses on functional capabilities of BizTalk Server, Oracle BPEL Process Manger and WebSphere Process Server for applications in Grid middleware Analyses on functional capabilities of BizTalk Server, Oracle BPEL Process Manger and WebSphere Process Server for applications in Grid middleware R. Goranova University of Sofia St. Kliment Ohridski,

More information

Concepts and Architecture of Grid Computing. Advanced Topics Spring 2008 Prof. Robert van Engelen

Concepts and Architecture of Grid Computing. Advanced Topics Spring 2008 Prof. Robert van Engelen Concepts and Architecture of Grid Computing Advanced Topics Spring 2008 Prof. Robert van Engelen Overview Grid users: who are they? Concept of the Grid Challenges for the Grid Evolution of Grid systems

More information

MIGRATING DESKTOP AND ROAMING ACCESS. Migrating Desktop and Roaming Access Whitepaper

MIGRATING DESKTOP AND ROAMING ACCESS. Migrating Desktop and Roaming Access Whitepaper Migrating Desktop and Roaming Access Whitepaper Poznan Supercomputing and Networking Center Noskowskiego 12/14 61-704 Poznan, POLAND 2004, April white-paper-md-ras.doc 1/11 1 Product overview In this whitepaper

More information

The GENIUS Grid Portal

The GENIUS Grid Portal The GENIUS Grid Portal (*) work in collaboration with A. Falzone and A. Rodolico EGEE NA4 Workshop, Paris, 18.12.2003 CHEP 2000, 10.02.2000 Outline Introduction Grid portal architecture and requirements

More information

Forschungszentrum Karlsruhe in der Helmholtz-Gemeinschaft. Global Grid User Support - GGUS - within the LCG & EGEE environment

Forschungszentrum Karlsruhe in der Helmholtz-Gemeinschaft. Global Grid User Support - GGUS - within the LCG & EGEE environment Global Grid User Support - GGUS - within the LCG & EGEE environment Abstract: For very large projects like the LHC Computing Grid Project (LCG) involving some 8,000 scientists from universities and laboratories

More information

Global Grid User Support - GGUS - in the LCG & EGEE environment

Global Grid User Support - GGUS - in the LCG & EGEE environment Global Grid User Support - GGUS - in the LCG & EGEE environment Torsten Antoni (torsten.antoni@iwr.fzk.de) Why Support? New support groups Network layer Resource centers CIC / GOC / etc. more to come New

More information

DataGrid Architecture Models V0.2 Lee Momtahan 02/04/02

DataGrid Architecture Models V0.2 Lee Momtahan 02/04/02 DataGrid Architecture Models V0.2 Lee Momtahan leemo@comlab.ox.ac.uk 02/04/02 Introduction The purpose of this document is to communicate a clear view of the DataGrid Architecture amongst DataGrid Application

More information

Data Management in an International Data Grid Project. Timur Chabuk 04/09/2007

Data Management in an International Data Grid Project. Timur Chabuk 04/09/2007 Data Management in an International Data Grid Project Timur Chabuk 04/09/2007 Intro LHC opened in 2005 several Petabytes of data per year data created at CERN distributed to Regional Centers all over the

More information

INFN Testbed status report

INFN Testbed status report L. Gaido Oxford July, 2-5 2001 1 Dedicated resources (available now) Quantum Grid: 3-4 PCs in 15 sites: Bari, Bologna, Cagliari, Catania, Cnaf, Ferrara, Lecce, Legnaro, Milano, Napoli, Padova, Parma, Pisa,

More information

The Lattice Project: A Multi-Model Grid Computing System. Center for Bioinformatics and Computational Biology University of Maryland

The Lattice Project: A Multi-Model Grid Computing System. Center for Bioinformatics and Computational Biology University of Maryland The Lattice Project: A Multi-Model Grid Computing System Center for Bioinformatics and Computational Biology University of Maryland Parallel Computing PARALLEL COMPUTING a form of computation in which

More information

WLCG and Grid computing at CERN

WLCG and Grid computing at CERN WLCG and Grid computing at CERN Andrea Sciabà Visit of the Korean National Health Insurance Service 18 March 2016, CERN, Switzerland What is the LHC The LHC is a particle accelerator 27 km of superconducting

More information

Instruments in Grid: the New Instrument Element

Instruments in Grid: the New Instrument Element Instruments in Grid: the New Instrument Element C. Vuerli (1,2), G. Taffoni (1,2), I. Coretti (1), F. Pasian (1,2), P. Santin (1), M. Pucillo (1) (1) INAF Astronomical Observatory of Trieste (2) INAF Informative

More information

A demonstration of the use of Datagrid testbed and services for the biomedical community

A demonstration of the use of Datagrid testbed and services for the biomedical community A demonstration of the use of Datagrid testbed and services for the biomedical community Biomedical applications work package V. Breton, Y Legré (CNRS/IN2P3) R. Météry (CS) Credits : C. Blanchet, T. Contamine,

More information

CMS Dashboard of Grid Activity

CMS Dashboard of Grid Activity Enabling Grids for E-sciencE CMS Dashboard of Grid Activity Julia Andreeva, Juha Herrala, CERN LCG ARDA Project, EGEE NA4 EGEE User Forum Geneva, Switzerland March 1-3, 2006 http://arda.cern.ch ARDA and

More information

Earth Observation. 86 esa bulletin 114 - may 2003 www.esa.int

Earth Observation. 86 esa bulletin 114 - may 2003 www.esa.int Earth Observation 86 esa bulletin 114 - may 2003 www.esa.int Earth Observation on the Grid Luigi Fusco & Pedro Goncalves ESA Earth Observation Science and Application Department, ESRIN, Frascati, Italy

More information

Cluster, Grid, Cloud Concepts

Cluster, Grid, Cloud Concepts Cluster, Grid, Cloud Concepts Kalaiselvan.K Contents Section 1: Cluster Section 2: Grid Section 3: Cloud Cluster An Overview Need for a Cluster Cluster categorizations A computer cluster is a group of

More information

The Grid-it: the Italian Grid Production infrastructure

The Grid-it: the Italian Grid Production infrastructure n 1 Maria Cristina Vistoli INFN CNAF, Bologna Italy The Grid-it: the Italian Grid Production infrastructure INFN-Grid goals!promote computational grid technologies research & development: Middleware and

More information

Solving the Mysteries of the Universe with Big Data

Solving the Mysteries of the Universe with Big Data Solving the Mysteries of the Universe with Big Data Sverre Jarp Former CERN openlab CTO Big Data Innovation Summit, Stockholm, 8 th May 2014 Accelerating Science and Innovation 1 What is CERN? The European

More information

Recent grid activities at INFN Catania(*) Roberto Barbera

Recent grid activities at INFN Catania(*) Roberto Barbera RecentgridactivitiesatINFNCatania(*) RobertoBarbera workincollaborationwithnicesrl (*) HEPiX/HEPNT2002,Catania,18.04.2002 CHEP2000,10.02.2000 1 RobertoBarbera DipartimentodiFisicadell UniversitàdiCataniaandINFNCatania

More information

Concepts and Architecture of the Grid. Summary of Grid 2, Chapter 4

Concepts and Architecture of the Grid. Summary of Grid 2, Chapter 4 Concepts and Architecture of the Grid Summary of Grid 2, Chapter 4 Concepts of Grid Mantra: Coordinated resource sharing and problem solving in dynamic, multi-institutional virtual organizations Allows

More information

EDG Project: Database Management Services

EDG Project: Database Management Services EDG Project: Database Management Services Leanne Guy for the EDG Data Management Work Package EDG::WP2 Leanne.Guy@cern.ch http://cern.ch/leanne 17 April 2002 DAI Workshop Presentation 1 Information in

More information

Plateforme de Calcul pour les Sciences du Vivant. SRB & glite. V. Breton. http://clrpcsv.in2p3.fr

Plateforme de Calcul pour les Sciences du Vivant. SRB & glite. V. Breton. http://clrpcsv.in2p3.fr SRB & glite V. Breton http://clrpcsv.in2p3.fr Introduction Goal: evaluation of existing technologies for data and tools integration and deployment Data and tools integration should be addressed using web

More information

GeoGrid Project and Experiences with Hadoop

GeoGrid Project and Experiences with Hadoop GeoGrid Project and Experiences with Hadoop Gong Zhang and Ling Liu Distributed Data Intensive Systems Lab (DiSL) Center for Experimental Computer Systems Research (CERCS) Georgia Institute of Technology

More information

Grid Scheduling Architectures with Globus GridWay and Sun Grid Engine

Grid Scheduling Architectures with Globus GridWay and Sun Grid Engine Grid Scheduling Architectures with and Sun Grid Engine Sun Grid Engine Workshop 2007 Regensburg, Germany September 11, 2007 Ignacio Martin Llorente Javier Fontán Muiños Distributed Systems Architecture

More information

For large geographically dispersed companies, data grids offer an ingenious new model to economically share computing power and storage resources

For large geographically dispersed companies, data grids offer an ingenious new model to economically share computing power and storage resources Data grids for storage http://storagemagazine.techtarget.com/magitem/0,291266,sid35_gci1132545,00.html by: Ray Lucchesi Storage Magazine Issue: Oct 2005 For large geographically dispersed companies, data

More information

Integrating a heterogeneous and shared Linux cluster into grids

Integrating a heterogeneous and shared Linux cluster into grids Integrating a heterogeneous and shared Linux cluster into grids 1,2 1 1,2 1 V. Büge, U. Felzmann, C. Jung, U. Kerzel, 1 1 1 M. Kreps, G. Quast, A. Vest 1 2 DPG Frühjahrstagung March 28 31, 2006 Dortmund

More information

NT1: An example for future EISCAT_3D data centre and archiving?

NT1: An example for future EISCAT_3D data centre and archiving? March 10, 2015 1 NT1: An example for future EISCAT_3D data centre and archiving? John White NeIC xx March 10, 2015 2 Introduction High Energy Physics and Computing Worldwide LHC Computing Grid Nordic Tier

More information

Bob Jones Technical Director bob.jones@cern.ch

Bob Jones Technical Director bob.jones@cern.ch Bob Jones Technical Director bob.jones@cern.ch CERN - August 2003 EGEE is proposed as a project to be funded by the European Union under contract IST-2003-508833 EGEE Goal & Strategy Goal: Create a wide

More information

Big Data and Storage Management at the Large Hadron Collider

Big Data and Storage Management at the Large Hadron Collider Big Data and Storage Management at the Large Hadron Collider Dirk Duellmann CERN IT, Data & Storage Services Accelerating Science and Innovation CERN was founded 1954: 12 European States Science for Peace!

More information

Analisi di un servizio SRM: StoRM

Analisi di un servizio SRM: StoRM 27 November 2007 General Parallel File System (GPFS) The StoRM service Deployment configuration Authorization and ACLs Conclusions. Definition of terms Definition of terms 1/2 Distributed File System The

More information

The Italian Grid Infrastructure (IGI) CGW08 Cracow. Mirco Mazzucato Italian Grid Infrastructure Coordinator INFN CNAF Director

The Italian Grid Infrastructure (IGI) CGW08 Cracow. Mirco Mazzucato Italian Grid Infrastructure Coordinator INFN CNAF Director The Italian Grid Infrastructure (IGI) CGW08 Cracow Mirco Mazzucato Italian Grid Infrastructure Coordinator INFN CNAF Director IGI -> National Grid Initiative Applications: Archeology Astronomy Astrophysics

More information

Grid Computing With FreeBSD

Grid Computing With FreeBSD Grid Computing With FreeBSD USENIX ATC '04: UseBSD SIG Boston, MA, June 29 th 2004 Brooks Davis, Craig Lee The Aerospace Corporation El Segundo, CA {brooks,lee}aero.org http://people.freebsd.org/~brooks/papers/usebsd2004/

More information

Deploying a distributed data storage system on the UK National Grid Service using federated SRB

Deploying a distributed data storage system on the UK National Grid Service using federated SRB Deploying a distributed data storage system on the UK National Grid Service using federated SRB Manandhar A.S., Kleese K., Berrisford P., Brown G.D. CCLRC e-science Center Abstract As Grid enabled applications

More information

Experiences with the GLUE information schema in the LCG/EGEE production Grid

Experiences with the GLUE information schema in the LCG/EGEE production Grid Experiences with the GLUE information schema in the LCG/EGEE production Grid Stephen Burke, Sergio Andreozzi and Laurence Field CHEP07, Victoria, Canada www.eu-egee.org EGEE and glite are registered trademarks

More information

Cloud Computing. Summary

Cloud Computing. Summary Cloud Computing Lecture 1 2011-2012 https://fenix.ist.utl.pt/disciplinas/cn Summary Teaching Staff. Rooms and Schedule. Goals. Context. Syllabus. Reading Material. Assessment and Grading. Important Dates.

More information

Vangelis Floros, GRNET S.A. 3 rd Open Source Software Conference March 22, 2008 NTUA, Athens Greece

Vangelis Floros, GRNET S.A. 3 rd Open Source Software Conference March 22, 2008 NTUA, Athens Greece Vangelis Floros, GRNET S.A. 3 rd Open Source Software Conference March 22, 2008 NTUA, Athens Greece Introduction What is a Grid? What is escience? Large Scientific Grids The example of EGEE Building Grid

More information

Database Services for Physics @ CERN

Database Services for Physics @ CERN Database Services for Physics @ CERN Deployment and Monitoring Radovan Chytracek CERN IT Department Outline Database services for physics Status today How we do the services tomorrow? Performance tuning

More information

Roberto Barbera. Centralized bookkeeping and monitoring in ALICE

Roberto Barbera. Centralized bookkeeping and monitoring in ALICE Centralized bookkeeping and monitoring in ALICE CHEP INFN 2000, GRID 10.02.2000 WP6, 24.07.2001 Roberto 1 Barbera ALICE and the GRID Phase I: AliRoot production The GRID Powered by ROOT 2 How did we get

More information

Retrieval on the Grid Results from the European Project GRACE (Grid Search and Categorization Engine)

Retrieval on the Grid Results from the European Project GRACE (Grid Search and Categorization Engine) Retrieval on the Grid Results from the European Project GRACE (Grid Search and Categorization Engine) FRANK SCHOLZE, WERNER STEPHAN Stuttgart University Library Main parts of this paper are based on the

More information

Grid Computing: A Ten Years Look Back. María S. Pérez Facultad de Informática Universidad Politécnica de Madrid mperez@fi.upm.es

Grid Computing: A Ten Years Look Back. María S. Pérez Facultad de Informática Universidad Politécnica de Madrid mperez@fi.upm.es Grid Computing: A Ten Years Look Back María S. Pérez Facultad de Informática Universidad Politécnica de Madrid mperez@fi.upm.es Outline Challenges not yet solved in computing The parents of grid Computing

More information

A Market-based Framework for Trading Grid Resources

A Market-based Framework for Trading Grid Resources A Market-based Framework for Trading Grid Resources Dr. JIE SONG Email: Jie.Song@sun.com Asia Pacific Science & Technology Center Sun Microsystems Inc. Agenda Motivation Grid Service Market Framework Prototype

More information

Distributed Database Access in the LHC Computing Grid with CORAL

Distributed Database Access in the LHC Computing Grid with CORAL Distributed Database Access in the LHC Computing Grid with CORAL Dirk Duellmann, CERN IT on behalf of the CORAL team (R. Chytracek, D. Duellmann, G. Govi, I. Papadopoulos, Z. Xie) http://pool.cern.ch &

More information

Scheduling and Resource Management in Computational Mini-Grids

Scheduling and Resource Management in Computational Mini-Grids Scheduling and Resource Management in Computational Mini-Grids July 1, 2002 Project Description The concept of grid computing is becoming a more and more important one in the high performance computing

More information

GRIP:Creating Interoperability between Grids

GRIP:Creating Interoperability between Grids GRIP:Creating Interoperability between Grids Philipp Wieder, Dietmar Erwin, Roger Menday Research Centre Jülich EuroGrid Workshop Cracow, October 29, 2003 Contents Motivation Software Base at a Glance

More information

Solving the Mysteries of the Universe with Big Data

Solving the Mysteries of the Universe with Big Data Solving the Mysteries of the Universe with Big Data Sverre Jarp CERN openlab CTO Big Data Innovation Summit, Boston, 12 th September 2013 Accelerating Science and Innovation 1 What is CERN? The European

More information

Betriebssystem-Virtualisierung auf einem Rechencluster am SCC mit heterogenem Anwendungsprofil

Betriebssystem-Virtualisierung auf einem Rechencluster am SCC mit heterogenem Anwendungsprofil Betriebssystem-Virtualisierung auf einem Rechencluster am SCC mit heterogenem Anwendungsprofil Volker Büge 1, Marcel Kunze 2, OIiver Oberst 1,2, Günter Quast 1, Armin Scheurer 1 1) Institut für Experimentelle

More information

Cloud Computing. Lecture 5 Grid Case Studies 2014-2015

Cloud Computing. Lecture 5 Grid Case Studies 2014-2015 Cloud Computing Lecture 5 Grid Case Studies 2014-2015 Up until now Introduction. Definition of Cloud Computing. Grid Computing: Schedulers Globus Toolkit Summary Grid Case Studies: Monitoring: TeraGRID

More information

GFAL and LCG data management. Jean-Philippe Baud CERN/IT/GD. HEPiX

GFAL and LCG data management. Jean-Philippe Baud CERN/IT/GD. HEPiX GFAL and LCG data management Jean-Philippe Baud CERN/IT/GD Agenda LCG Data Management goals Common interface Current status Current developments Medium term developments Conclusion LCG Data Management

More information

Status of Grid Activities in Pakistan. FAWAD SAEED National Centre For Physics, Pakistan

Status of Grid Activities in Pakistan. FAWAD SAEED National Centre For Physics, Pakistan Status of Grid Activities in Pakistan FAWAD SAEED National Centre For Physics, Pakistan 1 Introduction of NCP-LCG2 q NCP-LCG2 is the only Tier-2 centre in Pakistan for Worldwide LHC computing Grid (WLCG).

More information

The CMS analysis chain in a distributed environment

The CMS analysis chain in a distributed environment The CMS analysis chain in a distributed environment on behalf of the CMS collaboration DESY, Zeuthen,, Germany 22 nd 27 th May, 2005 1 The CMS experiment 2 The CMS Computing Model (1) The CMS collaboration

More information

Grids Computing and Collaboration

Grids Computing and Collaboration Grids Computing and Collaboration Arto Teräs CSC, the Finnish IT center for science University of Pune, India, March 12 th 2007 Grids Computing and Collaboration / Arto Teräs 2007-03-12 Slide

More information

A Performance Monitor based on Virtual Global Time for Clusters of PCs

A Performance Monitor based on Virtual Global Time for Clusters of PCs A Performance Monitor based on Virtual Global Time for Clusters of PCs Michela Taufer Scripps Institute & UCSD Dept. of CS San Diego, USA Thomas Stricker Cluster 2003, 12/2/2003 Hong Kong, SAR, China Lab.

More information

Chapter 7. Using Hadoop Cluster and MapReduce

Chapter 7. Using Hadoop Cluster and MapReduce Chapter 7 Using Hadoop Cluster and MapReduce Modeling and Prototyping of RMS for QoS Oriented Grid Page 152 7. Using Hadoop Cluster and MapReduce for Big Data Problems The size of the databases used in

More information

Information Technology Services

Information Technology Services Information Technology Services The purpose of an Information Technology Standard is to specify requirements for compliance with Old Dominion University Information Technology policies, other University

More information

Energy: electricity, electric grids, nuclear, green... Transportation: roads, airplanes, helicopters, space exploration

Energy: electricity, electric grids, nuclear, green... Transportation: roads, airplanes, helicopters, space exploration 100 Years of Innovation Health: public sanitation, aspirin, antibiotics, vaccines, lasers, organ transplants, medical imaging, genome, genomics, epigenetics, cancer genomics (TCGA consortium). Energy:

More information

An approach to grid scheduling by using Condor-G Matchmaking mechanism

An approach to grid scheduling by using Condor-G Matchmaking mechanism An approach to grid scheduling by using Condor-G Matchmaking mechanism E. Imamagic, B. Radic, D. Dobrenic University Computing Centre, University of Zagreb, Croatia {emir.imamagic, branimir.radic, dobrisa.dobrenic}@srce.hr

More information

EGEE a worldwide Grid infrastructure

EGEE a worldwide Grid infrastructure EGEE a worldwide Grid infrastructure Fabrizio Gagliardi Project Director EGEE CERN, Switzerland IFIC, 6 October 2005 www.eu-egee.org Presentation overview Data intensive science and the rationale for Grid

More information

PHENIX Job Submission/Monitoring in transition to the Grid Infrastructure

PHENIX Job Submission/Monitoring in transition to the Grid Infrastructure PHENIX Job Submission/Monitoring in transition to the Grid Infrastructure Andrey Y. Shevel, Barbara Jacak, Roy Lacey, Dave Morrison, Michael Reuter, Irina Sourikova, Timothy Thomas, Alex Withers 1 Brief

More information

Grid Computing vs Cloud

Grid Computing vs Cloud Chapter 3 Grid Computing vs Cloud Computing 3.1 Grid Computing Grid computing [8, 23, 25] is based on the philosophy of sharing information and power, which gives us access to another type of heterogeneous

More information

AN APPROACH TO DEVELOPING BUSINESS PROCESSES WITH WEB SERVICES IN GRID

AN APPROACH TO DEVELOPING BUSINESS PROCESSES WITH WEB SERVICES IN GRID AN APPROACH TO DEVELOPING BUSINESS PROCESSES WITH WEB SERVICES IN GRID R. D. Goranova 1, V. T. Dimitrov 2 Faculty of Mathematics and Informatics, University of Sofia S. Kliment Ohridski, 1164, Sofia, Bulgaria

More information

Finnish Grid Activities

Finnish Grid Activities Finnish Grid Activities Nordic Grid Neighbourhood Seminar January 20th, 2005 Michael Gindonis, Helsinki Institute of Physics (HIP) Arto Teräs, CSC - The Finnish IT Center for Science 1 Grid Research Activities

More information

US NSF s Scientific Software Innovation Institutes

US NSF s Scientific Software Innovation Institutes US NSF s Scientific Software Innovation Institutes S 2 I 2 awards invest in long-term projects which will realize sustained software infrastructure that is integral to doing transformative science. (Can

More information

The OGSA-DAI Project Databases and the Grid. Neil Chue Hong. Principal Consultant EPCC, Edinburgh N.ChueHong@epcc.ed.ac.uk

The OGSA-DAI Project Databases and the Grid. Neil Chue Hong. Principal Consultant EPCC, Edinburgh N.ChueHong@epcc.ed.ac.uk The OGSA-DAI Project Databases and the Grid Neil Chue Hong Principal Consultant EPCC, Edinburgh N.ChueHong@epcc.ed.ac.uk What is OGSA-DAI? 4It is a project: OGSA Data Access and Integration: funded by

More information

Hadoop & its Usage at Facebook

Hadoop & its Usage at Facebook Hadoop & its Usage at Facebook Dhruba Borthakur Project Lead, Hadoop Distributed File System dhruba@apache.org Presented at the The Israeli Association of Grid Technologies July 15, 2009 Outline Architecture

More information

irods at CC-IN2P3: managing petabytes of data

irods at CC-IN2P3: managing petabytes of data Centre de Calcul de l Institut National de Physique Nucléaire et de Physique des Particules irods at CC-IN2P3: managing petabytes of data Jean-Yves Nief Pascal Calvat Yonny Cardenas Quentin Le Boulc h

More information

Transnational Access to Mass Storage Capacity for Computational QCD

Transnational Access to Mass Storage Capacity for Computational QCD Transnational Access to Mass Storage Capacity for Computational QCD and Stefan Wollny Konrad-Zuse-Zentrum für Informationstechnik Berlin (ZIB), Takustr. 7, 14195 Berlin, Germany E-mail: stueben@zib.de,

More information

Hadoop & its Usage at Facebook

Hadoop & its Usage at Facebook Hadoop & its Usage at Facebook Dhruba Borthakur Project Lead, Hadoop Distributed File System dhruba@apache.org Presented at the Storage Developer Conference, Santa Clara September 15, 2009 Outline Introduction

More information

High Availability Databases based on Oracle 10g RAC on Linux

High Availability Databases based on Oracle 10g RAC on Linux High Availability Databases based on Oracle 10g RAC on Linux WLCG Tier2 Tutorials, CERN, June 2006 Luca Canali, CERN IT Outline Goals Architecture of an HA DB Service Deployment at the CERN Physics Database

More information

2. Requirements for Metadata Management on the Grid

2. Requirements for Metadata Management on the Grid Grid-Based Metadata Services Ewa Deelman 1, Gurmeet Singh 1, Malcolm P. Atkinson 2, Ann Chervenak 1, Neil P Chue Hong 3, Carl Kesselman 1, Sonal Patil 1, Laura Pearlman 1, Mei-Hui Su 1 1 Information Sciences

More information

Bibliography. University of Applied Sciences Fulda, Prof. Dr. S. Groß

Bibliography. University of Applied Sciences Fulda, Prof. Dr. S. Groß Slide III Bibliography 1) Abbas, A.: Grid Computing - A Practical Guide to Technology and Applications. Charles River Media, 2004. http://www.charlesriver.com/titles/gridcomputing.html 2) Berman, F.; et

More information

Global Grid User Support - GGUS - start up schedule

Global Grid User Support - GGUS - start up schedule Global Grid User Support - GGUS - start up schedule GDB Meeting 2004-07 07-13 Concept Target: 24 7 support via time difference and 3 support teams Currently: GGUS FZK GGUS ASCC Planned: GGUS USA Support

More information

Introduction to Grid Computing

Introduction to Grid Computing Introduction to Grid Computing What is the GRID? Grid Computing Technology Grid computing enables the virtualization of distributed computing and data resources such as processing, network bandwidth and

More information

This presentation provides an overview of the architecture of the IBM Workload Deployer product.

This presentation provides an overview of the architecture of the IBM Workload Deployer product. This presentation provides an overview of the architecture of the IBM Workload Deployer product. Page 1 of 17 This presentation starts with an overview of the appliance components and then provides more

More information

Scalable stochastic tracing of distributed data management events

Scalable stochastic tracing of distributed data management events Scalable stochastic tracing of distributed data management events Mario Lassnig mario.lassnig@cern.ch ATLAS Data Processing CERN Physics Department Distributed and Parallel Systems University of Innsbruck

More information

The EDG Workload Management System: release 2

The EDG Workload Management System: release 2 M. Sgaravatto n 1 The EDG Management System: release 2 Massimo Sgaravatto INFN Padova DataGrid WP1 massimo.sgaravatto@pd.infn.it Review of WP1 WMS architecture WP1 WMS architecture reviewed To apply the

More information

The LHC Computing Grid Project (LCG) and ROOT

The LHC Computing Grid Project (LCG) and ROOT The LHC Computing Grid Project (LCG) and ROOT Torre Wenaus, BNL/CERN LCG Applications Area Manager John Harvey, CERN EP/SFT Group Leader http://cern.ch/lcg/peb/applications ROOT Workshop, CERN October

More information

DAME Astrophysical DAta Mining Mining & & Exploration Exploration GRID

DAME Astrophysical DAta Mining Mining & & Exploration Exploration GRID DAME Astrophysical DAta Mining & Exploration on GRID M. Brescia S. G. Djorgovski G. Longo & DAME Working Group Istituto Nazionale di Astrofisica Astronomical Observatory of Capodimonte, Napoli Department

More information

19.7 Case study: the Grid 814 CHAPTER 19 WEB SERVICES

19.7 Case study: the Grid 814 CHAPTER 19 WEB SERVICES Chapter 19 Web Services.fm Page 814 Friday, March 18, 2005 3:13 PM 814 CHAPTER 19 WEB SERVICES The use of a common choreography description by a set of collaborating web services should result in more

More information

MANAGING SCIENTIFIC DATA WITH NDN

MANAGING SCIENTIFIC DATA WITH NDN MANAGING SCIENTIFIC DATA WITH NDN Chengyu Fan, Susmit Shannigrahi, Steve DiBenedetto, Catherine Olschanowsky, Christos Papadopoulos NDNcomm 2015 Sept 28, 2015 Los Angeles, CA Supported by NSF #13410999

More information

Virtual machine interface. Operating system. Physical machine interface

Virtual machine interface. Operating system. Physical machine interface Software Concepts User applications Operating system Hardware Virtual machine interface Physical machine interface Operating system: Interface between users and hardware Implements a virtual machine that

More information

The ENEA-EGEE site: Access to non-standard platforms

The ENEA-EGEE site: Access to non-standard platforms V INFNGrid Workshop Padova, Italy December 18-20 2006 The ENEA-EGEE site: Access to non-standard platforms C. Sciò**, G. Bracco, P. D'Angelo, L. Giammarino*, S.Migliori, A. Quintiliani, F. Simoni, S. Podda

More information

The Open Access initiative in the context of the European ICT Research Infrastructure Service

The Open Access initiative in the context of the European ICT Research Infrastructure Service The Open Access initiative in the context of the European ICT Research Infrastructure Service Open Access Conference Athens, 15 December 2008 Kyriakos Baxevanidis European Commission - DG INFSO Acting

More information

Cloud Computing Now and the Future Development of the IaaS

Cloud Computing Now and the Future Development of the IaaS 2010 Cloud Computing Now and the Future Development of the IaaS Quanta Computer Division: CCASD Title: Project Manager Name: Chad Lin Agenda: What is Cloud Computing? Public, Private and Hybrid Cloud.

More information

Digital libraries of the future and the role of libraries

Digital libraries of the future and the role of libraries Digital libraries of the future and the role of libraries Donatella Castelli ISTI-CNR, Pisa, Italy Abstract Purpose: To introduce the digital libraries of the future, their enabling technologies and their

More information

HPC and Grid Concepts

HPC and Grid Concepts HPC and Grid Concepts Divya MG (divyam@cdac.in) CDAC Knowledge Park, Bangalore 16 th Feb 2012 GBC@PRL Ahmedabad 1 Presentation Overview What is HPC Need for HPC HPC Tools Grid Concepts GARUDA Overview

More information

Grid Technology in Civil Engineering

Grid Technology in Civil Engineering University of Ljubljana Faculty of Civil and Geodetic Engineering, Institute of Civil Enegineering, Earthquake Engineering and Construction IT, Chair of Construction Informatics Grid Technology in Civil

More information

Batch and Cloud overview. Andrew McNab University of Manchester GridPP and LHCb

Batch and Cloud overview. Andrew McNab University of Manchester GridPP and LHCb Batch and Cloud overview Andrew McNab University of Manchester GridPP and LHCb Overview Assumptions Batch systems The Grid Pilot Frameworks DIRAC Virtual Machines Vac Vcycle Tier-2 Evolution Containers

More information

DSA1.4 R EPORT ON IMPLEMENTATION OF MONITORING AND OPERATIONAL SUPPORT SYSTEM. Activity: SA1. Partner(s): EENet, NICPB. Lead Partner: EENet

DSA1.4 R EPORT ON IMPLEMENTATION OF MONITORING AND OPERATIONAL SUPPORT SYSTEM. Activity: SA1. Partner(s): EENet, NICPB. Lead Partner: EENet R EPORT ON IMPLEMENTATION OF MONITORING AND OPERATIONAL SUPPORT SYSTEM Document Filename: Activity: Partner(s): Lead Partner: Document classification: BG-DSA1.4-v1.0-Monitoring-operational-support-.doc

More information

QosCosGrid Grid Technologies and Complex System Modelling

QosCosGrid Grid Technologies and Complex System Modelling QosCosGrid Grid Technologies and Complex System Modelling Pamela Burrage Krzysztof Kurowski Institute for Molecular Bioscience, University of Queensland, Australia Vision, objectives Complex systems (motivations)

More information

Managing Complexity in Distributed Data Life Cycles Enhancing Scientific Discovery

Managing Complexity in Distributed Data Life Cycles Enhancing Scientific Discovery Center for Information Services and High Performance Computing (ZIH) Managing Complexity in Distributed Data Life Cycles Enhancing Scientific Discovery Richard Grunzke*, Jens Krüger, Sandra Gesing, Sonja

More information

Introduction. MCSN N. Tonellotto Complements of Distributed Enabling Platforms

Introduction. MCSN N. Tonellotto Complements of Distributed Enabling Platforms Introduction 1 Distributed relating to a computer network in which at least some of the processing is done by the individual computers and information is shared by and often stored at the computers Enabling

More information

Integration of Cloud Computing and Cloud Storage

Integration of Cloud Computing and Cloud Storage Integration of Cloud Computing and Cloud Storage Chief Scientist,Renaissance Computing Institute Adjunct Prof. Computer Science, NC State University IEEE Mass Storage Conference Tutorial May 3, 2010 Outline

More information

EGEE is a project funded by the European Union under contract IST-2003-508833

EGEE is a project funded by the European Union under contract IST-2003-508833 www.eu-egee.org NA4 Applications F.Harris(Oxford/CERN) NA4/HEP coordinator EGEE is a project funded by the European Union under contract IST-2003-508833 Talk Outline The basic goals of NA4 The organisation

More information

Service Oriented Distributed Manager for Grid System

Service Oriented Distributed Manager for Grid System Service Oriented Distributed Manager for Grid System Entisar S. Alkayal Faculty of Computing and Information Technology King Abdul Aziz University Jeddah, Saudi Arabia entisar_alkayal@hotmail.com Abstract

More information

- from Clusters and Grids to Cloud computing

- from Clusters and Grids to Cloud computing Lecture - 1 Moving Ahead - from Clusters and Grids to Cloud computing Salman Toor salman.toor@it.uu.se Basic questions Why Cloud computing? What are the previous technologies? What was missing in the previous

More information

Cloud Models and Platforms

Cloud Models and Platforms Cloud Models and Platforms Dr. Sanjay P. Ahuja, Ph.D. 2010-14 FIS Distinguished Professor of Computer Science School of Computing, UNF A Working Definition of Cloud Computing Cloud computing is a model

More information

Configuration Management of Massively Scalable Systems

Configuration Management of Massively Scalable Systems 1 KKIO 2005 Configuration Management of Massively Scalable Systems Configuration Management of Massively Scalable Systems Marcin Jarząb, Krzysztof Zieliński, Jacek Kosiński SUN Center of Excelence Department

More information

PARALLEL & CLUSTER COMPUTING CS 6260 PROFESSOR: ELISE DE DONCKER BY: LINA HUSSEIN

PARALLEL & CLUSTER COMPUTING CS 6260 PROFESSOR: ELISE DE DONCKER BY: LINA HUSSEIN 1 PARALLEL & CLUSTER COMPUTING CS 6260 PROFESSOR: ELISE DE DONCKER BY: LINA HUSSEIN Introduction What is cluster computing? Classification of Cluster Computing Technologies: Beowulf cluster Construction

More information

Evolution of an Inter University Data Grid Architecture in Pakistan

Evolution of an Inter University Data Grid Architecture in Pakistan Evolution of an Inter University Data Grid Architecture in Pakistan Aslam Parvez Memon* and Shakil Akhtar** *SZABIST, Karachi, Pakistan **College of Information Technology, UAE University, UAE Abstract:

More information