Dominique Fouchez. 12 Fevrier 2011
|
|
|
- Shona Waters
- 10 years ago
- Views:
Transcription
1 données données CPPM 12 Fevrier 2011
2 The Data données one 6.4-gigabyte image every 17 seconds 15 terabytes of raw scientific image data / night 60-petabyte final image data archive 20-petabyte final database catalog (multiple trillion rows) 2 million real time events per night every night for 10 years
3 Data processing by the project données Data management process the images for the community Image simulation / calibration manage by system team
4 Management team (July 2012) données
5 Management team (July 2012) données
6 Data Management products données
7 Data processing for données SWG : Science Working Groups use data delivered by DM add level 3 processing (@ data center) combine with other data... DESC : Dark Energy Science Collaboration use data delivered by DM, mainly low level ( images) on pixel specific DE processing (psf - multifit etc) Combine with large set of non images...
8 The Data Management données Provide a highly reliable open source system to provide: Real time alerts catalog data products, image data yearly coadd realeases Provides the infrastructure to transport, process, and serve the data
9 Cyber infrastructure données
10 Processing pipelines données
11 Organisation for the Processing pipelines development données
12 Organisation for the Processing pipelines development données 1. Infrastructure, database and software deployment : evolving toward final and full size.
13 Organisation for the Processing pipelines development données 1. Infrastructure, database and software deployment : evolving toward final and full size. 2. Development of sofware : design / prototyping and test..
14 Organisation for the Processing pipelines development données 1. Infrastructure, database and software deployment : evolving toward final and full size. 2. Development of sofware : design / prototyping and test during data challenges (every 6 month / 1 year)
15 Organisation for the Processing pipelines development données 1. Infrastructure, database and software deployment : evolving toward final and full size. 2. Development of sofware : design / prototyping and test during data challenges (every 6 month / 1 year) 4. Next data challenge : Summer 2013 CCIN2P3 (+ france team) will process 1 half
16 French activities and commitment in computing and données
17 French activities and commitment in computing and données 1. 1 Half of Data (Challenge) at CC : from 2013, till the final design and the first light.
18 French activities and commitment in computing and données 1. 1 Half of Data (Challenge) at CC : from 2013, till the final design and the first light. 2. Simulation (and image reduction?) link with camera
19 French activities and commitment in computing and données 1. 1 Half of Data (Challenge) at CC : from 2013, till the final design and the first light. 2. Simulation (and image reduction?) link with camera 3. Work on images : CFHT LS images, (SDSS)
20 French activities and commitment in computing and données 1. 1 Half of Data (Challenge) at CC : from 2013, till the final design and the first light. 2. Simulation (and image reduction?) link with camera 3. Work on images : CFHT LS images, (SDSS) 4. IN2P3 : Science via DESC (and SWG)
21 French activities and commitment in computing and données 1. 1 Half of Data (Challenge) at CC : from 2013, till the final design and the first light. 2. Simulation (and image reduction?) link with camera 3. Work on images : CFHT LS images, (SDSS) 4. IN2P3 : Science via DESC (and SWG) Many opportunities : goodwill welcome
The LSST Data management and French computing activities. Dominique Fouchez on behalf of the IN2P3 Computing Team. LSST France April 8th,2015
The LSST Data management and French computing activities Dominique Fouchez on behalf of the IN2P3 Computing Team LSST France April 8th,2015 OSG All Hands SLAC April 7-9, 2014 1 The LSST Data management
Introduction to LSST Data Management. Jeffrey Kantor Data Management Project Manager
Introduction to LSST Data Management Jeffrey Kantor Data Management Project Manager LSST Data Management Principal Responsibilities Archive Raw Data: Receive the incoming stream of images that the Camera
Data analysis of L2-L3 products
Data analysis of L2-L3 products Emmanuel Gangler UBP Clermont-Ferrand (France) Emmanuel Gangler BIDS 14 1/13 Data management is a pillar of the project : L3 Telescope Caméra Data Management Outreach L1
Astrophysics with Terabyte Datasets. Alex Szalay, JHU and Jim Gray, Microsoft Research
Astrophysics with Terabyte Datasets Alex Szalay, JHU and Jim Gray, Microsoft Research Living in an Exponential World Astronomers have a few hundred TB now 1 pixel (byte) / sq arc second ~ 4TB Multi-spectral,
irods at CC-IN2P3: managing petabytes of data
Centre de Calcul de l Institut National de Physique Nucléaire et de Physique des Particules irods at CC-IN2P3: managing petabytes of data Jean-Yves Nief Pascal Calvat Yonny Cardenas Quentin Le Boulc h
Euclid. Mapping the geometry of the dark Universe. Conseil scientifique de l IN2P3. 25 Octobre 2012. A Ealet (CPPM)
Euclid Mapping the geometry of the dark Universe Conseil scientifique de l IN2P3 25 Octobre 2012 A Ealet (CPPM) Participation of IN2P3 to an ESA space mission OVERVIEW Euclid Scientific objectives The
LSST Data Management plans: Pipeline outputs and Level 2 vs. Level 3
LSST Data Management plans: Pipeline outputs and Level 2 vs. Level 3 Mario Juric Robert Lupton LSST DM Project Scien@st Algorithms Lead LSST SAC Name of Mee)ng Loca)on Date - Change in Slide Master 1 Data
Learning from Big Data in
Learning from Big Data in Astronomy an overview Kirk Borne George Mason University School of Physics, Astronomy, & Computational Sciences http://spacs.gmu.edu/ From traditional astronomy 2 to Big Data
LSST Data Management System Applications Layer Simulated Data Needs Description: Simulation Needs for DC3
LSST Data Management System Applications Layer Simulated Data Needs Description: Simulation Needs for DC3 Draft 25 September 2008 A joint document from the LSST Data Management Team and Image Simulation
Turning Big Data into Big Decisions Delivering on the High Demand for Data
Turning Big Data into Big Decisions Delivering on the High Demand for Data Michael Ho, Vice President of Professional Services Digital Government Institute s Government Big Data Conference, October 31,
Kevin Blair CEO, NewGround. Retail 2020 FUTURE BANK
Kevin Blair CEO, NewGround Retail 2020 FUTURE BANK Today, the amount of global big data = megabytes gigabytes terabytes petabytes exabytes SOURCE: Cisco Global Mobile Data Tracking Update (February 6,
MAST: The Mikulski Archive for Space Telescopes
MAST: The Mikulski Archive for Space Telescopes Richard L. White Space Telescope Science Institute 2015 April 1, NRC Space Science Week/CBPSS A model for open access The NASA astrophysics data archives
PMCS - WBS with Definition
02C Data Management Construction This WBS element provides the complete LSST Data Management System (DMS). The DMS has these main responsibilities in the LSST system: Process the incoming stream of images
Grid Computing in Aachen
GEFÖRDERT VOM Grid Computing in Aachen III. Physikalisches Institut B Berichtswoche des Graduiertenkollegs Bad Honnef, 05.09.2008 Concept of Grid Computing Computing Grid; like the power grid, but for
WFIRST- AFTA: European Interests
WFIRST- AFTA: European Interests Mark Cropper Mullard Space Science Laboratory University College London Contents 1. WFI interests Europe- wide 2. Coronagraph interests, UK only see Anthony BoccaleJ s
Milestone Solution Partner IT Infrastructure MTP Certification Report Scality RING Software-Defined Storage 11-16-2015
Milestone Solution Partner IT Infrastructure MTP Certification Report Scality RING Software-Defined Storage 11-16-2015 Table of Contents Introduction... 4 Certified Products... 4 Key Findings... 5 Solution
LSST and the Cloud: Astro Collaboration in 2016 Tim Axelrod LSST Data Management Scientist
LSST and the Cloud: Astro Collaboration in 2016 Tim Axelrod LSST Data Management Scientist DERCAP Sydney, Australia, 2009 Overview of Presentation LSST - a large-scale Southern hemisphere optical survey
Data Management So,ware Stack Intro
Data Management So,ware Stack Intro Mario Jurić LSST Data Management Project Scien:st SLAC DM Stack Working Mee:ng 10-12 December, 2012 1 LSST Data Management Tasks Processes the incoming stream of images
What Is Big Data? Craig C. Douglas University of Wyoming
What Is Big Data? Craig C. Douglas University of Wyoming What Is Big Data?... It Depends Unit Approximately 10 n Related to Kilobyte (KB) 1,000 bytes 3 Circa 1952 computer memory 32 KB Apollo 11 computer
How To Teach Data Science
The Past, Present, and Future of Data Science Education Kirk Borne @KirkDBorne http://kirkborne.net George Mason University School of Physics, Astronomy, & Computational Sciences Outline Research and Application
Challenges in e-science: Research in a Digital World
Challenges in e-science: Research in a Digital World Thom Dunning National Center for Supercomputing Applications National Center for Supercomputing Applications University of Illinois at Urbana-Champaign
TAP into NAU. TAP is a single tool. How it works. Transfer Academic Plan nau.edu/tap
TAP into and an degree: the catalog TAP into and an degree: the catalog TAP into and an degree: the catalog TAP into and an degree: the catalog TAP into and an degree: the catalog TAP into and an degree:
FRENCH RADIOSOUNDING AUTOMATION EXPERIMENTS ABSTRACT
FRENCH RADIOSOUNDING AUTOMATION EXPERIMENTS MAGALI STOLL, YVAN LEMAITRE, CHRISTOPHE BARRIERE, MÉTÉO-FRANCE, DIRECTION OF OBSERVING SYSTEMS UPPER AIR OBSERVING SYSTEMS DEPARTMENT 42 AVENUE G.CORIOLIS F
The CMS analysis chain in a distributed environment
The CMS analysis chain in a distributed environment on behalf of the CMS collaboration DESY, Zeuthen,, Germany 22 nd 27 th May, 2005 1 The CMS experiment 2 The CMS Computing Model (1) The CMS collaboration
The World-Wide Telescope, an Archetype for Online Science
The World-Wide Telescope, an Archetype for Online Science Jim Gray, Microsoft Research Alex Szalay, Johns Hopkins University June 2002 Technical Report MSR-TR-2002-75 Microsoft Research Microsoft Corporation
Summary of Data Management Principles Dark Energy Survey V2.1, 7/16/15
Summary of Data Management Principles Dark Energy Survey V2.1, 7/16/15 This Summary of Data Management Principles (DMP) has been prepared at the request of the DOE Office of High Energy Physics, in support
Condor Supercomputer
Introducing the Air Force Research Laboratory Information Directorate Condor Supercomputer DOD s Largest Interactive Supercomputer Ribbon Cutting Ceremony AFRL, Rome NY 1 Dec 2010 Approved for Public Release
IBM G-Cloud IBM SPSS Decision Management Software as a Service
IBM G-Cloud IBM SPSS Decision Management Software as a Service Service Definition 1 1. Summary 1.1 Service Description IBM SPSS Decision Management Software as a Service (SaaS) offers another way of delivering
Expert Group on Cloud Computing Services and Standards ( EGCCSS ) Formation of Working Groups
For Discussion on 27 May 2014 Paper EGCCSS-6-3 Expert Group on Cloud Computing Services and Standards ( EGCCSS ) Formation of Working Groups Purpose To propose the setting up of two Working Groups under
Canadian Astronomy Data Centre. Séverin Gaudet David Schade Canadian Astronomy Data Centre
Canadian Astronomy Data Centre Séverin Gaudet David Schade Canadian Astronomy Data Centre Data Activities in Astronomy Features of the astronomy data landscape Multi-wavelength datasets are increasingly
Introduction to Computer Graphics
Introduction to Computer Graphics Torsten Möller TASC 8021 778-782-2215 [email protected] www.cs.sfu.ca/~torsten Today What is computer graphics? Contents of this course Syllabus Overview of course topics
Extraction Heights for STIS Echelle Spectra
Instrument Science Report STIS 98-09 Extraction Heights for STIS Echelle Spectra Claus Leitherer and Ralph Bohlin March 1998 ABSTRACT The optimum extraction height (h) for STIS echelle spectra of 7 pixels
An IDL for Web Services
An IDL for Web Services Interface definitions are needed to allow clients to communicate with web services Interface definitions need to be provided as part of a more general web service description Web
Good Research Practice
Page 1 of 8 The Royal Veterinary College Introduction Preface Good Research Practice Funding bodies need to be assured of the quality and validity of research they fund and ensure their contractors are
ESKISP6055.01 Manage security testing
Overview This standard covers the competencies concerning with managing security testing activities. Including managing resources activities and deliverables. This includes planning, conducting and reporting
Product Overview. Payroll & Personnel Data Warehouse
Payroll & Personnel Data Warehouse Grapevine Solutions 2010 HAPPI HAPPI (History & Archiving for Payroll & Personnel Information) was developed by Grapevine Solutions to meet the needs of organisations
Auto-Classification for Document Archiving and Records Declaration
Auto-Classification for Document Archiving and Records Declaration Josemina Magdalen, Architect, IBM November 15, 2013 Agenda IBM / ECM/ Content Classification for Document Archiving and Records Management
VITO Centre of Image Processing
07/11/2013 VITO Centre of Image Processing Towards an Improved Scientific Exploitation of EO Data Sources in Support of Vegetation Monitoring Erwin Goor, VITO nv Outline» Introduction» The role & and objectives
Cloud Computing Where ISR Data Will Go for Exploitation
Cloud Computing Where ISR Data Will Go for Exploitation 22 September 2009 Albert Reuther, Jeremy Kepner, Peter Michaleas, William Smith This work is sponsored by the Department of the Air Force under Air
WHITE PAPER. Harnessing the Power of Advanced Analytics How an appliance approach simplifies the use of advanced analytics
WHITE PAPER Harnessing the Power of Advanced How an appliance approach simplifies the use of advanced analytics Introduction The Netezza TwinFin i-class advanced analytics appliance pushes the limits of
REFERENTIEL- SODERN. Autre référence / Other reference :
Classification: NC Page : 1/8 Autre référence / Other reference : Type: Sector procedureprocédure sectorielle. Description: Exigences qualité pour les fournisseurs de Sodern version anglaise / Quality
Migrating a (Large) Science Database to the Cloud
The Sloan Digital Sky Survey Migrating a (Large) Science Database to the Cloud Ani Thakar Alex Szalay Center for Astrophysical Sciences and Institute for Data Intensive Engineering and Science (IDIES)
Hexaware E-book on Predictive Analytics
Hexaware E-book on Predictive Analytics Business Intelligence & Analytics Actionable Intelligence Enabled Published on : Feb 7, 2012 Hexaware E-book on Predictive Analytics What is Data mining? Data mining,
University Information Sheet
Part A: University Information Sheet A.1- Brief description of your university: (history, main activities, number of students, courses offered, etc). The Sidi Mohammed Ben Abdellah University has been
Promises and Pitfalls of Big-Data-Predictive Analytics: Best Practices and Trends
Promises and Pitfalls of Big-Data-Predictive Analytics: Best Practices and Trends Spring 2015 Thomas Hill, Ph.D. VP Analytic Solutions Dell Statistica Overview and Agenda Dell Software overview Dell in
The PACS Software System. (A high level overview) Prepared by : E. Wieprecht, J.Schreiber, U.Klaas November,5 2007 Issue 1.
The PACS Software System (A high level overview) Prepared by : E. Wieprecht, J.Schreiber, U.Klaas November,5 2007 Issue 1.0 PICC-ME-DS-003 1. Introduction The PCSS, the PACS ICC Software System, is the
SQL Server PDW. Artur Vieira Premier Field Engineer
SQL Server PDW Artur Vieira Premier Field Engineer Agenda 1 Introduction to MPP and PDW 2 PDW Architecture and Components 3 Data Structures 4 PDW Tools Data Load / Data Output / Administrative Console
Gaming System Monitoring and Analysis Effort
for the Gaming System Monitoring and Analysis Effort DHS/S&T/PIA-025 October 11, 2012 Contact Point Douglas Maughan DHS S&T Cyber Security Division 202-254-6145 Reviewing Official Jonathan R. Cantor Acting
Data Management using irods
Data Management using irods Fundamentals of Data Management September 2014 Albert Heyrovsky Applications Developer, EPCC [email protected] 2 Course outline Why talk about irods? What is irods?
Unified Surveillance Video Analysis and Storage An innovative approach to improve retention, availability, and expand analytic capabilities.
Unified Surveillance Video Analysis and Storage An innovative approach to improve retention, availability, and expand analytic capabilities. NFF White Paper December 2014 ...Most current surveillance solutions
Constructing the Subaru Advanced Data and Analysis Service on VO
Constructing the Subaru Advanced Data and Analysis Service on VO Yuji Shirasaki on behalf of ADC National Astronomical Observatory of Japan Astronomy Data Center Contents Subaru Telescope and Instruments
Creating value in the mobile and fixed businesses
This presentation contains forward-looking information and statements about the Bouygues group and its businesses. Forwardlooking statements may be identified by the use of words such as will, expects,
NDS Environment - A Tour of France
FRENCH EXPERIENCE WITH NDS EIGHTH NDS USERS GROUP MEETING MEXICO 7-9 OCTOBER 2013 Aldine Fabreguettes The French National Agency for Medicines and Health Products Safety Narcotics and Psychotropics Department
CFHT STRIPE82 SURVEY (CS82)
CFHT STRIPE82 SURVEY (CS82) 160 degrees (130 effective) i=24.1, AB mag, 5σ, 2 aperture Seeing ~ 0.6 over full survey Pixel size : 0.187 Alexie Leauthaud. Jean-Paul Kneib. Martin Makler. Ludovic van Waerbeke.
Industrial Internet & Advanced Manufacturing
Industrial Internet & Advanced Manufacturing Marco Annunziata, Chief Economist April 08, 2014 Imagination at work. The Future of Work Advanced Manufacturing + The Industrial Internet + The Global Brain
Data Lake In Action: Real-time, Closed Looped Analytics On Hadoop
1 Data Lake In Action: Real-time, Closed Looped Analytics On Hadoop 2 Pivotal s Full Approach It s More Than Just Hadoop Pivotal Data Labs 3 Why Pivotal Exists First Movers Solve the Big Data Utility Gap
CS521 CSE IITG 11/23/2012
CS521 CSE TG 11/23/2012 A Sahu 1 Degree of overlap Serial, Overlapped, d, Super pipelined/superscalar Depth Shallow, Deep Structure Linear, Non linear Scheduling of operations Static, Dynamic A Sahu slide
EUDAT. Towards a pan-european Collaborative Data Infrastructure. Willem Elbers
EUDAT Towards a pan-european Collaborative Data Infrastructure Willem Elbers EUDAT / MPI-TLA Focus meeting: Data repositories SURF, Utrecht March 3, 2014 Outline EUDAT project EUDAT services Summary and
SGS System Requirements Review
SGS System Requirements Review John Hoar Lausanne 09/06/2015 Quick recap The Science Ground Segment transforms measurements made by the Euclid instruments into data products ready for scientific use. This
SAS BIG DATA SOLUTIONS ON AWS SAS FORUM ESPAÑA, OCTOBER 16 TH, 2014 IAN MEYERS SOLUTIONS ARCHITECT / AMAZON WEB SERVICES
SAS BIG DATA SOLUTIONS ON AWS SAS FORUM ESPAÑA, OCTOBER 16 TH, 2014 IAN MEYERS SOLUTIONS ARCHITECT / AMAZON WEB SERVICES AWS GLOBAL INFRASTRUCTURE 10 Regions 25 Availability Zones 51 Edge locations WHAT
Function Point: how to transform them in effort? This is the problem!
Function Point: how to transform them in effort? This is the problem! Gianfranco Lanza Abstract The need to estimate the effort and, consequently, the cost of a software project is one of the most important
Introduction to the Mathematics of Big Data. Philippe B. Laval
Introduction to the Mathematics of Big Data Philippe B. Laval Fall 2015 Introduction In recent years, Big Data has become more than just a buzz word. Every major field of science, engineering, business,
Oracle ClearTrial Cloud Service Service Descriptions and Metrics August 26, 2014. Metric Definitions... 2 Trial... 2 Instance... 2
Oracle ClearTrial Service Descriptions and Metrics August 26, 2014 Table of Contents Metric Definitions... 2 Trial... 2 Instance... 2 Service Descriptions... 2 Oracle ClearTrial Track -Trial... 2 Oracle