ASKAP Science Data Archive: Users and Requirements CSIRO ASTRONOMY AND SPACE SCIENCE (CASS)

Save this PDF as:
 WORD  PNG  TXT  JPG

Size: px
Start display at page:

Download "ASKAP Science Data Archive: Users and Requirements CSIRO ASTRONOMY AND SPACE SCIENCE (CASS)"

Transcription

1 ASKAP Science Data Archive: Users and Requirements CSIRO ASTRONOMY AND SPACE SCIENCE (CASS) Jessica Chapman, Data Workshop March 2013

2 ASKAP Science Data Archive Talk outline Data flow in brief Some radio astronomy terms Archive overview ASKAP operations Science users and use cases Requirements summary

3 Murchison Radio Observatory Pawsey Centre Worldwide Central processor: Processes the raw visibilities and outputs science data products. Science Data Archive Facility: Data storage and access to science data products

4 ASKAP Central Processor Data from same set of visibilities can be passed through three pipelines for Transient, Continuum and Spectral Line imaging. In principle this allows up to three different experiments to be scheduled simultaneously.

5 latitude Digression: Some radio astronomy terms Visibilities Fourier transform Images Radio astronomy images are created using a Fourier Transform of the observed visibilities. A single image is a map of a region of sky. This may show the sky brightness or another measured parameter. An image cube is a set of images for one region covering a range of frequency (spectral) channels. longitude

6 OH spectral Line emission from an evolved star Continuum image cubes (small-n) have up to 300 frequency channels (300 x 1 MHz) Spectral line image cubes (large-n) up to 16,200 spectral channels Transient observations generate one image every 5 seconds.

7 ASKAP Data Processing levels

8 ASKAP Science Data Archive (ASDA) ASKAP responsible for storage and access to level 5 data products. Survey Science Project Teams responsible for level 6 data validation. Astronomers and their science teams responsible for generation of level 7 enhanced data products.

9 ASDA: Schematic Data Management Layer HSM Tape libraries 2 x 20 PB Disk storage > 4 PB Science Database Nodes Portal Nodes Database cache storage 24 TB Web server 1 Web Server 2 Web Server 3 Web Server 3 V O S E R V I C E S F I R E W A L L Storage ~ 50 TB International astronomy community

10 ASDA Data Flows Schematic

11 ASDA Components Hierarchical Storage Management System with: 2 x 20 PB Tape Libraries (one as backup of other) several PB disk storage Science Database Management System 4 science database nodes with ~ 50 TBytes of storage Web servers with ~ 24 TBytes for cached data storage Virtual Observatory interface to external users Workflow processes User support and training Access to High Performance Computing facilities

12 ASDA: Data Management Layer The HSM will control the transfer of data to and from storage media. An additional data management layer is needed for the science archive. This will include software for: Generation of science metadata and updates to the science database User registration and data access controls User services including web interfaces, search forms, provision of Virtual Observatory services. Workflow management of interactions between the archive, the Central Processor, HSM and users.

13 Science Data Access Access to files, tables and search results will be provided through Virtual Observatory services: Simple Image Access Protocol returns link to images/cubes identified for a given position. Can generate image cut-outs. Cone searches: Returns table results such as positions and fluxes for sources detected within an area around agiven position. Table Access Protocol: Allows for complex querying of tables. For example could return a list of detections for sources above a given flux density with negative spectral indices (slopes). Note: Radio Astronomy is less familiar with using VO services than optical astronomy. Full adoption of VO services will require a cultural shift for users.

14 ASDA Primary Data Products Product Calibrated continuum visibility data (stored as measurement sets ) Continuum (small-n) images and image cubes Spectral line (large-n) image cubes Postage stamp images (quick look images produced with fewer pixels) Continuum source detection tables Spectral line source detections Transient source detections Transient light curves Project Source Catalogues Data type File File File File Table Table Table Table Table

15 Examples: File-based Data Products Product Parameters used Data Size 12-hr calibrated continuum visibility data set Set of three continuum images (restored, residual and model) Set of four polarisation continuum image cubes Single spectral line image cube 36 beams 300 channels 4 polarisations 666 baselines 5s integration Imsize = 10,800 pix 1 pol 1 spectral channel 3 images per position 300 spectral channels 4 pols Imsize = 3,600 pix 16,200 channels 1 pol 2.2 TBytes 5.4 GBytes 556 GBytes 839 GBytes

16 ASKAP Operations Longest baseline is 6 km. 36 antennas gives 630 baselines

17 ASKAP Operations Operated as part of the Australia Telescope National Facility (ATNF) Current ATNF Facilities are: Parkes radio telescope, Australia Telescope Compact Array, Mopra radio telescope, Long Baseline Array National Facility time is allocated on basis of scientific merit and technical feasibility Data taken on ATNF facilities are owned by CSIRO All ASKAP observations taken remotely (usually from Marsfield SOC) Science data available to all astronomers, in most cases with NO proprietary period

18 ASKAP Archive Users Astronomy Group Number of individuals Member of Survey Science Project teams 350 Member of Guest Science Project team 400 General astronomy community 750+ CASS Astrophysics group 30 Supported by CASS Science Operations 30 ivec Operations ~15 Technical developers tbd

19 ASKAP Science Survey Projects (SSPs) Projects need at least 1500 hours of telescope time 10 SSPs were selected in Their science goals are drivers of ASKAP development. SSP teams will work with CASS staff on aspects of science and archive commissioning. Project teams are responsible for their data validation i.e. will assess whether the data taken are of science quality. For bad data, observations may be retaken.

20 ASKAP science: EMU (C) Evolutionary Map of the Universe A deep radio continuum (small-n) survey that will study the evolution of star formation and massive back holes in galaxies on cosmological timescales. The survey is expected to detect about 70 million sources. EMU observations and archiving Observations taken in 12-hour schedule blocks with one block per field. Visibility data stored for 4 polarisation products and 300 spectral channels. Images archived for 1 pol and 1 spectral channel. 11 images per field for full characterisation of spectrum and image quality. Source detection tables stored

21 ASKAP science: WALLABY (S) WALLABY a blind spectral line (large-n) survey for neutral hydrogen. Survey is expected to detect spectral line emission from over 500,000 galaxies. WALLABY data processing and archives Central Processor will process 97 PB of visibility data NOT archived! WALLABY images made with 1 pol and 16,200 spectral channels. 3 images per field archived (residual, restored and mode). Two sets of Postage stamp images also archived. Each field divided into 350 and 1000 sub-regions. These allow for quick looks of smaller regions.

22 ASKAP science: VAST (T) VAST a survey for variable sources and slow transients. Examples are flare stars, intermittent pulsars, X-ray binaries, variable extragalactic sources. VAST data processing and archives VAST observations will mostly be taken in a piggy-back mode, using same input data stream taken for other projects. Visibility data are averaged to ~30 channels. VAST data produce one image every five seconds. Images are searched for sources and detections written to tables. Images are NOT archived. Typically about source detections in 5 seconds. Source detections written as table rows. Images are NOT archived.

23 Survey Science Projects: File-based data sizes in ASDA SSP Type Nfields Time per field Visibility data size (PB) Image data size (PB) EMU C POSSUM C WALLABY S NA 2.1 DINGO S NA 1.3 FLASH S NA 0.04 GASKAP S NA 0.9 VAST T Small NA Notes: Data sizes for two other SSPs: CRAFT (fast transients) and COAST (pulsars) are not included here.

24 Guest Science Projects Will be allocated up to 25% of time. May be scheduled together with SSPs (using parallel data processing pipelines). Science cases not yet established. Proposals will be submitted as for other ATNF facilities on a 6-month cycle. For 750 hours per semester, can expect about 5 10 GSPs to be scheduled.

25 ASDA Example Use Cases Group General use GSP teams Notes: Expertise Level Low Medium Example Use cases (preliminary) Download small number of full-sized images Download larger numbers of postage stamp images Cone searches identify sources of interest Carry out more complex table searches Download light curve information for given source(s) Comparison of ASKAP data with data from other facilities As above plus: Some post-data processing may be requested Some data visualisation Many (but not all) requirements may be largely satisfied through web + VO services.

26 ASDA science use: Survey Science Projects Group SSP teams Expertise Level High Example Use cases (preliminary) As for the general use and: Production of final Source Catalogues High-volume data downloads for data validation Re-analysis of visibility data to generate different images with different parameters Data visualisation across large sky regions Post-analysis of image cubes: image stacking Rapid follow up observations of transient sources Most SSP teams are likely to require access to High Performance Computing and temporary data storage. HPC services are outside the scope of the CASS ASKAP project. They can be provided by ivec and other groups.

27 Preliminary ASDA requirements Essential Requirements Data ingest from CP as the data products become available. Data transfer to tape and/or disk. Indefinite data storage, with backups for all archives files, tables and associated metadata. Potential for data archive to be mirrored to other locations. Disaster recovery plans to minimise loss of data following a crisis such as fire or flood. Efficient data retrieval from tapes and/or disks. Fast access to some data products (Source catalogues, tables, some images..) Extensible and scalable: Design should allow for additional data products, increased data storage and new technologies. User registration through OPAL (as for other ATNF facilities). Validation tool for SSP teams (set metadata flag).

28 Essential Requirements (continued) User queries through web interfaces. Data retrieval across internet using VO services. Queue management for multiple user requests. Administration-level access and tools for Operations staff from CASS and ivec. Highly Desirable Requirements ASKAP provision for SSP teams to upload final source catalogues. User support and access to HPC facilities through ivec.

29 Thank you Jessica Chapman CSIRO Astronomy and Space Science t e w atnf.csiro.au CSIRO ASTRONOMY AND SPACE SCIENCE

CSIRO ASKAP Science Data Archive: Overview, Requirements and Use Cases

CSIRO ASKAP Science Data Archive: Overview, Requirements and Use Cases ` CSIRO ASKAP Science Data Archive: Overview, Requirements and Use Cases ASKAP-SW-0017 Version 1.0 21 May 2014: Status: Open Release Authors: Jessica Chapman (CASS), Ben Humphreys (CASS), Matthew Whiting

More information

The Murchison Widefield Array Data Archive System. Chen Wu Int l Centre for Radio Astronomy Research The University of Western Australia

The Murchison Widefield Array Data Archive System. Chen Wu Int l Centre for Radio Astronomy Research The University of Western Australia The Murchison Widefield Array Data Archive System Chen Wu Int l Centre for Radio Astronomy Research The University of Western Australia Agenda Dataflow Requirements Solutions & Lessons learnt Open solution

More information

ASKAP Survey Science Projects

ASKAP Survey Science Projects ASKAP Survey Science Projects Lewis Ball Acting Director CSIRO Australia Telescope National Facility 1 September 2009 Introduction The Australian Square Kilometre Array Pathfinder (ASKAP) is a next generation

More information

Australian Virtual Observatory

Australian Virtual Observatory Australian Virtual Observatory International Astronomical Union GA 2003 Joint Discussion 08 17th-18th July 2003 Sydney David Barnes The University of Melbourne Our take on virtual observatories bring legacy

More information

Using the Parkes Pulsar Data Archive

Using the Parkes Pulsar Data Archive JART http://www.jart.ac.cn Using the Parkes Pulsar Data Archive J. Khoo 1, G. Hobbs 1, R. N. Manchester 1, D. Miller 2, J. Dempsey 2 1 CSIRO Astronomy and Space Science, Australia Telescope National Facility,

More information

Observer Access to the Cherenkov Telescope Array

Observer Access to the Cherenkov Telescope Array Observer Access to the Cherenkov Telescope Array IRAP, Toulouse, France E-mail: jknodlseder@irap.omp.eu V. Beckmann APC, Paris, France E-mail: beckmann@apc.in2p3.fr C. Boisson LUTh, Paris, France E-mail:

More information

Organization of VizieR's Catalogs Archival

Organization of VizieR's Catalogs Archival Organization of VizieR's Catalogs Archival Organization of VizieR's Catalogs Archival Table of Contents Foreword...2 Environment applied to VizieR archives...3 The archive... 3 The producer...3 The user...3

More information

Science Drivers for Big Data Joseph Lazio SKA Program Development Office & Jet Propulsion Laboratory, California Institute of Technology

Science Drivers for Big Data Joseph Lazio SKA Program Development Office & Jet Propulsion Laboratory, California Institute of Technology Science Drivers for Big Data Joseph Lazio SKA Program Development Office & Jet Propulsion Laboratory, California Institute of Technology 2010 California Institute of Technology. Government sponsorship

More information

This meeting of the AT Users Committee was held at the ATNF Headquarters from 26 th 27 th May 2010.

This meeting of the AT Users Committee was held at the ATNF Headquarters from 26 th 27 th May 2010. ATUC Report to the Director - May 2010 This meeting of the AT Users Committee was held at the ATNF Headquarters from 26 th 27 th May 2010. Attendance: Sarah Maddison (Chair), Stacy Mader (Secretary), Tara

More information

LSST Resources for the Community Lynne Jones University of Washington/LSST

LSST Resources for the Community Lynne Jones University of Washington/LSST LSST Resources for the Community Lynne Jones University of Washington/LSST 1 Data Flow Nightly Operations : (at base facility) Each 15s exposure = 6.44 GB (raw) 2x15s = 1 visit 30 TB / night Generates

More information

ETERNUS CS High End Unified Data Protection

ETERNUS CS High End Unified Data Protection ETERNUS CS High End Unified Data Protection Optimized Backup and Archiving with ETERNUS CS High End 0 Data Protection Issues addressed by ETERNUS CS HE 60% of data growth p.a. Rising back-up windows Too

More information

MAST: The Mikulski Archive for Space Telescopes

MAST: The Mikulski Archive for Space Telescopes MAST: The Mikulski Archive for Space Telescopes Richard L. White Space Telescope Science Institute 2015 April 1, NRC Space Science Week/CBPSS A model for open access The NASA astrophysics data archives

More information

1. Astronomy at millimetre/radio wavelengths

1. Astronomy at millimetre/radio wavelengths Talk Outline 1. Astronomy at millimetre/radio wavelengths 2. Examples of an Australian millimetre and radio telesecope and the kinds of observations that they do: i) the Mopra Telescope ii) the Australia

More information

Douglas Bock Assistant Director Operations ATUC, Oct 2011

Douglas Bock Assistant Director Operations ATUC, Oct 2011 ATNF Operations Douglas Bock Assistant Director Operations ATUC, Oct 2011 Today s program Operations focus! ATNF Operations Strategy Science Ops; Engineering Ops! Mopra observing statistics; impact; time

More information

Software challenges in the implementation of large surveys: the case of J-PAS

Software challenges in the implementation of large surveys: the case of J-PAS Software challenges in the implementation of large surveys: the case of J-PAS 1/21 Paulo Penteado - IAG/USP pp.penteado@gmail.com http://www.ppenteado.net/ast/pp_lsst_201204.pdf (K. Taylor) (A. Fernández-Soto)

More information

Archival of raw and analysed radar data at EISCAT and worldwide

Archival of raw and analysed radar data at EISCAT and worldwide Archival of raw and analysed radar data at EISCAT and worldwide Carl-Fredrik Enell, EISCAT Scientific Association COOPEUS workshop and EGI-CC kickoff, 11 March 2015 C-F Enell, EISCAT Radar data archival

More information

The Virtual Observatory: What is it and how can it help me? Enrique Solano LAEFF / INTA Spanish Virtual Observatory

The Virtual Observatory: What is it and how can it help me? Enrique Solano LAEFF / INTA Spanish Virtual Observatory The Virtual Observatory: What is it and how can it help me? Enrique Solano LAEFF / INTA Spanish Virtual Observatory Astronomy in the XXI century The Internet revolution (the dot com boom ) has transformed

More information

Tier 2 Nearline. As archives grow, Echo grows. Dynamically, cost-effectively and massively. What is nearline? Transfer to Tape

Tier 2 Nearline. As archives grow, Echo grows. Dynamically, cost-effectively and massively. What is nearline? Transfer to Tape Tier 2 Nearline As archives grow, Echo grows. Dynamically, cost-effectively and massively. Large Scale Storage Built for Media GB Labs Echo nearline systems have the scale and performance to allow users

More information

IBM Global Technology Services September 2007. NAS systems scale out to meet growing storage demand.

IBM Global Technology Services September 2007. NAS systems scale out to meet growing storage demand. IBM Global Technology Services September 2007 NAS systems scale out to meet Page 2 Contents 2 Introduction 2 Understanding the traditional NAS role 3 Gaining NAS benefits 4 NAS shortcomings in enterprise

More information

ALMA Overview. Leonardo Testi (European Southern Observatory)

ALMA Overview. Leonardo Testi (European Southern Observatory) ALMA Overview Leonardo Testi (European Southern Observatory) The ALMA Project Status Timeline towards Early and Full Science Operations Future Developments Atacama Large Millimeter Array At least 50x12m

More information

A New Data Visualization and Analysis Tool

A New Data Visualization and Analysis Tool Title: A New Data Visualization and Analysis Tool Author: Kern Date: 22 February 2013 NRAO Doc. #: Version: 1.0 A New Data Visualization and Analysis Tool PREPARED BY ORGANIZATION DATE Jeff Kern NRAO 22

More information

Canadian Astronomy Data Centre. Séverin Gaudet David Schade Canadian Astronomy Data Centre

Canadian Astronomy Data Centre. Séverin Gaudet David Schade Canadian Astronomy Data Centre Canadian Astronomy Data Centre Séverin Gaudet David Schade Canadian Astronomy Data Centre Data Activities in Astronomy Features of the astronomy data landscape Multi-wavelength datasets are increasingly

More information

Peter Quinn, Director, Astrophysical Virtual Observatory; European Southern Observatory, Garching, Germany

Peter Quinn, Director, Astrophysical Virtual Observatory; European Southern Observatory, Garching, Germany The International Virtual Observatory Robert Hanisch, Project Manager, US National Virtual Observatory, and Chair, International Virtual Observatory Alliance; Space Telescope Science Institute, Baltimore,

More information

Lecture 5b: Data Mining. Peter Wheatley

Lecture 5b: Data Mining. Peter Wheatley Lecture 5b: Data Mining Peter Wheatley Data archives Most astronomical data now available via archives Raw data and high-level products usually available Data reduction software often specific to individual

More information

Big Data, Big Challenges and new Paradigm for the Gaia Archive

Big Data, Big Challenges and new Paradigm for the Gaia Archive Big Data, Big Challenges and new Paradigm for the Gaia Archive Christophe Arviset Head of ESAC Science Data Centre 15/03/2016 Issue/Revision: 1.0 Reference: Gaia Archive's big data challenges Status: Issued

More information

How Does the Cloud Fit into Active Archiving. Active Archive Alliance Panel

How Does the Cloud Fit into Active Archiving. Active Archive Alliance Panel How Does the Cloud Fit into Active Archiving Active Archive Alliance Panel Active Archive Alliance Founded 2010 The Active Archive Alliance is a multi-vendor association aimed at evolving new technologies

More information

LSST and the Cloud: Astro Collaboration in 2016 Tim Axelrod LSST Data Management Scientist

LSST and the Cloud: Astro Collaboration in 2016 Tim Axelrod LSST Data Management Scientist LSST and the Cloud: Astro Collaboration in 2016 Tim Axelrod LSST Data Management Scientist DERCAP Sydney, Australia, 2009 Overview of Presentation LSST - a large-scale Southern hemisphere optical survey

More information

SAN Conceptual and Design Basics

SAN Conceptual and Design Basics TECHNICAL NOTE VMware Infrastructure 3 SAN Conceptual and Design Basics VMware ESX Server can be used in conjunction with a SAN (storage area network), a specialized high speed network that connects computer

More information

UNINETT Sigma2 AS: architecture and functionality of the future national data infrastructure

UNINETT Sigma2 AS: architecture and functionality of the future national data infrastructure UNINETT Sigma2 AS: architecture and functionality of the future national data infrastructure Authors: A O Jaunsen, G S Dahiya, H A Eide, E Midttun Date: Dec 15, 2015 Summary Uninett Sigma2 provides High

More information

The PACS Software System. (A high level overview) Prepared by : E. Wieprecht, J.Schreiber, U.Klaas November,5 2007 Issue 1.

The PACS Software System. (A high level overview) Prepared by : E. Wieprecht, J.Schreiber, U.Klaas November,5 2007 Issue 1. The PACS Software System (A high level overview) Prepared by : E. Wieprecht, J.Schreiber, U.Klaas November,5 2007 Issue 1.0 PICC-ME-DS-003 1. Introduction The PCSS, the PACS ICC Software System, is the

More information

The Very High Energy source catalogue at the ASI Science Data Center

The Very High Energy source catalogue at the ASI Science Data Center The Very High Energy source catalogue at the ASI Science Data Center a,b, F. Lucarelli a,b, L. A. Antonelli a,b, P. Giommi b E-mail: alessandro.carosi@oa-roma.inaf.it a INAF, National Institute for Astrophysics,

More information

ALMA Science Operations

ALMA Science Operations ALMA Science Operations Lars-Åke Nyman a, Paola Andreani b, John Hibbard c, Sachiko K. Okumura d a JAO, Alonso de Cordova 3107, Vitacura, Santiago, Chile b ESO, Karl-Schwarzschild-Strasse 2, D-85748 Garching

More information

Astrophysics with Terabyte Datasets. Alex Szalay, JHU and Jim Gray, Microsoft Research

Astrophysics with Terabyte Datasets. Alex Szalay, JHU and Jim Gray, Microsoft Research Astrophysics with Terabyte Datasets Alex Szalay, JHU and Jim Gray, Microsoft Research Living in an Exponential World Astronomers have a few hundred TB now 1 pixel (byte) / sq arc second ~ 4TB Multi-spectral,

More information

CSIRO Australia Telescope National Facility. OPAL Version 3. Online Proposal Applications & Links. Jessica Chapman Arkadi Kosmynin

CSIRO Australia Telescope National Facility. OPAL Version 3. Online Proposal Applications & Links. Jessica Chapman Arkadi Kosmynin CSIRO Australia Telescope National Facility OPAL Version 3 Online Proposal Applications & Links Jessica Chapman Arkadi Kosmynin Requirements Specification Document version 1.0 10 June 2009 1 Table of Contents

More information

This work was done in collaboration with the Centre de Données astronomiques de Strasbourg (CDS) and in particular with F. X. Pineau.

This work was done in collaboration with the Centre de Données astronomiques de Strasbourg (CDS) and in particular with F. X. Pineau. Chapter 5 Database and website This work was done in collaboration with the Centre de Données astronomiques de Strasbourg (CDS) and in particular with F. X. Pineau. 5.1 Introduction The X-shooter Spectral

More information

巨 量 資 料 分 層 儲 存 解 決 方 案

巨 量 資 料 分 層 儲 存 解 決 方 案 巨 量 資 料 分 層 儲 存 解 決 方 案 Lower Costs and Improve Efficiencies Cano Lei Senior Sales Consulting Manager Oracle Systems Oracle Confidential Internal/Restricted/Highly Restricted Agenda 1 2 3 Why Tiered Storage?

More information

ALMA Operations the role of the European ALMA Regional Centre Plans for cycle 1

ALMA Operations the role of the European ALMA Regional Centre Plans for cycle 1 ALMA Operations the role of the European ALMA Regional Centre Plans for cycle 1 Paola Andreani ALMA Operations Sites in Chile Antenna Operations Site (AOS) Operation Support Facility (OSF) 60 MB/s (peak)

More information

STORAGE. 2015 Arka Service s.r.l.

STORAGE. 2015 Arka Service s.r.l. STORAGE STORAGE MEDIA independently from the repository model used, data must be saved on a support (data storage media). Arka Service uses the most common methods used as market standard such as: MAGNETIC

More information

The Astronomical Data Warehouse

The Astronomical Data Warehouse The Astronomical Data Warehouse Clive G. Page Department of Physics & Astronomy, University of Leicester, Leicester, LE1 7RH, UK Abstract. The idea of the astronomical data warehouse has arisen as the

More information

HPC and Big Data. EPCC The University of Edinburgh. Adrian Jackson Technical Architect a.jackson@epcc.ed.ac.uk

HPC and Big Data. EPCC The University of Edinburgh. Adrian Jackson Technical Architect a.jackson@epcc.ed.ac.uk HPC and Big Data EPCC The University of Edinburgh Adrian Jackson Technical Architect a.jackson@epcc.ed.ac.uk EPCC Facilities Technology Transfer European Projects HPC Research Visitor Programmes Training

More information

Energy and Space Efficient Storage: Multi-tier Strategies for Protecting and Retaining Data

Energy and Space Efficient Storage: Multi-tier Strategies for Protecting and Retaining Data Energy and Space Efficient Storage: Multi-tier Strategies for Protecting and Retaining Data NOTICE This White Paper may contain proprietary information protected by copyright. Information in this White

More information

Ionospheric Research with the LOFAR Telescope

Ionospheric Research with the LOFAR Telescope Ionospheric Research with the LOFAR Telescope Leszek P. Błaszkiewicz Faculty of Mathematics and Computer Science, UWM Olsztyn LOFAR - The LOw Frequency ARray The LOFAR interferometer consist of a large

More information

An introduction to the CDS services and tools

An introduction to the CDS services and tools An introduction to the CDS services and tools Ada Nebot Gomez-Moran, Mark Allen, and the CDS team Centre de Données astronomiques de Strasbourg, France Introduction The CDS harbours three major programs,

More information

Amazon Cloud Storage Options

Amazon Cloud Storage Options Amazon Cloud Storage Options Table of Contents 1. Overview of AWS Storage Options 02 2. Why you should use the AWS Storage 02 3. How to get Data into the AWS.03 4. Types of AWS Storage Options.03 5. Object

More information

The safer, easier way to help you pass any IT exams. Exam : 000-115. Storage Sales V2. Title : Version : Demo 1 / 5

The safer, easier way to help you pass any IT exams. Exam : 000-115. Storage Sales V2. Title : Version : Demo 1 / 5 Exam : 000-115 Title : Storage Sales V2 Version : Demo 1 / 5 1.The IBM TS7680 ProtecTIER Deduplication Gateway for System z solution is designed to provide all of the following EXCEPT: A. ESCON attach

More information

IBM Tivoli Storage Manager Version 7.1.4. Introduction to Data Protection Solutions IBM

IBM Tivoli Storage Manager Version 7.1.4. Introduction to Data Protection Solutions IBM IBM Tivoli Storage Manager Version 7.1.4 Introduction to Data Protection Solutions IBM IBM Tivoli Storage Manager Version 7.1.4 Introduction to Data Protection Solutions IBM Note: Before you use this

More information

Concepts and Architecture of Grid Computing. Advanced Topics Spring 2008 Prof. Robert van Engelen

Concepts and Architecture of Grid Computing. Advanced Topics Spring 2008 Prof. Robert van Engelen Concepts and Architecture of Grid Computing Advanced Topics Spring 2008 Prof. Robert van Engelen Overview Grid users: who are they? Concept of the Grid Challenges for the Grid Evolution of Grid systems

More information

EChO Ground Segment: Overview & Science Operations Assumptions

EChO Ground Segment: Overview & Science Operations Assumptions EChO Ground Segment: Overview & Science Operations Assumptions Matthias Ehle & the Science Ground Segment Working Group EChO Science Operations Study Manager ESA-ESAC, Madrid Science Operations Department/Division

More information

Scientific Computing Meets Big Data Technology: An Astronomy Use Case

Scientific Computing Meets Big Data Technology: An Astronomy Use Case Scientific Computing Meets Big Data Technology: An Astronomy Use Case Zhao Zhang AMPLab and BIDS UC Berkeley zhaozhang@cs.berkeley.edu In collaboration with Kyle Barbary, Frank Nothaft, Evan Sparks, Oliver

More information

Protecting Information in a Smarter Data Center with the Performance of Flash

Protecting Information in a Smarter Data Center with the Performance of Flash 89 Fifth Avenue, 7th Floor New York, NY 10003 www.theedison.com 212.367.7400 Protecting Information in a Smarter Data Center with the Performance of Flash IBM FlashSystem and IBM ProtecTIER Printed in

More information

Charting the Transient Radio Sky on Sub-Second Time-Scales with LOFAR

Charting the Transient Radio Sky on Sub-Second Time-Scales with LOFAR Charting the Transient Radio Sky on Sub-Second Time-Scales with LOFAR Jason Hessels (ASTRON / UvA) with the LOFAR Pulsar Working Group (Stappers, van Leeuwen, Kramer, Karastergiou, Alexov, Hassall, Noutsos,

More information

Take Your Tax Practice to New Heights

Take Your Tax Practice to New Heights Take Your Tax Practice to New Heights TM AgileTax.com AGILE TAX Agile Tax is the first professional tax software designed exclusively for the cloud. The intuitive, easy-to-use online software takes full

More information

EA-ARC ALMA ARCHIVE DATA USER GUIDEBOOK

EA-ARC ALMA ARCHIVE DATA USER GUIDEBOOK EA-ARC ALMA ARCHIVE DATA USER GUIDEBOOK Prepared by James O. Chibueze NAOJ Chile Observatory Purpose of this handbook: This handbook is aimed at providing fundamental information on how and where to access

More information

DMF & Tiering Update. Kirill Malkin Director of Storage Engineering. September 2015

DMF & Tiering Update. Kirill Malkin Director of Storage Engineering. September 2015 DMF & Tiering Update Kirill Malkin Director of Storage Engineering September 2015 1 What s New in DMF? Data Migration Facility (DMF) Data management solution for HPC/HPDA Over 20 years of data management

More information

Holdings and distribution

Holdings and distribution NASA Data Center Annual Program Plan NASA Grant Number NNX09AF08G / STScI Grant J1346 March 2010 through February 2011 The Multi-Mission Archive at Space Telescope Science Institute (MAST) Phase II (Optical/UV

More information

Data Mining Challenges and Opportunities in Astronomy

Data Mining Challenges and Opportunities in Astronomy Data Mining Challenges and Opportunities in Astronomy S. G. Djorgovski (Caltech) With special thanks to R. Brunner, A. Szalay, A. Mahabal, et al. The Punchline: Astronomy has become an immensely datarich

More information

XenData Product Brief: SX-520 Series Servers for Sony Optical Disc Archives

XenData Product Brief: SX-520 Series Servers for Sony Optical Disc Archives XenData Product Brief: SX-520 Series Servers for Sony Optical Disc Archives The SX-520 Series of Archive Servers creates highly scalable Optical Disc Digital Video Archives that are optimized for broadcasters,

More information

An ArrayLibraryforMS SQL Server

An ArrayLibraryforMS SQL Server An ArrayLibraryforMS SQL Server Scientific requirements and an implementation László Dobos 1,2 --dobos@complex.elte.hu Alex Szalay 2, José Blakeley 3, Tamás Budavári 2, István Csabai 1,2, Dragan Tomic

More information

A Preliminary Summary of The VLA Sky Survey

A Preliminary Summary of The VLA Sky Survey A Preliminary Summary of The VLA Sky Survey Eric J. Murphy and Stefi Baum (On behalf of the entire Science Survey Group) 1 Executive Summary After months of critical deliberation, the Survey Science Group

More information

A Backup Strategy for Informatics Craig Strachan Version 1.0 31/01/2011

A Backup Strategy for Informatics Craig Strachan Version 1.0 31/01/2011 A Backup Strategy for Informatics Craig Strachan Version 1.0 31/01/2011 This document identifies the various types of backup being carried out in the School of Informatics, briefly considers the characteristics

More information

Cosmic Variability Study in Taiwan

Cosmic Variability Study in Taiwan Cosmic Variability Study in Taiwan Wen-Ping Chen Institute of Astronomy National Central University, Taiwan 2010 November 16@Jena/YETI Advantages in Taiwan: - Many high mountains - Western Pacific longitude

More information

WIYN ODI: Observing Process, Data Analysis and Archiving

WIYN ODI: Observing Process, Data Analysis and Archiving WIYN ODI: Observing Process, Data Analysis and Archiving Pierre Martin Yale Survey Workshop, October 2009 ODI: Scientific Challenges ODI is designed to take advantage of the best seeing conditions at WIYN.

More information

TECHNICAL GUIDELINES FOR APPLICANTS TO PRACE 7 th CALL (Tier-0)

TECHNICAL GUIDELINES FOR APPLICANTS TO PRACE 7 th CALL (Tier-0) TECHNICAL GUIDELINES FOR APPLICANTS TO PRACE 7 th CALL (Tier-0) Contributing sites and the corresponding computer systems for this call are: GCS@Jülich, Germany IBM Blue Gene/Q GENCI@CEA, France Bull Bullx

More information

Objectif. Participant. Prérequis. Pédagogie. Oracle Database 11g - Administration Workshop II - LVC. 5 Jours [35 Heures]

Objectif. Participant. Prérequis. Pédagogie. Oracle Database 11g - Administration Workshop II - LVC. 5 Jours [35 Heures] Objectif Back up and recover a database Configure Oracle Database for optimal recovery Administer ASM disk groups Use an RMAN backup to duplicate a database Automating Tasks with the Scheduler Participant

More information

Introduction to Optical Archiving Library Solution for Long-term Data Retention

Introduction to Optical Archiving Library Solution for Long-term Data Retention Introduction to Optical Archiving Library Solution for Long-term Data Retention CUC Solutions & Hitachi-LG Data Storage 0 TABLE OF CONTENTS 1. Introduction... 2 1.1 Background and Underlying Requirements

More information

Data Management Plan Extended Baryon Oscillation Spectroscopic Survey

Data Management Plan Extended Baryon Oscillation Spectroscopic Survey Data Management Plan Extended Baryon Oscillation Spectroscopic Survey Experiment description: eboss is the cosmological component of the fourth generation of the Sloan Digital Sky Survey (SDSS-IV) located

More information

XenData Product Brief: SX-550 Series Servers for LTO Archives

XenData Product Brief: SX-550 Series Servers for LTO Archives XenData Product Brief: SX-550 Series Servers for LTO Archives The SX-550 Series of Archive Servers creates highly scalable LTO Digital Video Archives that are optimized for broadcasters, video production

More information

Disk-to-Disk-to-Offsite Backups for SMBs with Retrospect

Disk-to-Disk-to-Offsite Backups for SMBs with Retrospect Disk-to-Disk-to-Offsite Backups for SMBs with Retrospect Abstract Retrospect backup and recovery software provides a quick, reliable, easy-to-manage disk-to-disk-to-offsite backup solution for SMBs. Use

More information

Aloke Guha. CTO, COPAN Systems. 23nd IEEE13th NASA Mass Storage Conference May 18, 2006

Aloke Guha. CTO, COPAN Systems. 23nd IEEE13th NASA Mass Storage Conference May 18, 2006 Data Archiving Using Enhanced MAID Aloke Guha CTO, 23nd IEEE13th NASA Mass Storage Conference May 18, 2006 Archive Data Needs Retention and Disposition Performance Corporate governance and regulatory guidelines

More information

XenData Archive Series Software Technical Overview

XenData Archive Series Software Technical Overview XenData White Paper XenData Archive Series Software Technical Overview Advanced and Video Editions, Version 4.0 December 2006 XenData Archive Series software manages digital assets on data tape and magnetic

More information

2011 European HyperWorks Technology Conference. Vladi Nosenzo, Roberto Vadori

2011 European HyperWorks Technology Conference. Vladi Nosenzo, Roberto Vadori 2011 European HyperWorks Technology Conference Vladi Nosenzo, Roberto Vadori 20 Novembre, 2010 2011 ABSTRACT The work described below starts from an idea of a previous experience of Reply, developed in

More information

Multi-Terabyte Archives for Medical Imaging Applications

Multi-Terabyte Archives for Medical Imaging Applications Multi-Terabyte Archives for Medical Imaging Applications This paper describes how Windows servers running XenData Archive Series software provide an attractive solution for storing and retrieving multiple

More information

Big Data and Cloud Computing for GHRSST

Big Data and Cloud Computing for GHRSST Big Data and Cloud Computing for GHRSST Jean-Francois Piollé (jfpiolle@ifremer.fr) Frédéric Paul, Olivier Archer CERSAT / Institut Français de Recherche pour l Exploitation de la Mer Facing data deluge

More information

Wide-Field Plate Database: Service for Astronomy

Wide-Field Plate Database: Service for Astronomy Wide-Field Plate Database: Service for Astronomy Milcho K. Tsvetkov To cite this version: Milcho K. Tsvetkov. Wide-Field Plate Database: Service for Astronomy. IMCCE. International Workshop NAROO-GAIA

More information

SCALABLE FILE SHARING AND DATA MANAGEMENT FOR INTERNET OF THINGS

SCALABLE FILE SHARING AND DATA MANAGEMENT FOR INTERNET OF THINGS Sean Lee Solution Architect, SDI, IBM Systems SCALABLE FILE SHARING AND DATA MANAGEMENT FOR INTERNET OF THINGS Agenda Converging Technology Forces New Generation Applications Data Management Challenges

More information

Indiana University Science with the WIYN One Degree Imager

Indiana University Science with the WIYN One Degree Imager Indiana University Science with the WIYN One Degree Imager Katherine Rhode (Indiana University, WIYN SAC member) Indiana University Department of Astronomy Nine faculty members, plus active emeritus faculty

More information

A6523 Signal Modeling, Statistical Inference and Data Mining in Astrophysics Spring 2015

A6523 Signal Modeling, Statistical Inference and Data Mining in Astrophysics Spring 2015 1/28/15 Astronomy 6523 Modeling, Inference & Data Mining in Astrophysics Professor J. Cordes 622 Space Sciences Building Tuesday Thursday 1:25-2:40 pm http:/www.astro.cornell.edu/~cordes/a6523 Bayesian

More information

Redefining Microsoft SQL Server Data Management. PAS Specification

Redefining Microsoft SQL Server Data Management. PAS Specification Redefining Microsoft SQL Server Data Management APRIL Actifio 11, 2013 PAS Specification Table of Contents Introduction.... 3 Background.... 3 Virtualizing Microsoft SQL Server Data Management.... 4 Virtualizing

More information

Introduction to the VO. Christophe.Arviset@esa.int ESAVO ESA/ESAC Madrid, Spain

Introduction to the VO. Christophe.Arviset@esa.int ESAVO ESA/ESAC Madrid, Spain Introduction to the VO Christophe.Arviset@esa.int ESAVO ESA/ESAC Madrid, Spain The way Astronomy works Telescopes (ground- and space-based, covering the full electromagnetic spectrum) Observatories Instruments

More information

MANAGING AND MINING THE LSST DATA SETS

MANAGING AND MINING THE LSST DATA SETS MANAGING AND MINING THE LSST DATA SETS Astronomy is undergoing an exciting revolution -- a revolution in the way we probe the universe and the way we answer fundamental questions. New technology enables

More information

Data Data Everywhere, We are now in the Big Data era

Data Data Everywhere, We are now in the Big Data era Data Data Everywhere, We are now in the Big Data era Mike Liddell 13 th March 2014 UUVs @ OI 2014 Contents Menu Background Big Data What is Generating our Big Data Physical Management of Big Data Optimisation

More information

Constructing the Subaru Advanced Data and Analysis Service on VO

Constructing the Subaru Advanced Data and Analysis Service on VO Constructing the Subaru Advanced Data and Analysis Service on VO Yuji Shirasaki on behalf of ADC National Astronomical Observatory of Japan Astronomy Data Center Contents Subaru Telescope and Instruments

More information

Archiving the insurance data warehouse

Archiving the insurance data warehouse REFLECTIONS Archiving the insurance data warehouse Surajit Basu, IBEXI Solutions Surajit Basu is an IT consultant with over 20 years of experience in design, development and implementation of enterprise-wide

More information

LOFAR Software Development Roadmap

LOFAR Software Development Roadmap Netherlands Institute for Radio Astronomy LOFAR Software Development Roadmap LOFAR User Meeting 01/06/2015, Hanno Holties The team: Adriaan Renting, Arno Schoenmakers, Arthur Coolen, Jan David Mol, Jan

More information

Oracle Database 10g: Backup and Recovery 1-2

Oracle Database 10g: Backup and Recovery 1-2 Oracle Database 10g: Backup and Recovery 1-2 Oracle Database 10g: Backup and Recovery 1-3 What Is Backup and Recovery? The phrase backup and recovery refers to the strategies and techniques that are employed

More information

Einstein Rings: Nature s Gravitational Lenses

Einstein Rings: Nature s Gravitational Lenses National Aeronautics and Space Administration Einstein Rings: Nature s Gravitational Lenses Leonidas Moustakas and Adam Bolton Taken from: Hubble 2006 Science Year in Review The full contents of this book

More information

Object Oriented Storage and the End of File-Level Restores

Object Oriented Storage and the End of File-Level Restores Object Oriented Storage and the End of File-Level Restores Stacy Schwarz-Gardner Spectra Logic Agenda Data Management Challenges Data Protection Data Recovery Data Archive Why Object Based Storage? The

More information

CONFIGURATION GUIDELINES: EMC STORAGE FOR PHYSICAL SECURITY

CONFIGURATION GUIDELINES: EMC STORAGE FOR PHYSICAL SECURITY White Paper CONFIGURATION GUIDELINES: EMC STORAGE FOR PHYSICAL SECURITY DVTel Latitude NVMS performance using EMC Isilon storage arrays Correct sizing for storage in a DVTel Latitude physical security

More information

8.1 Radio Emission from Solar System objects

8.1 Radio Emission from Solar System objects 8.1 Radio Emission from Solar System objects 8.1.1 Moon and Terrestrial planets At visible wavelengths all the emission seen from these objects is due to light reflected from the sun. However at radio

More information

A Virtual Tape Library Architecture & Its Benefits

A Virtual Tape Library Architecture & Its Benefits A Virtual Tape Library Architecture & Its Benefits Mas Omae Ultera Systems, Inc. A Decade of Enhancing Removable Storage 26052 Merit Cir., Laguna Hills CA 92653 Phone: +1-949-367-8800 FAX: +1-949-367-0758

More information

LSST Data Management. Tim Axelrod Project Scientist - LSST Data Management. Thursday, 28 Oct 2010

LSST Data Management. Tim Axelrod Project Scientist - LSST Data Management. Thursday, 28 Oct 2010 LSST Data Management Tim Axelrod Project Scientist - LSST Data Management Thursday, 28 Oct 2010 Outline of the Presentation LSST telescope and survey Functions and architecture of the LSST data management

More information

Backup and Recovery 1

Backup and Recovery 1 Backup and Recovery What is a Backup? Backup is an additional copy of data that can be used for restore and recovery purposes. The Backup copy is used when the primary copy is lost or corrupted. This Backup

More information

AGILE Data Center at ASDC: Overview, Catalogs and Science Tools

AGILE Data Center at ASDC: Overview, Catalogs and Science Tools AGILE Data Center at ASDC: Overview, Catalogs and Science Tools Carlotta Pittori, ASDC on behalf of the AGILE Data Center 12 th AGILE Science Workshop, May 8-9 2014 April 23, 2007: Launch! Equatorial orbit:

More information

The Giant Metre-wave Radio Telescope

The Giant Metre-wave Radio Telescope The Giant Metre-wave Radio Telescope - Suhas Naik-Satam Radio astronomy: Radio astronomy is the study of distant objects in the universe by collecting and analyzing the radio waves emitted by those objects.

More information

Solution Brief: Creating Avid Project Archives

Solution Brief: Creating Avid Project Archives Solution Brief: Creating Avid Project Archives Marquis Project Parking running on a XenData Archive Server provides Fast and Reliable Archiving to LTO or Sony Optical Disc Archive Cartridges Summary Avid

More information

Alexandria Overview. Sept 4, 2015

Alexandria Overview. Sept 4, 2015 Alexandria Overview Sept 4, 2015 Alexandria 1U System Block Diagram SAS Interface Board Zoneboard Zoneboard I2C UART SAS to SATA I2C 12V AC Power Supply Power 60w Supply Seagate Confidential Alexandria

More information

IBM Global Technology Services November 2009. Successfully implementing a private storage cloud to help reduce total cost of ownership

IBM Global Technology Services November 2009. Successfully implementing a private storage cloud to help reduce total cost of ownership IBM Global Technology Services November 2009 Successfully implementing a private storage cloud to help reduce total cost of ownership Page 2 Contents 2 Executive summary 3 What is a storage cloud? 3 A

More information

The Status of Astronomical Control Systems. Steve Scott OVRO

The Status of Astronomical Control Systems. Steve Scott OVRO The Status of Astronomical Control Systems Steve Scott OVRO Astronomical Control System Status 10 years ago radio control systems were much more sophisticated than optical. Why? Single pixel detectors

More information

Sales Tool. Summary DXi Sales Messages November 2009 6 NOVEMBER 2009. ST00431-v06

Sales Tool. Summary DXi Sales Messages November 2009 6 NOVEMBER 2009. ST00431-v06 Summary DXi Sales Messages November 2009 6 NOVEMBER 2009 ST00431-v06 Notice This Sales Tool contains proprietary information protected by copyright. Information in this Sales Tool is subject to change

More information

EMC NETWORKER AND DATADOMAIN

EMC NETWORKER AND DATADOMAIN EMC NETWORKER AND DATADOMAIN Capabilities, options and news Madis Pärn Senior Technology Consultant EMC madis.parn@emc.com 1 IT Pressures 2009 0.8 Zettabytes 2020 35.2 Zettabytes DATA DELUGE BUDGET DILEMMA

More information