Software challenges in the implementation of large surveys: the case of J-PAS
|
|
|
- Norman Riley
- 7 years ago
- Views:
Transcription
1 Software challenges in the implementation of large surveys: the case of J-PAS 1/21 Paulo Penteado - IAG/USP pp.penteado@gmail.com (K. Taylor) (A. Fernández-Soto)
2 Outline 2/21 J-PAS overview The problems: Going from survey strategy to taking daily observations. Going from daily observations to scientific data products. Going from scientific data products to scientific results. Survey execution: Strategy / Scheduler Observatory control Data products: Reduction Pipeline Generation of data products Archival, access and visualization pp.penteado@gmail.com
3 J-PAS overview 3/21 Javalambre PAU (Physics of the Accelerating Universe) Astrophysical Survey Two telescopes T80 (80 cm) and T250 (2.5 m): photometric survey. 54-point spectrum over Å redshift precision σ z 0.003(1 + z) for over 14 million galaxies to measure BAOs (Baryon Acoustic Oscillations) to constrain cosmological parameters. High spatial resolution, low spectral resolution of a large area of the sky; useful for cosmology, study of galaxy clusters, stellar populations (MW and nearby galaxies), variable objects (supernovae, variable stars, Solar System).
4 4/21 J-PAS overview T80 Telescope: 80 cm 1-CCD camera 2 FoV, 81 Mpixel 12 filters (J. Cenarro)
5 J-PAS overview 5/21 T250 telescope: 2.5 m 14-CCD camera 3 FoV 1.1 Gpixel 56 filters (K. Taylor) (J. Cenarro)
6 J-PAS overview - Filter systems: 6/21 T80: 12 filters: 4 SDSS (g,r,i,z) + 7: (A. Marín-Franch)
7 J-PAS overview - Filter systems: 7/21 T250: 2 wide-band 54 narrow-band filters: (A. Marín-Franch)
8 J-PAS overview - Survey: 8/21 Above 40 elevation; Away from the Galactic plane; Away from Galactic dust; Overlapping with SDSS; Wide shallow + Narrow deep coverage; 3 exposures over the whole area. (A. Fernández-Soto)
9 What is common between J-PAS and LSST? 9/21 Both surveys: Large-area coverage; Wide-field contiguous imaging; Multi-band imaging; Will produce large volumes of data to process, store and analyze. All face the same main software challenges: Going from survey strategy to taking daily observations. Going from daily observations to scientific data products. Going from scientific data products to scientific results.
10 Going from survey strategy to taking daily observations. 10/21 Survey execution software: Strategy Implements the strategy constraints (area x depth to cover, cadence, filter sequencing) to determine where to look each night: J-PAS survey simulation (A. Fernández-Soto): Simulations indicate where to change the strategy: Some regions can be observed deeper / more times Some strategies have too much overhead (pointing, instrument changes, etc.)
11 Survey execution software: Scheduler 11/21 Decides the exact observation sequence to take each night. Needs to be dynamic: Account for downtime, failures, weather changes (not binary) Decide on the compromise between fields (not all fields can be observed at optimal time) (A. Ederoclite)
12 Going from survey strategy to taking daily observations. 12/21 Survey execution software: Observatory control Integrates all systems. Ex (for J-PAS / PAU-SUL):
13 Going from daily observations to scientific data products. 13/21 Data products: Reduction Pipeline The pipeline is not only low level (radiometric, coadd, astrometric, etc.) In large datasets, user cannot process everything: The pipeline must generate many scientific products that might be needed (and put them in the catalog). Minimal set of necessary scientific products: Quality measurement (also used by the scheduler). Source location (in images) and identification (in source catalogs). Source classification (stars, galaxies, nebulae, Solar System). Basic measurements (aperture/psf/isophotal photometry, morphology). Derived measurements (redshift, distance, spectral type, stellar type, temperature, mass).
14 Going from scientific data products to scientific results. 14/21 Data products: volume DSS CDROMs GB (images) 2MASS DVDs, 43 GB (catalog) 10 TB (images) SDSS TB (catalog) 11.6 TB (images) JPAS T80 -??? (catalog) 87 TB (images) (~8 times SDSS) JPAS T250 -??? (catalog) 2.3 PB (images) (~ 210 times SDSS) (W. Schoenell) LSST: 200 PB Storing the data is not that hard: LSST will never fill more than 22 hard drives. Individual investigators will be able to maintain their own data copies to analyze as they choose. With the data set increasing linearly with time, and storage doubling every 1.6 years, the peak storage cost occurs 2.3 years into any survey. From A Letter to the NSF Astronomy Portfolio Review: LSST is Not Big Data, David Schlegel (Lawrence Berkeley National Lab),
15 Going from scientific data products to scientific results. 15/21 Data products: volume Storing the data is not that hard. Neither is processing. But accessing can be difficult: Since 80 s: Sequential read increased from 20MB/s to 80MB/s (fac of 4) Hard disk speed increased from 3k to 15k RPM (factor of 5) Storage cap. increased from 1GB to 2TB (factor of 2k!) (W. Schoenell) Storage (not only processing) will have to be massively parallel. New software solutions necessary to do parallel processing on the distributed storage (cannot just use a big storage server).
16 Going from daily observations to scientific data products. 16/21 Data products: Archival, access and visualization Archival is not just shoving all files in a hard drive. The success of any observation depends on how the data being: Systematically processed (a well-documented pipeline); Well-organized (a good database is necessary if one has more than a few observations); Accessible: The database must already have lots of fields with the variables that may be needed. VO protocols must be followed. Not optional! User-friendly access and visualization of queries is essential if the data is ever going to be used.
17 Millenium simulation interface GAVO (German Astrophysical VO). Going from scientific data products to scientific results. 17/21
18 Going from scientific data products to scientific results 18/21 One (true) tale: Cassini VIMS (Visual and Infrared Mapping Spectrometer) data: ~2x10 7 spectra of Titan recorded since 2004 (and still going) All processed data publicly accessible through NASA's PDS (Planetary Data System): All the usual metadata fields (exposure settings, geometry on target, etc.) But PDS was insufficient: After several years in the public archive, the first tropical lake was found only with the use of a new system which integrates database (with more fields, both metadata and data), flexible query evaluation and visualization.
19 19/21 Going from scientific data products to scientific results (Penteado 2012, in prep.) (Griffith, Lora, Turner, Penteado et al., 2012; submitted to Nature)
20 Going from scientific data products to scientific results 20/21 A promising new database system: SciDB ( A data model based on multidimensional arrays, not sets of tuples. A storage model based on versions and not update in place. Built-in support for provenance (lineage), workflows, and uncertainty. Scalability to 100s of petabytes and 1,000s of nodes with fault tolerance. Support for "external" data objects so that data sets can be queried and manipulated without ever having to be loaded into the database. Open source in order to foster a community of contributors and to ensure that data is never "locked up" a critical requirement for scientists.. Mapping layer will have interfaces to at least C++, Java, and Python (W. Schoenell)
21 Outline 21/21 J-PAS overview The problems: Going from survey strategy to taking daily observations. Going from daily observations to scientific data products. Going from scientific data products to scientific results. Survey execution: Strategy / Scheduler Observatory control Data products: Reduction Pipeline Generation of data products Archival, access and visualization pp.penteado@gmail.com
Indiana University Science with the WIYN One Degree Imager
Indiana University Science with the WIYN One Degree Imager Katherine Rhode (Indiana University, WIYN SAC member) Indiana University Department of Astronomy Nine faculty members, plus active emeritus faculty
Data Management Plan Extended Baryon Oscillation Spectroscopic Survey
Data Management Plan Extended Baryon Oscillation Spectroscopic Survey Experiment description: eboss is the cosmological component of the fourth generation of the Sloan Digital Sky Survey (SDSS-IV) located
Description of the Dark Energy Survey for Astronomers
Description of the Dark Energy Survey for Astronomers May 1, 2012 Abstract The Dark Energy Survey (DES) will use 525 nights on the CTIO Blanco 4-meter telescope with the new Dark Energy Camera built by
Migrating a (Large) Science Database to the Cloud
The Sloan Digital Sky Survey Migrating a (Large) Science Database to the Cloud Ani Thakar Alex Szalay Center for Astrophysical Sciences and Institute for Data Intensive Engineering and Science (IDIES)
Bringing Big Data to the Solar System. Paulo Penteado Northern Arizona University, Flagstaff (visiting David Trilling)
Bringing Big Data to the Solar System Paulo Penteado Northern Arizona University, Flagstaff (visiting David Trilling) pp.penteado@gmail.com http://www.ppenteado.net What is Big Data and why do we care?
The Challenge of Data in an Era of Petabyte Surveys Andrew Connolly University of Washington
The Challenge of Data in an Era of Petabyte Surveys Andrew Connolly University of Washington We acknowledge support from NSF IIS-0844580 and NASA 08-AISR08-0081 The science of big data sets Big Questions
Summary of Data Management Principles Dark Energy Survey V2.1, 7/16/15
Summary of Data Management Principles Dark Energy Survey V2.1, 7/16/15 This Summary of Data Management Principles (DMP) has been prepared at the request of the DOE Office of High Energy Physics, in support
Scientific Computing Meets Big Data Technology: An Astronomy Use Case
Scientific Computing Meets Big Data Technology: An Astronomy Use Case Zhao Zhang AMPLab and BIDS UC Berkeley zhaozhang@cs.berkeley.edu In collaboration with Kyle Barbary, Frank Nothaft, Evan Sparks, Oliver
A Preliminary Summary of The VLA Sky Survey
A Preliminary Summary of The VLA Sky Survey Eric J. Murphy and Stefi Baum (On behalf of the entire Science Survey Group) 1 Executive Summary After months of critical deliberation, the Survey Science Group
Astrophysics with Terabyte Datasets. Alex Szalay, JHU and Jim Gray, Microsoft Research
Astrophysics with Terabyte Datasets Alex Szalay, JHU and Jim Gray, Microsoft Research Living in an Exponential World Astronomers have a few hundred TB now 1 pixel (byte) / sq arc second ~ 4TB Multi-spectral,
J-PAS: low-resolution (R 50) spectroscopy over 8000 deg 2
J-PAS: low-resolution (R 50) spectroscopy over 8000 deg 2 C. López-Sanjuan J. Cenarro, L. A. Díaz-García, J. Varela, K. Viironen, & the J-PAS team Centro de Estudio de Física del Cosmos de Aragón 10th
Data Pipelines & Archives for Large Surveys. Peter Nugent (LBNL)
Data Pipelines & Archives for Large Surveys Peter Nugent (LBNL) Overview Major Issues facing any large-area survey/search: Computational power for search - data transfer, processing, storage, databases
Data analysis of L2-L3 products
Data analysis of L2-L3 products Emmanuel Gangler UBP Clermont-Ferrand (France) Emmanuel Gangler BIDS 14 1/13 Data management is a pillar of the project : L3 Telescope Caméra Data Management Outreach L1
Science and the Taiwan Airborne Telescope
Cosmic Variability Study in Taiwan Wen-Ping Chen Institute of Astronomy National Central University, Taiwan 2010 November 16@Jena/YETI Advantages in Taiwan: - Many high mountains - Western Pacific longitude
The LSST Data management and French computing activities. Dominique Fouchez on behalf of the IN2P3 Computing Team. LSST France April 8th,2015
The LSST Data management and French computing activities Dominique Fouchez on behalf of the IN2P3 Computing Team LSST France April 8th,2015 OSG All Hands SLAC April 7-9, 2014 1 The LSST Data management
LSST Data Management. Tim Axelrod Project Scientist - LSST Data Management. Thursday, 28 Oct 2010
LSST Data Management Tim Axelrod Project Scientist - LSST Data Management Thursday, 28 Oct 2010 Outline of the Presentation LSST telescope and survey Functions and architecture of the LSST data management
LSST Data Management System Applications Layer Simulated Data Needs Description: Simulation Needs for DC3
LSST Data Management System Applications Layer Simulated Data Needs Description: Simulation Needs for DC3 Draft 25 September 2008 A joint document from the LSST Data Management Team and Image Simulation
LSST Resources for Data Analysis
LSST Resources for the Community Lynne Jones University of Washington/LSST 1 Data Flow Nightly Operations : (at base facility) Each 15s exposure = 6.44 GB (raw) 2x15s = 1 visit 30 TB / night Generates
MANAGING AND MINING THE LSST DATA SETS
MANAGING AND MINING THE LSST DATA SETS Astronomy is undergoing an exciting revolution -- a revolution in the way we probe the universe and the way we answer fundamental questions. New technology enables
Virtual Observatory tools for the detection of T dwarfs. Enrique Solano, LAEFF / SVO Eduardo Martín, J.A. Caballero, IAC
Virtual Observatory tools for the detection of T dwarfs Enrique Solano, LAEFF / SVO Eduardo Martín, J.A. Caballero, IAC T dwarfs Low-mass (60-13 MJup), low-temperature (< 1300-1500 K), low-luminosity brown
Introduction to LSST Data Management. Jeffrey Kantor Data Management Project Manager
Introduction to LSST Data Management Jeffrey Kantor Data Management Project Manager LSST Data Management Principal Responsibilities Archive Raw Data: Receive the incoming stream of images that the Camera
Data-Intensive Science and Scientific Data Infrastructure
Data-Intensive Science and Scientific Data Infrastructure Russ Rew, UCAR Unidata ICTP Advanced School on High Performance and Grid Computing 13 April 2011 Overview Data-intensive science Publishing scientific
ASKAP Science Data Archive: Users and Requirements CSIRO ASTRONOMY AND SPACE SCIENCE (CASS)
ASKAP Science Data Archive: Users and Requirements CSIRO ASTRONOMY AND SPACE SCIENCE (CASS) Jessica Chapman, Data Workshop March 2013 ASKAP Science Data Archive Talk outline Data flow in brief Some radio
The Gaia Archive. Center Forum, Heidelberg, June 10-11, 2013. Stefan Jordan. The Gaia Archive, COSADIE Astronomical Data
The Gaia Archive Astronomisches Rechen-Institut am Zentrum für Astronomie der Universität Heidelberg http://www.stefan-jordan.de 1 2 Gaia 2013-2018 and beyond Progress with Gaia 3 HIPPARCOS Gaia accuracy
Libraries and Large Data
Libraries and Large Data Super Computing 2012 Elisabeth Long University of Chicago Library What is the Library s Interest in Big Data? Large Data and Libraries We ve Always Collected Data Intellectual
Data Mining Challenges and Opportunities in Astronomy
Data Mining Challenges and Opportunities in Astronomy S. G. Djorgovski (Caltech) With special thanks to R. Brunner, A. Szalay, A. Mahabal, et al. The Punchline: Astronomy has become an immensely datarich
The Solar Journey: Modeling Features of the Local Bubble and Galactic Environment of the Sun
The Solar Journey: Modeling Features of the Local Bubble and Galactic Environment of the Sun P.C. Frisch and A.J. Hanson Department of Astronomy and Astrophysics University of Chicago and Computer Science
Science Drivers for Big Data Joseph Lazio SKA Program Development Office & Jet Propulsion Laboratory, California Institute of Technology
Science Drivers for Big Data Joseph Lazio SKA Program Development Office & Jet Propulsion Laboratory, California Institute of Technology 2010 California Institute of Technology. Government sponsorship
Efficient data reduction and analysis of DECam images using multicore architecture Poor man s approach to Big data
Efficient data reduction and analysis of DECam images using multicore architecture Poor man s approach to Big data Instituto de Astrofísica Pontificia Universidad Católica de Chile Thomas Puzia, Maren
The Virtual Observatory: What is it and how can it help me? Enrique Solano LAEFF / INTA Spanish Virtual Observatory
The Virtual Observatory: What is it and how can it help me? Enrique Solano LAEFF / INTA Spanish Virtual Observatory Astronomy in the XXI century The Internet revolution (the dot com boom ) has transformed
LSST and the Cloud: Astro Collaboration in 2016 Tim Axelrod LSST Data Management Scientist
LSST and the Cloud: Astro Collaboration in 2016 Tim Axelrod LSST Data Management Scientist DERCAP Sydney, Australia, 2009 Overview of Presentation LSST - a large-scale Southern hemisphere optical survey
curation, analyses and interpretation of massive datasets opportunities are varied across disciplines
! Efficiency in scientific discovery through curation, analyses and interpretation of massive datasets! Uptake level and concentration on Big Data opportunities are varied across disciplines The nature
Visualization of Large Multi-Dimensional Datasets
***TITLE*** ASP Conference Series, Vol. ***VOLUME***, ***PUBLICATION YEAR*** ***EDITORS*** Visualization of Large Multi-Dimensional Datasets Joel Welling Department of Statistics, Carnegie Mellon University,
What is the Sloan Digital Sky Survey?
What is the Sloan Digital Sky Survey? Simply put, the Sloan Digital Sky Survey is the most ambitious astronomical survey ever undertaken. The survey will map one-quarter of the entire sky in detail, determining
Undergraduate Studies Department of Astronomy
WIYN 3.5-meter Telescope at Kitt Peak near Tucson, AZ Undergraduate Studies Department of Astronomy January 2014 Astronomy at Indiana University General Information The Astronomy Department at Indiana
Learning from Big Data in
Learning from Big Data in Astronomy an overview Kirk Borne George Mason University School of Physics, Astronomy, & Computational Sciences http://spacs.gmu.edu/ From traditional astronomy 2 to Big Data
and the VO-Science Francisco Jiménez Esteban Suffolk University
The Spanish-VO and the VO-Science Francisco Jiménez Esteban CAB / SVO (INTA-CSIC) Suffolk University The Spanish-VO (SVO) IVOA was created in June 2002 with the mission to facilitate the international
Towards the Detection and Characterization of Smaller Transiting Planets
Towards the Detection and Characterization of Smaller Transiting Planets David W. Latham 27 July 2007 Kepler MISSION CONCEPT Kepler Mission is optimized for finding habitable planets ( 10 to 0.5 M )
Big Data and Cloud Computing for GHRSST
Big Data and Cloud Computing for GHRSST Jean-Francois Piollé (jfpiolle@ifremer.fr) Frédéric Paul, Olivier Archer CERSAT / Institut Français de Recherche pour l Exploitation de la Mer Facing data deluge
An ArrayLibraryforMS SQL Server
An ArrayLibraryforMS SQL Server Scientific requirements and an implementation László Dobos 1,2 --dobos@complex.elte.hu Alex Szalay 2, José Blakeley 3, Tamás Budavári 2, István Csabai 1,2, Dragan Tomic
Virtual Observatories A New Era for Astronomy. Reinaldo R. de Carvalho DAS-INPE/MCT 2010
Virtual Observatories Virtual Observatories 1%%&'&$#-&6!&9:#,*3),!#,6!6#$C!&,&$D2 *:#%&+-3;& D&);&-$2!!"! "!" &,&$D2 %),-&,-!"#$%&'&#()*! $#%&!(!!! $ '!%&$ $! (% %)'6!6#$C!;#--&$G $! '!!! $#63#-3),G $!
Class 2 Solar System Characteristics Formation Exosolar Planets
Class 1 Introduction, Background History of Modern Astronomy The Night Sky, Eclipses and the Seasons Kepler's Laws Newtonian Gravity General Relativity Matter and Light Telescopes Class 2 Solar System
Galaxy Survey data analysis using SDSS-III as an example
Galaxy Survey data analysis using SDSS-III as an example Will Percival (University of Portsmouth) showing work by the BOSS galaxy clustering working group" Cosmology from Spectroscopic Galaxy Surveys"
Introduction to Scientific Data and Workflow Management
Introduction to Scientific Data and Workflow Management Michael Gertz gertz@ucdavis.edu Bertram Ludäscher ludaesch@ucdavis.edu Department of Computer Science University of California at Davis UC DAVIS
MAST: The Mikulski Archive for Space Telescopes
MAST: The Mikulski Archive for Space Telescopes Richard L. White Space Telescope Science Institute 2015 April 1, NRC Space Science Week/CBPSS A model for open access The NASA astrophysics data archives
Science Standard 4 Earth in Space Grade Level Expectations
Science Standard 4 Earth in Space Grade Level Expectations Science Standard 4 Earth in Space Our Solar System is a collection of gravitationally interacting bodies that include Earth and the Moon. Universal
On detecting variables using ROTSE-IIId archival data
Second Workshop on Robotic Autonomous Observatories ASI Conference Series, 2012, Vol. 7, pp 53 57 Edited by Sergey Guziy, Shashi B. Pandey, Juan C. Tello & Alberto J. Castro-Tirado On detecting variables
Top 10 Discoveries by ESO Telescopes
Top 10 Discoveries by ESO Telescopes European Southern Observatory reaching new heights in astronomy Exploring the Universe from the Atacama Desert, in Chile since 1964 ESO is the most productive astronomical
Australian Virtual Observatory
Australian Virtual Observatory International Astronomical Union GA 2003 Joint Discussion 08 17th-18th July 2003 Sydney David Barnes The University of Melbourne Our take on virtual observatories bring legacy
Creating A Galactic Plane Atlas With Amazon Web Services
Creating A Galactic Plane Atlas With Amazon Web Services G. Bruce Berriman 1*, Ewa Deelman 2, John Good 1, Gideon Juve 2, Jamie Kinney 3, Ann Merrihew 3, and Mats Rynge 2 1 Infrared Processing and Analysis
PISGAH ASTRONOMICAL RESEARCH INSTITUTE AND ASTRONOMICAL PHOTOGRAPHIC DATA ARCHIVE. www.pari.edu
PISGAH ASTRONOMICAL RESEARCH INSTITUTE AND ASTRONOMICAL PHOTOGRAPHIC DATA ARCHIVE www.pari.edu PARI 200ac (80ha) campus 6.2mi (10km) of roads 12 buildings 100,000 ft2 (9290 m2) Extensive Internet Infrastructure
The World-Wide Telescope, an Archetype for Online Science
The World-Wide Telescope, an Archetype for Online Science Jim Gray, Microsoft Research Alex Szalay, Johns Hopkins University June 2002 Technical Report MSR-TR-2002-75 Microsoft Research Microsoft Corporation
Lecture 5b: Data Mining. Peter Wheatley
Lecture 5b: Data Mining Peter Wheatley Data archives Most astronomical data now available via archives Raw data and high-level products usually available Data reduction software often specific to individual
How To Use The Wynn Odi
WIYN ODI: Observing Process, Data Analysis and Archiving Pierre Martin Yale Survey Workshop, October 2009 ODI: Scientific Challenges ODI is designed to take advantage of the best seeing conditions at WIYN.
The Sino-French Gamma-Ray Burst Mission SVOM (Space-based multi-band astronomical Variable Objects Monitor)
The Sino-French Gamma-Ray Burst Mission SVOM (Space-based multi-band astronomical Variable Objects Monitor) Didier BARRET on behalf of the SVOM collaboration didier.barret@cesr.fr Outline SVOM background
Adaptive Optics (AO) TMT Partner Institutions Collaborating Institution Acknowledgements
THIRTY METER TELESCOPE The past century of astronomy research has yielded remarkable insights into the nature and origin of the Universe. This scientific advancement has been fueled by progressively larger
PMCS - WBS with Definition
02C Data Management Construction This WBS element provides the complete LSST Data Management System (DMS). The DMS has these main responsibilities in the LSST system: Process the incoming stream of images
Commentary on Techniques for Massive- Data Machine Learning in Astronomy
1 of 24 Commentary on Techniques for Massive- Data Machine Learning in Astronomy Nick Ball Herzberg Institute of Astrophysics Victoria, Canada The Problem 2 of 24 Astronomy faces enormous datasets Their
High Energy Physics (HEP) Program Status Update to the AAAC Meeting. June 1, 2015
OFFICE OF SCIENCE High Energy Physics (HEP) Program Status Update to the AAAC Meeting June 1, 2015 Kathy Turner Program Manager, Cosmic Frontier Office of High Energy Physics Office of Science, U.S. Department
165 points. Name Date Period. Column B a. Cepheid variables b. luminosity c. RR Lyrae variables d. Sagittarius e. variable stars
Name Date Period 30 GALAXIES AND THE UNIVERSE SECTION 30.1 The Milky Way Galaxy In your textbook, read about discovering the Milky Way. (20 points) For each item in Column A, write the letter of the matching
LSST All Hands Meeting SLAC, December 4-8 2006 (MAP)
LSST All Hands Meeting SLAC, December 4-8 2006 (MAP) Monday, December 4 th Plenary Session Day One, Kavli Auditorium Project Status 1:00 Welcome; Project and MREFC Status D. Sweeney 1:40 Directors Report
Queued Service Observations with MegaPrime Phase 1 Proposal Submission for Semester 2010A
CFHT, Instruments, Queue Scheduling Queued Service Observations with MegaPrime Phase 1 Proposal Submission for Semester 2010A Updated 08/24/09 Table of Contents A - Introduction The QSO Project Document
Archival Science with the ESAC Science Archives and Virtual Observatory
Archival Science with the ESAC Science Archives and Virtual Observatory Deborah Baines Science Archives and VO Team Scientist European Space Agency (ESA) European Space Astronomy Centre (ESAC) Science
Challenges in e-science: Research in a Digital World
Challenges in e-science: Research in a Digital World Thom Dunning National Center for Supercomputing Applications National Center for Supercomputing Applications University of Illinois at Urbana-Champaign
Conquering the Astronomical Data Flood through Machine
Conquering the Astronomical Data Flood through Machine Learning and Citizen Science Kirk Borne George Mason University School of Physics, Astronomy, & Computational Sciences http://spacs.gmu.edu/ The Problem:
ASTRONOMICAL SURVEYS and BIG DATA
ASTRONOMICAL SURVEYS and BIG DATA International Symposium, dedicated to 50 th anniversary of Markarian Survey and 10 th anniversary of Armenian Virtual Observatory 5-9 October 2015, Byurakan, Armenia SCIENTIFIC
Using Photometric Data to Derive an HR Diagram for a Star Cluster
Using Photometric Data to Derive an HR Diagram for a Star Cluster In In this Activity, we will investigate: 1. How to use photometric data for an open cluster to derive an H-R Diagram for the stars and
NASA s Future Missions in X-ray Astronomy
NASA s Future Missions in X-ray Astronomy Nicholas E. White NASA s Goddard Space Flight Center Laboratory for High Energy Astrophysics Greenbelt, MD 20771 USA nwhite@lheapop.gsfc.nasa.gov Abstract The
Observatory Automation, Scheduling and Remote Sites
Observatory Automation, Scheduling and Remote Sites Presented by: Thomas C. Smith, Director, Dark Ridge Observatory And Tom Krajci, director, Astrokolkhoz Observatory Dallas Workshop on Space-age Alt-Az
Kathryn Forbes Neugent
Education Kathryn Forbes Neugent (805) 234-5390 KathrynNeugent@gmail.com Northern Arizona University, 2015 - present M.S. Degree Candidate in Physics with an emphasis on Planetary Science George Washington
Data Literacy For All: Astrophysics and Beyond (Astronomy is evidence-based forensic science, thus it is a data & information science)
Data Literacy For All: Astrophysics and Beyond (Astronomy is evidence-based forensic science, thus it is a data & information science) Kirk Borne George Mason University, Fairfax, VA www.kirkborne.net
Galaxy Morphological Classification
Galaxy Morphological Classification Jordan Duprey and James Kolano Abstract To solve the issue of galaxy morphological classification according to a classification scheme modelled off of the Hubble Sequence,
Reduced data products in the ESO Phase 3 archive (Status: 15 May 2015)
Reduced data products in the ESO Phase 3 archive (Status: 15 May 2015) The ESO Phase 3 archive provides access to reduced and calibrated data products. All those data are stored in standard formats. The
How To Teach Data Science
The Past, Present, and Future of Data Science Education Kirk Borne @KirkDBorne http://kirkborne.net George Mason University School of Physics, Astronomy, & Computational Sciences Outline Research and Application
Cloud Computing and Advanced Relationship Analytics
Cloud Computing and Advanced Relationship Analytics Using Objectivity/DB to Discover the Relationships in your Data By Brian Clark Vice President, Product Management Objectivity, Inc. 408 992 7136 brian.clark@objectivity.com
Canadian Astronomy Data Centre. Séverin Gaudet David Schade Canadian Astronomy Data Centre
Canadian Astronomy Data Centre Séverin Gaudet David Schade Canadian Astronomy Data Centre Data Activities in Astronomy Features of the astronomy data landscape Multi-wavelength datasets are increasingly
Observer Access to the Cherenkov Telescope Array
Observer Access to the Cherenkov Telescope Array IRAP, Toulouse, France E-mail: jknodlseder@irap.omp.eu V. Beckmann APC, Paris, France E-mail: beckmann@apc.in2p3.fr C. Boisson LUTh, Paris, France E-mail:
PART 1. Representations of atmospheric phenomena
PART 1 Representations of atmospheric phenomena Atmospheric data meet all of the criteria for big data : they are large (high volume), generated or captured frequently (high velocity), and represent a
SYLLABUS FORM WESTCHESTER COMMUNITY COLLEGE Valhalla, NY l0595. l. Course #:PHYSC 151 2. NAME OF ORIGINATOR /REVISOR: PAUL ROBINSON
SYLLABUS FORM WESTCHESTER COMMUNITY COLLEGE Valhalla, NY l0595 l. Course #:PHYSC 151 2. NAME OF ORIGINATOR /REVISOR: PAUL ROBINSON NAME OF COURSE: ASTRONOMY 3. CURRENT DATE: OCTOBER 26, 2011. Please indicate
Organization and Staffing
Large Synoptic Survey Telescope (LSST) Organization and Staffing Robert McKercher LPM-103 Latest Revision Date: September 3, 2013 Change Record Version Date Description Owner name 1 9/3/2013 Initial Version
Origins of the Cosmos Summer 2016. Pre-course assessment
Origins of the Cosmos Summer 2016 Pre-course assessment In order to grant two graduate credits for the workshop, we do require you to spend some hours before arriving at Penn State. We encourage all of
How To Process Data From A Casu.Com Computer System
CASU Processing: Overview and Updates for the VVV Survey Nicholas Walton Eduardo Gonalez-Solares, Simon Hodgkin, Mike Irwin (Institute of Astronomy) Pipeline Processing Summary Data organization (check
How To Write An Nccwsc/Csc Data Management Plan
Guidance and Requirements for NCCWSC/CSC Plans (Required for NCCWSC and CSC Proposals and Funded Projects) Prepared by the CSC/NCCWSC Working Group Emily Fort, Data and IT Manager for the National Climate
Big Data @ STScI. Enhancing STScI s Astronomical Data Science Capabilities over the Next Five Years
Big Data @ STScI Enhancing STScI s Astronomical Data Science Capabilities over the Next Five Years Science Definition Team Report March 15, 2016 1 Table of Contents 1 Executive Summary... 3 2 Charter of
Taming the Internet of Things: The Lord of the Things
Taming the Internet of Things: The Lord of the Things Kirk Borne @KirkDBorne School of Physics, Astronomy, & Computational Sciences College of Science, George Mason University, Fairfax, VA Taming the Internet
College of Science George Mason University Fairfax, VA 22030
College of Science George Mason University Fairfax, VA 22030 Dr. Sidney Wolff and the LSST Board of Directors LSST Corporation 933 N. Cherry Avenue Tucson, AZ 85721-0009 June 14, 2010 Dear Dr. Wolff and
Measuring the Doppler Shift of a Kepler Star with a Planet
Measuring the Doppler Shift of a Kepler Star with a Planet 1 Introduction The Doppler shift of a spectrum reveals the line of sight component of the velocity vector ( radial velocity ) of the object. Doppler
Explorations of the Outer Solar System. B. Scott Gaudi Harvard-Smithsonian Center for Astrophysics
Explorations of the Outer Solar System B. Scott Gaudi Harvard-Smithsonian Center for Astrophysics The Known Solar System How big is the solar system? a tidal R 0 M Sun M Galaxy 1/3 200,000AU How big is
Workload Characterization and Analysis of Storage and Bandwidth Needs of LEAD Workspace
Workload Characterization and Analysis of Storage and Bandwidth Needs of LEAD Workspace Beth Plale Indiana University plale@cs.indiana.edu LEAD TR 001, V3.0 V3.0 dated January 24, 2007 V2.0 dated August
astronomy 2008 1. A planet was viewed from Earth for several hours. The diagrams below represent the appearance of the planet at four different times.
1. A planet was viewed from Earth for several hours. The diagrams below represent the appearance of the planet at four different times. 5. If the distance between the Earth and the Sun were increased,
Real Time Big Data Processing
Real Time Big Data Processing Cloud Expo 2014 Ian Meyers Amazon Web Services Global Infrastructure Deployment & Administration App Services Analytics Compute Storage Database Networking AWS Global Infrastructure
HOW WILL ASTRONOMY ARCHIVES SURVIVE THE DATA TSUNAMI?
HOW WILL ASTRONOMY ARCHIVES SURVIVE THE DATA TSUNAMI? Astronomers are collecting more data than ever. What practices can keep them ahead of the flood? G. Bruce Berriman, NASA Exoplanet Science Institute,
Visualization and Astronomy
Visualization and Astronomy Prof.dr. Jos Roerdink Institute for Mathematics and Computing Science University of Groningen URL: www.cs.rug.nl/svcg/ Scientific Visualization & Computer Graphics Group Institute
The Legacy Value of Large Public Surveys: the SDSS Archive. Alexander Szalay The Johns Hopkins University
The Legacy Value of Large Public Surveys: the SDSS Archive Alexander Szalay The Johns Hopkins University Sloan Digital Sky Survey The Cosmic Genome Project Started in 1992, finished in 2008 Data is public
The Search for Dark Matter, Einstein s Cosmology and MOND. David B. Cline
The Search for Dark Matter, Einstein s Cosmology and MOND David B. Cline Astrophysics Division, Department of Physics & Astronomy University of California, Los Angeles, CA 90095 USA dcline@physics.ucla.edu
Building your Big Data Architecture on Amazon Web Services
Building your Big Data Architecture on Amazon Web Services Abhishek Sinha @abysinha sinhaar@amazon.com AWS Services Deployment & Administration Application Services Compute Storage Database Networking
Data Analytics at NERSC. Joaquin Correa JoaquinCorrea@lbl.gov NERSC Data and Analytics Services
Data Analytics at NERSC Joaquin Correa JoaquinCorrea@lbl.gov NERSC Data and Analytics Services NERSC User Meeting August, 2015 Data analytics at NERSC Science Applications Climate, Cosmology, Kbase, Materials,
The Large Synoptic Survey Telescope: Status Update
The Large Synoptic Survey Telescope: Status Update Steven M. Kahn LSST Director Mid-Decadal Review Committee December 13, 2015 LSST in a Nutshell The LSST is an integrated survey system designed to conduct
Bringing the Night Sky Closer: Discoveries in the Data Deluge
EARTH AND ENVIRONMENT Bringing the Night Sky Closer: Discoveries in the Data Deluge Alyssa A. Goodman Harvard University Curtis G. Wong Microsoft Research Th r o u g h o u t h i s t o r y, a s t r o n
DCA QUESTIONNAIRE V0.1-1 INTRODUCTION AND IDENTIFICATION OF THE DATA CENTRE
DCA QUESTIONNAIRE V0.1-1 INTRODUCTION AND IDENTIFICATION OF THE DATA CENTRE Introduction - The EuroVO-DCA Census questionnaire The Euro-VO Data Centre Alliance (http://www.euro-vo.org/pub/dca/overview.html)