PMCS - WBS with Definition

Transcription

1 02C Data Management Construction This WBS element provides the complete LSST Data Management System (DMS). The DMS has these main responsibilities in the LSST system: Process the incoming stream of images generated by the Camera Subsystem during observing to generate and archive the LSST nightly data products. Provide real-time information on data quality to the Observatory Control System (OCS) during observing, Reprocess archived data products as required to incorporate pipeline improvements and correct errors, and to generate longer-term LSST data products. Provide a VO-compliant interface that makes publicly available all generated data products. 02C.00 DM Level 2 Milestones This WBS Element defines the Level 2 milestones for this project. These milestones represent the top-level subsystem deliverables and interface milestones with other subystems. 02C.01 System Management This WBS element provides all activities related to the management and administration of the Data Management WBS elements. This includes all activities and support to maintain a core team responsible for the execution of all Data Management Tasks. Effort in this task insures compliance with Project level controls, documentation, and reporting. This also includes overseeing the management of DMS physical assets (hardware, software, facilities, documents) and support for travel and communications during the Construction Phase, as well as preparing for the same activities during the Operations Phase. This includes: Project management of resources, schedules, tasks, and deliverables Performance assurance of the DM project, including Science Data Quality Assessment Configuration control of all DMS elements and asset management for all DMS physical assets. 02C Project Management Page 1 of 45

2 02C This WBS element includes Project Management staff and supporting tools for the project. The primary deliverables are the Data Management sections of overall LSST PMCS-based project plans, schedules, budgets, and reports. Communications and collaboration tools for team management and coordination are also within this WBS element. Performance Assurance This WBS element includes Quality Planning, Quality Engineering, Quality Assessment, and Quality Control for the Data Management Subsystem. The primary deliverables include the Data Management sections of overall LSST system quality plans. These sections will include plans for Quality Metrics Collection and Analysis, Defect Tracking, Version/Release Control, and Change Management of the DMS. (Note, DMS Test Plans are under separate WBS elements.) This WBS element also includes Science Data Quality Assessment pipelines and toolkits for use by data analysts and scientists to assess the quality of the DMS-generated data. 02C Science Data Quality Assessment Pipeline documentation that implement the SDQA Pipeline capabilities. - SDQA Pipeline per Exposure - Retrieve SDQA Flags and Status for all Exposures - Compute Selected Summary SDQA Information - SDQA per Amplifier 02C Science Data Quality Analyst Toolkit documentation that implement the SDQA Toolkit capabilities. Page 2 of 45

3 This WBS element implements the following UML use cases: - Correlate SDQA Metric with Other Data - Correlate SDQA Metrics - Display SDQA Metrics - Analyze SDQA Metrics 02C.02 System Engineering This WBS element includes all system engineering activities for the LSST Data Management subsystem. This WBS elemen includes all subsystem level modeling and simulation development as well as subsystem and system level technical reviews of deliverables. The design optimization and as-built documentation of the subsystem requirements and design are covered under this WBS element. The principal deliverables of this WBS element are optimizations of and updates to the D&D phase products below: Subsystem trade-off studies, subsystems issues and risks, and an approach for their resolution/mitigation. Functional Requirements Specification: Define and maintain Data Management Subsystem (DMS) functional requirements. Operational Concept Document: Define and maintain Data Management Subsystem (DMS) operational concept. System Modeling Language (SysML) Model: Define and maintain Data Management Subsystem requirements, traceability, logical model and structure model, as part of the LSST system engineering model Unified Modeling Language (UML) Model: Define and maintain a UML model of the Data Management Subsystem (DMS) hardware and software requirements and design. 02C.03 Alert and Calibration Productions This WBS element is a summary element and includes software programs, configuration files, databases and data files, uni tests, component integration tests, and documentation implementing the Alert and Calibration Products Production Pipelines. This Alert Production element is deployed at both the Base Center and the Archive Center, while the Calibration Products Production element is deployed at the Archive Center only. Page 3 of 45

4 02C Image Processing Pipelines documentation that implement Image Processing capabilities. Instrument Signature Removal Pipeline removes the instrument signature and standardizes the input image and metadata into LSST format. Each slice operates on a single amplifier image. - Identify Calibration Products - Remove Cross-Talk - Mask Saturation - Remove Overscans - Remove Bias - Renormalize Variance - Apply Darks - Correct for Nonlinearity - Defringe - Flatten - SDQA for ISR CCD Assembly Pipeline assembles amplifiers from ISR into CCD Exposures. Page 4 of 45

5 - Assemble Ccd - Identify Defects - SDQA for CCD Assembly CR Split Handling Pipeline handles cosmic ray removal using the two split exposures per visit, algorithmically rejecting cosmics on each one and then doing a difference between them to find more cosmics. Background estimation is in this pipeline because it is required before Cosmic Ray Rejection. - Background Estimation - Find and Mask CRs - Simple Image Differencing - Mask and Sum Image Characterization Pipeline handles image characterization, including determination of PSF and WCS. It also handles initial CCD-exposure-level photometric calibration and background subtraction. It combines all of this information into a single Exposure. The CalibSources used for the determinations are saved. Each slice operates on a single CCD image and produces a science CCD exposure that is used throughout the rest of the Data Release Production. - Bright Star Detection - Bright Star Measurement - PSF Determination - Aperture Correction - WCS Determination - WCS Verification - CCD Photometric Calibration Page 5 of 45

6 - Exposure Generation 02C Association Pipelines documentation that implement Source Association capabilities. Night Source Association (Pipeline) - Correlates sources with known objects -- either fixed or moving objects predicted to be within the FOV -- to determine whether new sources have been detected. - Preprocess AstroObject Catalog - Calculate Object Zone Indices for Visit - Match DiaSources to AstoObjects - Update AstroObject and DiaSource Catalogs DiaSource Association Pipeline - handles association of DiaSources (sources derived from difference imaging) that have not already been associated with Moving Objects by DayMOPS. It associates them with prototype Objects from Deep Detection and Object Characterization. Any unassociated DiaSources are taken to be transients and new Objects are created for them. - Associate unassociated Moving Objects to Sources NOTE: This Pipeline is delivered as part of this WBS but is used in the Data Release Production. Page 6 of 45

7 Object/Merge Association Pipeline - Associate transient objects created by DiaSourceAssoc with Sources (via a spatial match), and Sources with moving objects. Merge together transient, moving, and deep-detection objects with object measurements and astrometry to produce final objects for ingestion into the data release Object catalogs. - Associate Moving Objects to Sources - Associate Sources with Transient Objects - Merge Source / Object Pairs NOTE: This Pipeline is delivered as part of this WBS but is used in the Data Release Production. Source Association - Correlates difference sources with known objects -- either fixed or moving objects predicted to be within the Field of View (FOV) -- to determine whether new objects have been detected, in which case a new object is created, stored and made available for association in future visits. The associations can be used to avoid issuing an alert for a difference source matching a known moving, variable or transient object. - Source Clustering - Source Cluster Attributes NOTE: This Pipeline is delivered as part of this WBS but is used in the Data Release Production. 02C Alert Generation Pipeline documentation that implement Alert Generation capabilities. Page 7 of 45

8 The Alert Generation Pipeline concludes the nightly processing of a visit and is performed on the full object entry. Each object s characteristics, including its prior history is checked against a set of rules that determines whether or not an alert will be issued, and if so, the type of alert. If an alert is to be generated, the object information, together with the subimages containing the sources are packaged into an alert structure, which is sent to the Archive center for dissemination to the community using the VOEvent mechanism. - Acquire Alerts - Package Alerts - Distribute Alerts 02C Calibration Products Production documentation implementing Calibration Products Production capabilities. Calibration Products Production generates the calibration products used to remove the instrument signature from exposures - Pre-sequence Exposure Catalog - Produce Crosstalk Matrix Pipeline - Produce Illumination Correction Exposure Pipeline - Produce Master Bias Exposure Pipeline - Produce Master Dark Current Exposure Pipeline - Produce Master Flat Exposure Pipeline - Produce Master Fringe Pipeline - Produce Pupil Ghost Exposure Pipeline Page 8 of 45

9 - Produce Atmospheric Models from Auxiliary Telescope Spectra 02C Application Framework AP & CPP documentation that implement the Application Framework for the Alert and Calibration Products productions. This WBS element implements the following UML Design elements: - Astro Object - Calibration - Catalogs - Detector - Exposure - Image - Mask Planes - Math - SDQA - Sky - Source The WBS element also supports the integration of third party libraries, e.g. boost, pynum, etc. 02C Moving Object Pipelines (Day and Page 9 of 45

10 documentation that implement Moving Object capabilities. Day MOPS Pipeline generates the predicted ephemerides for the current exposure's sky position. - Setup DayMOPS - Intra Night Linking - Inter Night Linking - Orbit Determination - Orbit Management NOTE: This Pipeline is delivered as part of this WBS but is used in the Data Release Production. Night MOPS Pipeline predicts locations of known objects expected to appear in difference images. - Compute Course Ephemerides for Night - Create Catalog of Known Solar System Objects within FOV Mask Moving Objects Pipeline handles the masking of moving objects from science CCD Exposures that are to be used for deep detection and object characterization. This element is parallelized so that each process operates on a single science CCD Exposure as well as (some type of) footprints for all moving objects that appear within that Exposure. Page 10 of 45

11 - Mask Footprints NOTE: This Pipeline is delivered as part of this WBS but is used in the Data Release Production. 02C Difference Imaging Pipeline documentation that implement Difference Imaging capabilities. The Difference Imaging Pipeline handles computation of Difference Exposures by subtracting Template Exposures from Exposures. It also detects and measures DiaSources on the resulting Difference Images. - Identify SkyTile - Extract Sky Map Image - Image Differencing - Difference Detection - Difference Measurement The Difference Forced Photometry Pipeline handles forced photometry on difference images to form ForcedDiaSources. Each slice operates on a single CCD difference image as well as the MovingObjects and transient Objects that occur within it. The WBS element implements the following UML activities: - Night MOPS Pipeline - Measure Sources NOTE: This Pipeline is delivered as part of this WBS but is used in the Data Release Production. Page 11 of 45

12 02C.04 Data Release Production This WBS element is a summary element that includes software programs, configuration files, unit tests, component integration tests, and documentation implementing the Data Release Production capabilities. This WBS element is deployed only at the Archive Center. - Single Frame Source Measurement Pipeline - PSF Match Pipeline - TemplateGen Pipeline - Deep Coadd Generation Pipeline - Deep Detection Pipeline - Difference Forced Photometry Pipeline - Astrometric Model Generation Pipeline - Galaxy Model Generation Pipeline - Photometric Calibration Pipeline Note that this WBS element is combined with the following UML activities in the Alert Production WBS element, and deployed at the Archive Center to provide the complete Data Release Production capability: - ISR Pipeline - CCD Assembly Pipeline - CRSplit Handling Pipeline - Image Characterization Pipeline - Difference Imaging Pipeline - DayMOPS Pipeline Page 12 of 45

13 - Mask Moving Objects Pipeline - Source Association Pipeline - DiaSource Association Pipeline - Object Merge/Association Pipeline 02C Photometric Calibration Pipeline documentation that implement the Photometric Calibration capabilities. Photometric Calibration Pipeline handles the global photometric calibration of all data products. It is expected to extract data from the database. - Determine system response for Exposure - Identify calibration stars in Exposure - Calculate color correction terms for calibration stars - Set up least squares system for self-calibration - Solve least squares system for self-calibration - Apply Calibration corrections to Sources 02C Astrometric Calibration Pipeline Page 13 of 45

14 documentation that implement Astrometric Calibration capabilities. In the Astrometric Model Generation Pipeline, each slice operates on all Sources within a given sky tile across all epochs. Before calibration can be performed, each Source must be associated with a deep Detection. - Build Transforms for Astrometric Calibration - Calibrate Object AstrometryApply Transform for Exposure-CCD-Group - Fit for Astrometric Transforms - Evaluate Convergence 02C Single Frame Measurement Pipeline documentation that implement Single Frame Measurement capabilities. The Single Frame Source Measurement Pipeline measures Sources on a single frame. It is parallelized such that each parallel process operates on a single science CCD Exposure. - Detect Source - Measure Sources - Compute Source Sky Coordinates Page 14 of 45

15 02C Image Coaddition Pipeline documentation that implement Image Coaddition capabilities. Deep Coadd Generation Pipeline handles the co-addition of masked Exposures to form deep templates. There will be 6 templates, one per filter. It is parallelized such that each parallel process operates on a single science CCD Exposure, adding it to the appropriate filter's deep co-add. - Generate ChiSquared Coadd - Convert Exposure to SkyMapImage PSFMatch Pipeline ssf-matches one exposure to another. Exposures must have the same WCS to get reasonable results. - Locate Relevant Images - Match PSF - Warp Exposure to SkyMap Image - Build Exposure Stack - Reject Outliers Template Generation Pipeline builds coadded SkyMapImage to be used to construct Template Exposure. - Generate Sky Tile Page 15 of 45

16 - Coadd without Outlier Rejection 02C Deep Detection Pipeline documentation that implement Deep Detection capabilities. Deep Detection Pipeline handles the generation of the deep panchromatic chi-square co-add and detection and measurement on that co-add. It is parallelized such that each parallel process operates on a single sky tile. - Acquire Sky Tile Metadata - Generate ChiSquared Coadd - Detect Sources - Measure Sources 02C Object Characterization Pipeline documentation that implement the Object Characterization capabilities. The Galaxy Model Generation Pipeline handles the projection of a detection from a coadd to the all the exposures that overlap it, using the list of projected detections to input a stack of postage stamp images, and fitting of a galaxy model. This includes forced photometry of the resulting shape on each image of the stack to form ForcedSources. Page 16 of 45

17 - Transform Detection from Coadd to Exposure - Postage Stamp Generation - Forward Modelling - Forced Photometry 02C Application Framework DRP 02C Application Framework DRP documentation that implement measurement capabilities. - Bright Star Detection - Bright Star Measurement - SF Source Detection - SF Source Measurement - Difference Measurement 02C.05 Science User Interface and Analysis Tools This WBS element is a summary element that contains the software, tools, and user interfaces specifying the Science User Interface and Analysis Tools which will support the following features: - Provide coherent and intuitive mechanisms for scientists to access the data products (images, catalogs, alerts), - Provide mechanisms for scientist to analyze and visualize the data sets and identify data subsets of interest - Provide mechanisms for scientists to download, export or import data subsets of interest to alternative storage locations or Page 17 of 45

18 into analysis environments - Provide a mechanism for scientists to store, manage and manipulate large data subsets of interest using Data Center resources - Facilitate federation of the data products with external data; - Facilitate federation of the data products with external data; - Serve documentation about the DMS and data products - Provide automated and human assistance in working with the DMS and data products; - Enable scientists to develop analytical codes by reusing existing DMS codes and integrating externally developed codes; - Execute the analytical codes on local or external platforms; - Capture and analyze the results of those codes. NOTE: The UML for this WBS element is in development at this time. 02C Basic Archive Access Tools documentation that implement the Archive access capabilities. Archive catalog data is selected based on user specifications of catalog properties. This WBS element implements the following UML use cases: - Review Scatter plots, Histograms, density plots - Review Statistical Summary - Review Time Series - Review Alerts - Review Orbit and Track properties of moving objects Page 18 of 45

19 - View Selected Catalog Data - Revise Selection Criteria - Retrieve Selected Catalog Data (to local storage, workspace or other location) - Access image data selected according to user specifications - View preview images - View Images - Review Data for Specific Region of Focal Plane - Review Metric Image - View Raw Exposure Image - View Processed Image - Overlay Catalog Data on Image - Associate Calibration Data with Image - Revise Selection Criteria - Retrieve Selected Image Data Archive Access provides means to identify and access a variety of pre-defined, summary data queries across the entire archive of data. This WBS element implements the following UML use cases: - Extract Time Sequence of Images - Extract Time Series for Objects - Execute Cone Search - Find All RR Lyrae within RA/DEC Region - Find All Galaxies with Given Properties - Find All g-band Stacked Images that Cover RA/DEC Region Page 19 of 45

20 - Query Catalog - Query Image Catalog 02C Data Analysis and Visualization Tools documentation that implement the Data Analysis and Visualization capabilities. Data visualization capabilities include the ability to view large-scale, multi-dimensional data in a variety of plots and views. Data Analysis permits querying, formatting, analyzing, and plotting data in a variety of 2D and 3D representations, with analytical overlays and statistical data. This WBS element implements the following UML use cases: - Review Scatter plots, Histograms, density plots - View Data From Auxiliary Telescope - Review Data from Facilities Database - Review data federated with external catalogs - Access catalog data selected according to user specifications on any catalog properties - Review 2D plot representation of Catalog Data - Review 3D plot representations of Catalog Data - Review higher-level graphic representations of Catalog Data - Review time-series movies of image data - Crop, re-project, overlay, create false color and mosaic image data - Analyze image data (surface brightness, aperture photometry, blink, subtract, ratio) Analytical capabilities include pre-defined query and analysis capabilities. Page 20 of 45

21 - Analyze Color-Color Diagram - Derive Galaxy Luminosity Function - Derive Stellar Luminosity Function - Discover groups and clusters of galaxies - Find all Lensed Quasar Candidates - Generate photometric redshift for a galaxy - Create Stellar Color-Color Diagram - Create Color-Magnitude Diagram - Create Corrected Color-Magnitude Diagram - Create Cleaned Color-Magnitude Diagram - Classify / Analyze Eclipsing Binaries - Search for Microlensed SN - Search for SN Light Echoes - Search for Planetary Transits - Calculate the Two Point Correlation Function of Galaxy Group. 02C Alert Subscription Toolkit documentation that implement the Alert Subscription capabilities. This WBS element implements the following UML use cases: - Subscribe to Alert Category - Create an Alert Filter - Retrieve Alerts Page 21 of 45

22 - Create Alert Category - Record Alerts - Deliver Alerts - Process Subscription Requests 02C User Assistance/Help Desk documentation that implement the User Assistance capabilities. This WBS element implements the following UML use cases: - Maintain Web page or equivalent to host User Support material - Post science user documentation such as User Guides, Data manuals etc. - Post science user information and news - Host information about obtaining access to Data Center Resources - Maintain a helpdesk with query tracking capability 02C User Workspace Toolkit documentation that implement the User Workspace capability. This WBS element implements the following UML use cases: - Upload data - Move data Page 22 of 45

23 - Organize data - Access data - Delete data 02C.06 Science Data Archive and Data Access Services This WBS element is a summary element that includes software programs, database tables, configuration files, unit tests, component integration tests, and documentation that implement Science Data Archive and Data Access Service capabilities, including: - Catalogs Alerts and Metadata - Image and File Archive - Database and File Access Services - Query Services - Catalog Construction - Virtual Observatory Interfaces These services provide the ability to ingest, index, federate, query, and administer DMS data products on distributed, heterogeneous storage system and data server architectures. All services will be implemented to provide reasonable fault-tolerance and autonomous recovery in the event software and hardware failures. 02C Science Data Archive This WBS element is a summary element that includes software programs, database tables, configuration files, unit tests, component integration tests, and documentation that implement Science Data Archive capabilities, including: - Catalogs Alerts and Metadata - Image and File Archive Page 23 of 45

24 - Image and File Archive These services provide the capability to store and retrieve LSST Data Products in terms of their application level "astronomical" semantics, mapping those semantics to physical, persistent versions of those data products in databases and files. 02C Catalogs Alerts and MetaData This WBS element includes software programs, database tables, configuration files, unit tests, component integration tests, and documentation that implement Catalogs Alerts and MetaData capabilities. Catalogs Alerts and Metadata implements all structured data that is stored in the DMS database system, including the schema and initial data for all database-based Data Products. This WBS element implements the following UML design elements: - Source Catalog - Object Catalog - Deep Object Catalog - Orbit Catalog - Meta-Data and Provenance This WBS element implements the following UML use cases: - Define Schema - Construct Per-Job Catalog - Construct Global Catalog - Construct Data Release Catalog - Run Data Ingest Converter - Run Data Ingest Page 24 of 45

25 - Persistence - Define Persistence Policies - Obtain and Configure Persistence Object - Obtain and Configure Storage Object for Persistence - Persist Data from Pipeline - Format and Send Persistable Object to Storage - Obtains and Configure Storage Object for Retrieval - Retrieve Persistable Object 02C Image and File Archive This WBS element includes software programs, database tables, configuration files, unit tests, component integration tests, and documentation that implement the Image and File Archive. The Image and File Archive includes all Images processed by the DMS. This WBS element implements the following UML design elements: - Raw Science Images - Calibrated Science Images - Subtracted Science Images Note: Calibrated and Subtracted Science Images will not be stored, but the Image Archive and Image Processing Pipeline will recreate them on demand from the input images and provenance. This WBS element implements the following UML use cases: - Access File Systems - Provenance - Read Provenance Page 25 of 45

26 26-Aug-11 09:14 - Re-Create Calibrated Science Image - Re-Create Subtracted Science Image 02C Data Access Services This WBS element is a summary element that includes software programs, database tables, configuration files, unit tests, component integration tests, and documentation that implement Data Access Service capabilities, including: - Database and File Access Services - Query Services - Catalog Construction - Virtual Observatory Interfaces These services provide the ability to ingest, index, federate, query, and administer DMS data products on distributed, heterogeneous storage system and data server architectures. All services will be implemented to provide reasonable fault-tolerance and autonomous recovery in the event software and hardware failures. 02C Database and File Access Services This WBS element includes software programs, database tables, configuration files, unit tests, component integration tests, and documentation that implement Database Services for developing, maintaining, configuring, and administering DMS database. The element includes off-the-shelf DBMS capabilities, as well as custom capabilities for partitioning and clustering, fault tolerance, and other advanced features. This WBS element implements the following UML use cases: - Access Database Systems Page 26 of 45

27 26-Aug-11 09:14 02C Query Services This WBS element includes software programs, database tables, configuration files, unit tests, component integration tests, and documentation that implement Query Services. The element includes off-the-shelf query capabilities, as well as custom Parallel queries, shared table scans, and other advanced features. This WBS element implements the following UML use cases: - Formulate and Submit Queries - Log Queries - Estimate Query Cost - Catch Hostile Queries - Process Query - Process Query Results 02C Catalog Construction Toolkit This WBS element includes software programs, database tables, configuration files, unit tests, component integration tests, and documentation that implement the Catalog Construction capabilities. The Catalog Construction Toolkit provides capabilities to create user-defined catalogs, extracted from L1/L2 data. This WBS element implements the following UML use cases: - Create Data Type - Create Metadata - Create Catalog Page 27 of 45

28 26-Aug-11 09:14 02C VO Interfaces This WBS element includes software programs, database tables, configuration files, unit tests, component integration tests, and documentation that implement the VO interace capabilities. VO Interfaces provide the capability to access all LSST data products via VO-standard formats and protocols. This WBS element implements the following UML use cases: - VORegistry access - VOTable import/export - VOEvent export - SIAP access - Access Time Series data 02C.07 Processing Middleware and Site Infrastructure This WBS element is a summary element that includes all infrastructure acquisition, configuration, test, and where required packaging and shipping for the Archive and Base Sites. It also includes the acquisition, configuration, deployment, and administration of all development tools and environments, including both development and integration clusters. Finally, it includes software programs, database tables, configuration files, unit tests, component integration tests, and documentation This WBS element implements the following UML design elements: - DMS Control and Management - Communications - Pipeline Construction Page 28 of 45

29 26-Aug-11 09:14 - Pipeline Execution File System Services - DMS Security These services provide the ability to execute DMS processing on distributed, heterogeneous computing and communications platforms, in parallel clustered and grid configurations. All services will be implemented to provide reasonable fault-tolerance and autonomous recovery in the event software and hardware failures 02C Processing Middleware This WBS element is a summary element that includes software programs, database tables, configuration files, unit tests, component integration tests, and documentation that implements the Processing Middleware capabilities. This WBS element implements the following UML design elements: - Control and Management Services (including communications) - Pipeline Construction Toolkit - Pipeline Execution Services - Security and Access Control Services - User Interface/Visualization Services - System Administration and Operations Services - File System Services - Environment and Tools (for software development and integration) These services provide the ability to execute DMS processing on distributed, heterogeneous computing and communications platforms, in parallel clustered and grid configurations. All services will be implemented to provide reasonable fault-tolerance and autonomous recovery in the event software and hardware failures 02C Control and Management Services Page 29 of 45

30 26-Aug-11 09:14 This WBS element includes software programs, database tables, configuration files, unit tests, and documentation that implement event notification services, orchestration services, and the Data Management Control System (DMCS). Event services are used to communicate among components of the DM System, including between pipelines in a production. Orchestration services are responsible for deploying pipelines and Policies onto nodes, ensuring that their input data is staged appropriately, distributing dataset identifiers to be processed, recording provenance, and starting pipeline execution. The DMCS at each site is responsible for initializing and running diagnostics on all equipment, including computing nodes, disk storage, tape storage, and networking. It establishes and maintains connectivity with the other sites including the Headquarters Site. It monitors the operation of all hardware and software. At the Base Center, the DMCS is responsible for interfacing with the Observatory Control System (OCS). At the Archive Center, the DMCS performs resource management for the compute cluster and initializes replication services. At each Data Access Center, the DMCS performs resource management for the Level 3 Data Products compute cluster and initializes replication services. This WBS element implements the following UML use cases: Event Logging - Configure Event Logging - Register Event Receiver - Register Event Transmitter - Publish Event Using Event System - Retrieve Event Using Event System Event Monitoring - Initialize Event Monitor - Process Incoming Event - Publish Event Using Event System Orchestration - Deploy and Execute Pipeline - Manage Pipeline Execution Page 30 of 45

31 26-Aug-11 09:14 - Terminate Pipeline Execution Archive Center DMCS - Execute Alert Production Reprocessing at Archive - Execute Data Release Processing - Process Calibrations - Process Observations - Register in VO - Replicate to Data Access Centers - Retire Previous Data Release - Initialize Archive Center - Monitor Archive Center - Reconfigure Hardware - Shut Down Base Center DMCS - Change to Calibration Mode - Change to Day Mode - Change to Maintenance Mode - Change to Observing Mode - Initialize Base Center - Monitor Base Center - Reconfigure Hardware - Shut Down Data Access Center DMCS - Initialize Data Access Center System - Monitor Data Access Center Page 31 of 45

32 26-Aug-11 09:14 - Reconfigure Hardware - Shut Down 02C Pipeline Construction Toolkit This WBS element includes software programs, database tables, configuration files, unit tests, and documentation that implement a framework library for constructing multi-processing, distributed pipelines. This WBS element implements the following UML use cases: - Create Component - Create Component Algorithm - Define Component Interface - Componentize and Add to Component Library - Create Pipeline - Define Parallelization and Communication - Define Processing Steps 02C Pipeline Execution Services This WBS element includes software programs, database tables, configuration files, unit tests, and documentation that implement Pipeline Execution capabilities. This WBS element implements the following UML design elements: - Pipeline Controllers/Managers - Logging and Trace Exceptions - Inter-process messaging Page 32 of 45

33 26-Aug-11 09:14 This WBS element implements the following UML use cases: - Pipeline Control - Define Execution Environment - Configure Access to Data - Configure Pipeline - Checkpoint / Restart Pipeline - Stop Pipeline Execution - Cleanup after Execution - Record Pipeline Execution Status - Monitor Pipeline Execution - Pipeline Execution 02C Security and Access Control Services This WBS element includes software programs, database tables, configuration files, unit tests, and documentation. This WBS element implements the following UML design elements: - Authorization/Authentication - Access Control (Enforcement) This WBS element implements the following UML use cases: - Administer Certificates - Administer Groups and Users Page 33 of 45

34 26-Aug-11 09:14 - Configure Security Profiles and Policies - Enforce Security Policies 02C User Interface/Visualization Services documentation. This WBS element implements the following UML design elements: - Create and configure displays and plots of static data and metadata in graphical and tabular forms - Create and configure static and continuously updating displays of real-time data streams in graphical and tabular forms This WBS element implements the following UML use cases: - Dynamic Display of Data and Metsdata - Static Display of Data and Metadata 02C System Administration and Operations Services documentation that implement the System Administration and Operations capabilities. System Administration and Operations Services is composed of low-level tools that monitor systems and provide informatio on current status. This WBS element implements the following UML design elements: - Provide an interface to formatted data describing the current operational state of the LSST DMS, including equipment health and workload, pipeline processing status, data query and ingest workload, data transfer workload, and DMS Page 34 of 45

35 26-Aug-11 09:14 health and workload, pipeline processing status, data query and ingest workload, data transfer workload, and DMS performance snapshots and trends. - Provide an interface to formatted data describing the internal state of LSST data archives and catalogs, including data integrity, usage patterns, and logical and physical schemas - Provide an interface to formatted data describing configuration of the LSST DMS subsystem. 02C File System Services This WBS element includes software programs, database tables, configuration files, unit tests, component integration tests, and documentation that implement distributed File System Services. File System Services provides file-based access over local storage, local area networks, and wide area networks. This WBS element also includes off-the-shelf distributed file system capabilities, e.g. irods, Lustre, gpfs. This WBS element implements the following UML use cases: - Access file-based data (local and remote) - Define file access for groups - Define file access for users 02C Environment and Tools This WBS element is a summary element that includes tools and data needed in the development, integration and test, and support of the DM systems and software. 02C Software Development Tools Notebook Topic This WBS element includes off-the-shelf tools for DMS software development, including: Page 35 of 45

36 26-Aug-11 09:14 - Compilers - Interpreters - Linkers - Make Utilities - Editors - Debuggers - Test Generators - CASE Tools - Version Control System - Document Generators - Profilers 02C Camera/DAQ Test Bed Notebook Topic This WBS element contains a test environment that includes the Camera/DAQ interfaces and a raw science image simulated data stream generator. It is provided by the Camera team under a Camera WBS element and configured for DM use under this element. 02C Telescope/OCS Interfaces Test Bed Notebook Topic This WBS element contains a test environment that includes the integrated control system interfaces, and simulated TCS/OCS command and status data streams generator. It is provided by the Telescope and Site team under a T&S WBS element and configured for DM use under this element. Page 36 of 45

37 26-Aug-11 09:14 02C Pre-Cursor and Simulated Data Server Notebook Topic This WBS element contains a data repository containing multiple terabytes of pre-cursor (e.g. SM/Essence, MACHO, DLS, DEC, Pan-STARRS, CFHTLS, TALCS, Subaru) and LSST simulated data for algorithm and scaling tests. 02C Virtual Observatory Repository Server Notebook Topic This WBS element includes a complete implementation of all basic VO Repository services and protocols to be used as a test environment for VO interfaces. 02C Test Data Notebook Topic This WBS element includes simulated or subset LSST data products of all types for use in DMS testing. It is derived from the Pre-Cursor and Simulated Data Server and is oriented at providing data to pipelines and productions during testing. 02C Test Harnesses and Scripts Notebook Topic This WBS element includes subsystem and system integration and test harnesses and scripts that exercise the DMS in all end-to-end usage scenarios. - The Subsystem Integration Test Harnesses and Scripts simulate the behavior and data stream of each DMS component. They implement the Test Cases created in the Data Management Subsystem Integration and Test element. - The System Integration Test Harnesses and Scripts simulate the other LSST subsystem and end user behavior and data streams (including driving test harnesses in the System Integration and Test element. 02C Operational Control and Monitoring System Page 37 of 45

38 26-Aug-11 09:14 Notebook Topic This WBS element contains end user tools that allow management and monitoring of all Data Management System operational status, including: system health and welfare, pipeline/production execution status, data transfer status, operational modes and schedules. This WBS is also known as the Data Management Systems Operations Center. This WBS element contains hardware and end-user display tools to monitor the LSST DMS subsystem. 02C Science Pipeline Toolkit Notebook Topic documentation that implement the Science Pipeline Toolkit capabilities. Science Pipeline Toolkit provides capabilities that permit users of the DMS to perform processing of LSST data with LSST open software and user-supplied codes, for Level 3 Data Product production. This WBS element implements the following UML use cases: - Upload user codes - Incorporate user code into pipeline - Select data to be processed - Select data to be stored - Configure pipeline execution - Execute pipeline - Monitor pipeline execution 02C Site Infrastructure This WBS element is a summary element that includes all infrastructure acquisition, configuration, test, and where required packaging and shipping for the Archive and Base Sites. It also includes the acquisition, configuration, deployment, and administration of all development tools and environments, including both development and integration clusters. Page 38 of 45

39 26-Aug-11 09:14 02C Archive Center Infrastructure This WBS element includes the acquisition, configuration, test, and operation of infrastructure for the Archive Center. 02C Data Acquisition Equipment Notebook Topic This WBS element includes a copy of the Mountain/Base DAQ equipment used for interface testing. It is provided by the Camera Subsystem. 02C Computing Servers Notebook Topic This WBS element includes the compute servers (including cluster infrastructure and fileservers for pipeline disk cache) tha host the pipelines that produce Data Release data products as well as all servers need to operate the Archive Center. This will be sized for re-processing the entire survey each year. 02C Workstations Notebook Topic This WBS element includes computers used primarily by on-site personnel to operate and maintain the archive and pipeline 02C High Speed Pipeline Disk Cache Notebook Topic This WBS element includes disks, controllers, and storage network connected directly to pipeline servers for processing data. These resources are sized and selected for the highest required data input/output rates. Page 39 of 45

40 26-Aug-11 09:14 02C Near-Line Archive Storage Notebook Topic This WBS element includes high-speed disk, controllers, storage network, and fileservers used to support Archive Center operations and serve as a secondary disk cache for the pipeline servers. 02C Deep Storage Notebook Topic This WBS element includes slower-speed disk, tape libraries, controllers, storage network and storage servers that provide long-term storage for all LSST archived data. 02C Archive Center Networks Notebook Topic This WBS element includes network switches, routers, cables, interfaces connecting servers, workstations, storage and communications within the Archive Center. 02C US Data Access Center Infrastructure 02C Data Access Servers Notebook Topic This WBS element includes the acquisition, configuration, test, and operation of infrastructure of the US DAC. This WBS element includes servers that carry out the operation and management of the Data Access Center including inter-site data replication and query services, but excluding fileservers/storage management. Page 40 of 45

41 26-Aug-11 09:14 02C High Speed Primary Storage Notebook Topic This WBS element includes high-speed disk, controllers, storage network, and fileservers used to support Data Access Center operations. The design is scaled-down version of the Archive Center Near-line Archive Storage. 02C Base Center Infrastructure This WBS element includes the acquisition, configuration, test, packing, and shipping of infrastructure for the Base Center. 02C Chilean DAC Infrastructure This WBS element includes the acquisition, configuration, test, packing, and shipping of infrastructure for the Chilean DAC. 02C Development and Integration Infrastructure This WBS element is a summary element and includes the acquisition, configuration, test, and operation of the Development and Integration clusters at the Archive Site. 02C Development Cluster and Workstations Notebook Topic Page 41 of 45

42 26-Aug-11 09:14 This WBS element includes a computing system that is architecturally the same as the Base Center infrastructure, but on a smaller scale in terms capacity/performance. It is used as a DMS software development and maintenance environment during Construction and Operations. It includes at least one of each type of DMS workstation, server, and communications end equipment. 02C Integration Cluster Notebook Topic This WBS element includes a computing system that is architecturally the same as the Archive Facility infrastructure, but on a smaller scale in terms of capacity/performance. It is used as a DMS integration environment during Construction and a support environment during Operations. This environment is the primary environment for duplicating/debugging DMS operational problems. During major DMS upgrades, the environment provides for a testing period in which the newly upgraded system operates in parallel with the current operational system to ensure smooth cut-over. 02C.08 Communications and Base Site This WBS element is a summary element that includes the infrastructure for the Base Center and the national and international networks connecting the Mountain Summit, Base, Archive, and Headquarters sites. 02C Base Center This WBS element is a summary element that includes the infrastructure that receives data from the camera DAQ subsystem and the Observatory Control System and performs all of the time critical computing and communications. The Base Center is hosted in the Base Facility, which is provided by the Telescope and Site WBS. 02C Data Acquisition Equipment This WBS element includes hardware that connects to the camera outputs to capture the raw data. Includes cables and data acquisition cards and dedicated computers. 02C Computing Servers Page 42 of 45

43 26-Aug-11 09:14 This WBS element includes servers located at the Base Center for Alert Production processing. 02C Workstations This WBS element includes workstations for operations use to control and monitor the systems at the Base Center. 02C Secondary Storage Farm This WBS element includes storage used for Base Center operations. 02C Summit/Base Primary Link This WBS element includes the fiber optic cables, end equipment, and associated tools for the Mountain Summit to Base Facility primary link. 02C Summit/Base Backup Link This WBS element includes the fiber optic cables, end equipment, and associated tools for the Mountain Summit to Base Center primary link. 02C Base Center Local Area Network Page 43 of 45

44 26-Aug-11 09:14 02C This WBS element includes the switches, routers, cables, and network management resources for the local area network at the Base Center. Chilean Data Access Center This WBS element is a summary element that includes the infrastructure for the Chilean Data Access Center. The Chilean Data Access Center provides a community access point for all LSST data, including computing and storage resources dedicated to end user processing to create Level 3 data products enabling the LSST science. 02C Data Access Servers This WBS element includes servers that carry out the operation and management of the Data Access Center including inter-site data replication and query services, but excluding fileservers/storage management. 02C High Speed Primary Storage This WBS element includes high-speed disk, controllers, storage network, and fileservers used to support Data Access Center operations. The design is scaled-down version of the Archive Center Near-line Archive Storage. 02C Long-Haul Networks This WBS element is a summary element that includes the infrastructure for the national and international networks connecting the Mountain Summit, Base, and Archive sites. 02C Chile National WAN Page 44 of 45

45 26-Aug-11 09:14 This WBS element includes the infrastructure for the national networks within Chile used in connecting the Base and Archive sites. 02C International Chile - US WAN This WBS element includes the long-haul/international Wide Area Networks connecting the Base Center with the Archive Center. These are likely to include AMPath and WHREN-LILA networks currently linking South America and the United States. 02C US National WAN This WBS element includes the infrastructure for the national networks within the US used in connecting the Base, Archive, Headquarters and Science Center sites. These are likely to include TeraGrid, internet2, National Lambda Rail, and ESNet. 02C.09 Data Management System Integration and Test This WBS element includes DM subsystem integration and test activities, including: - Creation of Integration Test Plans and Test Cases - Execution of Test Plans and Test Cases - Recording and Analysis of Integration Test Results - Debug and correcting defects found in testing - Data Product documentation and aids (print, web, media) Page 45 of 45