Ge Peng. In CollaboraBon with Ed Kearns and Nancy Ritchey NCDC/NOAA

Transcription

1 Stewardship Maturity Matrix - A Unified Framework for Assessing Data Quality and Usability PracBces Applied to Individual Digital Environmental Data Products Ge Peng NOAA s CooperaBve InsBtute for Climate and Satellite NC State University (CICS- NC) and NOAA s NaBonal ClimaBc Data Center (NCDC) Jeff PrivePe NCDC/NOAA In CollaboraBon with Ed Kearns and Nancy Ritchey NCDC/NOAA 10 July 2014 ESIP Summer MeeBng

2 Why Do We Need a Stewardship Maturity Matrix? Ø The value and quality of a data set depends in part on the stewardship pracbces applied a^er its producbon. HypotheBc quesbons to a data center: 1. Congress: Are your datasets compliant with the U.S. Data Quality Act? If not, then what? 2. Business: Is your product credible? Readily accessible with common data format? Sustainable? 3. Modelers: Is the quality of a rou7nely updated product being assessed? Problem: Most of data centers currently cannot readily convey - or even assess the level of stewardship prac7ces for its stakeholders or customers. No community scorecard exists. Ø This is a vulnerability and an opportunity! SoluBon: Define a Stewardship Maturity Matrix to assess stewardship prac7ces applied to individual data products

3 What Is ScienBfic Data Stewardship? AcBviBes to preserve or improve the informabon content, accessibility, and usability of environmental data and metadata (NRC, 2007) AcBviBes to ensure or improve the quality and usability of environmental data and metadata Ensuring Data Are Always Meaningful ScienBfic Data Stewards Data Quality Screening/ Assurance/ Control/ EvaluaBon/ Assessment/ Monitoring Environmental data are recorded observabons and measurements of the physical, chemical, biological, geological, or geophysical properbes or condibons of the oceans, atmosphere, space environment, sun, and solid earth, as well as correlabve data and related documentabon or metadata (NOAA, 2008). Common data format SpaBal/ temporal characterisbcs Uncertainty esbmates

4 How Do We IdenBfy Key Components and Define Levels of Stewardship Maturity Matrix Procedures Policies Processes Tasks Standards U.S. Laws Agencies recommendabon Data Archive Data/Metadata Management Data PreservaBon Data Governance Evaluate product Verify data integrity Experts recommendabon Data Quality Management Data Provenance Monitor data quality Non- FuncBonal Requirements FuncBonal Key Areas Key Matrix Components Community PracBces Relevant Measurable Progressive Independent (?)

5 Maturity Levels Follows CMMI level Structure Level 1 Ad Hoc Not Managed Level 2 Minimal Limit Managed Level 4 Advanced/Well Managed Community Best Prac7ces Level 3 Intermediate/Managed Community Good Prac7ces Level 5 Op7mal/Well Managed Measured, Controlled, Audit Reference Maturity Level Structure Capability Maturity Model Integra7on (CMMI) Levels of Maturity of Digital repository Climate Data Record Maturity Matrix (CDRMM)

6 Goals? General Simple Concise Assess & Convey & Path Forward Not to Reinvent Wheels Leveraging Community accepted good and best pracbces and standards Subject MaPer Experts The ESIP Community

7 Key Components of Stewardship Maturity Matrix ScienBfic Data Stewardship Data Quality Screen/Assurance/Monitor/Control Best PracBce Data Quality Assessment/ Validation Data Usability Data Integrity Production Sustainability Best PracBce Discovery and Access Preservation Transparency/Traceability

8 Who Could Use the Matrix? Data providers and scienbfic stewards Ø to evaluate and improve the quality and usability of their products against community best prac7ces Modelers, decision- support system users, and scienbsts Ø Ø to improve their products and uncertainty es7mates to make investment and use decision Data managers/stewards of data centers and repositories Ø Ø Ø to validate their compliance or lack of to community accepted stewardship prac7ce or standards to assess the current state to create a roadmap forward to improve or enhance its stewardship maturity of prac7ces applied to a certain product or all its holdings General data users Ø to make an educated choice on selec7ng or u7lizing a dataset

9 Your Comment & SuggesBon Are Appreciated! Should community best pracbces in each key component be given intermediate (level 3) or advanced (level 4) maturity level, i.e., benchmark or upper- bound? What is a Best PracBce? v Wikipedia: A best prac7ce is a method or technique that has consistently shown results superior to those achieved with other means, and that is used as a benchmark. v Cambridge DicBonary: a working method or set of working methods that is officially accepted as being the best to use in a par7cular business or industry, usually described formally and in detail. Dilbert is a Best PracBce

10 Your Comment & SuggesBon Are Appreciated! Should community best pracbces in each key component be given intermediate (level 3) or advanced (level 4) maturity level, i.e., benchmark or upper- bound? Need for mulbple data quality key components? DQ Assurance/Screen procedures are to ensure that the products meet the requirements building it right. DQ Assessment/ValidaBon is to ensure that the products are scien7fically sound building the right thing. DQ Monitor/Control procedures are to detect anomalous in the data input and/or outputs

11 Your Comment & SuggesBon Are Appreciated! Should community best pracbces in each key component be given intermediate (level 3) or advanced (level 4) maturity level, i.e., benchmark or upper- bound? Need for mulbple data quality key components? How can we Integrate our effort with the ESIP community? hpp:// mm betav10 hpp:// term- scienbficdatastewardship

12 Talking Points for Discussion The Data Quality Management Cycle Data Quality Key FuncBon Areas Plan: Requirements/SpecificaBons Do: Measure/Assess/Describe Check: Screen/Assurance/Monitor/Control Act: Resolve/Report/NoBfy Roles and responsibilibes? Consistent guidelines of good & best pracbces in each funcbon area? An efficient way to report, resolve, and nobfy Bmely and effecbvely?

13 Backup Slides

14

15

16

17 The Reference Model for an Open Archival InformaBon System (OAIS) The OAIS Reference Model apempts to comprehensively idenbfy the responsibilibes and components of an archival system, including: Ø the roles of people and insbtubons that interact in an archive: Producer, Management, and Consumer Ø the digital objects: informabon package Ø the major funcbons: six higher- level funcbons Ingest, Data Management, Archival Storage, Access, PreservaBon Management, and AdministraBon, represenbng thirty- three lower- level funcbons h\p:// eng/founda7on/oais/index.html Source: OAIS Overview 22Arp2008.ppt hpp://wiki.esipfed.org/index.php/ NOAA_Sessions_on_NIDIS,_CLASS,_the_Data_Centers,_and_much_much_more! Source: