Daten Verstehen - Archivieren

Size: px

Start display at page:

Download "Daten Verstehen - Archivieren"

Kelly Bates
7 years ago
Views:

1 Daten Verstehen - Archivieren Andreas Rauber Department of Software Technology and Interactive Systems Vienna University of Technology rauber@ifs.tuwien.ac.at

2 Outline Archiving Digital Preservation What is it? Why do we need it? Preserving Data How can we do it? - Physical: Storage - Logical: Migration, Emulation, Virtualization, - Semantic: Annotation From Data to Processes How to handle these? - Process Preservation - Software ESCROW - Data Citation

- Physical: Storage - Logical: Migration, Emulation, Virtualization, -

3 Why do we need Digital Preservation?

4 Why do we need Digital Preservation?

5 Why do we need Digital Preservation? A simple example Assume you are doing a statistical analysis on the population of Austria... Austria 1999 CIA map Wikimedia, 1999 Homann Heirs Map of Austria and Bohemia Wikimedia 1747 Mitchell Map of Austria, Hungary and Cary Map of Austria Wikimedia 1801 Transylvania Wikimedia 1850

6 Why do we need Digital Preservation? 3 levels of threat: physical, logical, semantic Essential for all digital objects - Office documents, accounting, photos, videos, s, - Scientific datasets, sensor data, databases, - Construction data, process logs, sales data, - Applications, simulations, business processes, All application domains - Cultural heritage data - egovernment, Public administration - Science / Research - Construction industry - Health, pharmaceutical industry - Aviation, control systems - Private data -

emails, - Scientific datasets, sensor data, databases, - Construction data, process logs, sales data, - Applications, simulations,

7 Outline Archiving Digital Preservation What is it? Why do we need it? Preserving Data How can we do it? - Physical: Storage - Logical: Migration, Emulation, Virtualization, - Semantic: Annotation From Data to Processes How to handle these? - Process Preservation - Software ESCROW - Data Citation

8 Bit-level preservation Maintain bit-sequence Redundant storage: - Lockss: lots of copies keeps stuff safe - Cloud Distributed storage physically separated Different technologies / platforms / production batches Controlled storage conditions Regular maintenance: tape rewinding, disc spinning, Maintain devices for accessing storage! Disaster recovery planning, monitoring and error detection Trade-off capacity, energy, effort, ease-of-use 8

Controlled storage conditions Regular maintenance: tape rewinding, disc spinning, Maintain devices for

9 Bit-level preservation Questions / discussion: How long do tapes / CDs / DVDs / HDDs / SSD last? What would be the most durable storage technologies? Is a DVD that lasts for 200 years a solution? What are the costs of bit-level preservation? What are the logistic challenges? Distribution and Trust? Are we allowed to store redundantly? - Copyright - Copy protection Am I allowed to store data in the cloud? 9

What are the costs of bit-level preservation? What are the logistic challenges? Distribution and Trust?

10 Logical Preservation Strategies Technology Museum Migration Emulation Standardization Virtualization Information Extraction Data Recovery, Data Archaeology All have advantages and disadvantages No single optimal solution All have to be re-applied, no one-stop solution 10

Data Recovery, Data Archaeology All have advantages and

11 Logical Preservation Summary Many preservation actions, subtle variations Changing object, environment Loss upon migration / emulation How to decide what to preserve? Significant Properties! How to detect/document what you lost? Range of strategies available, none is perfect Combination of strategies Which solution is best? Preservation Planning! No solution forever -> DP is a process! 11

Significant Properties! How to detect/document what you lost?

12 Semantic preservation Threats at semantic level - meaning of terms change: city names, - measurement scales, sensor sensitivity, change - interpretation of facts change: alcohol levels, - data context-dependent: re-calibration/sensor drift Rather long-term, but subtle to notice! Consider context of objects - purpose, setting, limitations, cultural context, related objects, 12

levels, - data context-dependent: re-calibration/sensor drift Rather long-term, but subtle to

13 Semantic preservation Approaches / solutions: - Semantic enrichment - Metadata - Migration at semantic level - Documentation of context - Tracing of metadata - Document intended meaning / interpretation How to decide which approaches are needed? - Re-use scenarios What is the context of data and data use? Is having the data sufficient? 13

Document intended meaning / interpretation How to decide which approaches are needed?

14 Outline Archiving Digital Preservation What is it? Why do we need it? Preserving Data How can we do it? - Physical: Storage - Logical: Migration, Emulation, Virtualization, - Semantic: Annotation From Data to Processes How to handle these? - Process Preservation - Software ESCROW - Data Citation

15 From Data to Processes Assume we know how to preserve data - Is this sufficient? Preserving data: Data Management Plans - describing data and context: provenance, authenticity, representation information,... - range of (ambiguous) definitions of context - But: mostly not actionable, not enforcable,... - BUT: data are (just) results of processes! Processes may be needed to - verify data - understand provenance - re-use process on new data - integrate data over time Process curation instead of data curation! But: HOW?

.. - range of (ambiguous) definitions of context - But: mostly not actionable, not enforcable,.

16 Process Preservation Capturing Context Need to capture and describe processes - SW, HW, organizational aspects, legal aspects, Analyze process, capture elements Represented in ontology Reasoning across process elements to find out what needs to be done to address risks Apply preservation actions

process, capture elements Represented in ontology Reasoning across

17 Process Preservation Technical Framework for ESCROW What to do with proprietary software? ESCROW: legal means to regulate deposit of source code Issue: no support to verify what has been deposited Technical framework automatically - Verifying completeness - Checking quality indicators - Identifying potential fraud Approach - Automatic build, similarity measures for build results - Software quality measures - Hash signatures of libraries, NISTNational Software Reference Library (NSRL)

framework automatically - Verifying completeness - Checking quality indicators - Identifying potential fraud Approach -

18 Process Preservation Data Citation Huge databases that keep growing and changing Goal: repeat a computation that was performed earlier with identical results - preserve the algorithms, process - But: data has changed same algorithm, different data - Keeping copy of data not possible for each computation New approach to identifying subsets of data - Time-stamped and versioned database - Time-stamped query + unique-sort + hash + PID Simple way to transparently and verifyably identify subsets of data to ensure repeatability of analyses,

data not possible for each computation New approach to identifying subsets of data - Time-stamped and versioned database -

19 Summary 3 levels of threat to (digital) information - Physical / Logical / Semantic Preservation actions to ensure continuous access and use - Storage / Migration-Emulation-Virtualization- / Documentation - Preservation Planning and Evaluation/Verification Preserving data + processes - Process context - ESCROW - Data Citation It s a challenging problem but essential to address Safeguarding investments made into data Address as part of operations rather than ex-post

Evaluation/Verification Preserving data + processes - Process context - ESCROW - Data Citation It s a challenging

20 Thanks Research projects - SCAPE, EU FP7 IP - TIMBUS, EU FP7 IP - APARSEN, EU FP7 NoE - 4C, EU FP7 CA Collaboration to Clarify the Cost of Curation

21 Thank you!

Data Management Plans - How to Treat Digital Sources

1 Data Management Plans - How to Treat Digital Sources The imminent future for repositories and their management Paolo Budroni Library and Archive Services, University of Vienna Tomasz Miksa Secure Business