Sustainable Digital Information. Corporate Memory

Transcription

1 Sustainable Digital Information Corporate Memory 10 September 2009

2 Family video/pictures next generation 2

3 LongRec DATA = DIGITAL ACCESS THROUGH AEONS 3+ year project, research and case studies - DNV R&I lead, 10 partners - Start October 2006, end Overall budget 27,6 MNOK, Norwegian Research Council grant 9.2 MNOK - 3 PhD theses in work 3

4 LongRec DATA = Digital Access Through Aeons Rea d Trust Find Understand 4

5 Project partners Nasjonalbiblioteket Norsk Regnesentral Utenriksdep. Riksarkivet InterPARES 3: Brønnøysundregistrene ICRI (Interdisciplinary Centre for Law and ICT), Katholieke Universiteit Leuven 5

6 Work Packages READ TRUST FIND UNDERSTAND COMPLIANCE 2 topics across packages 6

7 Information / Data Volume Trends READ Volume Explosion 2 stacks of books from Earth to Pluto Earliest written record Gutenberg London Library ( books) Internet Explosion 12 stacks of books from Earth to Sun 0.01% is stored on paper 3000 BC

8 Trends READ Storage shortage Peta Bytes info storage : Data created will be three times amount of available storage. Lots of data will be for immediate consumption only 8

9 Different medium over years If it s not digital, it s not accessible 9

10 National Library of Norway Current state of digitalization: 5% Total volume when today s collections are digitalized ( 2018) Estimated total volume: 37 Petabyte Estimated number of files: Percentage of completed digitalization 20 % 23,2% Hardware Support: 3 (4) years only!! 15 % 10 % 5 % 0 % 6.0% 2.2% 0.1% text images sound video 10

11 Migration Calculator Migration Calculator 6 parameters: - Size of digital objects - read/write bandwidth - read/write access time - file processing speed - network transfer bandwidth - the number of replicas. Two basic models: - Basic Migration - Migration with processing Two extensions: - Replication and verification 11

12 Time (s) Findings migration pilot doing verification by another CPU, the migration time will not be increased ,0 8000,0 6000,0 4000,0 2000,0 0, # folder MB-V-2CPU MB MB-V-1CPU 12

13 time (s) Findings migration pilot When doing migration with processing, the migration time will be decreased by running multiple processes the number of processes 13

14 Further work within READ (PhD) 1. Very large amounts of data in preservation systems (Calculator) 2. Lack of a comprehensive migration framework (Framework) 3. What metadata should be preserved for migration? (Metadata) 14 14

15 Information / Data Volume Trends within Trust More and more trusted Documents are getting digitized 3000 BC Time 15

16 Trusted Information life cycle 1) TRUSTED RECORD MANAGEMEMT The authenticity, integrity and completeness of records 2) TRUSTED DIGITAL REPOSITORY The sustained integrity of records after acquisition Reliable storage Continued usability and readability Controlled access 3) TRUSTED TRANSFER The sustained authenticity of records after migration, conversion and reformatting (verification challenges) Digital Repository 16

17 Record Management and trusted transfer 17

18 Digital Repository 18

19 Further work (PhD) General Trust model Guidelines to secure trust within a repository Checklist to secure trust within a repository Critical metadata to secure trust within a repository 19

20 Information / Data Volume Trends FIND Volume Explosion 90% of all data is unstructured (pictures, video, s, blogs, ) - no data model, no meta data 70% of all data belongs to individuals and are de-centralized stored - Video, Photos, web pages, ect Massive growth in multimedia information, less in textual information 2 stacks of books from Earth to Pluto Earliest written record Gutenberg London Library ( books) Internet Explosion 12 stacks of books from Earth to Sun 0.01% is stored on paper 3000 BC

21 Overview of the ECDL paper Problem: Due to decentralized nature and the lack of standards for date/time, it is difficult to find accurate and trustworthy timestamp for web documents. For a given document with uncertain timestamp, can the contents be used to determine the timestamp with a sufficiently high confidence? Let s me see This document is probably originated in 850 A.C. with 95% confidence. I found a bible-like document. But I have no idea when it was created? You should ask Guru! 21

22 Using Temporal Language Models for Document Dating Previous Work Temporal Language Models A non-time stamped document Partition Word 1999 tsunami tsunami Thailand 1999 Japan 1999 tidal wave 2004 tsunami 2004 Thailand Similarity Scores Score(1999) = earthquake Score(2004) = = 2 Most likely timestamp is Nattiya Kanhabua and Kjetil Nørvåg (Norwegian University of Science and Technology) 1/2

23 Find - Time traveler 23

24 FIND - Research questions The main research question: How to improve the quality of search in a document archive using temporal information? Q1. How to handle large number of documents retrieved? Q2. How to search with awareness of language changes? Q3. How to rank search results wrt. temporal information? 24

25 he impact of communication on language change High: many external contacts heterogeneous participants frequent exchange English Intensity of communication Low : Icelandic few external contacts homogeneous participants infrequent exchange Low: slow adoption of foreign words conservative and predictable orthographic system Speed of language change 25 High: high turn-over of words flexible, volatile orthographic system

26 The impact of globalization on changing information needs High: many international subcontractors complex, heterogenous customer base internationalized technological development DNV 2009 Low : Degree of globalized business limited dependency on business partners no transnational, few local business partners homogenous and stable customer needs Low: slow development of new knowledge stable, predictable information needs A local carpenter producing furniture in Copenhagen, 1930 Degree of changes in information 26 High: high change rate of knowledge volatile information needs

27 Timeline information changes Time zoom scroll-bar Primary Records Secondary data Decision support 27

28 How can semantics add value to information management? Maintenance of Metadata over time Maintenance of Master data over time Verification that your information structure is good over time Automatic maintenance like with Retention Etc. 28

29 Further work UNDERSTAND Information Governance regime Master data Search vs semantic technology supplement or overlap Assessing an organization's maturity in preserving semantic value of information assets More on Evolution 29

30 Increased laws and regulations Compliance Trends EUs 8 directive /EuroSox CDBA Enron MiFiD Solvency II Data Protection Act Norsk SOX Basel II HIPPA arkivlov GLBA FISMA FOIA

31 Input in 3 areas Toolbox - Maturity model related to compliance issues Retention - automatically Compliance tool the dream 31

32 Compliance Toolbox: Maturity related to Compliance 1. Corporate governance 2. Information governance 3. Information maturity 32

33 Toolbox 1: Managing Risks of Corporate Governance Corporate Governance Compliance Risks 33

34 Toolbox 2: 10 steps Information Governance 1. Decide Vision 2. Ensure corporate Management involvement 3. Create policies 4. Design guidelines and best practices 5. Define responsibilities 6. Assign IM Professionals 7. Identify the important information objects 8. Create an information architecture 9. Define required IM services incl SLA 10. Link the Metrics to companies goals and then monitor it VISION HOW WHAT 34

35 Toolbox 3 Information Maturity CMMI EIM Maturity Model (GARTNER) IBM Data Governance Council Maturity Model Aiim 35

36 Compliance retention/automatically Frame conditions: 1. Retention policy 2. Virtual organization 3. Decide which level you need retention - Information Category - Information type etc. RM Application: 1. An event trigger an retention application - Platform is disused - New or changed law - Updated retention policy etc 2. Which of the retention rules in the rule repository to apply - Dependent on: - Type of event - Type of object - Type of information object 3. The RM Application generates a set of information objects that need to be: - reviewed for retention workflow/information owner - updated retention date - or automatically deleted 36

37 Workflow Engine Basic Scenario 1: Managing retention for a given event DNVs Retention Manager 1. Submit trigging event 2. Make query & select documents 3. Select rules & calculate actions 4. Review, monitor, decide 5. Take action Retention Ontology Thing Engine for automatic deletion or update of metadata Interface for submitting events ProductionObject Platform Employee EventType Rule Condition DocType RuleContent NormalFlow OfTime EventType RIM DisuseOf Platform User Document Owner Process Contract TerminationOfE mployment Type Interface for reviewing retention candidates Interface for manual deletion or update of metadata Meridio Rule Repository Disuse platform Oseberg Employee Smith leaves the company Identify all documents related to Oseberg Identify all documents related to Smith Calculate all rules related to disuse of a platform AND the types of documents related to Oseberg Calculate all rules related to termination of employment and types of documents related to Smith 37 Find RIM for Oseberg Find RIM HR Review Oseberg documents Delete Osebergreports, but keep contract Set deletion date for Smithdocuments to Jan 1, 2019

38 Compliance Tool? Manual handling Performed by a tool External Law changes (region) Industry law changes Specific data Eks personal data DNV Broker ( lobing ) Tool? Relevant? JA Which internal Policy are involved? Nei Continue to Keep me updated Internal Review With feedback Decision by responsible Incorporate into the business Local storage? Retention time Information category/-type Information owner 38

39 Findings across work packages Cost Factors Commercialization 39

40 General cost factors in Digital preservation - Calculator OAIS Monitoring costs Security control costs Staff costs Maintenanc e costs Infrastructu re costs Standards costs Technology costs General cost factors Training costs Quality assurance costs Organisatio n costs Retention costs Selection costs Access costs 40

41 Development Path DNV 3rd party information management services DNV Trusted Online Service Information risk Management Digital Safe Information Maturity Assessment Service

42 Digital Safe Main functionality Long-term storage - Migration, conversion etc Compliance - Retention do automatically etc Digital Safe Trust - Fingerprints 3rd party role 42

43 Sleep like a baby - Assists you to secure sustainable records 43

44 Safeguarding life, property and the environment 44