System Requirements for Archiving Electronic Records PROS 99/007 Specification 1. Public Record Office Victoria

Transcription

1 System Requirements for Archiving Electronic Records PROS 99/007 Specification 1 Public Record Office Victoria Version 1.0 April 2000

2 PROS 99/007 Specification 1: System Requirements for Archiving Electronic Records 1 Table of Contents 1.0 Introduction Record Capture System Requirements Evidentiary Integrity Unique Files/Record Identifiers Document Capture Interface Metadata Capture Interface Metadata Control Record Linking Automatic Record Creation Record Creation Functionality Archive System Requirements Evidentiary Integrity Modifying information associated with Records Verification Access Control Disposal Destruction of Records/Files Auditing Reliability and Redundancy Refresh Record Transfer Unique Media Identifiers Record Discovery System Requirements Supporting documentation (Finding Aids, Thesauri) Access Control Access Logging User Interface Searching Verification Rendering Generic VERS Record Structure Records Encodings Files Unique Record Identifiers... 14

3 PROS 99/007 Specification 1: System Requirements for Archiving Electronic Records 2 This document specifies the system requirements for archiving electronic records in accordance with standards issued by Public Record Office Victoria. It should be used in conjunction with PROS 99/007 Standard for the Management of Electronic Records, PROS 99/007 Specification 2: VERS Metadata Scheme, and PROS 99/007 Specification 3: VERS Standard Electronic Record Format. 1.0 Introduction This specification provides an analysis of the functional requirements of a system which will capture, archive, manage, and provide access to electronic records. These functional requirements provide a summary of the issues to consider, and criteria to be defined, when specifying or evaluating a system that must produce or manage long term electronic records. While some of these requirements relate specifically to the VERS standard long term record format, most have general application to any archival system. The requirements have been organized into: Records Capture System Requirements Archive System Requirements Record Discovery System Requirements All three systems together form a Record Keeping System. There are many different ways in which these requirements could be fulfilled, depending on the applications from which records are captured and computer technology chosen. It is important to note that this document deals only with those functions needed to support archiving, and does not list functions supporting other uses of records. It is also important to note that the use of the three categories ( capture, archive, and discovery ) is for explanatory purposes only and that existing or future software and hardware products may support any or all of these functions under a variety of different names. 2.0 Record Capture System Requirements The capture function is achieved in a VERS compliant system when documents and metadata are converted to electronic records in the VERS standard format. 1 A Record Capture system may be integrated into an application, or it may be a separate component. Some implementations may store the record internally within the application and only convert the record to the VERS standard format when it is necessary to export the record. 2.1 Evidentiary Integrity To ensure evidentiary integrity, the Record Capture system must ensure that it is possible to demonstrate who created the record, when it was created, and that it has not been altered since creation. In the Record Capture System, evidentiary integrity is achieved 1 See PROS 99/007 Specification 3 VERS Standard Electronic Record Format. Available on the PROV website

4 PROS 99/007 Specification 1: System Requirements for Archiving Electronic Records 3 by capturing information which provides this integrity. Ensuring integrity of a record for its entire life within that system is the responsibility of the Archive system (see section 3.1). A capture system must be checked to ensure that information about who created the record or when it was created, cannot be forged by either users or system administrators. The VERS long term format uses digital signature technology to ensure that any alteration after creation can be detected.the Victorian government is preparing legislation to assist in the acceptance and application of e-commerce throughout the community. The legislation will clarify various matters relating to the use of electronic/digital signatures and the use of public and private keys. 2 Where digital signature technology is applied close attention should be paid to the protection of private keys to ensure they cannot be discovered by other users of the system. In addition, attention should be paid to the provision of metadata from the system or application such as time and date stamps. It should not be possible to forge a record by altering the computer system clock. In systems where the record is not immediately captured in the VERS standard format, but is kept within an application and protected by the system s own internal and external security, attention to the issue of evidentiary integrity is of particular importance. 2.2 Unique Files/Record Identifiers The system must have the capability to generate Record and File identifiers that are unique within the Victorian Government. This is necessary to facilitate integration following public office reorganizations, eventual whole-of-government access, and possible eventual management of the records by PROV (see section 6.0). 2.3 Document Capture Interface Any record content that is analogous to a paper document (e.g. report, memo, , drawing) must be captured in the standard document format. A public office may elect to capture documents in other formats in addition to the standard document format. For any content not analogous to a paper document, the public office must confer with PROV to determine an appropriate archivable format. 2.4 Metadata Capture Interface The Record Capture system must be capable of capturing the mandatory metadata specified in PROS 99/007 Specification 2 VERS Metadata Scheme. A public office may also elect to capture additional metadata for storage in the record. If additional metadata is to be captured, it is recommended that the public office confer with PROV about the best approach to extending the standard record format to accommodate the additional metadata. This will assist in maximizing interoperability of metadata between public offices. An analysis should be performed to ensure that the total cost of metadata capture is minimized while maximizing the accuracy of information captured. As far as possible, metadata should be automatically captured from the computer system, application, or other records, and not be entered manually. However, attention should be paid to the up-front cost of software 2 See PROS 99/007 Standard for the Management of Electronic Records, Appendix Two for a discussion of electronic records and the law.

5 PROS 99/007 Specification 1: System Requirements for Archiving Electronic Records 4 development work required to automatically capture metadata and the on-going cost of maintaining the resulting system. Obviously, the cost effectiveness will vary depending on the application, the technology used to implement the application, and the importance of the records produced. 2.5 Metadata Control The record capture system should be able to control the metadata entered by users during record creation. This could be done using data validation techniques, pick lists and/or thesauri and the automatic capture of date and time information from the system. The cost of creating and maintaining the information used by these techniques must be analyzed. Care should be taken that the system of controlling metadata is not so rigid that it prevents users from accurately describing the record. 2.6 Record Linking When records are created, the Record Capture system must allow users to easily link the newly created record with other records in the Archive. This linking is not the simple application of a pointer to another record but relates to the use of common metadata to link related records. For example it is possible to link the answer to an inquiry to the original letter to which it is responding by using common metadata in both records. Creating a link should be one of the first manual options which users are prompted to consider since previously generated metadata can be used as a source of automatically derived metadata for the newly created record. One of the most familiar metadata links could be a file reference. All records with the same file reference would be considered to be in the same file. 2.7 Automatic Record Creation A public office may require automatic creation of records without any user intervention. Automatic creation can form part of a workflow system or part of a batch system. An analysis of the process will need to be carried out to determine where records can be created and how the appropriate metadata can be automatically captured. 2.8 Record Creation Functionality The Record Capture system must be capable of encapsulating the captured documents and metadata into the standard electronic record format as VEOs. An application may keep the captured documents and metadata within its own storage system for a period of time, possibly years, and only export the record in the VERS standard format when it is necessary to move the record outside the application. In this case, it is important to note that the evidentiary integrity of the VEO only applies from the time the standard VEO was created. Proving evidentiary integrity for a record, in this case, requires proving the integrity of the application for the entire period prior to its conversion to VERS standard format. The discussion on evidentiary integrity in section 3.1 applies here. Public offices should carefully consider the migration strategies used to maintain records kept in applications for extended periods of time. Accuracy of records and the reliability of the migration process must be ensured.

6 PROS 99/007 Specification 1: System Requirements for Archiving Electronic Records Archive System Requirements The Archive System needs to manage the archived electronic records for as long as they must be kept. The Archive System function may be incorporated within an existing public office application, or it could be an entirely separate system. 3.1 Evidentiary Integrity To ensure evidentiary integrity, the Archive System must ensure that it is possible to demonstrate who created the record, when it was created, and that it has not been altered since creation. In the Archive System, this primarily involves ensuring that any modification of the record can be detected as the necessary provenance information has been captured at creation of the record. Records must be protected against modification by both users and system administrators. Attention to the issue of evidentiary integrity is of particular importance where the record is not immediately captured in the VERS standard format, but is kept within an application and protected by the internal and external security of the application. In this case, a statement must be obtained from the vendor affirming that records can only be accessed via specified methods, that all access methods log the identity of the accessor and the nature of any changes made to the record, and that the information in these access logs cannot be modified or deleted. This statement must cover all accesses by all users, including system administrators using management and administrative functions. It must not be possible for records to be destroyed or deleted outside the normal disposal function. It must not be possible to hide records by deleting their indexing or record discovery information; in other words, records cannot become unretrievable by making them unfindable. Protecting evidentiary integrity is difficult, as complex systems may contain software bugs, or undocumented access mechanisms. A public office may require an audit of the system and its design to check the integrity of the system. 3.2 Modifying information associated with Records It must be possible to modify the information associated with electronic records without compromising the evidentiary integrity of the record. The VERS standard electronic record format supports this. This process produces onion records. 3.3 Verification The system must be capable of verifying the digital signatures associated with a record. When requested by a user this must be done when retrieving a record. The system must also be capable of auditing the validity of the digital signatures of a random sample of records. Any failure to verify a signature must be logged and immediately brought to the attention of the administrator, as failure to verify a signature may indicate an attempt to forge or alter Records. A record must be made of the administrator s acknowledgment of the verification failure.

7 PROS 99/007 Specification 1: System Requirements for Archiving Electronic Records 6 If records are cached outside the Archive to improve discovery performance, it must be possible to automatically verify that the cached record is unchanged from the copy in the Archive. 3.4 Access Control The Archive System must support access control on objects within the Archive. This includes records disposal schedules, and access control policies themselves. At a minimum, the access control policy must be capable of preventing unauthorized retrieval of records or modification of access control policies. This is a minimum requirement, and public offices should carefully consider the access control policies that they will need. The types of objects that need to be protected, the types of access to be controlled, and the classes of users granted or denied access need to be specified. 3.5 Disposal It must be possible to appraise (that is, evaluate and determine the record s status) and, where authorised, transfer or destroy records in a controlled manner (that is, to dispose of records). It is important to note that records must not be destroyed without appropriate authorisation by the Keeper of Public Records. 3 The system must allow the inclusion of records disposal authorities (such as Disposal Schedules). It must be possible, following PROV authorisation, to alter the policy that applies to any File (a collection of Records) at any time without altering the File itself. This allows policy on disposal to change without modifying Files. The system must support the controlled disposal of Files. Disposal of a File will dispose of all the Records contained in the File. This reflects current practice in PROV and public offices. The system must record (as part of the File Record) the disposal of a File. The information recorded must include: the identity of the officer who authorized transfer or destruction the date the File was transferred or destroyed the disposal authority under which the File was disposed This provides an audit trail to track the fate of specific records. The system must not disclose a Record that has been destroyed. may still physically exist in the system (e.g. on a CD), but the system must never disclose it. 3.6 Destruction of Records/Files The system must have the ability to purge destroyed records from the system (i.e. physically remove all copies) on demand by the system administrators. Some public offices 3 See PROS 97/003 Destruction of Public Records. Available at the PROV website

8 PROS 99/007 Specification 1: System Requirements for Archiving Electronic Records 7 may need to ensure that destroyed records have been physically removed from the system (e.g. for legislative reasons). The system must record the fact of the records purge, the time it took place, and the identity of the operator who performed the purge. This provides an audit trail to track the fate of specific records. 3.7 Auditing The system must be capable of recording all events that affect records, including creation of records, import of records into the system, export of records from the system, sentencing and disposal/destruction of records. This audit trail makes it possible to track accidental or deliberate wrongful destruction of records. It must not be possible to modify the audit log, otherwise no trust could be placed in the audit trail. If required, all accesses to Records must be capable of being logged. The log will include what Records were retrieved, the identity of the user retrieving the Records, and the time. This allows unauthorized access to Records to be detected. 3.8 Reliability and Redundancy The system must not lose Records (e.g. by a disc crash) once they have been accepted by the system. A risk analysis should be undertaken on the system to identify any way records might be lost and appropriate backup strategies that might be implemented. 3.9 Refresh The system must have the ability to refresh the media on which records are stored (i.e. copy the contents from one piece of media to another medium). The accuracy of the refresh must be verified by ensuring that all Records which have not been marked for destruction have been copied, and that the contents of the Records have been copied accurately. The system may record the refresh (including time of refresh, operator invoking refresh (if any), and the identity of the media being refreshed). The system may reorganize Records on the media (e.g. merge the contents of two pieces of media onto one). In this case, the system may also record the source and disposition of each Record Record Transfer Custody of records must be capable of being transferred from the system where they were created to other systems and off-line. This may include transfer from a public office system to the PROV system, or from a public office system to an off-site record storage location which meets PROV storage standards. 4 4 See PROS 97/004 Transfer and Storage of Public Records. Available at the PROV website

9 PROS 99/007 Specification 1: System Requirements for Archiving Electronic Records 8 Importing or exporting of records from a system must be recorded. The identity of the operator exporting the record must be recorded. A reference to the new location of the record should be recorded Unique Media Identifiers If Records are stored on removable media (e.g. CDs), the system must have the capability to generate media identifiers that are unique within the system. If media are to be transferred to another public office or to PROV provision must be made to ensure that the media identifiers are unique or are made unique within all the offices (including PROV) involved in the transfer. Public offices should consult with PROV on the application of unique identifiers. 4.0 Record Discovery System Requirements 4.1 Supporting documentation (Finding Aids, Thesauri) The system must support the creation, storage, and use of documents that describe aspects of the Records and Files. The classes of documents that may be required include: Finding Aids. Finding Aids are documents providing context for Files and Records. Finding Aids may describe the functions and organization of public offices or describe the organization of records (i.e. the Series/File structure) within the public offices. Thesauri. Thesauri (pick lists, data dictionaries) define the precise meaning of terms used in records, particularly in titles and keywords. Certification Records. Certification Records provide a means of discovering what public key a particular user employed at a particular date. 4.2 Access Control The system must support access control on objects within the repository, including files, records, finding aids, disposal schedules, and the access control policies themselves. At a minimum, access control policies must be able to permit or deny individual users access to read individual records and files. More sophisticated access control (e.g. controlling access to collections of records, or restricting certain groups of users) may be required by a public office. Public offices should carefully evaluate the administrative costs of managing access control policies to determine the most cost-effective way of achieving the level of control they require. 4.3 Access Logging Public offices may require that Record Discovery systems log all record accesses. The information logged during a record access includes: the identity of the user requesting the record, the record requested, whether access was granted, and the system to which the record was sent.

10 PROS 99/007 Specification 1: System Requirements for Archiving Electronic Records User Interface It is necessary to support desktop user access to the records, files, and finding aids. Vendors must list any software required to be installed on desktop machines to access the Record Discovery system, and the hardware specifications of the desktop machines that can effectively run this software. Public offices may require access to records via the World Wide Web. Web interfaces should support access by world wide web browsers and utilise the hardware used by the public office. Particular care should be taken with the user interface design; it will be the primary way in which users will interact with the system. Care should be taken with the presentation of metadata and content to avoid confusion. The user interface should be as intuitive as possible for users. 4.5 Searching The system must support both searching and browsing for files and records. At a minimum, the system must support searching by the metadata contained within the record. A public office may require the ability to perform full text searching of the content of the records. The system must have the ability to construct boolean queries. The system must have the ability to sort the results of a query on the basis of a user specified list of metadata fields. The system must support hyperlinking of records by means of contextual information stored in the record and not through the use of electronic pointers. The system must support browsing from Finding Aids to the records browsing through the hierarchy of agency, series, file, records browsing through the related files and records which may include browsing through the records associated with a particular transaction 4.6 Verification The system must be capable of verifying the digital signatures associated with a record. When requested by a user this must be done when retrieving a record. The system must also be capable of auditing the validity of the digital signatures of a random sample of records. Any failure to verify a signature must be logged and immediately brought to the attention of the administrator, as failure to verify a signature may indicate an attempt to forge or alter Records. A record must be made of the administrator s acknowledgment of the verification failure. 4.7 Rendering The system must be capable of displaying the record as it appeared to the original creator of the record.

11 PROS 99/007 Specification 1: System Requirements for Archiving Electronic Records Generic VERS Record Structure The following section describes the structure of a generic record within VERS. Information for the metadata scheme used can be found in PROS 99/007 Specification 2. Technical information can be found in PROS 99/007 Specification 3. VERS Encapsulated Object Object Content Document Encoding Object Metadata Record Metadata Document Metadata Encoding Metadata Object Content Document 1 Encoding 1 Document Data Signature Block Document Encoding n Signature Block Document n Figure 1: Standard Electronic Record Structure The generic VERS record (VERS Encapsulated Object or VEO) can contain any type of object. The information in a VERS record includes some contextual information (or metadata) about the object, one or more signature blocks, and the signed object contained within the record (see Figure 2). VERS Encapsulated Object VEO Format Description Version Signed Object Object Metadata Object Type Object Type Description Object Creation Date Object Content Signature Block Signature Format Description Signature Date Signer Signature Certificate Block Signers Certificate Certificate Reference Signature Block Signature Format Description Signature Date Signer Signature Certificate Block Signers Certificate Certificate Reference Figure 2 Generic VERS Record

12 PROS 99/007 Specification 1: System Requirements for Archiving Electronic Records 11 Each signature block contains the result of signing of the record. The signature validates the object metadata and the object content. The information in the signature block includes the digital signature, the date of the signature, the identity of the signer, and the signer s public key. Inclusion of the signer s public key allows the signature to be verified irrespective of the availability of a certification authority (which may not last as long as the VEO). The final component is the object itself. Two basic object types have been defined: File and Record. Each are described in the following sections (sections 5.1 and 5.3). 5.1 Records A record has two parts, as shown in Figure 3. The first part is record level metadata that deals with the record as a whole. The second part contains the content of the record and contains one or more documents. Record Record Metadata Handle Metadata Context Metadata Policy Metadata History Metadata Document 1 Document 2 Document n Figure 3 VERS Record Object Record Metadata Record Metadata Handle Description Language Coverage Record Identifier VEO Identifier Context Agent Title Subject Relation Function Type Aggregation Level Format Location Transaction Policy Rights Management Disposal Mandate History Date Management History Use History Preservation History Figure 4 VERS Record Level Metadata The information includes: Handle Metadata. The unique identifier of this record and descriptive information.

13 PROS 99/007 Specification 1: System Requirements for Archiving Electronic Records 12 Context Metadata. (How this record relates to other records). This includes: information about the transaction documented by this record; who is responsible for the creation of the record; links to related records; information about the record itself and documentation as to the reliability of the system which generated and maintains the record. Policy Metadata. This metadata describes information about the policies which affect the record. This includes: who may access the record; how the record may be used; and disposal information. As policies may change rapidly this information may be stored within the system separately from the record itself. Some of this information, however, will need to be exported as part of the record if the record is transferred to another VERS repository. 5 In this situation, the system will form a new VERS record by wrapping the additional metadata around the original record to form an onion record. 6 History Metadata. This metadata describes the history of the record. As this list will be continually growing this information may be stored within the system separately from the record itself. Some of this information, however, will need to be exported as part of the record if the record is transferred to another VERS repository. 7 In this situation, the system will form a new VERS record by wrapping the additional metadata around the original record to form an onion record Documents A record may contain several documents. Each separate document is represented in the record as a separate unit. A document may be a set of data (for example, a report, a CAD file, a database, Web pages, or datasets). Different types of document will be encoded differently and will need different metadata, but the VEO has a standard structure for documents which can be customized to handle different document types. A document contains document level metadata, and one or more encodings of the document (see Figure 5). Document Document Metadata Description Document Source Encoding 1 Encoding 2 Encoding n Figure 5 VERS Document Structure The document level metadata contains descriptive information about the document (e.g. its purpose), and information about the source of the document (a textual description of the application that produced the document). For certain types of documents the source 5 See discussion in PROS 99/007 Specification 2 section See discussion in PROS 99/007 Specification 3 section See discussion in PROS 99/007 Specification 2 section See discussion in PROS 99/007 Specification 3 section 2.2.3

14 PROS 99/007 Specification 1: System Requirements for Archiving Electronic Records 13 description may be very detailed and may need to be structured. For example, if the document was a database, the source description could include the database schema to aid interpretation of the database. 5.2 Encodings An object may be encoded in many ways when it is included in the record. For example, a report could be encoded as a PDF file, or a Word file. A Document in the record can contain several representations of the same physical document. Each Encoding represents one representation of the physical document. It contains the encoded document, and encoding level metadata that describes how the physical document was encoded. Encoding Encoding Metadata File Encoding File Identifier File Rendering Document Data Figure 6 VERS Encoding Structure The encoding level metadata contains two elements. File Encoding. This is a textual description of the process used to convert the original data into the representation included in the record. To view the record, this process will have to be reversed and the description should be written to be clear to someone attempting to view the contents. File Rendering, contains a textual description of the process by which the document may be viewed (normally the reverse of the encoding process) and keywords which may be used by a computer program to reverse the encoding process. 5.3 Files File Records contain information about files (i.e. groups of records). Files are primarily used to assist in simplifying the discovery and management of a collection of related records. For example, it is significantly easier to manage a file (and all the records it contains) than to manage each record individually.

15 PROS 99/007 Specification 1: System Requirements for Archiving Electronic Records 14 File File Metadata File Disposal Agent Rights Management Title Subject Description Language Relation Coverage Function Date Type Aggregation Level Format Record Identifier Management History Use History Preservation History Location Disposal Mandate VEO Identifier Disposal Schedule Disposal Date Authorising Officer Figure 7 VERS File Structure The structure of a File is shown in Figure 7. It contains the following information: File Metadata. Information which describes the file (this information is similarly structured to Record Metadata). File Disposal. If the file has been disposed of, this element contains details about the disposal; when the file was disposed of, who authorized transfer or destruction, and what was the disposal authority. 6.0 Unique Record Identifiers Each VERS object must have a unique record identifier across the Victorian government. This allows records to be moved between agencies (e.g. during reorganizations) or be managed by one agency (e.g. PROV) without identifier clashes. 9 Until records are transferred records may have identifiers which are only unique within the agency which has custody of the records. However, once the records are to be transferred care should be taken to ensure that the record identifiers are unique within all agencies (including PROV) involved in the transfer. Agencies must contact PROV prior to the transfer of permanent records to PROV to obtain a unique domain number and for further information about unique record identifiers. 9 See PROS 99/007 Specification 2 section M99 for information on how to construct unique identifiers.