Research Data Understanding your choice for data placement

Transcription

1 Research Data Understanding your choice for data placement V1.4 Draft Aslam Ghumra 30 October 2014

2 Executive Summary This document is an overview of the technical infrastructure and supporting processes available for the support of Research Data Management (RDM) at the University of Birmingham. The appropriate management of research data throughout the data lifecycle, during and after the research project, is considered good research practice. This involves data management planning during the research proposal stage; looking after active data, its creation, processing, storage and access during the project; and data stewardship, long-term curation, publishing and reuse of archive data after the end of the project. Good RDM practice benefits all stakeholders in the research process: Researchers, will secure their data against loss or unauthorised access, and may increase research impact through publishing data; Research institutions may consider research data as special collections and will need to minimise risk to data and damage to reputation; Research Funders wish to maximise the impact of the research they fund by enabling reuse; Publishers may wish to add value to research papers by publishing the underlying data. DRAFT Research Data Understanding your Choice for data placement 2 14

3 Background Many research funders now mandate RDM procedures 1, particularly Data Management Planning (DMP), and the UK research councils policies have contributed to the RCUK common principles on data policy 2. Notice must be taken of the EPSRC Expectations 3 of organisations receiving EPSRC funding. These include the requirements that the organisation will: Publish appropriately structured metadata describing the research data they hold. Ensure that EPSRC-funded data is securely preserved for a minimum of ten years. Ensure that effective data curation is provided throughout the full data lifecycle. Institutions in receipt of EPSRC funding are expected to be compliant with these expectations by 1st May The University of Birmingham Research Data Management Policy 4 was developed in response to the RCUK principles 5 and EPSRC expectations 6. This states that the University will develop infrastructure and services to support research data management in consultation with researchers. The major functional components of the RDM technical infrastructure for the institution are: Research Data Store (RDS) Active (working) research data needs to be accessed rapidly, may require large computational resources and may require stringent security and access arrangements. A number of collaborative systems and virtual research environments have been developed to fulfil these requirements. Research Data Archive (RDA) will be an appropriate place for preservation and publishing of archive research data for which there is no discipline-based repository (e.g. UKDA 7 ) or data centre available. The catalogue and archive functions of the repository may be separated. PURE - Current Research Information System (CRIS) manages the metadata associated with researcher identity, project information, research costing, grant applications and awards. FindIT (Primo) holds the metadata records of published (but not necessarily open access) research data. Metadata capture system in order to identify, describe and document the research data as they are created, captured and processed and record the context, conditions, variables and instrument settings. This may be accomplished manually, by the researcher filling in forms, or automatically, concurrent with data capture, by using appropriate equipment. We are hoping to have this available by May These components may overlap in function, but need to be interoperable to provide seamless RDM. Alternative approaches to an infrastructure composed of diverse components, where ensuring interoperability may be problematic, are provided by data grids and micro-services 8 (e.g. middleware applications that allow researchers to manage data across grid infrastructures using third party software such as 7-zip). 1 Information adapted from the Report to the University of Sheffield Research Data by John Lewis Currently in draft, awaiting authorisation from Research Committee DRAFT Research Data Understanding your Choice for data placement 3 14

4 The technical infrastructure must fit into the researcher workflow, making RDM processes automatic and virtually invisible to the researcher as far as possible. This is so as not to burden the researcher with additional work or changes to their practice. Products, processes and practices that have been developed by a researcher community should be adopted, adapted and developed for the needs of other researchers, rather than new solutions developed. Data Workflow Data collected during a research project will include the research data themselves; experimental, observational, modelled data etc. together with the metadata that describes these data in detail and documentation describing the context of the research, details of the research project and the processes involved. Data here will be defined as the numerical and textual information collected by analysis or measurement from the research samples or objects but not the samples or objects themselves. For example, a collection of images or of tissue samples will not be considered data until there is textual or numerical information, such as identifiers (names or ID numbers), descriptions and relationships, associated with them. In this document, data refers to digital data, although the same management principles apply to analogue data formats. However, it is best practice to digitise research data, making its discovery and reuse easier. Once underway, a research project will collect or create raw data, which, during the project, will usually be processed to create derived or processed data. There may be many different iterations of processing, resulting in many sets of derived data. Eventually a set of results data will be selected as the basis of the research publication(s) output by the project. All these sets of data can be considered active data, which will need to be quickly accessible and easily shared between collaborators. All these sets of active data will need appropriate documentation to describe the processes involved in their creation and modification. After the project has finished, researchers will need to select data for curation on a long-term basis. This may have been decided in agreement with the research funder during the initial planning stage of the project. Curation in the context of RDM, refers to archiving, preservation and adding value through transformation and reuse. These archive data selected for curation, may need further processing (validation, cleaning, anonymisation or redaction) before submitting to an appropriate repository. The associated metadata will be needed to provide the necessary information for citation and re-use. There may be the facility to add new metadata or documentation, generated by data reuse, to the curated dataset. Data not required for curation needs to be disposed of in an appropriate manner. DRAFT Research Data Understanding your Choice for data placement 4 14

5 Projects finished data transferred Workflow diagram Data, WiP RDS [metadata] http [metadata] [ UoB Researcher ] PURE (CRIS) Web Portal FindIT@Bham [ Joe Public ] RDA [metadata] The above diagram shows how data becomes discovered by interested by parties (e.g. Joe Public). The data is created and placed into the RDS, once a paper has been published a record of the paper is placed in PURE; one of the attributes of the PURE record is to specify where the data resides. The PI requests storage in the RDA, where the data associated with the published paper is placed and this network location is also placed in the PURE record. When a user requests data via the web portal (FindIT), it makes a request to PURE to see if that data exists, (only data which is classified as publicly available is listed); if it does then metadata of the published paper and data are displayed. The user then has the option to review the paper and request (where available) the data associated with it, the data is given to the requestor in a timely and responsible manner. DRAFT Research Data Understanding your Choice for data placement 5 14

6 Research Data Store (RDS) The RDS is a storage service which has been procured for the active research data store. The RDS is made up of an IBM based storage solution, which uses v3700 storage enclosures and GPFS (General Parallel File System) to enable the datastore to be presented to front end servers. The front end servers are based on samba ( which allows us to present network drives (or shares) from the storage to your PC/MAC as a local drive. Access To get access to the RDS, principle investigator s (PI s) should put in a service desk call; once approved, 5Tb of storage will be allocated, additional capacity may be purchased 9 for further info. Once the storage has been allocated the PI will be asked to give a list of users who are able to access this storage, this will form the security group of who can access (e.g. read, write or both) the datastore. The PI will be given a network share in the form of \\RDS\user1-project, which can be mapped from their computer, this will be in a form of a CIFS share, which can be presented to Linux, MAC or Windows operating systems. Resilience The solution has been created to be highly resilient and is located at two data centres, if there is a major issue at one data centre we are able to work from the other. We have taken into account the majority of failures such as : Single disk failures Enclosure failures Inter-site link failure Site failure 9 Research Data Storage Pricing and Guidance for Research Grant Applications to add link> DRAFT Research Data Understanding your Choice for data placement 6 14

7 The following high level diagram shows us how the solution is setup to cate for resilience. Elms Road Data centre Aston Webb Data centre RDS Replication of data RDS [UoB Researcher] When data is placed on the RDS, the solution will automatically copy the data over to the other datacentre; researchers will not have to be involved in this procedure. At a point where there is a failure of any kind, the service will actively resume, however, depending upon the nature of the failure, it could take up to an hour for normal service to return. Backup When data is placed in the RDS it will be backed up and retained for the specified period (currently 6 months). Once a day the data is backed up using our current backup software (IBM Tivoli Storage Manager). The software will at the time of backup see which files have been created, amended or removed and will then take appropriate action, depending upon the policies set. If a file has been created and deleted before the backup has started, this file will not have been backed up. Due to the nature of the data on this storage, being work in progress and subject to a lot of change (transient files will be created and deleted). During 2015, a Hierarchical Storage Manager (HSM) will be introduced to move aged data (data not accessed for some time) from primary storage to a more economical location for maximum efficiency. This process will be automated and policy driven in accordance with some simple principles. DRAFT Research Data Understanding your Choice for data placement 7 14

8 Age of Data Less than 2 months Between 2 months and 4 months Between 4 months and 6 months Fast disk Slow disk Tape Tier of Storage In order for us to reduce the time taken for the restoration of the data we are going to keep 2 days worth of data (snaphots) on disk, so that users are able to recover files they may have inadvertently removed, without going back to tape. These snapshots will occur once a day, similar as the backup but at a different time, this enables the users to be self-sufficient. The diagram on the following page is a detailed explanation of how the hardware has been setup including all network connectivity, including SAN fabric, Infiniband 10Gb Ethernet. DRAFT Research Data Understanding your Choice for data placement 8 14

9 Elms Road RDS This diagram is a detailed explanation of how the hardware has been setup including all network connectivity, including SAN fabric, Aston Infiniband Webb 10Gb Ethernet. Login01 Login02 V3700 V SAN SAN SAN SAN GPFS01 GPFS02 GPFS01 GPFS02 10Gb IB 10Gb IB CTDB01 CTDB02 CTDB01 CTDB02 Quorum Services CLIENTS DRAFT Research Data Understanding your Choice for data placement 9 14

10 Research Data Archive (RDA) The RDA has been created specifically to store data associated with research papers. The solution was created by IT Services in conjunction with suppliers and the MidPlus consortium. The MidPlus consortium consists 4 universities, University of Warwick, University of Birmingham, University of Nottingham and Queen Mary, University of London. As part of a High Performance Computing (HPC) facility the consortium was tasked to create a compute and storage service to archive data. Birmingham is using this collaborative facility as its institutional RDA. The RDA is based on HP hardware and storage which utilise Simpana CommVault v10 software to archive data; the HP hardware are ProLiant servers and StorageWorks MDS 600. Access As with the RDS, access to this storage relies on the PI to log a service desk call, so that we can properly process the request and authenticate the PI is the person who owns the data. There will be an allocation of 1Tb of storage for each datasets associated to a published paper, each published paper will have its own allocated storage. If the PI requires more storage then there will be a charge 10. Once the storage has been allocated the PI will be asked to give a list of users who are able to access this storage, this will form the security group of who can access (e.g. read, write or both) the datastore. The PI will be given a network share in the form of \\RDA\user1-project, which can be mapped from their computer, this will be in a form of a CIFS share, which can be presented to Linux, MAC or Windows operating systems. After a period of time (i.e. 1 month), we will stop all write access to this datastore, this stops any inadvertent additions or deletions of the dataset, however the PI can ask for this to be extended for a short period of time. This has been done so that we are confident that the dataset has not been tampered in any way. Resilience The solution has been created to be highly resilient and is located at two data centres in two different cities, so if there is a major issue at one data centre we are able to work from the other. We have taken into account the majority of failures such as: Single disk failures Enclosure failures Site failure 10 Research Data Storage Pricing and Guidance for Research Grant Applications to add link> DRAFT Research Data Understanding your Choice for data placement 10 14

11 The following diagram shows us how the solution is setup to cater for resilience. DR in the following diagram refers to Disaster Recovery. University of Birmingham University of Nottingham RDA Data copied to Nottingham for DR RDA [UoB Researcher ] [Warwick Researcher ] [QMUL Researcher ] DR Access When data is copied over to the RDA, there are policies within the software to copy the data over to the University of Nottingham for DR purposes. In the event of a failure of any sort we are able to access the data, however depending upon the nature of the failure, it could up to 1 day for normal service to resume. Only the UoB staff can access their own data and is not available to anyone apart from themselves. DRAFT Research Data Understanding your Choice for data placement 11 14

12 The following is a detailed diagram of how the RDA has been setup DRAFT Research Data Understanding your Choice for data placement 12 14

13 PURE (CRIS) 11 Pure facilitates an evidence-based approach to our institution's research and collaboration strategies, assessment exercises and day-to-day business decisions. Research information is often fragmented across various systems and spreadsheets within an institution. This leads to administrative overhead, inconsistent and incomplete data and unreliable reports, all of which adversely affect business decisions. Pure aggregates our University s research information from numerous internal and external sources, and ensures the data that drives your strategic decisions is trusted, comprehensive and accessible in real time. A highly versatile centralised system, Pure enables us to build reports, carry out performance assessments, manage researcher profiles, enable research networking and expertise discovery and more, all while reducing administrative burden for researchers, faculty and staff. For each published paper we will allocate a directory / network share for the researcher to deposit their data. This network share will then be added to the PURE record which will then form a link to the data. The more information the researcher puts in their PURE record (metadata), the more likely it is citable. Metadata is data that supports the discovery, understanding and management of other data and information. It provides information about an item of data and its relevance so it can be easily found when needed. Capturing and maintaining the correct metadata is increasingly being viewed as perhaps the key to the reuse and preservation of digital objects. The metadata in the PURE record is then used on the RDA as an XML file to capture the dataset information. FindIT (Web portal) The web portal is piece of software from ExLibris, it is a Discovery and Delivery solution, which allows library users to enjoy the full benefit of a one-stop service for their research needs. Through a single search box, users can now search across the full breadth of content from the library s collection, exploring the library s locally-managed collections, along with the global and regional resources. The software has been integrated with our Library systems and our CRIS (PURE) so that articles, papers, journals etc. as well as associated metadata records are publically searchable. Metadata Capture System We are putting in place a java / web application to capture the metadata information of the files that have been added to the RDA. The benefit of this is twofold 1. Auditable Making sure the files that are listed are available, including security checks 2. Confidence Users are confident that the files they receive are secure and don t have hidden viruses, trojans etc. When the user finally finishes adding data onto the RDA, the user will then run an application to create an xml file which will hold information such as 11 DRAFT Research Data Understanding your Choice for data placement 13 14

14 creator title identifier language source pureid number of files filename id description format id size of file in KB id md5hash Glossary PI Principal Investigator, the lead academic on a research grant DRAFT Research Data Understanding your Choice for data placement 14 14