Research Data Storage Infrastructure Education Investment Fund Annual Business Plan

Transcription

1 Research Data Storage Infrastructure Education Investment Fund Annual Business Plan May 2014

2 Research Data Storage Infrastructure Education Investment Fund (EIF) Annual Business Plan Executive Summary Objectives Summary of Progress Programmes Risk Factors Background Information Objectives of the Project Outcomes of the Project Implementation Principles Context and Scope of the Project Context Scope Participating Organisations Management and implementation arrangements Relationship to other eresearch investments Funding arrangements Project Implementation Node Development (NoDe) Programme ReDS Programme Initial Research Data Storage Interim Research Data Storage Ongoing Research Data Storage (Top-up Funding) RDSI Board Allocation Process DaSh Programme Outline of DaSh Themes VePa programme Project Conclusion Management and Implementation Governance and Management Framework RDSI ABP Approved by the Department of Education Page 2 of 67

3 4.1.1 Roles Milestones Milestones for Project Resources RDSI Income and Co-investment RDSI Expenditure Total Investment Summary Performance Indicators APPENDIX A RDSI Risk Register APPENDIX B Terms of Reference and Operations B.1. Membership B.2. Meetings B.3 Expenses APPENDIX C Allocation Proposal Form Part A Part B RDSI ABP Approved by the Department of Education Page 3 of 67

4 Executive Summary This document is the fourth Annual Business Plan (ABP) for the Research Data Storage Infrastructure (RDSI) Project funded from the Education Investment Fund (EIF) under the Super Science Initiative covering the financial year July 2014 to December 2014 when the project operationally concludes. A reduced Project Office, consisting of the Project Director, Project Manager and Office Administrator will continue to manage the project until June 2015 to fulfil The University of Queensland s contractual obligations as Lead Agent.. This will include writing and submitting the final report. A funding agreement between the Commonwealth represented by the then Department of Industry, Innovation, Science and, Research (superseded by the Department of Industry, Innovation, Climate Change, Science, Research and Tertiary Education (DIICCSRTE) and now the Department of Education), henceforth referred to as the Department, and the University of Queensland (UQ) was executed on 23 rd December 2010, with the Project Office established on 27 th January A Deed of Variation was subsequently executed on 6 th May 2013 extending the RDSI Project operational conclusion to 31 st December 2014 and the contractual conclusion to 30 th June 2015, with a requirement to submit the final report by 31 st March The project was extended to strengthen the technical development of the data stores (aka Nodes) in a national network through the funding of specialised staff at each Node to support the effective ingest of data into those stores and to provide access to collections to researchers. A total 53 participating organisations are involved in the operation and use of this national distributed set of facilities. The aim of the RDSI Project is that researchers will be able to use and manipulate significant collections of data that were previously either unavailable or difficult to access and that there will be a consistent means of accessing this data. This has been successfully achieved through the development of a national network data stores, or Nodes, where the content held in those stores is allocated capacity through a priority and merit process and where that content can be readily accessed, analysed and reused in a coherently governed environment. The project extension has demonstrated that it is effectively accelerating the development of this national network of Nodes by funding staff at the Nodes to support and expedite deployment of their infrastructure and Merit allocation process and ingesting data. Currently over 33 PB of collections have been approved by the Nodes Merit allocation committees with over 9 PB of data available to researchers and this number is growing on a daily basis. As part of RDSI s outreach activities the project has consulted with a number of researchers in collaboration with Nodes and has developed a significant number of use cases describing how the availability of data in RDSI funded storage at Nodes is benefiting their research. These use cases, described as RDSI stories are available at RDSI s web site ( The RDSI project is assisting with Node sustainability beyond the project duration by: creating a robust network of data stores increasing the amount of time Nodes have to ingest data increasing the exposure of collections that were before unavailable to researchers creating a set of robust collaboration arrangements, and increasing the collective technical knowledge required to support the national storage infrastructure created by the RDSI project RDSI ABP Approved by the Department of Education Page 4 of 67

5 In October 2013, The Lead Agent, after advice from the RDSI Board recommended that the Department agree to an extension of the operational end date of the project until June 2015 with a revised date of 30 th September 2015 for submission of the final report and 31 st December 2015 for contractual completion of the project. This would align the dates of both the RDSI project and the RDSI NCRIS 2013 project. The proposal included recommendations to extend funding of the RDSI funded staff at Nodes until the operational completion on 30 th June 2015 and would require extension of the skeleton project office described earlier until 31 st December 2015 This recommendation has not, to date, been approved but if approval is provided, a review of the funding of RDSI funded staff at Nodes may be appropriate as this activity could be superseded by NCRIS 2013 funding. In any event, if the recommendation is approved, this ABP will need to be updated. The RDSI project is currently undergoing a project review. The review is being led by Professor Tom Cochrane and is expected to conclude in May The outcomes of the review will be considered by the RDSI Board, the Lead Agent and the Department and this ABP may need to be updated to accommodate approved recommendations from the review. The project, in consultation with Nodes and other eresearch projects, identified the need for funding the storage of large collections of research data, which would fall outside the scope of existing storage funding mechanisms within the project. The RDSI board and the Lead Agent have identified and approved a process The Board Allocation Process which would address this requirement. It has been supported by all Nodes after extensive consultation, which included a workshop in which all stakeholders participated. This process is described in the RDSI website and it is proposed that it should be adopted immediately. By the end of April 2014, estimated requests for funding research data storage at Nodes exceeds the capacity to fund such requests by a factor of 1.7. The RDSI Board will provide the Lead Agent with advice on the appropriate requests to fund based on approved principles and objectives defined by the project plan and approved business plans. Reports from Nodes indicate that, at the end of April 2014, a total of Petabytes of Research Data has been approved by Merit Allocation Committees for storage and that Petabytes of this has been ingested into Nodes Stores and is available for research use. It is anticipated that the project will meet its objectives within the timeframe and budget defined by both this ABP and the current ABP 2013/2014. RDSI ABP Approved by the Department of Education Page 5 of 67

6 1.1 Objectives The objectives of the RDSI Project are: Develop a national network of a limited number of data stores, or Nodes, where the content held in those stores is allocated capacity through a priority and merit process and where the content can be readily accessed, analysed and re-used in a coherently governed environment. There will be Primary and Additional Nodes. Create and develop a data storage infrastructure accessed through a common infrastructure layer provided by agencies within the sector, or commercial providers, or both. Connect the data storage infrastructure created by the RDSI Project to the Australian Research and Education Network (AREN) by a high bandwidth connection, funded and constructed under the Super Science National Research Networks (NRN) Project. This will include dedicated high speed connections between major Nodes. Support meritorious data collections through the Research Data Service Programme (ReDS). Encourage economies of scale through the Vendor Panel Programme (VePa). The intended outcomes of the project include: Providing significantly increased capacity for research data holdings. Developing and improving associated access and data sharing capabilities. Enhancing the capability and capacity to provide research data services to the sector. The result is intended to be a data storage framework that will assist institutions and researchers to more effectively use, manage, share and preserve much larger holdings of research data. 1.2 Summary of Progress The business plan principles and objectives remain substantially unchanged for with the project wrapping up its programmes and transferring services to the sector by December It is planned that the ReX system will be transferred to the Australian Access Federation (AAF) to maintain and continue to develop the system as new researcher identity and access management requirements emerge. The Vendor Panel developed in conjunction with CAUDIT will be transferred and maintained by CAUDIT. The DaShNet will continue to be maintained by AARNet. The LiveArc and ARMS will be maintained by Nodes along with the monitoring system and Nodes Security policies. The RDSI funded staff at Nodes used to accelerate and support the data ingestion and collections merit allocation will conclude in December Nodes intend to continue to support their infrastructure and make the collections available to the sector in line with the RDSI project objectives until June The RDSI Board has recommended to the Department to extend the project until June 2015 and continue to fund the RDSI funded staff at Nodes. The RDSI project Office would continue to work until 31 December 2015 to close down the project and provide the required reports to the Department. If this recommendation is accepted this ABP will need to be updated. In addition if approved by the Department the NCRIS 2013 funding will therefore transition the investments made through the RDSI (EIF) Project into a (long-term) sustainable data service through RDSI ABP Approved by the Department of Education Page 6 of 67

7 the single overarching goal of making operational Data as a Service for the Australian research community. These broad objectives build on the intent and achievements of the RDSI project by elevating the operational capability and support, in alignment with both the growth of collections, made available through the project, and the growth in expectations of research organisations and communities for higher levels of service (over and above the foundation access that is now in place) that are required to accelerate and improve the attainment of research outcomes. There has been significant progress against all milestones during the period. There are 50 milestones from the ABP, 84% are either completed, on track or an on-going activity and 16% are in progress. These are detailed at The adjustment of some project milestones in the period owe to anticipated early delivery of some activities and justified delays in others due to Node s intention to ensure RDSI infrastructure is co-located with other data infrastructure to ensure researchers have the best possible environment to work with. The project risks and mitigation strategies identified in the previous Project Plan are broadly unchanged and are revised periodically to analyse various project risk aspects. These are detailed at Appendix A. As a broad overview during the year , the project achieved the following outcomes: Oversaw the implementation of the ReDS acceleration activities with Nodes. Continued to manage Node contracts for distribution of data storage via the ReDS programme. Continued to support Nodes to purchase from the Vendor Panel. Continued to support the Vendor Panel RFP Vendor Evaluation Committee (VEC). Launched the ongoing VePa RFP. (Vendors are able to submit their applications at any time and the evaluation committee assesses on a need basis. This facilitates the entrance to the Vendor Panel.) At the time of writing this plan there were 18 vendors on the panel. ReDS programme continued to release funds to Nodes to enable storage infrastructure to be provisioned in advance of storage being allocated to data. ReDS programme continued to monitor data allocation and use. Continued engagement with stakeholders. Launched the Test Platform theme to foster Nodes implementation and future sustainability, this has been used to test components of the RDSI technical architecture. Refined the DaSh programme requirements and proposed new approaches to further support the Nodes technical architecture implementation. As a result Nodes volunteered to develop a programmatic data services interface, the LiveArc and Aspera integration, a dropbox style solution to be evaluated, and negotiated with the vendor an extension to the LiveArc license and maintenance for additional 12 months. Developed security policy guidelines for Nodes and Institutions. Developed Security Policy Model for Nodes. Developed Security Policy Model for Institutions. Continues to identify security related matters and provide input to Nodes. Implemented the DaShNet via AARNet. Continue to implement the RDSI Communications plan activities with Nodes and stakeholders. RDSI ABP Approved by the Department of Education Page 7 of 67

8 Maintain the RDSI website up to date with the latest information about the project, its programmes, subprojects and milestones progress, as well as publishing use case stories, newsletters, a security feed, a general tweeter feed, mailing lists and other social media tools to support the dissemination of project information to Nodes and stakeholders. RDSI Project Office participated in a number of Node meetings to discuss their architecture, implementation and timelines. RDSI Project Office continued to meet regularly with Node Steering Committee (Node SC) and Node Technical Advisory Committee (TAC). RDSI team continued to engage with stakeholders and had a strong presence at the eresearch Australasia Conference 2013, promoted awareness of the RDSI project to Nodes and researchers. Continues to monitor the Nodes implementation and publish Node performance information at the RDSI website. Additional activities carried out during ABP as per Department and/or Board request: Project Office assisted with requirements of the RDSI project review requested by the Department. Provided feedback to the Department on Terms of Reference and communications guidelines as requested. Provided feedback on reviewers and reference group. Supported and provided information to the reviewer as requested. Identified the need to fund large storage collections, developed guidelines for Nodes to participate. As a result received large storage collection proposals which are under review by the RAP at the time of writing this ABP. Assisted the RAP with logistics for evaluation and recommendation of proposals to the Board. A comprehensive description of the proposed Project Implementation during period is provided in section 3 with milestones outlined in section Programmes The Project has been undertaking activities across four key programme areas as follows: Node Development (NoDe): A programme to identify, strengthen and develop research data centres able to hold and process high data volumes. o Research data centres are supported either as a Primary Node or an Additional Node. o NoDe has funded the 6 Primary Nodes and 2 Additional Nodes. o Nodes are operational and provisioning data storage for researchers. Research Data Services (ReDS): A programme to identify research data holdings of lasting value and importance and to contribute funding to their development at the most appropriate Node. o Nodes are using the Merit Allocation process to identify collections of significant use for the sector and making these available. o ReDS programme has provided funds to support the storage of such collections. o ReDS programme monitors Node ingestion and storage use progress regularly. Data Sharing (DaSh): A programme to provide the widest possible range of general data sharing and movement infrastructure suitable for data intensive research activities. This programme comprised of two major components, a high performance network (DaShNet) and a DaSh Technical Architecture. DaSh extended the reach of the RDSI project by providing the RDSI ABP Approved by the Department of Education Page 8 of 67

9 DaShNet data transfer mechanisms to share data and by developing access to public storage services. DaSh funded, managed and in cooperation with Nodes deployed or is deploying a number of sub-projects listed herein further progress details are available on the RDSI website ( o ReX Identity management system for researcher identity and access to storage being developed by the AAF and Intersect. o LiveArc collection management system being delivered to Nodes and implemented as per their requirements. o Allocation Request Management System (ARMS) is an online application form for ReDS merit-allocated storage and supports assessment and allocation workflow. It is deployed at Nodes and being developed by QCIF. o Monitoring/Reporting system being deployed by Nodes to facilitate their information gathering and reporting on RDSI funded storage to the RDSI project Office. o Security policies and guidelines developed and available for Nodes to use as needed. o DaShNet project funded by NRN providing reliable high speed broadband connections to RDSI Nodes to enable researchers across Australia to access and use the data held at RDSI Nodes. Provide access to a public cloud infrastructure provider to support collaboration around research data. This activity has been contracted to AARNet. o Test platform run by Nodes to provide a test environment for the DaSh technical architecture and its sub-projects. o Created support materials - Nodes to create the resources needed to allow the effective future support of RDSI Node services. This activity has been funded by RDSI and achieved through the AeRO support project and the outcome is available at o o RDSI has provided a report on CKAN evaluation to Nodes and implementation is a matter for individual Nodes to decide. In consultation with Nodes identified additional activities required to support data access. The RDSI board approved the following activities to be carried out across ABPs periods: Programmatic Data Access Service: Tool to enable remote users to discover and access data collections via network-based data services. This activity will be allocated $399,164 and shall be managed by NCI with appropriate reporting to the RDSI project Office. Researcher file syncing and sharing system: Software to allow researchers using RDSI storage to sync and share files in a similar fashion to Dropbox. This activity will be allocated $25,000 for the development of a funding proposal. QCIF will manage the project with appropriate reporting to the RDSI project Office. A further $350,000 is reserved for this activity, however the board will decide on the amount released. LiveARC meets Aspera: Integrate Aspera with LiveARC to seamlessly exploit the benefits of both tools. The proposal will be allocated $54,625 and will be delivered by Architecta/SGI with support from Nodes. LiveARC extended licence: LiveARC is a single platform that enables researchers to manage supported data. LiveARC is a key component of the DaSh Technical Architecture and will be allocated $170,436. These funds will RDSI ABP Approved by the Department of Education Page 9 of 67

10 be used to grant Nodes with an extended LiveARC licence which incorporates an additional year of maintenance. Vendor Panel (VePa): A programme to establish a vendor panel for use by RDSI and the sector. The Vendor Panel was developed in conjunction with CAUDIT and will continue to be maintained by CAUDIT after the RDSI project concludes NoDe Programme The NoDe programme has identified, strengthened and developed research data centres able to hold and process high data volumes. A total of six primary Nodes were approved in Brisbane, Sydney, Canberra, Melbourne, Adelaide and Perth and two additional Nodes were approved in Townsville and Hobart. All eight Nodes have executed contracts with UQ. All Nodes are operational and approving collections for storage allocation, ingesting data and providing data to researchers. The status of Nodes is constantly updated and further details of each operator can be viewed on the Node Statuses page at the RDSI web site ( While there were delays on Nodes capability development these were due to actions by Nodes intended to ensure RDSI infrastructure is co-located with other data infrastructure to ensure researchers have the best possible environment to work with. Currently over 33 PB of collections have been approved by the Nodes Merit allocation committees with over 9 PB of data available to researchers and this number is growing on a daily basis. In addition if the large storage collections are approved this amount of storage will increase significantly. There is a chance that the demand for storage may exceed the RDSI funds envelope. It is anticipated that all Nodes will be reporting on their Node Asset Registry by August The NoDe programme has achieved its objectives in with all milestones completed and continues to carry out the on-going activities. For details refer to NoDe programme activities proposed for are available at section ReDS Programme The ReDS Programme through Nodes has identified research data holdings of lasting value and importance and contributed funding to their development at the most appropriate Node. Under the ReDS I Initial funding variation to the sub-contract with each Node, the RDSI ReDS programme has invested $1m for required storage for a collection Merit Allocated Storage (MAS) and $200K for collection development storage (CD) at each Primary Node. The programme has also invested $300K for required storage for a collection Merit Allocated Storage and $60K for collection development storage at each Additional Node. ReDS I approach was successful and the Department has approved that a similar Interim funding approach be used again. As this approach will be carried out across ABPs periods its details along with the RDSI Allocation process for large collections and the ongoing ReDS top-ups are described at section 3.2. It is anticipated that it will be appropriate and necessary to consult the Department on further activities conducted under the ReDS Programme which are not explicitly addressed in the Annual Business Plan. RDSI ABP Approved by the Department of Education Page 10 of 67

11 The ReDS programme has achieved its objectives in with all milestones complete or on target. For details refer to DaSh Programme The DaSh programme was designed to develop the DaShNet Network Architecture and the DaSh Technical Framework in consultation with stakeholders and propose an approach for RDSI-funded Nodes to use as basis to develop their infrastructure. Together, DaShNet and the Technical Framework were integrated to become the DaSh Technical Architecture which describes the network, data mover, security and identity, data access, cloud gateway, test platform and workflow automation capabilities for the RDSI-funded Nodes. Creating support materials for identified services was also considered a priority and the RDSI project identified the ability to address this through contribution from the AeRO Support Project. The majority of the DaSh activities listed in the milestones ( were either completed or are in progress. Seven activities missed the deadline and are in progress. The data mover activity has been incorporated into the DaShNet project and is progressing through AARNet. The identity system prototype was completed in December 2013 with the interim production deployment expected to be available in May The initial phase of ReX development is expected to be completed by July Additional features may be identified by the Nodes which will be considered. The identification of which cloud middleware options will be implemented by Nodes may be delayed. The implementation of a collections registry accessible by Nodes and RDSI Project called Allocation Request Management System (ARMS) has been developed and phase 1 is available to Nodes at Further development is expected to be completed in Q2/2014. The support of the implementation of widely discoverable data collections prototype is expected to be complete by Q2/2014. Additional activities to support the DaSh programme have been identified by Nodes and are detailed at section 3.3 to be carried out in Vendor Panel The Vendor Panel (VePa) programme was designed to provide options for potential RDSI Nodes and members of the Council of Australian University Directors of Information Technology (CAUDIT) to procure storage related infrastructure, software and services. The purposes for the programme were twofold. Firstly to allow Nodes, universities and other authorised users to avoid lengthy tendering processes by using an appropriately constructed panel and secondly to support volume pricing across Nodes and the wider Higher Education and Research Sector. A wide range of infrastructure and services are included on the panel and each Vendor has agreed to a Commonwealth Government Information Technology and Communications (GITC) compliant model contract (or its equivalent) which will significantly streamline the use of the panel across the university and research sector. The ongoing VePa RFP has been launched, vendors are able to submit their applications at any time, and the evaluation committee assesses on a need basis. This facilitates the entrance to the Vendor Panel.) Currently there are 18 vendors on the panel. The VePa programme has achieved all its objectives in with all milestones either completed or on-going. For details refer to VePa Programme activities for are available at section 3.4. RDSI ABP Approved by the Department of Education Page 11 of 67

12 1.2.2 Risk Factors Assessment and Mitigation The project risks and mitigation strategies identified in the Project Plan are broadly unchanged. RDSI carries out regular reviews of project risks with the RDSI Board and no risks were identified that weren t mitigated. The project in consultation with the Nodes identified the need for funding large storage allocation. The RDSI Research Allocation Panel (RAP) will provide their recommendation to the board for consideration and it is likely that the demand for storage will be larger than the RDSI ReDS funds envelope. It is likely that the ReDS programme funds will not be sufficient to meet the demands indicated by Nodes. Estimated requests for funding research data storage at Nodes exceeds the capacity to fund such requests by a factor of 1.7. The RDSI Board will provide the Lead Agent with advice on the appropriate requests to fund based on approved principles and objectives defined by the project plan and approved business plans. Reports from Nodes indicate that, at the end of April 2014, a total of Petabytes of Research Data has been approved by Merit Allocation Committees for storage and that approximate Petabytes of this has been ingested into Nodes Stores and is available for research use. RDSI funds the storage of two copies of collection at Nodes. RDSI ABP Approved by the Department of Education Page 12 of 67

13 2. Background Information The Super Science Initiative announced in the 2009 Budget included $50 million for Research Data Storage Infrastructure (RDSI) to enhance distributed data centre development to support nationally significant data assets. The Funding Agreement for the RDSI Project was signed with The University of Queensland (UQ) on 23 December A deed of variation was signed in April The RDSI project is building a national network of distributed data stores, or Nodes to enable research data to be readily accessed, analysed and re-used. The project will allow researchers and institutions to more effectively use, manage, share and preserve much larger amounts of research data and will support a national data environment at a scale that will enable new questions to be asked on topics and at scales not previously possible. This document is the fourth Annual Business Plan for the RDSI project covering the financial year July 2014 to June The benefits from better-managed and more accessible research data are being sought everywhere across the research sector. At the same time, the acceleration in the generation of data is outstripping growth in data storage capacities. The RDSI Project therefore represents a timely investment that is aimed at strengthening Australia s capabilities in data intensive research and data intensive research collaboration. In practical terms, this will be by providing significantly increased capacity for research data holdings; developing and improving associated access and data sharing capabilities; and enhancing the capability and capacity to provide research data services to the sector. The result is intended to be a data storage framework that will assist institutions and researchers to more effectively use, manage, share and preserve much larger holdings of research data. The expected benefits are to: improve the availability of quality research data for sharing and re-use and, as a result, expand the scale and scope of problems that Australian researchers may seek to address; improve research efficiency; and reduce institutional data storage costs and enable more extensive collaboration. The infrastructure may also assist institutions to: sustain a quality of research in the digital age that includes the reproducibility of results; meet the storage requirements of key research activities undertaken at that institution; and comply with the research data provisions of Universities Australia s Australian Code for the Responsible Conduct of Research. Other related projects funded as part of the Super Science Initiative include investments in the improved management of data; a significant increase in the supply of high performance computing; enhancements to research tools and improved support for research applications; improved connectivity to the Australian Research and Education Network (AREN); and a range of developments specific to discipline or capability requirements. The intended role of the RDSI Project within this broader set of investments is to increase holdings of research data valued by the research community. Key components of this role will be to support governance arrangements that underpin ingest regimes, data quality and access policies and support access services required by those data collections. The implementation of the RDSI Project will also RDSI ABP Approved by the Department of Education Page 13 of 67

14 enable the provision of such broader data sharing, movement and access services as are required at a national level to support data intensive research and research collaboration that are not otherwise available. Simultaneously, public infrastructure service providers are developing new approaches to data infrastructure. The outcomes of the RDSI Project would be positioned to assist the research sector pursue the economies of scale available from these new approaches. 2.1 Objectives of the Project The aim of the RDSI Project is that researchers will be able to use and manipulate significant collections of data that were previously either unavailable or difficult to access and that there will be a consistent means of accessing this data. This will be achieved through the development of a national network of a limited number of data stores, or Nodes, where the content held in those stores is allocated capacity through a priority and merit process and where that content can be readily accessed, analysed and reused in a coherently governed environment. There may be more than one category of Node. The Project will be realised through the creation and development of data storage infrastructure accessed through a common infrastructure layer and provided by agencies within the sector, or commercial providers, or both. Specifically, the intention is to increase the sharing and re-use of research related data, which might be sourced from any sector, and the collaborative activity based on the sharing of data, and not merely to achieve improved data retention. A national data environment of the scale planned will also enable new questions to be asked on topics and at scales not previously possible. The data storage infrastructure created by the RDSI Project will be connected to the AREN by a high bandwidth connection, funded and constructed under the Super Science National Research Networks (NRN) Project. This will include dedicated high speed connections between major Nodes. The resulting data environment is not intended to be exclusive to the use of RDSI Project stakeholders. Rather, the delivery of the RDSI Project is expected to contribute to the development of an open infrastructure that institutions, disciplines and research facilities can expand or extend to meet other requirements and into which other sector interests can add capability over time. 2.2 Outcomes of the Project The RDSI Project will deliver a national infrastructure able to hold significant collections of research data. That infrastructure will comprise: A small number of high capacity Nodes where each Node will contain multiple petabytes of storage configured to support the classes of access and retention appropriate to the research data held; A dedicated high speed inter-connection network with high bandwidth low latency connections that will support replication and data transfer between Nodes; A common access infrastructure to provide a uniform experience for common researcher access to all the data held; and Appropriate specialist access infrastructure hosting specialist access tools appropriate to the disciplines closely related to these national collections. As a result of this infrastructure: RDSI ABP Approved by the Department of Education Page 14 of 67

15 Researchers will be able to use and manipulate significant collections of data that were previously either unavailable to them or difficult for them to access; Research outcomes will be improved as a result of improved access to this data; Researchers will be able to access the data in a consistent manner which will support a general interface as well as discipline specific access; Researchers will be able to use the consistent interface established by this project for access to data collections at participating institutions and other locations as well as data held at the Nodes; In arriving at this infrastructure, the RDSI Project will leverage its purchase power to also develop a panel of vendors to help reduce storage costs across the Research and Education sector. 2.3 Implementation Principles Accordingly, the development and operation of the data storage infrastructure will be guided by the following principles: The RDSI project will provide data storage capabilities for identified research data holdings of lasting value and importance. The primary purposes should be to support collaboration around data and the development and use of research data collections. The RDSI should be managed to create a coherent national research data environment. The RDSI should hold research data, sourced from Australia and overseas, including from publicly funded research organisations and other sources relevant to research. The data held should be data intended for collaborative research purposes and should be accessible, for the purposes of research, subject to appropriate access rules. The quality of service should be appropriate to research needs. Access controls should support content providers in managing their own access rules. Overall, the implementation of the RDSI Project must establish, operate and provide access to Project infrastructure in accordance with the EIF principles, listed below: Principle 1: Principle 2: Principle 3: Projects should address national infrastructure priorities Projects should demonstrate high benefits and effective use of resources Projects should efficiently address infrastructure needs Principle 4: Projects should demonstrate they achieve established standards in implementation and management. The organisations participating in the Project must endeavour to establish, operate and/or provide access to the facilities in a manner which: takes into account the long-term strategic requirements of relevant research disciplines; enhances national and international collaboration in research; enhances research capability for relevant disciplines; provides for merit-based access to the facilities in accordance with the norms and expectations of the research community; has a strong emphasis on service provision to the research community; and has a strong emphasis on the effective use of data and compute resources RDSI ABP Approved by the Department of Education Page 15 of 67

16 2.4 Context and Scope of the Project Context The implementation of the RDSI Project will occur within a complex environment with competing sector, discipline and institutional priorities and varying technological requirements. The implementation of the RDSI Project will seek to realise economies of scale in a national data management and access system. While the data storage infrastructure may be composed of a combination of technologies from a number of suppliers, it is intended that all parts of it should be capable of being accessed through a common interface. The resulting data environment should support specific identified data needs, but also be easily augmented by institutions, disciplines and research facilities, on a needs or interest basis, at an incremental cost. Some of the drivers underpinning these objectives are as follows: At a very broad level, data intensive research and data re-use are important contributors to innovation excellence. In addition, improved data re-use also assists effective collaboration, providing a second contribution to innovation excellence and therefore future prosperity. At a research sector level, the growing requirement to manage research data better and to more easily support collaboration is ubiquitous. However, for the first time, the acceleration in the generation of data is outstripping growth in data storage capacities. Over time, the research sector will therefore need to devote more resources to the management of data. Experience from the commercial sector shows that very significant economies of scale can be achieved, using highly sophisticated implementation strategies, while providing accessible data storage and data analysis capabilities. Public and private storage providers and storage infrastructure providers are developing new approaches to data infrastructure that could be adapted to the advantage of the research sector. At the discipline level, developments in data gathering and data management also demonstrate a trend towards the development and adoption of shared global standards and open access protocols, enabling the re-use of data to become increasingly efficient and costeffective over time. The vast bulk of the research data that will need to be managed in 2015 has not yet been generated. Putting in place the foundations to better manage data in the future is an urgent requirement with potentially significant benefits. RDSI ABP Approved by the Department of Education Page 16 of 67

17 2.4.2 Scope The Project will undertake activities across four key programme areas as follows: Programme Area Node Development (NoDe) Research Data Services (ReDS) Data Sharing Services (DaSh) Vendor Panel (VePa) Scope To assist the development of research data storage services and their supporting infrastructure. To accelerate the aggregation of data and the co-location of data with analysis capabilities. To provide a uniform data access, sharing and movement capability commensurate with the importance, volume and complexity of the large volumes of data the RDSI will hold. A programme to establish a vendor panel for use by RDSI and the sector. The Project will ensure that: the needs and expectations of institutions are factored into overall strategy-setting, decisionmaking and management processes; services enabled by the Project s infrastructure are tailored to meet the specific needs of different research contexts; co-investors in the Project realise improved services for the researchers they support; the performance of the infrastructure is measured against improvements to research outcomes; Intellectual Property and other rights are managed in a way that permits the re-deployment of data from one jurisdiction to another, whenever possible; and Infrastructure is developed and installed for operation in enduring organisations, being the University of Queensland and the Project Sub-Contractors, as a basis for sustained operation of the Nodes. The scope of the Project will also account for the growing role of commercial providers, including public storage providers. It is likely that from time to time, commercial services will arise that can support or more effectively provide elements of the RDSI functionality. The Project will therefore ensure that investments are managed in a way that allows commercial services and software tools to be evaluated and adopted when appropriate over time. RDSI ABP Approved by the Department of Education Page 17 of 67

18 2.5 Participating Organisations Key participating organisations and their roles and responsibilities are described in the Table below. This list will be updated as necessary in Annual Business Plans. Organisation The University of Queensland (UQ) Node Operators (sub-contracting to UQ) Affiliates (will subcontract to UQ if receiving funding) Role and Responsibilities Signatory to the funding agreement with the Commonwealth Direct recipient of EIF Funds Responsible for managing distribution of EIF funds through subcontracting arrangements Responsible for meeting Commonwealth reporting and accountability requirements Responsible for overall management of the Project Establishes the RDSI Project Board and the Infrastructure Advisory Panel and Resource Allocation Panel Appoints the Project Director and Board Chair with the approval of the Department Appoints such support staff as agreed with the Department Node Operators are those organisations that agree to operate storage systems to support the RDSI services, and to make them available for open research access Node Operators will assist the University of Queensland in identifying important data assets for capacity allocations across the RDSI Nodes Node Operators will agree to allow and to support access to RDSI funded facilities needed by data sharing services funded under the DaSh programme, as determined by the RDSI Project Board. Affiliated operators are those organisations that either directly federate their own storage services with the RDSI DaSh Data Sharing Service or agree to allow access to their own data through a DaSh access mechanism. The DaShNet data transfer mechanism removed the need for Affiliates. The RDSI Project Infrastructure will be developed and implemented by the University of Queensland and other organisations acting as Sub-Contractors. All Node Operators funded by the RDSI project must be Sub-contractors to UQ. The RDSI Project Board will recommend all Sub-Contractors to the University of Queensland. UQ will seek agreement from the Department with sufficient advance notice that each Sub-Contractor is acceptable. This approval will be obtained before the University of Queensland enters into contractual arrangements with Sub-Contractors for the purposes of this Project. RDSI ABP Approved by the Department of Education Page 18 of 67

19 2.6 Management and implementation arrangements The University of Queensland will have overall responsibility for the Project in accordance with the reporting and accountability requirements outlined in the RDSI Funding Agreement. The Project will be managed through the arrangements described in section 4.1. The RDSI Project Board provides overall guidance and recommendations to the University of Queensland and the Project Director concerning the Project and will assist the Project Director refine and adjust the specifics of the activities outlined in the Project Plan. No other activity is undertaken without the prior written agreement of the Department. The University of Queensland sub-contracts with organisations to establish, operate and/or provide access to Project infrastructure as defined in the Project Plan. These Sub-Contractors must be agreed by the Department before the Sub-Contractors work may commence. The University of Queensland s sub-contract with a Sub-Contractor, where appropriate, require the Sub-Contractor to act in accordance with the principles set out above. Where relevant, the University of Queensland may identify a Sub-Contractor for a Sub-Project involving multiple organisations. The Sub-Contractor of such Sub-Projects may be contracted to manage and report on the entire Sub-Project and all funding and infrastructure associated with that Sub-Project. Each Sub-Contractor (Node) is required to ensure that the Project infrastructure operates for an agreed period determined on a case by case basis, but in all cases, extending at least to June Relationship to other eresearch investments The implementation of the RDSI Project will foster and leverage key linkages across other eresearch investments under the National Collaborative Research Infrastructure Strategy (NCRIS) Programme and the Super Science Initiative, including: NeCTAR The RDSI Project will rely on the parallel development of the NeCTAR Project funded under Super Science, as it is expected that NeCTAR s functions and capabilities will be more readily achieved by colocating significant components of its infrastructure with RDSI Nodes. ANDS The RDSI Project will support the processes and frameworks developed by the Australian National Data Service (ANDS) and will ensure alignment between the delivery of the RDSI Nodes and the use of ANDS infrastructure. NRN The National Research Network (NRN) Project, funded under Super Science, specifically includes provision for the high-bandwidth connection of data stores, represented by the RDSI Nodes, under this Project. It is expected that the NRN Project will respond appropriately to the identified needs under the RDSI. RDSI ABP Approved by the Department of Education Page 19 of 67

20 AAF The Australian Access Federation (AAF) provides federated identity services to its members who are universities and research organisations throughout Australia and New Zealand. The RDSI project will rely on the AAF for federated identity management. Rationale The rationale for this relationship between the projects is that research collaboration fundamentally depends on the easy sharing of content, and the RDSI Project will provide a foundational storage solution. This storage solution will operate with high reliability, making it likely to be more efficient to co-locate relevant services with RDSI Nodes, than to develop separate high reliability sites. This colocation would also reduce network costs associated with moving large volumes of data for analysis and re-use. It is important that RDSI arrangements ensure that appropriate research infrastructure services, including the eresearch services of NeCTAR and ANDS, also integrate fully with RDSI infrastructure. The RDSI project will therefore establish working relationships with the NeCTAR, NRN, AAF and ANDS Projects. 2.8 Funding arrangements The total Commonwealth funding for the Project was $50 million over three years: $24 million in , $13 million in and $13 million in , with the project variation these figures have been adjusted to fulfil the project objectives until the end of Funding is being provided from EIF under the Nation-building Funds Act 2008, which has the authority to fund the creation and development of research infrastructure. Funding is provided through the Funding Agreement signed between the University of Queensland and the Department signed on 23 December The Commonwealth funding is intended to support the development, deployment and commissioning of facilities and infrastructure by the Project, including agreed project management costs. The University of Queensland and its Sub-Contractors must meet the associated operational costs of those facilities and infrastructure including the employment of appropriate staff to manage and sustain them. Where specific Project infrastructure is to be developed or delivered by the University of Queensland, the approval, reporting and performance management processes will be the same as for any other party delivering Project infrastructure, except that no sub-contract will be required. RDSI ABP Approved by the Department of Education Page 20 of 67