California Digital Library Computing and Storage Resource Center. Infrastructure Description

Transcription

1 California Digital Library Computing and Storage Resource Center Infrastructure Description Version 2.0 September 26, 2008

2 Table of Contents 1 Overview Introduction Purpose of this Document Topology CSRC Environments Production Environment Replication and Disaster Recovery Environment Development and Staging Environment Departmental Computing Environment Oversight CSRC Distribution and Partners IR&C Kaiser Data Center UC Berkeley Hearst Data Center Distribution of Environments Interconnection of Sites Roles and Responsibilities CDL s Internal Organization Shared Responsibilities Future planning & developments IT Asset and Service Management Storage Next Generation Melvyl/WorldCat Local Google Book Digitization Web Archiving Service... 11

3 1 Overview 1.1 Introduction Over the past several years, the California Digital Library (CDL) has constructed the Computing and Storage Resource Center (CSRC). The CSRC is not a single facility, but rather a distributed network of CDL-owned resources, comprising several logical environments at three physical locations. It is a large-scale facility designed to support CDL s growing portfolio of critical and complex applications systems serving the UC academic community. It provides high-availability computing, database and storage resources for these applications, as well as environments for the development of new CDL applications, the testing of CDL and vendor applications and the operation of CDL s departmental applications. The CSRC is supported through service agreements between CDL and strategic partners who provide data center infrastructure and varying degrees of operations support. Specific responsibilities for planning, acquisition, physical hosting, operation and monitoring of the CSRC are shared between CDL personnel and those of CDL s strategic partners. CDL personnel have overall responsibility across all environments and physical locations, and varying levels of responsibility for installation, administration, operation and monitoring at the various sites. 1.2 Purpose of this Document This document describes the CSRC in terms of its current structure, components and scope; describes in detail the roles and responsibilities of CDL and its partners in its and operation; and highlights future CDL projects for which CSRC capacity planning is underway. 2 Topology 2.1 CSRC Environments The CSRC is divided into four discrete environments: Production Replication and Disaster Recovery Development and Staging Departmental Computing Production Environment The CDL production environment consists of servers, databases, programming language compilers and interpreters, storage hardware and software, network components, firewalls, monitoring systems, backup and recovery systems and 1

4 high availability systems. The environment supports CDL applications accessed directly by the UC academic community, primarily students and researchers. The environment also supports ongoing ingest into a massive digital preservation repository that serves the UC Libraries. Most of CDL s applications are interactive and web-based; and require 24x7 availability. Servers The following table summarizes the CDL servers that currently exist in the production environment: Mfr/OS/Version - names Model # Servers # Processors Status/Primary role Sun Solaris 9 clio, thalia V Active/Database cluster Sun Solaris 9 aretha, elektra, selene V (12 dual core) Active/ Preservation cluster Sun Solaris 10 zeus, bastet, leto V (20 dual core) Active/Bib Services cluster Sun Solaris 10 rhea M (2 dual core) Active/Bib Services cluster Sun Solaris 10 emu kiwi V (8 dual core) Active/Publishing; Digital special Collections; Web Production Sun Solaris 10 (the birds) x Active/Preservation; Mass digitization Total Storage The following table summarizes CDL storage arrays existing in the production environment: Mfr/Type Model Type Storage Status # Arrays Amt Sun Fibre Channel 15KRPM StorEdge 6130 Tier 1 6TB Active 1 Sun SATA 7.2KRPM StorEdge 6130 Tier 2 22TB Active Sun Fibre Channel 15KRPM StorEdge 6140 Tier TB Active 2 Sun SATA 7.2KRPM StorEdge 6140 Tier TB Active IBM Fibre Channel 8100 Tier 1 1(shared) 3TB Active Sun SATA 7.2KRPM StorEdge 3511 Tier TB Active Total TB Databases The following table summarizes CDL databases that currently exist in the production environment: Vendor Version # Databases Total Size Status Oracle 9i GB Active 2

5 Oracle 10g GB Active MySQL ~8 200 GB Active Total 8 ~1237GB These databases support the following production applications: Melvyl, uc-elinks, VD, IPAddress Database, Luna Imageservice, Publishing Program, Digital Preservation Repository, Web Archiving Service; and the Mass Digitization Metadata Database Replication and Disaster Recovery Environment This environment is currently under design. It will support of remote copies of content and standby instances of all high-priority applications, as determined by policy. Current planning includes consideration of leveraging the stage instances (see below) for disaster recovery Development and Staging Environment The development environment provides resources to CDL developers for building and enhancing applications, for constructing prototypes and for general experimentation. The staging environment provides a close-to-production hardware and software environment for testing new or modified CDL applications or new or modified vendor applications; within the same high availability failover framework as production. The development and staging environment is also used for conducting formal application pilots (often with participants from multiple UC campuses). Servers Mfr/OS/Version/names Model # Servers # Processors Status Sun Solaris 9 (old cassatt) V Pending Sun Solaris 9 hopper, clio V (6 dual core) Active erato Sun Solaris 10 escher, blake, V (12 dual core) Active klee Sun Solaris 10 hokusai, V (4 dual core) Active kandinsky Sun Solaris 10 leonardo, davinci V (16 dual core) Active/VCS staging cluster Sun Solaris 10 matisse V (2 dual core) Active/VCS staging cluster Sun Solaris 10, dp01 dp Active Linux SUSE: ingest1, ingest2, misc 5 ~10 Active wiki, rec-proto Linux Debian: vzhw02 Generic 1 2 Active Linux ES 9 alapay, voulkos misc 2 ~4 Retirements pending Capricorn Red Hat Petabox 1 (cluster) 40 Retirement pending Total

6 Storage Mfr/Type Model Type Storage Status # Arrays Amt Sun Fibre Channel 15KRPM StorEdge 6130 Tier TB Active 1 Sun SATA 7.2KRPM StorEdge 6130 Tier 2 20TB Active Sun Fibre Channel 15KRPM StorEdge 6130 Tier 1 Coming?TB 1 Online Sun SATA 7.2KRPM StorEdge 6130 Tier 2?TB Active Sun SATA 7.2KRPM StorEdge 3511 Tier 2 Coming 5 147TB Online Capricorn Petabox Tier 3 1 Retire. 64TB (cluster) pending Total TB Databases Vendor Version # Databases Total Size Status/Purpose Oracle 9i 4 Active/ Melvyl Oracle 10g 2 Active/VD MySQL 4.x Active/DPR, Luna, ~6 SF, IPAddress MySQL 5.x ~8 Active/WAS, MDID Total ~20 Databases in the development and staging environment support the following production applications: Melvyl, uc-elinks, VD, IPAddress Database, Luna Imageservice, Publishing Program, Digital Preservation Repository, Web Archiving Service; and the Mass Digitization Metadata Database. In addition, several experimental, prototype or potential production applications have databases in the development and staging environment: these have included American West, FindIt, IP Address Authorization, Mark Twain Papers, Web Archiving Service, Melvyl Recommender, ERMS Departmental Computing Environment This environment supports internal CDL computing by providing production database resources to internal CDL applications, file sharing and workgroup utilities to CDL staff such as CVS, and resources for Infrastructure (e.g. jumpstart servers). Servers Internal CDL applications are housed on servers in the production environment. File sharing is deployed on the servers listed below: Mfr/OS/Version Model # Servers # Processors Status 4

7 Sun Solaris 10 cassatt M (2 dual core) Active/Internal operations Sun SUSE Linux ES 9 oak V40z 1 2 (1 dual core; ) Active/Internal operations Sun SUSE Linux ES 9 junco x Active/Internal operations Apple Mac OS Server 10.4 diva serve G5 2 2 Retirement Pending Windows IIS VM wints1 VMware 1 2 (1 dual core) Active Windows IIS VM newdiva VMWare 1 2 (1 dual core) Pending Total 7 14 Storage Department wide file sharing is deployed on internal storage of the above servers, Mfr/Type Model # Arrays Storage Amt Status Apple serve G5 2 Retirement 1376 GB (servers) Pending Windows IIS VM newdiva VMWare IR&C SAN 1000 GB Pending Total GB Databases Vendor Version # Databases Total Size Status MySQL 4.x & 5.x ~13 Active Total 13 Databases in this environment include: GrantManager, Timesheet, Shared Content and shared content license tracking databases. 3 Oversight 3.1 CSRC Distribution and Partners The four CDL computing environments are distributed across the following physical locations: IR&C Data Center (Kaiser Center in Oakland) UC Berkeley Hearst Data Center (Berkeley) IR&C Kaiser Data Center The IR&C Kaiser Data Center, located on the third floor of the Kaiser Center in Oakland, is operated by the Information Resources & Communications (IR&C) unit of UC Office of the President. CDL s partnership with IR&C is governed by a Service Level Agreement (reviewed annually) and includes services provided by: Unix Services 5

8 Operations Network Services UC Berkeley Hearst Data Center The UC Berkeley Hearst Data Center, located at 2195 Hearst Street in Berkeley, is operated by UC Berkeley s Information Systems and Technology (IST) department. CDL s partnership with IST is governed by a Service Level Agreement (reviewed annually) and includes services provided by:: Data Center Operations, including UCBackup Network Operations Distribution of Environments The following table illustrates how the four CDL computing environments are distributed across these three physical locations: Computing Environment IR&C Kaiser Data Center UCB Hearst Data Center Production Replication and Disaster Recovery Development and Staging * Departmental Computing * CDL is in the process of moving all of its development and staging resources to the UCB data center; see the attached CSRC Map Interconnection of Sites The three physical locations are connected with high-speed Ethernet. All servers are equipped with multiple 1gigabit ports, with the exception of several older servers which are in the process of being phased out, and interconnections between sites can support even higher transmission rates in the future. 3.2 Roles and Responsibilities As stated in the introduction above, responsibilities for planning, acquisition, physical hosting, operation and monitoring of the CSRC are shared between CDL personnel and CDL s partners. CDL personnel have overall responsibility across all environments and physical locations, and varying levels of responsibility for installation, administration, operation and monitoring at the various environments and locations CDLʼs Internal Organization The CDL organization is comprised of a series of targeted programs, each having its own director, project team and technical team; and a series of service groups that provide various types of services to these programs. Among the service groups is the Infrastructure & Applications Support Group, which is responsible for 6

9 providing computing and storage resources to the programs through of the center. Program Technical Teams Each CDL program has a technical team consisting of a Technical Lead and one or more Programmer/Analysts. Each team is responsible for applications that support the program. Some applications are vendor-developed and some are developed by the team or through collaboration among multiple programs teams. Infrastructure & Applications Support Group The Infrastructure & Applications Support Group in CDL is charged with overall of the CSRC. Currently, the group consists of: A Group Manager A Lead Architect (proposed) Two Systems Architects (1 permanent; 1 staff contract A Database Architect A Systems Analyst The group is solely responsible for the following aspects of the center: Developing the overall integration architecture for computing systems, database systems, storage systems and network infrastructure Planning for hardware and software purchases to meet the requirements of CDL s programs, in accordance with the integration architecture Evaluating hardware and software products Executing purchases Administering databases (see below) Providing technical expertise to CDL s applications developers and applications integrators in specialized areas, such as database and storage interfaces Note that CDL s Infrastructure & Applications Support Group will at times collaborate with partners in areas of common interest (e.g.,. evaluation of storage hardware) but that CDL retains decision making responsibility for CDL s computing domain in these cases. A number of strategic CDL applications, including the Melvyl online catalog and the Request inter-library loan system, rely on the Oracle database system. Other strategic applications, both vendor-written and internally-developed, use MySQL, an open-source database system. Because these DBMS products are not supported by our partners, the I&AS group is responsible for all aspects of their administration Shared Responsibilities The following matrices indicate how responsibilities for CDL s infrastructure and applications are shared among CDL s Infrastructure & Applications Support Group, CDL s Program Technical Teams and CDL s partners across environments and locations. 7

10 Production Environment at IR&C Data Center CDL IR& C Responsibility Unix Operations Network I&AS Prog Services Services Server and storage hardware installation Storage provisioning Storage allocation and (note 1) Server provisioning (note 1) Server user/group administration (note 1) Server and storage consultation Common CDL software HA (VCS) Configuration Database software installation Database provisioning, tuning Database maintenance Database integrity and availability Database monitoring Database consultation Database ad hoc queries Network installation, configuration IP address, host name Application backup Application maintenance Application monitoring Infrastructure monitoring Application continuity planning Application continuity implementation Application problem resolution Application statistics gathering Note 1: CDL s I&AS Group is currently collaborating with IR&C s Unix Services Group in an effort to explore server virtualization and software that would enable more efficient deployment of (Linux) servers in the production environments. Development/Staging Environment at UCB IST Data Center CDL IR& C UCB IST Responsibility Unix Operations Net Ops I&AS Prog Services Server and storage hardware installation Storage provisioning Storage allocation and 8

11 Server provisioning Note 1 Note 1 Server user/group administration Common CDL software HA (VCS) Configuration Database software installation Database provisioning, tuning Database maintenance Database integrity and availability Database monitoring Database consultation Database ad hoc queries Network installation, configuration IP address, host name Application unit testing Application quality assurance Infrastructure monitoring Application problem resolution Note 1: CDL s I&AS Group is currently collaborating with IR&C s Unix Services Group in an effort to explore server virtualization and software that would enable more efficient deployment of (Linux) servers in the production environments. If successful, this VM strategy would have to be replicated in the development and staging environments, potentially in partnership with UCB s IST. Departmental Computing Environment Responsibility Server and storage hardware installation VM Server provisioning Storage provisioning Storage allocation and Server provisioning Server user/group administration Network installation, configuration IP address, host name CDL I&AS Prog Unix group IR& C Windows TechDesk Engineering Network Systems 9

12 Infrastructure monitoring Desktop Support 4 Future planning & developments A number of projects that are planned or underway will impact the CDL CSRC. The most significant of these are discussed below. 4.1 IT Asset and Service Management IAS is beginning explorations of more structured approaches to IT infrastructure and service, including appropriate adoption of frameworks such as the Information Technology Infrastructure Library (ITIL) of the ISO 2000 framework. It is hypothesized that adoption of these frameworks would lead to better capacity planning, documentation, change, and policy compliance. 4.2 Storage Several initiatives involving the distribution of CDL storage resources beyond the current environmental boundaries. IAS is working particularly closely with the Preservation program on these issues. Possible strategies include: Establishment of additional locations for storage, including gold level storage support from UCB and the San Diego Supercomputing Center. Experiments with denser storage such as Multiple Arrays of Idle Disks (aka MAID) Taking on the role of storage provider for campus researchers or other parts of the UC academic community Establishing partnerships with external storage providers, including other universities and commercial providers Sharing of storage, database, network and other expertise with campus or external partners 4.3 Next Generation Melvyl/WorldCat Local The University of California (UC) Libraries are collaborating with the Online Computer Library Center (OCLC) to create a pilot version of a replacement for the current Melvyl Catalog, which contains records for library holdings at all ten UC campuses and is one of the largest databases and application installations of its kind. The pilot will begin in April 2008 and will last at least six months. Following the completion of the pilot, a decision will be made about whether to move forward with it or to pursue other options. A move forward will have a large impact on the CDL Infrastructure components that support the current Melvyl catalog. 10

13 4.4 Google Book Digitization UC has entered into an agreement with Google, Inc. under which Google will digitize approximately 5 million books held in UC libraries, over the next 5 years. The University has chosen to preserve a master copy of all digitized books at the CSRC. It is projected that approximately 600 TB of SATA or other second tier storage will be needed to store the primary master copy of this content, and an additional 400 TB for a secondary copy. A new database will be required to hold administrative and descriptive metadata and 12 processors will be required for processing of the content. 4.5 Web Archiving Service Roll-out of the Web Archiving Service, a service for crawling and preserving web content (now in development), is scheduled for Fall This will require 15 processors, mostly dedicated to running web crawls, and an estimated 500 GB of storage per year in the production and replication environments. 11