CDL Database Administration Framework v. 1.1 August 2009

Transcription

1 CDL Database Administration Framework v. 1.1 August 2009 Contents Purpose of the framework 1 Terminology 1 Database administration conventions 1 Summary Table 1 Database design, location, and naming 2 Default database configuration 3 DBMS version control, upgrades, and patching 5 Maintenance procedures and windows 6 Capacity planning 7 Security 7 Extant database environment 8 Change & review history: 1. Nov. 2008: Framework outline endorsed in principle by and comments from TechCouncil, et al. 2. March 19, 2009: First draft distributed to TechCouncil & DirCab for comment 3. March 27, 2009: Summary of conventions; extant environment tables complete 4. April 2, 2009: First draft feedback due 5. August 18, 2009 Feedback incorporated; shared with TechCouncil for final review & endorsement

2 Purpose of the framework: 1. To describe the conventions for CDL s database infrastructure, including: a. Design, configuration, and various other management practices b. responsibilities (as distinct or shared between IAS and program/service area). 2. To document the extant database environment. Note: In this document declarative statements describing the CDL database environment and practices are in regular font. Descriptions of responsibility, including initiation and follow through, are italicized. Terminology Database owner: the CDL program/service representative(s) who has responsibility for the service/application for which the database is a component, or their delegate. An owner s delegate may include a 3 rd party such as a vendor (e.g. as with OCLC for the VDX application; Ex Libris for the SFX and Aleph/Melvyl applications). DBA: the designated database systems administrator(s) [currently Annita Auyang], or her/his designate. Database components: the objects tables, indexes, stored queries comprising an entire database. Database engine: the DBMS software used to create, manage, and use the database. Currently CDL deploys Oracle, and MySQL database engines (see 1.3 below) 1. Host: the physical or virtual host under which a database runs and can be located. Each host will have a name (DNS entry) and an IP. Hosts may run one or more applications and/or databases. IAS: Infrastructure and Applications Support; the CDL unit responsible for supplying database systems support and in which the DBA reports. 1. Database administration conventions Summary of explicit conventions (see below for more information) MySQL Oracle Version i; 10g Name variant of app or service name version of app or service name Ports ; Users <dbname>ro Sys, system, user accounts <dbname>rw Password see DBA or IASmgr see DBA or IAS mgr Virtual hostname varies varies of db server Data filesystem name /database/mysql /database2/mysql /db,/dblog,/dbarlog /vdxdb,/vdxdblog,/vdxdbarlog Backup filesystem name /mydbbk, /mydbbk2 /net/sysad- /dbbk,/arbk /vdxdb/cdlbkdaily, /vdxdblog/cdlbkdaily, 1 The preservation program also uses and assumes responsibility for the Postgres DBMS as part of the Storage Resource Broker (SRB) component of its architecture. Infrastructure & Applications Support - 1 -

3 repo/export/flash_archive/mysql_bck /vdxdbarlog/cdlbkdaily Script names Varies (see database.xls) Varies (see database.xls) 1.1. Database design, location, and naming Initial design To the best of its ability the CDL industry standard or best practices for database design (e.g. fully normalized relational databases). Database owners are responsible for design. The DBA may be consulted for initial design or design improvements. The DBA may require design modifications before creating and populating a production instance of a database Development, stage, production instances Normally, an empty database instance will be created upon request as a development instance and released to the owner for the construction of database components and data population. The owner will inform the DBA when development is complete and request a staging/qa instance of the database. The stage database components can be created as a copy of development or reconstructed by the owner. The owner will inform the DBA when staging/qa is complete, and request a production instance of the data. Database owners are responsible for requesting development, stage, and production instances, and for populating development and stage instances. The DBA is responsible for creation of empty development, stage, and production instances and should be consulted during the population of stage instances. The production database and components must be created by the DBA and populated by, or in coordination with, the DBA Naming and location Location and naming of databases follows conventions which support ease of administration (setup, migration from development to stage to production, backups, performance monitoring, failover, replication, etc.) Database hosts Database hosts are selected/assigned in consultation between CDL programs and IAS. Assignment takes into consideration the administration and cost efficiencies (license and support/maintenance costs) of consolidating databases on servers or clusters dedicated to database hosting, and on application design. Oracle databases are assigned to hosts through consultation between the owner, DBA, and other IAS principals. MySQL development databases are normally hosted on hopper.cdlib.org MySQL stage/qa dbs are normally hosted on asawa.cdlib.org. MySQL production dbs are normally hosted on clio.ucop.edu or thalia.ucop.edu, which are set up for failover from one node to another to provide high availability Database names Infrastructure & Applications Support - 2 -

4 Database names normally are built as semantic indicators of the associated application/service (e.g. dpr for digital preservation repository. In the past the environment was included in the name (e.g. dprstg, or dprprod for stage, or production instances). The current convention is to use the same name in the different environments to ease the replication and management of stored procedures, scripts, etc. across environments. The loss of contextual information when the names are the same ( which environment am I in? ) is offset by database user names and passwords that are specific to the instance. Exceptions to this convention can be requested if necessary Database ports Server ports for MySQL databases are normally drawn from a pool of ports from Ports are selected to avoid conflicts, e.g. when production databases failover from the primary node to the secondary node (see 1.2.4). Where possible going forward, the dev, stage, and production database versions will use the same port, to ease management of the firewall and other configurations across environments. Oracle listener ports are conventionally 1521 or in the range of Ports also play a role in database (and general infrastructure) security in that firewalls protecting all environments must be configured to let network traffic through to only the designated database ports on database servers and, when possible, only from known and trusted origins Default database configuration Users/owners and passwords Normally each database has four associated users: 1. the [database] system/root user (with all privileges for the database instance, not just specific databases within the instance), 2. the read only (ro) user for a database (e.g. in MySQL grant select; execute on stored procedures); 3. the read/write (rw) user (e.g. in MySQL grant select, execute stored procedures, insert, update, delete); the db owner (grant all);. Users with intermediate privileges, e.g. select, insert, update, and delete only can be requested to run applications that need intermediate privileges on a database. Database user names are recorded and maintained by the DBA (see section 2 Extant Database Environment). Database owners are responsible for requesting additional users or privileges. Requests are made to cdldba@ucop.edu which creates a work ticket. The DBA and the IAS Manager know or have access to all database passwords. Passwords will be changed from initial vendor-supplied defaults and will meet CDL security standards. Infrastructure & Applications Support - 3 -

5 Filesystems Database filesystems are established and named to emphasize efficient server, storage, and database management, unless an application requires a particular convention. Filesystems are located on network-mounted storage (primarily Sun SAN storage managed as a ZFS filesystem) Default settings Oracle defaults are prescribed by the related vendor applications. MySQL defaults include: Storage engine: InnoDB Character set: Logging: Backup strategy(ies) By default the CDL backs up database content and configuration information in all environments to aid recovery from unanticipated system problems and database corruption. Where possible the backup approach is designed to recover a database to the state immediately prior to failure. Otherwise the approach attempts to recover a database to known points in time, i.e. in daily, weekly, or quarterly increments. Backup usually is performed in two steps: 1) a disk-based copy is created using cold (db locked or shutdown), hot (db online and usable), or snapshot (storage manager s point-in-time copy) backup strategy; 2) filesystem level backup to tape using Tivoli Storage Manager (TSM) of the disk image produced in step #1. Additional disk copies may be substituted for tape backup when database size makes tape backup impractical (e.g. when creating the tape backup consumes too many TSM resources or more time than available in the backup schedule). The appropriate backup strategy is dependent, among other factors, on the capability of the database engine, the service program s tolerance for database downtime, the size of the database, and the preferred length of time for a restoration when necessary. In combination these factors may suggest database replication to assist with backup goals (see below). The backup approach is designed by the DBA in consultation with the Database owner and tested in development and/or stage before production. For each database, the backup strategy is designed by the DBA in consultation with the service program. It is documented by the DBA (see section 2 Extant Database Environment). The DBA and Database owner consult on testing/practicing restoration, which is encouraged but not required. Database owners and the DBA are mutually responsible for alerting each other if a corruption is detected and a restoration is needed. Requests for restoration or adjustments to the backup strategy are made to cdldba@ucop.edu which creates a work ticket Failover (VCS) set up Infrastructure & Applications Support - 4 -

6 In partnership with UCOP s IR&C and Unix systems units, CDL attempts to place production systems into high availability (aka HA ) status, normally using Veritas Cluster Server (VCS) software. Beginning in April 2009, production databases on the clio/thalia MySQL database servers are under VCS HA control. MySQL Database VCS conventions include: Databases and related resources (e.g. network storage on which they reside) are designated as either critical or non-critical. VCS scripts attempt to bring critical and non-critical resources back online three times before initiating any failover (move to the backup node) action. If non-critical resources cannot be brought online they are left in their faulted state for manual intervention the next business day. If critical resources cannot be brought online after three times, VCS scripts attempt to move all resources on that node (e.g. on clio) to the backup node (e.g. thalia). (This is done because 1) the number of databases involved makes scripting/managing separate failover for each one costly; 2) in many or most cases the underlying cause of failure (e.g. hardware failure of the host server) would require that all resources be failed over anyway.) The DBA works with the Database owner to determine if a database or related resource is critical, i.e. belongs to the pool of HA resources. As part of production implementation, the DBA is responsible for ensuring that a critical database is under HA control, and that HA is working Replication Database replication (a master and synchronized slave ) is available for MySQL databases in stage and production environments. A database replicant is established when necessary to address, in priority order, db activity load balancing (e.g. where querying/reporting would otherwise interfere with additions or updates), and/or disaster recovery, and/or backups; and/or ease of maintenance. The DBA works with the Database owner to determine if replication is needed and viable. The DBA sets up replication Monitoring, tuning, query optimization IAS makes available and the DBA manages tools for monitoring and analyzing performance of databases. MySQL Enterprise Monitor and Oracle Enterprise Manager are currently used. The DBA monitors the databases for high impact errors and error conditions. Database owners are responsible for requesting DBA consultation on performance, including query optimization and, in most cases, implementing associated database changes (see section 1.1) DBMS version control, upgrades, and patching Versions Infrastructure & Applications Support - 5 -

7 The CDL attempts to standardize the DBMS version in use for all databases using the same database engine. Exceptions are made when the DBMS is specified by or bundled into a vendor product. Currently CDL uses MySQL version and Oracle 9.i or 10g. New DBMS engines or upgrades to the in-use standard, and destined for production deployment, are determined on a case-by-case basis balancing the following factors: Availability of informal and formal (i.e. purchased) support Functional improvements and their match to business requirements Administrative considerations including security, staff impact, ease of backups, etc. In consultation with IAS principals and database owners, the DBA recommends and implements new DBMS engines or upgrades to in-use DBMS engines. Implementation schedules are arranged in consultation with all affected parties. Database owners and CDL program developers are responsible for alerting the IAS/DBA to out-of-cycle upgrade and feature possibilities. IAS will make best efforts to provide infrastructure resources for programs/developers to conduct sand-box explorations and pre-development work Patching The DBA regularly examines DBMS vendor announcements about bug fixes, security fixes and other changes made through the release of software patches. Bug patches are applied when relevant to CDL applications as determined by the DBA, Database owner, or Vendor whose application interacts with the DBMS. Security patches are always applied when the associated risk is moderate to high 2 and relevant to CDL implementations, and otherwise within 1 year of release. In consultation with IAS principals and database owners, the DBA recommends and applies patches Maintenance procedures and windows Work Requests Specific work requests are initiated and tracked using CDL s ticketing system (Footprints), specifically the cdldb ticket queue and address (cdldb@ucop.edu). CDL Programs and database owners are strongly encouraged to discuss activities with large database components with the IAS Manager and DBA before any tickets are submitted. Requests for new databases would normally start with a request for a development instance and should minimally include: Brief description of the purpose/business need for the database 2 In general security risk is summarized by the DBMS vendor, often using the Common Vulnerability Scoring Standard (CVSS) Infrastructure & Applications Support - 6 -

8 Brief description of whether this database is expected to have dev, stage, and production instances, and the approximate timing to establish those instances. A proposed name (see above). An initial size required and growth estimate. A need by date. A proposed back up strategy/schedule (see above) Stop/start scripts Scripts that gracefully stop and start databases are made available for all databases. The location of the scripts is documented (see section 2, below). Normally the DBA is responsible for the creation and all uses of start/stop scripts (e.g. pre and post cold backup; for VCS offline and online). Other uses of the scripts are negotiated as necessary Maintenance windows To the extent possible, database maintenance windows follow CDL conventions: Dev/stage: 7am-7pm M-F with advance notice and calendaring Production: as defined in the CDL-IR&C Production Systems Support SLA ( Every effort twill be made to notify Database owners of urgent and emergency database maintenance, but that maintenance will proceed when key infrastructure is in jeopardy Capacity planning Database growth and additions are included in annual capacity planning initiated by the IAS Manager with contributions from each CDL program and service (via TechCouncil). Infrastructure changes and gating factors for growth are discussed through regular communication between IAS and the organization, especially TechCouncil and Director s Cabinet Security The CDL administers its databases in compliance with UC, UCOP, and CDL technology security policies and practices ( Relevant security practices e.g. account management, password protection, firewall and other access/perimeter control are developed with database impact in mind and by default apply to database management. Specific practices are mentioned in the relevant sections of this document. UC and UCOP guidelines make clear that IT security is a shared responsibility of all who use IT resources. Within that context, IAS consults with CDL s TechCouncil and Director s Cabinet to create and manage IT security practices for the CDL. The IAS manager works with the DBA and Database owners to assess and implement database-centric security practices. Infrastructure & Applications Support - 7 -

9 2. Extant database environment. [Note: the first official version of this spreadsheet will be complete by 3/27/09] See which records: Database name, size and growth estimates, filesystem, user names, script locations, and backup details by physical host and virtual host. Other associated databse administration documents, e.g. upgrade project plans, are available at The DBA is responsible for documenting and maintaining information about the extant database environment. Database owners should inform the DBA if they notice inaccuracies in the documentation. Infrastructure & Applications Support - 8 -