1 Overview Oracle RAC Services Appendix As usage of the Blackboard Academic Suite grows and the system reaches a mission critical level, customers must evaluate the overall effectiveness, stability and scalability of the application. A critical element of this evaluation is the database. Typically, Blackboard customers maintain a single database, which introduces a critical point of failure. While there are many options to address this problem, Unix systems have the opportunity to take advantage of an Oracle technology known as Real Application Clusters ( RAC ). RAC solutions provide customers with enhanced services to increase redundancy and scalability of the Blackboard database, by allowing more than one server to manage the database itself. 1.1 Objectives The primary objective is to implement a RAC environment for the Blackboard Academic Suite. Ultimately, customers who are required to maintain Blackboard in a mission critical environment will be forced to address the Blackboard database from an availability standpoint. This service meets the requirement for improved availability by providing customers with greater redundancy and opportunity for scalability and improves the Blackboard system s overall availability. 1.2 Custom Development Approach Through its extensive experience and partnership with Oracle Corporation, Blackboard Consulting has developed a methodology to implement RAC in Blackboard environments. Its approach is broken into several, interrelated stages, designed to develop a robust solution that can be managed easily. The approach is highlighted below: Requirements Definition Solution Design Infrastructure Preparation Testing and Migration Deployment Maintenance As part of the development process, Blackboard will look to obtain the following objectives: Leverage Blackboard product knowledge and experience for efficient RAC implementation and configuration, minimizing maintenance and upgrade costs Leverage customer investment in the Oracle application by using best of breed solutions developed and proven by Oracle Corporation Develop the environment using standard practices that allow for ongoing customer management Test the environment to validate failover processes, in conjunction with existing business continuity practices Develop comprehensive documentation for future support and management of the environment 1.3 Benefits Oracle s Real Application Clusters ( RAC ) offer two distinct advantages: scalability and availability. With RAC, the Blackboard database can scale by distributing database load among several servers. RAC also provides a highly available solution. The database environment can be configured to automatically fail over to another server, should any database node fail. Most customers who employ RAC are looking for one or both of these advantages. Blackboard Inc. 2005 Page 1 of 9
The Blackboard consultants assigned to your project are highly experienced and trained in Oracle RAC and also have in-depth experience with the Blackboard Academic Suite. All projects are supervised by certified Oracle DBAs. These individuals have access to a broad set of Blackboard talent to assist them as necessary. This includes subject matter experts within the Consulting organization, and designers and developers from Software Engineering. Blackboard expects its approach to provide several critical benefits. By capitalizing on our institutional knowledge and Oracle experience, clients will receive a database solution that not only meets requirements and is thoroughly tested, but one that performs well, is appropriately scaled and integrates effectively with the core Blackboard product. 1.4 Qualifications Blackboard Consulting is 120 people strong across the world who have been implementing custom solutions for clients since 1996. In fact, Blackboard Consulting has the ability to truly work around the clock to support your elearning rollout if time is of an essence. Blackboard Consulting team members are trained and certified on Blackboard s Oracle RAC methodology, as well as on the technical architecture of the Blackboard Academic Suite products. All DBAs are Oracle certified, with a great deal of RAC experience. Blackboard Consulting Project Managers are experienced in IT project management best practices and processes. Blackboard technical consultants are expert on all applicable technologies, such as Java, PERL, SQL, Oracle, Linux, MS Windows, Solaris; as well as on accepted IT systems development life cycle processes. Blackboard Inc. 2005 Page 2 of 9
2 Conceptual RAC Architecture Blackboard Inc. 2005 Page 3 of 9
The implementation plan below describes the suite of activities that the Blackboard Consulting team and client project team will participate in to complete the proposed solution. Each section of activities has specific outcomes that track towards completion of the project for production purposes while also providing knowledge transfer for a level of client self sufficiency. 3 RAC Implementation Plan The following activities describe the services engagement in which your organization and the assigned Blackboard team will participate. 3.1 Project Preparation and Initiation The Blackboard Consulting team members assigned to your organization s engagement will: - Familiarize themselves with your project requirements, including the final Statement of Work and knowledge transfer from the Blackboard Sales team - Setup a centralized communication plan for tracking project activities, progress, issues and risks - Prepare materials for the project kick off conference call Blackboard Consulting will schedule and conduct a conference call kick off with the client team. During that call the following items will be covered: - Introduction of the Blackboard and client team members - Review of the scope of the engagement as defined in the Statement of Work - Describe the various client participants required for each stage of the project engagement. - Determine and create a high level project schedule, including scheduling of the onsite sessions After the kick off call, the Blackboard Consulting team prepares and delivers to the client a detailed agenda and supporting documentation for the project s onsite activities. If not already done, the client will install and configure a local version of Blackboard and Oracle to be used as a Test and Development platform 3.2 Project Management Throughout the engagement, the Blackboard Project Manager will provide expert management of all Blackboard performed activities. He/she will: - Act as the primary point of contact between the client project manager and the Blackboard team for all project-related topics - Define, document and update comprehensive project plans and project schedules - Track and report on progress against the project schedule - Coordinate team meetings and provide regular updates via conference calls for both Blackboard and client constituents - Provide regular status reports and bi-monthly budget tracking and project plan updates - Track and report on any project issues Blackboard Inc. 2005 Page 4 of 9
- Summarize and coordinate project planning across all activities and Blackboard groups Consulting, Training, ASP, Quality Assurance, and Product Development. 3.3 Requirements Definition This is the most critical portion of the project, as it is the foundation from which all development, testing and implementation activities stem. The process begins at the high-level from the expected outcomes described in the Statement of Work, and continues into the details, including environmental requirements, customer technical/hardware resources and development and data needs. At the conclusion of this phase, the client knows what will be developed. Customers typically find that their specific RAC needs are considerably more complex than originally anticipated. RAC solutions are complex and require careful planning. During this phase of the project, Blackboard Consultants are looking to define the high-level system requirements and understand the customer s unique environmental variables (i.e. does the infrastructure exist to support RAC. In the project s next phase, details will be identified and organized into a technical design. 3.4 Technical Design With the requirements and environmental variables identified, Blackboard Technical Consultants design the manner in which the solution will be developed. This solution design outlines the detailed process flow, the configuration necessary to implement it and the data requirements to support it. During this phase Blackboard Consulting capitalizes on Oracle knowledge and RAC experience. RAC solutions generally adhere to a consistent high-level design, however specific customer networks, hardware resources and infrastructure impact the detailed technical design considerably. 3.4.1 Technical Design Process The most important design feature to be considered is an infrastructure layout that eliminates single points of failure. As such, the Blackboard consultants consider the following, in addition to the RAC setup o Firewall o Network switches o AN/NAS array With the layout defined, the design process shifts focuses to the following areas: o Database nodes o Application server configuration o Database connection pool o Switch configuration Finally, the proposed solution is documented in a format that allows for a smooth transition into development. Critical components of the technical design include: o Hardware, database and network layout o Software and OS requirements o Database nodes and file system configuration Blackboard Inc. 2005 Page 5 of 9
3.5 Infrastructure Preparation In preparing the system for RAC, it is important to ensure that the infrastructure is capable of handling the designed solution. This straightforward process leverages details from the technical design and compares its requirements with the customer s infrastructure. More specifically, Consultants assess the following system components and configure the system for RAC. Network setup, including the cabling and switch installation (both load balancing and private network switches) as well as switch configuration RAC node setup, including the hardware installation, operating system installation and configuration Lastly, consultants install and configure RAC and then install the appropriate Academic Suite components, patch and configure it to interoperate with RAC 3.6 Development and Testing Blackboard s methodology requires unit testing at specific development intervals to avoid potential configuration conflicts. More specifically, customers expect their Blackboard application to operate under RAC as it did in the previous database configuration. Therefore unit and system testing focus on continuing the current end user functionality. Beyond that point, consultants test the RAC failover features under a variety of failure scenarios. The expectation is that if a single database node were to fail, the other nodes would carry the load and the system would remain available. 3.6.1 Testing Process Develop unit and system test cases from the system requirements and technical design Conduct internal configuration reviews for each segment of the system to verify accuracy against the requirements and design Conduct unit tests; modify and re-test configurations, as necessary Document unit test results Conduct system tests, such as database node failures; modify and re-test configurations, as necessary Document system test results Write internal technical support documentation Store tested code in Blackboard s code repository Migrate the solution from the customer s test to its production environment 3.7 Deployment At the conclusion of the testing process, Blackboard will deliver the final product to the client along with its associated documentation, which includes management instructions and system configuration guidelines. The Blackboard technical team will also complete internal turnover procedures to store the client s RAC code/configuration into Perforce, Blackboard s code repository and configuration management system, along with detailed documentation for future support and maintenance. 3.8 Project Closure At the conclusion of the project, Blackboard will furnish final documentation to the customer in the form of a project control binder. Blackboard Consulting will also complete internal turnover activities to the Blackboard Inc. 2005 Page 6 of 9
Blackboard Support organization for the initiation of Integration and Support Maintenance for the RAC solution. 3.8.1 Project Closure process Blackboard Consulting activities: - Develop and deliver the Project Final Report. This report documents the deployed system. It includes: o High-level system requirements o Technical solution design o Infrastructure requirements o RAC maintenance guidelines and failover scenarios o Test case results o Project Final Report - Fully document the RAC configurations in the Blackboard files, CRM tool and code repository - Complete activities necessary to initiate I&C Maintenance for the environment Blackboard Inc. 2005 Page 7 of 9
The schedule below reflects the necessary timelines required to fulfill the appropriate project milestones. Key to staying on schedule is making sure we have the correct client participants. Our objective is to follow the project schedule as closely as possible providing there are no significant scheduling challenges or changes in project scope. 4 Schedule During the term of the project the Blackboard project team will guide the client team through a series of exercises focused on defining, designing and implementing a robust RAC environment. The engagement begins with a series of requirements sessions that outline the process and capture applicable business rules and workflow. Next, Blackboard consultants design and implement the necessary technology and prepare for production rollout. The exact timeline is jointly developed during the project initiation and requirements definition phases, however a sample schedule is below. 4.1 Project Timeline 4.2 Scheduling Requirements [Figure1. Solution Project Plan] Blackboard Consulting s methodology assumes a certain level of effort from the customer team. Blackboard has found that this commitment is necessary to prepare customers for the ongoing demands of supporting new technology. The following table is intended to aid you in planning for this project. The specific client participants and schedule will be defined during the Project Initiation phase. Please note: Blackboard consulting resources are available on a first come first served basis and will be assigned shortly after agreement on proposed work and signature of a Statement of Work. Blackboard Inc. 2005 Page 8 of 9
4.3 Client Participants Project Task Roles Required Requirements Definition Executive Sponsor System Administrator Database Administrator System Administrator Technical Design Database Administrator Network Administrator Security Manager Infrastructure Preparation System Administrator Database Administrator Network Administrator Testing Network Administrator Database Administrator Deployment Network Administrator Database Administrator Project Closure Executive Sponsor 5 Conclusion Blackboard Inc. and Blackboard Consulting appreciate your interest in working with our skilled team of technology experts. We believe the aforementioned proposal and will provide your Blackboard environment with a higher level of availability and a more scalable infrastructure. Blackboard Inc. 2005 Page 9 of 9