Systems Load Testing Appendix

Transcription

1 Systems Lad Testing Appendix 1 Overview As usage f the Blackbard Academic Suite grws and its availability requirements increase, many custmers lk t understand the capability f its infrastructure. As part f this analysis, sund business cntinuity practices are required t maintain uptime cmmitments; hwever an understanding f the infrastructure s perfrmance threshlds prvides tremendus insight int lng-term system stability. Many custmers face challenges in understanding the infrastructure needs their Blackbard usage requires. Blackbard Cnsulting s Systems Lad Testing service prvides institutins with the means t maximize the perfrmance f their Blackbard envirnment. This service can address all aspects f perfrmance engineering, frm specific perfrmance issues t lng-term capacity planning. This service is designed t prvide a thrugh exercise in perfrmance engineering, the end-t-end prcess f cnsidering perfrmance thrughut the sftware develpment lifecycle, frm requirements analysis t develpment and deplyment. This type f engagement will help rganizatins wh are lking t achieve higher levels f perfrmance maturity, as shwn in the Perfrmance Prcess Maturity Mdel belw. 1.1 Objectives Perfrmance Prcess Maturity Mdel 1 The primary bjective f the Systems Lad Testing service is t perfrm the necessary activities t understand, reslve and ptimize the Blackbard envirnment fr peak perfrmance. Thrugh this service, custmers are able t prvide their users with a Blackbard Academic Suite that perfrms at the highest level pssible. Blackbard cnsultants evaluate the current infrastructure, perfrm a series f lad tests and evaluate the envirnment hlistically. Thereafter, cnsultants design a plan fr imprving all aspects f system perfrmance and capacity. Blackbard s perfrmance investigatin attempts t determine the cause f any actual r ptential perfrmance issues; and prvide custmers with apprpriate guidance and talent fr systematic reslutin. The gals f this perfrmance investigatin are: Evaluate the current traffic and usage patterns Define client expectatins and gals Measure current system capacity and scalability Evaluate current architecture Identify and crrect any ptential prblem areas Ensure peratinal stability Prvide shrt-term recmmendatins fr tuning and ptimizatin Develp lng term recmmendatins fr architectural changes t imprve perfrmance, quality f service and reliability as well as best practices Plan fr future upgrades 1 Prpsed by Michael Maddx, MCI Inc, Prceedings f CMG Page 1 f 9

2 1.2 Perfrmance Audit and Tuning Apprach Thrugh its years f experience, Blackbard Cnsulting has develped a perfrmance engineering methdlgy fcused n efficiency and quality. Its apprach is brken int several, interrelated stages, designed t develp a rbust slutin that can be managed easily. The apprach is highlighted belw. Interactive Wrkshps Architecture Review Traffic Analysis and Mdeling Perfrmance Measurement Analysis and Redesign 1.3 Benefits Blackbard expects its apprach t prvide several critical benefits. Thrugh the applicatin f expert knwledge, using a prven apprach, custmers will gain a thrugh understanding f hw their Blackbard envirnment system perfrms under varius lad scenaris. Beynd identificatin, Blackbard cnsultants investigate, reslve and plan fr ptimum perfrmance f the envirnment as a whle, including the netwrk, hardware and sftware design. Sme f the planned imprvements can be implemented rapidly and yield cnsiderable imprvement, hwever the lng-term, detailed perfrmance analysis will enable custmers t maximize their current infrastructure capability and design fr tmrrw s needs. Examples f perfrmance engineering benefits include: Identificatin and reslutin f perfrmance issues Validatin f hardware cnfiguratin s ability t handle anticipated lad Guidance n architecture grwth Prvisin f insight n future capacity planning, including quantificatin f current capacity and alignment f grwth gals with recmmendatins fr achieving desired capacity. 1.4 Qualificatins Blackbard Cnsulting maintains a glbal team f ver 120 prfessinal cnsultants. In fact, Blackbard Cnsulting has the ability t truly wrk arund the clck t supprt yur elearning rllut if time is f the essence. As Blackbard Cnsulting has been implementing and evaluating client envirnments arund the wrld since 1997, several key trends have emerged: 1) Blackbard custmers that utilized this service saw a direct impact n the stability and scalability f their prductin system. 2) Institutins received the necessary dcumentatin t address perfrmance issues befre they became a prblem 3) Custmers wh have utilized this service run sme f the highest perfrming Blackbard applicatins in the wrld 4) Institutins have acknwledged perfrmance during peak lad times was less than the testing prcess and, therefre did nt intrduce system utages r perfrmance prblems 2005 Page 2 f 9

3 2 Cnceptual Prcess Flw 2005 Page 3 f 9

4 The implementatin plan belw describes the suite f activities that the Blackbard Cnsulting team and client prject team will participate in t cmplete the prpsed slutin. Each sectin f activities has specific utcmes that track twards cmpletin f the prject fr prductin purpses while als prviding knwledge transfer fr a level client self sufficiency. 3 Implementatin Plan The fllwing activities describe the services engagement in which yur rganizatin and the assigned Blackbard team will participate. 3.1 Prject Preparatin and Initiatin Blackbard Cnsulting team members assigned t yur rganizatin s engagement Familiarizatin with yur prject requirements, including the final Statement f Wrk and knwledge transfer frm the Blackbard Sales team Setup f client centralized cmmunicatin plan fr tracking prject activities, prgress, issues, and risk Preparatin f materials fr the prject kick ff cnference call Cnference call kick ff with the client team cnducted by Blackbard Cnsulting Intrductin f Blackbard and yur team members Review scpe f the engagement as defined in the Statement f Wrk Review the prject plan at a high level Describe the varius client participants required fr each stage f the prject engagement Determine the prject schedule, including scheduling f the nsite sessins After the kick ff call, the Blackbard Cnsulting team prepares and delivers t the client a detailed agenda and supprting dcumentatin fr the prject s nsite activities 3.2 Prject Management Thrughut the engagement, the Blackbard Prject Manager will prvide expert management f all Blackbard perfrmed activities. He/she will: Act as the primary pint f cntact between the client prject manager and the Blackbard team fr all prject-related tpics Define, dcument and update cmprehensive prject plans and prject schedules Track and reprt n prgress against the prject schedule Crdinate team meetings and prvide regular updates via cnference calls fr bth Blackbard and client cnstituents Prvide regular status reprts and bi-mnthly budget tracking and prject plan updates. Track and reprt n any prject issues Summarize and crdinate prject planning acrss all activities and Blackbard grups Cnsulting, Training, ASP, Quality Assurance, and Prduct Develpment 2005 Page 4 f 9

5 3.3 Lad Testing Wrkshp Activities Prir t the perfrmance evaluatin prcess, it is critical t design a plan that addresses the knwn issues as well as specific custmer needs. Therefre, Blackbard Cnsultants invest significant energies in this phase f the prject. It begins with a series f wrking sessins t understand the public perceptins regarding system perfrmance. Further, Blackbard cnsultants meet with IT persnnel t understand effrts currently underway t manage system perfrmance and recent capital expenditures r system upgrades. Next, the system as a whle is dcumented, including the hardware and netwrk design as well as any applicable third-party applicatins and Blackbard cnfiguratins. This infrmatin allws fr the necessary baselining, prir t testing. Mre specifically, cnsultants lead a series f wrkshps at the client s facility. Thse include the fllwing: Varius infrmatin-gathering exercises, including interviews with key stakehlders, surveys f the user cmmunity, and ther activities designed t elicit lcal expectatins and the current perceptin f the system perfrmance Identify and dcument the current Blackbard architecture at all levels including, but nt limited t: sftware cmpnents, custmizatins, third-party integratins, hardware, netwrk envirnment, and database cnfiguratin Activities structured t identify existing business practices, map them t applicable Blackbard prcesses, and share best practices arund cmmn system issues Identify the bjectives f the engagement 3.4 Architecture Review With the baseline infrmatin assembled and an understanding f the current supprt infrastructure, cnsultants are able t review the current architecture. This prcess is expected t review the hardware design fr capacity and lng-term sustainability. Typically, custmers chse amng several design standards (scaling up r scaling ut fr example) when designing the Blackbard envirnment and while the preferred standard is nt critical, its implementatin and lng-term capacity is. Next, the Blackbard applicatin is evaluated fr the custmer s cnfiguratin decisins. Each f the critical cnfiguratin and tuning parameters are dcumented fr use in the next phase. Lastly, cnsultants review the netwrk, bth t understand hw the Blackbard Academic Suite is laid ut and t assess its capacity and ptential bttlenecks. With an understanding f the hlistic envirnment, Blackbard cnsultants review usage patterns in an attempt t mdel typical usage. Infrmatin is gathered frm system and applicatin lgs and used t determine areas f the applicatin mst cmmnly used and respnse times during varying times and under varying lads. During the architecture review phase, Blackbard Cnsultants will prvide the fllwing services: Perfrm a full audit f current system architecture (hardware, sftware, netwrk, cnfiguratin) Audit current and expected usage patterns t create data and behaviral mdels Scan fr ptential prblem areas (signs f abandnment, resurce saturatin, faulty architecture, errrs within lgs, etc.) 3.5 Perfrmance Measurement Using the results f the client interview and architecture reviews, Blackbard cnsultants define, dcument and perfrm the necessary lad testing activities. This phase begins with a detailed 2005 Page 5 f 9

6 measurement plan. This plan utlines the specific elements f the system t be measured, the expected utcmes as well as a baseline frm which cmparisns can be made. Specialized Blackbard tls will place lad n the client systems t simulate the behavir mdels develped during the previus phase. The tests begin with a thrugh evaluatin f the system as a whle, understanding specific bttlenecks. These issues are typically reslved in shrt-rder and mre detailed tests begin. Detailed tests are indented t use the custmer s specific behavir mdels and evaluate the envirnment s ability t respnd t user demand. Respnses are measured in terms f system capacity (i.e. limitatins) and reliability (i.e. scalability) in rder t determine weaknesses. Mre specifically, cnsultants will perfrm the fllwing: Specialized Blackbard tls will place lad n the client systems t simulate varius levels f real-life usage cnditins The test campaign will measure capacity, reliability, scalability, and user experience (respnse time) as well as identify ptential bttlenecks and htspts 3.6 Perfrmance Analysis The lad testing prcess generates a great deal f insight int capacity and system limitatins. Blackbard Cnsultants lead the custmer team thrugh a thrugh analysis f the resulting data sets. The specific areas f fcus during this analysis are: Netwrk respnse Hardware respnse Sftware respnse Blackbard cnfiguratins As part f the testing prcess, many f the smaller issues are identified and reslved. During this phase f the prject, the brader, lng-term issues are discussed. Blackbard cnsultants lead the discussins and, based n bth the data and results f the technical discussins, prvide the fllwing recmmendatins: Shrt-term ptimizatin and tuning pprtunities Lng-term architectural recmmendatins t achieve the perfrmance, reliability and quality f service gals Guidelines fr grwth and capacity planning This phase f the prject cncludes with a review f the recmmendatin and a series f discussins regarding the future f the system. These discussins fcus largely n scalability ptins, based n the custmer s chsen strategy and a review f the lng term capacity planning suggestins. Cnsultants then research the varius ptins available t the custmer team and mdel several scenaris. This infrmatin is used t understand the next steps regarding system perfrmance and dictates the lngterm strategy fr maintaining ptimum perfrmance. 3.7 Prject Clsure At the cnclusin f the prject, Blackbard will furnish final dcumentatin t the custmer in the frm f a prject cntrl binder. It includes: Interview Summaries Survey Results Architecture Summary Summary f Perfrmance Engagement Gals 2005 Page 6 f 9

7 Results f the System Audit Results f Lad Testing Dcumentatin f any Tuning/Optimizatin Changes made Recmmendatins Prject Final Reprt The dcumentatin will include all materials develped thrughut the prject as well as suggested next steps fr yur rganizatin Page 7 f 9

8 The schedule belw reflects the necessary timelines required t fulfill the apprpriate prject milestnes. Key t staying n schedule is making sure we have the crrect client participants. Our bjective is t fllw the prject schedule as clsely as pssible prviding there are n significant scheduling challenges r changes in prject scpe. 4 Schedule During the term f the prject the Blackbard prject team will guide the client team thrugh a series f exercise fcused n gaining a deep understanding f current system perfrmance and identifying any knwn r suspected prblem areas. The engagement begins with a series f client interviews and architecture reviews that utline the prcess and capture applicable business rules fr a perfrmance audit. Next, Blackbard cnsultants develp the necessary lad testing strategy. Finally, they prvide recmmendatins t bring the system up t the ptimum level f perfrmance. 4.1 Prject Timeline The average prject is estimated t be fur t five weeks frm prject initiatin. A rugh timeline is shwn abve with activities befre and after the nsite. 4.2 Scheduling Requirements Blackbard Cnsulting s methdlgy assumes a certain level f effrt frm the custmer team. Fr example, thse individuals with direct respnsibility fr the system s perfrmance and maintenance must be cmmitted during the three prject phases. Blackbard has fund that this immersin is necessary t prepare custmers fr the nging demands f the imprved system. The fllwing table is intended t aid yu in planning fr this prject: Please nte: Blackbard cnsulting resurces are available n a first cme first served basis and will be assigned immediately after agreement n prpsed wrk and signature f a Statement f Wrk 2005 Page 8 f 9

9 4.3 Client Participants Prject Task Client Interview Architecture Review Lad Testing Recmmendatins Rles Required Executive Spnsr Applicatin Administratr Prject Manager System Administratr Senir IT manager Database Administratr Business Analyst / Netwrk Administratr Educatinal Strategist Educatr panel Applicatin Administratr Database Administratr System Administratr Netwrk Administratr Applicatin Administratr Database Administratr System Administratr Netwrk Administratr Prject Manager System Administratr Senir IT manager Database Administratr Applicatin Administratr Netwrk Administratr 5 Cnclusin and Blackbard Cnsulting appreciate yur interest in wrking with ur skilled team f technlgy experts. We believe the afrementined prpsal and estimate will yu re yur needs by prviding the necessary insight and talent t imprve the perfrmance and capacity f yur Blackbard systems Page 9 f 9