Athletic Training and Pragini academic Participatin

Transcription

1 Online Assessment f Athletic Training Educatin Outcmes and Prgram Satisfactin W. David CARR, Ph.D. Athletic Training Educatin,The University f Kansas Lawrence, KS 66045, USA Stephen R. ROCKWELL, Ph.D. Schl f Accunting and Management Infrmatin Systems, The University f Tulsa Tulsa, OK 74104, USA Greg A. GARDNER, Ed.D. Athletic Training, The University f Tulsa Tulsa, OK 74104, USA and Elizabeth SWANN, Ph.D. Athletic Training, Nva Sutheastern University Frt Lauderdale-Davie, FL 33314, USA ABSTRACT This paper describes the develpment f the Online Assessment f Athletic Training Educatin system (OAATE), a tl fr assessing student achievement in the knwledge dmains necessary fr certificatin as an athletic trainer. The system als assesses students satisfactin with imprtant dimensins f their individual degree prgrams. By making use f current database and cmmunicatin technlgies, we have develped a system that addresses imprtant, unmet needs in the field f Athletic Training educatin. The design f the system makes it a dynamic, easily extensible tl that culd be applied in a wide variety f educatin dmains beynd its current setting. In additin, because f its Internet-based delivery system, the tl may be widely-used thrughut the wrld, with benefits accruing t students, prgram instructrs and administratrs, and researchers in the field f educatin. Keywrds: Educatin Assessment, Database, Infrmatin/Cmmunicatin Technlgies, Online Assessment. 1. INTRODUCTION It is a widely-held view that assessment f learning utcmes shuld be majr cmpnent f any educatin prgram. In higher educatin, it is cmmn t see this assessment ccurring at the degree prgram level, as well as at the individual curse level. T that end, in many disciplines there are multiple assessment tls available, ranging frm simple, internally develped instruments t thse created by natinal testing r certificatin agencies. In the field f Athletic Training educatin, as in many ther dmains, available assessment tls suffer frm a variety f shrtcmings that reduce their effectiveness r use, amng them There is n single, universally adpted instrument in use, and the multiple instruments available are nt standardized enugh t allw cmparisn acrss instruments. This makes cmparisn amng prgrams at the natinal level difficult r impssible. The csts f the tls may be restrictive fr smaller prgrams. They may nt measure all the dimensins desired by a particular prgram, and They are usually nt easily (if at all) mdifiable t include lcal assessment issues. They may nt capture attitudinal r ther parameters related t the assessed knwledge r perfrmance. They may nt be adapted in a timely fashin t changes in the external envirnment r t changes in prgram cntent and methdlgy. There may be a significant time delay between testing and assessment, especially fr paper-based instruments. The statistical analysis prvided by assessment services may nt include the specific analysis desired by a prgram. Fr example, lngitudinal assessment f student perfrmance ver a multiyear perid may need t be perfrmed by ad hc analysis, using tls external t the assessment service; r The analysis may nt prvide desired intra- and interprgram cmparisns. T help vercme these limitatins, we have designed, and are cntinuing t develp, an nline assessment system fr Athletic Training educatin prgrams at the university level. The system perfrms tw distinct assessment tasks. The first is assessment f student achievement in the twelve knwledge dmains identified by the Natinal Athletic Trainers' Assciatin Educatin Cuncil (NATA-EC). The secnd task is the students assessment f satisfactin and imprtance f their individual degree prgrams. 74 SYSTEMICS, CYBERNETICS AND INFORMATICS VOLUME 4 - NUMBER 6

2 Tgether, these assessments allw educatrs and administratrs t better evaluate their prgrams and their students prgress, as well as help them demnstrate the cntinuus assessment required by accreditatin standards. In additin, the system allws researchers t cllect and analyze data as part f a large scale, lngitudinal, crsssectinal research prgram that will prvide insight int the field f Athletic Training educatin. 2. METHODOLOGY Our assessment system is a brwser-based, nline system that allws Athletic Training prgrams t participate frm anywhere with Internet access. It has been designed fr three majr purpses: t assess the knwledge f graduating senirs wh are preparing fr the Natinal Athletic Trainers Assciatin Bard f Certificatin (NATABOC) certificatin exam; t measure students satisfactin and assessment f their degree prgrams, and t cllect crss-sectinal, lngitudinal data abut Athletic Training educatin, t allw researchers t better understand the efficacy f such educatin. Students cmplete a twelve part exam that tests twelve majr knwledge dmains within the field f athletic training, and they als cmplete a questinnaire that captures students evaluatins f their degree prgrams. The questinnaire and twelve dmain testing subsectins are discrete sessins, enabling students t cmplete the assessments ver multiple nline sessins. Allwing the assessments t be taken by knwledge area mimics the apprach taken by natinal certificatin exams in a number f prfessinal areas, and we anticipate it will increase adptin f ur system by schls with degree prgrams in Athletic Training. A larger number f participants increases the usefulness f the system t educatrs and researchers alike. The underlying technlgy used by the system makes it especially extensible and efficient, and allws the system t be mdified quickly and easily. This allws easy adaptatin in the dynamic envirnment f Athletic Training educatin. The technlgy being used includes relatinal database technlgy (Micrsft SQL Server), Micrsft Internet Infrmatin Server (IIS), Java, active server pages, and scripting languages (JavaScript and VBscript). The relatinal database is used bth t capture data fr later reprting and analysis and als t generate data fr dynamic web pages. These pages deliver differing cntent, depending n task and/r earlier respnses by the user. The web-based and database cmpnents are discussed at mre length in the fllwing sectins. Thse cmpnents are the keys t develping a system that vercmes the shrtcmings f currently available assessment tls described in the Intrductin sectin f this paper. 3. WEB-BASED ASSESSMENT The first shrtcming described in the previus sectin was the fact that there is n single, universally adpted instrument in use fr assessment f educatinal utcmes in the field f Athletic Training. The rt causes fr this are mst likely clsely assciated with several f the ther listed shrtcmings: the cst f the tls may be restrictive, the tls may nt measure all the dimensins desired by a particular prgram, and they are usually nt easily (if at all) mdifiable t include lcal assessment issues. We believe that a tl that vercmes these shrtcmings has the ptential t becme widely adpted, and thus prvide the brad baseline f data required t allw students, prgram directrs, and researchers t cmpare educatinal utcmes acrss the entire spectrum f prgrams. We further believe that OAATE is such a system. By develping a web-based assessment system, the prblems assciated with distributin, cllectin, and cst are greatly reduced. In fact, there is n cst t participating schls fr using the system itself, and as virtually all schls have Internet access and brwser-equipped cmputers, the incremental cst t schls shuld be near zer. The web site was develped using current XHTML specificatins. The adaptability and extensibility f the web site is facilitated by making use f Cascading Style Sheets (CSS) and dynamic web pages that draw much f their cntent frm an underlying database that stres web page cntent that might change frequently. The dynamic web pages allw cntent t be prduced and displayed n the web page when needed. There are three main type f dynamic cntent used by ur system: The test bank f knwledge dmain and satisfactin questins, Statistics based n the data cllected by thse questins, and Infrmatin based n data submitted by prgram directrs. The web-based interface fr the testing functins uses a multiple-chice questin frmat with which students are universally familiar. This type f test nt nly mimics the frmat f the certificatin tests taken by Athletic Training students, but als lends itself t a pint and click interface, using radi buttns fr answer selectin. Frm the prgram directr s perspective, the web-based system allws administratin f the assessment tl frm any cmputer with Internet access. Prgram directrs can register their prgrams, set up student accunts and passwrds fr their students, select a set f peer schls fr utcme cmparisns, and review a variety f reprts based upn test utcmes. SYSTEMICS, CYBERNETICS AND INFORMATICS VOLUME 4 - NUMBER 6 75

3 Figure 1: Self-Selectin Reprt The prgram directr s Self-Selectin Reprt is shwn in Figure 1. This reprt shws the average scre f the directr s students, and ranks thse scres cmpared t average scres f students at their peer schls. When the prgram directr selects the initial peer schl list, it is selected frm a system-generated list f all schls currently registered with OAATE. As new schls register, OAATE autmatically generates an t each currently registered prgram directr. The annunces the new registrant schl, t allw prgram directrs t add the schl t their peer schl lists, if desired. S in additin t giving prgram directrs the mst up t date list f scres, OAATE allws them t keep a current list f peer schls. In additin t the knwledge assessment questins, students are asked t rate their degree prgram n a number f dimensins. Thse dimensins are Use f Technlgy Career Placement Service Administratin Curse Instructin Advising Clinical Experience Instructr Availability Fr each dimensin, students prvide tw ratings. The first rating indicates the student s satisfactin with that dimensin f their prgram. The secnd rating indicates the student s assessment f the imprtance f that dimensin t them. As with the knwledge dmain scres, prgram directrs can see their students satisfactin/imprtance scres and rankings cmpared t their peer schl grup. The bttm prtin f the Self-Selectin Reprt is shwn in Figure 2. Figure 2: Satisfactin/Imprtance Scres and Ratings 76 SYSTEMICS, CYBERNETICS AND INFORMATICS VOLUME 4 - NUMBER 6

4 The Self-Selectin reprt shwn abve relates a prgram s scres t the self-selected set f peer schls. OAATE als prvides a number f reprts that reprt a schl s scres and cmpare them t the average scres f all participating schls. Figure 3 shws a prtin f this type f reprt. Yu ll ntice that the reprt als shws the multiyear trend in the schl s scres fr each Satisfactin/Imprtance dimensin. The reprts described here are just a sample f the wide variety f reprts that have r are being created fr OAATE. Summary statistics such as means, medians, maximums, and minimums will be available fr virtually any level f evaluatin frm individual assessment questin, t student, prgram, and natinal levels. Students receive immediate feedback in the frm f a diagnstic evaluatin f their perfrmance within each educatinal dmain, and fr questins where a student gave an incrrect answer, a reference is given, t pint the student t a specific bk chapter r article where the questin s cncepts are explained. Thus, OAATE ges beynd mst assessment tls by helping students find the resurces t imprve knwledge areas where they may be deficient. 4. THE DATABASE COMPONENT The engine that really drives OAATE s capabilities is the relatinal database that underlies it. Figure 4: shws a simple Entity-Relatinship-Diagram (ERD) fr the database. Missing frm the figure is a recursive relatinship f Institutins upn itself. That relatinship links a schl t its peer schls in the same table. By decupling questin cntent frm the web pages, we gain several imprtant advantages. First, a test s dmain questins may be drawn randmly frm a larger pl f dmain questins, helping t minimize cheating when giving the test in a grup setting. Secnd, test questins Figure 3: Satisfactin/Imprtance Summary Reprt r answers may be mdified (changed, added, deleted) withut requiring mdificatin t the web pages r underlying cde. Older questins can be cycled ut f the exam rtatin, and newer questins can be cycled in. If imprtant new dmain knwledge is discvered that wuld change a questin r require a new questin, that knwledge can be incrprated int the test as sn as it is knwn. This has bvius benefits frm the timeliness perspective, and it has great cst savings ver traditinal paper-based tests that might require discarding ptentially large numbers f utdated tests. Likewise, the satisfactin/imprtance questinnaire may be easily mdified t meet the changing needs f prgram directrs. While we can require that certain questins be answered by all schls, ther questins may be designated as ptinal, with the prgram directrs selecting the questins t be answered by their students. Figure 4: Database ERD SYSTEMICS, CYBERNETICS AND INFORMATICS VOLUME 4 - NUMBER 6 77

5 A future enhancement t OAATE will be the ability fr prgram directrs t add their wn satisfactin questins t the ptinal survey questin set. Thus, a larger, shared bdy f questins might be develped frm participating schls, adding the pssibility fr schls t learn frm the best practices f ther schls. This capability has already been designed and built, and it is currently underging testing, prir t its release int the prductin envirnment. Beynd the advantages listed abve, system extensibility and maintainability is perhaps the greatest benefit f using relatinal database technlgy fr string the large quantity f test, participant, and web page data required by OAATE. A lk at Figure 4 shws the variety f data currently cllected abut students and participating institutins. If researchers r participants wish t cllect data abut new attributes nt included in the riginal design, adding them is a relatively trivial task. Fr example, if future research finds a new factr related t student test perfrmance, it can easily be added as a field in the database, and the web page fr cllecting student data can be mdified with a relatively few lines f cde. The time delay between identifying a new attribute and adding it t OAATE can be a matter f hurs, if nt minutes It s difficult t veremphasize the pwer added t OAATE by using this database-driven, dynamic web page cmbinatin. Over the curse f system develpment, we encuntered numerus situatins in which we were saved by the design. Fr example, during develpment f questins fr the varius knwledge dmains, we became aware that there are at least tw different decmpsitins available fr ur dmain knwledge: the twelve dmain framewrk specified by the Natinal Athletic Trainers' Assciatin Educatin Cuncil and a secnd, cmmnly-used decmpsitin that divides athletic trainer knwledge int nly six categries. Because f the design f the database, by adding a single field t the questins table, we can identify questins by their secnd categry. Tests culd be cnstructed, and infrmatin reprted, using the alternate framewrk, shuld we s desire. Anther benefit f relatinal database technlgy is the relative ease with which data may be extracted by a variety f sftware applicatins. In ur web pages, we use SQL cde t generate the data fr the varius reprts, but we culd easily access the data frm within spreadsheet r statistics prgrams, fr mre advanced analysis. Using the Internet as ur cmmunicatin medium, we culd generate XML-based reprts fr sharing data sets amng researchers at different institutins. 5. CURRENT PARTICIPANT DEMOGRAPHICS OAATE is currently in use at a number f private and public universities. Table 1 shws selected demgraphics, t prvide insight int the current scpe f ur research. Table 1: Selected Demgraphics Ttal number f athletic training educatin prgrams: 352 Number f prgrams registered in OAATE: 96 Number f registered prgrams that have registered students 50 Number f students registered 439 Number f students tested 156 Befre students may use the OAATE system, their prgram directr must register their schl with OAATE researchers. Once registered, prgram directrs can enrll the students they want t participate withut further invlvement f the researchers. There is typically a time lag between the date a prgram directr registers students and the date students begin using the system, s the number f registered students typically exceeds the number wh have participated in testing. In additin, the ttals abve include schls that registered at the beginning f the current academic year, and students will mst likely take the assessments late in the semester, after the Table 1 numbers were generated. 6. FUTURE DIRECTIONS Because f the ease with which the system may be maintained and mdified, there are a number f ptential future enhancements t OAATE that may be envisined. Given the nature f the data cllected and the primary purpse f the system, sme f the mre bvius imprvements invlve reprting. We envisin prviding a much wider variety f statistics fr prgram directrs, including standard deviatins, medians, minimums, maximums, and mre. Multiyear reprts f peer schl and natinal averages wuld be an infrmative additin t the single schl data shwn in Figure 3. Beynd the infrmatin being prvided by OAATE, infrmatin frmat is anther area f ptential imprvement. Recently, a great deal f attentin and develpment effrt has been directed tward prducing digital dashbards t present infrmatin t infrmatin users. The term digital dashbard refers t a user interface where data is displayed in a graphical frmat, ften resembling the array f gauges and dials in the dashbard f an autmbile. The term is in such wide use nw that the term has mved beynd the autmbile metaphr and is being used t refer t an interface that integrates infrmatin frm multiple surces int a single, unified display, ften incrprating text, tables, and pictures. Mst ften, the underlying infrmatin fr the dashbard is stred in databases. There are a number f third-party vendrs f such tls, and it wuld be relatively easy t incrprate them int OAATE. The multiyear assessment scre averages might be presented as line r bar charts, r even as a gauge reminiscent f a gasline gauge, with prgram directr-specified warning levels that indicate when scres are mving int dangerusly lw levels. There culd be red light-green light metaphrs we re limited nly by imaginatin, it seems. 78 SYSTEMICS, CYBERNETICS AND INFORMATICS VOLUME 4 - NUMBER 6

6 Once we have implemented ur desired enhancements t OAATE, we intend t extend its use int ther knwledge dmains. The nature f the tests and reprts we have develped are applicable t many ther degree prgrams. The design structure f the system is such that neither the database nr web site require extensive mdificatin t be used in such a manner. It culd be accmplished in a few shrt stages: Clne the data base and web site. Mdify pages t reflect the new knwledge dmain. Change the test questins and the satisfactin/imprtance questins in the test database. Mdify a few systems resurce files that keep track f imprtant web site infrmatin. If a set f questins were already develped, it culd be a matter f as little as a week r less t have an OAATE-like system up and running in a different knwledge dmain. 7. CONCLUSIONS In this paper, we discussed sme f the aspects f OAATE, a system fr assessing educatinal utcmes fr students in Athletic Training degree prgrams at the university level. We explained the advantages derived by using a web-based system and underlying relatinal database instead f mre traditinal assessment methds. Thse advantages include the ability t quickly and easily adapt the system t changes in dmain knwledge and educatin cntent; reduced cst t users f the assessment tl when cmpared t alternative tls; and the ability t custmize the system fr a particular institutins needs. In ur Future Directins sectin, we described a number f enhancements and extensins t OAATE and explained hw it might be mdified t utilize it in different knwledge and educatin dmains. Thrugh the use f Infrmatin/Cmmunicatin Technlgies, OAATE imprves upn previus methds f assessing educatinal utcmes in the Athletic Training field and ffers us ther advantages as well. We believe that thrugh OAATE we will als be able t cllect a large set f crss-sectinal, lngitudinal data that can be used by educatin researchers t imprve Athletic Training educatin. The authrs gratefully acknwledge the significant effrts and assistance f students in the spring 2005 infrmatin systems prject curse at The University f Tulsa, especially Carrie J. Hersh, prject manager, and Matthew T. Snyder, team leader and senir prgrammer. SYSTEMICS, CYBERNETICS AND INFORMATICS VOLUME 4 - NUMBER 6 79