Coni Francis, PhD, RD Nutrition and Dietetics Program University of Northern Colorado
University of Northern Colorado Nutrition and Dietetics Program Alena Clark, PhD, MPH, RD Jamie Erskine, PhD, RD Albion College Department of Psychological Science Andrea P. Francis, PhD
Program Assessment Program Assessment Exam Development Reliability and Validity of Program Assessment Exam Comparison of On-Campus and Distance Learning Academic Integrity in Distance Learning Discussion Conclusions
Program improvement Student self-assessment Student practice for professional exams Accreditation requirement Institution reporting
Purpose Program assessment Student practice Student self-assessment Taken at the end of the senior year
Domain development Exam question writing Exam question revision Exam score review 100 point randomized computer administered exam
146 Students took the exam During academic years 2010-2011, 2011-2012 and 2012-2013 all eligible students Both tracks took the exam On- campus (n=48) Online distance (n= 98) Internal consistency Cronbach s alpha =.85
23 (15%) students retook the exam 2 months later Test-retest reliability Pearson s correlation Exam Mean SD r p Time 1 76.13 9.02 Time 2 76.70 6.24 0.52 0.01
Pearson s Correlation between Program assessment exam score Program GPA N=146 Mean SD r p Exam 75.73 9.77 GPA 3.43 0.39 0.54 >0.0001
50 (34%) students released their RD exam scores Pearson s Correlation between Program assessment exam score RD registration exam score N = 50 Mean SD r p Exam 75.96 8.83 RD Exam 28.68 3.67 0.35 0.013
Online learning is one of the fastest growing trends in educational technology. U.S. Department of Education, 2010 According to a 2010 meta-analysis by the U.S. Department of Education, students participating in online learning perform as well or better than those participating in classroom instruction. However, this had not been tested within dietetics education.
To determine if students enrolled in a didactic program in dietetics (DP) delivered online would perform on an end of program assessment exam as well or better than students taking the same course work through classroom instruction.
Mean program assessment exam scores from academic years 2010-2011 and 2011-2012 for both online instruction and classroom instruction were compared using student s t test
Track N Mean Exam Score Oncampus SD 48 72.08 11.53 Distance 98 77.51 8.28 0.005 p
Track N Female Male GPA Previous Degree On campus 48 46 (96%) 2 (4%) 3.25 18% Online 98 93 (95%) 5 (5%) 3.51* 65% *p = 0.002
Group N Age Black Hispanic Other White On campus 15 25.53 0 0 15 (100%) Online 38 34.32* 3 (8%) *p < 0.0001 2 (5%) 2 (5%) 31 (82%)
Since core knowledge standards are the same for all accredited DPs, it would be of interest to see if other institutions would find similar results using this or a similar exam. Since pass rate of the of the credentialing exam for registered dietitians (RD) is used as an indicator to determine the quality of accredited didactic programs in dietetics this exam may be a measure that can be used to validate RD pass rate improvement plans.
Loss of academic integrity is a concern across many college campuses Perception that distance education has more opportunities for loss of academic integrity Little research for online academic integrity
Require practices to prove the identification of the student being assessed U.S. Department of Education Most accreditors Most institutions
To determine if Online proctoring changes the assessment outcome for online students Online proctoring is an effective tool to use for distance student testing Students find the online proctoring service to be acceptable and easy to use.
On-campus students using the Testing Center Distance students taking the exam without a proctor with a set timer and instructions not to use outside materials Distance students using an online proctoring service (Proctor U).
ProctorU is an online proctoring service that allows exam takers to complete their assessment at home while still ensuring the integrity of the exam. Students can take exams from home seven days a week, connected to a live person during their exam. ProctorU is a member of the National College Testing Association. Proctors ensure common sense academic integrity standards by monitoring the student via web cam, monitoring what they are doing by watching the test takers screen in real time, and authenticating the student s identity using a multifactorial process. http://www.proctoru.com
Track N Female Male On-campus 48 46 (96%) Online - Proctored Online Not Proctored 47 44 (94%) 98 93 (95%) 2 (4%) 3 (6%) 5 (5%)
Group N Age Black Hispanic Other White Oncampus Online - Proctored Online Not Proctored 15 25.53 0 0 0 15 (100%) 47 33.94* 4 (9%) 38 34.32* 3 (8%) 3 (6%) 2 (5%) 3 (6%) 2 (5%) 37 (79%) 31 (82%) *p < 0.0001
Sum of Squares df Mean Square F Sig Between Groups 2854.897 2 1427.449 15.609.000 Exam Within Groups 17375.476 190 Total 20230.373 192 Between Groups 2.459 2 1.229 9.463.000 GPA Within Groups 24.685 190.130 Total 27.144 192
Track N Mean Assessment Exam Score SD On-campus 48 72.08 72.08 Online Proctored Online Not Proctored 47 68.04 9.859 98 77.51 a,b 8.281 a p = 0.014 b p < 0.0001
Track N GPA SD On-campus 48 3.25.45924 Online Proctored Online Not Proctored 47 3.51 a.33736 98 3.51 b. 31357 a p = 0.007 b p = 0.002
Online proctoring was an effective tool for distance student testing. All students were able to complete the exam in the specified time period Students found online proctoring to be easy to use and had no concerns about using it for testing. No student complaints Some students indicated they thought it was easy to use and a good idea.
Since this exam does not affect student grades, it is concerning to see that proctoring made such a significant difference in assessment exam scores for these online students. We are in the process of selecting a valid proctoring system for all online exams in our program for student-learning outcome measures for which grades or program assessment are dependent. It will be interesting to see if the GPA is affected for online students when all exams in online courses taken at UNC are proctored
Application of these academic integrity results could be used by other online programs/classes, as student identification and security of exams could be barriers for others to provide online learning experiences.
Proctored students delivered online instruction performed similarly on the end program assessment exam as students taking the same course work through classroom instruction. A valid proctoring system for online exams is crucial for all student learning outcome measures for which grades or program assessment are dependent.