Creating Rubrics for Distance Education Courses

1 Creating Rubrics for Distance Education Courses Introduction Kasey Fernandez University of Hawai i at Manoa 250 Campus Road Honolulu, HI 96822 kalcott@hawaii.edu Abstract: Rubrics have been proven to be an effective form of assessment in various educational settings. In distance education, rubrics are even more important, due to the lack of face-to-face communication. In order to help distance educators use rubrics in their courses, an online instructional module was designed to teach the basics of rubrics for distance education. The module was reviewed by educational technology graduate students interested in teaching distance education to test for effectiveness and relevance. Although participants were already quite knowledgeable about the topic, there was a slight increase in test scores after completing the instructional module. Participant feedback suggests that the module s content was valuable, however the module could be improved by adding tangible formative assessments and guided writing practice. Distance education can be an excellent medium for learning but in order for it to be effective, distance educators need to be mindful of the best practices of distance education and incorporate them into their own curriculum. Due to the geographical separation of the Hawaiian Islands, and the limited number of outer island Universities many outer island residents take distance education courses to complete degrees from Universities that they cannot easily commute to. Unfortunately, there is a higher dropout rate in distance education courses compared to face to face scenarios. In various studies, the attrition rate of students enrolled in distance education programs is often considerably higher than the comparable campus based student attrition rate (Simpson 2010, Patterson & McFadden 2009, Nash 2005). In a study by Patterson & McFadden (2009) 43% of the MBA candidates enrolled in the online track dropped out, compared to only 11% of the campus based students in the same major. A similar trend was found in Communications majors of which 23.5% of online students dropped out compared to a 4% attrition rate of campus based students. The researcher, a distance education student who has taken over 30 online courses from colleges in the state of Hawaii, suspects that some of the frustration that students experience in distance education courses may be caused due to a lack of assignment clarification, specification and communication. Student attrition and frustration in distance education it is a complicated issue, however using the best practices of distance education should help reduce learner frustration. Ambiguity of assignment expectations

2 can be misleading and ultimately damper student achievement. In distance education courses, much of the instruction is given in written form so it is important that assignment documents provide directions that are very clear. This is especially true for asynchronous courses. Cindy Rippe (2008) has similar opinions stating that: It is one thing to have a misunderstanding about an idea in discussion and quite another for a student to misinterpret a major project. If a final project is not communicated effectively, this could affect a student s grade and overall online learning experience. If a student has a negative learning experience, this may adversely affect retention. Instructors must go above and beyond to clearly communicate assignment directions and expectations. The best way to communicate these expectations is through rubric creation and implementation (Rippe 2008 p.3). A web-based instructional module called DE Rubrics: Creating Rubrics for Distance Education Use was developed to help potential distance educators use rubrics in their courses by providing them with enough information that they feel comfortable doing so. It was made with the intention to become a resource for a wide variety of distance educators and potential distance educators from elementary to college level. Teachers with years of teaching experience, but little distance education experience and distance educators who do not have an Education degree could also benefit from the module. Background Using rubrics is considered good teaching practice across the board. In Distance Education (DE) courses rubrics often become even more important. In order to clarify assignment requirements, the Western Interstate Commission for Higher Education (WICHE) Cooperative for Educational Technologies recommends providing a rubric or detailed checklist for every graded assignment that is given in a distance education course (Western Interstate Commission for Higher Education, 2009). WICHE (2009) recommends that these rubrics are given out at the beginning of each DE course so that students understand exactly how they will be graded before coursework begins. Not only does a well-created rubric or checklist make it easier for students to understand the specific requirements of each assignment or task, they also make it easier for teachers to fairly assess student learning. Well-developed rubrics allow teachers to assess students more objectively, reducing any biases that the instructor may have towards different styles of learners. Reeves and Stanford (2009 p.27) assert that classroom usage of rubrics could lead to a decrease in student confusion and increase in learning thus contributing to a win-win situation for everyone. Rubrics can be used for all types of authentic assessments, and recently distance educators have been investigating the value of using rubrics to assess student participation in online courses. Penny and Murphy (2009) found that using rubrics to assess forum participation in a transcript analysis project encouraged college students to reach a higher level of discussion. They suggest writing forum discussion rubrics to specifically address higher order thinking skills, stating that:

3 The inclusion of performance criteria and ratings that emphasize interactions with others, particularly the ability to share reflections, insights, information and resources with other members of the group, may aid in social bonding and lead to increased cognitive activity (p.816). This is particularly important since DE courses can easily overlook these areas due to the physical separation of the teacher and students. Another way to promote student interactivity and success in any course is to include opportunities for peer review. Murphy (2007) and Walvoord et. all (2008) support the use of rubrics for peer assessment. In Murphy s study, students at the Massachusetts Institute of Technology used rubrics to assess each other in an introductory physics class. Many students in the study found that they were able to learn how to correct errors in their own problem solving by using a rubric to correct their peers (Murphy, 2007). The above techniques promote the use of rubrics in DE courses and were included in the module. Participants The participants in my module included graduate students at University of Hawaii (UH) Manoa who are enrolled in the Educational Technology program as well as a few graduate students from other Universities. These students represent a diverse population of individuals including a large age range. All 14 participants have access to computers with reliable Internet connections and have experience using them. Many of these individuals aspire to become distance educators in the future. This module was designed to be especially helpful for those enrolled in the Certificate in Online Learning and Teaching (COLT) program. The majority of the participants reside in Honolulu. Other cities represented include Chicago, Philadelphia and San Jose. Eight of the participants were female and the other six were male. The majority of the participants were between the age of 25-39, while there were also quite a few participants in the 40-55 age range as well as two participants under the age of 25. The majority of the participants had previous teaching experience (11 out of 14) and seven of them had been teaching for more than five years. Three of the participants reported teaching a distance education course prior to completion of the module. Although only a few of the participants had prior experience teaching a distance education course, many of them had extensive experiences as a distance education student. Ten of the students (71%) had taken five or more distance education courses previously. Most of the participants had experience writing rubrics before reviewing my module however five of them had none. Methodology Prospective participants in the study were contacted by email. All participants signed an electronic consent form prior to participation. Each participant received a Starbucks egift card worth five dollars to compensate them for their time. All correspondence was confidential and the quizzes were filled out using codes to track pre- and post-tests rather

4 than names. The codes were numbers between 0-1000 that were randomly generated using a widget connected to the modules website. The Wix website creator was used to design the module for this instructional design project. Wix was chosen due to the flexible layout and the page arrangement options. When the module construction began Wix was only offering flash based websites, but has now added HTML5 that creates web site that are accessible on mobile devices. Future iteration of this website would be created using HTML 5 so that the entire site is accessible on mobile devices. To make the module appeal to various learners a variety of media such as videos, written examples and diagrams were used. The module was designed with minimal amounts of text on each page so that each section would incorporate interaction in the form of clicking from one page to the next. In this sense the module was designed to resemble an online tutorial or web quest rather than a traditional website. This was also designed to chunk the content. The module can be found on the website www.derubrics.com. The most complex section of the DE rubrics module was the How? section which covered the ways that one can create rubrics to use in DE settings. In order to teach how to write an analytic rubric and a holistic rubric, online guidebooks were provided which the users can flip through. Figure 1 shows the web page displaying the cover of the Holistic Writing Rubric guidebook. This was then reinforced with examples of the completed rubrics whose components were used as models in each of the books. Figure 1: Holistic Writing guidebook The tests and surveys were created using Wufoo forms. This particular tool works well with the Wix program as the forms are easily embedded into the pages. A disadvantage of

5 Wufoo forms is that there is no way to indicate an answer as right or wrong. Fortunately, there is an analytic section in which one can see the percentages of each response and it is easy to export the results into Microsoft Excel for analysis. The tests given comprised of a demographics survey combined with a pre-test and an attitudinal survey combined with a post-test. The pre- and post-tests were identical and included multiple choice and true or false questions while the participants attitudes towards the module were assessed using Likert scale questions. Some of the true/false responses to the questions on the pre- and post-test could vary from what is commonly thought of as the correct answer due to each participant s personal educational philosophy although all of the multiple choice questions were undeniably right or wrong. Results Results showed that the majority of the participants knew quite a bit about rubrics before completing the module. Despite this fact there was still some growth, the most significant being the identification of the main types of rubrics: holistic and analytic. Table 1 displays the results of the first five pre- and post-test questions which were delivered in a multiple choice format. Table 1: Results of Multiple Choice Questions Question Pre Test (# correct) Post Test (# correct) Which of the following are the main categories of rubrics? 5/14 (36%) 14/14 (100%) A rubric is... 13/14 (93%) 14/14 (100%) Rubrics are useful for... 13/14 (93%) 14/14 (100%) The first step to creating a rubric is to... 10/14 (71%) 12/14 (86%) The last step to creating a rubric is to... 10/14 (71%) 12/14 (86%) Table 2 summarizes the results of the True/False questions included in the Pre- and Posttests. Table 2: Results of True/ False Questions Question Rubrics need to be re-written every time the same assessment is assigned. Pre Test (# chose True ) Post Test (# chose True ) 5/14 (36%) 3/14 (21%) It is more important to use rubrics in face to face 0/14 (0%) 0/14 (0%)

6 instruction rather than in distance instruction. In most cases, it takes more time to write a rubric than it saves. In most cases, describing assessment details is just as efficient as using a rubric or checklist. 2/14 (14%) 3/14 (21%) 6/14 (43%) 5/14 (36%) After reviewing the attitudinal survey it appears that the learners generally liked the module and all either agree or strongly agree that the module was easy to navigate, appeals to a variety of learners, explains why rubrics are important, increased their confidence when it comes to writing rubrics for DE courses, and that the content was presented in a logical order. All but one agreed or strongly agreed that the module presented them with content that they will be able to use in their practice in the future, that the amount of instruction was appropriate, that the design is pleasing to look at and that they could create a rubric for any assignment using the resources provided in the module. In each of these areas one person was Undecided. There was only one area in which one person Disagreed and another was Undecided. This was in response to the question, The module s text was easy to read. This was initially puzzling to the researcher because the text in the instructional module was rather large and only standard fonts were used. After further contemplation the researcher found that the site dimensions were smaller than desired and they were later adjusted. This has most likely resolved the issue. Many of the participants liked the layout, flow and the linear progression of the module, how the information was chunked, as well as the pacing. The online guide books that were used to teach the steps of how to write an analytical and a holistic rubric received a large amount of positive feedback, although one person would prefer to have a hard copy of the instructions. When asked what they would improve about the module, the participants suggested that the module could use some sort of formative practice assessments such as mini quizzes or other activities embedded throughout the module. Some students also thought that there was some text that could be trimmed and replaced with other media. One student suggested that it would be helpful if the module loaded quicker. The researcher speculates that the loading issue may be resolved if the HTML 5 format was used, instead of Flash. Discussion Overall, participants reacted positively to the module, although there are areas for improvement. It appears that the content was appropriate for and relevant to the target audience. Many of them stated that they had learned something from the module, even though most already had experience in the subject matter.

7 A common complaint about the module was that it should include some sort of formative assessment to reinforce the concepts taught. It is something that had been considered prior to releasing the module, but unfortunately time ran out and implementation needed to begin. Once the research had begun the module was reviewed as is in order to test what really needed to be fixed after all students had attempted to complete the same module. To resolve this problem, review quizzes and games will be added to the module to provide students with formative self-assessments before future implementation. Another option to further instruction would be to require participants to create a rubric of their own which would be graded by the instructor after completion of the module. This would provide the learner with rubric writing practice. This was decided against for the purpose of this research because the module is quite long to begin with. During the first round of research participants were all busy students and professionals, many of whom were also implementing their own research at the time so it would have been too much to ask of them. In the future if the module is assigned as part of a college course, the students will be required to build a rubric of their own following the steps provided in the module. This assignment will reinforce the concepts taught in the module, and analysis of the student created rubrics would provide valuable insight concerning the effectiveness of the module. It would also provide the students with feedback so they could improve their rubric writing skills. In order to make this task easier for students, a printable version of the rubric writing instructions would be included as suggested by one of the participants. Additional post assessment questions featuring examples of rubrics which the students would need to judge as being correct or incorrect could also enhance the module. This would provide another way to assess the student s rubric writing capabilities. Conclusion DE Rubrics appears to be a valuable resource for prospective distance educators, although this study shows that there are a few things that could be adjusted which would most likely make the module more effective. Students would appreciate formative review assessments in order to solidify their learning. Also, results suggest that participants would learn more if they had to create a rubric of their own which would be assessed after completion of the module. In order for the module to reach its full potential it may be best to implement the module as part of a required course that would provide additional time, discussion and supplemental resources for teaching this complex subject matter. The module is versatile enough that the course could be part of a graduate degree, certification or professional development series to name a few. When the module is adjusted according to the procedures above it will be released online to serve as an open source resource for distance educators and distance education students. It will be reformatted in a way that will allow it to teach to a broader audience but will cover the same content. If the researcher finds a new opportunity to test the module, the second version of the module will be evaluated in order to continue to

8 improve and update the module. Then the test results of each round of research will be compared. The researcher believes that the above adjustments will enhance the module and there will be a positive increase in test scores and student retention if the updated version of the module is implemented in the future. References Murphy, S. (2007). Peer-assessment of Homework Using Rubrics. AIP Conference Proceedings, 951(1), 156-159. doi:10.1063/1.2820920 Nash, R. D. (2005). Course Completion Rates among Distance Learners: Identifying Possible Methods to Improve Retention. Online Journal of Distance Learning Administration, 8(4). Retrieved from http://www.westga.edu/~distance/ojdla/winter84/nash84.htm Patterson, B., McFadden, C. (2009). Attrition in Online and Campus Degree Programs. Online Journal of Distance Learning Administration, 12(2). Retrieved from http://www.westga.edu/~distance/ojdla/summer122/patterson112.html Penny, L., & Murphy, E. (2009). Rubrics for designing and evaluating online asynchronous discussions. British Journal Of Educational Technology, 40(5), 804-820. doi:10.1111/j.1467-8535.2008.00895.x Reeves, S., & Stanford, B. (2009). Rubrics for the Classroom: Assessments for Students and Teachers. Delta Kappa Gamma Bulletin, 76(1), 24-27. Rippe, C. (2008, July). Using Rubrics to Improve Teaching, Learning, and Retention in Distance Education, Online Classroom, 3-5. Retrieved from http://www.vcu.edu/cte/resources/newsletters_archive/oc0807.pdf Simpson, O. (2010, July16). 22% - can we do better? - The CWP Retention Literature Review Final report. Retrieved from http://www.ormondsimpson.com/userimages/retention%20literature%20revi ew.pdf Walvoord, M., Hoefnagels, M., Gaffin, D., Chumchal, M., Long, D. (2008). An Analysis of Calibrated Peer Review (CPR) in a Science Lecture Classroom. Journal of College Science Teaching, 37(4), 66-73. Western Interstate Commission for Higher Education (2009). Best practice strategies to promote academic integrity in online education v 2.0. Retrieved from http://www.wiche.edu/pub/13441