Quality Matters Rubrics: Ensuring Quality for Online Course Development

Transcription

1 Running Head: QUALITY MATTERS RUBRICS 1 Quality Matters Rubrics: Ensuring Quality for Online Course Development Team LARJ: Tiffany Linc, Adriana Greenlief, Roxanne Witherspoon and Julia Cutler The University of Akron

2 QUALITY MATTERS RUBRICS 2 Quality Matters Rubrics: Ensuring Quality for Online Course Development Virtual learning is a new movement in education that will continue to grow rapidly. Teachers need to recognize this growing trend and begin to understand it. As proven in the many articles, videos, and documents that are linked in the Quality Matters ( QM ) module, learners will quickly learn that online learning is neither temporary nor insignificant. The need for quality online instruction is evident. The goal of this module is furnished to the students that the purpose of QM is to enable learners to effectively use the QM rubrics to evaluate online learning content materials. From this point, the course of instruction can be successfully delivered if the instructional design process has been carefully considered. The process begins with learner analysis. In order to gather information to guide instruction, an online survey was sent out to Instructional Technology Master s students from The University of Akron. The data gathered from the survey informed instructional designers that the majority of the learners are between the ages of 26 and 35 years old and thirty percent are middle school educators followed by elementary educators at twenty percent. When asked about work experience, more than half of those responding to the survey have 1-5 years of experience in a field that could possibly involve designing online instruction. This response does not necessarily mean that they are currently designing online instruction but that they could be in the future. In an effort to gain even more insight on experiences in creating online instruction, learners were asked how often they have developed online instruction. More than seventy-five percent of the learners responded that they have created at least four online instructional experiences in some capacity. When the students were asked about their comfort level with various productivity tools (such as presentation software, contributing to Wikis, participating in

3 QUALITY MATTERS RUBRICS 3 online discussions, learning in an online environment, and integrating technology into curriculum), more than ninety percent responded that they are either comfortable or very comfortable. Before implementing instruction, more than fifty percent would either have a peer look over their materials or use their own best judgment in evaluating materials for quality. The second form of analysis that was performed was needs analysis. Needs analysis was used to identify performance gaps and to make effective intervention recommendations (Morrison, 2007). Our data provided information that 59.4% of the learners feel that evaluation of online materials is very important. However, the learners also indicated that a large number relied on peers (28.1%) to evaluate their materials or used their own judgment (25%) to determine the quality of their instruction. For the students who used peer review, implementation would be especially helpful as Quality Matters is a faculty-centered peer review process (MarylandOnline, 2010). Furthermore, most learners limited the methods used to evaluate the quality of the materials to ITSE NETS standards (76.7%) or the Common Core Standards. A minority of learners utilized the Quality Matters rubrics or standards (23.3%) to evaluate materials. The data collected exhibits that the majority of learners are not aware of Quality Matters Standards or they do not effectively utilize them. The importance of utilizing Quality Matters is emphasized by the statistic that 84.4% of the learners are likely to integrate technology into their curriculum in the future. Additionally, 35.5% of the learners predict a need to teach their students in an online or virtual environment within the next few years. According to those polled, 34.4% answered that they would have students utilize online resources and instruction by applying concepts. If Quality Matters were implemented into that process a rubric could be used to evaluate the success of the student and lead them to higher learning in accordance with Bloom s Taxonomy. Over fifty-percent of the

4 QUALITY MATTERS RUBRICS 4 survey respondents indicated that they review their materials weekly for relevance and rigor. This practice demonstrates a willingness to revise and improve upon current instructional materials and thus a need for the Quality Matters Rubric. To execute the above goals, a task analysis needs to be completed and a Subject Matter Expert ( SME ) is needed. The SME for this project is Dr. Cheryl Ward. Additionally, research has been gathered to use in instruction and to aid development of an effective design method. The Quality Matters module is organized in a procedural structure. The module requires the learner to complete an ordered sequence of steps to successfully implement the QM rubrics (Morrison, 2007). The steps will be introduced and explored through articles, videos, participation in discussion, and practical application of the QM rubric to existing 6-12 and higher education lessons. The learner will receive cues or feedback to enhance the learning experience within each method of instruction. Cues will be received through peer evaluation and instructor response. If this task analysis is implemented correctly, then the learners will enjoy great success in utilizing the QM rubrics to evaluate online learning. The design phase needs to be well planned to progress with construction of an effective learning module. This phase includes many steps beginning with a task analysis relating to the learners attributes. In the case of Quality Matters, the learners have a range of knowledge and skills relating to online learning. All of the learners have some familiarity as they are currently enrolled in an online graduate program. This enrollment ensures a basic entry-level competency in navigating an online module, but this still does not guarantee a basic familiarity with evaluation of online materials especially at varying grade levels. Learners use of online learning in other capacities varies from complete immersion through occupation to very little exposure outside of the graduate program. As a result it may be

5 QUALITY MATTERS RUBRICS 5 concluded that the learner might have hesitation or reluctance to learn this module due to personal beliefs regarding the relevance of online learning (especially outside of higher education as indicated in the survey). The learners will need to understand the advantages of the Quality Matters rubrics before venturing into the module (Morrison, 2007). Task Analysis: A. Learners will learn that online instruction is growing (specifically grades 6-12 and higher education) o o learners will read documents and watch videos supporting this idea learners will participate in a discussion of misconceptions, previous ideas, or what surprised them to reinforce of online learning's future and validity B. Quality Matters process evaluates online instruction o Learners will be introduced to Quality Matters through online videos and readings. o Learners will participate in a discussion and answer the question: Do you think QM is useful, give a situation where using QM would have been helpful and state why? C. QM rubric for use in 6-12 o o Learners will use rubric to evaluate already created online instruction Assignment will be submitted through a drop box on Springboard. D. QM rubric for use in higher education

6 QUALITY MATTERS RUBRICS 6 o o Learners will use rubric to evaluate already created online instruction Assignment will be submitted through a drop box on Springboard. Through the learning module, various learning theories will be used. The Behavioral Theory will be implemented by feedback being provided during discussions and the evaluation of assignments. By receiving feedback from peers and instructors, learners will be positively reinforced in their endeavors. The Social Learning Theory, which is defined to be the learning from others through observations and modeling (Morrison, 2007) is also utilized. Through this module, learners will acquire knowledge through the experiences of others as they are reading and responding to discussion prompts. Finally, the Cognitive Theory will be used in this learning module as learners are asked to apply the knowledge they acquire to real-world examples. Learners will be applying what they learn about the rubrics to evaluate already created online instruction. Therefore, information is more deeply processed and more likely to be transferred to long-term memory and retrievable in the future (Morrison, 2007). Upon completion of the module the learners will be able to successfully execute the following objectives: Demonstrate knowledge involving the growth and future of online instruction through participation in online discussion. Will be able to analyze online instruction using the QM Rubric (6-12). Will be able to analyze online instruction using the QM Rubric (higher education).

7 QUALITY MATTERS RUBRICS 7 Instructional Strategies utilized in this module will mirror our task analysis and will be completed in a one-week time frame. Strategies will include the reading of articles and watching online videos to overcome attitudes and/or preconceived ideas about the validity and the future of online learning. A demonstration of comprehension will occur in a discussion on misconceptions. As a segue into Quality Matters students will then watch a Quality Matters video that will connect the increasing trend in online learning to the validity of the Quality Matters rubric. Quality Matters will then be further explored through asynchronous instruction involving QM materials and rubrics. This will bring all learners to a basic competency level with the materials and when combined with the previous exercise will illustrate the advantages of QM. This will again be followed up by participation in discussion which allows adults to bring personal experiences and socialize; both of these accommodations are recommended for adult learners (Morrison, 2007). Finally, learners will use QM rubric to evaluate instructional materials for both 6-12 and higher education. This, like the other strategies, will stimulate working memory and maximize effectiveness of online instruction in accordance with Cognitive Load Theory (Edutech, 2009). Overall, this module will enlighten students to the future of online learners and give them the skill to use it effectively. The development phase needed to successfully combine media choice, relevant and important content materials, and assessments (both formative and summative), addressing the Quality Matters module. The first step was to choose media to successfully convey the content materials.

8 QUALITY MATTERS RUBRICS 8 To do so, a learning management system ( LMS ) was needed that allowed learner access to online videos, articles, lessons and assignments. We were able to use Springboard through the University of Akron that offered all of these attributes. This LMS was then modified and sequentially arranged to successfully fulfill the criteria for the course in an organized and ergonomic form. As discussed in the design phase, various articles and videos were needed that supported the future and validity of online learning. A large body of resources was collected and then individual resources were selected based on relevance and support of intended educational outcomes. Once these resources were scaled down to a few excellent options, the module started to take form. Overall, the module was made so that consistency in terminology was persistent throughout. Furthermore, the news section of the LMS was used as an initial presentation to give the learner an overview and the information needed to achieve the objective (Morrison, 2007). To ease the transition between categories in the modules, artifacts were arranged in an order that created a transition to the next subject (for example, placing the Evolution of Online Learning Video just prior to the QM and Online Learning section). The development also allowed for a variety of delivery methods in order to reach all of the learners learning types. Small-group was incorporated through the use of discussion, self-paced instruction was implemented by having the course take place asynchronously over a given time frame and a loose form of lecture was given by video, and these lectures were supported by articles. In the first segment of the module, learners were to become acquainted with the increase of and the future of online learning. Once this introductory material was reviewed, learners participated in a discussion that enabled them to discuss any misconceptions that they or they

9 QUALITY MATTERS RUBRICS 9 feel society holds about online learning. This discussion allowed learners to use the information found in the materials to support their thinking and participate with others. Through this participation, learners were able to increase their knowledge on the topic through what their peers wrote and respond. The second part of the module introduces the learners to the Quality matters program. Again, learners were exposed to videos and readings that gave them an overview of what the program is and the rubrics designed for assessing online courses. Once the course material was reviewed, learners participated in a discussion that allowed them to state whether they thought the rubrics had the potential to be useful and if they have had any experiences that they believe would have benefited from the implementation of the rubrics. The third and fourth parts of the module were actually parts of online courses that the students were to assess using the QM rubrics. In each, the first link encountered allowed the learners to know about the assignment that they were to complete while experiencing the online course material in both a 6-12 course and a higher education course. These assignments had learners evaluating the course using just a portion of the Quality Matters rubrics; those standards that are absolutely necessary in order for a course to even be considered worthy of Quality Matters approval. All of the latter heuristics, coupled with the methods of delivery, and content material selected will ensure an effective learning experience for the participating IT students. To assess that the learning objectives were met, two discussions and two assignments were completed. The discussions served as a method for learners to gain information and knowledge from their peers. Instructors were able to view and participate in these discussions to formatively assess their learners and influence the discussion to focus learning if needed. Finally, learners

10 QUALITY MATTERS RUBRICS 10 participated in two assignments where they actually used parts of the Quality Matters rubric to evaluate online courses. These assignments served as summative assessments and helped determine, along with the discussions, if the learning objectives were achieved. In the implementation phase of the ADDIE model, the solution to the design problem was ready to be tested. The proper research had taken place, instruction had been designed and developed, and it was time for full implementation. The learning problem, or issue, for this instructional design project is: Online learning is often created but not evaluated for quality. The learners were to evaluate online learning using the Quality Matters Rubrics. The design team decided to have the learners evaluate a higher ed module and also a module for 6-12 instruction. Before the learners were able to interact with the product, the instructional designers had to test the module for usefulness. Once all resource links (readings and videos), discussion prompts (appropriateness of questions), and assignments (clearness of expectations) were evaluated for functionality, clarity and relevance to topic, the module was ready for implementation with a test group. The implementation of this module was designed in Springboard and was conducted by the instructional design team. The module was developed with the target audience being IT Master s students or anyone who would find the need to evaluate online courses. The test group used in the initial implementation of the module came from the University of Akron s Instructional Technology Program s list. An was sent out asking for volunteers to participate in the one week module that could be used as field experience for their e-portfolio. As the module needed to be completed the following week, students were added to the module for three days as their requests came in. Twelve students were added to the Quality Matters Course Module by Team LARJ. Once students were added to the module, they were sent a welcome that included a

11 QUALITY MATTERS RUBRICS 11 reminder that this module was to be completed during the following week. Throughout the week, it became apparent that learners were unclear as to how to proceed through the module. A clarification of expectations was posted in the News section of Springboard and then ed to each of the participating members. As students participated in discussions, members of Team LARJ read and commented on the posts that were made. One member of the team LARJ posted her own experiences with online learning that could have used evaluation using the QM rubric, when she noticed that students were not responding to the part of the discussion that asked for them to share a time when they were involved with or witnessed a situation when Quality Matters would have been useful. As the week came to a close, a reminder was sent out to each of the students who volunteered to participate. The stated that there were two assignments that needed to be uploaded to a drop box in order for their participation to be completed. Finally, work completed by the learners was graded by the members of the instructional design team. The rubric was used by each member of the team to grade sections of the completed course work. Following grading, the rubric was scanned, turned into a PDF, and sent out to each of the participants via to evidence the completion of the module. In addition to some things that were already discussed, there are other changes that the design team has determined need to be made to this module. In the following section, we will address evaluation of this module and the problems that need to be fixed. When one thinks of evaluation, one usually thinks of the end of something or the last or final portion. In the ADDIE model, evaluation is performed throughout the entire process and after the implementation phase. Evaluation is an ongoing process because the designers would

12 QUALITY MATTERS RUBRICS 12 not want to spend tremendous amounts of time on a project for it to fail in the end. Formative evaluation is found in the entire process of ADDIE. In the analysis phase, evaluation is used to decide if there is an existing problem. Next, in the design phase objectives are evaluated to plan the specific type of instruction or project. The SME evaluates the project frequently to provide feedback and make sure the designers are moving in the right direction. Implementation sometimes appears to be the end of the phase, but it is not quite the end. Finally, in the evaluation stage the designers want feedback from the learners or users. At this point, it is important for the designers to remain objective and to recognize faults in the design instead of blaming the learners inability (Gomoll, 1992). Designers want to know if their instruction or project was useful and if they were successful in solving the problem. As previously mentioned, upon evaluation of the implementation of the module, it became apparent that students were unsure of how to proceed through the module. This may have been because the module was produced using Springboard and learners were less familiar than expected, or because the plan of action was not clearly defined by the design team. This problem was easy to fix. A message was placed in the News section and then ed to students with the hopes of clarifying the module. Another problem that needs to be remedied before future implementation is the existence of links that are important to the example courses, but are not required for learner completion or for their participation in the QM module. When learners clicked on discussions and drop box, they also saw links that were meant to be part of the example modules for 6-12 and higher education caused much confusion. Learners thought that they had to complete assignments and discussions that were only to be examples of online learning elements. An attempt was made to

13 QUALITY MATTERS RUBRICS 13 alleviate this problem by removing the visibility of the links when those tabs were clicked on. During implementation, the unnecessary drop box links were removed but the discussion links were not. This still permitted confusion regarding the discussion and minimized learner insight to the assignments that had involved the drop boxes. Therefore, before future implementation, a way must be found to allow the learners to see the discussions and drop box assignments when they are perusing the 6-12 and higher education modules, but not see them when they are accessing their own assignments and discussions. One way to do this might be to include screenshots of what the drop boxes would look like. Another issue arose with the two learning keys this module; the discussions that learners had with each other over the future of online learning and the use of the Quality Matters Rubrics. The success of this part of the module relied on the interaction that learners had with each other. This allowed learners to share their knowledge and learn from the knowledge of their peers. The instructions given in the discussion prompt did not require for learners to respond to the posts of others. Since the test group was very familiar with the discussion board and in most cases used to responding to at least two of (their) peers, many learners did respond to the posts of others. During future implementation, learners will be given the instructions to provide an initial post and respond to the posts of at least two others. These discussions served as two of four assessments embedded into the instruction to aid in evaluating the success of the module. The first was a discussion prompt created to assess whether or not learners met the intended goal of demonstrating knowledge involving the growth and future of online instruction. During this discussion, students were asked to discuss misconceptions of society and if they were surprised about anything that they found in the resources. Of the twelve students who signed up to participate in the class four students did not

14 QUALITY MATTERS RUBRICS 14 participate in the discussion, two students posted only an initial post, while the rest of the students provided an initial post as well as responding to the posts of at least one other. Based on the discussions, students are well aware of the growth and future of online instruction. Many were surprised about the number of students/schools that currently participate in online learning, especially those that are not higher education. The goal of the second discussion was to ensure that learners became familiar with the Quality Matters Program, understood its use, and saw value in using the Quality Matters rubrics. Once learners went through the learning materials, they were asked to discuss if they believed the Quality Matters Rubrics could be useful. They were also asked to share a situation that would have benefitted from the application of the Quality Matters rubric. Of the twelve students who signed up to participate in the class, four students did not participate in the discussion, two students posted only an initial post, while the rest of the students provided an initial post as well as responding to the posts of at least one other. Based on the discussions, learners did see value in the Quality Matters rubrics both from an evaluation point of view and as a way for educators to learn how to create online learning environments. The last two assessments were formative in nature and were meant to evaluate whether or not learners were able to meet the final learning objectives of analyzing online instruction using the QM Rubrics. This phase required the highest level of Bloom s Taxonomy and involved critical evaluation of a product and a process (Tiene, 2001). Through the use of the rubric to evaluate model courses, learners gained first-hand knowledge of how to use the rubric and thus were able to critique the value, strengths and weaknesses not only of the models but also of the rubric itself. Learners were also asked to include a summary of their experience that highlighted their concerns with the models and the rubrics.

15 QUALITY MATTERS RUBRICS 15 Half of the total intended participants completed this portion of the module. The low participation might imply that the module is too long or intensive for learners. However, the outlook by those who completed the module were positive. Length and intensity might need to be toned down before another implementation. Of those who completed the module most were very astute in their observations of the model courses. They noticed items intentionally absent from the models, and evaluated structure, reading easy, accessibility and other aspects of all major QM standards. To the contrary, one learner consistently missed the mark on both sample courses, but he also prefaced his papers with a note about how busy he is with the end of the school year and how this was not of high importance. With these considerations and the quality performance of students in the QM module, it is safe to say that, with the exception of aforementioned flaws, that this instructional design was successful in achieving its intended goal!

16 QUALITY MATTERS RUBRICS 16 References Edutech. (2009, March 21). Cognitive Load. Retrieved March 19, 2012, from Edutech Wiki: Gomoll, K. (1992). Springboard. Retrieved from The University of Akron: MarylandOnline. (2010). About QM. Retrieved March 19, 2012, from QM Program: Morrison, G. R. (2007). Designing Effective Instruction. Hoboken: Jonh Wiley & Sons, Inc. Tiene, D. &. (2001). Exploring current issues in education technology. New York: McGraw-Hill. Retrieved from Darden College of Education.