QM Research Grants FY11 The Development of Technological Pedagogical Content Knowledge (TPACK) in Instructors Using Quality Matters Training, Rubric, and Peer Collaboration: Cheryl Ward, Principle Investigator, University of Akron Findings: Pedagogy is central to the quality development of online course design and that the TPACK (Technological, Pedagogical, Content Knowledge) conceptual framework is key in enabling instructors to develop new schema for a re-conceptualization of content, pedagogy and technology. Posited that use of the Quality Matters process helps instructors develop this complex knowledge that enables them to discuss, develop and implement more effective online learning. Therefore, the purpose of the project was to study the process of how the QM Rubric and QM training help instructors develop TPACK knowledge that enables them to discuss, develop and implement more effective online learning. Here are the research questions that guided the study: Is the QM rubric consistent with the TPACK framework to help instructors construct knowledge in quality design and online instruction? How is the QM rubric implemented and integrated as a catalyst to inform and guide online instructors for quality design and instruction? An alignment between the QM Rubric and the TPACK conceptual framework was done to determine if any gaps existed between the rubric and the six areas of the framework. Three professors of instructional technology and three instructional technology students did independent alignment processes with the TPACK framework and the QM Rubric. One premise of this study was that the QM Program can inform and facilitate knowledge growth in the TPACK areas. The alignment indicated that the QM Rubric is fairly well aligned with the TPACK conceptual framework. It was interesting that a rubric that purports to only address the design elements of an online class aligns so highly with the pedagogical elements in the TPACK conceptual framework. This alignment supports initial contentions that the elements of the QM Rubric foster discussion and knowledge development in more areas than just design of these environments. Technological, pedagogical and content discussions are overlapping and connected in a dynamic way that result in the inability to discuss or work on them in isolation for online course development.
Four participants were recruited after they finished QM training. Multiple data resources collected in the study revealed that becoming online learners themselves for the QM training helped the participants understand the needs of online learners. The QM Rubric increased the instructors knowledge of the importance of aligning learning objectives to assessment, instructional activities, and technology integration. Even though QM training and the QM Rubric did not specifically introduce the Technology Pedagogical Content Knowledge conceptual framework to the instructors, it is clear that their knowledge increased in the areas of technology, pedagogy and content. The learning experiences they shared with the researchers about the QM training also demonstrated that they grew to be more sophisticated online instructors because of the way they designed, modified and implemented their online courses through the knowledge they gained. The data analysis results from this study suggest a developmental model that depicts a few key transitional points in order to become effective online instructors, and how QM training can effectively consider these transitional points to deliver the training more efficiently to enhance the quality of online courses with more explicit guidelines to not only course design, but permeate to the other aspects of online teaching and learning. Evaluating the Impact of the Quality Matters Review Process on Student and Faculty Perceptions of Course Quality: Tina Parscal, Principle Investigator, University of the Rockies Findings: Research Questions: What is the extent to which QM certification impacts student perceptions of a course? Which standards have the greatest impact on student perceptions of a course? In an enterprise model of course development, what do instructors perceive as the positive and negative aspects of teaching the same course before and after the QM course review process? UOR commitment to QM at institutional level: Although a QM review is generally a 6-8 week process per course, as the phases dictate, chronological progress is dependent on the outcome of each particular state. If course suggestions need to be applied, the overall length of the process would grow accordingly. These dependencies are critical to the overall lifecycle, but also impacted the data collection phases and survey launches associated with the grant research project. 10-12 participants (faculty and staff) are selected per quarter to complete the APPQMR training. Standard with the most impact: All 8 standards identified at least once Strengths of QM process: Collaboration (4), Alignment (4), Consistency (4)
Student survey responses: 4.3 (.68) The instructions on how to get started in the course were clear. (Standard 1) 4.18 (.88) The course outcomes seemed achievable. (Standard 2) 4.31 (.82) The course requirements for student interaction were clearly stated. (Standard 5) Unintended results of this study: In this research we aimed to determine student and faculty perceptions of quality as a result of the QM review process, we discovered that the majority of our courses are QM recognized on the first submission. Although this is not supportive of the grant research process, per se, it does indicate that embedding QM standards in the tools and templates that faculty and the curriculum team use to develop online courses are effective in support of the development of high quality courses. This has led to a high level of student perception of quality on pre-tests. The Faculty Experience in Preparing Faculty to Use the Quality Matters Model for Course Improvement, Carol Roehrs, Principle Investigator, University of Northern Colorado Findings: A descriptive qualitative study to gather information on most efficient and effective ways to use the program by examining the experiences of faculty with QM training and updating of their online courses. Six faculty members volunteered, two for each level of QM training [self (provided with QM Rubric only), short (attended informal QM training session, no follow-up support) and long (attended QM training and follow-up support made available)]. Following training, each course was self-reviewed by the study participant, informally peer reviewed, updated to reflect recommendations and officially QM reviewed. Participants were asked to comment on their experiences. An increase in Rubric scores from peer to QM reviews was seen as evidence that participants were able to improve their courses. Taking QM training helped faculty improve their courses, but how to change the course to meet the standards was sometimes unclear to participants, and the help of instructional designers in course improvement was valued. There was a trend toward more agreement of the self- and official-reviews for faculty who participated in the longer training and for the faculty that had more experience teaching online. Participants reported that they found QM principles and the Rubric to be a valuable approach to improving their online courses.
Feedback from the faculty peer reviewers was seen as helpful, as was feedback from the official QM review teams. Participants lamented the lack of time they had to learn how to use the Rubric and to participate in the study. Despite their ability to make improvements, instructors wanted to have help from experts when they were updating their courses so they could get things right the first time they worked on the course. The experiences of the faculty were positive overall. They liked the Rubric and whatever training they had. The pervasive theme in faculty comments was lack of time to assess and revise their courses, much less to participate in the study, which implies that new approaches or processes may need to be devised to support faculty efforts to improve their courses. These findings support administrative efforts to develop and support faulty peer review programs, perhaps starting with self-review, and to provide expert help with instructional design for faculty who are seeking to improve the quality of their online courses. Impact of Quality Matters Recognition on Course Attrition Rates, Salvatore Diomede, Principle Investigator, Cleveland State University Findings: Attrition in distance higher education is a common problem. Although recent studies have pointed to more and more colleges improving their online retention rates (Parry, 2010). There are many factors that go into determining whether or not a student is likely to continue with his or her studies. Diaz and Cartnal (2006) identified four broad categories of factors that help explain and predict attrition in distance education: student situations, student dispositions, institutional systems and course content. Course design, which QM attends to, falls under the category of institutional systems factors. Although course design is important, this prediction model shows that other factors, both internal and external to the student, play a role in explaining attrition, too. This project was an attempt by the research team to organize a baseline study examining whether course design has an impact on course attrition. In this study 11 courses were reviewed and improved during one semester (level of improvement was not detailed) and offered by the same instructor the next semester. While there is no statistical evidence that QM impacts students online experience and/or withdrawal rates, students responses to open-ended questions about their online experience do suggest QM standards are important to their positive learning experience. Student comments reflect three major themes: In order to reduce students frustration, courses should have clear course/unit-level objectives, organization, and instructor expectations; additionally, clear instruction on how to proceed and where to find various course components should be provided. Many
participants (especially those participants from non-qm courses) expressed concern about organizational and navigational issues. To promote students positive experience in an online course, sufficient interaction and engaging activities or materials are also necessary. Nonresponsiveness of instructors was noted as a frustration. Unfortunately it appears that some instructors believe online teaching is about presenting reading materials/notes and then testing what students have learned. From both QM standards and students perspectives, more interactive activities and materials are necessary, such as synchronous online interaction, case studies, videos, etc. While QM standards might facilitate students positive online learning experience, without an instructor s positive presence, students are very likely left with a negative online experience. Initially, we (researchers) were surprised to see quite a few very negative comments about online courses (even some QM-recognized courses). After more detailed examination of students feedback, we found that students were talking about instructor presence, which is more of a delivery-related aspect, rather than course design itself. Therefore, even if a course is well designed, it does not mean students will automatically glean a positive experience from the course. A well-designed course still needs an instructor s creative and diligent delivery. Discussion: Although this study as completed in a short, two-semester "turn-around" did not find statistically significant evidence either supporting or refuting QM s effect on withdrawal rates, there are ways to expand upon the research to tell a greater piece of the story of attrition. One important factor in future research would be to control for the delivery variable - meaning instructor level of interaction with students. According to our survey, instructor presence seems to have a direct effect on students perceptions about their online learning experience. This very likely impacts decisions students are making when deciding to persist or withdraw from a course. Therefore, to further understand the effects of QM recognition on attrition, a more accurate control for variables is necessary. Additionally, a longitudinal study may provide a more complete set of data for comparison. This study had only one semester of data available for the QM-recognized courses, which provided for a limited comparison to the three years of previous data for the pre-qm courses. Further studies could examine multiple semesters of post-qm recognition in comparison to an equal length of time of pre-qm offerings. Quality Matters Research Grants FY10 Project Title: Linking Course Design to Learning Processes Using the Quality Matters and Community of Inquiry Frameworks Researchers: Karen Swan, James J. Stukel Distinguished Professor of Educational Leadership, University of Illinois Springfield
Findings: 1. Results indicate no differences on the CoI survey but differences approaching significance on the outcome measures. On average, students in the spring 2010 class outperformed students in the fall 2009 classes on both the final exam and the research proposal, and their overall grade point averages were slightly higher. None of these differences were statistically significant as measured by independent samples t-tests, but differences in scores on the research proposal approached significance (p=.053). 2. Findings suggest that revising course around stated objectives resulted in better student outcomes related to them, especially concerning the ability to write a research proposal. Although there may be a ceiling effect operating with respect to outcome measures as well, the authors believe that greater numbers (from summer and fall 2010 sections of EDL 541) may yield significant results. (The study is on-going.) 3. Researchers suggested that student performance may have improved because the QM revision led instructors to focus on objectives and the mapping of objectives to outcomes, and that such focus translated into their activity in the course. This possibility will be explored further through qualitative means. 4. Finally, it should be pointed out that findings concerning course outcomes support the notion that the QM and CoI frameworks are orthogonal in nature. The linking of online course design to implementation and learning processes to course outcomes is long overdue in distance education. This ongoing study is not only a first step in that direction, but it employs what are probably the two most commonly used theoretical frameworks in online education in the process. Project Title: Quality Matters Rubric as "Teaching Presence": Application of Community of Inquiry Framework to Analysis of the QM Rubric s Effects on Student Learning Researcher: Anna A. Hall, PhD, Delgado Community College Findings: QM Rubric implementation as new design and organization : 1. Increases teacher and teaching presences 2. Reduces direct management on DB/ personalizes management by moving it to the teacher domain (personal e-mail) 3. Reduces student self-management (i.e., group concerns ) on DB, thus, indirectly reducing student social presence 4. Has a positive effect on student higher-order cognitive presence via higher teaching presence 5. Teacher presence, student social presence and QM positively affect course satisfaction 6. QM and higher-order cognitive presence positively affect DB grades
Project Title: The Impact of Quality Matters Standards on Student Perception of Online Courses Researcher: Evelyn Knowles, Park University Findings: Differences when surveying before/after by students and QM master reviewer 1. The certified master reviewer found that both courses did not meet many of the standards. However, the vast majority of students disagreed and concluded that the course met the standards. Yes, Yes, Yes answers on survey. 2. Discussion why the surprising differences? Since the students scored all of the QM standards positively on both the initial and redeveloped courses, this may indicate a discrepancy in their level of expectations when compared to the certified master reviewer s expectations. Perhaps, students did not access or read the content items necessary to meet the QM standards. Another possibility is that during the teaching function of delivery the "instructor" contributed clarifying statements and directions. Project Title: Maricopa Quality Matters Program Review Researchers: Christy Alarcon, Jennifer Strickland, Shelley Rodrigo; Maricopa Community Colleges 1. Survey and a focus group conducted by the Maricopa Center for Learning and Instruction of 20 of 67 QM-certified faculty and administrators for baseline data on impact of QM participation and dissemination throughout the expansive Maricopa Community College system (10 colleges, 4,000 faculty, 250,000 students). 2. Knowledge gained during QM training was shared informally through conversations (100%). A majority of responders reported sharing QM information at department committee meetings (75%), at college-wide meetings (60%) and by distributing the QM Rubric to colleagues (65%). 3. Alignment between learning objectives, outcomes/competencies and course materials was the area that had the most impact due to the QM training. Most faculty and administrators realized that alignment was the key thing to make the course design effective. 4. A prominent theme that emerged was that the biggest obstacle for faculty to develop online and hybrid courses is funding by their respective colleges. Our data has shown that the colleges with the most success with QM were those that implemented a funding model structure. 5. The creation of EDUlearning Community at MCC allowed faculty to become familiar with the QM process before deciding to submit their course for an official QM review. This is a department initiative that creates a sense of community among online and adjunct
faculty. Project Title: QM Standard 8 Pilot Project Researcher: Ed Bowen, Dallas TeleCollege Recommendation: Accessibility is as much about the user as it is about the course. Student involvement in accessibility efforts is strongly suggested. This project was designed to gather student input on thoughts and ideas about accessibility.