Transparency, Accountability, and Assessment of Student Learning

Transcription

1 Transparency, Accountability, and Assessment of Student Learning The Three-Legged Stool Paper presented in track 3 at the EAIR 37 th Annual Forum in Krems, Austria 30 August till 2 September 2015 Name of Author(s) Natasha A. Jankowski Contact Details Dr. Natasha A. Jankowski National Institute for Learning Outcomes Assessment (NILOA) College of Education, University of Illinois Urbana-Champaign 1310 S. Sixth St, Room 364 Champaign, IL USA njankow2@illinois.edu Key words Assessment/Evaluation; Higher education policy/development 1 Transparency, Accountability, and Assessment of Student Learning

2 Abstract Transparency, Accountability, and Assessment of Student Learning Transparency, accountability, and assessment of student learning form a three-legged stool from which institutions respond to the changing nature of education. Pulling from research conducted by the National Institute for Learning Outcomes Assessment, an organization in the United States focused on researching undergraduate student learning outcomes assessment, this paper explores the intersection of increased calls for transparency, accountability drivers shaping the work, and institutional responses to critiques of educational quality. The paper argues that the stool forms a basis upon which institutions craft arguments regarding their effectiveness and value an approach with implications to education systems beyond the United States. 2 Transparency, Accountability and Assessment of Student Learning

3 Presentation Transparency, Accountability, and Assessment of Student Learning: The Three-Legged Stool Institutions of higher education in the United States exist within a political environment requiring the public reporting of information in an effort to enhance transparency to various potential information consumers. Transparency has been viewed by the federal government of the US as a means to ensure accountability in concert with quality assurance processes undertaken by accreditation agencies. This paper argues that as accountability narratives shift over time, institutions modify the information they share and the arguments needed to assure external entities of their effectiveness and efficiency that they are in fact adding value to the students who attend. While this paper explores shifting narratives and institutional responses to them from a US context, the connection between transparency, accountability, and institutional responses in the form of assessment of student learning are salient issues to systems of higher education more broadly. To begin, the paper presents a brief exploration of each of the three: transparency, accountability, and assessment of student learning. Transparency Colleges and universities regularly have to report information on a variety of outcomes to various sources. Information provided includes graduation and retention rates, cost per student, return on investment, and employment outcomes to name a few. Generally, information on these metrics are shared externally in an effort to inform potential students such that they can utilize the information when selecting institutions to attend. Further, the information is thought to drive market-based competition between institutions to improve their outcomes and general efficiency (Jankowski & Provezis, 2014). While students rarely use the information when making decisions on where to attend and the college selection process involves metrics not included in the general reporting requirements, the federal government continues to ask for additional information to enhance transparency. The most recent of these was an attempt to develop an institutional scorecard for which all institutions would submit data. Yet, the view of transparency is one of making information available, rarely with additional contextual information or comparison information to enhance user comprehension. The framing of transparency has thus been one of providing consumer information to drive institutional improvement and foster data-driven decision-making in potential students. Accountability Accountability narratives involve examination of the role and purpose of higher education within the larger national system and educational environment. Of late, the focus has been upon whether institutions are functioning efficiently or effectively in terms of moving students through in a timely fashion and providing graduates who are deemed career ready by employers. Institutions are asked to cut costs, shorten time to degree, and align their offerings with employment needs to help drive economic growth. Thus, institutions are thought to be accountable when they can provide data showing that they are meeting the needs of multiple audiences and producing employable graduates. Institutions are held accountable through accreditation processes, such as regional accreditation, and reports to states. Further, institutions are examined based on the value they have added to learners as a measure to secure a return on investment in the form of attendance being worth the cost to achieve a degree. Assessment of Student Learning Colleges and universities have been assessing student learning since the 1980s when the Department of Education asked the accreditation agencies to examine institutional processes to systematically assure that 3 Transparency, Accountability and Assessment of Student Learning

4 students were learning, and to then utilize that information to improve institutional practices. Assessment of student learning occurs at multiple levels including program and institution, and involves the setting of learning outcome statements, the selection of measures to determine if students are meeting the stipulated outcomes, and use of the collected data to improve. With the diversity of institutional types in the US and the approach undertaken by the accreditation agencies to allow assessment to align with institutional mission, there have not been agreed upon learning outcome statements that transcend individual institutions. Two recent developments on this front have come in the form of the Degree Qualifications Profile currently being used by over 500 institutions and the work of the American Association of Colleges and Universities to outline Essential Learning Outcomes. In the accreditation process, institutions provide evidence that they not only have a plan to assess student learning, but also have implemented the plan and are actively using the results to improve. The Three-Legged Stool It is important to note that transparency, accountability, and assessment are not stagnant topics or processes, but ones that are dynamic and have shifted over time. Moreover, the three are interconnected such that as accountability narratives change, transparency requirements follow suit, and institutions then modify their approaches to assessing student learning. Yet, to this point, the approach has been one of reactionary tactics driven by compliance (Kuh, Ikenberry, Jankowski, Cain, Ewell, Hutchings, & Kinzie, 2015). Thus, this section presents the argument for the connected nature of the three legs before moving into a discussion of how institutions can utilize the understanding of the three-legged stool to provide counter narratives. In 2006, the US Department of Education formed a commission commonly referred to as the Spellings Commission whose report was critical of higher education. The report made the argument that to be held accountable in terms of demonstrating institutional value, colleges and universities needed to be more transparent. The analogy was made between higher education and a consumer purchase of a car in terms of having available information to make informed decisions on the purchase. While some within higher education argued against the usefulness of the analogy, the report further asked that institutions meet accountability and transparency needs through the use of three suggested standardized tests of student learning. Of note was a shift in the accountability dialogue away from presenting input measures as demonstration of effectiveness to one of examining output measures in the form of student learning. Examining student learning shifted the value added from attending college towards one inclusive of the knowledge and skills gained, an area previously unexamined outside of the accreditation processes. Further, it was uncommon for suggestions on how to assess student learning to be included in a department of education report, especially one in which colleges and universities worried a future mandate was possible. Institutions and organizations of higher education quickly responded by creating voluntary initiatives in which institutions would present the requested consumer information and administer the tests (Stassen, 2012). One such approach was the Voluntary System of Accountability (VSA) created by the American Association of State Colleges and Universities and the Association of Public and Land-grant Universities. Others included the University and College Accountability Network, Transparency by Design, and the Voluntary Framework of Accountability. Each of these approaches was a direct response to accountability narratives through designing electronic transparency efforts linked to how institutions assessed student learning. While the institutional response was quick to implement use of the standardized tests and to report the information, several difficulties began to emerge with the alignment of this approach to the principles of assessment of student learning. In an evaluation of the VSA page on student learning outcomes assessment, it was found that while institutions felt they had been responsive to external accountability demands, that the information was not transparent or useful to potential students and that the tests did not provide information which could be used to improve the institution and student learning (Jankowski, Ikenberry, Kinzie, Kuh, Shenoy, & Baker, 2012). In some ways, the strengthening of the leg of accountability forced the other two legs of assessment and transparency to respond to provide balance; however, in a manner that did not align with their purpose or intention. In addition to commission reports driving changes in transparency, accountability, and assessment, accreditation agencies have driven institutional responses in their approaches to quality assurance (Powell, 4 Transparency, Accountability and Assessment of Student Learning

5 2013). Not only has accreditation consistently been viewed as a driver of assessment efforts (Kuh & Ikenberry, 2009; Kuh, Jankowski, Ikenberry, & Kinzie, 2014), agencies have also begun to include policy statements on the transparency of information available to the public. Further, colleges and universities in states that require accountability reporting and have mandated assessment policies are more likely to have common learning outcome statements applicable to all undergraduate students, and to use standardized tests, national student surveys, and employer surveys (Ewell, Jankowski, & Provezis, 2010). The states with policies related to assessment also were more likely to have institutions with transparent reporting of assessment related information. Yet, each of these examples outlines reactions on the part of institutions to external narratives and demands thus they are compliance driven. When one leg drove the others, a balance was not reached and the other legs had to adjust. In the purview of assessment of student learning, institutions developed two processes: one for external reporting and one for internal improvement. However, accountability narratives are beginning to shift from an institution lens to a student level lens such that as students become increasingly mobile the unit of analysis in terms of success becomes the individual student. Prior transparency efforts reported aggregate information for an institution as a whole as opposed to information about programs or specific sub-populations of students. In addition, the assessment efforts designed for external reporting were not able to capture student-level information. In the most recent survey of provosts asking about assessment processes, an internal shift was reported in relation to the types of assessment used (Kuh, Jankowski, Ikenberry, & Kinzie, 2014). Provost respondents reported increases in the use of classroom-based assessments, rubrics, and portfolios, in addition to a widening of the range of approaches to assess student learning from three to five measures. Moreover, when asked which of the approaches used provided the most valuable information, provosts indicated that classroom-based assessments were the preferred source a student-level metric. While accreditation remained the driver for undertaking assessment, the uses of assessment shifted to more internal and improvement related foci from prior years. Institutions were shifting their focus from standardized tests to measures that provided a picture of student learning at an individual and an institutional-level when aggregated (Eubanks & Gliem, 2015; Richman & Ariovich, 2013). Conversations began emerging nationally regarding the design of classroom-based assignments which could be aligned to shared outcomes and provide evidence of student learning across a curriculum (Hutchings, Jankowski, & Ewell, 2014). Furthermore, several states began to examine if the use of rubrics might better inform accountability requirements through the formation of a Multi-State Collaborative (Crosson & Orcutt, 2014). However, rarely were institutions leading the discussion, active in altering the narrative, or intentional in the communication of information through transparent means. Counter-Narrative Development Instead of being driven by compliance efforts and accountability narratives, some institutions are beginning to provide counter-narratives in an effort to reframe and reshape the interaction between transparency, accountability, and assessment. In a cross-case analysis of nine case studies of colleges and universities using assessment results to improve student learning, transparent communication of information was a vital element of their efforts (Baker, Jankowski, Provezis, & Kinzie, 2012). Institutions not only used assessment results to communicate to external audiences, but also internally to advance assessment practice and improvement processes. These institutions engaged in shifting from compliance reporting to communication, but conceived of transparency not as making information available but of communicating an argument about their effectiveness or efficiency (Jankowski & Cain, 2015). They engaged in evidencebased storytelling where evidence of student learning is used in support of claims or arguments about improvement and accountability told through stories to persuade a specific audience. They argued that, as found in other studies (Blaich & Wise, 2011), what was needed was not additional data or wider dissemination, but a question driven approach to exploring information and evidence. Accountability narratives in part may be fueled by a lack of awareness or understanding of why institutions do the things they do as well as how colleges and universities respond to external demands such as accreditation (Ewell & Jankowski, 2015). Further, there may be a disconnect between what counts as evidence for different audiences. Policy makers may view evidence of quality assurance as graduation rates 5 Transparency, Accountability and Assessment of Student Learning

6 while accreditation agencies may view evidence in terms of documented processes and practices, while the general public may be more interested in evidence of employment upon graduation. Thus, targeting the argument to specific audiences allowed for the development of a narrative supported by evidence that was meaningful to the audience in question but also useful to the institution. The approach is very similar to the Toulmin s (2003) argumentation model whereupon an argument is made based upon the evidence used to make the argument, the claim or the point being argued, and the warrant which provides the connection between the claim and the evidence. Where institutions had not been active in the past involved in the claim and the warrant. While a variety of evidence was available, it was not being utilized in communicating with various audiences. Counter-narratives are available to shift the discourse once institutions began to actively compile the available data, make meaning of it in terms of what it means for the institution and the students. Then, institutions are in a position to argue that they are utilizing X opportunities or resources for Y students at this point in time, because they believe that it will either improve student learning or demonstrate their accountability. The inclusion of the warrant into this approach to argumentation and a vital element of the evidence-based storytelling involves outlining why an institution or program believes this to be the case. Why is it that they think providing education in the way that they are to the students they serve will lead to the outcomes in question or will demonstrate they are being accountable? The why is where the narrative can begin to shift. It was from communication between institutions and policy makers in the state of Massachusetts that the Multi-state Collaborative focused on alternative approaches to accountability and assessment emerged. It is also how institutions are able to navigate accreditation self-studies by explaining to the reviewers why they are engaging in the processes they are, not just that there are a variety of activities occurring on the campus but they are intentional and there is reasoning and evidence behind them. It can also serve to shift accountability and transparency from a compliance exercise to one of meaning to the institution. Conclusion and Implications This paper has put forth the argument that transparency, accountability, and assessment of studentlearning form a three-legged stool from which institutions may be active participants through crafting arguments regarding their effectiveness, value, and efficiency. The legs of the stools are connected and dynamic shifts in one leg lead to movement of the others to maintain a balance. Institutions, however, have been less active in the shifts and more responsive in nature. Yet, if the legs move apart, the stool will fall, and as the narratives change over time institutions are active participants in their environment and capable and able to effect change. The data reporting needs, what counts as evidence, and the audiences and arguments made change over time. Institutions who are active in the forefront of sharing their narrative, communicating about the approaches they are taking to help students learn and assessing that learning are able to present evidence in a larger argument regarding their efficiency and effectiveness. Implications of counter-narrative development and participation in the balancing of the three-legged stool are applicable throughout higher education. If our various audiences are not aware of or understand why we do the things we do, then there is space and opportunity for misconceptions to enter and for negative narratives to emerge. Several possible examples where this approach may be applicable is in responses to the rising cost of education, the value of attending an institution, the way an institution is run or operated, or even which students should attend. Being clear about the intentionality behind the institutional design of the educational experience may provide more meaningful information than consumer transparency efforts make available currently. Further, to make external arguments about institutional effectiveness, internal conversations around the data must occur. Such conversations foster a culture of data-driven decisionmaking and help to advance understanding of institutional mission and culture. Just as in classrooms of higher education, learning advances when we explain a concept to another classmate, colleague, or external audience. 6 Transparency, Accountability and Assessment of Student Learning

7 References Baker, G.R., Jankowski, N., Provezis, S., & Kinzie, J. (2012, July). Using assessment results: Promising practices of institutions that do it well. Urbana, IL: University of Illinois and Indiana University. National Institute for Learning Outcomes Assessment (NILOA). Blaich, C.F., & Wise, K.S. (2011, January). From gathering to using assessment results: Lessons from the Wabash national study. (Occasional paper No. 8). Urbana, IL: University of Illinois and Indiana University, National Institute for Learning Outcomes Assessment. Crosson, P., & Orcutt, B. (2014). A Massachusetts and multi-state approach to statewide assessment of student learning. Change: The Magazine of Higher Learning, May-June, Eubanks, D., & Gliem, D. (2015, May). Improving teaching, learning, and assessment by making evidence of achievement transparent. (Occasional Paper No. 25). Urbana, IL: University of Illinois and Indiana University, National Institute for Learning Outcomes Assessment. Ewell, P.T., & Jankowski, N.A. (2015). Accreditation as opportunity: Serving two purposes with assessment. In Using Evidence of Student Learning to Improve Higher Education, (pp ). San Francisco, CA: Jossey-Bass. Ewell, P., Jankowski, N., & Provezis, S. (2010, September). Connecting state policies on assessment with institutional assessment activity. Urbana, IL: University of Illinois and Indiana University, National Institute for Learning Outcomes Assessment (NILOA). Hutchings, P., Jankowski, N.A., & Ewell, P. (2014, November). Catalyzing assignment design activity on your own campus: Lessons from NILOA s assignment library initiative. Urbana, IL: University of Illinois and Indiana University, National Institute for Learning Outcomes Assessment (NILOA). Jankowski, N.A., & Cain, T.R. (2015) From compliance reporting to effective communication: Assessment and transparency. In Using Evidence of Student Learning to Improve Higher Education, (pp ). San Francisco, CA: Jossey-Bass. Jankowski, N.A., Ikenberry, S.O., Kinzie, J., Kuh, G.D., Shenoy, G.F., & Baker, G.R. (2012, March). Transparency & accountability: An evaluation of the VSA college portrait pilot. Urbana, IL: University of Illinois and Indiana University, National Institute for Learning Outcomes Assessment (NILOA). Jankowski, N., & Provezis, S. (2014). Neoliberal ideologies, governmentality, and the academy: An examination of accountability through assessment and transparency. Educational Philosophy and Theory,46(5), Kuh, G., & Ikenberry, S. (2009). More than you think, Less than we need: learning outcomes assessment in American Higher Education. Urbana, IL: University of Illinois and Indiana University, National Institute for Learning Outcomes Assessment (NILOA). Kuh, G.D., Ikenberry, S.O., Jankowski, N.A., Cain, T. R., Ewell, P., Hutchings, P., & Kinzie, K. (2015). Using Evidence of Student Learning to Improve Higher Education. San Francisco, CA: Jossey- Bass. Kuh, G.D., Jankowski, N., Ikenberry, S.O., & Kinzie, J. (2014). Knowing what students know and can do: The current state of student learning outcomes assessment in US colleges and universities. Urbana, IL: University of Illinois and Indiana University, National Institute for Learning Outcomes Assessment (NILOA). 7 Transparency, Accountability and Assessment of Student Learning

8 Powell, C. (2013). Accreditation, assessment, and compliance: Addressing the cyclical challenges of public confidence in American education. Journal of Assessment and Institutional Effectiveness, 3(1), Richman, W.A., & Ariovich, L. (2013, October). All-in-one: Combining grading, course, program, and general education outcomes assessment. (Occasional paper No. 19). Urbana, IL: University of Illinois and Indiana University, National Institute for Learning Outcomes Assessment. Stassen, M.L.A. (2012). Accountable for what? Journal of Assessment and Institutional Effectiveness, 2(2), Toulmin, S. E. (2003). The uses of argument (updated ed.). New York, NY: Cambridge University Press. U.S. Department of Education (2006). A test of leadership: Charting the future of U.S. higher education. Washington, DC.: Author. 8 Transparency, Accountability and Assessment of Student Learning