MAGNA ONLINE SEMINARS

Transcription

1 MAGNA ONLINE SEMINARS How Good Is Good Enough?: Setting Benchmarks or Standards Wednesday, Presented by: Linda Suskie Linda Suskie is vice president of the Middle States Commission on Higher Education. Prior positions include serving as associate vice president for assessment and institutional research at Towson University, Maryland, and as director of the American Association for Higher Education s Assessment Forum. She holds a bachelor s degree in quantitative studies from Johns Hopkins University and a master s in educational measurement and statistics from the University of Iowa Magna Publications Inc. All rights reserved. It is unlawful to duplicate, transfer, or transmit this program in any manner without written consent from Magna Publications. The information contained in this online seminar is for professional development purposes but does not substitute for legal advice. Specific legal advice should be discussed with a professional attorney.

2 Some Sources of Potential Peer Institutions Institutions using the same published instrument (contact the instrument s publisher for information) Information-sharing networks such as the Higher Education Data Sharing (HEDS) Consortium for private institutions Institutions with the same or similar Carnegie classifications ( Fellow members of a higher education organization or consortium, such as the American Association of Universities (AAU), the American Association of State Colleges and Universities (AASCU), the National Association of Independent Colleges and Universities (NAICU), or the Associated New American Colleges (ANAC) Fellow members of a state or regional system, for public institutions Peer institutions that the institution identifies, perhaps by using tools such as College Results Online (below) Some Public Sources of Information on Colleges and Universities Resource Sponsor Website Address College Navigator National Center for Education Statistics nces.ed.gov/collegenavigator/ College MatchMaker College Board collegesearch.collegeboard.com/search/index.jsp College Results Online Education Trust America s Best Colleges US News & World Report colleges.usnews.rankingsandreviews.com/college 100 Best Values in Public Colleges Kiplinger s College Portrait National Association of State Universities and Land Grant Colleges (NASULGC) and American Association of State Colleges and Universities (AASCU) U-CAN National Association of Independent Colleges and Universities (NAICU) Source: Suskie, L. (2009). Assessing student Learning: A Common Sense Guide, 2 nd ed. San Francisco: Jossey-Bass.

3 Examining Results from Multiple Perspectives: An Example Valley View University has a four-year graduation rate of 42%, for students entering as full-time, first-time freshmen. Is this good enough? Local standards-based 40% institutional goal YES External standards- 50% system goal NO based Peer-referenced 45% system average NO Best practice 60% at Mountain View U NO Historical 30% 3 years ago YES Strengths & weaknesses Capability 48% for women; 36% for men YES/ NO 20% leave for developmental reasons; 15% leave for NO preventable reasons Productivity $150 per retained student for retention programs; $175 at Sea View U YES Prepared by Linda Suskie, Middle States Commission on Higher Education

4 ENGL 102 (Writing for a Liberal Education) Grading Rubric In all categories, F = Fails to meet C Standard. Focus: Purpose, audience, and constraints Organization: Central idea, overall organization, & organizational devices Organization: Paragraph structure Content/Reasoning: Style/Expression: Sentence structure Style/Expression: Tone, word choice Paper has a clear purpose & shows awareness of audience. Complies with all constraints such as subject, organization, & length. A B C Paper clearly presents a central idea supported throughout the paper. Well-planned organization. Organizational devices (title, thesis statement, opening/closing paragraphs, transitions) are always effective & smooth. All paragraphs have clear points with effective organizational devices (e.g., topic sentences & transitions). Exceptionally sound reasoning; ideas & positions are well developed & supported with convincing evidence & relevant facts, examples, details, etc. All sentences are clear, well structured, & varied in pattern. Style options (tone, word choice) are appropriate for audience & purpose, varied, and make the paper interesting. Paper shows awareness of purpose & audience. Compiles with all constraints such as subject, organization, & length. Paper presents a central idea supported throughout the paper. Good overall organization; organizational devices are largely present, effective, & smooth. Virtually all paragraphs have clear points, organizational devices, & transitions. Very sound reasoning; ideas & positions are almost all well developed & supported with good evidence. Virtually all sentences are clear & well structured. Sentences may lack variety or a few may be awkward. A few style options (tone, word choices) are inappropriate for audience & purpose. Style options are somewhat varied and make the paper somewhat interesting. Paper shows limited awareness of purpose & audience. Complies with most constraints such as subject, organization, & length. Paper vaguely presents a central idea supported throughout the paper. Overall organization is good enough to be understandable, although devices may lack smoothness, be missing, or be ineffective. Most but not all paragraphs have clear points, organizational devices, & transitions. Reasoning is sound; ideas & positions are supported with some evidence but are not always well developed. Sentences are generally clear & well structured, but some sentences are awkward or unclear. Style options (tone, word choice) are largely reasonable for audience & purpose but some are inappropriate, or the paper is somewhat flat and dull. Style/Expression: Wordiness Paper is appropriately concise. Paper has a few wordy sentences/phrases. Paper has several wordy sentences/phrases. Grammar/Mechanics: Impeccable grammar, spelling, punctuation, & mechanics Linda Suskie, Middle States Commission on Higher Education Virtually no errors in grammar, spelling, punctuation, & mechanics Substantially free of errors in grammar, mechanics, etc.; errors do not impede meaning nor overly distract the reader.

5 Reflecting on Setting Benchmarks or Standards When you decide on grade cutoffs (the score, rating, or judgment point that separates, say, a B from a C) Which kind(s) of benchmark or standard do you use? Why? What does this say about you as a teacher? Should you consider other perspectives? Why or why not? How do you set each cutoff point? Whom do you involve in this decision? Whom would you like to involve? Should you try a different approach to setting grade cutoff points? Why or why not?

6 SUGGESTED READINGS ON SETTING TARGETS FOR UNDERSTANDING ASSESSMENT RESULTS Banta, T. W. (2008). Trying to clothe the emperor. Assessment Update, 20(2), 3-4, Banta, T. W. (2007). If we must compare Assessment Update, 19(2), 3-4. Banta, T. W., & Pike, G. R. Revisiting the blind alley of value added. Assessment Update, 19(1), 1-2, Campbell, D. T., & Stanley, J. C. (1963). Experimental and quasi-experimental designs for research. Chicago, IL: Rand McNally. Ewell, P. T. (2005). Power in numbers: The values in our metrics. Change, 37(4), Jones, D. P. (2002, April). Different perspectives on information about educational quality: Implications for the role of accreditation (CHEA Occasional Paper). Washington, DC: Council for Higher Education Accreditation. Jones, E., & Voorhees, R., with Paulson, K. (2002). Defining and assessing learning: Exploring competencybased initiatives. Washington, DC: National Center for Education Statistics. Retrieved June 2, 2003, from Livingston, S. A., & Zieky, M. J. (1982). Passing scores: A manual for setting standards on performance on educational and occupational tests. Princeton, NJ: Educational Testing Service. Miller, M. (2008). The Voluntary System of Accountability: Origins and purposes: An interview with George Mehaffy and David Shulenberger. Change, 40(4), Pascarella, E. T. (2001). Identifying excellence in undergraduate education: Are we even close? Change, 33(3), Peter D. Hart Research Associates, Inc. (2008). How should colleges assess and improve student learning? Employers views on the accountability challenge. Washington: Author. Pieper, S. L., Fulcher, K. H., Sundre, D. L., & Erwin, T. D. (2008). What do I do with the data now? Analyzing assessment information for accountability and improvement. Research & Practice in Assessment, 1. [ Pike, G. R. (2006). Assessment measures: Value-added models and the Collegiate Learning Assessment. Assessment Update, 18(4), 5-7. Pike, G. R. (1992). Lies, damn lies, and statistics revised: A comparison of three methods of representing change. Research in Education, 33, Pike, G. R. (2007). Response to Fulcher and Willse. Assessment Update, 19(5), Seybert, J. A. (2007). Benchmarking in community colleges: A current perspective. Assessment Update, 19(5), 3. Shepard, L. A. (1980). Standard setting issues and methods. Applied Psychological Measurement, 4, Stufflebeam, D. L. (2000). The CIPP model for evaluation. In D. L. Stufflebeam, G. F. Madaus, & T. Kellaghan (Eds.), Evaluation models: Viewpoints on educational and human services evaluation (2nd ed.) (Chapter 16) (Evaluation in Education and Human Services, Vol. 49). Boston, MA: Kluwer. Suskie, L. (2007.) Answering the complex question of How good is good enough? Assessment Update. Suskie, L. (2009). Setting benchmarks or standards. In Assessing student learning: A common sense guide, 2 nd ed. San Francisco: Jossey-Bass. Taylor, B. E., & Massy, W. F. (1996). Strategic indicators for higher education. Princeton, NJ: Peterson s. Compiled by Linda Suskie, Middle States Commission on Higher Education