Curricula Assessment Tool. April 2013

Transcription

1 Curricula Assessment Tool April 2013

2 Introduction: Programs are the foundation of Extension s educational strategies and curricula are at the center of mature programs. The Curricula Assessment Tool (CAT) was developed to: 1) evaluate existing educational curricula based on a set of criterion-referenced elements, and 2) to use in creating educational curricula. While there is much written about curricula development, there has not been a tool to help in the actual assessment until now. The CAT is a criterion-referenced assessment tool that permits multiple individuals to make judgments using common criteria with common definitions. The CAT is being used to assess the Health Insurance Literacy Initiative (HILI) curriculum. Your work in assessing the HILI curriculum is also a formative evaluation of the CAT. Therefore, we will be seeking your feedback on your experiences with the tool itself (as well as an assessment of the curriculum). Although created specifically for health insurance literacy, other topics could be substituted with appropriate vocabulary for the curricula under review. The Curricula Assessment Tool and its companions, the Program Assessment Tool and the Materials Assessment Tool are included in the forthcoming guide: Extension Education Theoretical Framework with Criterion-Referenced Assessment Tools. Instructions: 1) Assessing the curricula: This rubric evaluates curricula on a four-point scale (effective, good, fair, and ineffective). For each rating category, information is presented that clearly identifies how to assess the curricula according to the four-point scale. Use the scoring sheet to record your rating (see page 9 for scoring sheet). We have included a version of the CAT without the guiding text to type your comments (page 10). 2) Assessing the CAT: Remember, we are not only testing the curriculum, we are testing the CAT. Please insert comments about each section in the far rightcontent cell. You may also use track changes. Our goal is to improve the tool as well as the curriculum. 3) Returning Material: You will be returning to us by MAY 20: a) the scoring sheet for the curriculum; b) the CAT form with your comments about the curriculum (guiding text deleted) in the right-hand column; and c) the CAT with your comments about the CAT. Creators: Nicole Finkbeiner, MS and Bonnie Braun, PhD, University of Maryland Extension. Based on Coleman, et al, 2011 Journal of Extension article on reviewing curricula. your rating to: tmcoy1@umd.edu AND bbraun@umd.edu Page 1

3 Check the one you are reviewing: Educator s Guide Consumer s Guide University of Maryland Extension Curricula Assessment Tool Reviewed by: Content Theoretical foundation Researchbased Content Balanced Viewpoint Learning Objectives 4 points The curriculum is based on current education and behavioral change theory and research. The theoretical underpinnings of the curriculum are described. The content of the curriculum is researchbased, accurate, and current. The curriculum presents a balanced view of the topic, recognizing any aspects that are not yet clearly understood or open to debate. Includes clear, measurable learning and behavioral objectives. Objectives are clearly linked to theoretical underpinnings. 3 points or two pieces is based on current education and behavioral change theory and research. The theoretical underpinnings of the curriculum are mostly described. The content of the curriculum is mainly effective - all but one of the key components of effective curriculum (research-based, accurate, and current) are addressed. or two pieces presents a balanced view of the topic, recognizing any aspects that are not yet clearly understood or open to debate. or two pieces is tied to clear, measurable learning and behavioral objectives. Objectives are mostly linked to theoretical underpinnings. 2 points of the curriculum are not based on current education and behavioral change theory and research. The theoretical underpinnings of the curriculum are not described in detail. The content of the curriculum is missing more than one of the key components of effective curriculum - research-based, accurate, and current. of the curriculum do not present a balanced view of the topic, failing to recognize any aspects that are not yet clearly understood or open to debate. of the curriculum are not tied to clear, measurable learning and behavioral objectives. Objectives are poorly linked to theoretical underpinnings. Date: 1 point The curriculum is not based on current education and behavioral change theory and research. The theoretical underpinnings of the curriculum are not described. The content is not research-based, accurate, or current. The curriculum presents a one-sided view of the topic, failing to recognize any aspects that are not yet clearly understood or open to debate. Does not include clear, measurable learning and behavioral objectives. Comments Not enough information, not applicable, etc. Page 2

4 Target Input/ Outcomes Involvement Reflection of Diversity Respect for Diversity Comments Identifies the intended audience and is tailored to this audience. Builds on the strengths/assets, needs, and interests of learners. input was used to guide development of materials. Actively engages the audience in the learning process and promotes behavior change. Reflects the diversity, including health literacy, of the intended audience. Includes multilingual handouts and educational reinforcements when appropriate. Ideas and principles included in the curriculum respect all aspects of diversity including health literacy. or two components of the curriculum are tailored to the intended audience. or two pieces of the curriculum build on the strengths/assets, needs and interests of learners OR were guided by audience input. or two pieces actively engages the audience in the learning process and promotes behavior change. All content except for one or two pieces reflects the diversity, including health literacy, of the intended audience. Includes multilingual handouts and educational reinforcements when appropriate. All content, except for one or two ideas and principles included in the curriculum, respect all aspects of diversity including health literacy. More than one or two components of the curriculum are not tailored to the intended audience. More than one or two components of the curriculum do not build on the strengths/assets, needs and interests of learners OR were not guided by audience input. of the curriculum do not actively engage the audience or do not promote behavior change. of content do not reflect the diversity, including health literacy, of the intended audience OR the curriculum does not include multilingual handouts and educational reinforcements when appropriate. More than one or two ideas and principles included in the curriculum do not respect all aspects of diversity including health literacy. Does not clearly identify the intended audience. Does not build on the strengths/assets, needs, and interests of learners. input was not used to guide development of materials. Does not actively engage the audience in the learning process and does not promote behavior change. Does not reflect the diversity, including health literacy, of the intended audience. Does not include multilingual handouts and educational reinforcements when appropriate. Ideas and principles included in the curriculum do NOT respect all aspects of diversity including health literacy. Page 3

5 Readability Grammar Comments Tone and Reading Level Reflects standards of written English and correct grammar, spelling, punctuation, and mechanics. All health and insurance terminology is clear, correctly used and spelled throughout content. Correct health and insurance abbreviations are used throughout.. The curriculum is written at grade 6 or lower if intended for the general public. One to two grammatical, spelling, punctuation, or mechanical errors. Health and insurance terminology is somewhat clear and correctly used and spelled throughout most of content. Spelling mistakes are minor. Correct health and insurance abbreviations are mostly used throughout content. The curriculum is written at grade 6 or lower if intended for the general public. More than two grammatical, spelling, punctuation, or mechanical errors. Health and insurance terminology is frequently used incorrectly or is not clear and has misspellings. Health and insurance abbreviations are incorrect. The curriculum if for the general public is written at a higher level than grade 6. Not comprehensible. Not comprehensible and the curriculum is written at not written at the grade 6 level if for the general public. Organization Style of material Is logically and sequentially organized. Content displays evidence of understanding of principles of health literacy and plain language (format, font, visuals, color, text construction, depth, detail, complexity). or two pieces displays logical and sequential organization. or two pieces displays evidence of understanding of principles of health literacy and plain language (format, font, visuals, color, text construction, depth, detail, complexity). of the content are not logically and sequentially organized. of the content do not display evidence of understanding of principles of health literacy and plain language (format, font, visuals, color, text construction, depth, detail, complexity). Is not clearly organized. Content does not display evidence of understanding of principles of health literacy and plain language (format, font, visuals, color, text construction, depth, detail, complexity). Page 4

6 Utility Lesson Implementation /Preparation (for educators) References Comments Includes all the materials and information needed for implementing the lesson, acquiring support materials, and preparing for the class (including guidelines for lessons). All information except for one or two pieces needed for implementing the lessons, acquiring support materials, and preparing for the class is included. of information needed for implementing the lessons, acquiring support materials, and preparing for the class are missing. Does not include materials and information needed for implementing the lessons, acquiring support materials, or preparing for the class (including guidelines for lessons). Includes appropriate credit for all non-original material, references, and additional resources and information. All content, except for one or two pieces of credit for non-original material, references, and additional resources and information, are included. of credit are missing for nonoriginal material, references, and additional resources and information. Does not include credit for non-original material, references, and additional resources and information. Instructions Validity of program All instructions are easy to understand and follow. The curriculum has established strong validity and reliability - has been peer-reviewed, pilot tested and refined. All instructions except for one or two components are easy to understand and follow. All content except for one or two pieces of information verifying the validity and reliability of the curriculum are present - the user can establish with confidence that the curriculum has been peer-reviewed, pilot tested and refined. More than one or two components of the instructions are not easy to understand and follow. of information verifying the validity and reliability of the curriculum are missing - the user cannot establish with confidence that the curriculum has been peerreviewed, pilot tested or refined. Instructions are hard to understand and follow. The curriculum has not established strong validity and reliability- has not been peer-reviewed, pilot tested and refined. Page 5

7 Comments Activities Relevant resources Any activities used to reinforce the educational messages are practical to implement. All relevant resources, such as audiovisuals or websites, are included with the curriculum. All content except for one component of the activities used to reinforce the educational messages is practical to implement. All relevant resources, except for one audiovisual or website, are included with the curriculum. More than one component of the activities used to reinforce the educational messages are not practical to implement. More than one or two relevant resources, such as audiovisuals and websites, are missing from the curriculum. Any activities used to reinforce the educational messages are not practical to implement. Relevant resources, such as audiovisuals or websites, are not included with the curriculum. Source Citation Logic Model Source, author, and publication date are clearly and appropriately cited. All content except for one piece of citation material, including source, author, or publication date are clearly and appropriately cited. More than one piece of citation material, including source, author, or publication date, is missing from the curriculum. Source, author, and publication date are not clearly and appropriately cited. Process of implementation Includes a complete logic model or other appropriate programming planning and outcomes model. Describes recommended process for implementing the curriculum. Includes a logic model and all but one or two key pieces of information regarding the planning, implementation, and desired outcomes of the curriculum is included. All content except for one or two pieces of information regarding a recommended process for implementing the curriculum are described. Includes a logic model, but more than one or two key pieces of information regarding the planning, implementation, and desired outcomes of the curriculum is missing. of information regarding the process for implementing the curriculum are missing. Does not include a logic model or other appropriate programming planning and outcomes model. Does not describe the process for implementing the curriculum. Page 6

8 Evaluation tested Instruments More than one or two assessment instruments have not been audiencetested. Psychometrically Sound Instruments Link to Learning Objectives Evaluation Phases Comments Includes assessment instruments (e.g. checklists, questionnaires, observational instruments) that have been audience- tested. Includes assessment instruments (e.g. checklists, questionnaires, observational instruments) that demonstrate acceptable psychometric properties. Evaluation methods and items are clearly linked to learning objectives. Assessment instruments include those designed to be administered prior to, during, and after implementing the curriculum so that effectiveness can be established and reported. All assessment instruments except for one or two have been audience-tested. All assessment instruments except for one or two demonstrate acceptable psychometric properties. All content except for one or two items or methods for evaluation are clearly linked to learning objectives. All evaluation process content, except for one key component, is included - pre-tests, tests during implementation, or post/follow-up tests. More than one or two assessment instruments do not demonstrate acceptable psychometric properties. More than one or two items or methods for evaluation are not clearly linked to learning objectives. The evaluation is missing more than one of the key components of effective evaluation methods - pretests, tests during implementation, or post/follow-up tests. Does not include assessment instruments (e.g. checklists, questionnaires, observational instruments) that have been audience-tested. Does not include assessment instruments (e.g. checklists, questionnaires, observational instruments) that demonstrate acceptable psychometric properties. Evaluation methods and items are not linked to learning objectives. Assessment instruments do not include those designed to be administered prior to, during, or after implementing the curriculum. Page 7

9 Extension Curricula Assessment Tool - SCORING SHEET 4 points for effective rating 3 points for good rating 2 points for fair rating 1 point for ineffective rating N/A if the factor does not apply to this material FACTOR TO BE RATED SCORE COMMENTS 1. CONTENT a) Program grounded in theory b) Content grounded in research c) Viewpoint is balanced d) Learning objectives included 2. AUDIENCE a) Identifies target audience b) input utilized c) involved, engaged d) Diversity is reflected. e) Diversity is respected 3. READABILITY a) Accurate spelling/grammar b) Appropriate vocabulary and reading level c) Logical organization d) Material reflects principles of plain language and literacy Page 8

10 4. UTILITY a) Lesson Implementation and Preparation b) Appropriate references c) Easy to understand instructions d) Strong validity and reliability established e) Practical activities f) Relevant resources included g) Strong citation for program being reviewed h) Logic model included i) Describes process for implementing curriculum 5. EVALUATION a) -tested instruments b) Psychometrically-sound instruments c) Evaluation methods linked to learning objectives d) Pre-test, post-test methods Total score: Page 9

11 Check the one you are reviewing: Educator s Guide Consumer s Guide You can use this form (without the guiding text) to insert your own comments about the Guide and your rating. Target Input/ Outcomes Involvement Reflection of Diversity Respect for Diversity Comments. Comments Readability Grammar Tone and Reading Level Organization Style of material Page 10

12 Utility Lesson Implementation /Preparation (for educators) Comments References Instructions Validity of program Activities Relevant resources Source Citation Logic Model Process of implementation Evaluation -tested Instruments Comments Psycho-metrically Sound Instruments Link to Learning Page 11

13 Objectives Evaluation Phases References: Coleman, G., Byrd-Bredbenner, C., Baker, S., & Bowen, E. (2011). Best practices for extension curricula review. Journal of Extension, 49, Article 2T0T1. Available at: Page 12