Formative Assessment of the Design, Delivery, and Evaluation of Coach Education Programs in Canada. Dr. Bruce Mann February 26, 2013

Transcription

1 Formative Assessment of the Design, Delivery, and Evaluation of Coach Education Programs in Canada Dr. Bruce Mann February 26, 2013 The goal of this research is to answer one over-arching question do the Coach Education programs require revision? If so, which parts - should the Multi-Sport Learning Activities be revised? Should different content be added, such as more First Aid content, or Legal Education, or Sport Psychology? Should alternate learning venues be considered, such as online modules and quizzes? Assessment enhances learning (Hay, Dickens, Crudgington & Engstrom, 2012; Reade, 2012). This research will examine the Coach Education programs for relevance, timeliness and innovation along four dimensions the delivery, the environment, the content, and the learners (D.E.C.L.). Problem The National Coaching Certification Program (NCCP) in Canada is in the process of re-designing their Coach Education training programs to become more congruent with a lifelong learning perspective (Callary, Werthner & Trudel, 2011). The Coaching Association of Canada (CAC) has determined that coaching effectiveness, across all levels of the sport system, needs enhancing, an opinion that is shared by coach education observers. Burton and Gilliam (2012) for example, stated the lack of systematic assessment to provide accountability and value in coaching education has been a problem for over three decades (p. 207). Reade (2012) observed that, [In] the Canadian system, introductory coach education content is delivered in many sports to coaches in a workshop format over one or two weekends. The pragmatism and efficiency of such delivery is reasonable, but the effectiveness is questionable and it is not assessed. The coaches are not subjected to a formal assessment at the end of the workshop, and every coach passes the course. The need (demand) for coaches at the introductory level within the sport community is so great that there is likely no desire to eliminate interested individuals from the practice of coaching on the basis of the failure of a post-course assessment. The coaches are designated as trained, and are allowed to commence coaching. In most sports, there is no mandatory assessment after the course is complete, nor is there any requirement for any subsequent education. Should the introductory level coach later wish to progress to the next level and become certified, there is more opportunity for workshop-based education, and the coach can ask to be evaluated by an assessor. Even in this case however, the assessment is a snapshot, a one-time picture of the function of the coach. Given the complex and contextual nature of coaching, it is debatable whether assessment can be expected to be effective through a single snapshot (p. 199). One of the first strategic imperatives of the CAC s plan is to "strengthen the product development cycle to ensure relevant, timely and innovative programs and services" (Strategic Plan, 2010, p. 3). This is the goal of a formative evaluation namely

2 B.L. Mann Design, Delivery & Eval n of Coach Ed. 2 to determine if the Coach Education programs require revision, and if so, which parts? This proposal will focus on the design, delivery, and evaluation of the Multi-Sport Learning Activities for the Coach Education programs. The significance of the research is to attain a better understanding of the current status of the Multi-Sport Learning Activities among the coaches, coach educators and athletes, and to recommend changes to those materials. This particular research will ask the following questions: - Is the venue and teaching method appropriate for educating coaches? (i.e., Delivery) - What is Being Taught? What is Being Learned? (i.e., Content) - Who are the Coaches? Who are the Facilitators? (i.e., Learners) - Where s the Emphasis Placed in the Coach Education? (i.e., Environment) Literature Review It is generally recognised in the sports coaching literature, that coaches make a significant contribution to the learning of their athletes (Armour, 2010) and have a learning trajectory themselves (Lyle, 2002). The Strategic Plan of the Coaching Association of Canada (CAC) believes that Coach Education improves a coach s ability to meet the needs of participants and athletes" (2010, p. 2). Coach education is an important factor in the confidence of coaches. Recent Canadian research reported that coach education significantly affected coaching efficacy (Sullivan, Paquette, Holt, & Bloom, 2012). "Coaching efficacy" is a sport-specific concept defined as the extent to which coaches believe they have the capacity to affect the learning and performance of their athletes (p. 124). An Australian study of coaches of adolescent athletes (Vella, Oades, & Crowe, 2011) reported that coaches see themselves as responsible for facilitating eight inter-related and inter-dependent themes beyond on-field success, namely: competence, confidence, connection, character, life skills, climate, positive affect, and positive psychological capacities. In a recent Canadian study of coach education (Falcao, Bloom & Gilbert, 2012), coaches cited many benefits for themselves, their athletes, and their teams, including increased knowledge and a better understanding of their athletes. The coaches in the study perceived that the activities promoted cohesion and communication, while also contributing to the development of athlete competence, confidence, connection, and caring. Coach Education courses are not all the same. They can be made to have different emphases depending on the need for changes to the delivery, the environment, the content, and the learner factors, or D.E.C.L. Enhancing Coach Education courses therefore, as suggested by the CAC, becomes a function of changing one or more of these four factors the delivery, environment, content, learner, or D.E.C.L to improve Coaching. Figure 1 shows greater emphasis placed on the Learner (left), on the Environment (centre), and a Balanced emphasis on the right.

3 B.L. Mann Design, Delivery & Eval n of Coach Ed. 3 Figure 1. Graphical comparison of a Coach Education course by its emphasis. Adapted from Mann, B.L. (2005). Making your own educational materials part I. International Journal of Instructional Technology & Distance Learning 10(2). Method The design of this research is called formative evaluation. A formative evaluation is not concerned with matching learning outcomes to goals, nor does it examine learner satisfaction. Rather formative evaluation asks only one question, What revisions need to be made to the Coach Education programs? A formative evaluation must be both effective and enduring to ensure a long shelf-life. In this research, D.E.C.L. is used as both the model of evaluation and instructional design. D.E.C.L. stands for delivery, environment, content and learner factors that comprise achievement, shown in Figure 2. Delivery Environment Content Learner Scope Setting Mental Operations Required Attitude Presentation Climate Task Capacity Strategy Domain Demographics Sequencing Competence Figure 2. D.E.C.L. factors of Coach Education and associated variables. Adapted from Mann, B.L. (2005). Making your own educational materials part I. International Journal of Instructional Technology & Distance Learning 10(2). Is the Delivery Method Appropriate for Coaches & Facilitators? - D for Delivery Four variables comprise the delivery factor - scope, sequence, strategies, and presentation of the web course. Consideration of the scope would either the mean big picture the program, a course or a unit. Or its micro-structure as in a workshop, lesson, even a single objective. Evaluating the sequence of instruction could be examining the ordering of events or activities, or their pacing in the course schedule. Your research should consider the strategies, including both the media mix and the instructional devices. Consideration of the media mix means the design of the audio, video and graphics for possible effects on students cognitive load. Consideration of instructional devices would include a discourse analysis of asynchronous text discussion, asynchronous text chat, or

4 B.L. Mann Design, Delivery & Eval n of Coach Ed. 4 audio conferences. If it s a hybrid (e.g., web/face-to-face) course, the evaluator might also look at the amount and quality of whole group instruction, individualized instruction, work-embedded instruction, fault-free questioning, constructed answers, partial answers, elaborative interrogation. Presentation. Regarding the Presentation variables involved, its recommended to consider tactics, which are comprised of six functions, adapted from Gagne s events of instruction: 1) maintain attention; 2) secure a response; 3) provide reinforcement; 4) maintain interest; 5) facilitate retention, and 6) assess performance. Concerning the form of the presentation, consider how the mix of technologies is encoded by learners. Where s the Emphasis Placed in the Coach Education? - E is for Environment Two variables comprise the environment factor. In preparation for conducting your study, you would do well to account for the learning climate (e.g., K-12 school, business, government, medical & nursing, college, military) and its setting which could be characterized as online or hybrid (e.g., web/face-to-face). What is Being Taught? What is Being Learned? - C for Content Three variables comprise the content factor,. In the content domain, your formative evaluation should examine the mental operations required to learn the content, the tasks involved, and the learning domain. Mental operations required. Regarding the mental operations required to learn the content, consider the requirements to select and focus attention, student expectations, and guidance provisions. Is there time for retention, to organize, rehearse for storage, and retrieve and use. Finally there should be advice about learning transfer through lateral, vertical, problem solving or creative thinking activities. Tasks. Regarding the tasks involved its best to consider everything you can such as the basic skills that are assumed, any gender or cultural concerns, vocational aspects, and personal skills such as time management or goal setting. Learning domains. Concerning learning domains involved, it may be good to know the extent of motor and intellectual skills, verbal information learning and cognitive strategies to augment or supplant learner precepts. Finally, consider the buyin factor in the content, perhaps more accurately called the axiological variable. Who are they? - L for the Learners Attitude. Attitudes In the learner domain, attitude can have a direct effect on student performances in certain learning tasks. Attitudes are defined as likes and dislikes, with roots in social, emotional, behavioral, and cognitive experiences. Attitudes can also be value-laden (e.g., moral and religious beliefs, school pride, or work-ethic), selfconcept (e.g., academic, personal, professional), or motivational (e.g., goals, interests, perseverance). Capacity. Capacity relates to innate ability, as opposed to achievement, the competence variable. The definition of capacity includes intellectual abilities (e.g., verbal, mathematical, artistic, and social capabilities), as well as cognitive (e.g., perceiving, remembering, thinking, apprehending, sorting, and utilizing information) and physiological (e.g., perceptual development and motor dexterity) development. Regarding demographics, some evaluators like to collect demographic data to apply to

5 B.L. Mann Design, Delivery & Eval n of Coach Ed. 5 their data interpretation. Data shows differences among the learner performances when subjects are categorized on the basis of such variables as age, sex, and cultural background. Competence. Competence is the result of conscious activity, either a learning experience or another life event. Whereas competence is limited by an individual s natural capacities, it is influenced by the learner s attitudes and general profile characteristics. Competence includes prerequisite skills (e.g., information processing skills, basic skills, and content prerequisites) and experiential background (e.g., family, leisure time, social, vocational and educational background). Procedure Though there are many models of formative evaluation, this research will focus on the D.E.C.L. both for thick descriptions about the extant instructional material, and for the prescriptive advice for CAC s instructional designers. Dissemination Plan The dissemination plan relies on recommendations for revision based on data collected from the formative evaluation of D.E.C.L. factors in CAC s Coach Education programs. For The Facilitators & Coaches The dissemination plan for the Facilitators and Coaches in CAC s Coach Education programs will likely involve some instructor- and student- training. The recommendations will likely propose that Web-based instruction should replace at least some of the face-to-face classroom experiences. Some the facilitators may not be familiar with the proposed means and methods. Several Web-based products offer educators information and design tools for managing their Web courses. Moreover Web course management systems are remarkably similar to one another with many of the same features. Most systems present clickable icons spread out in rows on a graphical user interface. From discussions and workshops with instructors in Australia, Canada and the U.S, I found that educators assimilated into a Web course management system in three phases: lesson enhancement, resource-based teaching, and online learning environments (Mann, 1999). The phases are distinctive, the tasks are not. So educators working in different phases of can be doing similar tasks for different purposes (Mann 2000, 2006). For The Instructional Designers There should be plenty of data concerning the Delivery, Environments, Content, and Learners from which to revise the learning environments for Coach Education. From the literature the recommendation is for learning-oriented, authentic, valid and socially just assessment practices have much to offer both coach accreditation and continuing professional development, as well as the activity of sports coaching itself (Hay, Dickens, Crudgington & Engstrom, 2012, p. 196).

6 B.L. Mann Design, Delivery & Eval n of Coach Ed. 6 References Armour, K., (2010). The Learning Coach... the Learning Approach: Professional Development for Sports Coach Professionals, In J. Lyle and C. Cushion (Eds.). Sports Coaching: Professionalization and Practice, Elsevier, London, Burton, D. & Gillham, A. (2010). Exploring the Potential of Assessment Efficacy in Sports Coaching: A Commentary, International Journal of Sports Science & Coaching, 7, 2, 207. Callary, B., Werthner, P., & Trudel, P. (2011). A Brief Online Survey to Help a Sport Organization Continue to Develop its Coach Education Program, International Journal of Coaching Science, 5, 2, Coaching Association of Canada (2010). Strategic Plan Ottawa, ONT: National Coaching Certification Program. Coaching Association of Canada (2007). Coach Workbook. Competition - Introduction. Multi-Sport Learning Activities Part B. National Coaching Certification Program. St. John s: Newfoundland & Labrador. Coaching Association of Canada (2007). Reference Material. Competition - Introduction. Multi-Sport Learning Activities Part B. National Coaching Certification Program. St. John s: Newfoundland & Labrador. Coaching Association of Canada (2003). Coach Workbook. Teaching and Learning. Multi-Sport Learning Activities Part A Introductory Module. Contexts: Introduction to Competition Instruction Beginners. Ottawa, ONT: National Coaching Certification Program. Coaching Association of Canada (2003). Reference Material. Teaching and Learning. Multi-Sport Learning Activities Part A Introductory Module. Contexts: Introduction to Competition Instruction Beginners. Ottawa, ONT: National Coaching Certification Program. Coaching Association of Canada (2003). Coach Workbook. Teaching and Learning. Multi-Sport Learning Activities Part B Introductory Module. Contexts: Introduction to Competition Instruction Beginners. Ottawa, ONT: National Coaching Certification Program. Coaching Association of Canada (2003). Reference Material. Teaching and Learning. Multi-Sport Learning Activities Part B Introductory Module. Contexts: Introduction to Competition Instruction Beginners. Ottawa, ONT: National Coaching Certification Program. Falcao, W.R., Bloom, G.A., & Gilbert, W.D. (2012). Coaches Perceptions of a Coach Training Program Designed to Promote Youth Developmental Outcomes, Journal of Applied Sport Psychology, 24, Hay, P., Dickens, S., Crudgington, B., & Engstrom, C. (2012). Exploring the Potential of Assessment Efficacy in Sports Coaching. International Journal of Sports Science & Coaching, 7, 2, 187. Lyle, J. (2002). Sports Coaching Concepts: A Framework for Coaches Behaviour, Routledge, London. Mann, B.L. (2005). Making your own educational materials part I. International Journal of Instructional Technology & Distance Learning 10(2),

7 B.L. Mann Design, Delivery & Eval n of Coach Ed. 7 Mann, B.L. (2006). Research styles and the Internet. In Bruce L. Mann (Ed.). Selected styles in web-based educational research. (pp. 1-11) Hershey, PA: Idea Group Publishing. Mann, B.L. (2006). Conducting formative evaluations of instructional material. In Bruce L. Mann (Ed.). Selected styles in web-based educational research (pp ). Hershey, PA: Information Science Publishing Reade, I. (2012). Exploring the Potential of Assessment Efficacy in Sports Coaching: A Commentary, International Journal of Sports Science & Coaching, 7, 2, 187. Research Committee (2013). Call for Proposals. Coach Education Research, Retrieved 15 February Sullivan, P., Paquette, K., Holt, N., & Bloom, G. (2012). The Relation of Coaching Context and Coach Education to Coaching Efficacy and Perceived Leadership Behaviors in Youth Sport, The Sport Psychologist, 26, Vella, S., Oades, L., & Crowe, T. (2011). The Role of the Coach in Facilitating Positive Youth Development: Moving from Theory to Practice, Journal of Applied Sport Psychology, 23,

8 B.L. Mann Design, Delivery & Eval n of Coach Ed. 8 Financial Data Form APPLICANT: Prof. Bruce Mann, B.A., M.A., Ph.D., LL.M. TITLE OF PROJECT: Design, Delivery Evaluation of the Coach Education Programs PERSONNEL COSTS, JUSTIFICATION: Faculty Course Buy-Out - The minimum charge for a three-credit-hour-equivalent buy-out in recognition of the full costs associated with replacement instruction - $5, Research Assistant Research Assistant II - $23.73/hr x 18 hrs/wk = /wk x 4 wks = % = $ The Research Assistant II will have completed graduate courses in Instructional Design. The position requires the R.A. to search reference books and full-version online databases for formative evaluations of coach education programs. OTHER COSTS NOT COVERED ABOVE: Telephone Survey: $200. Photocopy paper, printing & Photocopying costs of the research material = $75. TOTAL FUNDS REQUESTED: $_6,995.56_ Signature of Applicant(s) Date