2013-2014 Assessment Report Program: Applied Behavioral Analysis, MS College of Liberal Arts Psychology Applied Behavioral Analysis, MS Expected Outcome 1: Proficiency in Application of ABA Procedures The Applied Behavior Analysis Program is a one-year intensive Master s program that involves three contiguous semesters of class instruction and practical training. Consistent with the Behavior Analysis Certification Board (BACB) Task List, graduates of the ABA program must demonstrate proficiency in the application of various behavior acquisition techniques including: behavioral chaining procedures, stimulus equivalence procedures, stimulus fading procedures, and methods for teaching verbal behavior (e.g., communication) among others. Assessment Method 1: Direct Skills Assessment Assessment Method Description Program faculty and practicum coordinators use behavioral skills training (BST) to teach students to implement various behavior-analytic assessment and intervention procedures. BST involves describing procedures, modeling procedures, observing the students engaged in the procedures, and providing feedback until mastery performance is demonstrated. Students are trained and evaluated under simulated conditions and with actual clients. See example rubrics provided below for discrete trial training, chaining, stimulus equivalence, stimulus fading, and verbal behavior. Each rubric includes a step-by-step outline of the materials and actions required by the student to correctly implement and document the effects of the specified intervention procedure. Each assessment was conducted under the context of simulations wherein a faculty member of practicum coordinator serves as a client who needs the specified intervention. All 12 ABA graduate students were evaluated on each area between 10/1/13 and 12/20/13. Each student received a score of 90% or higher on three or fewer opportunities with each assessment. 1
2
3
4
5
6
Findings Faculty members and practicum coordinators provided students feedback on their performance on individual procedures (using the grading rubrics) until achieving the mastery criterion of 90% correct or higher. Ultimately, all 12 students received a score of 90% or higher on three or fewer opportunities with each assessment. How did you use findings for improvement? As noted above, after the assessing faculty member or practicum coordinator provided individual feedback to the student who did not achieve the mastery criterion after a first attempt, the assessor provided up to two additional opportunities for the respective student to demonstrate proficiency with each procedure. Additional Comments Expected Outcome 2: Proficiency in Oral Presentations and APA Writing Style Student will provide fluent presentations on ABA procedures and demonstrate a working knowledge of APA style. Assessment Method 1: Capstone Paper Assessment Method Description In the second semester of the capstone experience, students write a comprehensive paper using APA style describing the application of various assessment and intervention approaches that were used during their practicum experience. This paper also includes a brief review of literature pertaining to the assessment and intervention procedures employed by the student. ABA procedures involving deliberate manipulations that are made to a learner s environment to facilitate their learning of specific behavior. These manipulations can typically be categorized as antecedent manipulations (things performed by a therapist to evoke the behavior of interest), consequence manipulations (things done by a therapist following a learner s engagement in a specific behavior), and motivational manipulations (things done by a therapist to optimize the consequences that are used for a given learner). The documents below are used by an ABA faculty member to provide specific feedback on various elements of Capstone Paper and Project. Each component of project must be rated as Satisfactory or higher by the evaluator. 7
8
9
10
11
12
13
14
Findings Currently in progress for all 12 graduate students. How did you use findings for improvement? If consistent weaknesses in specific topics (e.g., functional analysis procedures) are identified across several graduate students, we will increase the number of articles and lectures devoted to the topic in the respective courses. Additional Comments Assessment Method 2: Oral Presentation Assessment Method Description In the final two weeks of the third semester, students provide a 60-min presentation, which is based on a topic related to their capstone project, to colleagues, peers, and members of the public. 15
16
Findings Assessment will be conducted in mid to late July of 2014. The faculty advisor provides individual feedback during rehearsal of presentations as well as following the final presentation. Each student from the 2012-2013 gave an acceptable presentation. How did you use findings for improvement? As students have historically struggled to master APA style and appropriate presentation techniques, we have added specific training components to courses that are taught earlier in the ABA course sequence. Specifically, most students struggle to concisely describe the procedures and results from published studies using correct technical language. For example, our Behavior Principles course, which is one of the first courses in the sequence, now requires students to write an APA style review paper. Additional Comments Assessment Method 3: Coauthor Papers with ABA Faculty Members Assessment Method Description Students will coauthor scholarly papers with ABA faculty advisors. Findings Four of the 12 graduate students will have coauthored a paper that is currently under review. The overall percentage of students from this cohort who have or will coauthor papers with ABA faculty members is too low. 17
How did you use the findings for improvement? This is an area where improvement is needed. Program faculty members have considered adding one semester to the beginning of the program to increase students writing and research repertoires. In this way, students may have more developed skills sets when they begin working in clinical settings. These settings provide the most opportunities for conducting research and, subsequently, submitting papers for possible publication. Expected Outcome 3: Proficiency with Single-Subject Designs Consistent with the BCBA Task List, graduates of the ABA program must demonstrate proficiency in the application of single-subject experimental designs. Assessment Method 1: Oral Competency Evaluations Assessment Method Description Students are required demonstrate knowledge of single-subject methodology via oral competency examinations and article reviews provided by practicum coordinators and ABA program faculty. In part, students raise issues about internal validity, external validity, and data collection procedures. See competency checklist on pp. 19-21. Program faculty and practicum coordinators provide oral evaluations to students. The questions involve situations wherein specific single-subject experimental designs should and should not be used to evaluate the effects of behavioral interventions on specific dependent variables. Each student s progress is tracked over the course for all three semesters. Progress is reviewed monthly by the ABA program faculty and practicum coordinators 18
19
20
Findings Common weaknesses among 50% of the graduate students was an inability to distinguish between concurrent multiple baseline designs and nonconcurrent multiple baseline designs. In addition, about 25% of the graduate students struggle to identify suitable dependent variables for changing-criterion designs. Most of the 12 graduate students required multiple opportunities to describe the correct application of various single-subject experimental designs. By the end of June 2014, all 12 graduate students were able to provide specific examples for the correct use of each sub-type of single-subject experimental designs. How did you use findings for improvement? We provided remedial lectures on the subject matter noted above. In addition, we plan to set aside additional time to elucidate the distinctions with these experimental designs when the single-subject design course is taught to the next cohort of graduate students. Additional Comments Assessment Method 2: Comprehensive Exam Part II Assessment Method Description Part II was provided to all 12 graduate students on 6/2/14 (afternoon). Each 21
student was allotted 1 hour and 45 minutes to provide written responses to written questions. In Part II of the comprehensive examination (see below), students are required to demonstrate knowledge of clinical issues surrounding the application of single-subject design methodology and to recognize problems with some experimental designs. 22
23
24
. Findings Each of the 12 graduate students demonstrated proficiency with questions on single-subject methodology provided in Part II of the comprehensive exam. That is, each student correctly identified the type of experimental design that was or should be used to evaluate the effects of a given intervention on a specific dimension of a target behavior. Likewise, each student demonstrated the ability to correctly depict data graphically and to provide the correct conclusion about an independent variable based on graphically depicted data. How did you use findings for improvement? No action taken. Additional Comments Assessment Method 3: Comprehensive Exam Part IV 25
Assessment Method Description Part IV of the comprehensive exam was to all 12 graduate students on 6/4/14 (afternoon). Each student was allotted 1 hour and 45 minutes to provide written responses to written questions. In Part IV of the comprehensive examination, students are required to demonstrate knowledge of data collections methods and algorithms for calculating indices of interobserver reliability. See scoring rubric below. 26
Findings Eight of the 12 students demonstrated proficiency with questions provided in Part IV of the comprehensive exam. Specifically, proficient students correctly graphed raw data, correctly calculated the correct index score of agreement based on raw data using multiple calculation methods, and correctly described the application of continuous and discontinuous recording methods (for documenting behavioral events) and the dependent variables that are produced by each recording method. A common weakness for the four students who did not receive a proficiency rating was a failure to correctly describe the implementation of discontinuous recording methods, a failure to correctly describe the dependent variable generated with each discontinuous method, or both. How did you use the findings for improvement? The program director, Dr. Rapp, provided a remedial lecture on discontinuous recording methods on 6/18/14. Assessment Method 4 27
Capstone Project Assessment Method(s) Description Students complete a capstone project over two semesters that involves the application of two or more single-subject experimental designs to (a) assess the function of problem behavior, (b) evaluate client preferences for various activities, and (c) evaluate the effects of a behavioral intervention on one or more dependent variables. Students receive ongoing feedback from ABA program faculty and practicum coordinators during the conduct of these assessment and intervention activities. Findings Currently in progress. How did you use findings for improvement? Additional Comments Expected Outcome 4: Verbal and Written Proficiency in ABA Principles and Procedures Graduate students in the Applied Behavior Analysis (ABA) graduate program will demonstrate verbal proficiency by describing (both verbally and in writing) behavioral processes, behavioral principles, and application of assessment and intervention techniques in ABA. Assessment Method 1: Oral Competencies Assessment Method Description During the course of the 12-month program, students are engaged in a 750-hour practicum and are required to complete numerous oral competence examinations with practicum coordinators, who are board certified behavior analyst, or ABA program faculty. Performance of each graduate student is discussed during bi-monthly meetings with the ABA program faculty and the practicum coordinators. See competency checklist on pp.19-21. See http://www.bacb.com/downloadfiles/tasklist/bacb_fourth_edition_task_list.pdf for the Task List provided by the Behavior Analysis Certification Board (BACB), which accredits this program. Students are provided feedback on their performance until achieving our mastery criterion. Each student typically requires considerable feedback to reach mastery. Each student s progress is tracked over the course for all three semesters. Progress is reviewed monthly by the ABA program faculty and practicum coordinators. 28
Findings Most of the graduate students required more than one opportunity to demonstrate competence with each item. Ultimately, 10 of the 12 graduate student demonstrated competence across all areas by the end of June 2014. How did you use findings for improvement? Progress is reviewed monthly by the ABA program faculty and practicum coordinators. Feedback is provided on an on-going basis. After individual weaknesses have been identified with each graduate student, program faculty members and practicum coordinators provided additional instructional and assessment opportunities until the student demonstrated competence with each item. Additional Comments Assessment Method 2: Comprehensive Examination Assessment Method Description The four-part exam was provided over four, 1 hour and 45 minute sittings. Part I of the exam (given 6/2/14) evaluates knowledge of behavioral assessment techniques and behavioral intervention procedures. To obtain a proficient score on Part I, students must provide correct written descriptions of response definitions, data collection methods, behavior assessment methods, and to link an intervention to the reported function of the problem behavior. Part III of the exam evaluates knowledge of basic behavioral processes and principles. To obtain a proficient score, student must provide correct written descriptions of various behavior processes such as positive reinforcement, negative punishment, stimulus control, response generalization, and prompting procedures. See rubric for scoring the comprehensive exam below. 29
30
31
32
33
34
Findings Of the 12 graduate students, 10 demonstrated proficiency in their written responses to Part I of the comprehensive exam. The two students who did not demonstrated proficiency did not provide clear definitions of the behaviors that were targeted for intervention. In addition these students did not identify a sensitive data collection method to evaluate changes in the target behavior. In regard to Part III of the comprehensive exam (given 6/4/14), 9 of the 12 graduate students demonstrated proficiency. A common weakness in the written responses of the three who did not provide proficient responses was the failure to list several distinct behavioral interventions for addressing a given function of problem behavior. How did you use findings for improvement? The three students who demonstrated deficiencies with Part III of the comprehensive were directed to remedial readings prior to talking the BACB exam. Additional Comments Assessment Method 3: National certification exam in Applied Behavior Analysis Assessment Method Description The one-year intensive graduate program in ABA is designed to ensure that our students will be able to respond correctly to various questions about the application of behavioral principles, procedures, and interventions. As a direct measure of the ABA program curriculum 35
quality, we evaluate the percentage students who pass the BACB certification exam, which is required to obtain the certification credential in ABA. A high first-time pass rate for students from our program would suggest that our curriculum is sufficient. Findings All 10 students from the 2012-2013 cohort passed the certification exam with only one attempt (see attached document). Student in the 2013-2014 cohort will take the exam in August of 2014. Results of the exam are not available until October or November of 2013. 36
How did you use the findings for improvement? If the first-time pass rate drops below 70% for graduate students from the ABA program, we will need to make changes to the instructional courses in the ABA program. Additional Comments 37