Simulation for Development, Practice, and Assessment of Leadership Skills

Transcription

1 Simulation for Development, Practice, and Assessment of Leadership Skills Adam Nelson, Vangent, Inc. 806

2 Simulation for Development, Practice, and Evaluation of Leadership Skills Adam Nelson, EdD April 17, 2007 Copyright 2008 Vangent, Inc. All Rights Reserved. 1

3 THE MISSION: Develop officers leadership skills Target 42,000 learners, stationed around the world Make exercises relevant to each learner s role and workplace Minimize learner, mentor, and instructor time required The AF recognizes that leadership development is critical to the effectiveness of officers. To address this, they defined a set of leadership competencies required of all officers and indicative behaviors for different ranks / levels of performance. They decided to begin by developing an application-oriented masters degree program to provide officers with a common base of knowledge and skills on which to build. The mission we accepted was to Develop officers leadership competencies as defined by the AF. Design the program to be delivered via distance learning technology to the 42,000 active duty company grade officers i.e., lieutenants through captains - wherever they are based around the world. Make all exercises relevant to any learner s role and workplace, but enable each person to tailor the exercises to leverage work that they already need to do, or that will be helpful to their commanders. Make all readings and exercises as concise and focused on each learner s Air Force role as possible, in order to maximize applicability of the learning and to minimize the time required of mentors and instructors. 2

4 THE SOLUTION: Master s degree curriculum Delivered via Blackboard; SCORM and 508-conformant Use ILS for practice and assessment of soft skills The Air Force determined that an application-oriented, master s degree-level educational program was an appropriate vehicle. All officers have at least a bachelor s degree and a master s degree, while not required for promotion, can be a differentiator, so they believe that this will motivate people. In order to deliver this development to all officers, wherever they may be, and to accommodate schedule vagaries and deployments that may interrupt studies, an asynchronous distance learning approach was selected. The Air Force s existing Blackboard system is the delivery mechanism and courseware modules are formatted as Section 508-conformant HTML SCOs with embedded articles and textbook chapters. Application assignments are conducted in the real world, then Immersive Learning Simulations are used for practice and assessment of competencies without requiring subjective, qualitative evaluation by someone with subject matter expertise in every location. 3

5 In this session, you will learn: How we used simulation in this program to practice and assess soft skills How we use sims to assess and grade performance in defined competencies An efficient approach to designing rich, scenario-based ILSs In this session, you will learn how we use simulation in the Applied Leadership program to practice and assess soft skills, how we use this approach to assess and grade performance based on multiple factors (i.e., the defined competencies and supporting knowledge), and maybe most interesting, the efficient approach we use to designing rich, scenario-based, immersive learning simulations. I won t be talking about the ISD models and theories that inform the ISD approach we typically use when designing simulations. My associate Dan Campbell did that yesterday. In this session, I will focus more on the sim design process. That said, I do want to say just a little about the ISD model for the Air Force curriculum so that you understand you the sims fit into the instruction. 4

6 ISD Model ISD model involves typical DL master s-type exercises as well as simulations: > Expert introduction and synthesis, chapters and articles, discussion boards, multiple choice tests, real-world research, analysis, and writing assignments > Scenario-based simulations (AKA serious games ) for learning, practice, and assessment of leadership skills The ISD model is a hybrid of sorts. We call it A3, short for Acquire, Apply, Assess. A course is composed of units and each unit includes Acquire, Apply, and Assess modules. For the knowledge-transfer (or Acquire ) part of the courses, we begin with an approach that is typical for master s level DL courses. That is, an expert topic introduction with intros, syntheses of the assigned readings, and transitions between the readings helps learners build the supporting knowledge that enables them to develop the defined performance competencies. Readings are textbook chapters or journal articles. Video and audio are used sparingly to maximize accessibility in remote locations and to avoid hogging bandwidth everywhere. After completing the Acquire modules, learners take a multiple choice Self-Check quiz to ensure that they actually did the readings and they know the material well enough to do a good job on the application exercises. Apply exercises in every unit are designed to be tailored to each learner s real world role and location to make the learning relevant and to increase engagement and foster integration and transfer of material learned to improve performance. The goal was to design exercises that can yield useful information and products that help the learner or their command, so that each learner s unit and the larger Air Force benefit directly from the assignments. Discussion boards and other communication tools are available to help learners better understand and integrate knowledge. None of this so far is completely novel. Where the A3 model varies from most DL models is in the use of simulation to support practice and automated assessment of soft skills in application. This is an important requirement of this curriculum, because we are potentially targeting about 42,000 learners, dispersed all over the world (and moving around from time to time). It would be impossible for instructors or mentors to evaluate and provide feedback on every person s efforts and performance on all of the specific defined competencies we are trying to develop. 5

7 Scenario-Based Assessment Flight Commander course opening sequence So, in order to provide realistic opportunities for practice as well as to assess skill performance, we designed scenario-based simulations that also serve as assessment tools. We based the situations, decision points, and embedded information and questions on the defined performance competencies and the comprehension-level enabling objectives that support development of those skills. As components of this curriculum, we ve created 12 simulations. Each simulation, or scenario-based assessment as we call them, is designed to enable learners to practice and demonstrate the specific leadership competencies addressed in the unit. The competencies are grouped under competency headings, like Takes Care of People or Decision Making. Each simulation is designed as the culminating activity of a unit. It enables people to practice and demonstrate the competencies covered in that unit. The Flight Commander course that we re looking at here addresses the human aspects of Takes Care of People and Leveraging Diversity. As you saw, each scenario opens with the skills targeted in that unit, to remind the learner and set the stage. 6

8 Scenario-Based Assessment Shot of setup scene in commander s office We used an Anime look & feel and a fast pace to appeal to the target audience. CGOs are junior officers, ranging from about 21 year old 2 nd lieutenants to about 27 year old captains, so this is familiar to them. The last thing we want to do is put learners into A day in the life of a flight commander. That would be boring. The goal when designing each simulation is to identify a scenario and storyline that provides opportunities to demonstrate the defined skills, and to make it engaging and fun. The scenario in this assessment is that the CGO has been asked by his commander to take over command of a flight which has been poorly led, leading to inter-personal issues and a mission that s behind schedule. The learner has to identify the internal issues and take appropriate corrective action to complete the mission on-schedule. 7

9 Other Scenario Examples In a unit about warrior ethos and leadership bearing, learner plays liaison to a Hollywood war movie production, ensuring that the Air Force and officers behavior are portrayed accurately. In a unit about understanding cultures, learner is sent to a fictional African country after an earthquake to establish relationships with local people and set up a forward air base within a set time limit, in order to get relief supplies delivered. In other simulations, the learner is put in very different situations to provide opportunities to demonstrate the competencies addressed in their preceding units: In a unit about warrior ethos and leadership bearing, they play the role of liaison to a Hollywood war movie production, ensuring that the Air Force and officers behavior are portrayed accurately. In a unit about understanding cultures, they re sent to a fictional African country to establish relationships with local people and set up a forward air base within a set time limit, in order to get relief supplies delivered after an earthquake. In a unit about Speaking and Writing, they assist a lieutenant in developing and delivering a briefing to Iraqi military officers In a unit about Decision Making, they assist a Captain in using the Air Force s AFSO21 problem solving approach to make process improvements In a unit about Followership, they assist a Lieutenant in being both a follower and a leader while leading a team 8

10 Scenario-Based Assessment Shot of setup scene presenting learner role These simulations are designed so that the learner works in a third-person role, advising the protagonist on the additional information to seek out, people they should talk to, what decisions they should make, etc. The main characters sexes and ethnicities vary in the simulations to reflect the diversity of the Air Force. 9

11 Scenario-Based Assessment By clicking a Resources button to access additional information or resources, the learner can explore others opinions, get advice, look up reference information, or go to other locations on-base to get more info before forming a conclusion or making a decision. The screen shown here allows the learner to look at personnel records for several subordinates. Similar screens are used to present others perspectives or to get advice from peers. 10

12 Scenario-Based Assessment Shot of decision screen At key points, the learner is asked to make decisions about the best course of action. In some cases, there are finite choices as in this example. In other cases, the options are open-ended and realistic. The learner may not have the options presented like this and may have to rely on their own knowledge, experience, or the advice provided by other characters in the simulation if they choose to seek that out. 11

13 Scenario-Based Assessment Shot of question screen The learner is also asked about the underlying concepts that led them to the decisions they make in the scenario. Questions are embedded to check the learner s comprehension of key points from the instructional unit that support development of the targeted competencies. Elaborative feedback is presented immediately for these questions. 12

14 Scenario-Based Assessment Shot of open-ended action screen maybe the base map navigation tool? Should show the life meter Here s a scene where the learner reviews an interaction between the main character and a peer. By clicking the frames in the filmstrip, the learner advances through the scene. We use still images and audio to keep bandwidth requirement low and the script is on-screen for 508 compliance or in case anyone can t access or doesn t want the audio. 13

15 Scenario-Based Assessment Shot of open-ended action screen maybe the base map navigation tool? Should show the life meter We also embed questions and tasks within dialogue to make it more realistic. Sometimes, these require the learner to reach out for more information. This is an example of where the learner provides feedback on Captain Lyon s goal setting effort. At any point in the simulation, the learner can consult with classmates using the Blackboard Discussion Boards or , or by phone or by talking in person if they have classmates locally on the same base. 14

16 Scenario-Based Assessment Shot of Sit Rep (Situation Report i.e., interim feedback) screen We provide performance feedback in various ways. In the real world, you may make bad decisions and not know for a while, or maybe not ever, so sometimes we let a less-than-optimal decision go for a while. In many cases in the simulations though, we don t want to inadvertently reinforce misunderstandings or bad decisions, so while we may let the learner progress down an inappropriate path a little bit to see the consequences of their actions, we do provide feedback as soon as it s appropriate. Summary performance feedback is provided at key points in the scenario. For example, when a learner completes a major segment of a simulation, they get an interim performance report. The report tells the learner how they did with regard to each of the defined competencies addressed and how well they answered the comprehension questions. In addition, their overall score is presented. Elaborative feedback is presented on their performance in the completed parts of the scenario. At the end of the entire simulation, a similar After Action Report provides summary feedback for the entire experience, as well as a performance score and the option to submit the score to the Blackboard Gradebook. Learners can use the sim as many times as they want to without submitting a score and once they do submit, they can always choose to submit a later score to overwrite a score submitted earlier. The goal is to encourage learners to go through the scenario multiple times, so that they investigate different courses of action and decisions to see the outcomes of the best choices and also what may happen if they make less appropriate decisions. In this way, the scenario-based assessments serve to support practice and automated assessment of comprehension related to supporting knowledge and application of soft leadership skills. 15

17 Questions? USAF Applied Leadership program Simulation used to practice and assess soft skills Assesses and grades performance based on multiple factors Simulation Design Process Let s take a moment for questions before we continue. As a reminder, so far, we talked about the Air Force s performance development need related to leadership skills for CGOs and we looked at the distance learning program we designed to address that need. Then we looked at the way we designed the simulations to support practice and assessment of soft skills. Can anyone see parallels between this program and any of your performance development needs? Sims like these can be used within a blended curriculum, supporting classroom training, they can be used as elements of a web-based program, and they can even be designed to support discovery learning as well as skills practice and assessment, so that they essentially stand alone as complete learning, practice, and assessment modules. 16

18 Designing a Simulation Designing a Simulation Now let s look at the design process. Simulations can provide the wow factor and if they re designed well, they can be educationally sound, but it s a lot harder than designing traditional ILT or WBT, so I thought I should focus on the simulation design process we use. I m not saying that this is the only way to design a simulation, but it s one way that we ve found to be effective. 17

19 Designing a Simulation Technology is no longer the limiting factor Cognitive overload can lead to a logjam An efficient, iterative, process is critical for success First of all, let s set the stage. Simulations have been around for a long time now, but in the past, the limiting factor in sim design has been technological capability. Designers could come up with lots of cool ideas, but the time and cost involved in making those real were often prohibitive. Technology is still a factor to some degree, but for the most part, the limiting factor today is the instructional designer. I am an instructional designer, so I don t like to admit this, but designing simulations is much harder than designing instructor-led or even web-based training. 18

20 Conventional ISD Define terminal objectives Identify enabling objectives Determine assessment approach Gather or create content Design instruction It s difficult, because there are so many factors that have to be considered at the same time. Whether we design ILT or WBT, we re used to a pretty linear process: We define the desired outcomes the performance objectives We identify enabling objectives what do you need to know to achieve the performance objectives We determine how we ll assess achievement of the objectives Then we gather or create course content in various forms And we design instructional exercises to help people integrate knowledge and build skills in engaging ways Of course there is some looping in the process and some design efforts are harder than others, but I don t think I ve ever had an ILT or WBT design challenge that was as hard as a simulation design effort 19

21 Simulation Design Realistic scenarios to apply specific skills Underlying knowledge Force decision points but also allow choice Less-correct or wrong decisions Realistic consequences Engaging and rich Provide feedback Assess performance Assess underlying knowledge and skills Score performance and calculate a grade Designing simulations is harder, because there are so many factors to be considered. Cognitive overload is a big issue unless the process helps to manage the load. Just think about what these simulations do: Provide realistic scenarios designed to enable people to apply specific skills Be detailed enough so that key underlying knowledge and skills are demonstrated Provide defined, forced decision points and also allow self-directed information seeking and user control of navigation Allow people to make less-correct or even wrong decisions, and keep going - with different and realistic consequences than they would experience if they had made better decisions Provide sufficient richness to be engaging and to provide enough depth of information to make difficult decisions Provide sufficient and appropriate feedback at key points Assess performance or degree of skill in specific defined competencies. Sometimes we also want the sim to assess specific underlying knowledge and skills (we call these key points ) And of course they have to score performance against the defined competencies and calculate a grade. 20

22 Simulation Design Process Complete Prework Construct Storyline Develop Framework Write Scenarios/Storyboard In order to design this type of rich simulation, we use a four-step process that enables us to focus on one level of detail or element at a time, then we revisit each step as necessary and revise cyclically, going through the steps several times as needed, adding detail each time. Each step includes sub-tasks, but it s helpful to think about the process as being composed of four primary steps. First, we complete what we call the prework. That is, we define the skills to be demonstrated. We construct the high level storyline We develop the framework of the scenario And then we write the detail of the scenario and create the storyboards As I said, the process is cyclical. We may loop back several times within a step, and through all the steps, until we have the design done. Let s look at each step a little more 21

23 Complete Prework Defined competencies Key points > High-level understandings > Not detailed enabling objectives In the Complete Prework step, we identify what the sim has to make the learner do and demonstrate We re essentially defining the terminal and enabling objectives for the simulation. We list the competencies or skills that the learner has to demonstrate and the key points that the learner has to understand. The key points don t include all the detail of the enabling objectives for the instructional unit. They just target the most important of those and the high level understandings. For example, a key point might be an understanding that a supervisor s expectations of someone can impact that person s performance, whereas enabling objectives for the instructional unit might be much more detailed including knowledge of the Pygmalion Effect, Expectancy Theory, and other motivation theories. This task can normally be completed by an instructional designer who is familiar with the content, but may involve a SME. 22

24 Construct Storyline Generate outline of scenario > High level situation > Series of main scenes > Key decision points > Other decisions and questions When we Construct the Storyline, we re coming up with the general idea and outline of the entire scenario. This is difficult, because we need to create scenarios that are plausible and realistic so that people can imagine themselves in these situations, but they must be situations that would have the learner really use the knowledge and skills that are targeted for that sim, as identified in the prework stage. They also have to be relevant to Airmen in all sorts of different roles and in a lot of different environments. This task is completed by an instructional designer who is familiar with the content, and has to involve at least one SME, and usually input from members of the target audience. Once a general scenario storyline is constructed, we add detail by planning the sequence of scenes, the characters involved in each, and the main decision points. We go around several times, adding more detail to these elements each time. By the end of the process, we have a pretty good outline, including the situation, the scenes, the players in each one, the decision points, and other decisions that have to be made and the main information the learner will need to make the decisions and move through the simulation. You need to involve people who really know the content and the environment in designing simulations, to make sure that the storyline as well as all the dialogue is realistic. In this case, since I don t have any Air Force experience, I hired designers and SMEs for this project who are recently retired AF officers and we involved active duty CGOs to get ideas from actual members of the target audience. 23

25 Develop Framework > All choices in each scene and decision point > Consequences of each choice > Information needed to decide > Additional information and options > Transitions > Scoring impact Developing the framework is where we start getting into the detailed design. This step is done iteratively, focusing on one element at a time, because there are so many elements. In this step, we start with the high level scenario outline from the Storyline step and flesh out each scene. We have to plan all possible courses of action and options that will be available to users, including: Every right, wrong, and partially correct or less-than-ideal choice Where each possible choice leads the user what are the consequences? The specific information that users will need to make each decision or choose any particular course of action Additional information and actions that users can choose to access if they want to Transitional dialogue, scenes, and screens that will be needed to make the scenario flow smoothly And finally, a real subject matter expert works with the primary designers, to determine how each choice that can be made should impact scoring against each relevant competency and for the aggregate score You can see that even within this one step, there are a lot of elements and it would be unproductive to address them all in a linear way. You have to trust the process. 24

26 Write Scenarios/Storyboard Character descriptions Text Dialogue Media descriptions Reference information Feedback All other detail necessary Finally, we write and storyboard the simulation. This is where we write the character descriptions and the actual text, dialogue, media descriptions, reference information, feedback, and all other elements that become the content of the finished simulation. Again, this is an iterative process, with additional detail being added each time through. The storyboards go through a series of reviews and revisions, both internal and by the client team, before they are approved for production. Then the working draft simulation goes through additional reviews and revisions until it satisfies the designer s and other team members visions. 25

27 Questions? Simulation Design Process > Complete Prework > Construct Storyline > Develop Framework > Write Scenarios/Storyboard To recap, the sim design process we use is composed of four primary steps with several sub-tasks. First, we complete the prework to document exactly what knowledge and skills have to be applied and assessed Next we define the scenario and construct the high level storyline Then we flesh out the framework, detailing most aspects of the simulation Finally, we write the detail of the scenario and create the storyboards, then review and revise several times during the development process. Now we have some time for questions. 26