Mission Critical Role Project

Transcription

1 Mission Critical Role Project Job Competency Modeling for Critical Roles in Advanced Threat Response and Operational Security Testing Authors: MJ Assante DH Tobey TJ Vanderhorst Jr Contributors: R Huber B Rios L Barloon D McGuire Advanced Threat Response Panel Operational Security Testing Panel National Board of Information Security Examiners, doing business as Council on CyberSecurity Department of Homeland Security HSARPA, Cyber Security Division July 2013 This material is based on research sponsored by Air Force Research Laboratory under agreement number FA The US Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright notation thereon. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of Air Force Research Laboratory or the US Government.

2 TABLE OF CONTENTS EXECUTIVE SUMMARY... 3 INTRODUCTION... 6 The Importance of Competency Definition... 6 Practitioner Involvement... 9 Roles... 9 Scenario- Driven Competency Definition METHODS, ASSUMPTIONS, AND PROCEDURES Assumptions and Key Terms Vignettes: Defining Moments of Expert Performance Vignette Identification RESULTS AND DISCUSSION Cybersecurity Roles and Definition Process Results and Findings Summary Implications for Security Programs Implications for Challenges and Competitions Implications for Workforce Development Implications for Human Capital Management CONCLUSIONS REFERENCES APPENDICES

3 EXECUTIVE SUMMARY The search for existing cybersecurity technical talent has moved from a competition to a full- blown crisis for many organizations. Finding, developing, and retaining individuals that demonstrate valued technical skills is a difficult process with few tools and resources for hiring managers to rely upon. Competition can be so great that there now exist have some and have none organizations. This imbalance has resulted in industries and sectors that cannot overcome very real barriers and challenges to improve their security posture against the growing pool of sophisticated threat actors. Given the complex and multifaceted dependencies and relationships between modern organizations, this imbalance of cybersecurity technical talent reduces the cybersecurity posture of all organizations. The crisis label may seem extreme, but one only needs to look at the alarming increase in number of cybersecurity incidents affecting organizations across all industries and demographics. Numerous executive blue ribbon panel studies have been conducted by the defense community, civil agencies, and national security think tanks in an attempt to help address these concerns at a national scale. These studies have laid the groundwork for making talent identification more straight- forward and for expanding the overall talent pool through a federated model for pipeline development. All of these efforts focus on trying to answer the simple question of how does {organization name goes here} identify, pursue, capture, integrate, develop, and retain the talent necessary to prudently manage the risks posed by cyber threats? The current answer is for an organization to take the few cybersecurity technical staff they have and spend a portion of their time trying to find and recognize talent to hire and fill open job positions. This expert eyes approach has weaknesses and certainly does little for organizations that have been previously unsuccessful in attracting that type of talent. Tools, beyond professional certifications, are beginning to emerge to help public and private organizations in the identification of talent for hiring cybersecurity professionals. One category of tools is the use of cyber competitions (contests) or games to identify technical competency. Many of these competitions at the college and the early professional stage are high stakes contests for identifying those able to perform under the pressure of real world, job- relevant performance conditions. Many recruiters are finding these competitions to be an indicator of talent. They feel validated by the stiff competition with other recruiters in engaging winners in employment discussions. As with any game, there can be only a few winners, so what about the remaining field of competitors? How many are viable candidates to fill an organization s competency gaps? How can the competitions tool be further sharpened to find much needed talent with greater precision and in larger numbers? A quick inspection of competitions returns promising aspects that should contribute to the shared goal of finding and attracting talent. At their core, cybersecurity competitions, like other serious games, are expected to be an engaging learning environment (Hoffman, Rosenberg, Dodge, & The field of information security and cybersecurity (IS/Cyber) continues to undergo rapid expansion and change. Federal agencies are increasingly reliant on computer systems and networks to meet their mission requirements. While this has dramatically increased the speed and efficiency with which federal employees can do their jobs, it also creates vulnerabilities for the United States Government and its citizens. Therefore, IS/Cyber are becoming increasingly important as all agencies work to ensure that their systems are secure and their information remains intact and accessible to the right users. Chief Information Officers Council, Information Security Workforce Development Resource Guide December,

4 Ragsdale, 2005; Schepens & James, 2003; Schepens, Ragsdale, Surdu, & Schafer, 2002; White & Williams, 2005). They are expected to attract the best and brightest into the workforce by aligning instructional technology with what motivates the incoming generation of workers, and how they think and learn (Prensky, 2001). But we must ask further questions about the design of the competitions. Are these competitions engineered to reflect the current competency needs in the market? Are the existing needs defined well enough so that competitions can be honed to better suit this powerful purpose? The National Board of Information Security Examiners (NBISE) was established to study methods for rapidly developing cybersecurity job performance models by applying real world work scenarios to identify the desired knowledge, tool proficiency, and human abilities for specific cybersecurity job roles. The simple premise was to use an offensively- informed (i.e., representative cyber attack scenarios) and practitioner- focused process to identify competency elements over a short period of time. These models could then be used to validate job requirements and serve as a basis for the development of training curriculums, formative and summative measurement, and competitions and challenges. Based on the US Cyber Challenge (USCC) competition framework, the Department of Homeland Security (DHS) Science &Technology (S&T) sponsored an exploration of job performance model creation and its potential viability to assist the competition development community and talent acquisition programs. An important question arose: Can these models serve as a lens to identify talent by tuning existing scoring models for competitions? Also, can the process of identifying real world scenarios serve as a resource for competition design and developers in building a game/challenge? The very process of engaging practitioners filling the representative job roles brings immediate value by more clearly defining or simply validating the competencies that are being sought. Scenario- driven competency modeling can: 1. Expose future cyber defenders to realistic ground truth scenarios to prepare them for the demands of real world job performance 2. Highlight the critical mission areas and roles that are instrumental to organizations/agencies 3. Dissect the work being performed to highlight goals, objective metrics of performance, responsibilities, tasks, methods, tools, and, of course, varying knowledge, skill, and abilities 4. Illuminate the finely tuned, situated expertise that is able to address the emergent (unknown) problem or dilemma that cannot be adequately addressed by those with less expertise 5. Provide a library that can be used to structure exercises and drills to evaluate work processes and defense teams to identify short falls to be addressed and hone responses The purpose of this initial study is to demonstrate how to develop the components of a job performance model that may be used to support workforce development and/or to assist cybersecurity competitions to support aggregation and comparison of participant performance. The approach can be described in four steps: 1. Establish vignettes (or scenarios) that define situated expertise in job roles; 2. Detail the goals and objective metrics that determine successful performance; 3. Identify the responsibilities by job role necessary to achieve the objectives; 4. Detail the tasks, methods, and tools along with how competence may differ in level of fundamental or differentiating indicators of expertise, or the level of Volatility, Uncertainty, Complexity, Ambiguity (VUCA; Johansen, 2007) that indicates the difficulty of achieving that level of expertise. 4

5 The study was focused on two specific job roles that were identified as mission critical by a DHS study conducted by the Secretary s Homeland Security Advisory Committee (HSAC) i. NBISE leveraged its standing subject matter expert panels in Operational Security Testing and Advanced Threat Response to engage in the job performance model creation. The group was asked to analyze the list of DHS Mission Critical Job Roles, specified in the HSAC report (Task Force on CyberSkills: security- advisory- council- hsac#3). They selected two roles that were best represented by the seasoned practitioners on the panel and served as good starting points for DHS. The panel was then asked to brainstorm a series of scenarios (referred to as job vignettes) and select a single scenario that would exercise a large portion of the job performance model for the selected job role. The selected scenarios were further elaborated until they obtained a rich description of the story that best exercised the competencies of the selected job role. The scenario was organized into steps or logical stages representing the type of work being performed by the role to address the scenario. This was then used to identify the goals and responsibilities of the job role being modeled. The next series of exercises relied upon both the scenario and the job responsibilities to identify the necessary knowledge that is required, basic tool proficiency, and important underlying human abilities. This report describes the process and presents the outcomes of each panel exercise. It also captures the insights and observations from the subject matter experts as they review the outcome of their own exercises and evaluate the scenario- driven job performance model. The project team and panel members further considered how the resulting job performance models could be applied to strengthen cybersecurity workforce programs, initiatives, and frameworks with a special look at assisting in the design and tuning of cyber completions. In short, the integration of mission- critical role definitions with experiential game theory enables substantial improvement in cybersecurity competition program evaluation models and techniques. The purpose of this study was to demonstrate how to develop the components of a job performance model that may be used to support workforce development and/or to assess cybersecurity competitions to support aggregation and comparison of participant performance. Once such a validated model for scoring performance has been established, competition programs may be evaluated on outcome measures such as generalizability of scores, participant engagement, and support for growth and diversification of the workforce. In this way, a rigorous development of job competency models can directly support the rapid development of a capable workforce in a relatively short span of time, without much of the trial- and- error that has often accompanied the evolution of new professions. 5

6 INTRODUCTION Defining Mission Critical Roles The Importance of Competency Definition Cybersecurity is a contest of competence vulnerabilities are limitless because they emanate from constantly expanding human intellect, imagination, and ingenuity and are the artifacts of complexity. The mission of the cyber defender is therefore continually shifting; best practice heuristics have a half- life approaching zero as every day brings new attack vectors, exploitation techniques, or exfiltration targets. According to research on judgment and decision- making, work focused on such novel, highly variant, or rare problems defines a competency- based domain (Smith, Shanteau, & Johnson, 2004): Such tasks require decisions to be made and actions taken in the face of ambiguous and/or incomplete information. Time pressure is frequently great, and the penalties for failure are often severe. The research shows that competence- based professions have difficulty identifying and defining optimal performance. Optimal performance must be defined and measured differently by stage of expertise development and for the unique contributions of knowledge, skill, and ability. Learning curves are steep in competence- based professions. Thus, what is optimal performance differs greatly between beginners, or those merely proficient in methods and tools, and the skilled competent or expert performers. Mastery must be evaluated across the full multidimensionality of competence (Tobey, Reiter- Palmon, & Callens, 2012), combining the following: depth of understanding comprising knowledge; consistency of skills honed through practice; and the generative capacity of abilities by which knowledge and skill are adapted to effectively respond to increased volatility, uncertainty, complexity, and ambiguity that typify competence- based domains (Johansen, 2007; O Neil, Assante, & Tobey, 2012). Finally, while expertise may be apparent in hindsight, performance models that seek to predict or accredit competency must distinguish between fundamental tasks that define base levels of competence and the differentiating tasks in which both the methods used and outcomes achieved differ across stages of the learning curve (Tobey, 2011). Job performance in these domains is highly subject specific, method or tool specific, or scenario dependent. Measurement of optimal performance is more difficult because indicators are needed at a level of detail not typically found in competency models (Campion et al., 2011). Additionally, interrelationships among multiple competencies or across multiple job roles must be defined which are usually not identified in job task analysis, including: action or task- level competence, domain or subject competence, cognitive or intellectual competence, emotional or social competence, and meta- competence by which one may accurately gauge personal efficacy, engagement, and ethical stance in the performance of job duties (Le Deist & Winterton, 2005). Thus, in competence- based domains like cybersecurity the definition of mission- critical roles is highly situated (Lave & Wenger, 1991) grounded in scenarios whose truth is continually constructed as the interplay between attacker and defender play out in a contest of adaptive expertise (Assante & Tobey, 2011). Each scenario involves different goals, objective metrics of performance, responsibilities, tasks, methods, tools, and, of course, varying knowledge, skill, and abilities. 6

7 According to Brown, Collins and Duguid (1989), situated expertise becomes embedded through the interaction of declarative and procedural knowledge during skilled application. Individuals proficient in understanding of the domain must undergo cognitive apprenticeship by which the procedures they have learned become generalized and adaptive as a result of varied practice, collaboration, and reflection. Competence forms through repeated application of knowledge and skill. Novices, beginners and the proficient use reasoning based on procedures and rules, but these permit resolution of only well- defined (textbook) problems. The competent practitioner has developed skills through repetitive application enabling them to reason by causal models that demonstrate the situational awareness necessary to address ill- defined (unknown) problems. The expert and the master differ in the degree of ability developed to adapt these causal models and habituated skills. They reason through stories or, more accurately, vignettes that demonstrate finely tuned, situated expertise which is able to address the emergent (unknowable) problem or dilemma that cannot be adequately addressed by those with less expertise. In summary, mission- critical roles in cybersecurity must be defined at increasing depth of detail to align with the conceptualization and action repertoires of masters, experts, competent practitioners, proficient students, beginners, and novices. These decreasing orders of competency align with representation of situated expertise as vignettes (master level), goals and objectives (expert level), responsibilities (competent level), tasks (proficient level), tool procedures (beginner level), and domain knowledge (novice level) as the basis for performance. Optimal performance at each level differs. Accordingly, assessments or evaluations of performance should consider both the stage of expertise development and the fundamental or differentiating nature of the task. Finally, the development of mission- critical competence in cybersecurity requires the opportunity to fully engage in a contest of competence by which expertise is grounded in the volatility, uncertainty, complexity, and ambiguity that typifies the real- world environment of this competence- based domain. Developing Mission- Critical Competence Through Competition We contend that the effectiveness of a game feature is contingent on the ability of designers to align the complexity of the serious game with the limitations of human processing capacity Wouters, van der Spek and van Oostendorp (2009) Current practices in serious game research Cybersecurity competitions are serious games they are contests of competence that seek to edify and engage, more than to simply entertain (Garris, Ahlers, & Driskell, 2002; Vogel et al., 2006). Beyond their educational mission, cybersecurity competitions are developed to assist in recruiting and selecting the next generation workforce. These are high stakes contests for identifying those able to perform under the pressure of real world, job- relevant performance conditions. They involve competing goals, which must be prioritized and satisfied in a highly competitive setting. Though the best provide many forms of feedback, outperforming an adversary provides the ultimate indicator of a player s competence. But at their core, cybersecurity competitions, like other serious games, are expected to be an engaging learning environment (Hoffman, Rosenberg, Dodge, & Ragsdale, 2005; Schepens & James, 2003; Schepens, Ragsdale, Surdu, & Schafer, 2002; White & Williams, 2005). They are expected to attract the best and brightest into the workforce by aligning instructional technology with what motivates the incoming generation of workers, and how they think and learn (Prensky, 2001). Cybersecurity is not the only discipline seeking to use simulations and serious games to grow its workforce. Serious games and other challenges are used to entice young talent across a broad array of 7

8 science, technology, engineering, and math (STEM) disciplines (e.g., Mountain, 2004). For example, the ACM International Collegiate Programming contest has been operating for over 40 years and currently more than 7,000 teams and tens of thousands of students compete across nearly 90 countries. These contests have increased awareness of STEM professions. However, STEM competitions have yet to attract a diverse and growing workforce (Shilov & Yi, 2002). Enrollments continue to fall with grave implications for developing a sufficient and competent labor pool (Assante & Tobey, 2011). In a study comparing a broad spectrum of STEM competitions in general science, robotics, and cyber defense, Rursch, Luse and Jacobson (2010) found that despite increasing awareness of their respective discipline, competitions have failed to improve the diversity of the workforce, nor have they reduced the decline in numbers of people entering STEM- related careers. Recent studies suggest, however, that the promise of engaging the next generation in cybersecurity careers may be realized if there is better alignment between game design and the developmental stage of a participant s expertise. The design of games that present challenges adjusted to the learning state and competence of the player is called game balance (Kiili, 2005). For instance, Joiner et al (2011) discovered that game design factors have differential impact by gender. Women are more motivated than men to participate in games that adapt to their competence level and are focused on formative assessments that guide learning, rather than scores providing a summative assessment of achievement. Other studies applying game balance techniques are finding that they motivate students to achieve mastery in the discipline while at the same time increasing their persistence to learn. These studies show, as Phillips (2013) stated: A good teacher challenges her students, understands their struggles, and provides needed encouragement. A [good] game provides the same level of interaction, but with the added benefit of embedded assessments a student's progress is continually tracked The continual guidance towards higher and higher learning goals is called scaffolding. Adapting challenges based on the current level of the participant helps to develop critical thinkers that become engaged in and committed to a discipline, and increase motivation to learn. Thus, scaffolding is the difference between a serious game that increases awareness and one that fosters deep learning (Chin & Brown, 2000). However, the opposite effect the disengagement of the participant from a profession may result if a competition is too challenging. Wouters et al (in press) conducted a meta- analysis of the cognitive and motivational effects of serious games. They found students developing foundational competence through drill and practice saw no benefit from participating in a serious game. Further, the study showed that competitions that occur in a single, continuous session were actually less effective than traditional instruction. A recent study of the National Cyber League (NCL) inaugural competition season (Tobey, Pusey, & Burley, in press) offers an explanation for the failure of competitions to match or improve upon traditional classroom instruction. Overall, during the multi- event NCL season students who participated across multiple sessions showed a significant increase in all measures of engagement: dedication, absorption, and vigor towards participating in cybersecurity activities. However, there was also a notable decline in engagement for those with little experience in the field those who had participated in less than two events. These participants frequently dropped out of the competition before the season was over. The conclusion drawn from the data suggests that improved game balance that engages students through facilitating deep learning is needed in cybersecurity competitions if they are to accomplish the objective of expanding and enhancing the cybersecurity workforce: 8

9 The growth of participation in competition events is generally presumed to be increasing the number of entrants into the field. The analysis of the NCL data indicates that competitions may actually be constraining or detracting from this growth. It may be possible that competitions discourage those with little prior experience in cybersecurity. At a minimum, this analysis seems to strongly suggest more research is needed to understand the difference in change to perceived engagement between those with little to no experience and those who are entering their second (or greater) competition. (Tobey et al., in press) Practitioner Involvement To identify and develop the job performance components of the mission- critical roles, this project leverages the expertise of subject matter experts (SME) available in two of NBISE s job performance panels. NBISE job performance panels are assemblies of experts from industry stakeholders, government agencies, research institutions, service companies, and security product vendors who work to identify critical job roles that make up the cybersecurity workforce of today and tomorrow. They collaborate to define competency models for those roles and develop a standards- based library of validated assessment, curriculum, and simulation- based learning components. For this project, NBISE guided the Advanced Threat Response (ATR) and Operational Security Testing (OST) panels through the job competency definition process driven by scenarios that represent ground truth and properly capture the necessary job competencies. To facilitate the panel SMEs in this process, NBISE supported the panels with a technology suite that included tools for scenario (vignette) driven elicitation, collaboration, performance measurement, task characterization, and role identification. The 23- member ATR panel is focused on advanced cyber security threats such as advanced persistent threats and other highly sophisticated threats (see Appendix A for roster). The 31- member OST panel is focused on penetration testing, red teaming, and attacker emulation testing (see Appendix A for roster). With the involvement of these 54 SMEs from the two panels, this project supported over 700 hours of SME input to help DHS understand the job competency requirements for the mission- critical roles. Roles Mission critical is a term often used to identify those people that are unable to leave early in the face of an approaching foul weather system or have to brave the elements when everyone else has an unplanned day off. This particular definition captures some of our use of the term, but a more useful definition is functional job roles that bring the necessary know how, competencies, and practices to accomplish the mission of an organization. We focused our efforts on job roles that fall into the cybersecurity domain that contribute to the cybersecurity posture of an organization in a material and more direct manner. 9

10 These are the job roles that have very short line of sight to the health and security of an organization s information and communication technology. A recent published article ii coined these job roles as being in the Red Zone, to distinguish them from other cybersecurity job roles that are important but are less direct in impacting the security of deployed systems. Our definition of Mission Critical for this study is as follows: 1. Mission Critical is used to define the importance of the work to be performed by the cyber functional role as being critical to the defense of an organization/agency s information systems. 2. Functional Job Role is a label given to a category or classification of job roles based on their sharing a significant number of common goals (i.e., functions). 3. Job Role is a label given to a category or classification of job titles based on their sharing a significant number of common responsibilities or job duties. The universe of functional job roles that were evaluated for selection for this project came from recommendations put forward to the Secretary of DHS by the Homeland Security Advisory Committee (HSAC). NBISE asked its subject matter expert panels to analyze the list of DHS Mission Critical Job Roles, specified in the HSAC report and select two roles that were best represented by the seasoned practitioners on the panel and served as a reasonable starting point for DHS. The HSAC report stated: Red Zone Think about what the phrase red zone conveys in American football: when defensive players have their backs to the goal line, the situation demands peak performance because the threat is imminent and has to be turned back. Similarly, defender roles within the CI/KR sector s red zone need to be ever present and the capabilities of the individuals in those roles need to be fully developed to achieve peak performance. The pipeline of people moving into the workforce that have the necessary skills, knowledge, and capabilities to perform the critical red zone jobs compared to the pipeline of people exiting those positions is not balanced. This unbalanced condition seems to be worsening as the number of individuals exiting is increasing, the need across multiple sectors is growing, and the available programs or development capabilities have remained flat. Tim Conway, NBISE Smart Grid Cybersecurity panel chair, Control Engineering Magazine, April 2013 On June 6, 2012, Secretary Napolitano announced the formation of a Task Force on CyberSkills with a two- part mandate: first, to identify the best ways DHS can foster the development of a national security workforce capable of meeting current and future cybersecurity challenges; and second, to outline how DHS can improve its capability to recruit and retain that sophisticated cybersecurity talent. The HSAC report further detailed their rationale for calling out and defining Mission Critical Roles : In her tasking letter posing this challenge, Secretary Janet Napolitano said that DHS needs a workforce with specialized knowledge and skill to carry out its mission. The Task Force s first job was thus to identify those specialized skills without which DHS cannot meet its cybersecurity responsibilities (called mission- critical tasks and mission- critical skills ). Explicit definitions of the required skills are needed to enable DHS to differentiate between people who actually have those skills and people who may have knowledge in the area but no hands- on skills. Explicit definitions are also essential to meet the Task Force s charge to identify the most promising and effective 10

11 competitions, university programs, internships, private sector programs, and relevant federal government programs that may be valuable as partners or sources of talent for the Department. (HSAC Task Force on Cyber Skills) Table 1. List of DHS mission critical roles with alignment to ATR/OST roles Mission Critical Roles* ATR roles OST roles System and network penetration tester X Application penetration tester X Security monitoring and event analysis X Incident responder in- depth X Threat analyst/counter- intelligence Analyst X Risk assessment engineers Advanced forensics analysts for law enforcement Secure coders and code reviewers Security Engineers operations Security engineers/architects for building security in *From DHS HSAC CyberSkills Task Force Report Fall This table presents the Task Force s recommended list of mission- critical jobs. Both panels engaged in a discussion and a voting process to select the roles to develop scenario- driven job competency models. The initial project was scoped for two functional roles to serve as the basis for the underlying responsibility and accompanied competency model, but it is important to note that the brainstormed scenarios that were selected applied to multiple cybersecurity functional roles and can in many cases be used in future projects to help identify the responsibilities and competencies of non- selected or listed cyber roles. The ATR panel identified three jobs as aligned with their panel focus and membership; these jobs are Security monitoring and event analysis, Incident responder in- depth, and Threat analyst/counter- intelligence analyst. The OST panel identified two jobs as aligned with their panel focus and membership; these jobs are network and system penetration testing and application penetration testing. Table 2 includes descriptions for these five job roles as provided by the HSAC. ATR Table 2. Description of ATR and OST related HSAC identified Mission Critical Roles Role Description Security monitoring Identify indicators that show an incident has occurred and initiate swift and event analysis response, differentiating between those incidents that represent impotent attack vectors and those that need to be analyzed in- depth by the incident responders. Many other tasks are performed by the security monitoring and event analysis staff, but the ones described here are the critical tasks for which skills are in very short supply. Incident responder in- depth Implement proactive measures to contain the incident, including isolation, characterization, reverse engineering, assessment of capability and activity of malicious software that has been found on agency systems, identification of intruder local changes/suspect interactions, triggering of targets to evoke malicious behaviors, and development and deployment of eradication tools. Only 2% 10% of all malicious software needs to be put through this deep analysis; the remainder will be cleaned with anti- virus tools using current and updated signatures. However, the 2% 10% constitute the most dangerous payloads. 11

12 Threat analyst/ Counter- intelligence analyst System and network penetration tester Deploy deep and current knowledge of the attack surface, its most vulnerable and high value targets, and how its technical vulnerabilities may be exploited; maintain up to the minute situational awareness on what malicious actors are using and targeting; and develop techniques and program custom tools to detect local changes, identify suspect interactions, and watch for and respond to what the malicious actors are doing. More advanced teams also are able to understand the attackers motivation, language, organization, and social behaviors, as well as group the threat actors logically to create effective cyber profiles of groups, actors, and campaigns, thereby helping organizations become more proactive in their security posture and defense. Follow a systematic process to assess the ability of systems and networks to withstand sophisticated adversaries who have knowledge of the architecture and systems that are deployed. This is not social engineering or running a vulnerability testing tool or a packaged exploit tool, but rather a sophisticated technical testing of the configuration and pathways and interactions between systems that mimics the techniques employed by advanced adversaries. Application Test applications before they are deployed and when they are modified. penetration tester Identify the avenues that are most riddled with flaws and holes and that give malicious actors access to the most important content or systems. This is not only a tool- deployment task; it also requires deep understanding of the application being tested. Reference: Homeland Security Advisory Council s CyberSkills Task Force Report, Fall

13 Scenario- Driven Competency Definition iii The cybersecurity mission is to ensure the security, accuracy, and timely transfer of information (Seddigh et al., 2004). The ultimate goal is to provide assurance that a computer- based system is reasonably protected by reducing exploitable vulnerabilities and insecure behaviors, while maintaining an ability to detect and respond to security incidents and intrusion. This mission is therefore similar to that of other engineering professions, which assess and assure safety. Recently, best practices in safety assurance have been adopted by cybersecurity researchers seeking to develop an evidence- based approach to improve information assurance (Gandhi, Siy, & Wu, 2010; Goodenough, Lipson, & Weinstock, 2012). n this section, we will briefly describe the relevance and importance of safety assurance case modeling for our analysis of mission- critical roles in cybersecurity. The safety assurance case method was originally developed by Kelly and colleagues (Kelly & McDermid, 1997, 2001; Kelly & Weaver, 2004; Weaver, McDermid, & Kelly, 2002) to document, validate, and evolve safety assessments based on lessons learned from implementation of new technologies. A similar method is used to guide development of competency assessments in large- scale credentialing programs, such as those operated by the Educational Testing Service (Mislevy, Steinberg, & Almond, 2003). Across several studies, Kelly and colleagues showed that this scenario- based method facilitated common understanding of system vulnerabilities and faults across all stakeholders, e.g., system designers, safety professionals, industry regulators, and certifying authorities. Importantly, their studies also showed that it facilitated rapid adaptation of system designs and remedial actions necessary to reduce the number and negative impact of safety incidents. Accordingly, this method may address a critical issue facing the cybersecurity profession: the dynamic nature of threats and attack patterns require that mission- critical roles and task assignments are continually updated based on evidence gathered from the latest tactics, techniques and procedures used in cybersecurity incidents. This infusion of ground truth in workforce planning means that cybersecurity workforce programs must be well integrated and constantly adapted (Assante & Tobey, 2011). However, maintaining alignment and currency among the various workforce development programs and tools, such as training, simulations and certifications, is a constant challenge. For example, a recent application of the assurance case method to analyze cybersecurity workforce programs in the energy sector found several important job responsibility areas were either missing or showed wide variance in emphasis among competency frameworks, course designs, and certification programs (Assante et al., 2013). Goodenough, Lipson and Weinstock (2012) adapted the safety assurance case argument method to develop an evidence- based practice for information assurance that facilitates assessments of system safety, security and/or reliability. Their adaptation of the Goal- Structuring Notation Method for safety assurance (Kelly & Weaver, 2004) provides a step- by- step process for modeling cybersecurity scenarios. The first step in the assurance case method is the definition of a case that captures one or more critical vulnerabilities, system failures, recovery actions and consequences. Each case is further elaborated by creating a structured story or vignette that enumerates primary and subsidiary goals; objective measures of expected outcomes or operation; challenges to these goals and objectives introduced by one or more exemplar incidents; and the process steps, strategies (or job responsibilities), and tasks necessary to recognize and effectively respond to return the system to an acceptable operating state. Finally, tools are identified which will provide the evidence necessary to indicate whether a vulnerability has been detected or an intrusion has been thwarted. In this study, we applied the assurance case process to demonstrate how it may help to identify the situations and conditions that determine the development or demonstration of competence in mission- critical roles. As discussed above, cybersecurity work is characterized by decision- making that 13

14 must be made under high levels of uncertainty, ambiguity, and time pressure where optimal performance is difficult to decipher. Studies of similar jobs in military contexts (Gompert, 2007) show that performance in these contexts is highly situational, where decisions must be made on the fly (Franke, 2011). Success therefore depends on effective sensemaking (Weick, Sutcliffe, & Obstfeld, 2005). What differentiates the competent from the merely knowledgeable is the speed and accuracy of incident pattern recognition and classification into known scenarios. This finding is consistent with studies of chess masters which show that recall of game scenarios, especially during the first few moves, are highly predictive of the level of skill (Charness, 1991). Similarly, cybersecurity skill requires much more than rote memorization. Expert cybersecurity professionals, like their counterparts in military counterintelligence or chess mastery, must possess sufficient situational awareness (Endsley, 1995) to adapt the response to meet the unique requirements of the situation. These studies also show that simulation systems, such as those used in cyber competitions, can be effective training and assessment mechanisms if, and only if, the scenarios used are realistic and grounded in detailed case definitions (Zbylut & Ward, 2004). These studies show that fostering learning and the engagement necessary to create the active learning environment needed to develop adaptive expertise requires that the scenarios focus on tasks, tools, and methods that differentiate the performance of novices and beginners from those who are competent or expert cybersecurity professionals (Gandhi, Tobey, Reiter- Palmon, Yankelevich, & Pabst, 2013). Thus, a scenario- based approach to competency modeling will enable the development of challenges and assessments that are much more effective than a knowledge recall test at determining skill levels in a competence- based domain. 14

15 METHODS, ASSUMPTIONS, AND PROCEDURES Modeling Ground Truth Assumptions and Key Terms A primary purpose of this study was to demonstrate how a job performance modeling approach (Tobey, 2011; Tobey, Reiter- Palmon, & Callens, 2012) accelerates the process of job task analysis while improving the depth and breadth of analysis typically conducted in the preparation of a competency model for training or assessment, such as the NICE Framework (National Institute of Standards and Technology, 2011; Paulsen, McDuffie, Newhouse, & Toth, 2012), or in developing a cyber- competition design (Conklin, 2006; Schepens & James, 2003; Schepens et al., 2002). Figure 1. Basic Job Performance Model Process Vignettes: Defining Moments of Expert Performance Defining the context of job performance is essential because of the situated nature of expertise in cybersecurity. The term critical incident is often used to describe varying situations in which expertise is exhibited in competence- based domains (Benner, 1984; Boyatzis, 1982; Klein, 1998). Incident, as the word is used here, is not simply an event requiring a response. Instead, it represents a defining moment when differences in skill level are notable in clearly identifiable outcomes of action taken. This may be an actual or a potential event, and includes not only sense- and- respond situations, but also proactive or sustaining events critical to achievement of goals and objectives. Hence, the word incident here is more broadly defined. We therefore use the definition of incident proposed by John Flanagan, the inventor of the critical incident technique for task analysis: any observable human activity that is sufficiently complete in itself to permit inferences and predictions to be made about the person performing the act. To be critical, an incident must occur in a situation where the purpose or intent of the act seems fairly clear to the observer and where its consequences are sufficiently definite to leave little doubt concerning its effects (Flanagan, 1954, p. 327). However the incident name itself does not tell the whole story. In many cases, experts use an incident name to quickly convey a complex and diverse set of conditions and events in a simple, terse manner, especially when conversing with peers (Boje, 1991). Consequently, we prefer the term vignette because it signifies the need to extract the whole story, including several scenarios or differing perspectives of a critical incident (Boje, 1995; Tobey, 2007). Stories are frequently used by experts to 15

16 convert tacit into explicit knowledge for communicating an event to less experienced people (De Long, 2004; Tyler & Boje, 2008). Accordingly, the term vignette describes the collection of: a critical incident title or description; when the incident occurs (frequency and/or action sequence); what happens during the incident (problem or situation); who is involved (entities or roles); and where the incident might happen, now or in the future (systems or setting). Further definition of a vignette might include why it is important (severity or priority of response) and how the critical incident is addressed (method, tools, or abilities that may be needed). A collection of vignettes and the associated job context forms the basis for developing a Job Performance Model that may facilitate comparison with other jobs or to identify when an individual is performing the job as classified. Decomposing goals, responsibilities and tasks to guide assessment of ability We define a goal as a statement that expresses an action that must be successfully completed to accomplish the job mission, or to facilitate the accomplishment of another goal. The goal objective is defined as the measurable outcome that establishes the criteria by which the degree of success or effectiveness may be assessed. Job responsibilities are defined as action statements that result in outcome states that may be monitored or assessed to determine if an objective has been accomplished. Accordingly, responsibility statements use passive verbs, such as "ensure", "follow", or "obtain" that are not included in Bloom's taxonomy. Consistent with its use in task analysis, Schraagen (2006, p. 185) defines the word task as "what a person is required to do, in terms of actions and/or cognitive processes, to achieve a system goal." This definition implies that task statements must be written specifically to highlight the action verb that indicates the execution of the task. It is often the case, though not a requirement of task analysis, that the action verbs used to describe goals and tasks align with Bloom's taxonomy of action verbs (Anderson, Krathwohl, & Bloom, 2001; Bloom, 1956). This definition of a task also helps to clarify the definitions for elements of competency. The three components of competence (i.e., knowledge, skill, and ability) are independent dimensions which may be used to understand an individual s or team s level of competence within a three- dimensional space (Tobey, 2011; Tobey et al., 2012). Knowledge is defined as the understanding of a concept, strategy, or procedure. Thus, knowledge is measured by depth of understanding, from shallow to deep. Knowledge is therefore independent of task performance. Knowledge is identifiable by the capacity to encode, recall, or associate information, independent of context. For example, organizational knowledge is required to Understand what is important to the organization and what is mission critical. Skill is defined as the reliable application of knowledge in the accomplishment of a task to achieve desired outcomes. Thus, skill is measured by the degree of reliability, from inconsistent to consistent, in performance of a task. Skill is always task- specific and context- specific. Skill is identifiable by statements of accomplishment, such as Establish plan for secure storage and transmission of customer data. Ability is defined as a mental or physical capacity to transfer or transform knowledge and skills for application to new domains. Thus, ability is measured by the extent of knowledge and skill transfer, from narrow to broad, typically assessed through the use of physical or intelligence tests. Abilities are task- independent. Abilities include many forms of mental or physical manipulation (Guilford, 1956), e.g., dexterity, locomotion, memorizing, deducing, recognizing patterns, and planning. 16

17 Vignette Identification As a demonstration of the job performance modeling process (Tobey, 2011), the SME panel first defined two job roles that would guide the elicitation of job performance model components. During a complete job performance modeling process, the roles identified during this step would be categorized into functional roles. The list of functional roles would be discussed, or ranked, by the panel of subject matter experts (SMEs) who then select one or more functional roles to focus on for the remainder of the modeling process. This selection of functional roles establishes an important boundary condition for the Job Performance Model. A guide to the selection process may be the roles targeted by a sponsoring organization or roles identified in an existing competency model, such as the NICE Information Assurance Framework ("NICE Cybersecurity Workforce Framework," 2011) in the cybersecurity profession. The ATR subpanel decided to focus on the role of Security Monitoring and Event Analyst. The OST subpanel decided to focus on the role of System and Network Penetration Tester. With the two focal roles identified, the panel members then brainstormed a list of vignettes. In addition to providing a terse description of the critical incident, the panel added examples of the scenarios. For instance, the vignette Adversaries are collecting open source intelligence on your organization to be used for targeting and attack was further defined into a set of scenarios (see results section below), including Honeypots could be triggered based on web- scraping and Utilizing social networking sites to collect information about the company and employees. If this had been part of a complete job performance modeling process, the next step of the process would be to categorize the vignettes into a set of master vignettes (Tobey et al., 2012). However, as the purpose of this project was to demonstrate how the JPM process may be applied to develop mission- critical role definitions, only one master vignette for each subpanel, ATR and OST, was selected for further analysis. These two master vignettes were: Discovery of large amounts of sensitive data posted to internet with no clear signs of intrusion (ATR subpanel); and Conduct of a comprehensive Red Team penetration test against a sensitive national laboratory conducting advanced research with national security implications (OST subpanel). Each of the two master vignettes was further elaborated through a series of six focus group sessions. First, the descriptions of each critical incident was expanded by applying ante- narrative dynamic analysis (Boje, 2001; Tobey et al., 2012) to answer five questions about each incident: what, when, why, where, and how. Second, the SME panel brainstormed a list of process steps for responding to the critical incident. Third, the goals associated with each master vignette were elicited. Fourth, the SME panel brainstormed a list of job responsibilities for each goal. Fifth, a list of knowledge requirements, tools, and tasks (where abilities may be identified) were elicited for each master process step. Finally, the tasks for each process step were sorted by the SME panel into a list of abilities. This categorization was then analyzed to determine the relative importance of an ability to perform the target mission critical roles, both at the overall vignette level and for each process step. Job responsibilities defined in a job performance modeling process may bear some resemblance to the tasks defined during a traditional job task analysis or competency model. In job performance models they represent the starting point for decomposing a job into finer levels of detail. In effect, the responsibilities align with job duties often listed in job descriptions or performance evaluations. One fundamental difference between job performance modeling and previous approaches is the use of multiple roles at this step in the process. Guided by the vignette description, the SME panel defines responsibilities across the entire group of functional roles determined by the panel to provide the role boundary for the job performance model process. This approach enables elicitation of job overlap and 17