TOWARDS A NEEDS ASSESSMENT PROCESS MODEL FOR SECURITY, EDUCATION, TRAINING AND AWARENESS PROGRAMS: AN ACTION DESIGN RESEARCH STUDY

Transcription

1 TOWARDS A NEEDS ASSESSMENT PROCESS MODEL FOR SECURITY, EDUCATION, TRAINING AND AWARENESS PROGRAMS: AN ACTION DESIGN RESEARCH STUDY Lebek, Benedikt, Leibniz Universität Hannover, Königsworther Platz 1, 30167, Hannover, Germany, [email protected] Uffen, Jörg, Leibniz Universität Hannover, Königsworther Platz 1, 30167, Hannover, Germany, [email protected] Neumann, Markus, bhn Dienstleistungs GmbH & Co. KG, Hans-Lenze-Str. 1, Aerzen, Germany, [email protected] Hohler, Bernd, bhn Dienstleistungs GmbH & Co. KG, Hans-Lenze-Str. 1, Aerzen, Germany, [email protected] Abstract Employees are considered to be the weakest link in information systems (IS) security. Many companies and organizations started to implement security education, training and awareness (SETA) programs. These provide their employees awareness of information security risks and the necessary skills to protect a companies or organizations information assets. To ensure that SETA programs are efficiently aligned to an organization s objectives, it is essential to identify the most important areas on which to concentrate. In research, there is a lack of generic process models for conducting SETA needs assessments. In this study, we aim to close this gap by suggesting a systematic approach to capturing, evaluating, and depicting the current state of employees security awareness and behavior. Actual behavior is evaluated by determining the target values and measuring actual values with respect to security metrics. In order to contribute to both, practical and academic knowledge, we used an action design research (ADR) approach to draw general design principles from organizational intervention within an international engineering company. Keywords: SETA program, needs assessment, security behavior, security metrics, action design research, process model. 1

2 1 Introduction The proliferation of a wide variety of complex and multinational information security risks leads to major challenges for information security management (ISM). An important issue of IT managers is to determine how to create efficient and sustainable organizational information security. Since researchers refer to employees as the weakest link in information security (e.g. Bulgurcu et al., 2010; Spears & Barki, 2010) security education, training, and awareness (SETA) programs have garnered increasing attention. To maximize the number of prevented and deterred security breaches by explaining and predicting employees security-related behavior, researchers have begun to incorporate multidisciplinary theories, including theories from psychology, sociology, and pedagogy, into integrated information security success outcome models (Karjaleinen & Siponen, 2011). But a generally accepted approach that focuses on basic organizational requirements does not exist. Practitioners face the problem of how the theoretical constructs that were found to be determining employees behavior can be adopted. A gap between theoretically founded explanation of employees security behavior and the need of practitioners to know which interventions to apply can be identified (Workman et al. 2008). As a result, and due to the complex nature of the information security domain, organizations often face difficulties in managing an efficient and sustainable SETA approach when considering personnel security, user access control, and network security (Eloff & Eloff, 2005). In the development process and to ensure that SETA programs are efficiently aligned with organizational objectives, some areas need to receive more attention and in turn should receive more resources (Kruger & Kearney, 2006). To assist organizations in determining a risk and priority measurement, the purpose of this paper is to provide a systematic and individualized approach to capturing, evaluating, and depicting the state of employee security awareness and behavior. To specify organizational needs in enhancing security awareness and behavior, a process model that theorizes the needs assessment in an organizational context was developed and tested in an international engineering company. To build a bridge between research and practice, we adapted a research approach that is relatively new in IS research, namely action design research (ADR) by Sein et al. (2011). With the use of different cycles, ADR allows continuous interaction between researchers and practitioners in early stages. We explore the following research question: What are the design principles for developing and implementing a needs assessment process for SETA programs that considers an organization s individual context? The remainder of this paper is structured as follows: because of the relevance of ADR in this research study, an overview and theoretical basis is presented first, followed by a description of the evaluated process model. Subsequently, the process model is tested within the organizational setting: Actual behavior is evaluated by determining the target values and measuring actual values with respect to security metrics. We conclude with a discussion of general design principles that outline theoretical and practical implications, as well as limitations, and then give an outlook for future research. 2 Research Design The objective of this research study is to derive a needs assessment for a SETA program that can be applied within multiple organizations. Therefore, we chose the action design research approach by Sein et al. (2011) as the underlying research methodology. The term action design research was first mentioned by Iivari (2007) to describe the combination of action research (AR) and design research (DR). Motivated by an increasing debate about the gap between organizational relevance and methodological rigor (Lindgren, 2004; Iivari, 2007), Sein et al. (2011) introduced the ADR approach in order to close this gap by presenting an integrative research approach of AR and DR. In their ADR approach, the authors incorporated two challenges: First, by addressing a problem in a specific organizational setting, ADR takes the influence of practitioners and the ongoing interaction with researchers within the specific organizational context into account. Second, to meet the requirement of academic contributions, ADR designs and evaluates generalized IS artifacts that address a class of 2

3 problems through formalized learning from organizational intervention. Although Sein et al. (2011) primarily see technical products as the outcome of DR, we argue that the ADR approach is also applicable to an extended definition of the term artifact that includes organizational and social aspects of IS (Hrastinski et al., 2008), as well as concepts (Järvinen 2007), models, methods and instantiations (March & Smith, 1995; Hevner et al., 2004). We adopted the four stages of ADR as proposed by Sein et al. (2011): (1) problem formulation, (2) building, intervention and evaluation (BIE), (3) reflection and learning, and (4) formalization of learning (Figure 1). Stage 1: Problem formulation ADR Team Cycle 1 Cycle 2 Cycle 3 Cycle 4 Cycle 5 Contributions Stage 2: Building, intervention and evaluation (BIE) Researchers Practitioners (IT managers) Employees (End users) Applied methods Literature analysis Alpha version Semi-structured interviews Analytical hierarchy process (AHP) Beta version Online questionnaires Process model Design principles Process model for needs assessments Utility for needs assessment Goal question metric (GQM) Stage 3: Reflection and learning Stage 4: Formalization of learning Figure 1. Research design based on ADR approach by Sein et al The first stage was triggered by a problem perceived in the practical setting. In order to conduct a needs assessment for a SETA program, the target organization faced the problem of how to capture the actual level of employees security awareness and behavior. Based on a review of academic and practical literature, the specific practical problem was formulated as an instance of a broader class of problems. An ADR team was formed, made up of researchers from a German university and members of the SETA project team within the target organization, including the company s CIO and the security project manager. The shared competencies facilitated the problem definition and formulation. The adopted problem framing in stage one provides the baseline for the following stages. The BIE stage is consisting of five iterative cycles carried out in a real-world environment in order to build and continuously evaluate a process model to conduct a SETA needs assessment. An initial process model design ( alpha version ) was developed throughout cycle one and introduced to practitioners for the purpose of evaluation in cycle two. The first practical iteration did not shape the third level ( employees ) because of the needed expertise of designing the artifact. Based on feedback from the practitioners, the initial process model design was specified in cycle three ( beta version ). The applicability of the proposed needs assessment approach was tested within IT department of the target company. Feedback of the participating employees was used to refine the model again in cycle five until the final version was reached and adopted by the participating organization. In order to evaluate the process model, stage three (reflection and learning) was carried out simultaneously to the previous stage. On the basis of feedback from cycles one to four, this stage allowed to transfer experiences from 3

4 the specific problem solution within the target organization into knowledge that addresses the broad class of problems. It also helped to gain a clear understanding of the problem due to early evaluation. The fourth stage aims to provide a general solution for the broad class of problems as it outlines the results of this study as design principles. 3 Process Model Development Problem Formulation The present study emerged from a project for developing and implementing a SETA-program within an engineering company. The company operates in 60 countries from its headquarter in Germany with a total of 3,200 employees. The SETA-project is based on the NIST SP standard and consists of four phases: (1) program preparation, (2) program development and implementation, (3) program execution, and (4) program evaluation. Part of the first phase of the SETA-project is the execution of a needs assessment (cf. NIST SP ) to determine the extent of the lack of security awareness and potential need for action by using training- and awareness measures. Although company management is generally aware that the security behavior of employees plays an important role in any information security concept, employees security behavior was considered inadequate. To enhance security awareness, the company used general information security presentations on a regular basis. However, the state of employees actual security behavior was not monitored. As Abdulrazeg (2012) pointed out, security behavior cannot be improved if it cannot be measured, so we saw the need of a structured approach for capturing, evaluating and depicting the state of employees security awareness and behavior. A comprehensive literature review (for details see Lebek et al. 2013) based on the structured approach by Webster and Watson (2002) was conducted to access the current state of information security awareness research. We searched through ten databases: AISeL, ScienceDirect, IEEEXplore, JSTOR, SpringerLink, ACM, Wiley, Emerald, InformsOnline, Palgrave Macmillan. A list of search terms was pre-defined, including security awareness, awareness training, awareness program, awareness campaign, security education, security motivation, security behavior and personnel security. In total 113 articles were identified to be relevant. Results indicate that in the past decade of security awareness research, researchers mainly focused on the application of behavioral-cognitive models (Lebek et al., 2013). These models explain behavioral factors that raise employees security awareness. Researchers have begun to incorporate multidisciplinary theories, including theories from psychology, sociology, and pedagogy into integrated information security success outcome models (Karjalainen & Siponen, 2011). We determined a lack of general accepted meta- or process models that theorize the needs assessment in an organizational context. For the theoretical foundation of the proposed method presented in this paper, we made use of both theoretical and practical models and guidelines. On the theoretical side, we used prior work, for example from Kruger and Kearney (2006). The authors developed a prototype to measure security awareness levels of employees based on six focus areas. However, the needs assessment procedure was underrepresented. On the practical side, we adapted NIST SP , which provides guidelines for SETA needs assessments in organizations. Initial Process Model Design For the design of an initial process model we used primarily two data sources: (1) The results of the comprehensive literature review and (2) the results of semi-structured interviews which were conducted with 6 IT managers of the partner company. The interviews aimed at initially collecting requirements regarding the process needs assessment in the context of a SETA program. To analyze the results of both data sources coding methods were used according to Strauss and Corbin (1990). We applied open, axial, and selective coding to get categories, sub-categories, attributes and relationships out of the raw material. These constructs were finally used to design an initial rudimentary process model ( alpha version ) for identifying information security training and awareness needs (Figure 2). 4

5 Determining target values Measuring actual values Identifying roles and focus areas Weighting importance and risk Definition of target values Comparison and evaluation Measuring actual values Developing metrics Identifying measurement goals Figure 2. Sets up Process model for evaluating information security training and awareness needs The ADR team agreed that evaluating employees security behavior as a whole was inapplicable to determining information security training and awareness needs. For this reason, we decided to implement several perspectives on employees security behavior. First, we assumed that employees in different roles or positions demonstrate different security-related behavior, resulting in a role-based view. Secondly, we adopted the concept of focus areas from Kruger and Kearney (2006). For the purpose of this research study, focus areas are defined as critical risk areas in which the behavior of the employee is evaluated (e.g. use of mobile devices ). Because we assumed that each focus area contains a different risk potential, the focus areas need to be weighted amongst each other. Further, the ADR team supposed that each focus area is of differing importance for the different roles within the organization. For example, the focus area use of mobile devices is obviously less important for roles that do not use mobile devices in their work environment, such as application developers. On the other hand, the focus area is more important for roles with extensive use of mobile devices such as management. After the roles and focus area definition process, the measurement goals have to be defined. Applicable security metrics have to be identified based on the measurement goals. In information security research, the use of self-reporting data to determine employees information security behavior is predominant (e.g. Ifinedo 2011). However, the use of self-reported data to measure security-related behavior is prone to the problems of common method variance, consistency motif, and social desirability, and results may be biased (Workman et al., 2009). Therefore, the integration of empirical data that determines actual behavior (e.g. system monitoring data, incident records) into the measurement process is preferable. With the purpose of defining desired behavior, the importance and risk weightings have to be transformed into specific target values. In order to evaluate the gap between actual and desired behavior, a normalization process is needed to ensure that target and actual values are comparable. The general requirements for a needs assessment defined in the problem formulation stage were refined as shown in Table 1. Determination of desired behavior To determine desired behavior, different observation levels (i.e. roles, focus areas) must be considered. Each focus area must be weighted by its inherent risk potential. The importance of each focus area must be weighted for each role. Measuring employees actual behavior Applicable metrics must be developed based on the measurement goals. Reliable data sources must be included (e.g. system monitoring data, incident reports). Evaluation of the gap between actual and desired behavior Target values and actual values must be normalized in order to establish comparability (e.g. by using a points-based system). Training and awareness needs per role and focus area should be presented in a short table form which is intuitive to IT managers (named awareness map ). Table 1. General requirements for a needs assessment process 4 Target Value Determination Definition of Focus Areas and Roles Following the requirements set up by the initial design of the process model, it was necessary to define the observation levels (i.e. roles, focus areas) in the first instance. The employees roles were 5

6 predetermined by the organization s business processes. In order to get a valid theoretical foundation for defining the focus areas, we utilized an approach similar to the initial process model development as we used the perspectives of prior academic work from e.g. Drevin et al. (2007) and Kruger and Kearney (2006) and semi-structured interviews with IT experts within the company. In their research study, Drevin et al. (2007) derived a value-focused information security awareness approach whose fundamental objectives included a network of key areas that must be taken into account in security decisions. The authors identified thirteen mean objectives, e.g. maximize logical access control, minimize virus infection, and responsible use of and internet. Within these assisting objects, one limitation was that there was no generally accepted information security object with coherent areas or labels that addressed the assessment of information security behavior (Kritzinger & Smith, 2008; May & Dhillon, 2010; Torres et al., 2006). With the purpose of gaining a more practical view on relevant focus areas, we also considered several information security reports from recent years (e.g. Verizon 2011 Data Breach Investigation Report, KPMG - The e-crime Report 2011, CERT 2011 Cyber Security Watch Survey). Based on the literature analysis, a general list of focus areas was prepared. Due to their generic scope, each focus area had to be validated within the context of the target organization. For this purpose, an additional team was formed, consisting of six members who are well versed regarding the underlying topics (named expert team ). The expert team includes three IT managers, one information security manager, one governance, risk and compliance manager, and one IT security expert. We used semi-structured single interviews to present the focus areas to the expert team members. Tape recording supported the authors in collecting and analyzing data accurately. First, all interviewees were asked to select the focus areas that are relevant for the target organization and whether they considered any addition or changes of the focus areas to be necessary. A list of nine critical areas of information security awareness was resulting: access control, client workplace, storage media, mobile devices, software, internet, , handling of critical information, and physical safeguarding of the workplace. Second, each interviewee was asked to determine factors that accounted for each focus area in the project organization from his or her point of view. For example, for the focus area mobile devices, the interviewees named damage to devices, network access, apps, and securing of mobile devices. Focus Area Weighting and Target Value Definition In order to determine the inherent risk potential (RP) for each focus area and the importance (I) of each focus area per role, we made use of the analytic hierarchy process (AHP) as proposed by Saaty (1980). This method was developed to solve complex, multi-criteria decision problems. Four major arguments influenced our decision to use AHP: AHP provides explicit specifications in analysis, intuitiveness, validated measurement scales, and has robust built-in consistency assessments. Following the AHP approach, a specified number of questions were developed for pairwise comparison of the focus area measures. The weights were obtained from the members of the expert team and the company s CIO by using an online questionnaire. The results of the pairwise comparison were aggregated in a (n x n) comparison matrix. Normalized eigenvectors with a sum to one indicated the relative importance/inherent risk for the different focus area measures. For each individual judgment matrix, this procedure was used to derive the average risk and priority matrix for each focus area. Overall weights were built by calculating the average value of each expert s individual weightings of importance and inherent risk, resulting in one matrix for importance (I) for each focus area per role and one matrix for inherent risk potential (RP) for each focus area. The impact value (IV) of each focus area per role IV = I x RP was calculated (Table 2). Subsequently, the calculated impacts were used to determine target values on a scale ranging from 0 to 100 by using a spreadsheet application. In order to explain the awareness level, the following target corridors were derived in accordance with the expert team: = good; = average; poor; 24.9 and less = unacceptable. The lower limit of the section good (=75) was multiplied by (1+IV) for each focus area and role. In order to avoid having target corridors that were too small, a minimum size of the corridor good was set to 10 points. All other lower limits were raised by the same amount. The 6

7 resulting target corridors for two example focus areas, client workplace and mobile devices, are shown in Table 3. Focus areas Roles Onsite staff Management Server Application administration development Client workplace Mobile devices Table 2. Focus Areas Example of impact values Onsite staff Roles Server Management administration Application development G* A* P* G A P G A P G A P Client workplace Mobile devices *Lower limits of corridors: G = Good, A = Average, P = Poor Table 3. Example of target corridors 5 Actual Value Measurement Metrics Development Actual behavior was measured with security metrics. To select security metrics, we used the goalquestion-metric (GQM) approach introduced by Basili and Weis (1984). This validated approach facilitated the selection and implementation of useful metrics and aligned them to the identified focus areas. The GQM method was originally used to develop software metrics, but was also applied in literature in the context of security metrics (e.g. Hayden 2010; Abdulrazeg et al., 2012). In general, the GQM approach consisted of three steps (Ebert et al., 2005). First of all, a clear formulation of concrete goals for improving security behavior was required. The aim of the proposed needs assessment process is to measure employees behavior within organization specific focus areas. Consequently nine goals were derived directly from areas defined above. In the second step, questions were developed from the defined goals. For this purpose, we used the factors which were named during the expert team interviews to define the focus areas. The formulated questions related to the essential aspects of goal achievement. In the third step, the corresponding metrics were defined by the ADR team. Figure 3 shows an excerpt from the GQM approach used within the focus area mobile devices. Goal Appropriate use of mobile devices Questions Which apps are used by employees? How do employees secure the mobile devices? Metrics Number of installations of unauthorized apps Number of devices with installed unauthorized apps Frequency of use of data encryption Frequency of use of PINs PIN Complexity Frequency of leaving devices unattended Figure 3. Excerpt from the GQM approach for the focus area mobile devices Following this process, a total of one hundred metrics were developed for the nine defined focus areas. Subsequently, the results were discussed with the project company s information security manager and IT security expert. During this discussion it became apparent that some of the defined metrics were unnecessary. For example, since the use and complexity of PINs for mobile devices is inevitable due 7

8 to technical restrictions, the corresponding metrics were dropped. Other metrics were withdrawn since no explicit regulations had been defined within the company s security policies. Metrics collection Reliable data sources were determined to collect these metrics. Not every required metric could be obtained from either system monitoring data or incident management records (e.g. frequency of writing down passwords). Due to the sensitive context, additional methods for collecting the required data became necessary. We had to resort to employee self-reports. The use of questionnaires for this purpose provides several advantages (Malhotra 1999): first of all, structured questionnaires are easy to administer and provide reliable and comparable data since the respondents are limited to a predetermined set of answers. Moreover, online surveys can be distributed to all employees via the project company s communication systems and data is collected as soon as an employee finishes the questionnaire, providing a time advantage. However, questionnaires incur another difficulty, the phenomenon of social desirability (Oppenheim 1992; Fowler 1995; Malhotra 1999; Pauls & Crost, 2004). However, keeping the difficulties of obtaining security-related data in mind, (Katoulic et al., 2004) online questionnaires are least susceptible to social desirability and are therefore suitable for obtaining sensitive data (Malhotra 1999). Since questionnaires that control for social desirability have been proven to be inapplicable (Pauls & Crost, 2004), we opted not to implement these controls, but instead took additional steps to mitigate the social desirability effect and facilitate the employees motivation in participating. We assured participants that anonymity and confidentiality measures were in place and communicated the necessity of response accuracy (Fowler 1995). Furthermore, we followed the rules for questionnaire design proposed by Oppenheim (1992). As mentioned above, the proposed needs assessment approach was tested within the company s IT department. Consequently the survey was sent to all 50 IT employees, 29 of which returned a completed questionnaire. At the beginning of the online questionnaire, each participant had to select his or her role within the organization (e.g. onsite-staff, management, server administration, or application development). Based upon role specification, the online survey tool provided a specific set of questions for each participant. For example, the roles application development and server administration were not asked about mobile devices, since they do not use mobile devices during their work. The questionnaire was divided into two sections. In the first section, the employees were asked about security behavior in the focus areas that were relevant for their role according to the expert group weighting. In the second section, the employees were asked about their attitudes towards information security in the respective focus areas. Behavior Evaluation Subsequently, the collected data was normalized for comparison with the target corridors (cf. section 4) and evaluation of the gap. For this purpose, we used a scale ranging from 0 to 100. A score for both the behavior and the attitude measurement section was determined per role and focus area. During the process of collecting system monitoring data and incident records, we had to deal with insufficiently detailed data in some focus areas, i.e. metrics were not drilled down to organizational unit level or even role level. In order to achieve comparability, we resorted to the experience of the expert team members. We found that the experts were able to break down the global metrics to required details. Furthermore, the experts evaluated the metrics using a five-point Likert scale. Points Role specific corridors Score behavior: 89.1 ( ) Score attitude: 82.7 ( ) Score monitoring: 60.0 ( ) Good Average Poor Overall score (Ø): 77.3 ( ) Difference to corridor 'good': = Good, = Average, = Poor, = Inacceptable Table 4. Scores for role application development 8

9 Focus areas Roles Server Application Onsite staff Management administration development Client workplace (-16.43) (+1.9) (-4.92) (-7.73) Mobile devices (-23.75) (-14.97) n/a n/a = Good, = Average, = Poor, = Inacceptable; (+/- X) difference from overall score to the lower limit of the corridor good Table 5. Excerpt from the awareness map of the IT support process After that, the transformation process described above was carried out and a score was that averaged each evaluated metric was determined. The overall score was determined by averaging the three single scores (Table 4). In the last step, the overall scores were compared to the determined corridors for each role. The degrees of goal achievement were transferred to the awareness map (Table 5). The difference between each role s overall score and the lower limit of the respective corridor good was calculated. 6 Formalization of Learning and Discussion of Results Following the ADR approach, we reflected on each step during the problem formulation and BIE stages to learn from the practical intervention. Through formalization, the learning was transformed into general design principles with the purpose of contributing academic knowledge to the respective research field. The final results are presented in Table 6. Design principle Stakeholder integration Perspectives Weighted focus areas Applicable metrics Reliable data sources Normalization Awareness map Table 6. Description It is necessary to consider relevant stakeholders (i.e. management, experts, key-users) to reduce barriers within the organization and understand the purpose. Experts and key-users provide valuable experiences that complement measured data. Different observation levels should be integrated to enable a selective analysis of the current state of employees security behavior. The selection and combination of observation levels depends on the organizational context. Focus areas are critical risk areas of employees security behavior. To determine adequate target values, the risk potential and importance of each focus area has to be evaluated. A standardized process for developing metrics that correspond to organization-specific focus areas is a basic condition to ensure the validity and reliability of measuring employees security behavior. Instead of relying completely on employees self reports, the use of reliable data sources such as system monitoring should be aspired to. However, the integration of system monitoring data requires the establishment of a mature and detailed monitoring process. To make metrics comparable, normalization of data is needed. By depicting results from the evaluation process in an awareness map, needs for training and awareness measures can easily be identified. However, proper documentation of the measurement process is necessary to develop concrete measures. Set of Design Principles At the beginning of the project, the consolidation of stakeholders emerged as a necessary condition to successfully implement the needs assessment process for several reasons. First, the support of the company s top management was needed to emphasize the importance of a needs assessment process. Second, the expert team forms the connector to the human factor. By developing and weighting of focus areas, the expert team fits the needs assessment process to the individual requirements of the organization. Due to their practical experience, the experts were able to compensate for insufficient data from system monitoring. Through the early inclusion of key users, an understanding for the purpose of the project could be accomplished among the employees, which has been proven to be beneficial in the sensitive context of employees information security behavior. 9

10 With the purpose of providing a basis for determining and developing appropriate training and awareness measures, we emphasize the necessity of integrating different perspectives into the needs assessment process. Those perspectives can be roles and focus areas, as in our case, but also business units, departments, or business processes. The combination of several perspectives facilitates a variable consideration of employees security behavior within an organization. Focus areas constitute critical risk areas in which employees security behavior is evaluated. Although several propositions for focus areas exist in literature, organization-specific customizing is necessary (cf. Chapter 3.2). This requires a standardized selection process (e.g. expert interviews, focus group discussion). Based on the assumption that focus areas provide different inherent risk potentials and are of deferring importance for each role, a weighting process is needed. The adoption of the AHP approach turned out to be an applicable method of developing weights in this context. However, the use of online questionnaires to conduct pairwise comparisons entailed unanticipated difficulties. Even though a definition of each focus area was sent to the participants, the expert team members struggled to understand the focus areas. We solved the problem by individually explaining the focus areas to each expert team member as the problem occurred. The online questionnaire consisted of 180 pair wise comparisons, which meant a high workload for each expert team member. This led to a high number of questionnaires being incomplete. To avoid this problem, we recommend using a method that a priori allows interactions between researchers and participants (e.g. focus group discussions) to perform the AHP process. To measure employees actual behavior within the defined focus areas, applicable metrics had to be defined. The GQM approach provides a simple and easy way of developing metrics from the goals set up by the defined focus areas. Data from system monitoring or incident records is considered more reliable data than results from self-reported data. However, the use of questionnaires is necessary in order to gain full coverage of employees behavior and security related attitudes. Additionally, questionnaires are better for subsequent analysis, because results can be compared by using homogeneous scales. A major challenge emerged in regard to the inclusion of system monitoring data. Although we anticipated that adjustments would be necessary to make the data comparable, we discovered that the available data was not sufficiently detailed, e.g. metrics for unauthorized software installations were not drilled down to organizational unit level or even role level. A mature system monitoring process is a necessary precondition for successfully integrating system monitoring data into a SETA needs assessment process. By normalizing collected metrics, the measurements were made comparable. The use of a scale from 0 to 100 was proven to be applicable. The depiction of the degree of goal achievement in an awareness map enables managers to gain a fast initial overview of the current state of employees security behavior and to identify areas that need security training and awareness measures. Furthermore, through step-by-step documentation of the measurement process, a more detailed view of the identified needs was gained, thus providing a basis for developing training and awareness measures. 7 Limitations and Outlook This study is subject to the following limitations: First, in order to solve a specific organizational problem and derive solutions for a class of problems, an ADR approach was used. Even if this study has proven that ADR is suitable for drawing design principles for SETA needs assessment processes from a specific organizational context, only one organization participated in the research process. It can be argued that this fact challenges the generalizability of the study s findings, but Lee and Baskerville (2003) showed that a greater sample size within qualitative studies is not an indicator of greater generalizability. However, artifact quality might benefit from further evaluation and refinement by including several companies into a field study. In addition, cross-organizational differences may affect the needs assessment for SETA programs with regard to external variables. Future studies could investigate differences in branch or company size. The suggested needs assessment process was applied to one business process within the target company and measured employees security behavior in two out of nine focus areas. Since the suggested approach is repetitive 10

11 for each business process and focus area, we do not expect substantial changes to the general design principles when more processes and focus areas are included. However, the design principles can be refined through experience from practitioners and through employee feedback during an organization wide roll-out of the needs assessment process. The focus of this paper was to develop and validate an approach for needs assessments which represents the first step in the overall process of implementing a SETA program. It would be interesting for future research to investigate the long term experiences of the application of the proposed needs assessment approach. Particularly in the context of developing concrete information security awareness and training measures, the suggested approach has to prove its utility, which is part of an ongoing research process as mentioned in the problem formulation stage. In the course of this study, an organization specific list of security metrics was developed. It would be valuable if future research provides a generic list of security metrics in order to complement the proposed process model. 8 Conclusion This research study is a first step to provide a needs assessment process for SETA programs. Based on an ADR process, the gap between organizational objectives and current awareness was explored. For this purpose, we built an ADR team that consisted of researchers and IT managers from an international engineering company. We emphasized the target value definition and development of a reliable and valid measurement process as the two major challenges to conducting a SETA needs assessment within the target company. On this basis, initial requirements for a process model were developed and refined during several cycles of theoretical and organizational intervention until general design principles were set up. After considering the limitations, the suggested process model and particularly the proposed design principles contribute to practical and theoretical knowledge. This study is focusing on the gap between theoretically founded explanation of employees security behavior in academic literature and the need of practitioners to know which interventions to apply. From a practical perspective, the developed model assists organizations in implementing a needs assessment for SETA programs and builds directly on the NIST-SP standard. It supports IT managers in identifying and evaluating the undesired security behavior of employees and provides a basis for developing adequate training and awareness measures. On the theoretical side, this study contributes to scientific literature as it focuses on reducing the lack of generic process models in the context of employees security behavior. Whereas previous research is mainly focused on the adoption of different cognitive factors to explain and predict the security related behavior of employees (Lebek et al., 2013), this study facilitates the development of concrete training and awareness measures to improve employees behavior. The suggested needs assessment approach enables dynamic depiction of the current state of employees security behavior within organizations and its changes over time. This provides the basis for future research to test and evaluate the efficiency of different SETA measures in the organizational context. References Abdulrazeg, A.A.; Norwawi, N. & Basir, N. (2012): Security Measurement Based On GQM to Improve Application Security During Requirements Stage, International Journal of Cyber-Security and Digital Forensics 1(3), pp Basili, V. & Weiss, D. (1984), A Methodology for Collecting Valid Software Engineering Data, Software Engineering 10 (6), pp Bulgurcu, B.; Cavusoglu, H. & Benbasat, I. (2010): Information security policy compliance: An empirical study of rationality-based beliefs and information security awareness, MIS Quarterly 34(3), pp Drevin, L.; Kruger. H.A. & Steyn, T. (2007): Value-focussed assessment of ICT security awareness in an academic environment, Computer & Security 26(1), pp Ebert, C.; Dumke, R.; Bundschuh, M. & Schmietendorf, A. (2005); Best Practices in Software Measurement - How to use metrics to improve project and process performance, Springer, Berlin. 11

12 Eloff, J.H.P. & Eloff, M.M. (2005): Information Security Architecture. Computer Fraud & Security 11(1), pp Fowler, A. & Floyd J.Jr. (1995): Improving Survey Questions: Design and Evaluation, Applied Social Research Methods Series 38, SAGE Publications Inc., Thousand Oaks (CA). Hayden, L. (2012): IT Security Metrics - A Practical Framework for Measuring Security & Protecting Data, McGraw-Hill Publ. Comp. Hevner, A.; March, S.; Park, J. & Ram, S. (2004): Design Science in Information Systems Research, MIS Quarterly 28 (1), pp Hrastinski, S.; Carlsson, S.; Henningsson, S. & Keller, C. (2008): On How to Develop Design Theories for IS Use and Management, ECIS 2008 Proceedings, Paper 138. Ifinedo, P. (2011): Understanding information systems security policy compliance: An integration of the theory of planned behavior and the protection motivation theory, Computer & Security 31(1), pp Iivari, J. (2007): A paradigmatic analysis of information systems as a design science, Scandinavian Journal of Information Systems 19 (2), pp Järvinen, P. (2007): Action Research is Similar to Design Science, Quality and Quantity 41 (1), pp Karjalainen, M. & Siponen, M. (2011): Toward a New Meta-Theory for Designing Information Systems (IS) Security Training Approaches, Journal of the Association for Information Systems 12(8), Paper 3. Kruger, H.A. & Kearney, W.D. (2006): A prototype for assessing information security awareness, Computers & Security 25 (4), Pages Lebek, B.; Uffen, J.; Neumann, M.; Hohler, B. & Breitner, M.H. (2013): Employees Information Security Awareness and Behavior: A Literature Review, Proceedings of the HICSS Lee, A.S. & Baskerville, R.L. (2003): Generalizing Generalizability in Information Systems Research, Information Systems Research 14 (3), pp Lindgren, R.; Henfridsson, O. & Schultze, U. (2004): Design Principles for Competence Management Systems: A Synthesis of an Action Research Study, MIS Quarterly 28 (3), pp Malhotra, N.K. (1999): Marketing Research: An Applied Orientation, third edition, Prentice-Hall International Inc. March, S. & Smith, G. (1995): Design and Natural Science Research on Information Technology, Decision Support Systems 15, pp May, J. & Dhillon, G. (2010): A holistic approach for enriching information security analysis and security policy formation, ECIS 2010 Proceedings, Paper 146. Oppenheim, A. N. (1992): Questionnaire Design, Interviewing and Attitude Measurement, Continuum. Pauls, C.A. & Crost, N.W. (2004): Effects of faking on self-deception and impression management scales, Personality and Individual Differences 37, pp Saaty, T.L. (1980): Multicriteria Decision Making: The Analytic Hierarchy Process; McGraw-Hill. Sein, M.K.; Henfridsson, O.; Purao, S.; Rossi, M. & Lindgren, R. (2011): Action Design Research, MIS Quarterly 35 (1), pp Spears J.L. & Barki, H., (2010): User Participation in Information Systems Security Risk Management, MIS Quarterly, 34 (3), pp Strauss, A. & Corbin, J. (1990): Basics of Qualitative Research: Grounded Theory Procedures and Techniques, Sage Publications. Torres, J.M.; Sarriegi, J.M.; Santos, J. & Serrano, N. (2006): Managing information systems security: Critical success factors and indicators to measure effectiveness. ICIS 2006 Proceedings, pp Webster, J &Watson, R.T. (2002): Analyzing the Past to Prepare for the Future: Writing a Literature Review, MIS Quarterly 26, pp. xiii-xxiii. Workman, M.; Bommer, W.H. & Straub, D. (2008): Security lapses and the omission of information security measures: A threat control model and empirical test, Computers in Human Behavior 24, pp