1 computers & security 27 (2008) available at journal homepage: Consensus ranking An ICT security awareness case study H.A. Kruger a, *, W.D. Kearney b,1 a School of Computer, Statistical and Mathematical Sciences, North-West University, Potchefstroom Campus, Hoffman Street, Private Bag X6001, Potchefstroom 2520, South Africa b 40 Shalimar Rise, Currambine, Perth, WA 6028, Australia article info Article history: Received 22 October 2007 Received in revised form 26 May 2008 Accepted 9 July 2008 Keywords: Information security awareness Consensus ranking Assignment problem Maximize agreement heuristic Decision making abstract There are many disciplines where the problem of consensus ranking plays a vital role. Decision-makers are frequently asked to express their preferences for a group of objects, e.g. new projects, new products, candidates in an election, etc. The basic problem then becomes one of combining the individual rankings into a group choice or consensus ranking. The objective of this paper is to report on the application of two management science methodologies to the problem of identifying the most important areas to be included in an Information Communications Technology (ICT) security awareness program. The first methodology is based on the concept of minimizing the distance (disagreement) between individual rankings, while the second one employs a heuristic approach. A realworld case study from the mining industry is presented to illustrate the methods. ª 2008 Elsevier Ltd. All rights reserved. 1. Introduction Information security has become crucial to the continuous wellbeing of modern organisations and an information security solution should be a fundamental component in any organisation (Thomson et al., 2006). Information is regarded as an asset (Pipkin, 2000) and as such is exposed to a wide variety of threats and vulnerabilities that require a combination of technical and procedural controls to mitigate risks. Companies often spend huge amounts of money and time on implementing technical solutions, while the human factor in information security receives less attention. Technical solutions are of course necessary to address vulnerabilities to viruses, denial of service attacks, etc. However, the involvement of humans in information security is equally important and many examples exist where human activity can be linked to security issues. One such example can be found in the area of social engineering, where phishing (fraudulent acquisition of sensitive information) has become one of the major problems associated with humans and their levels of awareness. Kerstein (2005) reported that, according to Gartner, between May 2004 and May 2005 approximately 1.2 million computer users in the United States suffered losses caused by phishing. These losses were valued at $929 million. Companies in the United States also lose an estimated $2 billion annually as their clients fall victim to these scams. Statistics from the Association for Payment Clearing Services (APACS) revealed that losses from web banking fraud in the United Kingdom, which were mainly the result of phishing scams, rose by 90% from 12.2 million in 2004 to 23.2 million in 2005 (Finextra, 2006). A key defence in the fight against security incidents that involve human activity, such as the phishing scams referred to above, is the use of ICT security awareness programs. In * Corresponding author. Tel.: þ ; fax: þ addresses: (H.A. Kruger), (W.D. Kearney). 1 Tel.: þ /$ see front matter ª 2008 Elsevier Ltd. All rights reserved. doi: /j.cose
2 computers & security 27 (2008) general, the goal of such an awareness program would be to increase awareness of the importance of information systems security and the possible negative effects of a security breach or failure (Hansche, 2001). The importance of security awareness programs is also emphasized in the South African National Standard on Information Security, where one of the objectives of human resources security is given as to ensure that all employees, contractors and third party users are aware of information security threats and concerns, their responsibilities and liabilities, and are equipped to support organizational security policy in the course of their normal work, and to reduce the risk of human error (SANS 27001, 2006). The development and implementation of ICT security awareness programs imply that appropriate awareness material, activities and actions be developed, implemented and monitored. A wide variety of such material and possible actions is usually available to choose from, and a final decision on what to use and where to focus attention is normally based on the different views obtained from different managers. To obtain the necessary resources in order to develop and implement an awareness program, it is necessary to identify the most important areas on which to concentrate effort and money. This is not always obvious even if the areas have been identified, there may still be a problem in terms of which areas, if any, are more important than others. Questions such as whether all areas should receive equal resources or whether some areas should be regarded as more important and therefore receive more resources may be problematic. The same is true for awareness material which material is more important and should be used more extensively? The problem then becomes one of combining the different management opinions into a group or consensus choice. To assist in determining a consensus priority ranking of security awareness areas and/or security awareness promotion material, this paper investigates the application of two existing management science methodologies to obtain a consensus ranking from different role players. The first of the two methods that are briefly explained in the next section is based on work carried out by Cook (2006) and makes use of the concept of minimizing the disagreement, or distance, between individual rankings through solving a linear assignment problem. Another overview of the technique can also be found in Cook and Seiford (1978). The second method that was applied to the problem makes use of a heuristic called the maximize agreement heuristic (MAH) developed by Beck and Lin (1983). The remainder of the paper is structured as follows: In Section 2 the two models to optimise disagreements are briefly introduced, while Section 3 implements the models in a real-world case study. Concluding comments are presented in the last section. 2. Consensus ranking The basic problem in a consensus priority ranking scheme is one of combining individual rankings into a group choice or consensus ranking given a set of individual rankings on a finite set of alternatives. Ranking problems can be classified into two basic categories, viz. cardinal problems and ordinal problems. A cardinal ranking formulation requires an individual to express a degree of preference in the ranking, while this is not necessary in ordinal formulations. Ordinal problems are called complete ordinal rankings when there are no ties in the ranking and when the transitivity property is present. The problem of combining individual ordinal rankings into consensus has been studied for many years and a number of procedures have been developed to deal with the problem. The simplest form of group consensus is majority rule. Kendall (1962) has proposed an approach where individuals preferences, represented as priority factors, are simply added together and then the average is taken as the consensus choice. Two different techniques in this study were applied to the problem of identifying the most important areas to be considered in an ICT security awareness program and the choice of awareness promotion material to be used. A brief introduction to each of the two techniques follows Distance-based approach Cook and Seiford developed a theory of distance between Kendall s priority factors and proposed a median consensus ranking based on distance. A good description of the axioms, mathematical representation and proof of existence of a unique distance function can be found in Cook and Seiford (1978). An excellent overview of distance-based and ad hoc consensus models in ordinal preference ranking is given in Cook (2006). Shi et al. (1996) have already used this solution technique in a practical situation to determine consensus priority for information systems requirements. The nature of the security awareness areas prioritization problem described in this paper fits the framework of ordinal ranking problems and is suitable for the method developed by Cook and Seiford. In general, the formulation can be described as follows. Consider n individuals and m objects (security awareness areas). Let r ij be the rank of the ith individual on the jth object (i ¼ 1,., n and j ¼ 1,., m). If c j is the consensus rank for the jth object, then the ith individual s absolute distance (disagreement) from the consensus ranking is represented by d i ¼ Xm rij c j ði ¼ 1;.; nþ: X n i¼1 The total distance of all individuals can then be expressed by d i ¼ Xn i¼1 X m rij c j : If c j is set equal to an index number k (k ¼ 1,., m) the total distance can be rewritten as X m d jk ; where d jk ¼ Xn rij k : i¼1 This represents the sum of distances between a consensus rank k and all n individuals rank on the jth object. The best consensus ranking then becomes the one for which the total distance is a minimum. The problem can now be represented by the following assignment problem.
3 256 computers & security 27 (2008) Min P m P m k¼1 d jkx jk Subject to P m x jk ¼ 1 P m k¼1 x jk ¼ 1 x jk 0; with 1 if cj ¼ k; x jk ¼ 0 otherwise: ðk ¼ 1;.; mþ; ðj ¼ 1;.; mþ; In the following section, the two approaches are applied in a real-life case study to determine the best consensus ranking of selected information security awareness areas and awareness material used in an awareness program. 3. Case study 3.1. Background The assignment problem is capable of handling large problems and can readily be solved by most linear programming software. Solution procedures and the structure of an assignment problem are discussed extensively in the literature and details can be found in Taylor (2002), for example Heuristic approach A simple procedure, called the maximize agreement heuristic (MAH), which can be used to arrive at a consensus ranking and that maximizes agreement among decision-makers, was developed by Beck and Lin (1983). Examples of how this heuristic was implemented in other studies can be found in Tavana et al. (1996), Tavana (2003) and Kengpol and Tuominen (2006). The heuristic was also used in this paper for comparative reasons and the purpose of this section is to introduce briefly the mechanics of the MAH. The MAH requires the construction of an agreement matrix A, where each element a ij represents the number of decisionmakers who ranked object i higher than object j. Positive and negative preference vectors, P and N, are then calculated using P i ¼ Xn and N i ¼ Xn a ij a ji ði ¼ 1;.; nþ; ði ¼ 1;.; nþ: Each P i is a row total that represents the total agreement for each object i, i.e. the total number of times object i is preferred over all other objects. Similarly, N i is a column total representing the total disagreement for each object j, i.e. the total number of times object j is not preferred when compared with all other objects. If any entry in the P vector or N vector is zero, that object would be placed at the bottom or the top of the final consensus ranking, respectively. If no zero entries exist, the difference P i N i, for all i, is considered. The largest difference is evaluated and if it is positive, the object is placed at the top of the final consensus ranking. If it is negative, the object will be placed at the bottom. It is often easier to complete the consensus ranking from the most to the least important ranking, in which case the largest positive difference (instead of the absolute difference) is used to indicate the next ranking. The placed object is now deleted from the agreement matrix and a new agreement matrix is constructed. The process is then repeated until all objects have been placed in the final ranking. Ties are dealt with arbitrarily. One of the largest international gold mining companies agreed to assist with the project. The company is a global African gold producer with 25 operations in 11 countries and is listed on a number of stock exchanges such as the Johannesburg Securities Exchange, New York Stock Exchange, etc. Over 6 million ounces of gold are produced annually, and it has one of the world s largest reserves, resource bases and focused exploration activities around the globe. Operations include both deep and open pit mines and more than people are employed in countries such as South Africa, Namibia, Ghana, Mali, Argentina, Brazil, USA and Australia. Like any other organisation with ICT assets, senior management realized that a key defence against ICT security breaches would be to raise the general level of information security awareness and to educate all computer users in the basics of information security. The objective was to prevent, or at least reduce, human-related security incidents, for example, phishing. As a result, a comprehensive process was started to develop an ICT awareness program. During the last quarter of 2003 the roll-out of the programme commenced. One of the priorities was to narrow the focus of the program into a manageable size and at the same time ensure that all important areas are covered. After careful deliberation and following a risk elimination process, the program was focused on six areas, viz. always adhere to company policies, keep passwords and personal identification numbers (PINs) secret, use and the Internet with care, be careful when using mobile equipment, report incidences like viruses, theft and losses, and be aware that all actions have consequences. The program was rolled out to all computer users and awareness material was made available in English, Spanish, French and Portuguese. The six main awareness materials used included video presentations, personal presentations, a website on the company s intranet, brochures, posters in offices, and articles in the company s in-house magazine. Following the implementation of the program a twofold business need arose. Firstly, there was a need to evaluate the success and effectiveness of the program, and secondly, a need to confirm that the six areas and six awareness
4 computers & security 27 (2008) materials were the correct ones. The first concern was addressed through the development of a comprehensive tool to measure awareness levels of staff (Kruger and Kearney, 2006). The second issue was addressed through the use of consensus ranking techniques described in this paper and case study. The motivation for reviewing the focus areas and awareness materials to determine whether new ones should be added or existing ones excluded from the awareness program can be found in ordinary business principles that impact ICT awareness programs. Business goals, technology and work environments are subject to constant change to ensure that an ICT awareness program is properly aligned with changes and company objectives, periodic reviews of areas to be covered and material to be used should be conducted. Resources, such as money and effort, are necessary for any new or follow-up awareness campaigns. To ensure that they are effectively employed, it is important to know where to concentrate these resources. Once focus areas and material have been identified it is also necessary to determine the more important areas and material within the group of identified objects. It is very seldom that all identified aspects are of equal importance, and money and effort should not necessarily be evenly spread among identified focus areas and/or awareness material. Another issue concerning the priority rankings is the measuring of the effectiveness of the awareness program. For example, theft of mobile equipment should be a higher risk in South Africa than in Australia. Priority rankings would therefore enable the incorporation of importance weights in a measuring tool and ensure more accurate measurements of awareness levels. One way of addressing these issues is to present a list of possible focus areas and awareness materials to the right role players to rank them. The rankings should then be converted into a consensus ranking where the top x number of ranked objects are chosen for the program. The consensus ranking can also serve as an importance ranking from which importance weights can be derived Methodology A very simple questionnaire was designed to present the six focus areas and six awareness materials to selected senior managers in each region (country). Respondents were then asked to rank them in order of importance from 1 (most important) to 6 (least important). In addition, they were asked to add any new items if necessary, and to include these new items in their importance rankings. Questionnaires and communications were translated into Spanish, French and Portuguese where appropriate. A small number of senior decision-makers in each region were selected to participate. A personal from the Manager IT Risk and Compliance was sent to each of them, explaining the exercise and requesting them to complete and return the questionnaire. Twenty-two useable rankings were received, which represents a 63% response rate. The reason for the small number of participants was that only those senior managers in each region who had a direct influence on the company strategy and business goals were targeted Results As per agreement with the company, the actual ratings of decision-makers may not be revealed. None of the respondents have added any new items. This was seen as confirmation that the six focus areas and the six awareness materials used in the program were currently appropriate and relevant. Processing of the data was therefore focused on arriving at a consensus ranking to assist with providing importance rankings (weights) and thereby assisting with management information regarding the concentration of effort and money. Responses received were converted into two distance matrixes, one for the focus areas and one for the awareness material, according to the discussion in Section 2. The Solver function of Excel was then used to solve the final assignment problems. For purposes of comparison the maximize agreement heuristic was also applied to the responses. Table 1 presents the results for the six focus areas and Table 2 the results for the six awareness materials used in the awareness program. It can be seen from the two tables that there were no significant differences between the distance-based solution and the MAH. In both cases the top three rankings contain the same focus areas and awareness material although video presentations and posters exchanged first and second positions in Table 2. The middle column in each table indicates importance weights for each ranked object. The ranking orders were used to assign these importance weights to the areas and materials. The weights would be useful when measuring awareness levels, or they can be used to influence the allocation of resources. A very simple way of deriving the importance weights was used. The ranking orders were normalized to be between 0 and 1 and were then assigned in reverse order to the focus areas and the awareness materials. E.g. the focus area keep passwords secret (Table 1) has the highest weight of (6/(1 þ 2 þ 3 þ 4 þ 5 þ 6)) and actions carry consequences the lowest weight of This may then imply that 29% of the awareness budget should be spent on the focus area keep passwords secret, while only about 5% should go to the actions carry consequences area. Even though the distance-based approach may have more than one optimal solution in certain cases, it is also clear that Table 1 Consensus rankings for focus areas Rank Consensus ranking Importance Consensus ranking based on the assignment problem weight using MAH 1 Keep passwords and PINs secret 2 Adhere to company policies 3 Use and the Internet with care Keep passwords and PINs secret Adhere to company policies Use and the Internet with care 4 Report incidences Careful when using mobile equipment 5 Careful when using mobile equipment Report incidences 6 All actions have consequences All actions have consequences
5 258 computers & security 27 (2008) Table 2 Consensus rankings for awareness material Rank Consensus ranking based on the assignment problem Importance weight Consensus ranking using MAH 1 Video presentations Posters in offices 2 Posters in offices Video presentations 3 Personal presentations Personal presentations 4 Brochures Brochures 5 Website on company s intranet 6 Articles in in-house magazine Website on company s intranet Articles in in-house magazine the two techniques do not necessarily give the same consensus rankings. The fact that more than one optimal answer may be possible when using the distance-based approach was not seen as a problem in this case study it simply means that if for example there are two optimal answers (consensus ratings) that are both of equal importance an arbitrary choice between the two can be made to allocate, for example, resources. The case study suggested that the distance-based approach be considered for obtaining consensus rankings. Not only does it provide an optimal answer, but the application is extremely easy calculating the distance matrix is easy and straightforward, while any standard linear programming software (e.g. the solver function of Excel) can be used to solve the standard assignment problem. The MAH is also a useful tool with easy steps that can be successfully applied. However, in this study the application of the MAH was found to be somewhat tedious due to the iterative nature of calculations with different matrixes a problem that will be aggravated when dealing with many objects that need to be ranked. The development of an automated tool should be considered when using the MAH. An optimal answer is also not guaranteed by the MAH. One of the limitations of the distance-based approach, as used in this study, is the fact that all rankings should be complete (no ties), while the MAH is capable of handling incomplete rankings. Missing values in responses should also be discarded. However, in this case study, there were only a few objects (six in each case) to be ranked and none of the participating decision-makers had any problems in complying with the request of providing complete rankings with no missing values. In other cases or circumstances this might not be the case. The problem of consensus ranking discussed in this paper is not limited to only a few or a small number of objects to be ranked. Although psychological research indicates that the maximum number of objects for which most decision-makers can make meaningful judgements will vary from five to nine (Patton et al., 1983), a greater number of objects can easily be handled by grouping them into groups of, say, nine objects each (Shi et al., 1996) Management response to the consensus ranking model It was difficult to verify senior management s response to the consensus ranking model with each manager involved in the exercise as they were located in different parts of the world where they formed part of the senior strategic management team of the mining group. One of the senior managers, the Manager IT Risk & Compliance, who deals regularly with the respondents and who assisted in performing the case study, was used as a representative opinion from senior management. According to the Manager IT Risk & Compliance, one of the major advantages observed during the case study was the use of the simple and easy questionnaire that was distributed to respondents. It consisted only of a list of the six objects to be ranked in order of preference/importance no weights or any other information or evaluations were requested. This may sound insignificant but it is a known fact that people, especially senior decision-makers, do not normally like to complete questionnaires. In a previous project, which is related to this one, an attempt was made to measure the security awareness levels of staff based on the six focus areas (Kruger and Kearney, 2006). Importance ratings were determined by using the Analytic Hierarchy Process (AHP), where a decision-maker would give his/her preference by means of pairwise comparisons. Many practical problems arose from this, e.g. getting managers to participate as they did not understand the AHP completely; combining the different managers pairwise comparisons into one overall comparison; inconsistent pairwise comparisons; and a waste of time redoing the exercise every time comparisons were inconsistent. None of these problems existed during this case study, something that can mainly be attributed to the use of the simple questionnaire. Other positive feedback received can be summarised as follows: a short and easy process to obtain consensus from different role players; a process that is easily understood by all, which encouraged participation; no comebacks in the form of additional meetings or questionnaires to try and resolve deadlocks; and the provision of a formal and transparent framework to achieve consensus previously, attempts to achieve consensus were mostly during discussions, something that was often susceptible to problems caused by group dynamics. 4. Conclusion ICT security awareness programs have become one of the key defenses in the fight against security incidents involving the human factor, and sufficient material exists to assist organisations with delivering proper awareness programs. These programs are normally focused on specific areas of concern and may include a variety of awareness materials such as posters, presentations, brochures, etc. To obtain and justify resources needed for such an awareness program and to comply with the business principle of efficient and effective use of resources, while at the same time addressing business objectives, a consensus choice of focus areas, awareness material and importance rankings are required. To address this problem, this paper described a consensus ranking method based on the concept of minimizing the distance between individual rankings. The method was demonstrated in a mining environment with satisfactory results. First, the traditional questionnaire was replaced by a single form containing only the list of objects to be ranked.
6 computers & security 27 (2008) Senior decision-makers were then asked to rank the objects in order of preference/importance, and, finally, the consensus ranking was calculated by constructing a distance matrix and solving a standard assignment problem. For purposes of comparison, the rankings were also heuristically evaluated. The distance-based method was extremely easy to apply and promises to provide management with information that will assist in identifying important security awareness areas, the allocation of resources to these areas and the provision of more accurate measuring opportunities of security awareness levels. In general the study has shown that the use of a formal consensus ranking technique not only saves time and money, but may also provide a better understanding of the relevance and importance of those factors influencing an ICT security awareness program. Applying the techniques described in the paper assists in identifying and prioritizing improvement opportunities in an easy and transparent way and will enable decision-makers more accurately to address security awareness problems, such as phishing scams, through focused awareness programs. The techniques discussed are not limited only to situations described in the case study but could be used in any decision-making situation associated with ICT security. Acknowledgement The authors would like to thank the two anonymous referees for their constructive comments that helped improve the paper. The authors alone are responsible for any errors and omissions. Part of this research was supported by the National Research Foundation in South Africa. Grant reference FA references Beck MP, Lin BW. Some heuristics for the consensus ranking problem. Computers and Operations Research 1983;10(1):1 7. Cook WD. Distance-based and ad hoc consensus models in ordinal preference ranking. European Journal of Operation Research 2006;172: Cook WD, Seiford LM. Priority ranking and consensus formation. Management Science 1978;24(16): Finextra. UK phishing fraud losses double. Available from: <http://www.finextra.com/fullstory.asp?-id¼15013>; 2006 [accessed June 2006]. Hansche S. Designing a security awareness program: part 1. Information System Security January/February 2001: Kendall M. Rank correlation methods. 3rd ed.; New York. Kengpol A, Tuominen M. A framework for decision support systems: an application in the evaluation of information technology for logistics firms. International Journal of Production Economics 2006;101(1): Kerstein PL. How can we stop phishing and pharming scams?. Available from: <http://www.csoonline.com/talkback/ html>; 2005 [accessed June 2006]. Kruger HA, Kearney WD. A prototype for assessing information security awareness. Computers & Security 2006;25: Patton JM, Evans JH, Barry LL. A framework for evaluating internal audit risk. Research report number 25. Altamonte Springs, FL: The Institute of Internal Auditors, Inc.; Pipkin DL. Information security. Protecting the global enterprise. Upper Saddle River, NJ: Prentice Hall; SANS 27001:2006. South African National Standard. Information technology security techniques information security management systems requirements. SANS 27001:2006, the identical implementation of ISO/IEC 27001: st ed. Pretoria: Standards South Africa (a Division of SABS); Shi Y, Specht P, Stolen J, Vanwetering F. A consensus ranking for information system requirements. Information Management & Computer Security 1996;4(1):10 8. Tavana M. CROSS: a multicriteria group-decision-making model for evaluating and prioritizing advanced-technology projects at NASA. Interfaces 2003;33(3): Tavana M, Kennedy DT, Joglekar P. A group decision support framework for consensus ranking of technical manager candidates. Omega 1996;24(5): Taylor BW. Introduction to management science. 7th ed. Prentice Hall; Thomson K, Von Solms R, Louw L. Cultivating an organisational information security culture. Computer Fraud & Security October 2006;2006(10):1 11. H.A. Kruger is an Associate Professor in the School of Computer, Statistical and Mathematical Sciences at the North-West University (Potchefstroom Campus) in South Africa. He previously worked for a large international mining company and has a number of years experience in Information Risk Management. He has a PhD in Computer Science, a MCom (Information Systems) and an MSc (Mathematical Statistics). His current interests include decision modeling and the use of linear programming models. W.D. Kearny currently works as a Manager, Risk and Assurance. He has over 20 years experience in Information Risk Management in a number of positions in large international companies. He has an MSc degree, numerous diplomas and earned a number of certifications, including CISA and CIA.