Study of Security Awareness Training Steve Kruse,Security Principal @ RSA Bill Pankey, Consultant @Tunitas Group Paradox Explanations Conclusions Innovation Norway Feb 04, 2010 CSI / FBI Computer Crime Survey http://www.gocsi.com Annual study of infosec events & security practice in US Estimates losses from computer crime Type and frequency of exploits Survey of security practice / response to threats Commonly used baseline for Risk assessments Security budget justification 1
2007 CSI Finding: Minimal UAT* Budgets http://i.cmpnet.com/v2.gocsi.com/pdf/csisurvey2007.pdf % of security budget spent on UAT 22% 48% 6% 2007 Security budgets average between 3-5% of total IT budget * UAT: Security Awareness Training Counter Indications Convention wisdom among US security practitioners is that end users are the greatest security threat: 85% of security breaches involve end users ~ Forester End user web surfing primary source of network infection ~ Symantec Email borne compromise of end user accounts has been has been the initial entry vector for 100% of advanced persistent threats (APT) ~ Mandiant. Any basic handbook of computer security 2
The Paradox You can t service pack the end user - Mike Nash, Microsoft Is there an inverse Pareto principle i at work: spend the least on the biggest problem? High recognition of end user involvement with security breaches Relatively low spending on user security awareness training Is the security industry guilty of risk management malpractice? Study resolved to understand / explain this apparent paradox 2009 Survey of Security Awareness & Training http://tinyurl.com/uatsurvey 57 questions UAT budgeting & rationale UAT metrics & accountability UAT practices Respondents recruited from: UseNet / yahoo UAT interest groups Metricon audience Disclaimer: No sampling control Suspect bias in self-report survey results: More, rather than less interest in UAT than among typical security practitioners Natural tendency to over-estimate the quality of their security program 3
Sample validates CSI Finding Survey sample places only slightly greater emphasis on UAT than CSI study 55% less than 2% of security budget 13% greater than 8% of security budget Difference are either not significant or indicate a very slight increased emphasis on UAT Security UAT budget in context True cost of UAT typically masked: Security budgets typically are not burdened with the cost of the users time spent on UAT Only 1 respondent reported that UAT training time is charged against security budget Total cost of user training time is likely to be the most costly component of UAT 93% of respondent companies require UAT for all employees at hire; 75% annual refresher training 53% of respondents require at least 1 hr per employee 4
Security UAT budget in context (2) Security program s UAT expenditure is either very efficient or very wasteful of company (human) resources UAT budget includes preparation, delivery and management costs for a significant expenditure of corporate resources Normalized UAT management cost metric: security program UAT budget / total # of user training hours * loaded labor rate Measure of efficiency? or inattention? Questions about the rationale for UAT? Is there formal documentation ti of anticipated i t UAT benefits? If so, where? Security plan: 65% Security policy: 42% (pro-forma?) UAT business case: 35% Individual campaign proposal: 5% But, often there is no accountability for benefit statements: 60% of respondents report no management review or approval of the above 5
UAT Rationale What is used to justify the commitment t of users time to UAT Regulatory requirement: 73% Unspecified security benefit : 65% External Auditor report: 45% Expected increase in user accountability: 15% Basis for rejection of requests for increase in mandated UAT (100 % of respondents)? Unspecified management priorities: 70% Weak business case: 16% Lack of UAT Effectiveness Performance Measurement Few companies tracked meaningful measures of UAT performance Training completion / compliance rate: 100% (User) Behavioral \ attitude measures: 13% Correlation w/ security incident metrics: 7% Management satisfaction determined by: CSO \ CISO: 67% CIO: 25% Compliance: 13% HR: 13% 6
Illusory UAT Management you can t manage what you don t measure Drucker 60% claim success of UAT to Reduce security incidents Address root causes of security breaches Increase compliance with security policy While focusing on cost, Rate of training completion But Avoiding collection of UAT performance / effectiveness metrics Few Meaningful UAT Metrics Focus on activity rather than benefit realization. Security Metrics, Jacquith % of staff completing security awareness training? (6) Correlation of tailgating rates w/ training latency Metrics for IT Service Management, ITSM % of staff not at optimal training level? (10) Poll of callers to service desk SP800-55 Performance Measurement Guide for Information Security, NIST % of employees in security roles receiving specialized security training (1) 7
Do UAT metrics obscure the security objective? Common UAT metrics available to establish industry baselines may miss the point Implicitly assume the effectiveness of training, ie, the results of management W/o the relevance, credibility & appropriateness of the training, the completion rate indicates nothing about the security value of the UAT program. Currently these measures are primarily cost metrics reflecting the scale of resource (end user time) consumption Meaningful UAT Metrics imply Strategy Security role of the end user is specific to company and its security strategy. Is the user, a: Threat to be engineered around? An actor whose behavior needs the constraint of policy? A source of detective or corrective control? All of the above? Some of the above? The view of user determines the objectives of UAT (improve user performance in security role) What and how much risk awareness Will vary with industry and company culture These UAT objectives will determine metrics Missing industry benchmarks 8
Standardization of Security Awareness Effectiveness of UAT has to be measured against expectations of security strategy Some user roles are expected to have more than less general security / risk awareness Some use roles expected to take specific action in response to events Standardization of user expectation facilitates development of appropriate metrics and establishing meaningful industry baselines Maturity Model for User Security Awareness Blissfully unaware Little recognition or acceptance of most information security threats At this level, l prevalent view is that t information security is a property of IT systems and largely a matter of architecture and configuration. Security largely independnet of user behavior. Consciously incompetent Some recognition that there is a information security threat, but: Poor risk assessment skill and intuition Uncertain of action needed to protect company information assets. Will do nothing rather than create further harm Compliant Aware of risks identified in company policy Will take action identified in company security policy Risk aware Considers information security risk in performance of company duties, but Unsure of appropriate action; sometime will report incidents Competent & Practiced Takes appropriate action within scope of role; otherwise reports incidents 9
UAT Goals re Targeted Maturity Level Blissfully unaware User: heads down routine data processing roles No UAT? Complaint User: Industry w/ little discretionary access control (e.g. banking). Users locked down by restrictive policy. UAT: Policy training Metric: Correlate policy violations with training latency Risk aware User: Industry w/ significant discretionary access (e.g. health) UAT: Policy training + Threat identification Metric: # of anomalies reported by users Example Scenario: End user sees what could be a company owned laptop in an unlocked car in the facility parking lot. What is the end user expected to do? Blissfully unaware: Where the company security tolerates the unaware user, e.g. where whole disk encryption has been implemented for all company laptops p ~ nothing Compliant: Where company policy prescribes all security obligations ~ only what is described in policy Risk aware: Where company security model depends upon actions of end users ~ alert company facility manager; security officer 10
Conclusions Little accountability for UAT beyond compliance with regulatory mandates Few, if any performance metrics Focus on cost Where there is no accountability, the optimal strategy is to reduce the absolute UAT expenditure UAT management requires new UAT performance metrics Correlate UAT with specific security benefits Lack of industry UAT performance benchmarks Meaningful metrics determined by targeted maturity levels 11