After the Attack: RSA's Security Operations Transformed

After the Attack: RSA's Security Operations Transformed Ben Smith, CISSP RSA Field CTO (East), Security Portfolio Senior Member, ISSA Northern Virginia 1

The Environment ~ 2,000 security devices ~55M security events per hour ~60K employees 350 sites 85 countries Core intellectual property 2

Structure of Pre-Breach CIRC Eyes on Glass Analysis Forensic Coordination Remediation Rule/Report Creation Workflow Development RSA CIRC 2009-2011 L1 L2 L3 3

Technology Pre-Breach SharePoint Incident Tracking Security Operations (RSA Archer) Vulnerability Risk Management Security Controls Firewall IPS Proxy Windows AV Clients/Server s File Servers RSA SIEM Databases Data Discovery (RSA DLP) Log Analysis Reporting Event Forensics NAS/SAN Endpoints Limited Real-Time Response Limited Visibility 4

The Initial Vector in the Attack 1 Two rounds of phishing emails Some clues about the email lead us to believe that this was from some slightly dated research on employees 2 Launch Zero-day attack One user opened email attachment (an Excel spreadsheet) which launches a Flash zero-day X 3 Attacker gains access to other machines Zero-day exploit installs backdoor (Poison Ivy RAT Variant) which enables extraction of memory resident password hashes X X X X 5

From Compromise to Exfiltration 4 Attacker initiates separate network connection using credentials obtained in the earlier phase of the attack 5 Attacker moves laterally through organization, relying heavily on escalation of privileges, to systems containing disparate information that when combined, allow compromise of targeted information ATTACKER 6 Attacker removes data and stages it on a file share within the network External Server 7 Files are encrypted and attacker tries to exfiltrate from several servers before finding a successful exit path from within the organization 6

Additional Attack Details Focused and coherent attacks Times of attacks may be carefully choreographed Attackers move rapidly to the target: they know what they want, and the order in which to get it Months or years of reconnaissance and preparation allow the attacker to move with exacting precision With prior experience, preparation and tools, attackers exploit people and processes more than weaknesses in infrastructure Remote (attacker) hosts may be modified to match internal naming structure Attackers may exhibit very detailed knowledge of the people, processes and infrastructure People continue to be the easiest target in any organization Fresh malware may be used Compiled just hours before the initial attack event Specifically crafted malware yields no known signature to block 7

The Business Impact ~$70 Million Write-Down Cost of remediation, IT, investigation, consulting, lost revenue, lost productivity Six months focused on remediating authenticators All Authentication-related marketing & sales activity stopped for six months Impact to trust Customers: How could you do this to us? Why didn t you contact us first? Lost some customers permanently All despite no risk to customers 8

The Organizational Impact Nobody fired! Changed product/services portfolio Spent more on investigative tools vs. checkboxes Gained authority - now more than just a token seller Recognized for honesty by announcing breach 9

Biggest Impact: CIRC Reorganization 2009-2011 CIRC Today Cyber Threat Intelligence L1 L2 L3 CIRT Advanced Tool & Tactics - Eyes on Glass - Analysis - Forensic - Coordination - Remediation - Rule/Report Creation - Workflow Development Content Analytics - Specific functions - Reduces scope creep - Focused workflow 10

Critical Results The Ability to Measure Event time to Assignment to Escalation to Resolution Average time to closure Scan every image file for.exe content 55M events per hour become 2-3K incidents per month 90 incidents for 94 person hours per day Time to closure is now about 1 day Program can scale! 11

People and Process Define dwell time From point of trigger to eyes on (analyst assigned) Separation of duties Analytics (CIRT Tier 1) Advanced Tools & Tactics Threat intelligence Integrated investigation Visible to analyst 12

Lessons learned/implemented by RSA Post-breach analysis indicated that threat intelligence would have played a major role in detecting this activity earlier on Actions: Built a dedicated Cyber Intelligence group Bought multiple commercial intelligence feeds Joined multiple threat sharing groups Custom developed a threat intelligence portal / database Developed in-house OSINT gathering program But at that time, we found 13

Observed Threat Intel Issues Some threat intel vendors don t understand the difference between Intelligence and Information A bad IP with no context is not actionable! Impact: Resources wasted on searching Why am I searching proxy logs for the IP address of a mail server that was used in a phishing campaign? 14

Observed Threat Intel Issues Lack of widely adopted standard for sharing threat intel or IoCs Many IoCs are still shared in unmapped formats CSV, text files, HTML posts, vendor-specific XML Impact: Resources wasted on logging into various portals, mailing lists, feeds and then normalizing the IoCs Impact: Human errors when transferring data 15

Observed Threat Intel Issues Limited platforms/applications to house threat intel Avalanche, MITRE CRITs, ThreatConnect Sharing, reviewing/approving, retiring Have you ever retired an IoC? How big are your block lists? Impact: IoC lifecycle management difficulty Impact: Increased burden on security controls 16

Observed Threat Intel Issues Quality of product from vendors varies Some do a good job of vetting indicators However, we still see 8.8.8.8 listed as bad Impact: Potential for operational issues Impact: Developed custom tools to vet IoCs 17

Observed Threat Intel Issues Justifying the expense to management Lack of obvious wins Early failures due to poor third-party intelligence Still not finding all the bad stuff A lot of custom development 18

Lessons learned/implemented by RSA Reviewed Threat Intel sources Removed those that fail to provide context Took a hard look at those who don t provide structured IoC delivery, regardless of context Understood each vendor s focus area Do you need Cybercrime Intel or just APT? Migrated from custom Portal to CRITs Still required substantial code changes to support EMC workflow Developed capability to integrate with multiple sharing standards 19

What is next? Efficiency Tracking incident false positive rate based on threat intelligence source Assign confidence values to sources Feedback to source vendor Correlating alerts across multiple data sources to add contextual elements to Incident record When alert from DNS fires, check proxy / firewall logs for contextual data and add to Incident 20

What is next? Harvesting IoCs Malware Intelligence Program Leverages Yara, VirusTotal, Cuckoo, Internal DB Search for new samples of specific Threat Actor tools each night and programmatically extracts IoCs Passive DNS Internally generated and commercial Used to pivot on known IoCs to find more 21

Lessons learned/implemented by RSA Organizational maturity is required! Threat intel isn t the silver bullet Need to manage expectations Expensive Both in $$$ and human capital Requires constant care and feeding New vendor offerings, quality of data Doesn t always produce tangible results No hits today. Intel failure or nothing going on? 22

Lessons learned/implemented by RSA If you are shopping for external threat intelligence, understand: Threat Intel quality varies widely Get some samples before signing the contract Ask your peers Threat Intel requires manual data entry Amount is proportional to the number of sources This is improving, more support for standards [STIX, TAXII] Threat Intel will likely require custom coding Portal/database, workflow integration, federation/sharing [CRITs] 23

Final Thoughts & Recommendations 24

Restoring Trust After a Data Breach Stabilize the patient Know what you have and prioritize by risk and value Harvest system state information from your production systems Compare what you have to what you deployed Remove suspect systems from the environment and return to a trustworthy state Continuously monitor and validate to prevent re-compromise Communicate in a way that builds trust and confidence Dwayne Melançon (2014), Restoring Trust After a Data Breach [RSA Conference APJ 2014] http://www.rsaconference.com/writable/presentations/file_upload/cds-w08-restoring-trust-after-a-data-breach.pdf 25

The Single Most Effective Security Control The most valuable assets to any company are informed, aware, and vigilant employees A well-defined security policy will take the guesswork out of what is appropriate? employee behavior If I could have chosen anything, technology or otherwise, that would have prevented or lessened the attack against RSA it would have been a more aware employee base. James Lugabihl Critical Incident Response Center (CIRC) Manager, EMC Global Security Office 26

Questions? http://bensmith.se/twitter http://bensmith.se/linkedin 27