Building a Modern Security Engineering Organization.

Transcription

1 Building a Modern Security Engineering Organization

2 Who is this guy anyway? Built and led the Etsy Security Team Spoiler alert: what this presentation is about Recently co-founded Signal Sciences to productize effective AppSec approaches

3 This talk is a collection of lessons learned from building and adapting a security team

4 For security teams, the world has changed in fundamental ways: Code deployment is now near-instantaneous

5 For security teams, the world has changed in fundamental ways: Code deployment is now near-instantaneous Merging of development and operations means more people with production access

6 For security teams, the world has changed in fundamental ways: Code deployment is now near-instantaneous Merging of development and operations means more people with production access Cost of attack has significantly dropped

7 Near-instantaneous deployment?

8 A technical diagram of tradi7onal waterfall code deployment

9 What is this shifting to?

10 Etsy pushes to production 30 times a day on average

11 Constant iteration in production via feature flags, ramp ups, A/B testing

12 But doesn t the rapid rate of change mean things are less secure?!

13 Actually, the opposite is true

14 They key to realize is vulnerabilities occur in all development methodologies But there s no such thing as an out-ofband patch in continuous deployment

15 They key to realize is vulnerabilities occur in all development methodologies But there s no such thing as an out-ofband patch in continuous deployment

16 Compared to: We ll rush that security fix. It will go out in about 6 weeks. - Former vendor at Etsy

17 What makes continuous deployment safe?

18

19 Source: h<p:// topics- in- con7nuous- deployment

20 The same culture of graphing and monitoring inherent to continuous deployment can be used for security too

21 Surface security info for everyone, not just the security team

22

23 Don t treat security as a binary event

24 *Mullets sold separately Building a (k- )rad culture

25 In the shift to continuous deployment, speed increases by removing organizational blockers

26 Trying to make security a blocker means you get routed around

27 Instead, the focus becomes on incentivizing teams to reach out to security

28 Keys to incentivizing conversation: Don t be a jerk. This should be obvious, but empathy needs to be explicitly set as a core part of your teams culture.

29 Keys to incentivizing conversation: Don t be a jerk. This should be obvious, but empathy needs to be explicitly set as a core part of your teams culture. Make realistic tradeoffs. Don t fall in to the trap of thinking every issue is critical. Ex: Letting low risk issues ship with a reasonable remediation window buys you credibility for when things actually do need to be addressed immediately.

30 Keys to incentivizing conversation: Coherently explain impact. This would allow all our user data to be compromised if the attacker did X & Y paints a clear picture, where The input validation in this function is weak does not.

31 Keys to incentivizing conversation: Coherently explain impact. This would allow all our user data to be compromised if the attacker did X & Y paints a clear picture, where The input validation in this function is weak does not. Reward communication with security team. T-Shirts, gift cards, and high fives all work (shockingly) well.

32 Keys to incentivizing conversation: Take the false positive hit yourself. Don t send unverified issues to dev and ops teams. When issues come in, have the secteam verify and make first attempt at patch. Scale via team leads. Build relationships with technical leads from other teams so they make security part of their teams culture.

33 Keys to incentivizing conversation: Take the false positive hit yourself. Don t send unverified issues to dev and ops teams. When issues come in, have the secteam verify and make first attempt at patch. Scale via team leads. Build relationships with technical leads from other teams so they make security part of their teams culture.

34 Access restric7ons

35 Startups begin with a simple access control policy: Everyone can access everything

36 As organization grow there will be more pressure to institute access policies

37 The key to remember is don t take away capabilities

38 Methodology: 1. Figure out what capability is needed 2. Build an alternate way to perform the needed function in a safe way 3. Transition the organization over to the safe way 4. Alert on any usage of the old unsafe way

39 Methodology: 1. Figure out what capability is needed 2. Build an alternate way to perform the needed function in a safe way 3. Transition the organization over to the safe way 4. Alert on any usage of the old unsafe way

40 Methodology: 1. Figure out what capability is needed 2. Build an alternate way to perform the needed function in a safe way 3. Transition the organization over to the safe way 4. Alert on any usage of the old unsafe way

41 Methodology: 1. Figure out what capability is needed 2. Build an alternate way to perform the needed function in a safe way 3. Transition the organization over to the safe way 4. Alert on any usage of the old unsafe way

42 EX: SSH access to production systems

43 Security policy goal: Eliminate unneeded access to production systems Why do developers do it? Ex: To view error logs Build alternate approach: Send the logs to central logging service (ex: elasticsearch, splunk, etc) Publicize the new tooling to the organization After majority of transition, alert on any logins to production systems by non-sysops

44 Security policy goal: Eliminate unneeded access to production systems Why do developers do it? Ex: To view error logs Build alternate approach: Send the logs to central logging service (ex: elasticsearch, splunk, etc) Publicize the new tooling to the organization After majority of transition, alert on any logins to production systems by non-sysops

45 Security policy goal: Eliminate unneeded access to production systems Why do developers do it? Ex: To view error logs Build alternate approach: Send the logs to central logging service (ex: elasticsearch, splunk, etc) Publicize the new tooling to the organization After majority of transition, alert on any logins to production systems by non-sysops

46 Security policy goal: Eliminate unneeded access to production systems Why do developers do it? Ex: To view error logs Build alternate approach: Send the logs to central logging service (ex: elasticsearch, splunk, etc) Publicize the new tooling to the organization After majority of transition, alert on any logins to production systems by non-sysops

47 Increasing a<acker cost

48 Specifically, some thoughts on: Bug Bounties Attack simulations/pentesting

49 Bug Boun7es

50 Bug bounties are tremendously useful. If you re not working towards launching one, strongly consider it.

51 Common concerns about launching a bounty: 1. Budgetary concerns. Money is almost never the main motivation for researchers, you can launch a bounty with just a hall of fame and still get great submissions. 2. Risk of inviting attacks. You re already getting attacked continuously, you re just not getting the results.

52 Common concerns about launching a bounty: 1. Budgetary concerns. Money is rarely the main motivation for participants, you can launch a bounty with just a hall of fame and still get great submissions. 2. Risk of inviting attacks. You re already getting attacked continuously, you re just not getting the results.

53 Common concerns about launching a bounty: 1. Budgetary concerns. Money is rarely the main motivation for participants, you can launch a bounty with just a hall of fame and still get great submissions. 2. Risk of inviting attacks. It s the Internet. You re already getting pentested continuously, you re just not receiving the report.

54 The ultimate goals of a bug bounty are threefold: 1. Incentivize people to report issues to you in the first place 2. Drive up cost of vulnerability discovery and exploitation for attackers 3. Provide an external validation of if your security program is working (or not)

55 The ultimate goals of a bug bounty are threefold: 1. Incentivize people to report issues to you in the first place 2. Drive up cost of vulnerability discovery and exploitation for attackers 3. Provide an external validation of if your security program is working (or not)

56 The ultimate goals of a bug bounty are threefold: 1. Incentivize people to report issues to you in the first place 2. Drive up cost of vulnerability discovery and exploitation for attackers 3. Provide an external validation of where your security program is working (and where it s not)

57 Before you launch, record what vulnerability classes you expect to see and what you don t. Compare this against the issues actually reported.

58 Before you launch, record what vulnerability classes you expect to see and what you don t. Compare this against the issues actually reported.

59 Keep metrics on: Number of bugs reported and severities Time to remediation of reported issues You want both of these metrics to trend down over time

60 Practical considerations: Inform all teams before bounty launch, especially non-engineering teams Ex: Customer Support Attacks will start almost immediately For Etsy bug bounty launch, time from announcement to first attack: 13min

61 Practical considerations: Inform all teams before bounty launch, especially non-engineering teams Ex: Customer Support Attacks will start almost immediately For Etsy bug bounty launch, time from announcement to first attack: 13min

62 Practical considerations: Your first 2-3 weeks will be intense. Have as many people as you can dedicated to triage and response

63 Practical considerations: Operationally review any helper systems for scaling problems beforehand When x traffic hits helper systems your security team uses, what falls over? Money almost never the overriding factor, hall of fame is Researchers are generally great to interact with

64 Practical considerations: Operationally review any helper systems for scaling problems beforehand When x traffic hits helper systems your security team uses, what falls over? Money is almost never the main motivation for bounty participants, hall of fame credit is Researchers are generally great to interact with

65 Practical considerations: Operationally review any helper systems for scaling problems beforehand. When x traffic hits helper systems your security team uses, what falls over? Money is almost never the main motivation for bounty participants, hall of fame credit is Key to great researcher interaction is frequent and transparent communication

66 XXX Running effective attack simulations

67 Problems with pentesting are well understood in the offensive community but not as well in the defensive community

68 Pentests typically result in a list of enumerated known vulnerabilities to be patched, not data on how a real attacker would operate against a given environment

69 Attack simulations should be done to learn how attackers are likely to achieve goals against your organization NOT to show compromise is possible (spoiler alert: it is.)

70 Use this attack data to focus where/how to build detection mechanisms

71 From an organizational side, attack simulations compliment vulnerability enumeration/compliance/etc

72 Four keys to effective attack simulations: 1. Goal oriented Obtain domain admin, read the CEOs , view credit card data, Ask attack team for input on goals, they ll come up with ones you didn t think of 2. Full ganization in scope Have attack team call a contact if they re about to do something risky several week simulat ion

73 Four keys to effective attack simulations: 1. Goal oriented Obtain domain admin, read the CEOs , view credit card data, Ask attack team for input on goals, they ll come up with ones you didn t think of 2. Full organization in scope Have attack team call a contact if they re about to do something risky Ex: Instead of throwing an exploit that lands most of the time, grant access to the target system with temporary credentials

74 Four keys to effective attack simulations: 3. Simulate realistic compromise patterns Start the attack team on a: standard laptop/desktop to simulate phishing/clientside compromise database or web server to simulate SQL injection/rce 0days aren t cheating, they re reality. Attack team should be encouraged to use them. Break simulation down into iterations: Don t spend the full engagement time on only round of testing, once one team achieve goal(s), then swap in new attack team to achieve the same goal(s) Ex: We try to run 3-4 iterations per several week simulation

75 Four keys to effective attack simulations: 3. Simulate realistic compromise patterns Start the attack team on a: standard laptop/desktop to simulate phishing/clientside compromise database or web server to simulate SQL injection/rce 0days aren t cheating, they re reality. Attack team should be encouraged to use them. 4. Break simulation down into iterations: Don t spend the full engagement time on only round of testing, once one team achieve goal(s), then swap in new attack team to achieve the same goal(s) Ex: We try to run 3-4 iterations per several week simulation

76 The project output should be attack chains showing how attack team went from A->B->C to achieve goals, what steps they took and why

77 Just as importantly, what steps they didn t take Ex: We didn t try to find internal network diagrams on your wiki because zone transfers were enabled so we could got enough data about your network from that

78 Remember, the goal is to simulate realistic attack behaviors and patterns that can be used to enhance detection

79 In addition, simulate varying attack profiles from quick & loud to quietly maintaining persistence

80 Over multiple iterations learn what behaviors overlap between attackers and what strong signals of lateral movement in your environment look like

81 TL;DR (The section formerly known as Conclusions )

82 Adapt security team culture to DevOps and continuous deployment by: Surfacing security monitoring and metrics Incentivize discussions with the security team When creating policy, don t take away capabilities Drive up attacker cost through bug bounty programs, countering phishing, and running realistic attack simulations

83