Passwords are for Chumps

Who Am I?! David Veuve Sales Engineer for Major Accounts in Northern California! dveuve@splunk.com! Former Splunk Customer (For 3 years, 3.x through 4.3)! Security Guy! Primary Author of Splunk Search Usage app! David on Splunk Answers 2

! Why Single Sign On (SSO)?! SeUng up SSO on Windows! SeUng up SSO on Linux! SeUng up SSO via SAMLv2! A liwle something extra! Wrap up Agenda! All config files (where possible for Windows) will be posted to GitHub at the end of the presenta[on 3

Disclaimer During the course of this presenta[on, we may make forward- looking statements regarding future events or the expected performance of the company. We cau[on you that such statements reflect our current expecta[ons and es[mates based on factors currently known to us and that actual events or results could differ materially. For important factors that may cause actual results to differ from those contained in our forward- looking statements, please review our filings with the SEC. The forward- looking statements made in the this presenta[on are being made as of the [me and date of its live presenta[on. If reviewed a`er its live presenta[on, this presenta[on may not contain current or accurate informa[on. We do not assume any obliga[on to update any forward- looking statements we may make. In addi[on, any informa[on about our roadmap outlines our general product direc[on and is subject to change at any [me without no[ce. It is for informa[onal purposes only, and shall not be incorporated into any contract or other commitment. Splunk undertakes no obliga[on either to develop the features or func[onality described or to include any such feature or func[onality in a future release. 4

What is Wrong with Passwords! Diminish adop[on! Dispropor[onately discourage the users you really want Execu[ves/Managers, Business Users! Fundamentally insecure 5

Detail: Passwords are Fundamentally Insecure! People write them on post- it notes! People create simple ones! People type them into phishing websites! People reuse them across many websites hwp://xkcd.com/792/ hwp://xkcd.com/936/ 6

! Easier adop[on! More secure Benefits of Single Sign On! Facilitates High Availability Search Head Pooling works bewer with SSO enabled ê Allows you to fail over without a user no[cing 7

Limita[ons of Splunk SSO! Single Sign On depends on an external proxy that will handle the authen[ca[on piece, and then pass the username in an HTTP header to Splunk! Even with Single Sign On handling authen[ca[on, we s[ll need an LDAP connec[on to assign users to individual roles. This is not typically an issue for internal deployments, but is a greater issue for SAML deployments Can cover standard roles [To be filled in] 8

Single Sign On - Defini[on! Single sign- on (SSO) is mechanism whereby a single ac6on of user authen6ca6on and authoriza6on can permit a user to access all computers and systems where he has access permission, without the need to enter mul6ple passwords hwp://www.opengroup.org/security/sso/! In prac[ce: Users are automa[cally logged in without typing in a password 9

Common Single Sign On Methods! Ac[ve Directory AD has supported SSO via NTLM and others for years! Kerberos Core to Ac[ve Directory and widely used in Linux / OSX! SAML Commonly used for online systems! Smart Card (or One Time Password) Can be implemented by one of the above, or a hook into Ac[ve Directory to intercept and service authen[ca[on accounts! Several others employing similar core theories 10

How to Decide Which Method! Windows Server Environment: Windows Authen[ca[on Easiest setup in my experience! Linux Server Environment: Kerberos S[ll easy! Splunk hosted via external cloud (or with 3 rd party SSO such as Okta, PingIden[ty, etc.): SAML Most Challenging approach! 3 rd Party Proxy / Load Balancer Likely Kerberos, but depends on product 11

Splunk Setup

1. Set up LDAP Authen[ca[on 2. Map LDAP Groups 3. Update server.conf 4. Update web.conf Splunk Setup Steps 13

LDAP Configura[on! Frequently done by Splunk Users hwp://docs.splunk.com/documenta[on/splunk/6.1.3/security/configureldapwithsplunkweb! From Splunk Web, Access Controls 14

server.conf and web.conf Setup! server.conf trustedip Indicates that the local splunkd will trust the user coming from splunkweb ê (Remember that indexers implicitly trust the search head, so this only happens on the search head)! web.conf trustedip Indicates that splunkweb will trust the user coming from your upstream proxy/other device SSOMode Indicates whether local logons are allowed remoteuser Indicates what header parameter the user string will be put into 15

Security Quick Tip! Limit the number of trusted IPs you have configured on splunkweb, as they will be able to masquerade as any user! If you have tools.proxy.on = true, and see your worksta[on s IP address in /debug/sso, turn off tools.proxy.on and don t add every worksta[on to the trustedip list 16

Demo Splunk Setup 17

Demo - Splunk LDAP Setup 18

Demo server.conf! server.conf Refers to the local splunkd Remember that splunkweb running on the same box will communicate with splunkd via 127.0.0.1 19

Demo web.conf! web.conf Refers to the local splunkweb SSOMode ê Permissive allows either SSO or direct access to splunkd ê Strict SSO only (cannot log in with local auth seungs if locked out, must modify via conf files) trustedip ê IP of Proxy remoteuser ê Parameter containing username tools.proxy.on ê Required for old versions of Apache. This is turned on in a bunch of examples, but for none of the systems I ve used has it actually been necessary 20

Windows Op[on

Core Technologies at Play! Func[oning Splunk Install! Ac[ve Directory Infrastructure! IIS Web Server (2012 R2 in my test, but known to work at least through 2008) Plarorm addons: ê ARR hwp://www.iis.net/downloads/microso`/applica[on- request- rou[ng ê ISAPI Module ê ISAPI Filters Module Free Third Party ê ISAPI_Rewrite3 hwp://www.isapirewrite.com/ Allows you to add authen[cated user name to header 22

High Level Process 1. Configure Authen[ca[on for IIS Site 2. Configure Reverse Proxy for IIS Site 3. Configure URL_Rewrite to empty Accept Encoding Workaround for UI quirk 4. Configure ISAPI_Rewrite3 to put REMOTE- USER header 23

Windows Authen[ca[on Diagram! Users will hit the IIS Server, which will authorize them via Integrated Windows Authen[ca[on! Requests will then be proxied to Splunk! Splunk will perform authoriza[on via LDAP Groups! Users will get a seamless authen[ca[on and authoriza[on experience, and be greeted by the Splunk page! 24

Challenges! By default, Splunk will use gzip encoding, but that doesn t work with IIS ARR rou[ng rules. As a result, we need to store the original Accept Encoding in a header, wipe it, and then replace it. That will be seen in the example! IIS does not support wri[ng the authen[cated user informa[on into a header. This is why we need the external ISAPI_Rewrite3 Lite module. Fortunately, we can use the free Lite module by offloading the rou[ng! (Neither of these issues exist on Linux, or should exist on 3 rd party proxies or load balancers) 25

Why Third Party (ISAPI_Rewrite3 Lite)! ISAPI_Rewrite3 by Helicon is a great way to port configura[ons over from Apache! In par[cular, it allows us to set a header a`er the authen[ca[on part completes, which is not possible out of the box with IIS! There are two versions of ISAPI_Rewrite3 free and commercial For this configura[on, we only need the free version. The commercial version adds addi[onal proxy capabili[es which are delivered by IIS ARR 26

Demo Enabling Authen[ca[on 27

Demo Enabling Reverse Proxy 28

Demo Configure URL Rewrite 29

Demo Workaround for URL Rewrite Quirk 30

Demo - Helicon 31

Demo Successful SSO Debug 32

Demo Successful Logon 33

Troubleshoo[ng! Wireshark Verify that communica[on to your search head has the proper field populated! Debug page hwp://youriisserver/debug/sso! IIS Detailed Debug Logs By default, IIS will only show you the major error code (e.g., 500). If you turn on detailed logs, it will also show the more detailed logs, e.g.: ê HTTP Error 500.52 - URL Rewrite Module Error. Outbound rewrite rules cannot be applied when the content of the HTTP response is encoded ("gzip") 34

Troubleshoo[ng with Wireshark! Capture relevant traffic (port 8000)! Then look for the actual headers being passed in the HTTP message 35

Troubleshoo[ng with Debug SSO! Great source for ensuring your seungs are correct! Look par[cularly for the SSO Mode, trustedips and the Remote user HTTP Header. This has to be the same as what is seen inwireshark! Hopefully your setup will lookjust like this 36

Troubleshoo[ng with IIS Logs! By default IIS logs aren t very helpful. While troubleshoo[ng, turn on detailed logs for your site! Just click on Error Logs, then Edit Feature SeUngs, then Detailed Logging 37

Linux Op[on

! Working Splunk Installa[on! Linux Kerberos! Apache Web Server mod_auth_kerb mod_proxy mod_rewrite Core Technologies! Ac[ve Directory (or other Kerberos Store) 39

High Level Process! Create AD Service Account! Create keytab! Configure Linux Host Kerberos! Configure Apache to use mod_auth_kerb! Configure Apache to revers proxy using mod_proxy! Configure Request Header to set Remote User 40

Linux Authen[ca[on Diagram! Users will hit the Apache Server, which will authorize them via Kerberos to AD! Requests will then be proxied to Splunk! Splunk will perform authoriza[on via LDAP Groups! Users will get a seamless authen[ca[on and authoriza[on experience, and be greeted by the Splunk page! 41

Challenges! Biggest challenge with this approach is that there are many different sets of instruc[ons on the internet. This approach, end to end, worked in my environment 42

Demo Create AD User! Nothing complex about the user account can be anything 43

Demo Create Keytab! Copy- paste from internet. Note that this will reset the password! ktpass - princ {PRINCIPAL NAME} - mapuser {username@fqdn} - crypto {YourChoice} - ptype KRB5_NT_PRINCIPAL - pass {LookAtMyLongPassword} - out {Path\to\keytab} 44

Demo - Configure Linux Host Kerberos! Change the realm to your local realm! Note that this should probably match your users desktop config i.e., if they log into mydomain.local and you re hos[ng this site on mydomain.com, you will need to configure IE/Firefox/Chrome to try a kerberos Auth 45

Demo - Configure Apache to use auth_kerb! Change the realm and AuthName to your local realm/domain FQDN! Configure the Krb5KeyTab to where you copied the file over from your domain controller! KrbMethodK5Passwd allows users without kerberos to authen[cate via password! Require valid- user tells Apache that authen[ca[on is required 46

Demo - Configure Apache to Reverse Proxy! This leverages and requires mod_proxy to work, but is a prewy straighrorward config beyond that! The last two lines are the heart of the config behind the scenes, take anything going to myserver/* and send a parallel request to hwp://127.0.0.1:8000/*! If moun[ng your web path at a different directory, consider the root_endpoint seung! hwp://www.davidveuve.com/tech/proxying- splunk- with- ssl/ 47

Demo Configure Remote User Header! Unlike with Windows, here we can leverage a simple config to insert the remote user into the REMOTE- USER header! In seung this up, I tried several awempts to get the remote_user properly inserted this is the one that finally worked 48

Demo PuUng it all together! All the configura[on for my environment lives in /etc/hwpd/ conf.d/splunksso.conf! The en[re configura[on is here 49

Troubleshoo[ng! Paralleling the Windows troubleshoo[ng, there are three great tools for troubleshoo[ng on Linux: Apache Logs (hey, it s super easy to Splunk those!) Debug SSO Splunk Endpoint tcpdump 50

Troubleshoo[ng with Apache Logs! Make sure your keytab is in the right path!! Make sure your web server name matches your principal name! 51

Troubleshoo[ng with Debug SSO! Great source for ensuring your seungs are correct! Look par[cularly for the SSO Mode, trustedips and the Remote user HTTP Header. This has to be the same as what is seen in tcpdump! Hopefully your setup will look just like this 52

Troubleshoo[ng with tcpdump! Great to verify that the reverse proxy actually works and that the seungs are correct! Look par[cularly for the the Remote user HTTP Header 53

SAML Op[on

Core Technologies! Working Splunk Installa[on! Linux Host (CentOS 6.0 for this demo) yum install xmlsec1 xmlsec1- openssl xmlsec1- openssl- devel openssl hwpd mod_ssl Install EPEL on your RHEL- type box to get the xmlsec1s Lasso! Apache Web Server mod_auth_mellon! SAMLv2 Iden[ty Provider Recommend that to get started, you leverage a known working partner such as Okta (used here) or PingIden[ty. Then adapt to your own SAMLv2 55

High Level Process! Install host dependencies! Set up Iden[ty Provider (e.g., Okta/PingIden[ty/etc.)! Set up mellon config! Set up mod_auth_mellon config! Based almost completely on Paul Stout s excellent guide: hwp://blogs.splunk.com/2013/10/09/splunksso- using- saml- through- okta/ 56

SAMLv2 Authen[ca[on Diagram! Users will hit the Okta Server, which will authorize them and then forward them (via POST) to the Splunk server, which does not have to be accessible to Okta (can be behind the VPN)! Requests will then be proxied to Splunk! Splunk will perform authoriza[on via LDAP Groups! Users will get a seamless authen[ca[on and authoriza[on experience, and be greeted by the Splunk page! 57

Challenges! The provided versions of mod_auth_mellon / lasso only work for hwpd 2.2. There will be a conflict if you try to install on 2.4, and when I tried a newer version of mod_auth_mellon (0.7.0 instead of 0.5.0) it never worked, and never errored out Recommend that you set up first on 2.2 (RHEL or equivalent 5.x or 6.x, verify with hwpd - v) as it s a known working version! SAMLv2 is a notoriously finicky setup with lots of moving parts. Recommend that you start with a known working combina[on (e.g., Okta has a no- limit free version for a single app), then make incremental changes to move to your own implementa[on 58

On Groups! The major downside to SAMLv2 in Splunk is that it will only handle authen[ca[on. You will s[ll need to set up groups to handle authoriza[on, which would require an LDAP connec[on 59

Demo Install Host Dependencies! wget hwp://dl.fedoraproject.org/pub/epel/6/x86_64/epel- release- 6-8.noarch.rpm! rpm - ivh epel- release- 6-8.noarch.rpm! yum install hwpd xmlsec1 xmlsec1- openssl xmlsec1- openssl- devel mod_ssl openssl! Disable or tune selinux (/etc/selinux/config)! Set your hostname to match your principal name (e.g., splunk.dvsplunk.com)! wget hwps://dev.entrouvert.org/redhat/6/rpms/x86_64/lasso- 2.3.6-1.el6.x86_64.rpm! wget hwps://dev.entrouvert.org/redhat/6/rpms/x86_64/ mod_auth_mellon- 0.5.0-1.el6.x86_64.rpm! rpm - ivh lasso- 2.3.6-1.el6.x86_64.rpm! rpm - ivh mod_auth_mellon- 0.5.0-1.el6.x86_64.rpm 60

Demo Set up Iden[ty Provider (IdP)! Very easy with Okta Add Applica[on Provide URL Provide Default Relay State and username 61

Demo Grab IdP Metadata! Also very easy with Okta: 62

Demo Set up Mellon Config! Paul Stout s previously- linked- to guide includes a handy script that will set up the suppor[ng mellon files for Splunk: 63

Demo Set up mod_auth_mellon! The Mellon config is prewy straighrorward, and very copy- pasteable! For an explana[on of the ProxyPass configura[on, please see the Linux Config sec[on 64

Troubleshoo[ng! The recommended troubleshoo[ng tools for this configura[on are iden[cal to those for normal Linux systems: Apache Logs (hey, it s super easy to Splunk those!) Debug SSO Splunk Endpoint tcpdump 65

Troubleshoo[ng with Apache Logs! Make sure your keytab is in the right path!! Make sure your web server name matches your principal name! 66

Troubleshoo[ng with Debug SSO! Great source for ensuring your seungs are correct! Look par[cularly for the SSO Mode, trustedips and the Remote user HTTP Header. This has to be the same as what is seen in tcpdump! Hopefully your setup will look just like this 67

Troubleshoo[ng with tcpdump! Great to verify that the reverse proxy actually works and that the seungs are correct! Look par[cularly for the Remote user HTTP Header 68

Shameless Plug

Splunk Search Usage! Splunk Search Usage Analysis and Adop[on Tracking, with security reports! hwp://www.davidveuve.com/go/ssu 70

Wrap Up

Wrap Up! Three Op[ons for Single Sign On: Windows Web Server Easy Linux Web Server Easy SAML Achievable, recommend a packaged solu[on if you need this (e.g., Okta, PingIden[ty, etc.)! SSO gives you more security, greater adop[on, and less headache! You can probably set this up in your environment in < 1 hr! Check out the Splunk Search Usage app to bewer understand users and broaden adop[on! 72

! That was a lot of material, right? Config Files GitHub! Get all the configs here: hwp://www.davidveuve.com/go/conf- sso hwp://xkcd.com/565/ 73

THANK YOU