Hadoop Elephant in Active Directory Forest. Marek Gawiński, Arkadiusz Osiński Allegro Group

Size: px

Start display at page:

Download "Hadoop Elephant in Active Directory Forest. Marek Gawiński, Arkadiusz Osiński Allegro Group"

Sherilyn Wilkerson
8 years ago
Views:

1 Hadoop Elephant in Active Directory Forest Marek Gawiński, Arkadiusz Osiński Allegro Group

3 Agenda Goals and motivations Technology stack Architecture evolution Automation integrating new servers Making AD users and groups visible to Linux Making architecture non-vulnerable to AD service inaccessibility Auto-deployment clients software on desktops

groups visible to Linux Making architecture non-vulnerable to AD

4 Allegro Hadoop cluster in numbers 4 terabytes RAM 2 petabytes disk space 47 datanodes 79 projects 612 users

5 Goals and motivations Secured cluster Central authentication and authorisation Compliance for real and project users and groups Cluster resources available from desktop Integrating new servers automatically Making whole architecture non-vulnerable for failures or timeouts to AD Auto-deployment and autoconfiguration of Hadoop clients software on users desktops

new servers automatically Making whole architecture non-vulnerable for failures or

6 Technology stack Cloudera CDH5 MIT Kerberos Microsoft Active Directory FreeIPA sssd puppet msktutil Hadoop desktop client

7 History - FreeIPA+FreeIPA Kerberos In te rn al ha do op FreeIPA User cr ed Chec k gro Kerberos Service Ticket ups Check user/pass s Secured Hadoop cluster Local groups management User/pass Client Kerberos KDC

8 History - FreeIPA+own Kerberos Secured Hadoop cluster Chec Internal hadoop creds k gro Kerberos Service Ticket ups Check user/pass FreeIPA User Local groups management User/pass Client Kerberos KDC Kerberos KDC MIT

Ticket ups Check user/pass FreeIPA User Local groups

9 History - FreeIPA+own Kerberos+AD In te rn al ha do op FreeIPA User cr ed Chec k gro ups Ch kg ec Kerberos Service Ticket Local groups management ps u ro Check user/pass s Secured Hadoop cluster User/pass Client Kerberos KDC MIT Us e r/p s Check user/pass as AD User&Groups AD Kerberos

management ps u ro Check user/pass s Secured Hadoop cluster User/pass

10 Final - own Kerberos+AD In te rn al ha do op cr ed s Secured Hadoop cluster Ch kg ec Kerberos Service Ticket ps u ro Client Kerberos KDC MIT Us e r/p s Check user/pass as AD User&Groups AD Kerberos

11 Integrating new Linux servers automatically with AD Kerberos keytab user e t a Cre AD Kerberos Msktutil Create AD User&Groups princip al

12 Integrating new Linux servers automatically with AD define get_ad_keytab ( $path = '',...) {... $realm = 'SOME_REALM' $pass = hiera('hadoop_prod/ad/krb_manager_pass') $principal = "${title}/${host}@${realm}" $command = "echo ${pass} kinit _hadoop_manager@${realm}; \ /usr/local/bin/add_ad_princ.sh ${title} ${host} ${path}; kdestroy"... msktutil -c -s $PRINCIPAL --upn $PRINCIPAL -k $KEYTAB \ --computer-name $COMPUTER_NAME \ --server $SERVER_KRB \ --realm $REALM \ -b $USER_LDAP_ROOT \ --dont-expire-password \ --description "\"$DESCRIPTION\"" \ --user-creds-only

$kinit _hadoop_manager@${realm}; \ /usr/local/bin/add_ad_princ.sh ${title} ${host} ${path}; kdestroy".$

13 Integrating new Linux servers automatically with AD klist -ket Keytab name: FILE:/etc/krb5.keytab KVNO Timestamp Principal /17/ :26:45 (aes256-cts-hmac-sha1-96) 1 08/17/ :26:45 host/nn1.local@ipa.realm (aes128-cts-hmac-sha1-96) 1 08/17/ :26:45 host/nn1.local@ipa.realm (des3-cbc-sha1) 1 08/17/ :26:45 host/nn1.local@ipa.realm (arcfour-hmac) 1 08/17/ :26:45 host/nn1.local@ipa.realm (camellia128-cts-cmac) 1 08/17/ :26:45 host/nn1.local@ipa.realm (camellia256-cts-cmac) 4 08/17/ :30:23 91c76848bc458b62e67$@AD.REALM (arcfour-hmac) 4 08/17/ :30:23 91c76848bc458b62e67$@AD.REALM (aes128-cts-hmac-sha1-96) 4 08/17/ :30:23 91c76848bc458b62e67$@AD.REALM (aes256-cts-hmac-sha1-96) 4 08/17/ :30:23 host/nn1.local@ad.realm (arcfour-hmac) 4 08/17/ :30:23 host/nn1.local@ad.realm (aes128-cts-hmac-sha1-96) 4 08/17/ :30:23 host/nn1.local@ad.realm (aes256-cts-hmac-sha1-96)

realm (aes256-cts-hmac-sha1-96) 1 08/17/2015 13:26:45 host/nn1.local@ipa.realm (aes128-cts-hmac-sha1-96) 1 08/17/2015 13:26:45 host/nn1.local@ipa.realm (des3-cbc-sha1) 1 08/17/2015 13:26:45 host/nn1.

14 Integrating new Linux servers automatically with AD Separated Subtree in AD structure

15 System Security Services Daemon Identity and authentication Multiple providers (FreeIPA, LDAP, AD) High availability for backends Provides PAM and NSS modules Caching > 1.11.x - stable support for AD forest auth

AD) High availability for backends Provides PAM and

16 System Security Services Daemon /etc/sssd/sssd.conf [domain/ad.realm] id_provider = ad ad_server = h1, h2, h3 ad_backup_server = hb1, hb2, hb3 auth_provider = ad chpass_provider = ad access_provider = ad enumerate = False krb5_realm = AD.REALM ldap_schema = ad ldap_id_mapping = True cache_credentials = True ldap_access_order = expire ldap_account_expire_policy = ad ldap_force_upper_case_realm = true fallback_homedir = /home/ad.realm/%u default_shell = /bin/false ldap_referrals = false AD schema with no modifications root@nn1:~# id _hc_tech_prod tr "," "\n" uid= (_hc_tech_prod) gid= (domain users) groups= (domain users) (_gr_hc_users_common) (_gr_hc_hadoop_prod) (_gr_hc_project1_prod) (_gr_hc_project2_prod)

$realm/%u default_shell = /bin/false ldap_referrals = false AD schema with no modifications root@nn1:~# id _hc_tech_prod tr "," "\n" uid=1827653611(_hc_tech_prod) gid=1827600513(domain users)$

17 Making whole architecture nonvulnerable for failures Active Closest DC Fallback servers in Remote DC Local filesystem nss cache /etc/sssd/sssd.conf [nss] memcache_timeout = 3600

18 Auto-deployment and autoconfiguration on desktops Install script for Hadoop Client on desktops Refresh configs with currently prod environment Support for HDFS/YARN/Hive/Spark [marek.gawinski:~/allehadoop] $ sh env.sh Password for marek.gawinski@ad.realm: ************** [marek.gawinski:~/allehadoop] $ klist Ticket cache: FILE:/tmp/krb5cc_ Default principal: marek.gawinski@ad.realm Valid starting Expires 09/04/15 23:31:35 09/05/15 09:31:35 renew until 09/11/15 23:31:33 Service principal krbtgt/ad.realm@ad.realm

realm: ************** [marek.gawinski:~/allehadoop] $ klist Ticket cache: FILE:/tmp/krb5cc_1511317717 Default principal: marek.

19 Auto-deployment and autoconfiguration on desktops [marek.gawinski:~/allehadoop] Found 8 items drwxr-xr-x - marek.gawinski drwxr-xr-x - marek.gawinski drwxr-xr-x - marek.gawinski drwx marek.gawinski drwxr-xr-x - marek.gawinski -rw-r--r-3 marek.gawinski -rw-r--r-3 marek.gawinski drwxr-xr-x - marek.gawinski $ hdfs dfs -ls hadoop hadoop hadoop hadoop hadoop hadoop hadoop hadoop [marek.gawinski:~/allehadoop] $ hive hive (default)> show databases; OK database_name tpch_benchmarks... xwing_poc Time taken: seconds, Fetched: 72 row(s) hive (default)> set hive.execution.engine = tez; hive (default)> select count(*) from table1; 02:00 21:01 10:43 02:35 13:11 15:26 12:30 16:21.Trash.hiveJars.sparkStaging.staging oozie1 ozzietest1.hql pwd.txt tables

20 Auto-deployment and autoconfiguration on desktops

21 Auto-deployment and autoconfiguration on desktops

22 Auto-deployment and autoconfiguration on desktops

23 Auto-deployment and autoconfiguration on desktops

24 Benefits One standard for access control to all company resources Every new employee automatically can play with Hadoop with no additional effort One password to all systems

25 Thank you! Questions?

System Security Services Daemon

System Security Services Daemon System Security Services Daemon Manages communication with centralized identity and authentication stores Provides robust, predictable caching for network accounts Can cache