Centrify Identity and Access Management for Hortonworks

Transcription

1 Centrify Identity and Access Management for Hortonworks Integrion Guide Abstract Centrify Server Suite is an enterprise-class solution th secures Hortonworks Da Plform leveraging an organizion s existing Active Directory infrastructure to deliver identity, access control, privilege management and user-level auditing CENTRIFY CORPORATION. ALL RIGHTS RESERVED. PAGE 1

2 Informion in this document, including URL and other Internet Web site references, is subject to change without notice. Unless otherwise noted, the example companies, organizions, products, domain names, addresses, logos, people, places and events depicted herein are fictitious, and no associion with any real company, organizion, product, domain name, address, logo, person, place or event is intended or should be inferred. Complying with all applicable copyright laws is the responsibility of the user. Without limiting the rights under copyright, no part of this document may be reproduced, stored in or introduced into a retrieval system, or transmitted in any form or by any means (electronic, mechanical, photocopying, recording, or otherwise), or for any purpose, without the express written permission of Centrify Corporion. Centrify may have pents, pent applicions, trademarks, copyrights, or other intellectual property rights covering subject mter in this document. Except as expressly provided in any written license agreement from Centrify, the furnishing of this document does not give you any license to these pents, trademarks, copyrights, or other intellectual property Centrify Corporion. All rights reserved. Centrify, DirectControl and DirectAudit are registered trademarks and Centrify Suite, DirectAuthorize, DirectSecure and DirectManage are trademarks of Centrify Corporion in the United Stes and/or other countries. Microsoft, Active Directory, Windows, Windows NT, and Windows Server are either registered trademarks or trademarks of Microsoft Corporion in the United Stes and/or other countries. The names of actual companies and products mentioned herein may be the trademarks of their respective owners CENTRIFY CORPORATION. ALL RIGHTS RESERVED. PAGE 2

3 Contents Contents... 3 Overview... 4 Planning for Active Directory Integrion... 4 Cluster Creion Pre-Requisites... 5 Preparing Active Directory... 6 Setup Centrify Zones and setup Roles for Linux login... 7 Setup Hortonworks Cluster with Centrify... 8 Setup the Virtual Machines... 8 Install Centrify on each node in the cluster... 9 Install Hortonworks on each node in the cluster... 9 Enable Security Verify Proper Operion Verify Active Directory managed Service Accounts Finishing the Security Wizard and Testing Services Setting Long Term Account Maintenance Zone enable Hadoop Accounts Validing Your Cluster s Security Conclusion How to Contact Centrify CENTRIFY CORPORATION. ALL RIGHTS RESERVED. PAGE 3

4 Overview Centrify Server Suite is an enterprise-class solution th secures even the most complex Hadoop environments leveraging an organizion s existing Active Directory infrastructure to deliver access control, privilege management and user-level auditing. Centrify Server Suite secures the industry's broadest range of mission-critical servers from identityreled insider risks and outsider tacks, making security and regulory compliance repeable and sustainable. The solution leverages existing Active Directory infrastructure to centrally manage authenticion, access controls, privileged identities, policy enforcement and compliance for onpremises and cloud resources. Centrify Server Suite provides Identity, Access and Privilege Management for the Hortonworks Da Plform:! Simplifying AD integrion for Hortonworks to run in secure mode! Automing service account credential management! Simplifying access with AD-based user single sign-on authenticion! Ensuring regulory compliance with least privilege and auditing! Developer SDKs for secure client applicion access to Hadoop NOTE: This document provides the configurion guidance for multiple Hortonworks clusters to be managed within an Active Directory environment. The key to multiple clusters in Active Directory is the addition of a cluster prefix to the associed Hortonworks Kerberos principals or Active Directory Account Name. Without the cluster prefix, Kerberos principals for the accounts for each cluster would have the same name User Principal Name (UPN). These account names (UPN) must be unique within the Active Directory domain. Planning for Active Directory Integrion Hadoop s security implemention uses Kerberos which is built into Active Directory. As a result all principals are user principals and th there will be an Active Directory account for each service th requires a keytab. From an implemention perspective a 2-node cluster with 6 unique distributed services will require 12 Active Directory accounts where each will require a unique Kerberos keytab file. Centrify provides a centralized access control and privilege management solution built on top of Active Directory th simply requires the Centrify agent software to be installed on every node within the cluster while administrion is performed through Microsoft Management Consoles on an administror s Windows computer CENTRIFY CORPORATION. ALL RIGHTS RESERVED. PAGE 4

5 Cluster Creion Pre-Requisites There are several common requirements such as you must have an Active Directory environment running, you will need a Windows workstion joined to the domain where you can run administrive consoles and you will need several Linux systems on which to install Hortonworks. Centrify software You should request a free trial of Centrify Server Suite if you don t already have access to Centrify software from just specify Hadoop in the Comments field. You can find the Centrify Documention online here after you register for a free trial and setup your Centrify Account here Naming convention You should outline a naming convention for all Hadoop components th will reside in AD. Ideally you will be able to identify the cluster in the names. But keep in mind the limitions of the Active Directory samaccountname th has a maximum length of 20 characters and must be unique across the Active Directory environment.! You will need an Active Directory OU for managing all your Hadoop clusters such as OU=Hadoop. You may have to ask your Active Directory team to cree this OU for you. The technical lead or Hadoop admin should be given full control of this Hadoop OU. Your Active Directory Domain Admin will need to delege administrive rights of this OU to your technical lead.! Each cluster should have it s own OU in order to independently manage it s nodes and service accounts. The OU name should reflect the name of the cluster; e.g. HWC9. This is usually creed within an OU th was creed by the AD staff and deleged to you so th you can cree an OU for each Hortonworks cluster and manage the accounts and policies yourself.! Centrify uses Zones as a logical container for storing the Linux access and privilege permissions for the selected Active Directory users who you authorize to access your Hortonworks cluster. You will setup a unique Zone for each Hortonworks cluster you deploy in order to ensure separion of duties and enable deleged administrion. This Centrify Zone containing the Linux identity, access and privilege informion is stored within the OU th was creed for you in the steps above. Use the child zone name as the same name for the cluster prefix, e.g. HWC9. Servers and Hortonworks software Additionally, you will need the following:! At least 2 Linux systems th are compible with Hortonworks to use for the Hadoop nodes. Ideally the Ganglia and Nagios monitoring services are setup.! Access to Hortonworks Da Plform software CENTRIFY CORPORATION. ALL RIGHTS RESERVED. PAGE 5

6 ! Preferably the organizion is running their own Hadoop repository/repo (this speeds up any setup) Preparing Active Directory Cree Active Directory OUs (Organizional Unit is just a container for AD objects). For this task you may need your Active Directory administror to perform the first step and grant you deleged permission to manage this top level OU for! Cree the Hadoop OU; e.g. OU=Hadoop, DC=Company, DC=com! Then for each Cluster cree another OU under OU=Hadoop; e.g. OU=HWC9, OU=Hadoop, DC=Company, DC=Com! Next in order to make it easier to manage nodes in the cluster separe from the Service accounts, you may also want to cree a set of child OUs with OU=Nodes and OU=Users 2015 CENTRIFY CORPORATION. ALL RIGHTS RESERVED. PAGE 6

7 Setup Centrify Zones and setup Roles for Linux login Start with the Centrify Server Suite Quick Start Guide to install the Management Consoles and to setup your Centrify Zone with the approprie Roles to grant AD users with login rights to the Linux systems you will join to Active Directory in the next step.! Run the approprie setup program from the Management ISO for Windows 32-bit or 64-bit on a Windows administror s workstion. The setup program simply copies the necessary files to the local Windows computer, so there are no special permissions required to run the setup program other than permission to install files on the local computer. Follow the prompts displayed to select the type of suite to install and which components to install.! Open Access Manager to start the Setup Wizard and cree the containers for Licenses and Zones. You can accept the default locions or use cree a Centrify organizional unit for the containers.! In Access Manager, cree a new zone with the default options. For example, cree a new zone named Hadoop.! In Access Manager, add Active Directory users to the new zone. These are the users you will grant access permission to login to the Hadoop cluster.! Select the new Hadoop zone.! Right-click, then select Add User to search for and select existing Active Directory users.! Select Define user UNIX profile and deselect assign roles.! Accept the defaults for all fields.! Cree a child zone.! Select the Hadoop zone.! Right-click, then select Cree Child Zone.! Type a name for the zone, for example, HWC9 and an optional description, then click Next and Finish to cree the new child zone CENTRIFY CORPORATION. ALL RIGHTS RESERVED. PAGE 7

8 ! Assign a role for the users you added to the Hadoop zone. User profiles are inherited by child zones, so the users you added to Hadoop, automically have a profile in HWC1. To login to a machine, a user requires a profile and a role assignment. DirectManage provides a default UNIX Login role th you can assign to enable users to login.! Expand Child Zones, HWC9, and Authorizion.! Select Role Assignments, right-click, then click Assign Role.! Select the UNIX Login role from the results and click OK.! Click Add AD Account, then search for one of the Active Directory user you added to the Hadoop zone. Select this user and click OK. Setup Hortonworks Cluster with Centrify Setup the Virtual Machines! Provision 2 new Centos 6.x virtual machines:! C9n1.centrifyimage.vms ( ), 2 processors, 8GB RAM, 1 HD (40gb)! C9n2.centrifyimage.vms ( ), 2 processors, 8GB RAM, 1 HD (40gb)! Cree the corresponding DNS A records in the approprie DNS Zone, in this case we are using centrifyimage.vms DNS zone. Make sure to setup the proper reverse DNS entries as well.! One each Hadoop node:! Perform a yum upde! Disable and stop the iptables service (chkconfig iptables off && service iptables stop)! Enable the ntpd service (chkconfig ntpd on) 2015 CENTRIFY CORPORATION. ALL RIGHTS RESERVED. PAGE 8

9 ! Disable selinux (edit /etc/selinux/config)! Set the directive to enabled = 0 on the /etc/yum/pluginconf.d/refresh-packagekit.conf! Cree /etc/security/keytabs directory (mkdir p /etc/security/keytabs)! On the c9n1 Node or your first node you will need to:! Run the ssh-keygen command and copy the contents of id_rsa.pub to /root/.ssh/authorized_keys.! Attempt an ssh connection as root to the second node, c9n2.centrifyimage.vms! Copy the /root/.shh/authorized_keys file to c9n2:/root/.ssh Install Centrify on each node in the cluster Install the Centrify Agent and join the nodes to Active Directory.! After downloading Centrify agents disk image, just copy the approprie tgz file from the ISO to the Nodes, un pack the file and run the install.sh! Install.sh will ask several questions if you run it interactively which is suggested this first time, however the installion can be automed with a custom config file for silent installion. Just install Standard Edition of Centrify Suite and do not join Active Directory, we will need to do th after making a few changes to the configurion files.! Edit the /etc/centrifydc/centrifydc.conf file and uncomment he adclient.krb5.service.principals line and remove the http principal. Note: this step is required or the cluster will not start. Centrify should not cree serviceprincipalname for the http service since Hortonworks will need to do this ler.! Join your zone (adjoin z zone c container V u user domain name) adjoin z HWC9 c ou=hwc9,ou=hadoop,dc=company,dc=com V u <your AD loginname> company.com! Optional: Install the Centrify Audit agent and enable audit (rpm Uvh centrifyda- <version>)! The computer should join AD and then you will need to reboot. At this point, you should be able to login with an AD userid and password for the user you granted login rights to previously. Install Hortonworks on each node in the cluster Hortonworks will be installed on the first node in the cluster, in this case th is c9n1.! On c9n1, login as root CENTRIFY CORPORATION. ALL RIGHTS RESERVED. PAGE 9

10 ! Add the Hortonworks repo: wget copy the ambari.repo file to /etc/yum.repos.d! Note: This is OK, the Centos6 repo seems to be down the time of this writing.! Install the epel repository (yum install epel-release)! Confirm the repos (yum repolist)! Install the ambari server (yum install ambari-server) The server install will prompt you for dependencies and to accept the Oracle JDK EULA.! Run the ambari-server setup program and accept all the defaults.! Start the ambari server (ambari-server start)! On the Welcome page, name your cluster (e.g. HWC9)! On the Select Stack page, select HDP 2.1! One the install Options page > Target hosts, enter the FQDNs for the Hadoop servers and in the host registrion page, paste the contents of hadoop1:/root/.ssh/id_rsa 2015 CENTRIFY CORPORATION. ALL RIGHTS RESERVED. PAGE 10

11 ! On the Confirm Hosts page, the installion of the Ambari agents will start! In the Choose Services page, uncheck every service but following. This is to limit the cluster so it does not consume all fo the resources of your machine. (This is especially helpful if you are running on VMs on a laptop.)! HDFS! Nagios! Ganglia 2015 CENTRIFY CORPORATION. ALL RIGHTS RESERVED. PAGE 11

12 ! Zookeeper! Mapreduce2 / YARN! In the Assign Masters page as well as the Assign Slaves and Clients page, accept the defaults.! In the Customize Services page, set up a password and for the Nagios component.! Also, on the Customize Services page, select Misc, to add a cluster pre-fix ( hwc9- to mch the name of your cluster entered earlier) to all users and groups. Note: This step allows for multiple clusters within Active Directory and must be done before Hadoop software deployment.! Select Accept for the changes to the various services 2015 CENTRIFY CORPORATION. ALL RIGHTS RESERVED. PAGE 12

13 ! Then press Next, and next in the Review Page. This will take you to the Install, Start and Test Page progress window! In the Summary page, press Complete. At this point you will be taken to the Ambari Dashboard. The startup of some of the services may have timed out, so you may have to stop all services then restart all CENTRIFY CORPORATION. ALL RIGHTS RESERVED. PAGE 13

14 Enable Security The next step is to configure the cluster to opere in secure mode leveraging the Kerberos th was enabled by the Centrify agent on each of the nodes.! In Ambari, go to Admin > Security and click Enable Security! In the Get Started page, press next! In the Configure Services > General, specify the realm name or Active Directory Domain Name, realms must be all uppercase (CENTRIFYIMAGE.VMS). Note: make sure to use the cluster pre-fix hwc9- on the user principal names for both hdfs and ambari-qa 2015 CENTRIFY CORPORATION. ALL RIGHTS RESERVED. PAGE 14

15 ! In the Cree Principals and Keytabs page, click the Download CSV button and export it to excel. Review the results and you ll realize th there are reusable principals like ambari-qa and hdfs as well as host-specific principals like http. You will return to the wizard once the keytabs are genered. Service Account Creion in Active Directory Centrify Server Suite 2015 will provide tools th autome the creion and distribution of these service accounts. If you are using Centrify Server Suite or prior, you should use the following instructions.! Open an SSH session with an AD user (who can eleve to root) or as root to both servers.! On both servers, set the proper ACLs for the /etc/security/keytabs folder chown root:hwc9-hadoop /etc/security/keytabs chmod 750 /etc/security/keytabs 2015 CENTRIFY CORPORATION. ALL RIGHTS RESERVED. PAGE 15

16 ! On c9n1, use centrify s adkeytab command to cree the Kerberos keytabs and service (headless) accounts for ambari-qa, hdfs, and hbase adkeytab --new --upn [email protected] --keytab /etc/security/keytabs/smokeuser.headless.keytab -c ou=users,ou=hwc9,ou=unix -- ignore -V hwc9-ambari-qa adkeytab --new --upn [email protected] --keytab /etc/security/keytabs/hdfs.headless.keytab -c ou=users,ou=hwc9,ou=unix --ignore -V hwc9-hdfs! Copy via scp the headless keytabs for ambari-qa, hdfs, hbase to c9n2:/etc/security/keytabs! On nodes c9n1 & c9n2, use adkeytab to cree the keytabs for the node specific principals adkeytab --new -P HTTP/[email protected] -U HTTP/[email protected] --keytab /etc/security/keytabs/spnego.service.keytab -c ou=users,ou=hwc9,ou=unix --ignore -V c9n1-http adkeytab --new -P nn/[email protected] -U nn/[email protected] --keytab /etc/security/keytabs/nn.service.keytab -c ou=users,ou=hwc9,ou=unix --ignore -V c9n1-nn adkeytab --new -P HTTP/[email protected] -U HTTP/[email protected] --keytab /etc/security/keytabs/spnego.service.keytab -c ou=users,ou=hwc9,ou=unix --ignore -V c9n2-http adkeytab --new -P nn/[email protected] -U nn/[email protected] --keytab /etc/security/keytabs/nn.service.keytab -c ou=users,ou=hwc9,ou=unix --ignore -V c9n2-nn! Set the proper security for both files in both hosts with the following script: cd /etc/security/keytabs chown hwc9-hdfs:hwc9-hadoop dn.service.keytab chown hwc9-falcon:hwc9-hadoop falcon.service.keytab chown hwc9-hbase:hwc9-hadoop hbase.* chown hwc9-hdfs:hwc9-hadoop hdfs.headless.keytab chown hwc9-hive:hwc9-hadoop hive.service.keytab chown hwc9-mapred:hwc9-hadoop jhs.service.keytab chown hwc9-nagios:hwc9-hadoop nagios.service.keytab chown hwc9-yarn:hwc9-hadoop nm.service.keytab chown hwc9-hdfs:hwc9-hadoop nn.service.keytab chown hwc9-oozie:hwc9-hadoop oozie.service.keytab chown hwc9-yarn:hwc9-hadoop rm.service.keytab chown hwc9-ambari-qa:hwc9-hadoop smokeuser.headless.keytab chown root:hwc9-hadoop spnego.service.keytab chown hwc9-storm:hwc9-hadoop storm.service.keytab chown hwc9-zookeeper:hwc9-hadoop zk.service.keytab chmod 400 * chmod 440 *headless* chmod 440 spnego*! On each individual host, cree the host-specific principals. E.g. for the zookeeper principal! Cree a Kerberos ticket for the AD user with privedlge to cree the keytabs >kinit >adkeytab --new -P zookeeper/[email protected] --keytab /etc/security/keytabs/zk.service.keytab -c ou=users,ou=hwc9,ou=unix --ignore -V c9n1-zookeeper Entering the above keytab will cree the following output ADKeyTab version: CentrifyDC Options use machine ccache: no domain: centrifyimage.vms server: null gc: null user: null container: ou=users,ou=hwc9,ou=unix 2015 CENTRIFY CORPORATION. ALL RIGHTS RESERVED. PAGE 16

17 account: c9n1-zookeeper trust: no des: no Attempting bind to centrifyimage.vms site:demo-site server:dc.centrifyimage.vms: ccache:file:/tmp/krb5cc_ Bind successful to server dc.centrifyimage.vms Attempting bind to GC domain:centrifyimage.vms site:demo-site gcserver:dc.centrifyimage.vms ccache:file:/tmp/krb5cc_ Bound to GC server:dc.centrifyimage.vms domain:centrifyimage.vms Searching for AD Object: filter = (samaccountname=c9n1-zookeeper), root = DC=centrifyimage,DC=vms Searching for AD Object: filter = (samaccountname=c9n1-zookeeper$), root = DC=centrifyimage,DC=vms AD Object not found. Building Container DN from OU=USERS,OU=HWC9,OU=UNIX Account 'CN=c9n1-zookeeper,OU=USERS,OU=HWC9,OU=UNIX,DC=centrifyimage,DC=vms' does not exist Search for account in GC: filter = (samaccountname=c9n1-zookeeper), root = DC=CENTRIFYIMAGE,DC=VMS SAM name 'c9n1-zookeeper' not found in GC Problem to cree account; try again with no password required Searching for AD Object: filter = (samaccountname=c9n1-zookeeper), root = DC=centrifyimage,DC=vms AD Object found: CN=c9n1-zookeeper,OU=Users,OU=HWC9,OU=Unix,DC=centrifyimage,DC=vms Key Version = 1 Adding managed account keys to configurion file: c9n1-zookeeper Changing account 'c9n1-zookeeper' password with user '[email protected]' credentials. Searching for AD Object: filter = (samaccountname=c9n1-zookeeper), root = DC=centrifyimage,DC=vms AD Object found: CN=c9n1-zookeeper,OU=Users,OU=HWC9,OU=Unix,DC=centrifyimage,DC=vms Key Version = 2 Success: New Account: c9n1-zookeeper! Repe for all principals th correspond to each host. Note: Centrify Server Suite 2015 will provide tools th autome the creion and distribution of these service accounts. Verify Proper Operion Verify Active Directory managed Service Accounts In ADUC, browse to the Hadoop/HWC9 OU, you should see your Service Account in AD. On each host, you should see the keytabs with the approprie permissions: 2015 CENTRIFY CORPORATION. ALL RIGHTS RESERVED. PAGE 17

18 Now you re ready to return to the Ambari Security Wizard. Finishing the Security Wizard and Testing Services In the Cree Principals and Keytabs page, click Apply. At this point, Ambari will reconfigure all the services to use Kerberos for authenticion. Once complete, press Done and you ll be returned to the Ambari Dashboard. Note: Depending on how your cluster performs, you may see a Failed message in the page, but don t worry, this may mean th you have to start some services manually. For example, in my environment, I had to start the NameNode and Nagios service manually CENTRIFY CORPORATION. ALL RIGHTS RESERVED. PAGE 18

19 Setting Long Term Account Maintenance Centrify s Direct Control automically maintains the keytab entries th are part of the machine account when adclient changes machine password every 28 (default value) days. Other keytab are NOT automically refreshed, such those creed for Hadoop. A script could issue an adkeytab -C th will upde keytab for the specified account because user tells Active Directory the password, so Direct Control will upde the account, and get a new kvno. The upshot of the above is the accounts (Hadoop principals) should have passwords set to never expire and those accounts not used for management locked CENTRIFY CORPORATION. ALL RIGHTS RESERVED. PAGE 19

20 Zone enable Hadoop Accounts The Ambari installer will automically cree a number of local accounts on the node with the cluster prefix hwc9-. In addition, the RPM installer will cree additional accounts without the cluster prefix (see below). postgres:x:26:26:postgresql Server:/var/lib/pgsql:/bin/bash hwc9-ambari-qa:x:1001:501::/home/hwc9-ambari-qa:/bin/bash hwc9-nagios:x:1002:502::/home/hwc9-nagios:/bin/bash hwc9-yarn:x:1003:501::/home/hwc9-yarn:/bin/bash hwc9-nobody:x:1004:501::/home/hwc9-nobody:/bin/bash hwc9-hdfs:x:1005:501::/home/hwc9-hdfs:/bin/bash hwc9-mapred:x:1006:501::/home/hwc9-mapred:/bin/bash hwc9-zookeeper:x:1007:501::/home/hwc9-zookeeper:/bin/bash hwc9-tez:x:1008:501::/home/hwc9-tez:/bin/bash rrdcached:x:496:493:rrdcached:/var/rrdtool/rrdcached:/sbin/nologin zookeeper:x:495:492:zookeeper:/var/run/zookeeper:/bin/bash hdfs:x:494:491:hadoop HDFS:/var/lib/hadoop-hdfs:/bin/bash After zone enabling all of the above accounts th have a cluster prefix the local accounts can be removed from all nodes in the cluster CENTRIFY CORPORATION. ALL RIGHTS RESERVED. PAGE 20

21 In the example above the cluster specific accounts hwc9-nagios, hwc9-yarn, etc are linked to normal AD accounts nagios, yarn except the headless accounts. The headless accounts are cree during keytab creion with specific UPN and are cluster wide. However, the headless accounts still must be zone enabled. Validing Your Cluster s Security First you should verify th users cannot access the cluster without having logged into Active Directory to obtain their Kerberos credential which is now required to gain access to the cluster. In the following session, you will see th the initial Hadoop command and mapreduce job will fail since the user dwirth does not have a valid Kerberos ticket. Using username "dwirth". CentOS release 6.5 (Final) Kernel el6.x86_64 on an x86_64 Last login: Fri Oct 24 14:23: from dc.centrifyimage.vms [dwirth@c9n2 ~]$ whoami dwirth [dwirth@c9n2 ~]$ id uid= (dwirth) gid= (dwirth) groups= (dwirth),650(uni -adm) [dwirth@c9n2 ~]$ hadoop fs -ls /user 14/10/24 14:24:59 WARN ipc.client: Exception encountered while connecting to the server : javax.security.sasl.saslexception: GSS initie failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)] ls: Failed on local exception: java.io.ioexception: javax.security.sasl.saslexception: GSS initie failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]; Host Details : local host is: "c9n2.centrifyimage.vms/ "; destinion host is: "c9n1.centrifyimage.vms":8020; [dwirth@c9n2 ~]$ yarn jar /usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar pi Number of Maps = 16 Samples per Map = /10/24 14:25:17 WARN ipc.client: Exception encountered while connecting to the server : javax.security.sasl.saslexception: GSS initie failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)] java.io.ioexception: Failed on local exception: java.io.ioexception: javax.security.sasl.saslexception: GSS initie failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]; Host Details : local host is: "c9n2.centrifyimage.vms/ "; destinion host is: "c9n1.centrifyimage.vms":8020; 2015 CENTRIFY CORPORATION. ALL RIGHTS RESERVED. PAGE 21

22 org.apache.hadoop.net.netutils.wrapexception(netutils.java:764) org.apache.hadoop.ipc.client.call(client.java:1414) org.apache.hadoop.ipc.client.call(client.java:1363) org.apache.hadoop.ipc.protobufrpcengine$invoker.invoke(protobufrpcengine.java:206) com.sun.proxy.$proxy14.getfileinfo(unknown Source) sun.reflect.nivemethodaccessorimpl.invoke0(nive Method) sun.reflect.nivemethodaccessorimpl.invoke(nivemethodaccessorimpl.java:57) sun.reflect.delegingmethodaccessorimpl.invoke(delegingmethodaccessorimpl.java:4 3) java.lang.reflect.method.invoke(method.java:606) org.apache.hadoop.io.retry.retryinvocionhandler.invokemethod(retryinvocionhandl er.java:190) org.apache.hadoop.io.retry.retryinvocionhandler.invoke(retryinvocionhandler.jav a:103) com.sun.proxy.$proxy14.getfileinfo(unknown Source) org.apache.hadoop.hdfs.protocolpb.clientnamenodeprotocoltranslorpb.getfileinfo(cl ientnamenodeprotocoltranslorpb.java:699) org.apache.hadoop.hdfs.dfsclient.getfileinfo(dfsclient.java:1762) org.apache.hadoop.hdfs.distributedfilesystem$17.docall(distributedfilesystem.java:1 124) org.apache.hadoop.hdfs.distributedfilesystem$17.docall(distributedfilesystem.java:1 120) org.apache.hadoop.fs.filesystemlinkresolver.resolve(filesystemlinkresolver.java:81) org.apache.hadoop.hdfs.distributedfilesystem.getfilestus(distributedfilesystem.ja va:1120) org.apache.hadoop.fs.filesystem.exists(filesystem.java:1398) org.apache.hadoop.examples.quasimontecarlo.estimepi(quasimontecarlo.java:278) org.apache.hadoop.examples.quasimontecarlo.run(quasimontecarlo.java:354) org.apache.hadoop.util.toolrunner.run(toolrunner.java:70) org.apache.hadoop.examples.quasimontecarlo.main(quasimontecarlo.java:363) sun.reflect.nivemethodaccessorimpl.invoke0(nive Method) sun.reflect.nivemethodaccessorimpl.invoke(nivemethodaccessorimpl.java:57) sun.reflect.delegingmethodaccessorimpl.invoke(delegingmethodaccessorimpl.java:4 3) java.lang.reflect.method.invoke(method.java:606) org.apache.hadoop.util.programdriver$programdescription.invoke(programdriver.java:7 2) org.apache.hadoop.util.programdriver.run(programdriver.java:145) org.apache.hadoop.examples.exampledriver.main(exampledriver.java:74) sun.reflect.nivemethodaccessorimpl.invoke0(nive Method) sun.reflect.nivemethodaccessorimpl.invoke(nivemethodaccessorimpl.java:57) sun.reflect.delegingmethodaccessorimpl.invoke(delegingmethodaccessorimpl.java:4 3) java.lang.reflect.method.invoke(method.java:606) org.apache.hadoop.util.runjar.main(runjar.java:212) Caused by: java.io.ioexception: javax.security.sasl.saslexception: GSS initie failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)] org.apache.hadoop.ipc.client$connection$1.run(client.java:677) java.security.accesscontroller.doprivileged(nive Method) javax.security.auth.subject.doas(subject.java:415) org.apache.hadoop.security.usergroupinformion.doas(usergroupinformion.java:1594 ) org.apache.hadoop.ipc.client$connection.handlesaslconnectionfailure(client.java:640 ) org.apache.hadoop.ipc.client$connection.setupiostreams(client.java:724) org.apache.hadoop.ipc.client$connection.access$2800(client.java:367) org.apache.hadoop.ipc.client.getconnection(client.java:1462) 2015 CENTRIFY CORPORATION. ALL RIGHTS RESERVED. PAGE 22

23 org.apache.hadoop.ipc.client.call(client.java:1381) more Caused by: javax.security.sasl.saslexception: GSS initie failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)] com.sun.security.sasl.gsskerb.gsskrb5client.evaluechallenge(gsskrb5client.java:21 2) org.apache.hadoop.security.saslrpcclient.saslconnect(saslrpcclient.java:411) org.apache.hadoop.ipc.client$connection.setupsaslconnection(client.java:550) org.apache.hadoop.ipc.client$connection.access$1800(client.java:367) org.apache.hadoop.ipc.client$connection$2.run(client.java:716) org.apache.hadoop.ipc.client$connection$2.run(client.java:712) java.security.accesscontroller.doprivileged(nive Method) javax.security.auth.subject.doas(subject.java:415) org.apache.hadoop.security.usergroupinformion.doas(usergroupinformion.java:1594 ) org.apache.hadoop.ipc.client$connection.setupiostreams(client.java:711) more Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt) sun.security.jgss.krb5.krb5initcredential.getinstance(krb5initcredential.java:147) sun.security.jgss.krb5.krb5mechfactory.getcredentialelement(krb5mechfactory.java:12 1) sun.security.jgss.krb5.krb5mechfactory.getmechanismcontext(krb5mechfactory.java:187 ) sun.security.jgss.gssmanagerimpl.getmechanismcontext(gssmanagerimpl.java:223) sun.security.jgss.gsscontextimpl.initseccontext(gsscontextimpl.java:212) sun.security.jgss.gsscontextimpl.initseccontext(gsscontextimpl.java:179) com.sun.security.sasl.gsskerb.gsskrb5client.evaluechallenge(gsskrb5client.java:19 3) more [ Now th the Hortonworks cluster is using Centrify for Active Directory based authenticion, the user Diana Worth can now login using her Active Directory credentials directly the console prompt or could use a Kerberized SSH client such as Centrify s version of PuTTY on her Windows computer to get Single Sign-on to the Cluster. Once logged in, she will have Kerberos credentials from Active Directory and then will be able to run a Hadoop job such as the example used below th computes the value of Pi. Since the cluster is now running in secure mode, users without Kerberos will not be able to successfully submit a job to the cluster. dwirth@c9n2 ~]$ kinit Password for [email protected]: [dwirth@c9n2 ~]$ hadoop fs -ls /user Found 2 items drwxr-xr-x - dwirth dwirth :38 /user/dwirth drwxrwx--- - hwc9-ambari-qa hwc9-hdfs :19 /user/hwc9- ambari-qa [dwirth@c9n2 ~]$ yarn jar /usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar pi Number of Maps = 16 Samples per Map = 1000 Wrote input for Map #0 Wrote input for Map #1 Wrote input for Map #2 Wrote input for Map #3 Wrote input for Map #4 Wrote input for Map #5 Wrote input for Map # CENTRIFY CORPORATION. ALL RIGHTS RESERVED. PAGE 23

24 Wrote input for Map #7 Wrote input for Map #8 Wrote input for Map #9 Wrote input for Map #10 Wrote input for Map #11 Wrote input for Map #12 Wrote input for Map #13 Wrote input for Map #14 Wrote input for Map #15 Starting Job 14/10/24 14:25:48 INFO client.rmproxy: Connecting to ResourceManager c9n2.centrifyimage.vms/ : /10/24 14:25:48 INFO hdfs.dfsclient: Creed HDFS_DELEGATION_TOKEN token 6 for dwirth on : /10/24 14:25:48 INFO security.tokencache: Got dt for hdfs://c9n1.centrifyimage.vms:8020; Kind: HDFS_DELEGATION_TOKEN, Service: :8020, Ident: (HDFS_DELEGATION_TOKEN token 6 for dwirth) 14/10/24 14:25:49 INFO input.fileinputform: Total input phs to process : 16 14/10/24 14:25:49 INFO mapreduce.jobsubmitter: number of splits:16 14/10/24 14:25:49 INFO mapreduce.jobsubmitter: Submitting tokens for job: job_ _ /10/24 14:25:49 INFO mapreduce.jobsubmitter: Kind: HDFS_DELEGATION_TOKEN, Service: :8020, Ident: (HDFS_DELEGATION_TOKEN token 6 for dwirth) 14/10/24 14:25:50 INFO impl.yarnclientimpl: Submitted applicion applicion_ _ /10/24 14:25:50 INFO mapreduce.job: The url to track the job: 14/10/24 14:25:50 INFO mapreduce.job: Running job: job_ _ /10/24 14:26:00 INFO mapreduce.job: Job job_ _0003 running in uber mode : false 14/10/24 14:26:00 INFO mapreduce.job: map 0% reduce 0% 14/10/24 14:26:08 INFO mapreduce.job: map 6% reduce 0% 14/10/24 14:26:09 INFO mapreduce.job: map 13% reduce 0% 14/10/24 14:26:16 INFO mapreduce.job: map 19% reduce 0% 14/10/24 14:26:17 INFO mapreduce.job: map 25% reduce 0% 14/10/24 14:26:23 INFO mapreduce.job: map 31% reduce 0% 14/10/24 14:26:25 INFO mapreduce.job: map 38% reduce 0% 14/10/24 14:26:29 INFO mapreduce.job: map 44% reduce 0% 14/10/24 14:26:33 INFO mapreduce.job: map 50% reduce 0% 14/10/24 14:26:36 INFO mapreduce.job: map 56% reduce 0% 14/10/24 14:26:40 INFO mapreduce.job: map 63% reduce 0% 14/10/24 14:26:45 INFO mapreduce.job: map 69% reduce 0% 14/10/24 14:26:48 INFO mapreduce.job: map 69% reduce 23% 14/10/24 14:26:50 INFO mapreduce.job: map 75% reduce 23% 14/10/24 14:26:54 INFO mapreduce.job: map 75% reduce 25% 14/10/24 14:26:55 INFO mapreduce.job: map 81% reduce 25% 14/10/24 14:26:57 INFO mapreduce.job: map 81% reduce 27% 14/10/24 14:27:00 INFO mapreduce.job: map 88% reduce 27% 14/10/24 14:27:03 INFO mapreduce.job: map 88% reduce 29% 14/10/24 14:27:05 INFO mapreduce.job: map 94% reduce 29% 14/10/24 14:27:06 INFO mapreduce.job: map 94% reduce 31% 14/10/24 14:27:10 INFO mapreduce.job: map 100% reduce 31% 14/10/24 14:27:11 INFO mapreduce.job: map 100% reduce 100% 14/10/24 14:27:12 INFO mapreduce.job: Job job_ _0003 completed successfully 14/10/24 14:27:13 INFO mapreduce.job: Counters: 49 File System Counters FILE: Number of bytes read=358 FILE: Number of bytes written= FILE: Number of read operions=0 FILE: Number of large read operions=0 FILE: Number of write operions=0 HDFS: Number of bytes read=4454 HDFS: Number of bytes written=215 HDFS: Number of read operions=67 HDFS: Number of large read operions=0 HDFS: Number of write operions=3 Job Counters Launched map tasks=16 Launched reduce tasks=1 Da-local map tasks=16 Total time spent by all maps in occupied slots (ms)=83721 Total time spent by all reduces in occupied slots (ms)=33925 Total time spent by all map tasks (ms)=83721 Total time spent by all reduce tasks (ms)=33925 Total vcore-seconds taken by all map tasks=83721 Total vcore-seconds taken by all reduce tasks= CENTRIFY CORPORATION. ALL RIGHTS RESERVED. PAGE 24

25 Total megabyte-seconds taken by all map tasks= Total megabyte-seconds taken by all reduce tasks= Map-Reduce Framework Map input records=16 Map output records=32 Map output bytes=288 Map output merialized bytes=448 Input split bytes=2566 Combine input records=0 Combine output records=0 Reduce input groups=2 Reduce shuffle bytes=448 Reduce input records=32 Reduce output records=0 Spilled Records=64 Shuffled Maps =16 Failed Shuffles=0 Merged Map outputs=16 GC time elapsed (ms)=546 CPU time spent (ms)=9890 Physical memory (bytes) snapshot= Virtual memory (bytes) snapshot= Total committed heap usage (bytes)= Shuffle Errors BAD_ID=0 CONNECTION=0 IO_ERROR=0 WRONG_LENGTH=0 WRONG_MAP=0 WRONG_REDUCE=0 File Input Form Counters Bytes Read=1888 File Output Form Counters Bytes Written=97 Job Finished in seconds Estimed value of Pi is ~]$ Conclusion As you can see, the job executed properly and provided the desired output with the value of Pi after successful login via Active Directory. Centrify Server Suite the industry s most widely deployed solution for securing identity on Linuxand Windows-based servers and applicions provides several benefits for Hadoop and Big Da environments including:! Simple and secure access to Hadoop environments. Centrify makes it simple to run Hadoop in secure mode by leveraging existing identity management infrastructure Active Directory without the hassle of introducing alternive solutions th do not scale and are not enterprise ready. Centrify Server Suite also saves money by letting organizions leverage existing skill sets within the enterprise.! Single sign-on for IT administrors and big da users. By extending the power of Active Directory s Kerberos and LDAP capabilities to Hadoop clusters, Centrify Server Suite lets organizions leverage existing Active Directory-based authenticion for Hadoop administrors and end users. New SSO functionality in Big Da environments makes users more productive and secure by allowing them to login in as themselves, rher than sharing privileged accounts.! Secure machine-to-machine communicions. Centrify Server Suite automes Hadoop service account management within Active Directory. By automing machine-to-machine 2015 CENTRIFY CORPORATION. ALL RIGHTS RESERVED. PAGE 25

26 credential management, Centrify not only secures user identity but also system and service account identity.! Reduced identity-reled risks and greer regulory compliance. The reality is th Hadoop environments store most if not all of an organizion s most important da. Centrify Server Suite tracks user activity back to an individual in Active Directory, thereby making da more secure. Centrify also reports on who did wh across Hadoop clusters, nodes and services. And, by enforcing access controls and least-privilege security across Hadoop, Centrify delivers cost-effective compliance through combined access and activity reporting.! Certified solution for superior compibility and support. Centrify has worked closely with Hortonworks and has received product certificion. This ensures product compibility and technical support collaborion between customers, Hortonworks and Centrify CENTRIFY CORPORATION. ALL RIGHTS RESERVED. PAGE 26

27 How to Contact Centrify North America (And All Locions Outside EMEA) Centrify Corporion 3393 Octavius Dr, Suite 100 Santa Clara, CA United Stes Sales: +1 (669) Online: Europe, Middle East, Africa (EMEA) Centrify EMEA Lilly Hill House Lilly Hill Road Bracknell, Berkshire RG12 2SJ United Kingdom Sales: +44 (0) CENTRIFY CORPORATION. ALL RIGHTS RESERVED. PAGE 27