Cloudera Navigator Installation and User Guide

Transcription

1 Cloudera Navigator Installation and User Guide

2 Important Notice (c) Cloudera, Inc. All rights reserved. Cloudera, the Cloudera logo, Cloudera Impala, and any other product or service names or slogans contained in this document are trademarks of Cloudera and its suppliers or licensors, and may not be copied, imitated or used, in whole or in part, without the prior written permission of Cloudera or the applicable trademark holder. Hadoop and the Hadoop elephant logo are trademarks of the Apache Software Foundation. All other trademarks, registered trademarks, product names and company names or logos mentioned in this document are the property of their respective owners. Reference to any products, services, processes or other information, by trade name, trademark, manufacturer, supplier or otherwise does not constitute or imply endorsement, sponsorship or recommendation thereof by us. Complying with all applicable copyright laws is the responsibility of the user. Without limiting the rights under copyright, no part of this document may be reproduced, stored in or introduced into a retrieval system, or transmitted in any form or by any means (electronic, mechanical, photocopying, recording, or otherwise), or for any purpose, without the express written permission of Cloudera. Cloudera may have patents, patent applications, trademarks, copyrights, or other intellectual property rights covering subject matter in this document. Except as expressly provided in any written license agreement from Cloudera, the furnishing of this document does not give you any license to these patents, trademarks copyrights, or other intellectual property. For information about patents covering Cloudera products, see The information in this document is subject to change without notice. Cloudera shall not be liable for any damages resulting from technical errors or omissions which may be present in this document, or from use of this document. Cloudera, Inc Page Mill Road Bldg 2 Palo Alto, CA info@cloudera.com US: Intl: Release Information Version: 2.0.x Date: April 16, 2015

3 Table of Contents Cloudera Navigator Installation and User Guide...5 Introducing Cloudera Navigator...7 Cloudera Navigator Requirements...9 Cloudera Manager Requirements...9 Cloudera Navigator Auditing Component...9 Cloudera Navigator Metadata Component...10 Installing and Upgrading Cloudera Navigator...13 Cloudera Navigator Auditing Component...15 Cloudera Navigator Audit Server...15 Audit Log Properties...18 Service Auditing Properties...18 Audit Events...20 Viewing Audit Events...21 Filtering Audit Events...21 Downloading Audit Events...22 Downloading HDFS Directory Access Permission Reports...24 Cloudera Navigator Metadata Component...27 Cloudera Navigator Metadata Server...27 Metadata...30 Search Properties...31 Accessing Metadata...34 Modifying Business Metadata...36 Lineage Diagrams...41 Displaying a Template Lineage Diagram...43 Displaying an Instance Lineage Diagram...45 Displaying the Template Lineage Diagram for an Instance Lineage Diagram...45 Downloading a Lineage File...45 Tables...65 Displaying Table Schema...65

4 Displaying Pig Tables...66

5 Cloudera Navigator Installation and User Guide Cloudera Navigator Installation and User Guide This guide explains how to install, configure, and use Cloudera Navigator. Cloudera Navigator Installation and User Guide 5

6

7 Introducing Cloudera Navigator Introducing Cloudera Navigator Cloudera Navigator is a fully integrated data management tool for the Hadoop platform. Data management capabilities are critical for enterprise customers that are in highly regulated industries and have stringent compliance requirements. Cloudera Navigator provides two categories of functionality: Auditing data access and verifying access privileges - Cloudera Navigator allows administrators to configure, collect, and view audit events, to understand who accessed what data and how. Cloudera Navigator also allows administrators to generate reports that list the HDFS access permissions granted to groups. Cloudera Navigator tracks access permissions and actual accesses to all entities in HDFS, Hive, HBase, and Cloudera Impala to help answer questions such as - who has access to which entities, which entities were accessed by a user, when was an entity accessed and by whom, what entities were accessed using a service, which device was used to access, and so on. Cloudera Navigator auditing supports tracking access to: HDFS data accessed through HDFS, Hive, HBase, Cloudera Impala services HBase and Impala operations Hive metadata Sentry access Searching metadata and visualizing lineage - Cloudera Navigator metadata management features allow DBAs, data modelers, business analysts, and data scientists to search for, amend the properties of, and tag data entities. In addition, to satisfy risk and compliance audits and data retention policies, it supports the ability to answer questions such as: where did the data come from, where is it used, and what are the consequences of purging or modifying a set of data entities. Cloudera Navigator supports tracking the lineage of HDFS files and directories, Hive tables and columns, MapReduce and YARN jobs, Hive queries, Pig scripts, Sqoop jobs, and Oozie workflows. Cloudera Navigator Installation and User Guide 7

8

9 Cloudera Navigator Requirements Cloudera Navigator Requirements Cloudera Manager Requirements This section describes the Cloudera Navigator requirements for Cloudera Manager. Cloudera Navigator 2 is available with Cloudera Manager 5.1. For information on the requirements for installing Cloudera Manager, see Requirements for Cloudera Manager. Cloudera Navigator Auditing Component This section describes the databases, service versions, and audited operations supported by the Cloudera Navigator auditing component. Supported Audit Databases The Cloudera Navigator auditing component supports the following databases for storing audit events: MySQL - 5.0, 5.1, 5.5, and 5.6 Oracle 11gR2 PostgreSQL - 8.4, 9.1, and 9.2 Supported Service Versions and Audited Operations HDFS - Minimum supported version: CDH The captured operations are: Operations that access or modify a file's or directory's data or metadata Operations denied due to lack of privileges HBase - Minimum supported version: CDH Note: In CDH versions less than 4.2.0, for grant and revoke operations, the operation in log events is ADMIN In simple authentication mode, if the HBase Secure RPC Engine property is false (the default), the username in log events is UNKNOWN. To see a meaningful user name: 1. Click the HBase service. 2. Click the Configuration tab. 3. Select Service-wide > Security. 4. Set the HBase Secure RPC Engine property to true. 5. Save the change and restart the service. Hive - Minimum supported versions: CDH 4.2.0, CDH for operations denied due to lack of privileges. The captured operations are: Operations (except grant, revoke, and metadata access only) sent to HiveServer2 Operations denied due to lack of privileges Cloudera Navigator Installation and User Guide 9

10 Cloudera Navigator Requirements Note: Actions taken against Hive via the Hive CLI are not audited. Therefore if you have enabled auditing you should disable the Hive CLI to prevent actions against Hive that are not audited. In simple authentication mode, the username in log events is the username passed in the HiveServer2 connect command. If you do not pass a username in the connect command, the username is log events is anonymous. Hue - Minimum supported version: CDH The captured operations are: Operations (except grant, revoke, and metadata access only) sent to Beeswax Server Note: You do not directly configure the Hue service for auditing. Instead, when you configure the Hive service for auditing, operations sent to the Hive service through Beeswax appear in the Hue service audit log. Cloudera Impala - Minimum supported version: Cloudera Impala The captured operations are: Queries denied due to lack of privileges Queries that pass analysis Sentry The captured operations are: Operations sent to the HiveServer2 and the Hive Metastore Server roles Add and delete roles, assign roles to groups and remove roles from groups, create and delete privileges, grant and revoke privileges Operations denied due to lack of privileges Note: You do not directly configure the Sentry service for auditing. Instead, when you configure the Hive service for auditing, grant, revoke, and metadata operations sent to the Hive Metastore Server appear in the Hive service audit log. Cloudera Navigator Metadata Component This section describes the product versions, components, and browsers supported by the Cloudera Navigator metadata component. Supported Product Versions CDH and higher for all components except Pig. For Pig, CDH and higher. The supported components are: HDFS Hive MapReduce Oozie Pig Sqoop 1 - all Cloudera connectors are supported. YARN 10 Cloudera Navigator Installation and User Guide

11 Cloudera Navigator Requirements Supported Browsers for the Metadata UI Firefox 11 or later Google Chrome Safari 5 or later Cloudera Navigator Installation and User Guide 11

12

13 Installing and Upgrading Cloudera Navigator Installing and Upgrading Cloudera Navigator Required Role: Cloudera Navigator is implemented as two roles in the Cloudera Management Service: Navigator Audit Server and Navigator Metadata Server. You can add Cloudera Navigator roles while installing Cloudera Manager for the first time, into an existing Cloudera Manager installation, or while upgrading an existing Cloudera Manager installation. Configuring a Database for the Cloudera Navigator Auditing Component When you install Cloudera Navigator you choose the database to store audit events. You can choose either an embedded PostgreSQL database or an external database. For information on supported databases, see Supported Audit Databases on page 9. For information on setting up an external database, see Installing and Configuring Databases. Adding Cloudera Navigator in a New Cloudera Manager Installation 1. Install Cloudera Manager following the instructions in Installing Cloudera Manager. 2. In the first page of the Cloudera Manager installation wizard, choose one of the license options that support Cloudera Navigator: Cloudera Enterprise Data Hub Edition Trial Cloudera Enterprise Flex Edition Data Hub Edition 3. If you have elected Cloudera Enterprise, install a license: a. Click Upload License. b. Click the document icon to the left of the Select a License File text field. c. Navigate to the location of your license file, click the file, and click Open. d. Click Upload. Click Continue to proceed with the installation. 4. In the first page of the Add Services procedure, check the Include Cloudera Navigator checkbox. 5. If you have chosen to use an external database, provide the Cloudera Navigator Audit Server database properties in the Database Setup page. Adding Cloudera Navigator in an Existing Cloudera Manager Installation 1. Add and start the Cloudera Navigator roles: Adding and Starting the Navigator Audit Server Role on page 15 Adding and Starting the Navigator Metadata Server Role on page 28 Adding Cloudera Navigator While Upgrading an Existing Cloudera Manager Installation 1. Upgrade Cloudera Manager following the instructions in Upgrading Cloudera Manager. 2. Add and start the Cloudera Navigator roles as described in Adding Cloudera Navigator in an Existing Cloudera Manager Installation on page 13. Cloudera Navigator Installation and User Guide 13

14 Installing and Upgrading Cloudera Navigator Deleting Cloudera Navigator Roles 1. Do one of the following: Select Clusters > Cloudera Management Service > Cloudera Management Service. On the Status tab of the Home page, in Cloudera Management Service table, click the Cloudera Management Service link. 2. Click the Instances tab. 3. Check the checkboxes next to the Navigator Audit Server and Navigator Metadata Server roles. 4. If the role instance is running, select Actions for Selected > Stop and click Stop to confirm the action. 5. Select Actions for Selected > Delete. Click Delete to confirm the deletion. Upgrading Cloudera Navigator To upgrade Cloudera Navigator, upgrade Cloudera Manager following the instructions in Upgrading Cloudera Manager. Important: Cloudera does not provide an upgrade path from the Navigator Metadata component in Cloudera Navigator 1.2 to the Cloudera Navigator 2.0 release. If you are upgrading from Cloudera Navigator 1.2, you must perform a clean install of Cloudera Navigator 2.0. Therefore, if you have Cloudera Navigator roles from a previous beta release: 1. Delete the Navigator roles. 2. Remove the contents of the Navigator Metadata Server storage directory. 3. Add the Navigator roles according to the process described in Adding and Starting the Navigator Audit Server Role on page 15 and Adding and Starting the Navigator Metadata Server Role on page Clear the cache of any browser that had previously used the 1.2 release of the Navigator Metadata component. Otherwise, you may observe errors in the Navigator Metadata UI. 14 Cloudera Navigator Installation and User Guide

15 Cloudera Navigator Auditing Component Cloudera Navigator Auditing Component The Cloudera Navigator auditing component provides data auditing and access features. The architecture of the Cloudera Navigator auditing component is illustrated below. The Cloudera Navigator auditing component is implemented as an add-on to Cloudera Manager; all Cloudera Navigator auditing component functions (configuration and audit log review) are accessed through the Cloudera Manager Admin Console. When the Cloudera Navigator auditing component is configured, plug-ins that enable collection of audit events are added to the HDFS, HBase, and Hive (that is, the HiveServer2 and Beeswax servers) services. The plug-ins write the audit events to an audit log on the local filesystem. Cloudera Impala records audit events directly in an audit log file. The Cloudera Manager Agent monitors the audit log files and sends these events to the Navigator Audit Server. The Cloudera Manager Agent retries any event that it fails to transmit. As there is no in-memory transient buffer involved, once the audit events are written to the audit log file, they are guaranteed to be delivered (as long as filesystem is available). The Cloudera Manager Agent keeps track of current audit event offset in the audit log that it has successfully transmitted, so on any crash/restart it picks up the audit event from the last successfully sent position and resumes. Audit logs are rotated and the Cloudera Manager Agent follows the rotation of the log. The Agent also takes care of purging old audit logs once they have been successfully transmitted to the Navigator Audit Server. If a plug-in fails to write audit event to audit log file, it can either drop the event or shut down the process in which they are running (depending on the configured queue policy). The Navigator Audit Server stores events in the Navigator Audit DB. Cloudera Navigator Audit Server Describes how to add and configure the Navigator Audit Server role. Required Role: Adding and Starting the Navigator Audit Server Role Before adding the Navigator Audit Server role, configure the database where audit events are stored. Cloudera Navigator Installation and User Guide 15

16 Cloudera Navigator Auditing Component 1. Do one of the following: Select Clusters > Cloudera Management Service > Cloudera Management Service. On the Status tab of the Home page, in Cloudera Management Service table, click the Cloudera Management Service link. 2. Click the Instances tab. 3. Click the Add Role Instances button. The Customize Role Assignments page displays. 4. Assign the Navigator role to a host. a. Customize the assignment of role instances to hosts. The wizard evaluates the hardware configurations of the hosts to determine the best hosts for each role. The wizard assigns all worker roles to the same set of hosts to which the HDFS DataNode role is assigned. These assignments are typically acceptable, but you can reassign role instances to hosts of your choosing, if desired. Click a field below a role to display a dialog containing a pageable list of hosts. If you click a field containing multiple hosts, you can also select All Hosts to assign the role to all hosts or Custom to display the pageable hosts dialog. The following shortcuts for specifying hostname patterns are supported: Range of hostnames (without the domain portion) Range Definition [1-4] host[1-3].company.com host[07-10].company.com Matching Hosts , , , host1.company.com, host2.company.com, host3.company.com host07.company.com, host08.company.com, host09.company.com, host10.company.com IP addresses Rack name Click the View By Host button for an overview of the role assignment by hostname ranges. 5. When you are satisfied with the assignments, click Continue. The Database Setup page displays. 6. Configure database settings: a. Choose the database type: Leave the default setting of Use Embedded Database to have Cloudera Manager create and configure required databases. Make a note of the auto-generated passwords. Select Use Custom Databases to specify external databases. 1. Enter the database host, database type, database name, username, and password for the database that you created when you set up the database. b. Click Test Connection to confirm that Cloudera Manager can communicate with the database using the information you have supplied. If the test succeeds in all cases, click Continue; otherwise check and correct the information you have provided for the database and then try the test again. (For some servers, if you are using the embedded database, you will see a message saying the database will be created at a later step in the installation process.) The Review Changes page displays. 7. Click Continue. The Review Changes page displays with no configuration changes. 8. Click Finish. The Instances page displays. 9. Check the checkbox next to the Navigator Audit Server role. 10. Select Actions for Selected > Start. Click Start to confirm the action. 16 Cloudera Navigator Installation and User Guide

17 Configuring the Navigator Audit Server Log Directory 1. Do one of the following: Select Clusters > Cloudera Management Service > Cloudera Management Service. On the Status tab of the Home page, in Cloudera Management Service table, click the Cloudera Management Service link. 2. Click the Configuration tab. 3. Click the Navigator Audit Server Default Group category. 4. Set the Navigator Audit Server Log Directory property. 5. Click Save Changes. 6. Click the Instances tab. 7. Check the checkbox next to the Navigator Audit Server role. 8. Select Actions for Selected > Restart. Configuring the Navigator Audit Server Data Expiration Period You can configure the number of hours of audit events to keep in the Navigator Audit Server database as follows: 1. Required Roles: 2. Set the Navigator Audit Server Data Expiration Period property. 3. Click Save Changes. 4. Click the Instances tab. 5. Check the checkbox next to the Navigator Audit Server role. 6. Select Actions for Selected > Restart. Configuring the Audit Server to Mask Personally Identifiable Information This feature is available in Cloudera Navigator or later. Personally identifiable information (PII) is information that can be used on its own or with other information to identify or locate a single person, or to identify an individual in context. The PII masking feature allows you to specify credit card number patterns (from major credit issuers) that are masked in audit events, in the properties of entities displayed in lineage diagrams, and in information retrieved from the Audit Server database and the Metadata Server persistent storage. Note: Cloudera Navigator Auditing Component Masking Social Security numbers is not supported in this release. Masking is not applied to audit events and lineage entities that existed before the mask was enabled. 1. Expand the Navigator Audit Server Default Group category. 2. Click the Advanced category. 3. Configure the PII Masking Regular Expression property with a regular expression that matches the credit card number formats to be masked. The default expression is: (4[0-9]{12}(?:[0-9]{3})?) (5[1-5][0-9]{14}) (3[47][0-9]{13}) (3(?:0[0-5] [68][0-9])[0-9]{11}) (6(?:011 5[0-9]{2})[0-9]{12}) ((?: \\d{3})\\d{11}) which is constructed from the following subexpressions: Visa - (4[0-9]{12}(?:[0-9]{3})?) MasterCard - (5[1-5][0-9]{14}) American Express - (3[47][0-9]{13}) Diners Club - (3(?:0[0-5] [68][0-9])[0-9]{11}) Cloudera Navigator Installation and User Guide 17

18 Cloudera Navigator Auditing Component Discover - (6(?:011 5[0-9]{2})[0-9]{12}) JCB - ((?: \\d{3})\\d{11}) If the property is left blank, PII information is not masked. 4. Click Save Changes. Audit Log Properties Describes auditing log properties and how to configure the log properties. The following properties apply to the audit log file: Audit Log Directory - The directory in which audit event log files are written. By default, this property is not set if Cloudera Navigator is not installed. Note: If the value of this property is changed, and service is restarted, then the Cloudera Manager Agent will start monitoring the new log directory for audit events. In this case it is possible that not all events are published from the old audit log directory. To avoid loss of audit events, when this property is changed, perform the following steps: 1. Stop the service. 2. Copy audit log files and (for Impala only) the impalad_audit_wal file from the old audit log directory to the new audit log directory. This need to be done on all the nodes where Impala daemons are running. 3. Start the service. Maximum Audit Log File Size - The maximum size of the audit event log file before a new file is created. The unit of the file size is service dependent: HDFS, HBase, Hive - MiB Impala - lines (queries) Number of Audit Logs to Retain - Maximum number of rolled over audit logs to retain. The logs will not be deleted if they contain audit events that have not yet been propagated to Audit Server. Configuring Audit Logs 1. Click a supported service. 2. Click the Configuration tab. 3. Configure the log properties in the following categories: Impala - Impala Daemon Default Group > Logs HBase, HDFS, and Hive - Service-Wide > Logs 4. Edit the audit log properties. 5. Click Save Changes. 6. Restart the service. Service Auditing Properties Describes service auditing properties and how to configure the properties. Each service that supports auditing configuration has the following properties: Enable Collection - Controls whether the Cloudera Manager Agent tracks a service's audit log file. A validation check is performed for all life cycle actions (stop/start/restart). If the Enable Collection flag is selected and 18 Cloudera Navigator Installation and User Guide

19 Cloudera Navigator Auditing Component the Audit Log Directory property is not set, the validator displays a message that says that the Audit Log Directory property must be set to enable auditing. Event Filter - A set of rules that capture properties of auditable events and actions to be performed when an event matches those properties. Event Tracker - A set of rules for tracking and coalescing events. This feature is used to define equivalency between different audit events. When events match, according to a set of configurable parameters, only one entry in the audit list is generated for all the matching events. Queue Policy - The action to take when the audit event queue is full. The options are Drop or Shutdown. When a queue is full and the queue policy of the service is Shutdown, before shutting down the service, N audits will be discarded, where N is the size of the Cloudera Navigator Audit Server queue. Note: If the queue policy is Shutdown, the Impala service is shut down only if Impala is unable to write to the audit log file. It is possible that an event may not appear in the audit event log due to an error in transfer to the Cloudera Manager Agent or database. In such cases Impala will not shut down and will keep writing to the log file. When the transfer problem is fixed the events will be transferred to the database. The Event Filter and Event Tracker rules for filtering and coalescing events are expressed as JSON objects. For information on the structure of the objects, see the description on the configuration page within the Cloudera Manager Admin Console. The default event filter discards events generated by the internal Cloudera and Hadoop users (cloudera-scm, hdfs, hbase, hive, mapred, solr, and dr.who) and that affect files in the /tmp directory. Configuring Service Auditing Properties 1. Click a service that supports auditing. 2. Click the Configuration tab. 3. Click the Cloudera Navigator category. The Service-Wide category displays. 4. Edit the properties. 5. Click Save Changes. 6. Restart the service. Configuring Impala Event Logging To control whether the Impala daemon logs events to the audit log: 1. Click the Impala service. 2. Click the Configuration tab. 3. Expand the Impala Daemon Default Group > Logs category. 4. Edit the Enable Impala Audit Event Generation checkbox setting. 5. Click Save Changes. 6. Restart the service. Audit Logging to Syslog The Audit Server logs all audit records into a Log4j logger called auditstream. The log messages are logged at the TRACE level, with the attributes of the audit records. By default, the auditstream logger is inactive because the logger level is set to FATAL. It is also connected to a NullAppender, and does not forward to other appenders (additivity set to false). To record the audit stream, configure the auditstream logger with the desired appender. For example, the standard SyslogAppender allows you to send the audit records to a remote syslog. Cloudera Navigator Installation and User Guide 19

20 Cloudera Navigator Auditing Component The Log4j SyslogAppender supports only UDP. An example syslog configuration would be: $ModLoad imudp $UDPServerRun 514 # Accept everything (even DEBUG messages) local2.* /my/audit/trail.log It is also possible to attach other appenders to the auditstream to provide other integration behaviors. You can audit events to syslog in two formats: JSON and RSA EnVision. To configure audit logging to syslog, do the following: 1. Do one of the following: Select Clusters > Cloudera Management Service > Cloudera Management Service. On the Status tab of the Home page, in Cloudera Management Service table, click the Cloudera Management Service link. 2. Click the Configuration tab. 3. Search for Navigator Audit Server Logging Advanced Configuration Snippet. 4. Click the Value field and depending on the format type, enter: log4j.logger.auditstream = TRACE,SYSLOG log4j.appender.syslog = org.apache.log4j.net.syslogappender log4j.appender.syslog.sysloghost = hostname log4j.appender.syslog.facility = Local2 log4j.appender.syslog.facilityprinting = true To configure the specific stream type, enter: Format JSON RSA EnVision Properties log4j.additivity.auditstream = false log4j.additivity.auditstreamenvision = false 5. Click Save Changes to commit the changes. Example Log Messages Format JSON RSA EnVision Log Message Example Jul 23 11:05:15 hostname local2: {"type":"hdfs","allowed":"true","time":" ", "service":"hdfs-1", "user":"root","ip":" ","op":"mkdirs","src":"/audit/root","perms":"rwxr-xr-x"} Cloudera Navigator 1 type="hive",allowed="false",time=" ", service="hive-1",user="systest",impersonator="",ip="/ ",op="query", optext="select count(*) from sample_07",db="default",table="sample_07",path="/user/hive/warehouse/sample_07",objtype="table" If a particular field is not applicable for that audit event, it is omitted from the message. Audit Events An audit event is an event that describes an action that has been taken for a service, role, or host. In Cloudera Manager, audit event logs display: Service, role, and host life cycle events recorded by Cloudera Management Service roles. For further information, see Audit Events in the Cloudera Manager Monitoring and Diagnostics Guide Service access events recorded by the Cloudera Navigator Audit Server. Such audit events are described below. 20 Cloudera Navigator Installation and User Guide

21 Audit Event Properties The following properties can appear in an audit event entry: Date - Date and time the action was performed. Command - The action performed. Source - The object affected by the service action. User - The name of the user that performed the action. Impersonator - If the action was requested by another service, the name of the user that invoked the service action on behalf of the user. The Impersonator field will always show when Sentry is not enabled. The Impersonator field will show for other services than Hive when Sentry is enabled. IP Address - The IP address of the host where the service action occurred. Service - The name of the service that performed the service action. Role - The name of the role that performed the service action. Viewing Audit Events Cloudera Navigator Auditing Component You can view audit events for all services or for a specific service. To view audit events, follow the appropriate procedure: Object All Services Service Procedure 1. Click Audits in the Cloudera Manager Admin Console top navigation bar. 1. In the Cloudera Manager Admin Console, click a service that supports auditing. 2. Click the Audits tab on the service navigation bar. Audit event entries are ordered with the most recent at the top. Events that represent denied access are labeled Denied, red text, and a pink background. Filtering Audit Events You filter on generated audit events in the audit UI by selecting a time range and adding filters. You can use the Time Range Selector or a duration link ( ) to set the time range. (See Time Line in Cloudera Manager Monitoring and Diagnostics Guide for details). When you select the time range, the log displays all events in that range. Note that the time it takes to perform a search will typically increase for a longer time range, as the number of events to be searched will be larger. Adding a Filter Click the icon that displays next to a property when you hover in one of the event entries. A filter containing the property, operator, and its value is added to the list of filters at the left and Cloudera Manager redisplays all events that match the filter. Click the Add a filter link. A filter control is added to the list of filters. Cloudera Navigator Installation and User Guide 21

22 Cloudera Navigator Auditing Component 1. Choose a property in the drop-down list. You can search by properties such as Username, Service, Command, or Role. The properties vary depending on the service or role. 2. If the property allows it, choose an operator in the operator drop-down list. 3. Type a property value in the value text field. For some properties, where the list of values is finite and known, you can start typing and then select from a list of potential matches. To match a substring, use the like operator and specify % around the string. For example, to see all the audit events for files created in the folder /user/joe/out specify Source like %/user/joe/out%. 4. Click Search. The log displays all events that match the filter criteria. 5. Click to add more filters and repeat steps 1 through 4. Removing a Filter 1. Click the at the right of the filter. The filter is removed. 2. Click Search. The log displays all events that match the filter criteria. Downloading Audit Events You can download audit events in the Audit UI or using the Audit API. An audit event contains the following fields: service, username, command, ipaddress, resource, allowed, timestamp, operationtext. The structure of the resource field depends on the type of the service: HDFS - A file path. Hive, Hue, Impala, and Sentry - database:tablename HBase - table family:qualifier For Hive, Hue, Impala, and Sentry events, operationtext contains the operation string. Downloading Audit Events Using the Audit UI 1. Display the audit log. 2. Specify desired filters and time range. 3. Click the Download CSV button. A file named history.csv is downloaded. HDFS Audit Log Example service,username,command,ipaddress,resource,allowed,timestamp,operationtext, HDFS,cloudera,setPermission, ,/user/hive,false," T00:59:34.430Z", HDFS,cloudera,getfileinfo, ,/user/cloudera,true," T00:59:22.667Z", HDFS,cloudera,getfileinfo, ,/,true," T00:59:22.658Z", In this example, the first event access was denied, and therefore the allowed field has the value false. Hive and Sentry Example - via downloaded CSV file The following records list Hive operations to create and load a table and Sentry operations to created roles and grant privileges: service,username,command,ipaddress,resource,allowed,timestamp,operationtext Hive,admin,LOAD, ,default:sample_08,true," T20:21:03.510Z","LOAD DATA INPATH '/user/admin/sample_08' OVERWRITE INTO TABLE sample_08" Hive,admin,LOAD, ,:,true," T20:21:03.509Z","LOAD DATA INPATH '/user/admin/sample_08' OVERWRITE INTO TABLE sample_08" Hive,admin,CREATETABLE, ,default:sample_08,true," T20:21:01.899Z","CREATE TABLE `sample_08` ( `code` string, `description` string, `total_emp` int, `salary` int ) ROW FORMAT DELIMITED FIELDS TERMINATED BY ' ' 22 Cloudera Navigator Installation and User Guide

23 Cloudera Navigator Auditing Component STORED AS TextFile" Hive,hive,GRANT_ROLE, ,:,true," T20:19:26.718Z","GRANT ROLE default_admin TO GROUP sentrydefaultadmin" Hive,hive,GRANT_PRIVILEGE, ,default:,true," T20:19:26.149Z","GRANT ALL ON DATABASE default TO ROLE default_admin" Hive,hive,CREATEROLE, ,:,true," T20:19:25.761Z","CREATE ROLE default_admin" Hive,hive,GRANT_ROLE, ,:,true," T20:19:20.515Z","GRANT ROLE global_admin TO GROUP sentryadmin" Hive,hive,GRANT_ROLE, ,:,true," T20:19:20.063Z","GRANT ROLE global_admin TO GROUP hive" Hive,hive,GRANT_PRIVILEGE, ,server1:,true," T20:19:19.382Z","GRANT ALL ON SERVER server1 TO ROLE global_admin" Hive,hive,CREATEROLE, ,:,true," T20:19:18.281Z","CREATE ROLE global_admin" Downloading Audit Events Using the Audit API You can filter and download audit events using the Cloudera Manager audit API. See Audit API. Hive and Sentry Example - via audit API To download the same audits events using the API, issue the request which returns the following JSON items: { "items" : [ { "timestamp" : " T20:21:03.510Z", "service" : "Hive", "username" : "admin", "ipaddress" : " ", "command" : "LOAD", "resource" : "default:sample_08", "operationtext" : "LOAD DATA INPATH\n '/user/admin/sample_08' OVERWRITE INTO TABLE sample_08", "allowed" : true { "timestamp" : " T20:21:03.509Z", "service" : "Hive", "username" : "admin", "ipaddress" : " ", "command" : "LOAD", "resource" : ":", "operationtext" : "LOAD DATA INPATH\n '/user/admin/sample_08' OVERWRITE INTO TABLE sample_08", "allowed" : true { "timestamp" : " T20:21:01.899Z", "service" : "Hive", "username" : "admin", "ipaddress" : " ", "command" : "CREATETABLE", "resource" : "default:sample_08", "operationtext" : "CREATE TABLE `sample_08` (\n `code` string,\n `description` string,\n `total_emp` int,\n `salary` int )\nrow FORMAT DELIMITED\n FIELDS TERMINATED BY '\t'\nstored AS TextFile", "allowed" : true { "timestamp" : " T20:19:26.718Z", "service" : "Hive", "username" : "hive", "ipaddress" : " ", "command" : "GRANT_ROLE", "resource" : ":", "operationtext" : "GRANT ROLE default_admin TO GROUP sentrydefaultadmin", "allowed" : true { "timestamp" : " T20:19:26.149Z", "service" : "Hive", "username" : "hive", Cloudera Navigator Installation and User Guide 23

24 Cloudera Navigator Auditing Component } "ipaddress" : " ", "command" : "GRANT_PRIVILEGE", "resource" : "default:", "operationtext" : "GRANT ALL ON DATABASE default TO ROLE default_admin", "allowed" : true { "timestamp" : " T20:19:25.761Z", "service" : "Hive", "username" : "hive", "ipaddress" : " ", "command" : "CREATEROLE", "resource" : ":", "operationtext" : "CREATE ROLE default_admin", "allowed" : true { "timestamp" : " T20:19:20.515Z", "service" : "Hive", "username" : "hive", "ipaddress" : " ", "command" : "GRANT_ROLE", "resource" : ":", "operationtext" : "GRANT ROLE global_admin TO GROUP sentryadmin", "allowed" : true { "timestamp" : " T20:19:20.063Z", "service" : "Hive", "username" : "hive", "ipaddress" : " ", "command" : "GRANT_ROLE", "resource" : ":", "operationtext" : "GRANT ROLE global_admin TO GROUP hive", "allowed" : true { "timestamp" : " T20:19:19.382Z", "service" : "Hive", "username" : "hive", "ipaddress" : " ", "command" : "GRANT_PRIVILEGE", "resource" : "server1:", "operationtext" : "GRANT ALL ON SERVER server1 TO ROLE global_admin", "allowed" : true { "timestamp" : " T20:19:18.281Z", "service" : "Hive", "username" : "hive", "ipaddress" : " ", "command" : "CREATEROLE", "resource" : ":", "operationtext" : "CREATE ROLE global_admin", "allowed" : true } ] Downloading HDFS Directory Access Permission Reports For each HDFS service you can download a report that details the HDFS directories a group has permission to access. 1. In the Cloudera Manager Admin Console, navigate to the Reports page: (Cloudera Manager 4) Click Reports. (Cloudera Manager 5) Click Clusters > ClusterName > Other > Reports. 2. In the Directory Access by Group row, click CSV or XLS. The Download User Access Report pop-up displays. a. In the pop-up, type a group and directory. 24 Cloudera Navigator Installation and User Guide

25 Cloudera Navigator Auditing Component b. Click Download. A report of the selected type will be generated containing the following information path, owner, permissions, and size for each directory contained in the specified directory that the specified group has access to. Cloudera Navigator Installation and User Guide 25

26

27 Cloudera Navigator Metadata Component The Cloudera Navigator metadata component provides data discovery and data lineage management functions. The architecture of the Cloudera Navigator metadata component is illustrated below. The Navigator Metadata Server performs five main functions: Obtains connection information about the services whose data it manages from the Cloudera Manager Server Extracts entity metadata from the services at periodic intervals Indexes the metadata Implements a graphical UI for searching the metadata and displaying lineage relationships between entities Implements a REST API for retrieving metadata Cloudera Navigator Metadata Server Describes how the Cloudera Navigator Metadata Server extracts metadata from the entities managed by Cloudera Manager, and how to add and configure the Navigator Metadata Server role. Required Role: About Metadata Extraction The Navigator Metadata Server extracts metadata from the following resource types from the listed servers: HDFS - Extracts HDFS metadata at the next scheduled extraction run after an HDFS checkpoint. However, if you have High Availability enabled, metadata is extracted as soon as it is written to the JournalNodes. Hive - Extracts database and table metadata from the Hive Metastore Server. MapReduce - Extracts job metadata from the JobTracker. The default setting in Cloudera Manager retains a maximum of five jobs, which means if you run more than five jobs between Navigator extractions, the Navigator Metadata Server would extract the five most recent jobs. Oozie - Extracts Oozie workflows from the Oozie Server. Cloudera Navigator Installation and User Guide 27

28 Pig - Extracts Pig script runs from the JobTracker or Job History Server. Sqoop 1 - Extracts database and table metadata from the Hive Metastore Server. YARN - Extracts job metadata from the Job History Server. If an entity is created at time t0 in the system, that entity will be extracted and linked in Navigator after the extraction poll period (default 10 minutes) plus a service-specific interval as follows: HDFS: t0 + extraction poll period + HDFS checkpoint interval (default 1 hour) HDFS + HA: t0 + extraction poll period Hive: t0 + extraction poll period + Hive maximum wait time (default 60 minutes) Adding and Starting the Navigator Metadata Server Role 1. Do one of the following: Select Clusters > Cloudera Management Service > Cloudera Management Service. On the Status tab of the Home page, in Cloudera Management Service table, click the Cloudera Management Service link. 2. Click the Instances tab. 3. Click the Add Role Instances button. The Customize Role Assignments page displays. 4. Assign the Navigator role to a host. a. Customize the assignment of role instances to hosts. The wizard evaluates the hardware configurations of the hosts to determine the best hosts for each role. The wizard assigns all worker roles to the same set of hosts to which the HDFS DataNode role is assigned. These assignments are typically acceptable, but you can reassign role instances to hosts of your choosing, if desired. Click a field below a role to display a dialog containing a pageable list of hosts. If you click a field containing multiple hosts, you can also select All Hosts to assign the role to all hosts or Custom to display the pageable hosts dialog. The following shortcuts for specifying hostname patterns are supported: Range of hostnames (without the domain portion) Range Definition [1-4] host[1-3].company.com host[07-10].company.com Matching Hosts , , , host1.company.com, host2.company.com, host3.company.com host07.company.com, host08.company.com, host09.company.com, host10.company.com IP addresses Rack name Click the View By Host button for an overview of the role assignment by hostname ranges. 5. Click Finish. The Instances page displays. 6. Check the checkbox next to the Navigator Metadata Server role. 7. Select Actions for Selected > Start. Click Start to confirm the action. Configuring the Navigator Metadata Server Storage Directory Describes how to configure where the Navigator Metadata Server stores extracted data. The default is /var/lib/cloudera-scm-navigator. 1. Do one of the following: Select Clusters > Cloudera Management Service > Cloudera Management Service. 28 Cloudera Navigator Installation and User Guide

29 On the Status tab of the Home page, in Cloudera Management Service table, click the Cloudera Management Service link. 2. Click the Configuration tab. 3. Click the Navigator Metadata Server Default Group. 4. Specify the directory in the Navigator Metadata Server Storage Dir property. 5. Click Save Changes. 6. Click the Instances tab. 7. Check the checkbox next to the Navigator Metadata Server role. 8. Select Actions for Selected > Restart. Configuring the Navigator Metadata Server Port Describes how to configure the port on which the Navigator Metadata UI is accessed. The default port is Do one of the following: Select Clusters > Cloudera Management Service > Cloudera Management Service. On the Status tab of the Home page, in Cloudera Management Service table, click the Cloudera Management Service link. 2. Click the Configuration tab. 3. Select Navigator Metadata Server Default Group > Ports and Addresses. 4. Specify the port in the Navigator Metadata Server Port property. 5. Click Save Changes. 6. Click the Instances tab. 7. Check the checkbox next to the Navigator Metadata Server role. 8. Select Actions for Selected > Restart. Configuring the Metadata Server to Mask Personally Identifiable Information This feature is available in Cloudera Navigator or later. Navigator Metadata Server Sizing and Performance Recommendations Two activities determine Navigator Metadata Server resource requirements: Extracting metadata from the cluster and creating relationships Querying The Navigator Metadata Server uses Solr to store, index, and query metadata. Indexing happen during extraction. Querying is fast and efficient because the data is indexed. Memory and CPU requirements are based on amount of data that is stored and indexed. With 6 GB of RAM and 8-10 cores Solr can process 6 million entities in minutes or 80 million entities in 8 to 9 hours. Any less RAM than 6GB and will result in excessive garbage collection and possibly out-of-memory exceptions. For large clusters, Cloudera advises at least 8 GB of RAM and 8 cores. The Solr instance runs in process with Navigator, so the Java heap for the Navigator Metadata Server should be set according to the size of cluster. By default, during the Cloudera Manager first run installation wizard the Navigator Audit Server and Navigator Metadata Server are assigned to the same host as the Cloudera Management Service monitoring roles. This configuration works for a small cluster, but should be updated before the cluster grows. You can either change the configuration at installation time or move the Navigator Metadata Server if necessary. Moving a Navigator Metadata Server Role Cloudera Navigator Metadata Component 1. Stop the Navigator Metadata Server role, delete it from existing host, and add it to a new host. 2. If the Solr data path is not on NFS/SAN, move the data to the same path on the new host. 3. Start the Navigator Metadata Server role. Cloudera Navigator Installation and User Guide 29

30 Enabling Hive Metadata Extraction in a Secure Cluster The Navigator Metadata Server uses the hue user to connect to the Hive Metastore. The hue user is able to connect to the Hive Metastore by default. However, if the Hive service Hive Metastore Access Control and Proxy User Groups Override property and/or the HDFS service Hive Proxy User Groups property have been changed from their default values to settings that prevent the hue user from connecting to the Hive Metastore, Navigator Metadata Server will be unable to extract metadata from Hive. If this is the case, modify the Hive service Hive Metastore Access Control and Proxy User Groups Override property and/or the HDFS service Hive Proxy User Groups property so that the hue user can connect as follows: 1. Go to the Hive or HDFS service. 2. Click the Configuration tab. 3. Expand the Service-Wide > Proxy category. 4. In the Hive service Hive Metastore Access Control and Proxy User Groups Override field or the HDFS service Hive Proxy User Groups field, click the Value column, and click to add a new row. 5. Type hue. 6. Click Save Changes to commit the changes. 7. Restart the service. Metadata The Cloudera Navigator Metadata component manages metadata about the entities in a Hadoop cluster and relationships between the entities. The Cloudera Navigator metadata schema defines the types of metadata that are available for each entity type it supports. The types of metadata defined by the Cloudera Navigator Metadata component include: the name of an entity, the service that manages or uses the entity, type, path to the entity, date and time of creation, access, and modification, size, owner, purpose, and relationships parent-child, data flow, and instance of between entities. For example, the following shows the property sheet of a file entity: There are two classes of metadata: technical metadata - metadata defined when entities are extracted. You cannot modify technical metadata. business metadata - metadata added to extracted entities. You can add and modify business metadata before or after entities are extracted. 30 Cloudera Navigator Installation and User Guide

31 Metadata Indexing After metadata is extracted it is indexed and made available for searching by an embedded Solr engine. The Solr schema indexes two types of metadata: entity properties and relationships between entities. You can search entity metadata using the Navigator Metadata UI. Relationship metadata is implicitly visible in lineage diagrams and explicitly available in a lineage file. Search Syntax Search in the Cloudera Navigator Metadata component is implemented by an embedded Solr engine that supports the syntax described in LuceneQParserPlugin. You construct search strings by specifying the value of a default property, property name-value pairs, or user-defined name-value pairs using the syntax: Property name-value pairs - propertyname:value, where propertyname is one of the properties listed in Search Properties on page 31. value is a single value or range of values specified as [value1 TO value2]. In a value, * is a wildcard. In property name-value pairs you must escape special characters :, /, and * with the backslash character \. For example, filesystempath:\/user\/admin. User-defined name-value pairs - up_propertyname:value. To construct complex strings, join multiple property-value pairs using the or and and operators. Example Search Strings Cloudera Navigator Metadata Component Filesystem path /user/admin - filesystempath:\/user\/admin Descriptions that start with the string "Banking" - description:banking* Sources of type MapReduce or Hive - sourcetype:mapreduce or sourcetype:hive Directories owned by hdfs in the path /user/hdfs/input - owner:hdfs and type:directory and filesystempath:\/user\/hdfs\/input Job started between 20:00 to 21:00 UTC - started:[ t20:00:00.000z TO T21:00:00.000Z] User-defined key-value project-customer1 - up_project:customer1 Note: When viewing MapReduce jobs in the Cloudera Manager Activities page, the string that appear in a job's Name column equates to the originalname property. Therefore, to specify a MapReduce job's name in a search, use the following string: (restype:mapreduce) and (originalname:jobname), where jobname is the value in the job's Name column. Search Properties A reference for the search schema properties. Default Properties The following properties can be searched simply by specifying a property value: type, filesystempath, inputs, jobid, mapper, mimetype, name, originalname, outputs, owner, principal, reducer, and tags. Common Properties Name description group name Type text caseinsensitivetext ngramedtext Description Description of the entity. The group to which the owner of the entity belongs. The overridden name of the entity. If the name has not been overridden, this value is empty. Names cannot contain spaces. Cloudera Navigator Installation and User Guide 31