SECURITY IMPLEMENTATION IN HADOOP. By Narsimha Chary( ) Siddalinga K M( ) Rahman( )

Transcription

1 SECURITY IMPLEMENTATION IN HADOOP By Narsimha Chary( ) Siddalinga K M( ) Rahman( )

2 AGENDA What is security? Security in Distributed File Systems? Current level of security in Hadoop! Security features to be incorporated in HDFS to make it robust.

3 What is Security? Protection of information and property from theft, corruption, or natural disaster, while allowing the information and property to remain accessible and productive to its intended users Processes and mechanisms by which sensitive and valuable information and services are protected from publication, tampering or collapse by unauthorized activities or untrustworthy individuals and unplanned events respectively.

4 Security in Distributed File System Private Clouds are more or less secure as they are deployed within the locales of the organization and is also secured by firewall Public Clouds are prone to all sorts of danger as you know where your data is residing at any instance

5 Current level of security in Hadoop Current version of Hadoop has very basic rudimentary implementation of security which is advisory access control mechanism. Hadoop doesn t strongly authenticate the client, it simply asks the underlying Unix system by executing `whoami` command Anyone can communicate directly with a Datanode (without the need for communicating with the Namenode)and ask for blocks if you have the block location details (This was experimented at the recent Cloudera's Hadoop Hackathon)

6 With current provisions. Hadoop cluster may be prone to following attacks Unauthorized clients can impersonate authorized users and access the cluster. One can get the blocks directly from the Datanodes by bypassing the Namenode. Eavesdropping/sniffing of data packets being sent by Datanodes to client. (Can this be resolved by using secure socket over a regular socket? YES, YES! But

7 Proposed Solutions Authentication of users/clients accessing the Hadoop cluster using Kerberos Protocol Authorization for accessing data residing over the HDFS (by granting and revoking capabilities)

8 A little about Kerberos Protocol Network authentication protocol Developed at MIT in the mid 1980s Available as open source or in supported commercial software

9 How does Kerberos work? Instead of client sending password to application server: Request Ticket from authentication server Ticket and encrypted request sent to application server How to request tickets without repeatedly sending credentials? Ticket granting ticket (TGT)

10 How does Kerberos work?: Ticket Granting Tickets

11 Kerberos contd

12 Some of the notations used A -> B : M #denotes a message M from node A to node B K U A and K R A :The public and private keys of Node A K AB : Key shared between A and B {M} K AB : A message M encrypted with K AB <M> K R A : A message signed with K R A C, N, D: Client, Namenode, Datanode

13 1. Authentication The Namenode checks the details of the request and if the client is a valid user, accordingly issues/doesn't issue the Ticket T C -> N: request_ticket, T S, hash<request_ticket,t S,K CN > N-> C : T

14 Authentication contd Message exchange between client C and Datanode D to establish shared key amongst them. C -> D: {(K CD, T S, nonce)k R C}K U D, T D -> C: nonce', hash(nonce',k CD ) The client sends the ticket T along with a shared key K CD that it wants to establish with the Datanode D, the client also sends a nonce so that the Datanode can verify the freshness of the message.

15 Authentication contd To complete the ticket establishment step, the Datanode has to respond to a nonce challenge. T = <ID U,K U C,IV,T S,T E >K R M K CD = hash<iv,k U D,random_data>

16 Authentication contd T contains the user Id, public key, initialization vector and the ticket s lifetime. Shared key is computed by hashing the IV with the Datanode s public key and some arbitrarily random data.

17 2. Capabilities To read data from the HDFS the client has to obtain block locations and capabilities from Namenode before it goes to Datanodes. C->N : read (path),t S, hash<read(path),t S,K U CN> N->C : block_locations, hash<block_locations> The capabilities are embedded into block location information and signed by the Namenode. The Datanode verifies the capabilities and accordingly allows to read or doesn t.

18 Capabilities cont Description of capability information embedded into the block location information. The sign (With Namenode's private key) of the capability and block id is also embedded C = ID,permissions,path Sign = <c,block_id>k R N

19 Revocation of capabilities Capabilities can potentially be re-used by clients to read the data from HDFS at any time after when they were issued. However the file permissions change over period of time. Revocation of capabilities needs to be done, in order to prevent replay attacks. Capabilities issued by Namenode will have an expiry period ( say 1hr) and this can be configured in hadoop-site.xml

20 Revocation of Capabilities contd The client has to get a renewal ticket issued by the Namenode and has to present it to the Datanode for every request after expiry of the capabilities. If the renewal ticket is not presented, the Datanode will deny the request. Revocation of capabilities is done actively by Namenode. This is done by sending the message to thedatanodes to deny the particular capabilities.

21 Difficulties faced Integrating Kerberos protocol with the HDFS Frame work is quite a task! Need a more efficient design on Granting and Revoking capabilities

22 Conclusion The overhead with the introduction of capabilities is low but secures the data access for only clients which have been issued the capabilities by Namenode. However, if the filesize is lessthan 64MB, the overhead still remains the same as a single block, for smaller files the overhead would be substantial. Although the performance overhead at the Datanode isn t significant for 64MB or larger block size, it can be reduced further by caching the capabilities for each block.

23 Thank You for your time!