Is Hadoop Enterprise ready?

Transcription

1 Is Hadoop Enterprise ready? Building Hadoop cluster Krzysztof Adamski

2 Agenda About ISP Team Architecture Automated Hadoop deployment Monitoring Security Q&A

3 About ING Services Polska

4 ISP Service Catalogue

5 ISP promotes ambitious goals % 4 100% 86% Headcount FTEs WPC (actuals 2014) Process maturity assessed by Ernst & Young Average systems availability (actuals 2014) General IT controls of KPIs on target

6 ISP has been growing as a solid business partner 35+ Business partners 191 SLAs 18 Countries 8.1 Customer satisfaction (1-10 scale, Q4 2014) Security monitoring, Remote management, System hosting, Security services Services

7 The team

8 The A team Don t hire, train them! Break out of the silo mentality DevOps Agile Let them choose their own tools Automation

9 Architecture

10 Hadoop deployment options

11 Cloud vs on-premise Legal and Regulatory Issues (e.g. data locality, limited responsibility) Network speed (we are talking BIG data) Time to market Initial costs

12 Basic network principles Machines should be on an isolated network from the rest of the data center Machines should have static IPs Reverse DNS should be setup Top-of-the-rack switches hadoop servers are quite chatty Multi-homed networks are tricky

13 VLAN configuration example VLAN Fabric NIC Port Function Failover vlan160_mgmt A eth0 Management, User connectivity Fabric failover to B vlan12_hdfs B eth1 Hadoop Fabric failover to A vlan11_data A eth2 SAN/NAS access, ETL Fabric failover to B Cisco reference architecture

14 ToR vs Cisco ref. architecture

15 Linux general recommendations Use FQDNs required by Ambari, Kerberos Disable IPTables since we are within isolated network Disable SELinux enabling it can be very challenging Set swappiness to 1 Set ulimits to 64k Disable Transparent Huge Pages Disable atime Enable NTP JBOD for hadoop drives RAID1 for system drives (if dedicated)

16 What else do we need? Code repository e.g. Stash, GitLab Open Source package repository for Python (pip), Perl (cpan), R (cran), Maven Repository Manager Integration tools e.g Jenkins Stepping stone (edge) server Other RDBMS to store aggregates e.g. MySQL, PostgreSQL Data scientists server RStudio, Ipython etc.

17 Did you know?

18 Hadoop DR strategy No inherent cross data center replication DistCp can be used for large inter/intra-cluster copying Data can be ingested into two separate hadoop clusters Wandisco Non-Stop Hadoop

19 Automated deployment

20 RHEL Kickstart installation Bladelogic jobs to provision software components e.g. monitoring agents, security monitoring components Bladelogic jobs to harden RHEL security according to best practicies Red Hat Satellite as package distribution and versioning center

21 UCS Manager - organisation Let Hadoop team manager servers themself create organization Create server profile template Create profiles from a template

22 UCS Manager fabric interconnect

23 Ambari

24 Ambari

25 Ambari HA wizard

26 Ambari blueprints

27 Ambari blueprint example { "configurations" : [ { "configuration-type" : { "property-name" : "property-value", "property-name2" : "property-value" } }, { "configuration-type2" : { "property-name" : "property-value" } }... ], "host_groups" : [ { "name" : "host-group-name", "components" : [...

28 Ambari REST API curl -u admin:$password -i -H 'X-Requested-By: ambari' -X PUT -d '{"RequestInfo": {"context" :"Start HDFS via REST"}, "Body": {"ServiceInfo": {"state": "STARTED"}}}' /CLUSTER_NAME/services/HDFS curl -u admin:$password -H 'X-Requested-By: ambari' -X GET " onents/?servicecomponentinfo/category.in(slave,master)&host_ components/hostroles/host_name=clusternode&fields=host_compo nents/hostroles/component_name,host_components/hostroles/sta te

29 Leverage docker

30 Did you know? Upgrading hadoop stack can be still a painful (80 man pages) process Ref /Ambari_Upgrade_v170/Ambari_Upgrade_v170.pdf

31 Monitoring

32 Hadoop Availability Monitoring (service health)

33 Hadoop metrics monitoring

34 Did you know? Check your region and language settings ;)

35 Security

36 Hadoop security Hadoop is not a single product, choose your components wisely Up until recently there was no single point for user managment Maintaining ACL in HDFS is a painful process No out of the box Active Directory integration

37 Hadoop ring of defense

38 Apache Knox Gateway

39 Is there anything we can do? 1. Do not store sensitive data within Hadoop 2. Separate Hadoop environment in a separate network zone (dedicated vlan/s, firewall filtered traffic) 3. Kerberize cluster environment a) Watch for unkerberized components b) Keep your keytabs safe 4. LDAP for central user managment 5. Manager your ACLs start simple with POSIX groups 6. Auditting

40 IPA At the most basic level, Red Hat Identity Management is a domain controller for Linux and Unix machines.

41 IPA server client communication

42 IPA

43 Did you know? IPA 3 for RHEL 6 has issues when installing using external CA option

44 Central user and policy managment

45 Ranger

46 Where to continue from here? hadoop distribution best practicies Reference architecture papers /Cluster_Plan_Gd_v22/Cluster_Plan_Gd_v22.pdf

47 Interesting books and docs

48 Q&A

49