Chef Patterns at Bloomberg Scale HADOOP INFRASTRUCTURE TEAM. Freenode: #chef-bach

Transcription

1 CHEF PATTERNS AT BLOOMBERG SCALE HADOOP INFRASTRUCTURE TEAM Freenode: #chef-bach

2 BLOOMBERG CLUSTERS 2 APPLICATION SPECIFIC Hadoop, Kafka ENVIRONMENT SPECIFIC Networking, Storage BUILT REGULARLY DEDICATED BOOTSTRAP SERVER Virtual Machine DEDICATED CHEF-SERVER

3 WHY A VM? 3 LIGHTWEIGHT PRE-REQUISITE Low memory/storage Requirements RAPID DEPLOYMENT Vagrant for Bring-Up Vagrant for Re-Configuration EASY RELEASE MANAGEMENT MULTIPLE VM PER HYPERVISOR Multiple Clusters EASY RELOCATION

4 SERVICES OFFERED 4 REPOSITORIES APT Ruby Gems Static Files (Chef!) CHEF SERVER KERBEROS KDC PXE SERVER DHCP/TFTP Server Cobbler ( Bridged Networking (for test VMs) STRONG ISOLATION

5 BUILDING BOOTSTRAP 5 CHEF AND VAGRANT Generic Image (Jenkins) NETWORK CONFIGURATION CORRECTING KNIFE.RB CHEF SERVER RECONFIGURATION CLEAN UP (CHEF REST API) CONVERT BOOTSTRAP TO BE AN ADMIN CLIENT Secrets/Keys

6 BUILDING BOOTSTRAP CHEF-SOLO PROVISIONER # Chef provisioning bootstrap.vm.provision "chef_solo" do chef chef.environments_path = [[:vm,""]] chef.environment = env_name chef.cookbooks_path = [[:vm,""]] chef.roles_path = [[:vm,""]] chef.add_recipe("bcpc::bootstrap_network") chef.log_level="debug" chef.verbose_logging=true chef.provisioning_path="/home/vagrant/chef-bcpc/" CHEF SERVER RECONFIGURATION NGINX, SOLR, RABBITMQ # Reconfigure chef-server bootstrap.vm.provision :shell, :inline => "chef-server-ctl reconfigure" 6

7 BUILDING BOOTSTRAP CLEAN UP (REST API) ruby_block "cleanup-old-environment-databag" do block do rest = Chef::REST.new(node[:chef_client][:server_url], "admin", \ "/etc/chef-server/admin.pem") rest.delete("/environments/generic") rest.delete("/data/configs/generic") ruby_block "cleanup-old-clients" do block do system_clients = ["chef-validator", "chef-webui"] rest = Chef::REST.new(node[:chef_client][:server_url], "admin", \ "/etc/chef-server/admin.pem") rest.get_rest("/clients").each do client if!system_clients.include?(client.first) rest.delete("/clients/#{client.first}") 7

8 BUILDING BOOTSTRAP 8 CONVERT TO ADMIN (BOOTSTRAP_CONFIG.RB) ruby_block "convert-bootstrap-to-admin" do block do rest = Chef::REST.new(node[:chef_client][:server_url], "admin", "/etc/chef-server/admin.pem") rest.put_rest("/clients/#{node[:hostname]}",{:admin => true}) rest.put_rest("/nodes/#{node[:hostname]}", { :name => node[:hostname], :run_list => ['role[bcpc-bootstrap]'] } )

9 CLUSTER USABILITY 9 CODE DEPLOYMENT APPLICATION COOKBOOKS RUBY GEMS Zookeeper, WebHDFS CLUSTERS ARE NOT SINGLE MACHINE Which machine to deploy Idempotency; Races

10 DEPLOY TO HDFS 10 USE CHEF DIRECTORY RESOURCE USE CUSTOM PROVIDER directory /projects/myapp do mode 755 owner foo recursive true provider BCPC::HdfsDirectory

11 DEPLOY KAFKA TOPIC 11 USE LWRP Dynamic Topic; Right Zookeeper PROVIDER CODE AVAILABLE AT # Kafka Topic Resource actions :create, :update attribute :name, :kind_of => String, :name_attribute => true attribute :partitions, :kind_of => Integer, :default => 1 attribute :replication, :kind_of => Integer, :default => 1

12 KERBEROS 12 KEYTABS Per Service / Host Up to 10 Keytabs per Host WHAT ABOUT MULTI HOMED HOSTS? Hadoop imputes _HOST PROVIDERS WebHDFS uses SPNEGO SYSTEM ROLE ACCOUNTS TENANT ROLE ACCOUNTS AVAILABLE AT

13 LOGIC INJECTION 13 Statutory Warning Code snippets are edited to fit the slides which may have resulted in logic incoherence, bugs and un-readability. Readers discretion requested. COMPLETE CODE CAN BE FOUND AT Community cookbook Wrapper custom recipe

14 LOGIC INJECTION 14 WE USE COMMUNITY COOKBOOKS Takes care of standard install, enable and starting of services NEED TO ADD LOGIC TO COOKBOOK RECIPES Take action on a service only when conditions are satisfied Take action on a service based on depent service state

15 LOGIC INJECTION 15 VANILLA COMMUNITY COOKBOOK: template ::File.join(node.kafka.config_dir, 'server.properties') do source 'server.properties.erb'... helpers(kafka::configuration) if restart_on_configuration_change? notifies :restart, 'service[kafka]', :delayed service 'kafka' do provider kafka_init_opts[:provider] supports start: true, stop: true, restart: true, status: true action kafka_service_actions

16 LOGIC INJECTION VANILLA COMMUNITY COOKBOOK: template ::File.join(node.kafka.config_dir, 'server.properties') do source 'server.properties.erb'... helpers(kafka::configuration) if restart_on_configuration_change? notifies :restart, 'service[kafka]', :delayed #----- Remove ----# service 'kafka' do provider kafka_init_opts[:provider] supports start: true, stop: true, restart: true, status: true action kafka_service_actions #----- Remove----# 16

17 LOGIC INJECTION 17 VANILLA COMMUNITY COOKBOOK 2.0: template ::File.join(node.kafka.config_dir, 'server.properties') do source 'server.properties.erb... helpers(kafka::configuration) if restart_on_configuration_change? notifies :create, 'ruby_block[pre-shim]', :immediately #----- Replace----# include_recipe node["kafka"]["start_coordination"]["recipe"] #----- Replace----#

18 LOGIC INJECTION 18 COOKBOOK COORDINATOR RECIPE: ruby_block 'pre-shim' do # pre-restart no-op notifies :restart, 'service[kafka] ', :delayed service 'kafka' do provider kafka_init_opts[:provider] supports start: true, stop: true, restart: true, status: true action kafka_service_actions

19 LOGIC INJECTION 19 WRAPPER COORDINATOR RECIPE: ruby_block 'pre-shim' do # pre-restart done here notifies :restart, 'service[kafka] ', :delayed service 'kafka' do provider kafka_init_opts[:provider] supports start: true, stop: true, restart: true, status: true action kafka_service_actions notifies :create, 'ruby_block[post-shim] ', :immediately ruby_block 'post-shim' do # clean-up done here

20 SERVICE ON DEMAND 20 COMMON SERVICE WHICH CAN BE REQUESTED Copy log files from applications into a centralized location Single location for users to review logs and helps with security Service available on all the nodes Applications can request the service dynamically

21 SERVICE ON DEMAND 21 NODE ATTRIBUTE TO STORE SERVICE REQUESTS default['bcpc']['hadoop']['copylog'] = {} DATA STRUCTURE TO MAKE SERVICE REQUESTS { } 'app_id' => { 'logfile' => "/path/file_name_of_log_file", 'docopy' => true (or false) },...

22 SERVICE ON DEMAND 22 APPLICATION RECIPES MAKE SERVICE REQUESTS # # Updating node attributes to copy HBase master log file to HDFS # node.default['bcpc']['hadoop']['copylog']['hbase_master'] = { 'logfile' => "/var/log/hbase/hbase-master-#{node.hostname}.log", 'docopy' => true } node.default['bcpc']['hadoop']['copylog']['hbase_master_out'] = { 'logfile' => "/var/log/hbase/hbase-master-#{node.hostname}.out", 'docopy' => true }

23 SERVICE ON DEMAND 23 RECIPE FOR THE COMMON SERVICE node['bcpc']['hadoop']['copylog'].each do id,f if f['docopy'] template "/etc/flume/conf/flume-#{id}.conf" do source "flume_flume-conf.erb action :create... variables(:agent_name => "#{id}", :log_location => "#{f['logfile']}" ) notifies :restart,"service[flume-agent-multi-#{id}]",:delayed service "flume-agent-multi-#{id}" do supports :status => true, :restart => true, :reload => false service_name "flume-agent-multi" action :start start_command "service flume-agent-multi start #{id}" restart_command "service flume-agent-multi restart #{id}" status_command "service flume-agent-multi status #{id}"

24 PLUGGABLE ALERTS 24 SINGLE SOURCE FOR MONITORED STATS Allows users to visualize stats across different parameters Didn t want to duplicate the stats collection by alerting system Need to feed data to the alerting system to generate alerts

25 PLUGGABLE ALERTS ATTRIBUTE WHERE USERS CAN DEFINE ALERTS default["bcpc"]["hadoop"]["graphite"]["queries"] = { 'hbase_master' => [ { 'type' => "jmx", 'query' => "memory.nonheapmemoryusage_committed", 'key' => "hbasenonheapmem", 'trigger_val' => "max(61,0)", 'trigger_cond' => "=0", 'trigger_name' => "HBaseMasterAvailability", 'trigger_dep' => ["NameNodeAvailability"], 'trigger_desc' => "HBase master seems to be down", 'severity' => 1 },{ 'type' => "jmx", 'query' => "memory.heapmemoryusage_committed", 'key' => "hbaseheapmem",... },...], namenode' => [...]...} Query to pull stats from data source Define alert criteria 25

26 TEMPLATE PITFALLS 26 LIBRARY FUNCTION CALLS IN WRAPPER COOKBOOKS Community cookbook provider accepts template as an attribute Template passed from wrapper makes a library function call Wrapper recipe includes the module of library function

27 TEMPLATE PITFALLS WRAPPER RECIPE... Chef::Resource.s(:include, Bcpc::OSHelper)... cobbler_profile "bcpc_host" do kickstart "cobbler.bcpc_ubuntu_host.preseed" distro "ubuntu mini-x86_ FUNCTION CALL IN TEMPLATE... d-i passwd/user-password-crypted password 'cobbler-root-password-salted')}"%> d-i passwd/user-uid string...

28 TEMPLATE PITFALLS 28 MODIFIED FUNCTION CALL IN TEMPLATE... d-i passwd/user-password-crypted password 'cobbler-root-passwordsalted')}"%> d-i passwd/user-uid string...

29 DYNAMIC RESOURCES 29 ANIT-PATTERN? ruby_block "create namenode directories" do block do node[:bcpc][:storage][:mounts].each do d dir = Chef::Resource::Directory.new("#{mount_root}/#{d}/dfs/nn", run_context) dir.owner "hdfs" dir.group "hdfs" dir.mode 0755 dir.recursive true dir.run_action :create exe = Chef::Resource::Execute.new("fixup nn owner", run_context) exe.command "chown -Rf hdfs:hdfs #{mount_root}/#{d}/dfs" exe.only_if { Etc.getpwuid(File.stat("#{mount_root}/#{d}/dfs/").uid).name!= "hdfs " }

30 DYNAMIC RESOURCES 30 SYSTEM CONFIGURATION Lengthy Configuration of a Storage Controller Setting Attributes at Converge Time Compile Time Actions? MUST WRAP IN RUBY_BLOCK S Does not Update the Resource Collection Lazy s everywhere: Guards: not_if{lazy{node[ ]}.call.map{ }}

31 SERVICE RESTART 31 WE USE JMXTRANS TO MONITOR JMX STATS Service to be monitored varies with node There can be more than one service to be monitored Monitored service restart requires JMXtrans to be restarted**

32 SERVICE RESTART 32 DATA STRUCTURE IN ROLES TO DEFINE THE SERVICES "default_attributes" : { "jmxtrans :{ "servers :[ { "type": "datanode", "service": "hadoop-hdfs-datanode", "service_cmd": "org.apache.hadoop.hdfs.server.datanode.datanode" }, { "type": "hbase_rs", "service": "hbase-regionserver", "service_cmd": org.apache.hadoop.hbase.regionserver.hregionserver" } ] }... Depent Service Name String to uniquely identify the service process

33 SERVICE RESTART 33 JMXTRANS SERVICE RESTART LOGIC BUILT DYNAMICALLY jmx_services = Array.new jmx_srvc_cmds = Hash.new node['jmxtrans']['servers'].each do server jmx_services.push(server['service']) jmx_srvc_cmds[server['service']] = server['service_cmd'] service "restart jmxtrans on depent service" do service_name "jmxtrans" supports :restart => true, :status => true, :reload => true Store the depent service name and process ids in local variables action :restart jmx_services.each do jmx_dep_service subscribes :restart, "service[#{jmx_dep_service}]", :delayed only_if {process_require_restart?("jmxtrans","jmxtrans-all.jar, jmx_srvc_cmds)} Subscribes from all depent services What if a process is re/started externally?

34 SERVICE RESTART 34 def process_require_restart?(process_name, process_cmd, dep_cmds) tgt_proces_pid = `pgrep -f #{process_cmd}`... tgt_proces_stime = `ps --no-header -o start_time #{tgt_process_pid}`... ret = false restarted_processes = Array.new dep_cmds.each do dep_process, dep_cmd dep_pids = `pgrep -f #{dep_cmd}` if dep_pids!= "" dep_pids_arr = dep_pids.split("\n") dep_pids_arr.each do dep_pid Start time of the service process Start time of all the service processes on which it is depent on Compare the start time dep_process_stime = `ps --no-header -o start_time #{dep_pid}` if DateTime.parse(tgt_proces_stime) < DateTime.parse(dep_process_stime) restarted_processes.push(dep_process) ret = true...

35 ROLLING RESTART 35 AUTOMATIC CONVERGENCE AVAILABILITY HOW High Availability Toxic Configuration Check Masters for Slave Status Synchronous Communication Locking

36 ROLLING RESTART 36 FLAGGING Negative Flagging flag when a service is down Positive Flagging flag when a service is reconfiguring Deadlock Avoidance CONTENTION Poll & Wait Fail the Run Simply Skip Service Restart and Go On Store the Need for Restart Breaks Assumptions of Procedural Chef Runs

37 ROLLING RESTART 37 SERVICE DEFINITION HADOOP_SERVICE "ZOOKEEPER-SERVER" DO DEPENDENCIES ["TEMPLATE[/ETC/ZOOKEEPER/CONF/ZOO.CFG]", "TEMPLATE[/USR/LIB/ZOOKEEPER/BIN/ZKSERVER.SH]", "TEMPLATE[/ETC/DEFAULT/ZOOKEEPER-SERVER]"] PROCESS_IDENTIFIER "ORG.APACHE.ZOOKEEPER... QUORUMPEERMAIN" END

38 ROLLING RESTART 38 SYNCH STATE STORE Zookeeper SERVICE RESTART (KAFKA) VALIDATION CHECK Based on Jenkins pattern for wait_until_ready! Verifies that the service is up to an acceptable level Passes or stops the Chef run FUTURE DIRECTIONS Topology Aware Deployment Data Aware Deployment

39 WE ARE HIRING JOBS.BLOOMBERG.COM: Hadoop Infrastructure Engineer DevOps Engineer Search Infrastructure Freenode: #chef-bach