Hardware Monitoring with the new IPMI Plugin v2

Transcription

1 Hardware Monitoring with the new IPMI Plugin v2 Werner Fischer, Technology Specialist Thomas-Krenn.AG 6. OSMC / Nuremberg / Germany 29th November 2011

2 Introduction who I am Werner Fischer working for a Server vendor who I am not Linux user since 2001 Kernel or H/W developer slide 2/37

3 Introduction who is Server & accessories "Made in Germany" based in Freyung, Bavaria serving all over Europe slide 3/37

4 Some questions... Should I use monitoring? slide 4/37

5 Some questions... Should I use monitoring? It depends on what you want to do in your free time... remember yesterday evening? slide 5/37

6 Some questions... All drives of the RAID 6 O.K.? slide 6/37

7 Some questions... All network connections O.K.? slide 7/37

8 Some questions... All FANs O.K.?? slide 8/37

9 Some questions... All power supplies O.K.?? slide 9/37

10 Some questions... Can we monitor all these servers? any IPMI compatible server slide 10/37

11 Some questions... Can we monitor all these servers? Easily? any IPMI compatible server slide 11/37

12 Some questions... Can we monitor all these servers? Easily? With one single tool? any IPMI compatible server slide 12/37

13 Some questions... Can we monitor all these servers? Easily? With one single tool? slide 13/37

14 Agenda 1) IPMI overview 2) Plugin implementation 3) Live demo 4) Common pitfalls slide 14/37

15 Intelligent Platform Management Interface IPMI developed by Intel, HP, NEC, Dell 1998: IPMI v : IPMI v : IPMI v2.0 slide 15/37

16 IPMI main features Monitoring (temp, fans,...) Recovery Control (power on/off/reset) Logging (System Event Log) Inventory (FRU information) slide 16/37

17 IPMI overview LAN Connector Serial Connector access req. username & password Remote Mmgt. Card (KVM over IP,...) Auxillary IPMB Connector PCI mgmt. bus Network (LAN) Controller LAN interface Serial Port Sharing Serial/Modem interface BMC Serial Controller Chassis mgmt. (Satellite Controller) NVS Storage SDR SEL FRU FRU Sensors & Controls Fan sensor Temp. sensor Power control Reset control private mgmt. busses FRU M/B Serial Controller ICMB bridge IPMB Baseboard Management Controller (BMC) access req. root privileges ICMB System interface FRU Temp. s. Memory board Temp. sensor Chassis board FRU Redundant Power board Processor board System bus Motherboard slide 17/37

18 IPMI Channel Privilege Levels (LAN access) Privilege Level Allows query sensors User nearly all IPMI commands Operator but no changing of out-ofband interfaces Administrator all IPMI commands use privilege level 'User' for monitoring purposes slide 18/37

19 Example: remote control with ipmitool ~]$ ipmitool I lan H \ [user@adminpc ~]$ ipmitool I lan H \ U admin power status U admin power status Password: Password: Chassis Power is off Chassis Power is off [user@adminpc ~]$ [user@adminpc ~]$ [user@adminpc ~]$ ipmitool I lan H \ [user@adminpc ~]$ ipmitool I lan H \ U admin power on U admin power on Password: Password: Chassis Power Control: Up/On Chassis Power Control: Up/On [user@adminpc ~]$ [user@adminpc ~]$ [user@adminpc ~]$ ipmitool I lan H \ [user@adminpc ~]$ ipmitool I lan H \ U admin power status U admin power status Password: Password: Chassis Power is on Chassis Power is on [user@adminpc ~]$ [user@adminpc ~]$ slide 19/37

20 IPMI Sensor Classes (1/2) Discrete Threshold multiple states possible: up to 15 states each state is reflected by a bit multiple state bits can active changes event status on: analog reading compared to threshold values can provide: generic states sensor-specific states provides: analog reading of the sensor discr. threshold comparison status bit other class similar to discrete: OEM: discrete sensor where the meaning of the states (offsets) are OEM defined slide 20/37

21 IPMI Sensor Classes (2/2) Discrete ~]# ipmitool sdr get "PS2 Status" ~]# ipmitool sdr get "PS2 Status" Sensor ID : PS2 Status (0x71) Sensor ID : PS2 Status (0x71) Entity ID : 10.2 (Power Supply) Entity ID : 10.2 (Power Supply) Sensor Type (Discrete): Power Supply Sensor Type (Discrete): Power Supply States Asserted : Power Supply States Asserted : Power Supply [Presence detected] [Presence detected] [Power Supply AC lost] [Power Supply AC lost] Assertion Events : Power Supply Assertion Events : Power Supply [Presence detected] [Presence detected] [Power Supply AC lost] [Power Supply AC lost] Assertions Enabled : Power Supply Assertions Enabled : Power Supply [Presence detected] [Presence detected] [Failure detected] [Failure detected] [Predictive failure] [Predictive failure] [Power Supply AC lost] [Power Supply AC lost] [...] [...] Deassertions Enabled : Power Supply Deassertions Enabled : Power Supply [...] [...] Threshold [root@test ~]# ipmitool sdr get "Fan 1" [root@test ~]# ipmitool sdr get "Fan 1" Sensor ID : Fan 1 (0x50) Sensor ID : Fan 1 (0x50) Entity ID : 29.1 (Fan Device) Entity ID : 29.1 (Fan Device) Sensor Type (Analog) : Fan Sensor Type (Analog) : Fan Sensor Reading : 5719 (+/ 0) RPM Sensor Reading : 5719 (+/ 0) RPM Status : ok Status : ok Nominal Reading : Nominal Reading : Normal Minimum : Normal Minimum : Normal Maximum : Normal Maximum : Lower critical : Lower critical : Lower non critical : Lower non critical : Positive Hysteresis : Positive Hysteresis : Negative Hysteresis : Negative Hysteresis : Minimum sensor range : Unspecified Minimum sensor range : Unspecified Maximum sensor range : Unspecified Maximum sensor range : Unspecified Event Message Control : Per threshold Event Message Control : Per threshold Readable Thresholds : lcr lnc Readable Thresholds : lcr lnc Settable Thresholds : lcr lnc Settable Thresholds : lcr lnc Threshold Read Mask : lcr lnc Threshold Read Mask : lcr lnc Assertion Events : Assertion Events : Assertions Enabled : lnc lcr Assertions Enabled : lnc lcr Deassertions Enabled : lnc lcr Deassertions Enabled : lnc lcr slide 21/37

22 IPMI Sensor Types ipmi sensors L root@test:~# ipmi sensors L Temperature Temperature Voltage Voltage Current Current Fan Fan Physical_Security Physical_Security Platform_Security_Violation_Attempt Platform_Security_Violation_Attempt Processor Processor Power_Supply Power_Supply Power_Unit Power_Unit Cooling_Device Cooling_Device [ ] [ ] slide 22/37

23 Example: query sensors with FreeIPMI ~]# ipmimonitoring ~]# ipmimonitoring Record_ID Sensor Name Sensor Group Monitoring Status Record_ID Sensor Name Sensor Group Monitoring Status Sensor Units Sensor Reading Sensor Units Sensor Reading [...] [...] 17 Fan 5 Fan Nominal RPM Fan 5 Fan Nominal RPM Fan 6 Fan Nominal RPM Fan 6 Fan Nominal RPM PS1 AC Current Current Nominal A PS1 AC Current Current Nominal A PS2 AC Current Current Nominal A PS2 AC Current Current Nominal A [...] [...] 36 Physical Scrty Physical Security Critical N/A 36 Physical Scrty Physical Security Critical N/A 'General Chassis Intrusion' 'General Chassis Intrusion' slide 23/37

24 Example: interpret discrete sensors( FreeIPMI) cat /etc/freeipmi/freeipmi_interpret_sensor.conf cat /etc/freeipmi/freeipmi_interpret_sensor.conf [ ] [ ] ## IPMI_Physical_Security ## IPMI_Physical_Security ## # IPMI_Physical_Security_No_Event Nominal # IPMI_Physical_Security_No_Event Nominal # IPMI_Physical_Security_General_Chassis_Intrusion Critical # IPMI_Physical_Security_General_Chassis_Intrusion Critical # IPMI_Physical_Security_Drive_Bay_Intrusion Critical # IPMI_Physical_Security_Drive_Bay_Intrusion Critical [ ] [ ] # IPMI_Power_Supply_No_Event Nominal # IPMI_Power_Supply_No_Event Nominal # IPMI_Power_Supply_Presence_Detected Nominal # IPMI_Power_Supply_Presence_Detected Nominal # IPMI_Power_Supply_Power_Supply_Failure_Detected Critical # IPMI_Power_Supply_Power_Supply_Failure_Detected Critical # IPMI_Power_Supply_Predictive_Failure Critical # IPMI_Power_Supply_Predictive_Failure Critical # IPMI_Power_Supply_Power_Supply_Input_Lost_AC_DC Critical # IPMI_Power_Supply_Power_Supply_Input_Lost_AC_DC Critical [ ] [ ] ## IPMI_Memory ## IPMI_Memory ## # IPMI_Memory_No_Event Nominal # IPMI_Memory_No_Event Nominal # IPMI_Memory_Correctable_Memory_Error Warning # IPMI_Memory_Correctable_Memory_Error Warning # IPMI_Memory_Uncorrectable_Memory_Error Critical # IPMI_Memory_Uncorrectable_Memory_Error Critical slide 24/37

25 IPMI System Event Log (SEL) stored in non-volatile storage ~]# ipmitool sel elist ~]# ipmitool sel elist 40 06/21/ :29:29 Power Supply PS1 Status Power Supply AC lost Asserted 40 06/21/ :29:29 Power Supply PS1 Status Power Supply AC lost Asserted 54 06/21/ :29:29 Power Unit Power Redundancy Fully Redundant 54 06/21/ :29:29 Power Unit Power Redundancy Fully Redundant 68 06/21/ :29:29 Power Unit Power Redundancy Redundancy Lost 68 06/21/ :29:29 Power Unit Power Redundancy Redundancy Lost 7c 06/21/ :29:29 Power Unit Power Redundancy Non Redundant: Sufficient from Redundant 7c 06/21/ :29:29 Power Unit Power Redundancy Non Redundant: Sufficient from Redundant [...] [...] 2fc 06/21/ :20:32 Physical Security Physical Scrty General Chassis intrusion Asserted 2fc 06/21/ :20:32 Physical Security Physical Scrty General Chassis intrusion Asserted ~]# ipmitool sel elist ~]# ipmitool sel elist Power Supply PS1 Status Power Supply AC lost Asserted Power Supply PS1 Status Power Supply AC lost Asserted Power Unit Power Redundancy Fully Redundant Power Unit Power Redundancy Fully Redundant Power Unit Power Redundancy Redundancy Lost Power Unit Power Redundancy Redundancy Lost Power Unit Power Redundancy Non Redundant: Sufficient from Redundant Power Unit Power Redundancy Non Redundant: Sufficient from Redundant [...] [...] Physical Security Physical Scrty General Chassis intrusion Asserted Physical Security Physical Scrty General Chassis intrusion Asserted slide 25/37

26 Agenda 1) IPMI overview 2) Plugin implementation 3) Live demo 4) Common pitfalls slide 26/37

27 Plugin implementation Bash script uses FreeIPMI, gawk #./check_ipmi_sensor H f /etc/ipmi config/ipmi.cfg #./check_ipmi_sensor H f /etc/ipmi config/ipmi.cfg IPMI Status: OK 'System Temp'= 'FAN 1'= 'FAN IPMI Status: OK 'System Temp'= 'FAN 1'= 'FAN 2'= 'FAN 3'= 'FAN 4'= 'FAN 2'= 'FAN 3'= 'FAN 4'= 'FAN A'= 'Vcore'= '3.3VCC'= '12V'= A'= 'Vcore'= '3.3VCC'= '12V'= 'VDIMM'= '5VCC'= ' 12V'= 'VBAT'= 'VDIMM'= '5VCC'= ' 12V'= 'VBAT'= 'VSB'= 'AVCC'= 'VSB'= 'AVCC'= slide 27/37

28 Plugin implementation Bash script uses FreeIPMI, gawk #./check_ipmi_sensor H f /etc/ipmi config/ipmi.cfg v 2 #./check_ipmi_sensor H f /etc/ipmi config/ipmi.cfg v 2 IPMI Status: OK 'System Temp'= 'FAN 1'= 'FAN IPMI Status: OK 'System Temp'= 'FAN 1'= 'FAN [ ] [ ] System Temp = (Status: Nominal) System Temp = (Status: Nominal) CPU Temp = 'Low' (Status: Nominal) CPU Temp = 'Low' (Status: Nominal) FAN 1 = (Status: Nominal) FAN 1 = (Status: Nominal) FAN 2 = (Status: Nominal) FAN 2 = (Status: Nominal) FAN 3 = (Status: Nominal) FAN 3 = (Status: Nominal) [ ] [ ] AVCC = (Status: Nominal) AVCC = (Status: Nominal) Chassis Intru = 'OK' (Status: Nominal) Chassis Intru = 'OK' (Status: Nominal) PS Status = 'Presence detected' (Status: Nominal) PS Status = 'Presence detected' (Status: Nominal) slide 28/37

29 Plugin implementation clear illustration in webinterfaces slide 29/37

30 Agenda 1) IPMI overview 2) Plugin implementation 3) Live demo 4) Common pitfalls slide 30/37

31 Agenda 1) IPMI overview 2) Plugin implementation 3) Live demo 4) Common pitfalls slide 31/37

32 Common pitfalls sensors with state N/A [ ] [ ] 12 CPU1 Temp OEM Reserved N/A N/A N/A 'OEM Event = 0000h' 12 CPU1 Temp OEM Reserved N/A N/A N/A 'OEM Event = 0000h' 13 CPU2 Temp OEM Reserved N/A N/A N/A 'OEM Event = 0000h' 13 CPU2 Temp OEM Reserved N/A N/A N/A 'OEM Event = 0000h' [ ] [ ] solution shortest-term: exclude (-x opt.) solution short-term: FreeIPMI update tkwiki.cc/freeipmi-na-sensor slide 32/37

33 Common pitfalls unrecognized events [ ] [ ] 40 Status Cable/Interconnect Nominal N/A 40 Status Cable/Interconnect Nominal N/A 'Cable/Interconnect is connected' 'Cable/Interconnect is connected' 41 RAC Status Module/Board N/A N/A 'Unrecognized Event = 41 RAC Status Module/Board N/A N/A 'Unrecognized Event = 0001h' 'Unrecognized Event = 0002h' 'Unrecognized Event = 0004h' 0001h' 'Unrecognized Event = 0002h' 'Unrecognized Event = 0004h' 42 OS Watchdog Watchdog 2 Nominal N/A 'OK' 42 OS Watchdog Watchdog 2 Nominal N/A 'OK' [ ] [ ] solution shortest-term: ignore unrec. e. tkwiki.cc/freeipmi-unrec-event slide 33/37

34 Agenda 1) IPMI overview 2) Plugin implementation 3) Live demo 4) Common pitfalls some conclusions slide 34/37

35 Conclusions (1/2) Download: Mailing List: lists.thomas-krenn.com Thanks for your contribution: Nikolaus Filus, Timme Katz, Lars Meuser, Sebastian Mörchen, Gustav Olsson, Holger Paschke, Andy Spiegl, Ulrich Zehl slide 35/37

36 Conclusions (2/2) Monitor hardware with Icinga & IPMI Problems? They will tell you! It'll save you time & money slide 36/37

37 Get German article on the plugin for free at tkwiki.cc/ipmi-plugin Thanks for your time!