heck What the is wrong with my server?!? Josh Malone Systems Administrator National Radio Astronomy Observatory Charlottesville, VA

Transcription

1 heck What the is wrong ^ with my server?!? Josh Malone Systems Administrator National Radio Astronomy Observatory Charlottesville, VA 1

2 Agenda Intro to Monitoring Nagios Install/Config Usage Custom plugins Internet protocols 101 SMTP IMAP HTTP Packet sniffing for dummies Intro to Troubleshooting Tools telnet, openssl grep, sed ps, lsof, netstat 2

3 MONITORING 3

4 Automated Monitoring Workflow 4

5 Monitoring Packages: Open Source Pandora FMS Opsview Core Naemon Captialware ServerStatus Core Sensu All Trademarks and Logos are property of their respective trademark or copyright holders and are used by permission or fair use for education. Neither the presenter nor the conference organizers are affiliated in any way with any companies mentioned here. 5

6 Monitoring Packages: Commercial Nagios XI Groundwork PRTG network monitor CopperEgg WhatsUp Gold PRTG network monitor op5 (Naemon) All Trademarks and Logos are property of their respective trademark or copyright holders and are used by permission or fair use for education. Neither the presenter nor the conference organizers are affiliated in any way with any companies mentioned here. 6

7 Why Automatic Service Monitoring? Spot small problems before they become big ones Learn about outages before your users do Checklist when restoring from a power outage Gives you better problem reports than users Problems you might never spot otherwise Failed HDDs in RAIDs Full /var partitions Logs not rotating System temperature rising 7

8 Why Automatic Service Monitoring? Capacity planning Performance data can generate graphs of utilization RAM, Disk, etc. Availability reports - CAUTION Easy to generate -- even easier to generate wrong Make sure your configurations actually catch problems Will also include problems with Nagios itself :( If you re going to quote your availability numbers (SLAs, etc.) make sure you understand what you re actually monitoring. 8

9 Without Monitoring With Monitoring! The Internet s down - fix it!!! dhcp out of leases dhcp server down dns server not responding ethernet switch down ISP link down / saturated 9

10 Without Monitoring With Monitoring! ZOMG! Our web site is down! O Noes!!! connectivity issues web server down apache not running web server disk full server load too high 10

11 Nagios Open source host / service monitoring package Nagios Ain't Gonna Insist On Sainthood Originally released in 1999 as NetSaint Available in 2 versions: Core and XI Nagios Core: Open-source, freely available Nagios XI: Commercial Free license for up to 7 hosts Available as source installer or VMware appliance 11

12 Terminology Host - A logical (physical / virtual) machine running an OS Service - A resource available on a host Network service (www, dns, imap) Local resource (free RAM, disk space, system load) Plugin - An executable that checks something Add-on - An extension to Nagios that adds functionality Graphing, Trending, SNMP trap reporting, etc. 12

13 What s a plugin? Plugins actually run the service or host checks. Each plugin monitors a different type of service Data from plugin is communicated to Nagios using a (very) simple API Plugins can also report Performance Data (perfdata) to be graphed or tracked Requires a perfdata add-on (or Nagios XI) Plugins can be written in any language Perl plugins can run using Nagios s embedded perl interpreter for increased performance 13

14 INSTALLING NAGIOS 14

15 A word of caution... Installing Nagios on Linux is much easier than Mac. While most Linux distros are still shipping Nagios 3.5.x, they usually come with all the pre-requisite packages available via their native package managers. This makes it much easier to build 4.x from source. Small to medium environments don t require much of a server to run Nagios. Mac installations will require you to install numerous prerequisites, especially for the plug-ins. Many of the pre-reqs can be installed using MacPorts, Homebrew and cpan. If you can run on Linux, do it. 15

16 About Nagios Replacements When Nagios went commercial, the open-source community decided that it needed not one, not two, but three replacements for Nagios: Icinga and Naemon (forks of Nagios) and Shinken (a drop-in replacement). Most linux distros are now shipping one or more of these compatible replacements rather than the official Nagios Core. Not a single distro I checked is shipping Nagios 4. Either Shinken, Naemon or Icinga should work fine using the material covered in the tutorial, but I have only briefly tested Icinga and have not tested Shinken or Naemon at all. 16

17 About Nagios Plugins source On January 15, 2014, Nagios Enterprises forked the original code from the Nagios plugins project, effectively creating two sources for the Nagios plugins. The original maintainers of the plugins package renamed it to monitoring plugins and now host their distribution at monitoring-plugins.org Nagios Enterprises appears to be actively maintaining the nagios-plugins package, including security fixes This tutorial will use the official Nagios plugins package 17

18 Requirements C compiler (Xcode + cmdline tools) Perl 5.8+ (for many plugins) Web server (for web interface) PHP 5.3 gd (for statusmap and trends CGIs) rrdtool, perl-rrd, php-gd (for PNP4Nagios) 18

19 Nagios 4 - brew or build from source Latest MacPorts version of nagios is boo :( Uses old worker code and not as efficient as 4.x Latest Homebrew version is yay! brew install nagios! We want Nagios at least fixes lots of bugs and is much faster 19

20 Create the user and group Nagios needs a user / group to run as Create user nagios with group nagios Can use System Preferences -- no need to resort to command line 20

21 Create group nagios Go to Users and Groups, click the + Change New Account: selector to Group Set name to nagios 21

22 Create user nagios Click + again Create a Standard account Account name: nagios Set a password for security 22

23 Create user nagios Right-click the new nagios account and select Change group to nagios Set shell to /usr/bin/false 23

24 Download Nagios Core Nagios plugins package PNP4Nagios 24

25 Installing Nagios on OSX Install Xcode and command line tools Install gd library using MacPorts (or homebrew) cpan! port install gd2! brew install gd! install Net::SNMP! Extract the tarball Configure and install nagios./configure --with-gd-inc=/opt/local/include \ --with-gd-lib=/opt/local/lib \ --with-httpd-conf=/etc/apache2/other! 25

26 Compile problems: ld: archive has no table of contents for architecture x86_64 Probably a makefile bug Solution: run ranlib lib/libnagios.a 26

27 Installation from source sudo -s! make install! make install-config! make install-webconf install-exfoliation! make install-commandmode! install -m 755 -o root -g wheel daemon-init \! /usr/local/nagios/nagios.init! htpasswd -c /usr/local/nagios/etc/htpasswd.users nagios 27

28 Build and Install Plugins The plugins have LOTS of prerequisites LDAP libs SNMP utils samba client openssl libs Installed most prerequisites via MacPorts Homebrew should work as well!./configure --prefix=/usr/local/nagios \! --enable-perl-modules \! --with-mysql=/opt/local/lib/mysql5 \! --with-openssl=/opt/local/ 28

29 Install pnp4nagios Pre-reqs: rrdtool, perl RRD modules, php-gd!./configure --with-nagios-user=nagios \! with-nagios-group=nagios!! make! # Install without npcd (bulk mode)! make install-config install-processperfdata \! install-html install-plugins 29

30 Configure pnp4nagios cp /usr/local/pnp4nagios/etc/nagios.cfg-sample \! /usr/local/nagios/etc/pnp4nagios.cfg! cp /usr/local/pnp4nagios/etc/misccommands.cfg-sample \ /usr/local/nagios/etc/objects/misccommands.cfg 30

31 Create LaunchDaemon <?xml version="1.0" encoding="utf-8"?>! <!DOCTYPE plist PUBLIC -//Apple Computer//DTD PLIST 1.0//EN! >! <plist version="1.0">! <dict>! <key>label</key>! <string>org.nagios.nagios</string>! <key>programarguments</key>! <array>! <string>/usr/local/nagios/bin/nagios</string>! <string>-d</string>! <string>/usr/local/nagios/etc/nagios.cfg</string>! </array>! <key>keepalive</key>! <true/>! </dict>! </plist> 31

32 Web server / PHP Enable OSX web server Set PHP timezone to avoid a warning echo "date.timezone = 'America/New_York'" >>/etc/php.ini 32

33 CONFIGURING NAGIOS 33

34 Configuration Nagios has 2 parts to configure: the daemon and the cgi s Configure the CGIs to define access levels (authorization) Configure the daemon Commands Check commands Misc commands (how to notify, event handlers) Services Contacts Monitoring and alerting policies 34

35 Enable authentication in the CGIs cgi.conf use_authentication=1! Expects your web server to authenticate users Only allow admin users to issue commands and view config details authorized_for_system_information=...! authorized_for_configuration_information=...! authorized_for_system_commands=...! authorized_for_all_service_commands=...! authorized_for_all_host_commands=...! Allow any authenticated user to view Nagios status authorized_for_all_services=*! authorized_for_all_hosts=* 35

36 nagios.cfg Main configuration file Typically includes additional cfg files for commands, services, etc. ( objects ) cfg_file=/path/to/file! cfg_dir=/path/to/dir/of/configs! Interval length How many seconds are Nagios s default unit of time If you need to monitor a service more than once per minute, you can lower this from 60 interval_length=60 36

37 nagios.cfg Timeouts (in seconds):! service_check_timeout=60! host_check_timeout=30! event_handler_timeout=30! notification_timeout=30! ocsp_timeout=5! perfdata_timeout=5! If you have checks, event handlers, etc. that take a long time to run, increase the appropriate timeout value 37

38 Included default config files commands.cfg contacts.cfg localhost.cfg printer.cfg switch.cfg templates.cfg timeperiods.cfg Define commands called in other files Define contacts to be notified Example services for the local system, example printer, example network switch, etc. Templates for most objects; useful as a starting point for inheritance Def. periods of time (24x7, daytime, etc) 38

39 Inheritance All objects in Nagios support inheritance via the use <template> directive. Templates can inherit from other templates All templates must include the register 0 directive so Nagios knows it s just a template Templates include common settings for different obj types normal_check_interval, retry_check_interval, notification_interval, notification_period, notification_options, max_check_attempts, check_period 39

40 Object Configuration: Overview Inheritance 40 Reference / Assignment

41 Inheritance Example define service {!!name generic-service!!normal_check_interval! 5!!retry_check_interval!! 1!!max_check_attempts!! 3!!! register!!!!!! 0! }!! define service {!! use!!!!! generic-service!!! host_name!!! webserv!! service_description HTTP!! check_command!!! check_http! } Check every 5 minutes. Notify after service is down for ~3 minutes. 41

42 Basic Configuration Example Monitor a mail server that provides IMAP, SMTP and webmail Server IP address is name hornet All services are running on standard ports Not concerned with SSL services for this demo 42

43 Example: Host Templates define host {!! name!!!!!!!!!!!! generic-host!! notifications_enabled! 1!! event_handler_enabled! 1!! flap_detection_enabled! 1!! process_perf_data! 1!! retain_status_information! 1!! retain_nonstatus_information! 1!! notification_period! 24x7!! check_period!!!!!!!!! 24x7!! register! 0! } 43

44 Example: Host Templates define host {!! name!!!!!!!!!!!! generic-server!! use!!!!!!!!!!!! generic-host!! check_interval!!!!!!!! 5!! retry_interval!!!!!!!! 1!! max_check_attempts!!!!!! 5!! check_command!!!!!!!! check-host-alive!! notification_options!!!!!! d,u,r!! notification_interval!!!!! 120!! contact_groups!!!!!!!! sysadmins!! register!!!!!!!!!! 0! } 44

45 Example: Define Contacts define contact {!! contact_name!!!!! alice!! use!!!!!!!! generic-contact!! alias!!!!!!!! Alice Admin!! !!!!!!!! }! define contact {!! contact_name!!!!! bob!! use!!!!!!!! generic-contact!! alias!!!!!!!! Bob Admin!! !!!!!!!! }! define contactgroup {!! contactgroup_name!!!! sysadmins!! alias!!!!!!!! All admins!! members!!!!!!! alice,bob! } 45

46 Example: Define the Server define host {!! name!!!!!!!!!!! hornet!! use!!!!!!!!!!! generic-server!! host_name!!!!!!!!! hornet!! host_address!!!!!!!! ! } Most configuration inherited from generic-server Server will be checked every 5 minutes Problems will be re-checked every minute Alerts will be sent to contact group sysadmins after 5 failed checks 46

47 Example: Service Template define service{! name generic-service! active_checks_enabled 1! passive_checks_enabled 1! parallelize_check 1! check_freshness 0! notifications_enabled 1! event_handler_enabled 1! flap_detection_enabled 1! process_perf_data 1! check_period 24x7! max_check_attempts 5! normal_check_interval 5! retry_check_interval 1! notification_options w,u,c,r! notification_interval 60! notification_period 24x7! register 0! } 47

48 Example: Define the Services define service {!! use!!!!!!!!!!! generic-service!! host_name!!!!!!!!! hornet!! service_description!!!!! HTTP!! check_command!!!!!!! check_http! } Service will be checked every 5 minutes Problems will be re-checked every minute Alerts will be sent to contacts assigned to the host hornet 48

49 Example: Define check command define command {! command_name check_smb! command_line $USER1$/check_disk_smb -H $HOSTADDRESS$ -s "$ARG1$" -u nagios -p foobar -w "$ARG2$" -c "$ARG3$"! } Check command uses macros $USER1$ is set in nagios.cfg and is usually the full path to plugins installation directory $ARG1$, $ARG2$, etc. are macros that refer to arguments when calling the check_smb command $HOSTADDRESS$ is the IP of the host that is being checked 49

50 Example: Define New Service define service {!! use!!!!!!!!! generic-service!! host_name!!!!!!! hornet!! service_description!!! Docs share!! check_command!!!!! check_smb!is docs!80%!95%! } Passes 3 arguments to the check_smb check command SMB share IS docs will be checked every 5 minutes Problems will be re-checked every minute Alerts will be sent to contacts assigned to the host hornet 50

51 Running Plugins on a Remote Host Sometimes you need the plugin to actually run on the monitored host Checking local resources like disk, ram, etc. Multiple options for remote plugin execution check_by_ssh - ssh to host and run local plugin nrpe ncpa - the Nagios Remote Plugin Executor - a full-featured Nagios monitoring agent 51

52 USING NAGIOS 52

53 Navbar Overview Main window 53

54 Host summary Services View Service summary 54

55 Click on Services - Critical 55

56 Host and Service Groups Organize services or hosts into groups by function, etc. Can disable alerts, schedule downtime, etc. on whole group Can show availability report for a whole group Group services by desired reporting capability Groups get a unique URL so you can send a single link to check on a group of hosts Great for PHBs! Also great for delegated IT departments 56

57 Service Groups 57

58 Acknowledging an Outage Click on service name (or hostname) that has the problem Under Service Commands Click Acknowledge this service problem You must enter a comment about why you are acknowledging the problem (i.e., Bob is working on it ) Click Commit 58

59 Acknowledging an Outage Click Here 59

60 Acknowledging an Outage 60

61 The Tactical Overview Displays overview of monitored services and hosts Shows if Any services / hosts have notifications disabled Any services / hosts are flapping Active / passive checks enabled / disabled Warning / Critical / Okay breakdown 61

62 The Tactical Overview 62

63 Silencing All Alerts Don t ever do this! You *will* forget to turn them back on At least make sure someone is watching the dashboard! Hidden at bottom of sidebar Click Process Info 63

64 Silencing All Alerts Disable notifications Failed Nagios upgrade causes every plugin to fail Some other problem with Nagios server itself Disable event handlers (in case one is going crazy) 64

65 CUSTOM PLUGINS 65

66 Custom Plugins Nagios can monitor anything you can write a script to check Simple API: just write text to stdout and exit with a value You can write plugins in ANY language you choose! bash, python, tcl, expect perl (Nagios has embedded perl interpreter for speed) C, C++ Huge collection of plugins available at: Be wary of some community plug-ins! Test first!!! 66

67 The API exit code: 0: OK - everything is normal. Carry on. 1: Warning - a metric s warning threshold was exceeded 2: Critical - service not running or a metric s critical threshold was exceeded 3: Unknown - plugin was unable to run properly stdout: Brief summary of the state of the service. Generally included in pages so keep length short Not mandatory; does not affect Nagios s interpretation of the service state Can include performance data 67

68 Performance Data Metrics about the state of the service Can be used to generate graphs showing trends, etc. Performance data processing requires some external add-on like PNP4Nagios 68

69 Performance Data Perfdata is separated from main plugin output by vertical bar character (pipe symbol) Format: name=value[unit];[w thresh];[c thresh];[min];[max]! Minimal output: name=value Examples: bytes= temp=69.5f;78;85;50;100 mailq=34messages;200;400 disk_root=85% 69

70 Example Plugin - Time Machine Let s write a plugin to check that a machine is being backed up by Time Machine Figure out how to check this via command line tmutil latestbackup /Volumes/Time Machine Backups/Backups.backupdb/agrajag/ We can parse this last part into a timestamp with date -jf '%Y-%m-%d-%H%M%S' ' ' +%s! Beware differences between GNU date and BSD date Compare to date +%s! Simple arithmetic to see if we ve crossed a threshold 70

71 Minimal Plugin - Time Machine #!/bin/bash! <read warning and critical thresholds>!! now=$(date +%s)! latestbackup=$(tmutil latestbackup sed -e 's/.*\///' )! latest=$(date -jf '%Y-%m-%d-%H%M%S' "$latestbackup" +%s)! delay=$(( now - latest ))! if [ $delay -gt $critical ]; then state=2! elif [ $delay -gt $warning ]; then state=1! else state=0! fi! exit $state 71

72 TROUBLESHOOTING 72

73 Basic troubleshooting resources Logs!!!! You might have to enable or raise the log level Verbose / debug output --debug --verbose -v or -vvvv options run servers in the foreground Watch the traffic or service output Telnet Packet capture (tcpdump, wireshark, etc.) Examine processes ps, lsof, netstat, strace Network / DNS tests ping, dig, traceroute, nmap 73

74 It s Log, Log, Log! /var/log/* Common log tools grep tail head less Not sure, which file to look at? Sort by modification time: ls -lt! Learn about regular expressions for searching with grep 74

75 Common Log Actions Watch a logfile in real-time tail -f logfile! Print the last 100 lines tail -100 logfile! Search for specific text - case insensitive grep -i -e some string logfile! Exclude lines with specific text grep -v -e some string -e other string logfile! Truncate (clear) a log file without disturbing file permissions > logfile 75

76 Common Log Regular Expressions ^string string$ program: String at beginning of line String at end of line Find a program in syslog file (colon) 8\.8\.8\.8 IP addresses must escape the dot progname.*error prog.*(error warn) Line that mentions a program and error Find error or warn on the line 76

77 Processes ps aux List all processes on the system and the user it s running as lsof List all open files and which process / user is using it Great for finding why you can t unmount a filesystem Make sure a process is using the files you think it is lsof also lists listening network processes lsof grep IPv4 77

78 Netstat Display info about network sockets and connections BSD (Mac) and GNU (Linux) versions differ significantly Linux: -l -p Show listening processes Show process PID! Both: -n Show numerical output - skip DNS resolution 78

79 Query DNS with dig dig is part of the OSX base and most Linux distros Query DNS without involving host s resolver or cache Query for exact record type Query a specific DNS server Use dig to ensure that your name servers are answering queries properly Compare to output of host <query> to spot resolver problems!! dig [@<server>] <query> [<record type>] 79

80 Common dig Queries : jmalone@gargravarr; ; <<>> DiG P1 ; (1 server found)! ;; global options: +cmd! ;; Got answer:! ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 41627! ;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 0!! ;; QUESTION SECTION:! ; IN A!! ;; ANSWER SECTION:! IN CNAME quordlepleen.cv.nrao.edu.! quordlepleen.cv.nrao.edu IN A !! 80

81 Common dig Queries ; <<>> DiG P1 nrao.edu in ns! ; (1 server found)! ;; global options: +cmd! ;; Got answer:! ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 1187! ;; flags: qr rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 0!! ;; QUESTION SECTION:! ;nrao.edu. IN NS!! ;; ANSWER SECTION:! nrao.edu IN NS cv3.cv.nrao.edu.! nrao.edu IN NS zia.aoc.nrao.edu.! nrao.edu IN NS sadira.gb.nrao.edu. 81

82 Telnet & Openssl We can use telnet and openssl to open a TCP connection to a service on any port on communicate directly to the service as if we were a client program Use telnet for un-encrypted services telnet <hostname> <port>! <port> can be numeric or name from /etc/services Use openssl for an SSL- or TLS-protected service openssl s_client -connect <host>:<port> 82

83 Telnet Example: web server : jmalone@gargravarr; telnet 80! Trying ! Connected to Escape character is '^]'.! GET / HTTP/1.0!! HTTP/ OK! Date: Mon, 17 Mar :03:31 GMT! Expires: -1! Cache-Control: private, max-age=0! Content-Type: text/html; charset=iso ! Set-Cookie: PREF=ID=a725ac8...; expires=wed, 16-Mar :03:31 GMT; path=/; domain=.google.com! Set-Cookie: NID=67=O11S-VZ1...; expires=tue, 16-Sep :03:31 GMT; path=/; domain=.google.com; HttpOnly! Server: gws!! <!doctype html><html itemscope="" itemtype=" <head><meta content="search the world's information, including webpages, images, videos and more... 83

84 Same thing but HTTPS $ openssl s_client -connect CONNECTED( )! depth=2 /C=US/O=GeoTrust Inc./CN=GeoTrust Global CA! verify error:num=20:unable to get local issuer certificate! verify return:0! ---! Certificate chain! 0 s:/c=us/st=california/l=mountain View/O=Google Inc/CN= i:/c=us/o=google Inc/CN=Google Internet Authority G2! 1 s:/c=us/o=google Inc/CN=Google Internet Authority G2! i:/c=us/o=geotrust Inc./CN=GeoTrust Global CA! 2 s:/c=us/o=geotrust Inc./CN=GeoTrust Global CA! i:/c=us/o=equifax/ou=equifax Secure Certificate Authority! ---! (certificate block)! subject=/c=us/st=california/l=mountain View/O=Google Inc/CN= issuer=/c=us/o=google Inc/CN=Google Internet Authority G2! ---! SSL handshake has read 3238 bytes and written 444 bytes! ---! New, TLSv1/SSLv3, Cipher is RC4-SHA! Server public key is 2048 bit! Secure Renegotiation IS supported! Compression: NONE! Expansion: NONE! SSL-Session:! Protocol : TLSv1! 84

85 SMTP Simple Mail Transfer Protocol RFC 821, 2821, 5321 Simple vocabulary: HELO! Note: the angle brackets are included around the address MAIL FROM: <address>! RCPT TO: <address>! DATA! QUIT 85

86 SMTP Example : jmalone@gargravarr; telnet smtp.cv.nrao.edu smtp! Connected to cv3.cv.nrao.edu.! Escape character is '^]'.! 220 cv3.cv.nrao.edu ESMTP Sendmail cv-ws-8.12; Sun, 16 Mar 2014! HELO gargravaar.local! 250 cv3.cv.nrao.edu Hello va-xx.dhcp.xx [184.xx], pleased to meet you! MAIL FROM: <user@example.com>! <user@example.com>... Sender ok! RCPT TO: <jmalone@nrao.edu>! <jmalone@nrao.edu>... Recipient ok! DATA! 354 Enter mail, end with "." on a line by itself! Subject: This is a test!! Hi. This is a test message.!.! s2h2udah Message accepted for delivery! QUIT! cv3.cv.nrao.edu closing connection! Connection closed by foreign host. 86

87 IMAP Example $ openssl s_client -connect hornet:imaps! * OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE AUTH=PLAIN AUTH=LOGIN AUTH=CRAM-MD5 AUTH=X-PLAIN-SUBMIT] Dovecot ready.!. login nagios foobar!. OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE SORT SORT=DISPLAY THREAD=REFERENCES THREAD=REFS MULTIAPPEND CATENATE UNSELECT ] Logged in!. list "" *! * LIST (\HasNoChildren) "." "Deleted Messages"! * LIST (\HasNoChildren) "." "Drafts"! * LIST (\HasNoChildren) "." "Junk"! * LIST (\HasNoChildren) "." "Sent Messages"! * LIST (\HasNoChildren) "." "INBOX"!. OK List completed.!. logout! * BYE Logging out!. OK Logout completed.! closed 87

88 IMAP Example: Bad auth $ openssl s_client -connect hornet:imaps! * OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE AUTH=PLAIN AUTH=LOGIN AUTH=CRAM-MD5 AUTH=X-PLAIN-SUBMIT] Dovecot ready.!. login nagios wrongpassword!. NO [AUTHENTICATIONFAILED] Authentication failed.!! Cannot tell if problem is username or password Check server logs bash-3.2# grep dovecot /var/log/system.log! Jul 6 14:35:55 hornet dovecot[1533]: auth: Error: od(nagios, ): Credentials could not be verified, username or password is invalid.! bash-3.2# id nagios! uid=1025(nagios) gid=20(staff) groups=20(staff), 403(com.apple.sharepoint.group.2),402(com.apple.sharepoint.group. 1),404(com.apple.sharepoint.group.3),12(everyone), 61(localaccounts) 88

89 IMAP Example: Bad auth mech $ openssl s_client -connect hornet:imaps! * OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE AUTH=LOGIN AUTH=CRAM-MD5 AUTH=X-PLAIN-SUBMIT] Dovecot ready.!. login nagios foobar!. NO [ALERT] Unsupported authentication mechanism.!!! In this case, the plain mechanism was disabled by the server. Had to enable this mech to support a legacy client that only did plaintext auth. 89

90 PACKET SNIFFING 90

91 Packet Sniffing Packet sniffing examines live network traffic on a connection Watch contents of network packets during a connection to look for problems Two main tools to cover Wireshark: graphical packet sniffing tool tcpdump: command-line packet sniffing tool Other tools available EtherApe - Linux GUI network monitor iptraf - Linux TUI connection monitor 91

92 Wireshark X11 application - requires XQuartz on Originally known as Ethereal Powerful capture filters and protocol decoders Warning: In the past, bugs in the protocol decoders have led to remote attack vectors when running Wireshark as root. You have been warned. Can also capture traffic on one machine using tcpdump and load the cap file into Wireshark on another 92

93 Start new Wireshark Overview capture 93

94 Select Interface Wireshark Overview Enter capture filter Start capture 94

95 Capture Filters Capture filters are different from display filters Capture filters use libpcap syntax (same as tcpdump)! host Only capture packets involving this host! host Exclude specific IP address Useful for chatty hosts you don t care about dst port 80 arp - Traffic send to Any traffic involving the www port - Just watch ARP traffic (look for storms) 95

96 Capture Filters Can use logical operators to AND or OR filters dst or dst host and host host and (host or host )! tcp portgrange Traffic from privileged ports! 96

97 Different from capture filters Display Filters ip.addr == Any traffic involving host! ip.addr == Traffic not involving host NOTE: don t use ip.addr!= not the same filter ip.dst == tcp.port == 80 arp - Traffic destined for Any non-ssl web traffic - Any ARP traffic 97

98 Display Filters Can use logical operators to AND or OR filters arp or tcp.port == 22 ip.addr == and tcp.port == 53 ip.addr == and (tcp.port==53 or tcp.port==5353) Other types of matching ip.addr contains http.request.uri matches jsp$ 98

99 tcpdump Command-line packet sniffer for unix systems Can filter captured traffic based on IP, port, protocol, etc. Can save pcap files for use with other decoder software Use options -s 0 -w <filename> The -s 0 tells it to snarf the whole packet not just the headers scp file back to your machine and analyze with Wireshark Uses the same syntax for capture filters as Wireshark and other libpcap applications. 99

100 THANK YOU! 100