TROUBLESHOOTING FIREWALLS

Transcription

1 TROUBLESHOOTING FIREWALLS SESSION 1 Agenda Understanding the Concepts PIX and FWSM Troubleshooting Tools PIX and FWSM Common Issues Cisco IOS Firewall Concepts Cisco IOS Firewall Common Issues and Troubleshooting 2 Printed in USA.

2 Agenda Understanding the Concepts PIX and FWSM Troubleshooting Tools PIX and FWSM Common Issues Cisco IOS Firewall Concepts Cisco IOS Firewall Common Issues and Troubleshooting 3 How the PIX Processes a Packet Private Public Entering PIX Packet Processed Against ASA Leaving PIX Adaptive Security Algorithm Randomize Sequence Numbers Xlate and Connection Objects Stateful Inspection Security Levels Other rules 4 Printed in USA.

3 PIX Packet Processing Recv No No Existing ACL Pkt Conn Permit Yes Yes 4 Match xlate Yes L7 NAT Sec Checks NAT IP Header Egress Interface 5 6 Drop 1. Receive packet 2. Existing connection? 3. Permit by Inbound ACL on interface? 4. Match translation rule (nat, static) 5. NAT embedded IP and perform security checks/randomize sequence number 6. NAT IP header 7. Pass packet to outgoing interface 8. Layer 3 route lookup? 9. Layer 2 next hop? 10. Transmit packet L3 Route Yes L2 Addr Yes Xmit Pkt 7 Drop Drop 5 PIX Release Process Interim Images 6.0(2.100) 6.0(2.101) 6.0(2.122) Bug Fixes Committed to Multiple Trains CSCdv (2) 6.0(3) 6.0(4) Maintenance Trains 6.1(2) 6.1(3) 6.1(4) 6.1(5) 6.2(2) 6.2(3) 6.3(2) 6.3(3) 6.0(1) 6.1(1) 6.2(1) 6.3(1) Major Feature Releases 6 Printed in USA.

4 Agenda Understanding the Concepts PIX and FWSM Troubleshooting Tools PIX and FWSM Common Issues Cisco IOS Firewall Concepts Cisco IOS Firewall Common Issues and Troubleshooting 7 PIX and FWSM Troubleshooting Tools Syslog and ICMP Packet Capture Show Commands Output Interpreter PDM (3.0) 8 Printed in USA.

5 What Are Syslogs? Archival methods Primary mechanism to record traffic TO and THROUGH the PIX One of the best troubleshooting tools Archival Methods Debugging Methods Console Syslog Server Syslog SNMP Internet SSH Client SNMP Server Buffered 9 Syslog Level vs. Number of Messages Logged Log Level Description Emergencies Alerts Critical Errors Warnings Notifications Informational Debugging # of Messages (Sum) 0 41 (41) 21 (62) 74 (136) 56 (192) 21 (213) 95 (308) 15 (323) * Based on PIX Printed in USA.

6 Configuring and Using Syslog Define internal syslog server Log error (level 3) messages and below to syslog server Log alerts (level 1) to the internal buffer You must enable logging Log to a SSH/Telnet session as necessary pixfirewall (config) # logging host inside pixfirewall (config) # logging trap 3 pixfirewall (config) # logging buffered 1 pixfirewall (config) # logging on pixfirewall (config) # logging monitor 4 11 What Are Modifiable Syslog Levels [no] logging message <syslog_id> level <level> Modifiable Syslog Levels Allows one to move any syslog message to any level Example: You want to record what commands are being executed on the PIX; message records this information, but by default it is at level 7 (Debug) %PIX : User johndoe executed cmd: show run The problem is we don t want to log all 322 other syslogs that are generated at debug level Levels: 0 Emergency 1 Alert 2 Critical 3 Errors 4 Warnings 5 Notifications 6 Informational 7 Debugging 12 Printed in USA.

7 How to Create Modifiable Syslog Levels [no] logging message <syslog_id> level <level> SOLUTION: Lower syslog message to level 3 (Error) pix(config)# logging message level 3 - or pix(config)# logging message level error Now our syslog looks as follows: %PIX : User johndoe executed cmd: show run To restore the default syslog level: pix(config)# no logging message level error - or pix(config)# logging message level 7 13 Notes on ICMP 1 3 Bob Ping Inside 2 DMZ Ping Outside Internet Ping 1 User Can Only Ping the Local Interface of the PIX 2 User Cannot Ping Remote Interface of the PIX 3 ICMP Echo-Replies Must be Explicitly Permitted thru the PIX 14 Printed in USA.

8 Debug ICMP Trace Private Network Internet 1 User Able to Access Internet? No 2 Can Aser Access Private Network? Yes 3 Check Syslog Messages? 4 Test IP Connectivity Ping 5 pixfirewall (config) # debug icmp trace EXAMPLE OF DEBUG ICMP TRACE TO TEST SUCCESSFUL IP CONNECTIVITY ICMP echo-request from inside: to ID=3239 seq=4369 length=80 ICMP echo-request: translating inside: to outside: ICMP echo-reply from outside: to ID=3239 seq=4369 length=80 ICMP echo-reply: untranslating outside: to inside: Disabling Debug Commands undebug all In 6.3 You Can Now Turn Off All Debugs Globally by Issuing no debug all" and "undebug all or un all for Short pix(config)# show debug debug icmp trace debug sip pix(config)# un all pix(config)# show debug pix(config)# 16 Printed in USA.

9 PIX and FWSM Troubleshooting Tools Syslog and ICMP Packet Capture Show Commands Output Interpreter PDM (3.0) 17 Packet Capture capture <capture-name> [access-list <acl-name>] [buffer <buf-size>] [ethernet-type <type>] [interface <if-name>] [packet-length <bytes>] Capture command first introduced in PIX 6.2; it depreciates the debug packet command Capture sniffs packets on an interface that match an ACL Key steps: Create an ACL that will match interesting traffic Define the capture and bind it to an access-list and interface View the capture on the PIX, or copy it off in pcap format 18 Printed in USA.

10 Packet Capture (Cont.) Traffic can be captured both before and after it passes through the PIX; one capture on the Inside interface, one capture on the Outside interface Capture buffer saved in RAM (default size 512kb) Default is to stop capturing when buffer is full Default packet-length is 68 bytes (typically set to 1500) Copy captures off via TFTP or HTTPS Capture In Inside Capture Out Outside 19 Packet Capture: Example No. Time Source Destination Protocol Info Example: User on the Inside with TCP an 4511>http[SYN] IP of Seq= is having Ack= a problem accessing Cisco.com TCP( ); http>4511[syn,ack] the Seq= user is getting NATed to TCP 4511>http[ACK] Seq= HTTP GET /HTTP/1.1 Step 1: Create ACL for both Inside and Outside Interface Access-list 100 permit tcp host host eq 80 Access-list 100 permit tcp host eq 80 host Access-list 101 permit tcp host host eq 80 Access-list 101 permit tcp host eq 80 host Step 2: Create Captures on Both Inside and Outside Interface capture out access-list 100 interface outside packet-length 1500 capture in access-list 101 interface inside packet-length 1500 Step 3: Have Inside User Access Step 4: Copy the Captures off to a TFTP Server copy capture:out-web tftp:// pcap copy capture:in-web tftp:// pcap OR copy using https: 20 Printed in USA.

11 PIX and FWSM Troubleshooting Tools Syslog and ICMP Packet Capture Show Commands Output Interpreter PDM (3.0) 21 An Introduction to Show Commands Show Perfmon Displays Current and Past Info Related to the PIX Show Blocks CLI Tool Used to Extract Information from the PIX for Information or Troubleshooting Show Memory Show cpu usage Show xlate <detail> Show conn <detail> Show interface Show traffic Used to Monitor the Health of the PIX and Draw a Baseline for your Network Show Processes 22 Printed in USA.

12 Show Conn and Show Conn <detail> pixfirewall (config) # show connection 2 in use, 2 most used TCP out :23 in :1026 idle 0:00:22 bytes 1774 flags UIO UDP out :31649 in :1028 idle 0:00:14 bytes 540 flags d pixfirewall (config) # show connection detail 2 in use, 2 most used Flags: A awaiting inside ACK to SYN, a awaiting outside ACK to SYN, B initial SYN from outside, D DNS, d dump, E outside back connection, F outside FIN, f inside FIN, G group, H H.323, I inbound data, M SMTP data, m SIP media, O- outbound data, P inside back connection, q SQL*Net data, R outside acknowledged FIN, R UDP RPC, r inside acknowledged FIN, S awaiting inside SYN, s awaiting outside SYN, T SIP, t SIP transient, U up TCP outside: /23 dmz: /1026 flags UIO UDP outside: /31649 dmz: /1028 flags d 23 Connection Termination Reasons %PIX : Teardown TCP connection number for interface_name:real_address/real_port to interface_name:real_address/real_port duration time bytes number [reason] [(user)] Reset-I Reset-O TCP FINs FIN Timeout SYN Timeout Xlate Clear Deny SYN Control Uauth Deny Reset was from Inside Reset was from Outside Normal Close Down Sequence Force Termination After 15 Seconds Force Termination After 2 Min Command Line Removal Terminate by Application Inspection Back Channel Initiation from Wrong Side Deny by URL Filter Unknown Catch All Error 24 Printed in USA.

13 Show Xlate and Show Xlate <detail> pixfirewall (config) # show xlate 3 in use, 3 most used PAT Global (0) Local ICMP id 340 PAT Global (1024) Local (1028) PAT Global (1024) Local (516) pixfirewall (config) # show xlate detail 3 in use, 3 most used Flags: D DNS, d dump, I identity, I inside, n no random, o outside, r portmap, s static TCP PAT from inside: /1026 to outside: /1024 flags ri UDP PAT from inside: /1028 to outside: /1024 flags ri ICMP PAT from inside: /21505 to outside: /0 flags ri 25 Xlate Flags FLAG DESCRIPTION s Static Translation Slot d Dump Translation Slot on Next Clearing Cycle r Port Map Translation n No Randomization of TCP Sequence Number o Outside Address Translation i Inside Address Translation D DNS A RR Rewrite I Identity Translation from NAT 0 26 Printed in USA.

14 Example Connection through PIX 1. PIX receives an initial SYN packet from the Inside. The SYN is permitted by the access-list, a translation (xlate) is built up, and the connection is also created with the flags "saa". 2. The Outside device responds to the SYN packet with a SYN+ACK. The connection flags are updated to reflect this, and now show "A". 3. The Inside device responds to the SYN+ACK with an ACK and this completes the TCP 3-way handshake, and the connection is now considered "up" (U flag). 4. The Outside device sends the first data packet. The connection is updated and an "I" is added to the flags to indicate the PIX received Inbound data on that connection. 5. Finally, the Inside device has sent a data packet and the connection is updated to include the "O" flag SYN+ACK Data 42 Connection Flags saa UIO Inside Outside Server Client 27 Example Connection Teardown 1. PIX receives a FIN packet from the Inside. As the FIN passes through the PIX, it updates the connection flags by adding an f to indicate that the FIN was received on the Inside interface. 2. The Outside device immediately responds to the FIN packet with a FIN+ACK. The connection flags are updated to reflect this, and now show UfFR". 3. The Inside device responds to the FIN+ACK with a final ACK and the PIX tears down the connection. Thus, there are no more connection flags, because the connection no longer exists. 31 FIN+ACK 2 Connection Flags UfUfFR UfFRr Inside Outside Server Client 28 Printed in USA.

15 Show CPU Usage First introduced in PIX OS version 6.0(1) Under normal conditions the PIX CPU should stay below 30% (baseline as per Network); if the CPU reaches 100% the PIX will start dropping packets The show cpu usage command displays the CPU over time as a running average An Example pixfirewall# show cpu usage CPU utilization for 5 seconds = 1%; 1 minute: 2%; 5 minutes: 1% A Note The Percentage Usage Prints as NA (Not Applicable) if the Usage is Unavailable For the Specified Time Interval; this Can Happen if the User Asks For CPU Usage Before the 5-second, 1-minute, or 5-minutes 29 Show Traffic The Show Traffic Command Displays the Traffic, in Packets and in Bytes, Out Each Interface of the PIX An Example pixfirewall# show traffic outside: received (in secs): packets bytes 2370 pkts/sec bytes/sec transmitted (in secs): packets bytes 2093 pkts/sec bytes/sec inside: received (in secs): packets bytes 2097 pkts/sec bytes/sec transmitted (in secs): packets bytes 2363 pkts/sec bytes/sec 30 Printed in USA.

16 Show Blocks The show blocks command, along with the show cpu usage command, are useful in determining whether the PIX is being overloaded The blocks are internal storage locations, similar to queues on a router; a packet is stored in a block until the PIX can process it and place it on the outbound interface xmit queue An Example pixfirewall# show blocks SIZE MAX LOW CNT What are these blocks used for? 256 = Stateful Failover, Syslog, TCP module 1550 = Ethernet Packets, buffering url filtered packets = Only used for the Gig Ethernet cards 31 Show Local-Host The show local-host command displays the translation and connection slots for all local hosts The clear local-host command stops traffic on all local hosts The clear local-host <ip_address> command stops traffic on the local host specified by its IP address An Example pixfirewall# show local-host Interface inside: 822 active, 823 maximum active, 0 denied local host: < >, TCP connection count/limit = 0/unlimited TCP embryonic count = 0 TCP intercept watermark = unlimited UDP connection count/limit = 63/unlimited AAA: Xlate(s): PAT Global (41166) Local (39075) Conn(s): UDP out :8943 in :63556 idle 0:01:31 flags - 32 Printed in USA.

17 Show Tech-Support Enhancements (6.3) The show tech output was enhanced to include some additional show commands that can be used to troubleshoot memory and performance issues On the right are the commands included in the show tech output; note: They are in order; (New commands are in red) show version show clock show memory show conn count show xlate count show blocks show interface show cpu usage show process show failover show traffic show perfmon show running-config 33 Show Output Filters show <cmd> begin include exclude grep [-v] <regular_exp> Output Filters Have Been Added to PIX 6.3, Similar to the Ones in IOS; to Use them, at the End of Show <command>, Use the Pipe Character Followed By: begin include exclude grep [-v] <regular_exp> to Filter the Show Output Begin Start Displaying the Output Beginning at the First Match of the RegEx, and Continue to Display the Remaining Output include Display any line that matches the RegEx exclude Display any line that does not match the RegEx grep same as include grep v same as exclude 34 Printed in USA.

18 PIX and FWSM Troubleshooting Tools Syslog and ICMP Packet Capture Show Commands Output Interpreter PDM (3.0) 35 Output Interpreter GREAT TOOL TO CATCH COMMON CONFIGURATION ERRORS Select the Output In Question Paste the Output 36 Printed in USA.

19 Output Example Snip of Output Example Of Messages 37 PIX and FWSM Troubleshooting Tools Syslog and ICMP Packet Capture Show Commands Output Interpreter PDM (3.0) 38 Printed in USA.

20 Using PDM for Monitoring Device Information Interface Statistics Active VPN Tunnels Current System Utilization: CPU, Memory, Traffic 39 Using PDM for Monitoring PDM Can Graph Up to 4 Different Graphs at Once Great for Monitoring Trends 40 Printed in USA.

21 Agenda Understanding the Concepts PIX and FWSM Troubleshooting Tools PIX and FWSM Common Issues Cisco IOS Firewall Concepts Cisco IOS Firewall Common Issues and Troubleshooting 41 Common Issues PIX Common Issues Accessing the Internet Accessing internal network from the Internet Issues with traffic between interfaces PIX not redirecting Failover Password recovery High CPU Out-of-memory issues FWSM Common Issues Configuration issues Passing traffic outbound Understanding failover 42 Printed in USA.

22 Accessing the Internet x INSIDE OUTSIDE x Internet.2 Web Server Problem: Accessing the Internet Troubleshooting: P ermissions T ranslation R outing 43 Permissions (Access-Lists) x INSIDE OUTSIDE x Internet.2 Web Server Verify if any Access-lists Are Applied Pixfirewall (config) # access-list acl permit tcp host any eq www Pixfirewall (config) # access-group acl in interface inside Note If you Have an Access-list Applied on the Inside Interface, Check to Make Sure Traffic is Permitted Outbound; Remember, there is an Implicit Deny at the End of an Access-list 44 Printed in USA.

23 Translation x INSIDE OUTSIDE x Internet.2 Web Server 1 Verified Permissions Verify Translation Commands global (outside) netmask nat (inside) global (outside) 1 interface nat (inside) static (inside,outside) netmask Routing INSIDE.3.1 OUTSIDE.1.2 Internet x x x 1 Verified Permissions 2 Verified Translation Verify Route Commands on the PIX Check to make sure the pix has the correct default gateway pixfirewall(config)# route outside If trying to access the internet from behind a layer 3 device, verify the PIX has a route to that network Pixfirewall (config)# route inside Printed in USA.

24 Common Issues PIX Common Issues Accessing the Internet Accessing internal network from the Internet Issues with traffic between interfaces PIX not redirecting Failover Password recovery High CPU Out-of-memory issues FWSM Common Issues Configuration issues Passing traffic outbound Understanding failover 47 Accessing Internal Network From Internet Web Server x INSIDE OUTSIDE x Internet Problem: Internal web server not accessible to users on the Internet Troubleshooting: P ermissions T ranslation R outing 48 Printed in USA.

25 Permissions (Access-Lists) Web Server x INSIDE OUTSIDE x Internet Traffic has to be Explicitly Allowed into the PIX from a Lower Security to a Higher Security CHECK TO MAKE SURE YOU HAVE PERMITTED INTERESTING TRAFFIC EXPLICITLY Pixfirewall (config) # access-list acl permit tcp any host eq http Pixfirewall (config) # access-group acl in interface outside If you have an access-list applied, check to make sure traffic is permitted inbound Remember, there is an implicit deny at the end of an access-list 49 Translation Web Server x INSIDE OUTSIDE x Internet A Static Translation is Required to Pass Inbound Traffic Verify Translation Commands Pixfirewall(config) #static(inside,outside) netmask Example of a Syslog Message With No Static Defined : No translation group found for tcp src outside: /35550 dst inside: /80 50 Printed in USA.

26 Routing Issues Web Server x INSIDE OUTSIDE x Internet Check to make sure the PIX has the correct default gateway pixfirewall(config)# route outside Verify the PIX has a route to the internal network pixfirewall(config)# route inside Other Issues to Consider Confirm default gateway on your Web Server Verify your layer 3 device is routing correctly 51 Common Issues PIX Common Issues Accessing the Internet Accessing internal network from the Internet Issues with traffic between interfaces PIX not redirecting Failover Password recovery High CPU Out-of-memory issues FWSM Common Issues Configuration issues Passing traffic outbound Understanding failover 52 Printed in USA.

27 Issues with Traffic Between Interfaces DMZ (30) 1 INSIDE (100) OUTSIDE (0) 3 DMZ 1 (40) 2 1 Static and Access-List/Conduit 2 3 Static and Access-List/Conduit Static or a NAT/Global Statement 53 Common Issues PIX Common Issues Accessing the Internet Accessing internal network from the Internet Issues with traffic between interfaces PIX not redirecting Failover Password recovery High CPU Out-of-memory issues FWSM Common Issues Configuration issues Passing traffic outbound Understanding failover 54 Printed in USA.

28 PIX Is NOT Redirecting INSIDE OUTSIDE x x x Internet 1 PIX Will Not Handle Redirects 2 Change User s Default Gateway to Be the Layer 3 Device 3 Modify the Layer 3 Device s Default Gateway to Be the PIX 55 Common Issues PIX Common Issues Accessing the Internet Accessing internal network from the Internet Issues with traffic between interfaces PIX not redirecting Failover Password recovery High CPU Out-of-memory issues FWSM Common Issues Configuration issues Passing traffic outbound Understanding failover 56 Printed in USA.

29 PIX Failover Active/Standby vs. Primary/Secondary Serial Failover/LAN Failover Stateful Failover A Failover ONLY occurs when either PIX determines the Standby PIX is healthier than the Active PIX Both PIXes swap MAC and IP Addresses when a failover occurs Active Unit Stateful Serial Lan FO Standby Unit Level 1 syslogs will give reason of failover 57 Commands to Verify Failover Is Active Pixfirewall (config)# show failover Failover On Cable status: Normal Reconnect timeout 0:00:00 This host: Primary - Standby (Failed) Active time: 7140 (sec) Interface 0 ( ): Normal Interface 1 ( ): Normal Other host: Secondary - Active Active time: 30 (sec) Interface 0 ( ): Normal Interface 1 ( ): Normal 58 Printed in USA.

30 Why Will Failover Happen? Power Failure Network Activity Test ARP Test Link Test Ping Test Failover Cable Failure 59 Using Syslogs to Troubleshoot Failover No Failover Hello Seen on Serial Cable for 30 + Seconds; This Ensures that Failover is Running Properly on the Other PIX An Interface Did not Pass One of the 4 Failover Tests (Link up, Interface Traffic, ARP Test, Broadcast Ping) No Proper ACK for 15+ Seconds After a Command has Been Sent on the Serial Cable Message on Secondary when Primary Did Not Respond %PIX : (Secondary) No response from other firewall (reason code = 3) Example of Syslogs when Primary Was Active, and It Lost One of Its Interfaces %PIX : Line protocol on Interface intf2, changed state to down %PIX : (Primary) Link status Down on interface 1 %PIX : (Primary) Switching to STNDBY interface check, mate is healthier %PIX : (Secondary) Switching to ACTIVE mate want me Active 60 Printed in USA.

31 Common Issues PIX Common Issues Accessing the Internet Accessing internal network from the Internet Issues with traffic between interfaces PIX not redirecting Failover Password recovery High CPU Out-of-memory issues FWSM Common Issues Configuration issues Passing traffic outbound Understanding failover 61 PIX Password Recovery Password recovery can only be performed by uploading the Password Recovery utility to the PIX from Monitor mode via TFTP Password recovery will also remove any AAA commands A password recovery utility is created for each major PIX release (6.1, 6.2, 6.3 ) however, the utilities are backwards compatible EX: np63.bin for PIX 6.3 and prior releases 62 Printed in USA.

32 Example of Password Recovery monitor> interface 1 Inside TFTP monitor> address monitor> file np63.bin monitor> gateway monitor> server monitor> tftp tftp np63.bin@ via Received bytes Do you wish to erase the passwords? [yn] y The following lines will be removed from the configuration: enable password 8Ry2YjIyt7RRXU24 encrypted passwd 2KFQnbNIdI.2KYOU encrypted aaa authentication serial console LOCAL aaa authentication telnet console LOCAL aaa authentication ssh console LOCAL aaa authentication enable console LOCAL Do you want to remove the commands listed above from the configuration? [yn] y Passwords and aaa commands have been erased. Rebooting.. 63 Common Issues PIX Common Issues Accessing the Internet Accessing internal network from the Internet Issues with traffic between interfaces PIX not redirecting Failover Password recovery High CPU Out-of-memory issues FWSM Common Issues Configuration issues Passing traffic outbound Understanding failover 64 Printed in USA.

33 High CPU Usage on the PIX Problem: PIX CPU Running Very High A Quick Overview of the Show Processes Command Number of msec this Process Has been on the CPU The Name of the Process pixfirewall(config)# show processes PC SP STATE Runtime SBASE Stack Process Hsi 001eab19 008a5a a4aec 3628/4096 arp_timer Lsi 001f00bd 00a28dbc a27e /4096 FragDBGC Lwe 00119abf 02d280dc 0055b d /4096 dbgtrace Lwe 003e d2a26c 00557dd d /8192 Logger Crd 001e26fb c 00557d / poll Lsi 00300a29 04c0f c0e57c 3944/4096 xlate clean For more information on the output of the show processes command, see: 65 High CPU Usage on the PIX Step 1: Determine What Process Is Eating the CPU Take the Difference in Output of Two Show Processes over a Period of Time The Following Output Was a Diff of the Processes Taken 1 Minute Apart Process_Name Runtime (msec) Logger pix/intf poll 9250 i82543_timer 4180 i82542_timer 2230 In one minute, these Processes Account for 44 Seconds of CPU Time ~ 73% The Interface Polling Processes Always Run, and Are not Counted in the CPU Usage 66 Printed in USA.

34 High CPU Usage on the PIX Step 2: Focus on the Processes with High CPU Time Logging Is Taking up Much of the CPU, Let s Review What We Have Configured to Log This is Cumulative pixfirewall(config)# show log Since the PIX Was Syslog logging: enabled Last Rebooted Standby logging: disabled Console logging: disabled Monitor logging: disabled Buffer logging: level alerts, 0 messages logged Trap logging: level warnings, messages logged Logging to lab History logging: disabled... pixfirewall(config)# show log Syslog logging: enabled Buffer logging: level alerts, 0 messages logged Trap logging: level warnings, messages logged Logging to lab Notice the Change Over a Few Minutes 67 High CPU Usage on the PIX Syslog Server Is Controlled by a Different Group Enable Buffered Logging to Same Level as Syslog Server, and Examine the Buffered Messages pixfirewall(config)# show log Buffer logging: level warnings, messages logged Trap logging: level warnings, messages logged Logging to lab PIX s Interface Address : IDS:2001 ICMP unreachable from to on interface lab : IDS:2001 ICMP unreachable from to on interface lab : IDS:2001 ICMP unreachable from to on interface lab : IDS:2001 ICMP unreachable from to on interface lab : IDS:2001 ICMP unreachable from to on interface lab : IDS:2001 ICMP unreachable from to on interface lab 68 Printed in USA.

35 High CPU Usage on the PIX EXAMINE IDS CONFIGURATION pixfirewall(config)# show run grep audit ip audit name IDS info action alarm ip audit interface lab IDS Syslog service was down on the syslog server ICMP Unreachable was generated by syslog server for each syslog message the PIX sent it PIX s IDS configuration also logged every ICMP Unreachable message, creating the exponentially increasing problem Syslog Server LAB OUTSIDE ICMP Unreachable Syslog Message IDS Syslog Message 69 High CPU Usage on the PIX SOLUTION: Bring back up syslog service on server Take server offline Configure PIX to not log IDS ICMP Unreachable messages ip audit signature 2001 disable -or - no logging message pixfirewall# show run grep signature ip audit signature 2001 disable pixfirewall# show cpu usage CPU utilization for 5 seconds = 2%; 1 minute: 50%; 5 minutes: 99% 70 Printed in USA.

36 High CPU Usage on the PIX SUMMARY: Examine the DIFF of two show processes taken over a one minute interval Find the process taking up the highest amount of CPU (excluding the polling processes) Take actions to lower that processes CPU time Re-examine the CPU output, and repeat as necessary 71 Common Issues PIX Common Issues Accessing the Internet Accessing internal network from the Internet Issues with traffic between interfaces PIX not redirecting Failover Password recovery High CPU Out-of-memory issues FWSM Common Issues Configuration issues Passing traffic outbound Understanding failover 72 Printed in USA.

37 Out of Memory PROBLEM: Users are unable to access the Internet No new connections are working; All old (long lived) connections continue to work Step 1: Check the Syslogs %PIX : Memory allocation Error %PIX : Memory allocation Error Step 2: Check the Amount of Free Memory Available Hardware: PIX-515E, 64 MB RAM pixfirewall# show memory Free memory: bytes Used memory: bytes Total memory: bytes 73 Out of Memory Step 3: What Eats Up Memory (RAM) on the PIX? PIX image (run from RAM) Configuration IPSec Database Xlates (translations) Connections What Can Eat Up 64MB on a PIX-515 E Step 4: Let s Check the Translations pixfirewall# show xlate 251 in use, 258 most used PAT Global (2379) Local (52716) PAT Global (2378) Local (20276) Global Local PAT Global (2255) Local (12783) PAT Global (2382) Local (39197) PAT Global (2254) Local (43543) A Small Global Pool is Used, Overloading to a PAT Address Varied Source IPs 74 Printed in USA.

38 Out of Memory Step 5: Check the Connections pixfirewall# show conn in use, most used TCP out :80 in :26749 idle 0:00:19 Bytes 312 flags OIU TCP out :80 in :32209 idle 0:00:14 Bytes 239 flags OIU TCP out :21 in :32893 idle 0:00:48 Bytes 0 flags saa TCP out :80 in :18664 idle 0:00:46 Bytes 934 flags OIU TCP out :80 in :46712 idle 0:00:17 Bytes 8394 flags OIU TCP out :135 in :62296 idle 0:00:15 Bytes 0 flags saa TCP out :80 in :51864 idle 0:00:32 Bytes 902 flags OIU TCP out :80 in :52301 idle 0:00:03 Bytes 7813 flags OIU TCP out :80 in :51532 idle 0:00:24 Bytes 3891 flags OIU TCP out :80 in :28585 idle 0:00:28 Bytes 239 flags OIU TCP out :80 in :44945 idle 0:00:48 Bytes 9717 flags OIU TCP out :80 in :19578 idle 0:00:06 Bytes 2348 flags OIU TCP out :80 in :28149 idle 0:00:47 Bytes 419 flags OIU TCP out :135 in :43703 idle 0:00:12 Bytes 0 flags saa... Q: WHY IS THE CONNECTION COUNT SO HIGH? 75 Out of Memory Take a Look at the Traffic Load pixfirewall# show traffic outside: received (in secs): 1475 packets bytes 59 pkts/sec bytes/sec transmitted (in secs): packets bytes 6704 pkts/sec bytes/sec inside: received (in secs): packets bytes 7208 pkts/sec bytes/sec transmitted (in secs): 1050 packets bytes 42 pkts/sec 4746 bytes/sec Traffic Flow Outside Inside Vast Majority of Traffic Is Coming in the Inside Interface and Going Out the Outside Interface 76 Printed in USA.

39 Out of Memory Step 6: Review What We Know and Take Action pixfirewall# show conn count in use, most used pixfirewall# show xlate count 251 in use, 258 most used Conn Count Is Very High, but xlate Count Is Low Many connections per xlate Probably one, or a few hosts, are generating the vast majority of connections Most likely due to a virus on the host(s) 77 Out of Memory Step 7: Find the Host(s) Generating All the Connections pixfirewall# show local-host include host count/limit local host: < >, TCP connection count/limit = 0/unlimited UDP connection count/limit = 0/unlimited local host: < >, TCP connection count/limit = 2/unlimited UDP connection count/limit = 0/unlimited local host: < >, TCP connection count/limit = 0/unlimited UDP connection count/limit = 0/unlimited... local host: < >, TCP connection count/limit = /unlimited UDP connection count/limit = 0/unlimited Only Show Lines that Have the Word host OR count/limit In Them Host Is Eating Up All the Connections, and They Are TCP Based Connections 78 Printed in USA.

40 Out of Memory Step 8: Now that We Found the Host, Let s Look at the Connections it Is Generating pixfirewall# show local-host Interface inside: 250 active, 250 maximum active, 0 denied local host: < >, TCP connection count/limit = /unlimited TCP embryonic count = UDP connection count/limit = 0/unlimited Xlate(s): Global Local Conn(s): TCP out :135 in :34580 idle 0:01:43 Bytes 0 flags saa TCP out :135 in :8688 idle 0:01:43 Bytes 0 flags saa TCP out :135 in :7774 Connections idle 0:01:43 to Bytes 0 flags saa TCP out :135 in :39193 idle Random 0:01:43 Bytes 0 flags saa TCP out :135 in :16462 Destinations idle 0:01:43 on Bytes 0 flags saa TCP out :135 in :30322 idle TCP/135 0:01:43 Bytes 0 flags saa TCP out :135 in :41116 idle MS Blaster 0:01:43 Bytes 0 flags saa TCP out :135 in :59163 idle 0:01:43 Bytes 0 flags saa TCP out :135 in :2978 idle 0:01:43 Bytes 0 flags saa TCP out :135 in :41589 idle 0:01:43 Bytes 0 flags saa... All Connections Are Embryonic (Refer to Slide 23) 79 Out of Memory Step 9: Limit Infected Host(s) Impact on Network PIX Provides Two Methods to Limit the Number of Connections per Host: TCP Intercept Max Connections Which One can be Used Here? TCP Intercept Won t Help Because the Source Address Is VALID Limiting the Maximum Number of Connections Each Internal Host Can Have Is the Only Option 80 Printed in USA.

41 Out of Memory Step 9: Limit Infected Host(s) Impact on Network Configure the MAX TCP Connections for NATed Hosts to be 50: pixfirewall(config)# nat (inside) NOTE: The Local-host MUST be Cleared Before the New Connection Limits Are Applied pixfirewall(config)# clear local-host pixfirewall(config)# show local-host Interface inside: 250 active, 250 maximum active, 0 denied local host: < >, TCP connection count/limit = 50/50 TCP embryonic count = 50 TCP intercept watermark = unlimited UDP connection count/limit = 0/unlimited... The Infected Host is Limited to 50 TCP Connections 81 Out of Memory Take One Last Look at the Memory and Connection Counts After Applying the TCP Connection Limit pixfirewall# show conn count 126 in use, most used pixfirewall# show memory Free memory: bytes Used memory: bytes Total memory: bytes Things Look Much Better Now Question: How Could we Configure the PIX so the Connection Limit Was only Applied to the One Host ( ) which Was Infected with the Virus? nat (inside) nat (inside) Printed in USA.

42 Still Can t Resolve It? Open a TAC Case If After Using All Your Troubleshooting Tools You Still cannot Resolve the Problem, Please Open a TAC Case: At a minimum include: Detailed problem description Output from show tech Optionally include: Syslogs captured during time of problem Sniffer traces from both interfaces using the capture command (capturing only the relevant packets, and saved in pcap format) 83 PIX/FWSM Best Practices Enable ip verify reverse-path on all interfaces Set Embryonic and Maximum connection counts on static and nat statements Configure logging to syslog server Move messages you want to see to lower levels, instead of raising logging levels and capturing messages you don t want to see Disable telnet access, use SSH for management access Enable Authentication for Management access (Console/SSH/Telnet/Enable); use TACACS+, RADIUS or LOCAL database Restrict DMZ access inbound to your Internal Networks Baseline CPU load, connection counts, xlate counts, and traffic (per interface) 84 Printed in USA.

43 Agenda Understanding the Concepts PIX and FWSM Troubleshooting Tools PIX and FWSM Common Issues Cisco IOS Firewall Concepts Cisco IOS Firewall Common Issues and Troubleshooting 85 FWSM Code Base History Feature Releases PIX 6.0(1) 6.1(1) 6.2(1) 6.3(1) FWSM 1.1(1) New Features OSPF, crashinfo, etc. 1.1(2) Maintenance Releases Feature Releases 1.1(3) Port Features Bug Fixes 2.2(1) New Features Virtual FW, Transparent FW, etc. 86 Printed in USA.

44 FWSM Bug Fix Process Feature Releases PIX 6.0(1) 6.1(1) 6.2(1) 6.3(1) Bug Found and Fixed in PIX CSCdw27548 Cloned to FWSM Port Features Bug Fixes CSCeb16339 FWSM 1.1(1) 2.2(1) Fix Ported to FWSM 1.1(2) 1.1(3) Maintenance Releases CSCeb76295 Bug Fixes Committed to Multiple Trains Printed in USA.

45 Understanding the Configuration Catalyst OS 7.6(3) 6K> (enable) set vlan 30,60 firewall-vlan 5 vlans 30,60 declared secure for firewall module K> (enable) session 5 FWSM# nameif vlan30 outside security0 nameif vlan60 inside security100 ip address outside ip address inside Native IOS 12.2(17a)SX1 Router#! vlan 30,60 firewall vlan-group 1 30,60 firewall module 5 vlan-group FWSM# nameif vlan30 outside security0 nameif vlan60 inside security100 ip address outside ip address inside Slot 5 Vlan 30 OUTSIDE INSIDE / Vlan / Printed in USA.

46 Agenda Understanding the Concepts PIX and FWSM Troubleshooting Tools PIX and FWSM Common Issues Cisco IOS Firewall Concepts Cisco IOS Firewall Common Issues and Troubleshooting 91 Cisco IOS Firewall Platform Compatibility Cisco IOS Firewall was introduced in 12.0(5)T Flash and RAM requirements vary depending on the router platform Router platforms include: Small Office: 800* and ubr900 series Branch Offices: 1600, 2500, 2600, and 3600 series WAN and high throughput: 7100, 7200, 7500, and RSM 92 Printed in USA.

47 What Are the Features of Cisco IOS Firewall? Stateful Packet Inspection Intrusion Detection Per-user Authentication and Authorization DoS Detection and Prevention Dynamic Port Mapping Java Blocking Real Time Alerts and Audit Trail Basic and Advanced Traffic Filtering Policy Based Multi-Interface Support 93 What Is CBAC INSIDE ip inspect name foo tcp interface ethernet0 ip inspect foo in OUTSIDE access-list 101 deny ip any any interface ethernet1 ip access-group 101 in A:a A:a A:a A:a A:a SYN SYN + ACK ACK RST RST B:b B:b B:b B:b B:b Allow return traffic Deny traffic initiating from outside 94 Printed in USA.

48 How Does CBAC Work and Its Effects Can this Crash My Router Do I Inspect All Interfaces INTERNAL NETWORK e0 EXTERNAL NETWORK e1 What Do I Inspect? Why Do I Inspect? and or and Why Access-Lists Dynamic ACL s? 95 Didn t CBAC Create Dynamic ACL s Firewall ACL Bypass: Introduced in 12.3(4) T ACL bypassing subjects the packet to one search the inspection session search against the list of session identifiers that already exist on the interface Established Sessions Session CC ( :32955)=>( :23) tcp SIS_OPEN Created 00:00:10, Last heard 00:00:06 Bytes sent (initiator:responder) [140:298] In SID [23:23]=> [32955:32955] on ACL 101 (15 matches) Out SID [23:23]=> [32955:32955] on ACL 102 Router# show access-list Extended IP access list 101 deny udp any any (20229 matches) deny tcp any any permit ip any any (6 matches) Extended IP access list 102 deny udp any any deny tcp any any permit ip any any (1 match) 96 Printed in USA.

49 State Tables and Session Maintenance State Tables: Maintains session state information Updated when a packet is inspected at the firewall's interface State table updates will permit return traffic through the firewall for packets traveling within a permissible session The state table is updated to remove the temporary opening in the access list when the session terminates Session Maintenance Use of timeouts and threshold values to manage session state information Timeouts used to prevent DoS attacks by dropping half-open connections and freeing up network resources Threshold values used to prevent DoS attacks by controlling the number of half-open sessions 97 Thresholds What Are Thresholds Set for the total number of half-open TCP and UDP sessions Set for the total number of half-open session based on time Set for the total number of half-open TCP-only sessions per host Configure Thresholds ip inspect max-incomplete {high low} This will modify the number of allowed max-incomplete sessions from the defaults of 500/400 ip inspect one-minute {high low} This will modify the number of unestablish sessions allowed in one minute from the defaults of 500/400 ip inspect tcp max-incomplete host <value> This will modify the number of incomplete sessions one particular host is allowed to have at any time from the default of Printed in USA.

50 Agenda Understanding the Concepts PIX and FWSM Troubleshooting Tools PIX and FWSM Common Issues Cisco IOS Firewall Concepts Cisco IOS Firewall Common Issues and Troubleshooting 99 Show Commands show ip access-list show ip inspect name inspection-name show ip inspect config show ip inspect config show ip inspect interfaces show ip inspect session [detail] show ip inspect all show ip inspect stat /120t/120t5/iosfw2/iosfw2_2.htm# Printed in USA.

51 Example of Show Commands Router# show ip inspect session Established Sessions Session 25A3318 ( :20)=>( :46068) ftpdata SIS_OPEN Session 25A6E1C ( :46065)=>( :21) ftp SIS_OPEN Router# show ip inspect name myinspectionrule Inspection Rule Configuration Inspection name myinspectionrule tcp timeout 3600 udp timeout 30 ftp timeout 3600 The following is sample output for the show ip inspect interfaces command: Interface Configuration Interface Ethernet0 Inbound inspection rule is myinspectionrule tcp timeout 3600 udp timeout 30 ftp timeout 3600 Outgoing inspection rule is not set Inbound access list is not set Outgoing access list is not set 101 Debugging CBAC Audit Trails ip inspect audit-trail Generic Debug debug ip inspect object-creation debug ip inspect object-deletion debug ip inspect events debug ip inspect timers debug ip inspect detail Transport Level Debugs debug ip inspect tcp debug ip inspect udp Application Protocol Debugs debug ip inspect protocol 102 Printed in USA.

52 Common Debugging Techniques If access-list may be a suspect: Study the logic of your list or try defining an additional broader list: access-list # permit tcp any any access-list # permit udp any any access-list # permit icmp any any int <interface> ip access-group # in out Use an extended access-list with a log option at the end: access-list 101 deny ip host host log access-list 101 permit ip any any If the ip inspect list is suspect, try debug ip inspect <type_of_traffic> : Feb 14 12:41: : 3d05h: CBAC* sis pak 16D0DC TCP P ack seq (2) ( :11109) => ( :23) 103 Configuration Issues THE MOST COMMON CONFIGURATION ERROR IS THE DIRECTION OF INSPECTION PROTECTED NETWORK e0 UNPROTECTED NETWORK e1 Inspect Inbound on e0 Inspect Outbound on e1 and or and Access List Inbound on e1 Access List Inbound on e1 104 Printed in USA.

53 Cisco IOS FW Dropping Packets Base Line Your Network Adjust Your Threshold Values As Needed Check Your Access-Lists Verify Your Inspect Statements Check for Asymmetrical Routing 105 Tips for Troubleshooting CBAC If Traffic is Being Denied: See if an access-list Is not denying traffic; remove the access-group and see if traffic in question Is permitted; If Possible Apply ExtendedAccess-lists Log Your Deny Statements Temporarily Router(config)#ip Access-list Extended IOSFW Router(config-ext-nacl)#deny ip any any Log Debugs on the Router: CBAC related debugs will give a lot of information if CBAC Is working the way it is supposed to be and return traffic Is permitted Debug IP Packet Detail: Router(config) # access-list 101 tcp host host Router # debug ip packet detail Printed in USA.

54 Helpful Resources PIX Firewall -bin/tablebuild.pl/pix FWSM c4fee.shtml IOS FW xtocid Complete Your Online Session Evaluation! WHAT: WHY: Complete an online session evaluation and your name will be entered into a daily drawing Win fabulous prizes! Give us your feedback! WHERE: Go to the Internet stations located throughout the Convention Center HOW: Winners will be posted on the onsite Networkers Website; four winners per day 108 Printed in USA.

55 109 Printed in USA.