TROUBLESHOOTING FIREWALLS SESSION 1 Agenda Understanding the Concepts PIX and FWSM Troubleshooting Tools PIX and FWSM Common Issues Cisco IOS Firewall Concepts Cisco IOS Firewall Common Issues and Troubleshooting 2 Printed in USA.
Agenda Understanding the Concepts PIX and FWSM Troubleshooting Tools PIX and FWSM Common Issues Cisco IOS Firewall Concepts Cisco IOS Firewall Common Issues and Troubleshooting 3 How the PIX Processes a Packet Private Public Entering PIX Packet Processed Against ASA Leaving PIX Adaptive Security Algorithm Randomize Sequence Numbers Xlate and Connection Objects Stateful Inspection Security Levels Other rules 4 Printed in USA.
PIX Packet Processing 1 2 3 Recv No No Existing ACL Pkt Conn Permit Yes Yes 4 Match xlate Yes L7 NAT Sec Checks NAT IP Header Egress Interface 5 6 Drop 1. Receive packet 2. Existing connection? 3. Permit by Inbound ACL on interface? 4. Match translation rule (nat, static) 5. NAT embedded IP and perform security checks/randomize sequence number 6. NAT IP header 7. Pass packet to outgoing interface 8. Layer 3 route lookup? 9. Layer 2 next hop? 10. Transmit packet 8 9 10 L3 Route Yes L2 Addr Yes Xmit Pkt 7 Drop Drop 5 PIX Release Process Interim Images 6.0(2.100) 6.0(2.101) 6.0(2.122) Bug Fixes Committed to Multiple Trains CSCdv33495 6.0(2) 6.0(3) 6.0(4) Maintenance Trains 6.1(2) 6.1(3) 6.1(4) 6.1(5) 6.2(2) 6.2(3) 6.3(2) 6.3(3) 6.0(1) 6.1(1) 6.2(1) 6.3(1) Major Feature Releases 6 Printed in USA.
Agenda Understanding the Concepts PIX and FWSM Troubleshooting Tools PIX and FWSM Common Issues Cisco IOS Firewall Concepts Cisco IOS Firewall Common Issues and Troubleshooting 7 PIX and FWSM Troubleshooting Tools Syslog and ICMP Packet Capture Show Commands Output Interpreter PDM (3.0) 8 Printed in USA.
What Are Syslogs? Archival methods Primary mechanism to record traffic TO and THROUGH the PIX One of the best troubleshooting tools Archival Methods Debugging Methods Console Syslog Server Syslog SNMP Internet SSH Client SNMP Server Buffered 9 Syslog Level vs. Number of Messages Logged Log Level 0 1 2 3 4 5 6 7 Description Emergencies Alerts Critical Errors Warnings Notifications Informational Debugging # of Messages (Sum) 0 41 (41) 21 (62) 74 (136) 56 (192) 21 (213) 95 (308) 15 (323) * Based on PIX 6.3 10 Printed in USA.
Configuring and Using Syslog Define internal syslog server Log error (level 3) messages and below to syslog server Log alerts (level 1) to the internal buffer You must enable logging Log to a SSH/Telnet session as necessary pixfirewall (config) # logging host inside 10.1.1.5 pixfirewall (config) # logging trap 3 pixfirewall (config) # logging buffered 1 pixfirewall (config) # logging on pixfirewall (config) # logging monitor 4 11 What Are Modifiable Syslog Levels [no] logging message <syslog_id> level <level> Modifiable Syslog Levels Allows one to move any syslog message to any level Example: You want to record what commands are being executed on the PIX; message 111009 records this information, but by default it is at level 7 (Debug) %PIX-7-111009: User johndoe executed cmd: show run The problem is we don t want to log all 322 other syslogs that are generated at debug level Levels: 0 Emergency 1 Alert 2 Critical 3 Errors 4 Warnings 5 Notifications 6 Informational 7 Debugging 12 Printed in USA.
How to Create Modifiable Syslog Levels [no] logging message <syslog_id> level <level> SOLUTION: Lower syslog message 111009 to level 3 (Error) pix(config)# logging message 111009 level 3 - or pix(config)# logging message 111009 level error Now our syslog looks as follows: %PIX-3-111009: User johndoe executed cmd: show run To restore the default syslog level: pix(config)# no logging message 111009 level error - or pix(config)# logging message 111009 level 7 13 Notes on ICMP 1 3 Bob Ping Inside 2 DMZ Ping Outside Internet Ping 1 User Can Only Ping the Local Interface of the PIX 2 User Cannot Ping Remote Interface of the PIX 3 ICMP Echo-Replies Must be Explicitly Permitted thru the PIX 14 Printed in USA.
Debug ICMP Trace Private Network Internet http://www.cisco.com 1 User Able to Access Internet? No 2 Can Aser Access Private Network? Yes 3 Check Syslog Messages? 4 Test IP Connectivity Ping 5 pixfirewall (config) # debug icmp trace EXAMPLE OF DEBUG ICMP TRACE TO TEST SUCCESSFUL IP CONNECTIVITY ICMP echo-request from inside:10.1.1.2 to 209.165.201.2 ID=3239 seq=4369 length=80 ICMP echo-request: translating inside:10.1.1.2 to outside:209.165.201.22 ICMP echo-reply from outside:209.165.201.2 to 209.165.201.22 ID=3239 seq=4369 length=80 ICMP echo-reply: untranslating outside:209.165.201.22 to inside:10.1.1.2 15 Disabling Debug Commands undebug all In 6.3 You Can Now Turn Off All Debugs Globally by Issuing no debug all" and "undebug all or un all for Short pix(config)# show debug debug icmp trace debug sip pix(config)# un all pix(config)# show debug pix(config)# 16 Printed in USA.
PIX and FWSM Troubleshooting Tools Syslog and ICMP Packet Capture Show Commands Output Interpreter PDM (3.0) 17 Packet Capture capture <capture-name> [access-list <acl-name>] [buffer <buf-size>] [ethernet-type <type>] [interface <if-name>] [packet-length <bytes>] Capture command first introduced in PIX 6.2; it depreciates the debug packet command Capture sniffs packets on an interface that match an ACL Key steps: Create an ACL that will match interesting traffic Define the capture and bind it to an access-list and interface View the capture on the PIX, or copy it off in pcap format 18 Printed in USA.
Packet Capture (Cont.) Traffic can be captured both before and after it passes through the PIX; one capture on the Inside interface, one capture on the Outside interface Capture buffer saved in RAM (default size 512kb) Default is to stop capturing when buffer is full Default packet-length is 68 bytes (typically set to 1500) Copy captures off via TFTP or HTTPS Capture In Inside Capture Out Outside 19 Packet Capture: Example No. Time Source Destination Protocol Info 15 148.701751 Example: 209.165.201.4 User on 198.133.219.25 the Inside with TCP an 4511>http[SYN] IP of 10.1.1.12 Seq=27007623614 is having Ack=0 16 148.704086 a problem 198.133.219.25 accessing 209.165.201.4 Cisco.com TCP(198.133.219.25); http>4511[syn,ack] the Seq=979356760.. user is 17 148.705398 getting 209.165.201.4 NATed to 198.133.219.25 209.165.201.4. TCP 4511>http[ACK] Seq=2707623615 18 148.701751 209.165.201.4 198.133.219.25 HTTP GET /HTTP/1.1 Step 1: Create ACL for both Inside and Outside Interface Access-list 100 permit tcp host 209.165.201.4 host 198.133.219.25 eq 80 Access-list 100 permit tcp host 198.133.219.25 eq 80 host 209.165.201.4 Access-list 101 permit tcp host 10.1.1.12 host 198.133.219.25 eq 80 Access-list 101 permit tcp host 198.133.219.25 eq 80 host 10.1.1.12 Step 2: Create Captures on Both Inside and Outside Interface capture out access-list 100 interface outside packet-length 1500 capture in access-list 101 interface inside packet-length 1500 Step 3: Have Inside User Access www.cisco.com Step 4: Copy the Captures off to a TFTP Server copy capture:out-web tftp://10.1.1.10 pcap copy capture:in-web tftp://10.1.1.10 pcap OR copy using https: https://<pix_ip>/capture/out-web/pcap 20 Printed in USA.
PIX and FWSM Troubleshooting Tools Syslog and ICMP Packet Capture Show Commands Output Interpreter PDM (3.0) 21 An Introduction to Show Commands Show Perfmon Displays Current and Past Info Related to the PIX Show Blocks CLI Tool Used to Extract Information from the PIX for Information or Troubleshooting Show Memory Show cpu usage Show xlate <detail> Show conn <detail> Show interface Show traffic Used to Monitor the Health of the PIX and Draw a Baseline for your Network Show Processes 22 Printed in USA.
Show Conn and Show Conn <detail> pixfirewall (config) # show connection 2 in use, 2 most used TCP out 192.150.49.10:23 in 10.1.1.15:1026 idle 0:00:22 bytes 1774 flags UIO UDP out 192.150.49.10:31649 in 10.1.1.15:1028 idle 0:00:14 bytes 540 flags d pixfirewall (config) # show connection detail 2 in use, 2 most used Flags: A awaiting inside ACK to SYN, a awaiting outside ACK to SYN, B initial SYN from outside, D DNS, d dump, E outside back connection, F outside FIN, f inside FIN, G group, H H.323, I inbound data, M SMTP data, m SIP media, O- outbound data, P inside back connection, q SQL*Net data, R outside acknowledged FIN, R UDP RPC, r inside acknowledged FIN, S awaiting inside SYN, s awaiting outside SYN, T SIP, t SIP transient, U up TCP outside: 192.150.49.10/23 dmz:10.1.1.15/1026 flags UIO UDP outside: 192.150.49.10/31649 dmz:10.1.1.15/1028 flags d 23 Connection Termination Reasons %PIX-6-302014: Teardown TCP connection number for interface_name:real_address/real_port to interface_name:real_address/real_port duration time bytes number [reason] [(user)] Reset-I Reset-O TCP FINs FIN Timeout SYN Timeout Xlate Clear Deny SYN Control Uauth Deny Reset was from Inside Reset was from Outside Normal Close Down Sequence Force Termination After 15 Seconds Force Termination After 2 Min Command Line Removal Terminate by Application Inspection Back Channel Initiation from Wrong Side Deny by URL Filter Unknown Catch All Error 24 Printed in USA.
Show Xlate and Show Xlate <detail> pixfirewall (config) # show xlate 3 in use, 3 most used PAT Global 192.150.49.1(0) Local 10.1.1.15 ICMP id 340 PAT Global 192.150.49.1 (1024) Local 10.1.1.15(1028) PAT Global 192.150.49.1 (1024) Local 10.1.1.15(516) pixfirewall (config) # show xlate detail 3 in use, 3 most used Flags: D DNS, d dump, I identity, I inside, n no random, o outside, r portmap, s static TCP PAT from inside:10.1.1.15/1026 to outside:192.150.49.1/1024 flags ri UDP PAT from inside:10.1.1.15/1028 to outside:192.150.49.1/1024 flags ri ICMP PAT from inside:10.1.1.15/21505 to outside:192.150.49.1/0 flags ri 25 Xlate Flags FLAG DESCRIPTION s Static Translation Slot d Dump Translation Slot on Next Clearing Cycle r Port Map Translation n No Randomization of TCP Sequence Number o Outside Address Translation i Inside Address Translation D DNS A RR Rewrite I Identity Translation from NAT 0 26 Printed in USA.
Example Connection through PIX 1. PIX receives an initial SYN packet from the Inside. The SYN is permitted by the access-list, a translation (xlate) is built up, and the connection is also created with the flags "saa". 2. The Outside device responds to the SYN packet with a SYN+ACK. The connection flags are updated to reflect this, and now show "A". 3. The Inside device responds to the SYN+ACK with an ACK and this completes the TCP 3-way handshake, and the connection is now considered "up" (U flag). 4. The Outside device sends the first data packet. The connection is updated and an "I" is added to the flags to indicate the PIX received Inbound data on that connection. 5. Finally, the Inside device has sent a data packet and the connection is updated to include the "O" flag. 13 5 SYN+ACK Data 42 Connection Flags saa UIO Inside Outside Server Client 27 Example Connection Teardown 1. PIX receives a FIN packet from the Inside. As the FIN passes through the PIX, it updates the connection flags by adding an f to indicate that the FIN was received on the Inside interface. 2. The Outside device immediately responds to the FIN packet with a FIN+ACK. The connection flags are updated to reflect this, and now show UfFR". 3. The Inside device responds to the FIN+ACK with a final ACK and the PIX tears down the connection. Thus, there are no more connection flags, because the connection no longer exists. 31 FIN+ACK 2 Connection Flags UfUfFR UfFRr Inside Outside Server Client 28 Printed in USA.
Show CPU Usage First introduced in PIX OS version 6.0(1) Under normal conditions the PIX CPU should stay below 30% (baseline as per Network); if the CPU reaches 100% the PIX will start dropping packets The show cpu usage command displays the CPU over time as a running average An Example pixfirewall# show cpu usage CPU utilization for 5 seconds = 1%; 1 minute: 2%; 5 minutes: 1% A Note The Percentage Usage Prints as NA (Not Applicable) if the Usage is Unavailable For the Specified Time Interval; this Can Happen if the User Asks For CPU Usage Before the 5-second, 1-minute, or 5-minutes 29 Show Traffic The Show Traffic Command Displays the Traffic, in Packets and in Bytes, Out Each Interface of the PIX An Example pixfirewall# show traffic outside: received (in 124.650 secs): 295468 packets 167218253 bytes 2370 pkts/sec 1341502 bytes/sec transmitted (in 124.650 secs): 260901 packets 120467981 bytes 2093 pkts/sec 966449 bytes/sec inside: received (in 124.650 secs): 261478 packets 120145678 bytes 2097 pkts/sec 963864 bytes/sec transmitted (in 124.650 secs): 294649 packets 167380042 bytes 2363 pkts/sec 1342800 bytes/sec 30 Printed in USA.
Show Blocks The show blocks command, along with the show cpu usage command, are useful in determining whether the PIX is being overloaded The blocks are internal storage locations, similar to queues on a router; a packet is stored in a block until the PIX can process it and place it on the outbound interface xmit queue An Example pixfirewall# show blocks SIZE MAX LOW CNT 4 1600 1597 1600 80 400 399 400 256 500 495 499 1550 1444 1170 1188 16384 2048 1532 1538 What are these blocks used for? 256 = Stateful Failover, Syslog, TCP module 1550 = Ethernet Packets, buffering url filtered packets 16384 = Only used for the Gig Ethernet cards 31 Show Local-Host The show local-host command displays the translation and connection slots for all local hosts The clear local-host command stops traffic on all local hosts The clear local-host <ip_address> command stops traffic on the local host specified by its IP address An Example pixfirewall# show local-host Interface inside: 822 active, 823 maximum active, 0 denied local host: <10.1.2.74>, TCP connection count/limit = 0/unlimited TCP embryonic count = 0 TCP intercept watermark = unlimited UDP connection count/limit = 63/unlimited AAA: Xlate(s): PAT Global 209.165.201.26(41166) Local 10.1.2.74(39075) Conn(s): UDP out 209.165.201.1:8943 in 10.1.2.74:63556 idle 0:01:31 flags - 32 Printed in USA.
Show Tech-Support Enhancements (6.3) The show tech output was enhanced to include some additional show commands that can be used to troubleshoot memory and performance issues On the right are the commands included in the show tech output; note: They are in order; (New commands are in red) show version show clock show memory show conn count show xlate count show blocks show interface show cpu usage show process show failover show traffic show perfmon show running-config 33 Show Output Filters show <cmd> begin include exclude grep [-v] <regular_exp> Output Filters Have Been Added to PIX 6.3, Similar to the Ones in IOS; to Use them, at the End of Show <command>, Use the Pipe Character Followed By: begin include exclude grep [-v] <regular_exp> to Filter the Show Output Begin Start Displaying the Output Beginning at the First Match of the RegEx, and Continue to Display the Remaining Output include Display any line that matches the RegEx exclude Display any line that does not match the RegEx grep same as include grep v same as exclude 34 Printed in USA.
PIX and FWSM Troubleshooting Tools Syslog and ICMP Packet Capture Show Commands Output Interpreter PDM (3.0) 35 Output Interpreter GREAT TOOL TO CATCH COMMON CONFIGURATION ERRORS Select the Output In Question Paste the Output 36 Printed in USA.
Output Example Snip of Output Example Of Messages https://www.cisco.com/cgi-bin/support/outputinterpreter/home.pl 37 PIX and FWSM Troubleshooting Tools Syslog and ICMP Packet Capture Show Commands Output Interpreter PDM (3.0) 38 Printed in USA.
Using PDM for Monitoring Device Information Interface Statistics Active VPN Tunnels Current System Utilization: CPU, Memory, Traffic 39 Using PDM for Monitoring PDM Can Graph Up to 4 Different Graphs at Once Great for Monitoring Trends 40 Printed in USA.
Agenda Understanding the Concepts PIX and FWSM Troubleshooting Tools PIX and FWSM Common Issues Cisco IOS Firewall Concepts Cisco IOS Firewall Common Issues and Troubleshooting 41 Common Issues PIX Common Issues Accessing the Internet Accessing internal network from the Internet Issues with traffic between interfaces PIX not redirecting Failover Password recovery High CPU Out-of-memory issues FWSM Common Issues Configuration issues Passing traffic outbound Understanding failover 42 Printed in USA.
Accessing the Internet 10.10.10.x INSIDE OUTSIDE.3.1 http://www.xyz.com.1.2 63.1.1.x Internet.2 Web Server Problem: Accessing the Internet Troubleshooting: P ermissions T ranslation R outing 43 Permissions (Access-Lists) 10.10.10.x INSIDE OUTSIDE.3.1 http://www.xyz.com.1.2 63.1.1.x Internet.2 Web Server Verify if any Access-lists Are Applied Pixfirewall (config) # access-list acl permit tcp host 10.10.10.3 any eq www Pixfirewall (config) # access-group acl in interface inside Note If you Have an Access-list Applied on the Inside Interface, Check to Make Sure Traffic is Permitted Outbound; Remember, there is an Implicit Deny at the End of an Access-list 44 Printed in USA.
Translation 10.10.10.x INSIDE OUTSIDE.3.1 http://www.xyz.com.1.2 63.1.1.x Internet.2 Web Server 1 Verified Permissions Verify Translation Commands global (outside) 1 63.1.1.10 63.1.1.50 netmask 255.255.255.0 nat (inside) 1 10.10.10.0 255.255.255.0 global (outside) 1 interface nat (inside) 1 10.10.10.0 255.255.255.0 static (inside,outside) 63.1.1.51 10.10.10.3 netmask 255.255.255.255 45 Routing.2.1.2 INSIDE.3.1 OUTSIDE.1.2 Internet 192.168.1.x 10.10.10.x 63.1.1.x 1 Verified Permissions 2 Verified Translation Verify Route Commands on the PIX Check to make sure the pix has the correct default gateway pixfirewall(config)# route outside 0.0.0.0 0.0.0.0 63.1.1.2 If trying to access the internet from behind a layer 3 device, verify the PIX has a route to that network Pixfirewall (config)# route inside 192.168.1.0 255.255.255.0 10.10.10.3 46 Printed in USA.
Common Issues PIX Common Issues Accessing the Internet Accessing internal network from the Internet Issues with traffic between interfaces PIX not redirecting Failover Password recovery High CPU Out-of-memory issues FWSM Common Issues Configuration issues Passing traffic outbound Understanding failover 47 Accessing Internal Network From Internet Web Server.1 10.10.10.x INSIDE OUTSIDE.1.1.2 63.1.1.x Internet http://www.xyz.com Problem: Internal web server not accessible to users on the Internet Troubleshooting: P ermissions T ranslation R outing 48 Printed in USA.
Permissions (Access-Lists) Web Server.1 10.10.10.x INSIDE OUTSIDE.1.1.2 63.1.1.x Internet http://www.xyz.com Traffic has to be Explicitly Allowed into the PIX from a Lower Security to a Higher Security CHECK TO MAKE SURE YOU HAVE PERMITTED INTERESTING TRAFFIC EXPLICITLY Pixfirewall (config) # access-list acl permit tcp any host 63.1.1.10 eq http Pixfirewall (config) # access-group acl in interface outside If you have an access-list applied, check to make sure traffic is permitted inbound Remember, there is an implicit deny at the end of an access-list 49 Translation Web Server.1 10.10.10.x INSIDE OUTSIDE.1.1.2 63.1.1.x Internet http://www.xyz.com A Static Translation is Required to Pass Inbound Traffic Verify Translation Commands Pixfirewall(config) #static(inside,outside) 63.1.1.10 10.10.10.1 netmask 255.255.255.255 Example of a Syslog Message With No Static Defined 305005: No translation group found for tcp src outside:200.1.1.1/35550 dst inside:63.1.1.10/80 50 Printed in USA.
Routing Issues Web Server.1 10.10.10.x INSIDE OUTSIDE.1.1.2 63.1.1.x Internet http://www.xyz.com Check to make sure the PIX has the correct default gateway pixfirewall(config)# route outside 0.0.0.0 0.0.0.0 63.1.1.2 Verify the PIX has a route to the internal network pixfirewall(config)# route inside 172.16.171.0 255.255.255 10.10.10.2 Other Issues to Consider Confirm default gateway on your Web Server Verify your layer 3 device is routing correctly 51 Common Issues PIX Common Issues Accessing the Internet Accessing internal network from the Internet Issues with traffic between interfaces PIX not redirecting Failover Password recovery High CPU Out-of-memory issues FWSM Common Issues Configuration issues Passing traffic outbound Understanding failover 52 Printed in USA.
Issues with Traffic Between Interfaces DMZ (30) 1 INSIDE (100) OUTSIDE (0) 3 DMZ 1 (40) 2 1 Static and Access-List/Conduit 2 3 Static and Access-List/Conduit Static or a NAT/Global Statement 53 Common Issues PIX Common Issues Accessing the Internet Accessing internal network from the Internet Issues with traffic between interfaces PIX not redirecting Failover Password recovery High CPU Out-of-memory issues FWSM Common Issues Configuration issues Passing traffic outbound Understanding failover 54 Printed in USA.
PIX Is NOT Redirecting INSIDE OUTSIDE.2.1 172.16.171.x.2.1.1 10.10.10.x 63.1.1.x Internet 1 PIX Will Not Handle Redirects 2 Change User s Default Gateway to Be the Layer 3 Device 3 Modify the Layer 3 Device s Default Gateway to Be the PIX 55 Common Issues PIX Common Issues Accessing the Internet Accessing internal network from the Internet Issues with traffic between interfaces PIX not redirecting Failover Password recovery High CPU Out-of-memory issues FWSM Common Issues Configuration issues Passing traffic outbound Understanding failover 56 Printed in USA.
PIX Failover Active/Standby vs. Primary/Secondary Serial Failover/LAN Failover Stateful Failover A Failover ONLY occurs when either PIX determines the Standby PIX is healthier than the Active PIX Both PIXes swap MAC and IP Addresses when a failover occurs Active Unit Stateful Serial Lan FO Standby Unit Level 1 syslogs will give reason of failover 57 Commands to Verify Failover Is Active Pixfirewall (config)# show failover Failover On Cable status: Normal Reconnect timeout 0:00:00 This host: Primary - Standby (Failed) Active time: 7140 (sec) Interface 0 (192.168.1.1): Normal Interface 1 (172.16.171.54): Normal Other host: Secondary - Active Active time: 30 (sec) Interface 0 (192.168.1.3): Normal Interface 1 (172.16.171.55): Normal 58 Printed in USA.
Why Will Failover Happen? Power Failure Network Activity Test ARP Test Link Test Ping Test Failover Cable Failure 59 Using Syslogs to Troubleshoot Failover 1 2 3 No Failover Hello Seen on Serial Cable for 30 + Seconds; This Ensures that Failover is Running Properly on the Other PIX An Interface Did not Pass One of the 4 Failover Tests (Link up, Interface Traffic, ARP Test, Broadcast Ping) No Proper ACK for 15+ Seconds After a Command has Been Sent on the Serial Cable Message on Secondary when Primary Did Not Respond %PIX-1-103001: (Secondary) No response from other firewall (reason code = 3) Example of Syslogs when Primary Was Active, and It Lost One of Its Interfaces %PIX-4-411002: Line protocol on Interface intf2, changed state to down %PIX-1-105007: (Primary) Link status Down on interface 1 %PIX-1-104002: (Primary) Switching to STNDBY interface check, mate is healthier %PIX-1-104001: (Secondary) Switching to ACTIVE mate want me Active 60 Printed in USA.
Common Issues PIX Common Issues Accessing the Internet Accessing internal network from the Internet Issues with traffic between interfaces PIX not redirecting Failover Password recovery High CPU Out-of-memory issues FWSM Common Issues Configuration issues Passing traffic outbound Understanding failover 61 PIX Password Recovery Password recovery can only be performed by uploading the Password Recovery utility to the PIX from Monitor mode via TFTP Password recovery will also remove any AAA commands A password recovery utility is created for each major PIX release (6.1, 6.2, 6.3 ) however, the utilities are backwards compatible EX: np63.bin for PIX 6.3 and prior releases http://www.cisco.com/warp/public/110/34.shtml 62 Printed in USA.
Example of Password Recovery monitor> interface 1 Inside TFTP monitor> address 14.36.1.88 monitor> file np63.bin monitor> gateway 14.36.1.1 monitor> server 172.18.108.26 monitor> tftp 172.18.108.26 14.36.1.88 tftp np63.bin@172.18.108.26 via 14.36.1.1... Received 92160 bytes Do you wish to erase the passwords? [yn] y The following lines will be removed from the configuration: enable password 8Ry2YjIyt7RRXU24 encrypted passwd 2KFQnbNIdI.2KYOU encrypted aaa authentication serial console LOCAL aaa authentication telnet console LOCAL aaa authentication ssh console LOCAL aaa authentication enable console LOCAL Do you want to remove the commands listed above from the configuration? [yn] y Passwords and aaa commands have been erased. Rebooting.. 63 Common Issues PIX Common Issues Accessing the Internet Accessing internal network from the Internet Issues with traffic between interfaces PIX not redirecting Failover Password recovery High CPU Out-of-memory issues FWSM Common Issues Configuration issues Passing traffic outbound Understanding failover 64 Printed in USA.
High CPU Usage on the PIX Problem: PIX CPU Running Very High A Quick Overview of the Show Processes Command Number of msec this Process Has been on the CPU The Name of the Process pixfirewall(config)# show processes PC SP STATE Runtime SBASE Stack Process Hsi 001eab19 008a5a74 00557910 0 008a4aec 3628/4096 arp_timer Lsi 001f00bd 00a28dbc 00557910 0 00a27e44 3832/4096 FragDBGC Lwe 00119abf 02d280dc 0055b070 0 02d27274 3688/4096 dbgtrace Lwe 003e4425 02d2a26c 00557dd8 74440 02d28324 6936/8192 Logger Crd 001e26fb 0533940c 00557d88 6070290 05338484 3684/4096 557poll Lsi 00300a29 04c0f504 00557910 0 04c0e57c 3944/4096 xlate clean For more information on the output of the show processes command, see: http://www.cisco.com/en/us/partner/products/hw/vpndevc/ps2030/products_tech_note09186a008009456c.shtml 65 High CPU Usage on the PIX Step 1: Determine What Process Is Eating the CPU Take the Difference in Output of Two Show Processes over a Period of Time The Following Output Was a Diff of the Processes Taken 1 Minute Apart Process_Name Runtime (msec) Logger 25940 pix/intf3 18410 557poll 9250 i82543_timer 4180 i82542_timer 2230 In one minute, these Processes Account for 44 Seconds of CPU Time ~ 73% The Interface Polling Processes Always Run, and Are not Counted in the CPU Usage 66 Printed in USA.
High CPU Usage on the PIX Step 2: Focus on the Processes with High CPU Time Logging Is Taking up Much of the CPU, Let s Review What We Have Configured to Log This is Cumulative pixfirewall(config)# show log Since the PIX Was Syslog logging: enabled Last Rebooted Standby logging: disabled Console logging: disabled Monitor logging: disabled Buffer logging: level alerts, 0 messages logged Trap logging: level warnings, 5919412 messages logged Logging to lab 172.18.173.123 History logging: disabled... pixfirewall(config)# show log Syslog logging: enabled Buffer logging: level alerts, 0 messages logged Trap logging: level warnings, 6172472 messages logged Logging to lab 172.18.173.123 Notice the Change Over a Few Minutes 67 High CPU Usage on the PIX Syslog Server Is Controlled by a Different Group Enable Buffered Logging to Same Level as Syslog Server, and Examine the Buffered Messages pixfirewall(config)# show log Buffer logging: level warnings, 31527 messages logged Trap logging: level warnings, 6453127 messages logged Logging to lab 172.18.173.123 PIX s Interface Address... 400011: IDS:2001 ICMP unreachable from 172.18.173.123 to 14.36.1.88 on interface lab 400011: IDS:2001 ICMP unreachable from 172.18.173.123 to 14.36.1.88 on interface lab 400011: IDS:2001 ICMP unreachable from 172.18.173.123 to 14.36.1.88 on interface lab 400011: IDS:2001 ICMP unreachable from 172.18.173.123 to 14.36.1.88 on interface lab 400011: IDS:2001 ICMP unreachable from 172.18.173.123 to 14.36.1.88 on interface lab 400011: IDS:2001 ICMP unreachable from 172.18.173.123 to 14.36.1.88 on interface lab 68 Printed in USA.
High CPU Usage on the PIX EXAMINE IDS CONFIGURATION pixfirewall(config)# show run grep audit ip audit name IDS info action alarm ip audit interface lab IDS Syslog service was down on the syslog server ICMP Unreachable was generated by syslog server for each syslog message the PIX sent it PIX s IDS configuration also logged every ICMP Unreachable message, creating the exponentially increasing problem Syslog Server LAB OUTSIDE ICMP Unreachable Syslog Message IDS Syslog Message 69 High CPU Usage on the PIX SOLUTION: Bring back up syslog service on server Take server offline Configure PIX to not log IDS ICMP Unreachable messages ip audit signature 2001 disable -or - no logging message 400011 pixfirewall# show run grep signature ip audit signature 2001 disable pixfirewall# show cpu usage CPU utilization for 5 seconds = 2%; 1 minute: 50%; 5 minutes: 99% 70 Printed in USA.
High CPU Usage on the PIX SUMMARY: Examine the DIFF of two show processes taken over a one minute interval Find the process taking up the highest amount of CPU (excluding the polling processes) Take actions to lower that processes CPU time Re-examine the CPU output, and repeat as necessary 71 Common Issues PIX Common Issues Accessing the Internet Accessing internal network from the Internet Issues with traffic between interfaces PIX not redirecting Failover Password recovery High CPU Out-of-memory issues FWSM Common Issues Configuration issues Passing traffic outbound Understanding failover 72 Printed in USA.
Out of Memory PROBLEM: Users are unable to access the Internet No new connections are working; All old (long lived) connections continue to work Step 1: Check the Syslogs %PIX-3-211001: Memory allocation Error %PIX-3-211001: Memory allocation Error Step 2: Check the Amount of Free Memory Available Hardware: PIX-515E, 64 MB RAM pixfirewall# show memory Free memory: 714696 bytes Used memory: 66394168 bytes ------------- ---------------- Total memory: 67108864 bytes 73 Out of Memory Step 3: What Eats Up Memory (RAM) on the PIX? PIX image (run from RAM) Configuration IPSec Database Xlates (translations) Connections What Can Eat Up 64MB on a PIX-515 E Step 4: Let s Check the Translations pixfirewall# show xlate 251 in use, 258 most used PAT Global 209.165.201.26(2379) Local 10.1.1.132(52716) PAT Global 209.165.201.26(2378) Local 10.1.1.227(20276) Global 209.165.201.25 Local 10.1.1.102 PAT Global 209.165.201.26(2255) Local 10.1.1.125(12783) PAT Global 209.165.201.26(2382) Local 10.1.1.175(39197) PAT Global 209.165.201.26(2254) Local 10.1.1.34(43543) A Small Global Pool is Used, Overloading to a PAT Address Varied Source IPs 74 Printed in USA.
Out of Memory Step 5: Check the Connections pixfirewall# show conn 147456 in use, 147456 most used TCP out 64.102.144.194:80 in 10.1.1.38:26749 idle 0:00:19 Bytes 312 flags OIU TCP out 64.101.22.236:80 in 10.1.1.74:32209 idle 0:00:14 Bytes 239 flags OIU TCP out 64.102.147.77:21 in 10.1.1.48:32893 idle 0:00:48 Bytes 0 flags saa TCP out 64.103.31.215:80 in 10.1.1.136:18664 idle 0:00:46 Bytes 934 flags OIU TCP out 64.101.19.69:80 in 10.1.1.235:46712 idle 0:00:17 Bytes 8394 flags OIU TCP out 64.101.205.10:135 in 10.1.1.139:62296 idle 0:00:15 Bytes 0 flags saa TCP out 64.101.200.200:80 in 10.1.1.83:51864 idle 0:00:32 Bytes 902 flags OIU TCP out 64.102.80.27:80 in 10.1.1.66:52301 idle 0:00:03 Bytes 7813 flags OIU TCP out 64.103.95.35:80 in 10.1.1.231:51532 idle 0:00:24 Bytes 3891 flags OIU TCP out 64.102.206.172:80 in 10.1.1.223:28585 idle 0:00:28 Bytes 239 flags OIU TCP out 64.102.57.106:80 in 10.1.1.135:44945 idle 0:00:48 Bytes 9717 flags OIU TCP out 64.102.21.85:80 in 10.1.1.20:19578 idle 0:00:06 Bytes 2348 flags OIU TCP out 64.101.25.203:80 in 10.1.1.170:28149 idle 0:00:47 Bytes 419 flags OIU TCP out 64.101.86.97:135 in 10.1.1.54:43703 idle 0:00:12 Bytes 0 flags saa... Q: WHY IS THE CONNECTION COUNT SO HIGH? 75 Out of Memory Take a Look at the Traffic Load pixfirewall# show traffic outside: received (in 25.000 secs): 1475 packets 469050 bytes 59 pkts/sec 18762 bytes/sec transmitted (in 25.000 secs): 167619 packets 9654480 bytes 6704 pkts/sec 386179 bytes/sec inside: received (in 25.000 secs): 180224 packets 10410480 bytes 7208 pkts/sec 416419 bytes/sec transmitted (in 25.000 secs): 1050 packets 118650 bytes 42 pkts/sec 4746 bytes/sec Traffic Flow Outside Inside Vast Majority of Traffic Is Coming in the Inside Interface and Going Out the Outside Interface 76 Printed in USA.
Out of Memory Step 6: Review What We Know and Take Action pixfirewall# show conn count 147456 in use, 147456 most used pixfirewall# show xlate count 251 in use, 258 most used Conn Count Is Very High, but xlate Count Is Low Many connections per xlate Probably one, or a few hosts, are generating the vast majority of connections Most likely due to a virus on the host(s) 77 Out of Memory Step 7: Find the Host(s) Generating All the Connections pixfirewall# show local-host include host count/limit local host: <10.1.1.131>, TCP connection count/limit = 0/unlimited UDP connection count/limit = 0/unlimited local host: <10.1.1.51>, TCP connection count/limit = 2/unlimited UDP connection count/limit = 0/unlimited local host: <10.1.1.236>, TCP connection count/limit = 0/unlimited UDP connection count/limit = 0/unlimited... local host: <10.1.1.99>, TCP connection count/limit = 146608/unlimited UDP connection count/limit = 0/unlimited Only Show Lines that Have the Word host OR count/limit In Them Host 10.1.1.99 Is Eating Up All the Connections, and They Are TCP Based Connections 78 Printed in USA.
Out of Memory Step 8: Now that We Found the Host, Let s Look at the Connections it Is Generating pixfirewall# show local-host 10.1.1.99 Interface inside: 250 active, 250 maximum active, 0 denied local host: <10.1.1.99>, TCP connection count/limit = 146608/unlimited TCP embryonic count = 146606 UDP connection count/limit = 0/unlimited Xlate(s): Global 209.165.201.21 Local 10.1.1.99 Conn(s): TCP out 64.101.32.157:135 in 10.1.1.99:34580 idle 0:01:43 Bytes 0 flags saa TCP out 64.103.108.191:135 in 10.1.1.99:8688 idle 0:01:43 Bytes 0 flags saa TCP out 64.100.205.160:135 in 10.1.1.99:7774 Connections idle 0:01:43 to Bytes 0 flags saa TCP out 64.101.182.19:135 in 10.1.1.99:39193 idle Random 0:01:43 Bytes 0 flags saa TCP out 64.102.218.45:135 in 10.1.1.99:16462 Destinations idle 0:01:43 on Bytes 0 flags saa TCP out 64.100.21.120:135 in 10.1.1.99:30322 idle TCP/135 0:01:43 Bytes 0 flags saa TCP out 64.101.25.195:135 in 10.1.1.99:41116 idle MS Blaster 0:01:43 Bytes 0 flags saa TCP out 64.103.17.219:135 in 10.1.1.99:59163 idle 0:01:43 Bytes 0 flags saa TCP out 64.102.201.141:135 in 10.1.1.99:2978 idle 0:01:43 Bytes 0 flags saa TCP out 64.103.176.75:135 in 10.1.1.99:41589 idle 0:01:43 Bytes 0 flags saa... All Connections Are Embryonic (Refer to Slide 23) 79 Out of Memory Step 9: Limit Infected Host(s) Impact on Network PIX Provides Two Methods to Limit the Number of Connections per Host: TCP Intercept Max Connections Which One can be Used Here? TCP Intercept Won t Help Because the Source Address Is VALID Limiting the Maximum Number of Connections Each Internal Host Can Have Is the Only Option 80 Printed in USA.
Out of Memory Step 9: Limit Infected Host(s) Impact on Network Configure the MAX TCP Connections for NATed Hosts to be 50: pixfirewall(config)# nat (inside) 1 0.0.0.0 0.0.0.0 50 0 NOTE: The Local-host MUST be Cleared Before the New Connection Limits Are Applied pixfirewall(config)# clear local-host 10.1.1.99 pixfirewall(config)# show local-host 10.1.1.99 Interface inside: 250 active, 250 maximum active, 0 denied local host: <10.1.1.99>, TCP connection count/limit = 50/50 TCP embryonic count = 50 TCP intercept watermark = unlimited UDP connection count/limit = 0/unlimited... The Infected Host is Limited to 50 TCP Connections 81 Out of Memory Take One Last Look at the Memory and Connection Counts After Applying the TCP Connection Limit pixfirewall# show conn count 126 in use, 147456 most used pixfirewall# show memory Free memory: 47716152 bytes Used memory: 19392712 bytes ------------- ---------------- Total memory: 67108864 bytes Things Look Much Better Now Question: How Could we Configure the PIX so the Connection Limit Was only Applied to the One Host (10.1.1.99) which Was Infected with the Virus? nat (inside) 1 10.1.1.99 255.255.255.255 50 0 nat (inside) 1 0.0.0.0 0.0.0.0 0 0 82 Printed in USA.
Still Can t Resolve It? Open a TAC Case If After Using All Your Troubleshooting Tools You Still cannot Resolve the Problem, Please Open a TAC Case: http://www.cisco.com/techsupport/servicerequest/ At a minimum include: Detailed problem description Output from show tech Optionally include: Syslogs captured during time of problem Sniffer traces from both interfaces using the capture command (capturing only the relevant packets, and saved in pcap format) 83 PIX/FWSM Best Practices Enable ip verify reverse-path on all interfaces Set Embryonic and Maximum connection counts on static and nat statements Configure logging to syslog server Move messages you want to see to lower levels, instead of raising logging levels and capturing messages you don t want to see Disable telnet access, use SSH for management access Enable Authentication for Management access (Console/SSH/Telnet/Enable); use TACACS+, RADIUS or LOCAL database Restrict DMZ access inbound to your Internal Networks Baseline CPU load, connection counts, xlate counts, and traffic (per interface) 84 Printed in USA.
Agenda Understanding the Concepts PIX and FWSM Troubleshooting Tools PIX and FWSM Common Issues Cisco IOS Firewall Concepts Cisco IOS Firewall Common Issues and Troubleshooting 85 FWSM Code Base History Feature Releases PIX 6.0(1) 6.1(1) 6.2(1) 6.3(1) FWSM 1.1(1) New Features OSPF, crashinfo, etc. 1.1(2) Maintenance Releases Feature Releases 1.1(3) Port Features Bug Fixes 2.2(1) New Features Virtual FW, Transparent FW, etc. 86 Printed in USA.
FWSM Bug Fix Process Feature Releases PIX 6.0(1) 6.1(1) 6.2(1) 6.3(1) Bug Found and Fixed in PIX CSCdw27548 Cloned to FWSM Port Features Bug Fixes CSCeb16339 FWSM 1.1(1) 2.2(1) Fix Ported to FWSM 1.1(2) 1.1(3) Maintenance Releases CSCeb76295 Bug Fixes Committed to Multiple Trains 87 88 Printed in USA.
Understanding the Configuration Catalyst OS 7.6(3) 6K> (enable) set vlan 30,60 firewall-vlan 5 vlans 30,60 declared secure for firewall module 5 ----------------- 6K> (enable) session 5 FWSM# nameif vlan30 outside security0 nameif vlan60 inside security100 ip address outside 10.30.1.2 255.255.255.0 ip address inside 10.60.1.1 255.255.255.0 Native IOS 12.2(17a)SX1 Router#! vlan 30,60 firewall vlan-group 1 30,60 firewall module 5 vlan-group 1 ----------------- FWSM# nameif vlan30 outside security0 nameif vlan60 inside security100 ip address outside 10.30.1.1 255.255.255.0 ip address inside 10.60.1.1 255.255.255.0 Slot 5 Vlan 30 OUTSIDE INSIDE 10.30.1.0/24 10.30.1.1 10.60.1.1 Vlan 60 10.60.1.0/24 89 90 Printed in USA.
Agenda Understanding the Concepts PIX and FWSM Troubleshooting Tools PIX and FWSM Common Issues Cisco IOS Firewall Concepts Cisco IOS Firewall Common Issues and Troubleshooting 91 Cisco IOS Firewall Platform Compatibility Cisco IOS Firewall was introduced in 12.0(5)T Flash and RAM requirements vary depending on the router platform Router platforms include: Small Office: 800* and ubr900 series Branch Offices: 1600, 2500, 2600, and 3600 series WAN and high throughput: 7100, 7200, 7500, and RSM 92 Printed in USA.
What Are the Features of Cisco IOS Firewall? Stateful Packet Inspection Intrusion Detection Per-user Authentication and Authorization DoS Detection and Prevention Dynamic Port Mapping Java Blocking Real Time Alerts and Audit Trail Basic and Advanced Traffic Filtering Policy Based Multi-Interface Support 93 What Is CBAC INSIDE ip inspect name foo tcp interface ethernet0 ip inspect foo in OUTSIDE access-list 101 deny ip any any interface ethernet1 ip access-group 101 in A:a A:a A:a A:a A:a SYN SYN + ACK ACK RST RST B:b B:b B:b B:b B:b Allow return traffic Deny traffic initiating from outside 94 Printed in USA.
How Does CBAC Work and Its Effects Can this Crash My Router Do I Inspect All Interfaces INTERNAL NETWORK e0 EXTERNAL NETWORK e1 What Do I Inspect? Why Do I Inspect? and or and Why Access-Lists Dynamic ACL s? 95 Didn t CBAC Create Dynamic ACL s Firewall ACL Bypass: Introduced in 12.3(4) T ACL bypassing subjects the packet to one search the inspection session search against the list of session identifiers that already exist on the interface Established Sessions Session 814063CC (192.168.1.116:32955)=>(192.168.101.115:23) tcp SIS_OPEN Created 00:00:10, Last heard 00:00:06 Bytes sent (initiator:responder) [140:298] In SID 192.168.101.115[23:23]=>192.168.1.117[32955:32955] on ACL 101 (15 matches) Out SID 192.168.101.115[23:23]=>192.168.1.116[32955:32955] on ACL 102 Router# show access-list Extended IP access list 101 deny udp any any (20229 matches) deny tcp any any permit ip any any (6 matches) Extended IP access list 102 deny udp any any deny tcp any any permit ip any any (1 match) 96 Printed in USA.
State Tables and Session Maintenance State Tables: Maintains session state information Updated when a packet is inspected at the firewall's interface State table updates will permit return traffic through the firewall for packets traveling within a permissible session The state table is updated to remove the temporary opening in the access list when the session terminates Session Maintenance Use of timeouts and threshold values to manage session state information Timeouts used to prevent DoS attacks by dropping half-open connections and freeing up network resources Threshold values used to prevent DoS attacks by controlling the number of half-open sessions 97 Thresholds What Are Thresholds Set for the total number of half-open TCP and UDP sessions Set for the total number of half-open session based on time Set for the total number of half-open TCP-only sessions per host Configure Thresholds ip inspect max-incomplete {high low} This will modify the number of allowed max-incomplete sessions from the defaults of 500/400 ip inspect one-minute {high low} This will modify the number of unestablish sessions allowed in one minute from the defaults of 500/400 ip inspect tcp max-incomplete host <value> This will modify the number of incomplete sessions one particular host is allowed to have at any time from the default of 50 98 Printed in USA.
Agenda Understanding the Concepts PIX and FWSM Troubleshooting Tools PIX and FWSM Common Issues Cisco IOS Firewall Concepts Cisco IOS Firewall Common Issues and Troubleshooting 99 Show Commands show ip access-list show ip inspect name inspection-name show ip inspect config show ip inspect config show ip inspect interfaces show ip inspect session [detail] show ip inspect all show ip inspect stat http://www.cisco.com/univercd/cc/td/doc/product/software/ios120/120newft /120t/120t5/iosfw2/iosfw2_2.htm#12583 100 Printed in USA.
Example of Show Commands Router# show ip inspect session Established Sessions Session 25A3318 (10.0.0.1:20)=>(10.1.0.1:46068) ftpdata SIS_OPEN Session 25A6E1C (10.1.0.1:46065)=>(10.0.0.1:21) ftp SIS_OPEN Router# show ip inspect name myinspectionrule Inspection Rule Configuration Inspection name myinspectionrule tcp timeout 3600 udp timeout 30 ftp timeout 3600 The following is sample output for the show ip inspect interfaces command: Interface Configuration Interface Ethernet0 Inbound inspection rule is myinspectionrule tcp timeout 3600 udp timeout 30 ftp timeout 3600 Outgoing inspection rule is not set Inbound access list is not set Outgoing access list is not set 101 Debugging CBAC Audit Trails ip inspect audit-trail Generic Debug debug ip inspect object-creation debug ip inspect object-deletion debug ip inspect events debug ip inspect timers debug ip inspect detail Transport Level Debugs debug ip inspect tcp debug ip inspect udp Application Protocol Debugs debug ip inspect protocol 102 Printed in USA.
Common Debugging Techniques If access-list may be a suspect: Study the logic of your list or try defining an additional broader list: access-list # permit tcp any any access-list # permit udp any any access-list # permit icmp any any int <interface> ip access-group # in out Use an extended access-list with a log option at the end: access-list 101 deny ip host 171.68.118.100 host 10.31.1.161 log access-list 101 permit ip any any If the ip inspect list is suspect, try debug ip inspect <type_of_traffic> : Feb 14 12:41:17 10.31.1.52 56: 3d05h: CBAC* sis 258488 pak 16D0DC TCP P ack 3195751223 seq 3659219376(2) (10.31.1.5:11109) => (12.34.56.79:23) 103 Configuration Issues THE MOST COMMON CONFIGURATION ERROR IS THE DIRECTION OF INSPECTION PROTECTED NETWORK e0 UNPROTECTED NETWORK e1 Inspect Inbound on e0 Inspect Outbound on e1 and or and Access List Inbound on e1 Access List Inbound on e1 104 Printed in USA.
Cisco IOS FW Dropping Packets Base Line Your Network Adjust Your Threshold Values As Needed Check Your Access-Lists Verify Your Inspect Statements Check for Asymmetrical Routing 105 Tips for Troubleshooting CBAC If Traffic is Being Denied: See if an access-list Is not denying traffic; remove the access-group and see if traffic in question Is permitted; If Possible Apply ExtendedAccess-lists Log Your Deny Statements Temporarily Router(config)#ip Access-list Extended IOSFW Router(config-ext-nacl)#deny ip any any Log Debugs on the Router: CBAC related debugs will give a lot of information if CBAC Is working the way it is supposed to be and return traffic Is permitted Debug IP Packet Detail: Router(config) # access-list 101 tcp host 10.1.1.1 host 192.168.1.1 Router # debug ip packet detail 101 106 Printed in USA.
Helpful Resources PIX Firewall http://www.cisco.com/univercd/cc/td/doc/product/iaabu/pix/pix_sw/v_63/index.htmhttp://www.cisco.com/cgi -bin/tablebuild.pl/pix http://www.cisco.com/pcgi-bin/support/browse/psp_view.pl?p=hardware:pix&s=software_configuration http://www.cisco.com/pcgibin/support/browse/psp_view.pl?p=hardware:pix&s=troubleshooting#known_problems FWSM http://www.cisco.com/univercd/cc/td/doc/product/lan/cat6000/cfgnotes/78_14450.htm http://www.cisco.com/univercd/cc/td/doc/product/lan/cat6000/cfgnotes/fwsm/index.htm http://www.cisco.com/en/us/products/hw/modules/ps2706/products_qanda_item09186a0080 0c4fee.shtml IOS FW http://www.cisco.com/pcgibin/support/browse/psp_view.pl?p=software:cisco_ios_firewall&s=implementation_and_configuration http://www.cisco.com/warp/partner/synchronicd/cc/pd/iosw/ioft/iofwft/prodlit/fire_qa.htm http://www.cisco.com/univercd/cc/td/doc/product/software/ios120/120newft/120t/120t5/iosfw2/iosfw2_2.htm# xtocid135950 107 Complete Your Online Session Evaluation! WHAT: WHY: Complete an online session evaluation and your name will be entered into a daily drawing Win fabulous prizes! Give us your feedback! WHERE: Go to the Internet stations located throughout the Convention Center HOW: Winners will be posted on the onsite Networkers Website; four winners per day 108 Printed in USA.
109 Printed in USA.