ANATOMY OF A CODE RED II ATTACK IMPROVING FIREWALL SECURITY USING PATROL FOR CHECK POINT FIREWALL-1 BILL KENNON, SENIOR SOFTWARE CONSULTANT BMC SOFTWARE, INC 1
The Surprise Attack As the NT Systems Administrator for the XYZ Widget Corporation, you ve had a typically busy week. In between the numerous phone calls, staff/status/planning meetings, assisting the DBA staff with a database upgrades, testing changes to the DNS server, installing additional user account licenses for a Sales support application, and assisting the Network Administrator with remote dialup email access to the Web site, you re still tuning the new Web-based Commerce application and server environment to handle a steady rise in widget sales. You noticed that the primary IIS Web Server is operating a little slower than usual today. However, since the new Commerce site went live, Web Server transaction volume normally peaks in the early afternoon and slower CPU and memory performance isn t unusual during this time period. You make a mental note to take a closer look after lunch, just to be sure. It s 2:50 PM and you ve finally taken a moment to eat your lunch when your phone rings again. Only this time it s the Vice President of Sales wanting to know why the new Commerce site suddenly can t be accessed for the customer demonstration that is going to take place in exactly 10 minutes! You have to put her on hold to check your beeper it s the Chief Security Officer and no sooner do you promise the Vice President that you ll get right on it than who should appear at your door but your boss wanting to know What s wrong with the Web site?. Together, you conference in the Chief Security Officer only to find that the Firewall Administrator is reporting a significant spike in log entries affecting memory and CPU consumption on the firewall node protecting the primary IIS Server. Within the past 15 minutes, the Network Administrator detected router problems including a large number of ARPs/ARP storms in the network, excessive ARP Input memory use, and high CPU utilization. An urgent call from Operations confirms that the primary IIS server is rapidly grinding to a halt, and two other IIS servers are quickly reaching critical CPU and memory saturation levels. It appears that somehow a virus is resident within your secure perimeter and your new Commerce system is under hostile attack. How could this be? XYZ Widget Corporation takes security very seriously. The latest firewall and virus detection technologies are deployed and regularly maintained. You know that attacks can t be prevented, but XYZ Widget Corporation has been able to fend off previous intrusion attempts (from script kiddies as they re known within the industry) with little to no negative impact. Is it possible that this attack may have been carried out by a professional, known within the industry as a blackhat? More importantly, how do detect and stop this attack? It s at that moment that you recall this morning s warning about a brand new Denial of Server (DoS) attack called Code Red II. In fact, your To-Do list includes a new entry to search for patches from Microsoft. A quick scan of the Code Red II warning gives you several options to prevent the attack, such as blocking all traffic to port 80, the well known WWW port, but it will take precious time to locate the infected device(s), build the access-list to deny IP packet access to port 80, and apply it inbound on the interface 1
facing the infection source. Every minute that it takes to identify and respond means another minute of downtime, unhappy customers, distressed Vice Presidents, and lost business. Regardless of your reaction time and the resultant level of impact to XYZ s bottom line, this is an event you won t soon forget. This time the attacker wins, and you ll resign yourself to learning a painful, timeconsuming, expensive lesson. But you resolve that next time you ll be ready. You realize that budget constraints and the pressure to bring critical systems back online as quickly as possible will make it tough to diagnose hacker attacks. But if you could only understand how this attack occurred and, even better, if you could track the violator s actions as they happened, you could develop the forensics to prevent this scenario in the future 2
PATROL Detecting Code Red II Code Red II represents one of the latest in a series of increasingly devious and malicious threats to systems and networks worldwide. Simply perform an Internet search for Code Red II or Nimda to view a list of various organizations that were affected by these two recent attacks despite significant investments in firewall and related security protection. In today s climate of rapid technical evolution, constant environment changes, and malicious hackers, Code Red II, Nimda, and similar threats are successful despite the most stringent measures. These measures include enterprise Security Policies, the latest firewall technology, and strictest adherence to test, upgrade, and maintenance procedures. Although attacks cannot be prevented, technology is available that provides the detection, notification, and recovery capabilities not only to reduce Code Red II (and similar) threats but to also gather knowledge about the attack that can be used to frustrate and/or trap attackers in the future. PATROL for Check Point FireWall-1 from BMC Software, Inc., provides comprehensive firewall health management, detection and notification of unusual activity and problems, as well as the ability to automatically recover from detected impacts. BMC recently released PATROL for Check Point FireWall-1 v1.3 containing enhanced Denial of Service (DoS) capabilities to notify administrators of early DoS symptoms and help administrators prevent these attacks from causing network and system problems. The following is a real-life example of BMC Software, Inc. s PATROL performance in a network environment that was threatened by Code Red II. The Setting On Friday, August 3, 2001, one or more computers inserts the Code Red II virus into the internal network of a large corporation. Kennon2.xyz.com is the NT server where the Check Point FireWall-1 v4.1 management module and PATROL for Check Point FireWall-1 v1.3 are installed. This server is a 300mhz Pentium II processor with 128mb of memory and 16gb IDE hard drive, and runs NT 4.0 service pack 6. It is triple homed, with 172.23.5.36 defined as the external / real IP address and 192.168.15.1 defined as an internal / NAT address range (SCDEMO and 2815A domains). 10.0.0.1 is the second internal / NAT address range and is largely idle. What is unique about this firewall is that it performs as a honeypot, positioned inside the network to look at the corporate network as the outside or untrusted Internet zone. The remainder of this paper presents the attack in total as witnessed and documented by PATROL. There are numerous points during the attack where unusual activity is detected, normally leading to immediate preventive action. However, the attack was allowed to complete its activities for the purpose of documenting the entire procedure. The SCDEMO domain comprises a test lab of roughly 20 servers including Web servers that is used for testing and education. It is typically never accessed by any outside/external contacts and is not published as an available resource. Normally, the 3
SCDEMO Web servers are very lightly loaded except for programmed tests, which were not taking place during the attack. The Attack Begins Based on traffic analysis, at approximately 2:30PM, the virus initiates port scans and intrusion attempts on specific networked machines. Normally kennon2.bmc.com runs at about 10% CPU utilization, 70mb memory free, and about 200 log entries (Firewall events) per minute. Refer to Chart 1 below. During normal activity the log entries appear rhythmic, with predictable peaks and valleys reflecting business activity levels and traffic patterns on the network. Most networks display a similar rhythmic activity cycle, making it possible to extrapolate the baselines needed to understand variances from normal behavior. Chart 1: Normal Activity, average of 200 log entries per minute 4
On Saturday morning August 4, the log entries spike to over 200,000 per minute in about 20 minutes. This indicator alone is enough to warn an administrator of attack. In addition, for the next several days kennon2.xyz.com continues to report unusually high levels of log activities, occasional spikes in port scan attacks, and abnormally high levels of accepted and dropped packets to port 80. Chart 2: Anomalous Log Entry Behavior begins Saturday morning 5
Chart 3. The firewall anomaly tracks with Port 80 activity Web Servers are Probed While there are web servers in the SCDEMO domain, there is no reason for Port 80 traffic to be going to those servers from the other side of the firewall (the XYZ corporate network). Clients accessing the Web servers in the SCDEMO domain normally originate within that domain, so no URLs in the domain are explicitly published for external use. During the attack, port 80 inbound activities increases from 0 to about 50 packets dropped (Refer to Chart 3). There are no firewall policy rules stating that port 80 traffic should be dropped except for traffic directed at the firewall itself. The firewall explicitly drops all traffic defined to its own IP address (172.23.5.36) except for that originating from certain management workstations, so the excessive amount of traffic directed at port 80 is highly suspect. During the same period the firewall shows several thousand port 80 packets accepted per discovery cycle. So, while it appears that the high volume of port 80 traffic directed at the firewall represents attempts to breach the firewall host itself, the accepted port 80 6
traffic indicates the same sort of scanning against the hosts inside the firewall. In Charts 5 and 6, we see the accepted port 80 packets correlating well with the log entries, and showing an activity level well above average. Again, port 80 packets are generally near zero for this network, so settings for each network should be baselined in order to detect what type of activity would be considered abnormal. Chart 4: These are port 80 packets accepted 7
Chart 5: Chart of port 80 inbound (accepted) and log entries show perfect correlation Chart 5 shows both the port 80 inbound (accepted) and log entries. The correlation in the time line is perfect and indicates that port 80 was the targeted port in this particular case. Note that log entries indicate an increase of 400% to 500% of actual port 80 traffic during peak periods probably machines scanning for servers to attack. After the ferocious period of scanning, the log entries and port 80 traffic merge (Refer to Chart 6). 8
Chart 6: Activity during worm activity merge of log entries and port 80 inbound Chart 6 shows a time slice from a period of less activity. It is possible to see the sawtooth pattern of the port activity and log activity which at this time is a perfect 1 to 1 correlation. Apparently scans have all but ceased and now infected machines are trying to insert the worm into other servers. Attempts to infect three patched machines behind the firewall fail but apparently continue for some time. This is one persistent worm. Finding the Culprits Interestingly, when the first attack hit on Saturday morning, only a few directed port scans were recorded by the firewall as such. As Chart 7 indicates, the attacks occurred in singular fashion, and were destined to specific ports inside the firewall. However, any port scan by a machine on the inside should be considered cause for alarm. Several scans are by the same machine an absolute cause for alarm. 9
Chart 7: DoS attacks for timeline (Port Scan) The annotated datapoints in Chart 8 show how PATROL traps and displays specific IP addresses that originate the port scan. If XYZ Corporation had installed PATROL on the firewall management machine, PATROL would have detected this activity, alerted on it, and issued a recovery action, such as turning off a segment on a switch that owned the attacking machine. This action alone could have thwarted or at least delayed the attack. 10
Chart 8 Chart 9 To maximize the forensic capability of PATROL, it is important to be able to analyze history of at least 4 days, so that an attack over a period of time (a weekend) can be dissected. With the use of annotation and an effective notification system, such as PATROL Event Manager, operations support staff can receive immediate detailed notification of anomalous activity during off hours. Chart 9 shows the first machine that committed a port scan, in this case a spoof attack where a machine in the external (172.x.x.x) network uses an internal (192.168.158.99) IP address to call itself. 11
Chart 10: Infected machine inside SCDEMO 192.168.15 network Chart 10 shows an infected machine behind the firewall. PATROL detection demonstrates that a machine inside the network is attempting an outbound scan. Again, this represents a point where a recovery action could shut down the attack. This is obviously an infected machine. Summary As noted previously, there is no method to prevent attacks like Code Red II. However, had the XYZ security management personnel deployed PATROL in a honeypot implementation as described in this paper, this attack could have been detected and stopped before it could run its course. When the DoS attack started, the honeypot would attract packets sent at the abnormally high level. PATROL would have satisfied critical security management requirements quickly notify and identify the attacking source, and immediately identify the location of the attack. By setting alarm thresholds to trigger notification, and utilizing annotation to trap source/location information, PATROL would have provided rapid assistance to determine the addresses and networks of the attackers. With this information in hand, administrators could isolate the internal machine/s that are sending packets, or, if they are coming from the Internet, to program the router to drop those packets from the networks where the attacks were originating. 12