Managing a Malware Outbreak

Sality A Case Study Authors: Mike Andrews Senior Principal Consultant Jerry Pierce Principal Consultant Shawn Baker Senior Consultant

Table of Contents Managing a Malware Outbreak... 1 Sality A Case Study... 1 Table of Contents... 1 Introduction... 2 Initial Infection... 2 Lessons Learned... 3 Propagation... 4 Lessons Learned... 5 Watching the Behavior... 5 Lessons Learned... 8 Remediation... 8 Workstations... 9 Servers... 9 File Server / NAS / SAN... 9 Lessons Learned... 10 Conclusion... 10 About Foundstone Professional Services... 11

Introduction For the vast majority of organizations, finding out they are under attack from a virulent, aggressive piece of malware is a stressful prospect. Not knowing what the malware is doing, or is capable of doing, can be terrifying. Some malware is what we call nuisance-ware. It displays annoying messages, modifies Internet browser or registry settings, or disables some functionality. Most other malware, however, can have serious impact on business continuity. Malicious malware can modify and delete data, install backdoors and keystroke loggers, and transmit files and information beyond network perimeters. They can even modify their own behavior over time by downloading patches and additional malware. There are many sources of information that detail how individual pieces of malware operate and their behavioral characteristics on a system. This whitepaper chronicles one of our incident response investigations where a multi-national organization with several sites discovered they had the Sality virus rampaging through their environment. We focus on the steps that were taken to contain and eradicate the malware and the resulting lessons learned. Initial Infection Generally, malware outbreaks start slowly. The first device to get infected is often a workstation or laptop that does not have adequate defenses. Anti-virus software is not installed, not current, or disabled. The operating system and applications are not patched. Sometimes a user inadvertently puts themselves, and company assets, at risk by installing browser plug-ins, cruising questionable web sites, or clicking on suspicious links in email messages. In this particular investigation, the initial infection was probably caused by a file being opened from an untrusted site on the Internet, or brought into the organization via a USB thumb drive. Despite the huge amount of malware in existence, there is a lot of commonality among them. There are only so many strategies that can be employed to stay resident on a machine and propagate. Therefore, as long as antivirus clients are updated regularly, antivirus software from any vendor has a good chance of catching an infection attempt. However, in this particular Sality outbreak there were two issues. First, due to a configuration oversight, workstations were not receiving timely antivirus signature updates. In fact, some workstations were close to a year out-of-date with their virus signatures. A weakened security posture certainly helped the virus slip into the organization. The second contributing factor is the ability of recent virus variants to avoid detection by antivirus engines. Malware programmers subtly change an existing virus variant so that it does not match a known signature. This arms race between malware authors and antivirus vendors means that there is a period of increased risk between the time a new variant has been released, and when antivirus companies can detect and successfully stop/clean the infection. Some malware 2 www.foundstone.com 1.877.91.FOUND

has the ability to phone home and patch itself or download new code / instructions. During this incident, we observed this capability. Figure 1 shows a steady infection rate of the AG strain of the Sality virus over time. Notice the sudden jump in infections from other Sality strains at about the same point in time. This is often caused by new versions leveraging previously compromised machines to gain a foothold. Figure 1: Virus infection and eradication by strain One other factor that assisted this particular virus in propagating was the fact many workstations did not have a particularly secure setup. As is often the case, users had local administrative privileges so that they can install software, or because some critical application will not run without privileged access rights. Newer operating systems, such as Windows Vista, discourage users from always having administrative rights by prompting users when such elevated privileges are required (e.g. the much maligned User Account Control). Restricting administrative permissions severely limits the opportunity for a piece of malware to gain a permanent hold on a workstation. Lessons Learned Ensure that antivirus signatures are up-to-date and all protection mechanisms are enabled. Enable On-Access file scanning to scan files before they are accessed. Prevent program execution from TEMP folders. Many files get downloaded to a user s computer when surfing the Internet. These files are placed in a user s temporary folder and executed from there. Files downloaded from the Internet may be infected with malicious code. 3 www.foundstone.com 1.877.91.FOUND

Block creation of autorun.in files. Much of the malware seen in recent months creates autorun.inf files on file shares and removal media. When an infected flash drive or network share is accessed, the operating system will look for the existence of these files. If found, the file is automatically executed which means that malware could execute with essentially no user interaction. In addition disable autorun / autoplay on all devices. For the Sality virus in particular, creation of read-only autorun.inf files in the root of all local drives and network shares will help to prevent the virus from spreading via this mechanism. Enable Access Protection in your antivirus software. This allows antivirus software to protect itself from being reconfigured or disabled. Set alerts when a process attempts to disable the antivirus software engine. These alerts advise when a user, or malicious software, attempts to stop the antivirus software to copy potentially unwanted programs (PUPs) or pieces of malware. Leverage enterprise antivirus products. Ensure your centralized system is regularly monitored or reviewed so outbreaks can be identified and responded to quickly. Early identification and eradication of malware infections may help prevent a single host infection from becoming a wide-spread epidemic. Train users to be aware of strange behavior of their workstations and report any incidents to IT. Remove file extension associations in the registry that are not used, or only used maliciously (e.g. PIF files). Restrict, or remove local administrative rights. Utilize third party User Access Control (UAC) software to grant access for applications that may require elevated privileges. All other actions are executed as a regular authenticated user account. Third party software, such as BeyondTrust s Privilege Manager, will allow for support across multiple workstation operating systems. Use the Run As functionality in Windows. Programs that require elevated privileges can be executed as a privileged user while other functions are performed as a regular authenticated user. This will require a second user account to execute the applications requiring elevated privileges. Disable the ability to access USB devices where appropriate. Propagation Malicious code comes in all shapes and sizes with various propagation methods. Some require no user interaction to spread (worms). Some are parasitic in nature, attach to files, and wait to be executed (viruses). Some malware does not try to replicate, but rather, affects the behavior of the machine (Trojan-horses). In order to increase infection rates and propagate further, there are also blended threats that can use a variety of these techniques. 4 www.foundstone.com 1.877.91.FOUND

Sality is a virus that propagates by infecting or dropping files with the hope users will execute them with increased access permissions. Two main propagation vectors are used by Sality; infecting EXE files and dropping autorun.in files pointing to the infected executable files. Whenever a user executes an infected file or browses to a directory that contains an autorun.inf file, or inserts a USB device containing the autorun.inf file, the virus is executed. Once executed, the virus has a particular payload (see Watching the Behavior below), but it also attempts to spread to other devices. Sality achieves this by searching for mapped drives and infecting portable executable (PE) files stored on them as well as any files on USB media that is inserted into the machine. A potentially devastating prospect occurs if a domain administrator logs onto an infected machine, as the virus is now able to execute with additional privileges. It will attempt to open network share(s) with all computers it discovers on the network and, if successful, start to infect files on those drives as well. Because the domain administrator has the permissions to access a large number of machines with elevated rights, including server machines, the virus is able to spread very quickly. Lessons Learned Configure workstations to discourage automatic execution of code, disable autorun / autoplay and remove file extension associations in the registry that are not used, or only used maliciously (e.g. PIF files). Restrict permissions on shared folders and consider disabling auto-mount. Never allow users with domain administrator credentials to log directly into a workstation when a virus is loose in the enterprise. Users should log on to their workstations as a non-privileged (regular) user and use administrative accounts only as necessary to perform administrative job functions. Watching the Behavior Reverse engineering a piece of malware to determine its behavior is best left to experts. Malware behavior is often complex, obfuscated, and not always visible immediately. Malware can sleep and only wake after a random period of time or when a certain event occurs. Additionally, without careful procedures, there s the chance of inadvertently spreading the infection. For a known bit of malicious code, looking up its behavioral characteristic is easy on sites such as http://vil.nai.com. Anyone can submit files that they think contain malware (see http://vil.nai.com/vil/submit-sample.aspx ). Other sites which offer free analysis of suspected malicious files are http://www.virustotal.com and http://www.threatexpert.com. If deemed necessary using some free tools, malware behaviors can be observed and some basic analysis can be performed. Process Monitor is a very useful free tool available at http://technet.microsoft.com/en- 5 www.foundstone.com 1.877.91.FOUND

us/sysinternals/bb896645.aspx. If you identify a particular file or process that is behaving suspiciously, this tool can capture and filter events related to the process. From this view it is easy to see if the process is attempting to access other files or registry keys, and if it is communicating on the network. Clearly, if a process such as Notepad.exe is attempting to write to numerous files or is sending network packets, something is seriously amiss. Figure 2: Process Monitor capturing file, registry and network information Once a malicious file has been identified, it can be transferred to a virtual machine environment that is disconnected from the main network and isolated from its host machine as well. Having a test network available may be necessary to wake certain parts of the malware s behavior. Once again, start Process Monitor, and watch what happens. Although behaviors differ between different variants of Sality, here s what we observed from one such infection. A user, browsing the Internet, visits a legitimate website. Unfortunately, this website has been compromised and an executable file is downloaded to the user s TEMP folder, from where it is executed. Within seconds of this program running, registry entries are created which indicate a Sality infection. A few seconds later, Notepad starts as a child of the explorer.exe process and begins communicating on the network. Approximately two hours later, two more executables are downloaded into the user s TEMP folder and executed. At this point the malware is only running on a single machine. If the user has mounted shares that contain executable files, the malware will infect these shared executables and the outbreak begins. In this case, had this organization s antivirus product been properly configured, it would have prevented the initial infection by disallowing the execution of code from the user s TEMP folder. 6 www.foundstone.com 1.877.91.FOUND

Given this information two critical activities must be performed: 1. Check that the antivirus solution identifies the malware and successfully detects and removes it. If not, the vendor should be contacted and provided with a sample of the virus so that they can work on a solution. 2. Identify any sensitive data the malware is trying to access and any external sites on the Internet it is trying to communicate with. These sites should be blocked at the firewall/proxy for both incoming and outgoing traffic. These sites should be blocked by both the URI and IP addresses. Monitoring the firewall and/or proxy will give an idea of the level of infection and IP addresses of machines known to be infected so they can be taken off the network. Although there is much more to analyzing the behavior of malicious code, these basic principles can help in understanding what a piece of malware is doing and the answers to these critical questions: What is the scope of the infection? What is the behavior of the threat and what is at risk? Has any sensitive data been compromised? Once these questions are answered, you can go on to the clean-up operation. Sality is a Windows portable executable infector, meaning that it will search for executable files and wrap itself around them. When an infected file is executed, it will spread the contagion further. It primarily relies on two methods of infection once it has reached the inside of a corporate network. First, it searches the root folder of all local hard drives for executable files that it can infect. In some instances, an autorun.inf file will be created in the root folder of the drive. It will also place a malicious autorun.inf file on the root of any USB device attached to an infected machine. These autorun.inf files will cause anyone browsing the root folder of the device to execute the malicious content if autorun is not disabled. The second method of infection is enumeration of file shares. Sality searches for all mounted file shares, and will look for executable files within those shares to infect. This can be a very effective way to propagate if file servers are located that host applications. In most organizations, the virus similarly arrives via delivery from a compromised website. Once the virus takes hold inside an organization, the spread is typically very slow until it manages to reach executable files on one or more file servers. Unfortunately, many organizations do not have antivirus applications deployed on their file servers. Once a file server is infected with Sality, it will spread very rapidly as users access infected executables on file shares. It is usually at this point IT and Security staff become aware of the infection. Once detected, Sality cannot always be successfully removed from an infected file. In some instances, the executable must be deleted. This, of course, means the application is no longer available until the file has been replaced. On a file server, replacing one, two or even a dozen executables files is not difficult. Having to 7 www.foundstone.com 1.877.91.FOUND

replace executable files, including operating system files, on hundreds or thousands of workstations is another thing entirely. It is important to note that the Sality virus is constantly modified and changed by the malware writers. We have identified variants of Sality that perform keystroke logging. Most variants phone home to look for instructions, additional tools, or even updates of the virus itself. If the virus manages to infect a system with Administrator level credentials, every file on that system is at risk of infection and data files contained on the system are at a serious risk of being transmitted to outside parties. If the virus has managed to be executed with Domain Administrator level privileges, the risk dramatically increases. Lessons Learned Suspect files can be submitted to antivirus vendors for automated analysis. If a malicious file has been discovered but your antivirus solution does not recognize it as nefarious, contact the vendor. Most antivirus vendors share information with each other. New variants of malware appear quickly and sharing malware behavior patterns helps everyone stay protected. Proxy outgoing traffic to create a choke point where it can be monitored and blocked. Consider an authenticated proxy, or personal outbound firewall, so malicious code cannot phone home without your knowledge. Categorize sensitive data within your organization and store it in a centralized location. This makes it easier to determine if sensitive information has been accessed by the malicious code. Remediation Now that you know what the malware is, the scope of the infection, and what resources it manipulates, a plan can be put in place to contain and remediate its impact. Many antivirus vendors produce stand-alone disinfectors, or stingers, for particular variants of malware. Most of these tools are free to download and use. For instance, McAfee offers a stinger which is available at http://vil.nai.com/vil/stinger/default.aspx. These stingers are very helpful when you have a serious virus outbreak. However, these should not be considered a comprehensive solution. The malware they identify is often very limited, and they do not get updated as regularly as a traditional antivirus engine. Most of these stand-alone disinfectors, or stingers, perform four tasks. First, they scan memory looking for compromised processes. If any hooks are found, they are removed. Second, the stinger scans local storage for infected files. Depending on configuration settings, a subset of files known to host malware or all files are scanned. We recommend that you scan all files on a suspect machine. Third, the stinger will attempt to clean any infected files it locates. If it cannot clean a file, it will ignore it, quarantine it, or delete it. Again, this action is configurable. Finally, the stinger will rescan the system to confirm it is has been remediated. 8 www.foundstone.com 1.877.91.FOUND

Remember, these tools do not have any on access protection mechanism. If one of these phases fails (e.g. removing the threat from memory), the malware can re-infect files that have been cleaned. Workstations If a workstation is infected, use a stinger or your antivirus product to do a full scan. Be sure to scan all files. Note any infections discovered and if the file was repaired, quarantined, or deleted. To be certain the device was successfully cleaned, reboot and run the scan again. There should be no further identification of the malware. When multiple scans continue to find something, it may be necessary to re-image the device. This is often faster than trying additional manual remediation steps. A very effective way to remediate stubborn infections is to use a virtual boot machine such as BartPE http://www.nu2.nu/pebuilder/. BartPE provides a memory resident version of Windows XP bootable from a CD/DVD. Virus scanning tools can be included on the boot disk. This allows the scanning of local drives without the potentially infected host operating system running. Servers In most cases, servers can be remediated in the same manner as workstations, although it usually takes longer due to the large number of files. Update the antivirus scanning software and perform a full system scan of all files on the server. Patch the system for all known vulnerabilities and note if any of the detected files could not be cleaned. This is where server remediation differs from workstations. With workstations, if an infected binary cannot be cleaned, there is usually another workstation that will have a good copy of the binary. With servers, there may be only one server running a particular application, and it is often a mission critical application. You may need to take that server offline for remediation. If remediation efforts fail on any server, the server will need to be rebuilt and updated before it can be brought back online. File Server / NAS / SAN Shared file systems are the hardest systems to clean effectively. The large number of files to scan means this effort can take a while, and if the server is still online, a user accessing the store increases the chance of reinfection or propagation. The best approach is to have an antivirus solution running on the fileserver or on the interface to the NAS / SAN that performs on-access scans for both reads and writes. This may cause performance issues in the short term. However, during a widespread infection, this is the best method of trying to protect the file servers. If a full scan is to be performed, the most effective method is to take the fileserver/nas/san off-line. This may be an inconvenience to users and cause business disruption, but it is the best way to ensure the server is malware free. 9 www.foundstone.com 1.877.91.FOUND

Another strategy you can use to speed up the remediation process is to have multiple machines mounting different file shares from your file servers and each performing scans on a portion of your storage array. Once everything is remediated, it is still not the end of the incident. Careful monitoring of antivirus logs and network monitoring has to continue for at least a month to be sure that every machine is malware free. Just like a forest fire, one last small flare-up can set everything in motion again. Vigilance is key. Lessons Learned Booting devices to a memory resident operating system using a tool like BartPE is a very effective method for remediating infected machines. Determine early in the incident what machines require manual remediation and which ones should be re-imaged. It is often faster and takes less effort to re-image a machine. If reimaging, ensure a known clean image that has been disconnected from the infected network is used. Servers and SAN/NAS devices need special treatment and a clear operational decision based on business impact and risk. Be mindful that the malware may be in your backup media. Restore data within the known infection period with care, and mark such backups to be used with caution. Remove a recent, known good backup media set out of rotation and secure in a safe location for at least six months. Continue monitoring your antivirus solution, network traffic, and educate users of the incident so any potential flare-ups can be caught and addressed quickly Conclusion Malware outbreaks are going to happen. Given the number of new malware variants released on a regular basis, the IT and security community is looking at a long battle. But hope is not lost. IT staff can better position themselves for the likelihood of a malware infection. Recent trends in malware infections have shown that there are some preparatory steps which can be taken to minimize the effects of a malware outbreak and aid in the removal of the infection. Business impact can be minimized by implementing some security best practices. Keep systems patched for all known security vulnerabilities. Limit user access on workstations by removing local administrator access. Disallow domain administrator access logins at the workstation. Keep virus software scanning engines and definition files up to date. Maintain and review antivirus logs. Disable autorun / autoplay of all devices. Configure antivirus software to alert and prevent execution of autorun.inf files. Enable an On Access scans for both reads and writes. Prevent execution of programs from any TEMP directory. Disable all unnecessary services at startup. 10 www.foundstone.com 1.877.91.FOUND

Enable a web proxy requiring authentication to help prevent malware from connecting to the Internet. Educate users about common malware infection vectors including web surfing, flash memory devices, email attachments, etc. as part of security awareness training. By implementing these security practices, an organization can help minimize the threat from a malware outbreak on their network and expedite the process of cleaning and remediation. About Foundstone Professional Services Foundstone Professional Services, a division of McAfee. Inc. offers expert services and education to help organizations continuously and measurably protect their most important assets from the most critical threats. Through a strategic approach to security, Foundstone identifies and implements the right balance of technology, people, and process to manage digital risk and leverage security investments more effectively. The company s professional services team consists of recognized security experts and authors with broad security experience with multinational corporations, the public sector, and the US military. 11 www.foundstone.com 1.877.91.FOUND