In-Band Methods of Virtual Machine Detection

Transcription

1 GRADUATE OPERATING SYSTEMS 1 In-Band Methods of Virtual Machine Detection Estefan Ortiz & Cory Hayes University of Notre Dame {eortiz, chayes3}@nd.edu Abstract In a proof of concept paper, the authors show that it is possible to run a virtual machine based rootkit (VMBR) that is able to gain access to system resources and avoid detection and effective removal. A VMBR essentially installs a virtual machine monitor and runs the native operating system as a guest. Once installed, the VMBR can run a myriad of malicious services. Three general types of malicious services that can be executed without detection from the system are: services that do not interact with the target system, services that observe the system in effort to gain sensitive information, and services that disrupt the targeted system. Attacks that do not interact with the system are of a particular concern because they can include distributed denial-of-service attacks and phishing web servers. It is under this premise that we examine the viability of detecting traces of a client (host) who is connected to a web server and communicates via a virtual machine. Index Terms Virtual Machine Detection 1 INTRODUCTION A virtual machine monitor runs in the software layer of the operating system and allows for the installation of multiple virtual machines which are defined as efficient, isolated duplicates of a real machine [2]. Virtual machine monitors must meet the following three requirements: programs running in a virtual machine monitor should have performances similar to what they would be in a physical machine, a majority of the virtual processes must be executed by the host processor, and the virtual machine monitor should have complete control over system resources [2]. Virtual machines are primarily known for use by software developers to test products on various architectures before release, but other advantages have been recently targeted and exploited. Chen and Noble [3] discuss how the isolation of virtual services from the physical host machine provides the benefit of security and portability, with virtual machine monitors having the ability to prevent and detect intrusions by observing all events occurring within a virtual machine while the host machine remains unaffected. A malicious program needs to be aware if the surrounding environment is a virtual one so that the program s behavior can be modified accordingly to prevent detection since the programs are unable to harm the host machine in this setting. Detection especially needs to be avoided because malicious programs are potentially open for analysis in the closed virtual environment. Through careful theorems and proofs, Gueron and Seifert [4] show that it is impossible to detect true virtual machine monitors. However, Gueron and Seifert also acknowledge that most implemented virtual machine monitors only come close to the strict criteria of what is considered to be a virtual machine monitor and a common observation seen in most virtual machine monitor detection methods are the indirect information leaks caused by imperfect isolation. Our project assumes Paper submitted December 16, 2011 the role of the ethical hacker by exploring attempts to infer information about a remote environment. The goal is for our project to function as a potential additional resource for future work where systems are modified to hide any telling aspects that compromise the identity of both physical and virtual environments and thwart malware anti-detection attempts. 2 RELATED WORK Paleari et. al [5] describe a method to determine if a program is being executed by a virtualized CPU. To accomplish the task of emulator detection, the authors examine the byte level behavior of a given set of instructions for an actual CPU and an emulated CPU and check the output byte information. Any disagreement between the output for the real and emulated CPU leads to the creation of one or more red pills. Red pills are tests that malicious programs can execute to determine if the environment is a real or virtual one. The tests are based on machine instructions that return some form of system information or have behavior that differs between real and virtual environments. These red pills were subsequently tested by well-known malware analysis programs, which ultimately resulted in effective evasion of detecting attempts. The primary focus of their work was to automate the generation of effective red pills and removal of unreliable ones. Perhaps the most dangerous tool created for malware anti-detection is the SubVirt framework introduced by King and Chen [1]. SubVirt is a manifestation of one of the ultimate goals of malicious programs: to gain control over the infected machine and prevent the machine from detecting the malware s presence. SubVirt installs a virtual machine-based rootkit underneath the host operating system and completely conceals its activity from most intrusion detectors by running the native operating system as a guest. The approach is based on the protection ring concept of operating systems where

2 GRADUATE OPERATING SYSTEMS 2 lower levels of a system control or have more privileges than upper levels. This installation at a lower position allows the rootkit to run all kinds of malicious services to attack the targeted physical machine while remaining nearly undetectable. Potential solutions are presented in the paper through a combination of using security software both above and below the virtual machinebased rootkit to monitor inconsistencies in CPU overhead, memory, and virtualizing I/O devices. In contrast to the above work where attackers try to detect a virtual environment and/or hide from it, a recent paper by Vishnani et. al [6] directly addresses security analysis by introducing a tool to trick malware. Their approach targets the six main types of malware s detection attempts: hardware fingerprinting, registry checking, memory checking, virtual machine communication channel checking, timing analysis, and process checking. By dynamically adding code in various parts of executables, their tool VMdetectGuard is able to mimic a native environment and throw off malware attempting to analyze the system. Both Garfinkel and Rosenblum [7] and Kourai and Chiba [8] discuss approaches that attempt to prevent attacks by using a virtual monitor that isolates its detection systems outside of the physical machine to limit malware anti-detection attempts. Zhu and Chen [9] note that some detection techniques implemented by malware attempt to call instructions that typically originate in the operating system kernel, such as SIDT, SLDT and SGDT from the Intel instruction set. If any of these calls are done in the user-level, as is the case with malware programs, then this may be a flag that some malware is attempting to gather information about the environment. Scanning malware for these instructions is difficult, Zhu and Chen developed a plugin called Malaware that runs suspicious programs in an emulator and examines program behavior on the instruction level. To combat the initial limitation of only being able to recognize known virtual machine techniques, they also developed another approach to detect any attempts of malware changing its execution path, allowing for the dynamic discovery of new virtual machine detection techniques. Another possible path for detection from a server is via in-band methods where TCP/IP packets are monitored or analyzed for odd behavior. Chen et. al [10] focus on artifacts left behind by virtual and physical environment interactions that are hard to conceal, such as the perturbed clock information of timestamps found in the TCP packets of hosting virtual machine monitors. They note that TCP timestamp clocks increase with a fixed frequency between 1Hz and 1000Hz depending on the host operating system, and since there are only a few common TCP clock frequencies, a machine can be reliably identified by the frequency. While native operating systems rely on accurate hardware-triggered interrupts, guest operating systems in a virtual machine rely on software interrupts created by the virtual machine monitor, and these interrupts can be potentially delayed or lost completely. These delays or lost interrupts are reflected in the timestamp information and can be used to detect the presence of a virtual machine. Our project follows along this path by attempting to infer information about a remote client by analyzing the TCP/IP packets sent between the two machines. 3 IN-BAND METHODS OF VIRTUAL MACHINE DETECTION This paper examines the feasibility of determining if a client is connecting to a server which is hosted via a virtual machine. In particular, our research focuses on a structured way of examining data packets exchanged from a client to a server hosted on a virtual machine and data packets gathered from a client to the same server hosted on the native operating system. We developed a general man-in-the-middle framework to collect packets for the various underlying operating systems and server configurations. The Man-in-the-Middle platform created was used to examine the following two server host configurations: 1) The server is hosted on the native operating system. 2) The server is hosted on a virtual machine running on top of the native operating system. The two hosted server scenarios can conceptually be seen in Figure 1(a) and Figure 1(b) respectively. Each setup found in Figure 1 displays the general structure of our man in the middle setup for packet gathering. The following sections provide the details for each of the major components specified by the man in the middle system. Fig. 1. Experimental setup 3.1 Man-in-the-Middle In general, a MITM configuration can be used to intercept and disrupt or manipulate packets bound from a

3 GRADUATE OPERATING SYSTEMS 3 source to a given destination. Although packet manipulation is one possible way that a MITM can be used to elicit a response from a server to indicate if it is being hosted on a VM, we consider the MITM to be a passive listener of packets being transferred from client to server. Thus, in the setup described in Section 3, the MITM will consist of a packet sniffer that resides on the client side monitoring traffic to and from the client s network interface card (NIC). Wireshark [11] was the packet sniffing software used to create the described MITM. Wireshark is an open source packet analyzer based on pcap (packet capture) which has a user interface allowing various real-time filtering options during the packet capturing process. Due to being built on top of pcap, Wireshark has the capability to capture and interpret various communication protocols such as Transmission Control Protocol/Internet Protocol (TCP/IP), User Datagram Protocol (UDP), and Internet Control Message Protocol (ICMP). Our designed man-in-the-middle configuration made use of Wireshark because of its ability to quickly analyze packets as they are transmitted from client to server. However, in our analysis we made use of pcap and command line functions to convert the saved output of Wireshark into a more usable form. While Wireshark has the capability to capture different protocols, our research focused strictly on TCP/IP communication. The Experimental section describes in detail how Wireshark and pcap were used to gather and prepare packet data for analysis. 3.2 Client and Server To estimate if there are any differences found in the packet transmissions from a client to a server hosted on a native of virtual operating system, we created a simple switched network in which to run a set of probing experiments. Below are brief descriptions of the client and server and the roles they play in the experimental process Server In our experiments we examine two different native operating systems and a respective virtual machine that matched the underlying native operating system. The two different operating system examined are Ubuntu 32bit version and Windows Vista Ultimate 32bit. Table 1 provides a summary of the underlying respective hardware for each operating system to act as a server in our experiments. We used VirtualBox [12] as the x86 virtualization software package developed by Oracle to match each of the underlying examined operating systems. The remaining component in our setup is the server. For the server portion of the configuration we used Apache Web Server version 2.2. For each operating system we installed the aforementioned version of Apache. Similarly, for each virtual machine running on top of the native host, we installed and set up the same version of Apache. TABLE 1 Software and Hardware Specifications Client With the general framework as described in Section 3.2, the client consists of a machine that will make a connection to the web server send a malformed packet and then disconnect. The two different client configurations are given below: 1) Client: Windows Vista Machine (4GB RAM, 2.6GHz Intel Core 2 Duo, Windows Vista Ultimate 32-bit) 2) Client: Mac (4GB RAM, 2.4GHz Intel Core i5, MacOSX version ) 4 EXPERIMENTAL DESIGN To determine if there are observable differences in the packet data that has been sent from a server within a virtual machine verses a server that has sent a packet from the native operating system, we examine a pair of metrics to determine which information from the TCP/IP packet data is a potential indicator that a server is connecting from a virtual machine. To gain some understanding of the effectiveness of the chosen metrics we run a series of experiments that limit the influence that outside connections and multi-hop networks have on our contrived system. In other words, our experiments will aim to keep the number of packets produced for a given simulation setup consistent. In addition, we will attempt to maintain the order in which the packets are sent and received. The following sections describe the metrics used in our evaluation of locating differences in packet information as well as executed experiments and their respective results. 4.1 Developed Metrics The approach taken in this paper is a hierarchical one with respect to the metrics used to illustrate differences in packet data. Thus, the first metric is a high-level examination measuring the round-trip time for the client and server to initially establish a connection. The second metric measures the fractional amount by which two packets disagree in terms of bits. The two different metrics to compare packet information: 1) Round-Trip Time: Time difference between a synchronize packet (SYN) sent by the client to the server and the acknowledgement (ACK) packet received by the client from the server. 2) Difference in Bits: Fractional Hamming distance calculation comparing two packets.

4 GRADUATE OPERATING SYSTEMS Round-Trip Time The difference in time comparison is concerned solely with the synchronize packet (SYN) sent by the client to the server and the acknowledgement (ACK) packet sent from the server to the client. For each experiment, during the connection attempt from the client we capture the synchronize packet, its respective time stamp, and the acknowledgement (ACK) packet sent from the server and its respective time stamp. The time difference calculation is then made on the elapsed time taken from SYN to ACK and the onset of establishing a connection. Figure 2 displays this comparison for a given experiment. Fig. 2. Example of Time Difference Comparison Difference in Bits This metric examines the fractional Hamming distance between two packets. Briefly, the Hamming distance of a binary code is the number of bits that disagree when comparing two bit codes. Thus the higher the Hamming distance, the more two bit codes are not alike. The fractional Hamming is essentially a normalized Hamming distance by some given factor. The normalizing factor that we use is the number of bits actually compared. For example if two codes are the same length of 128 bits then the Hamming distance found between them will be divided by 128. However, it s not always the case that two codes are the same length and in such cases we compare up to the length of the shortest code. Thus, when the bit codes differ in size the normalizing factor will be the length of the shortest code. Figure 3 gives an example of two packets being compared bitwise. Fig. 3. Example of Bit Difference Calculation The Hamming distance signals that there exists a difference in the two packets being compared, and this signal leads to the analysis of the packets at the bit level to determine which TCP/IP fields correspond to the discrepancies. An example of a differing packet data is provided for each experiment when applicable. 4.2 Experiments An Apache server follows the standard HTTP format when communicating over a network. All of our experiments involve communicating with the Apache server using a malformed TCP/IP packet. A malformed packet is any packet that does not follow the standard HTTP format, which is typically in the format of Request, Requested Item, HTTP Version, a header, an empty line, and an optional message body terminated with CRLF. When the client is trying to establish a connection with the server, these correctly formatted HTTP requests are done automatically. After that, a custom malformed message that only contains a simple message and no header information is sent from the client to the server. The malformed packet used in the first three experiments contains the message Aloha. The client-server exchange is generated by a small Matlab script. The script initializes the connection to the Apache server by specifying the port and ip address, waits 10 seconds, sends the malformed message Aloha, and then terminates the connection. This process is done twenty-five times in each experiment. Wireshark captures the network traffic generated by the Matlab script and displays individual packet data in a legible format. The first 160 bits of each packet correspond to IP header and padding information while the remaining bits correspond to the TCP information. Each experiment is set up where the virtual machine operating system matches the operating system of the host machine. The targeted output for each experiment contains a consistent number and order of packets sent throughout the establishment of the client-server connection, the transmission of the custom malformed message, and the closing of the connection for each of the twentyfive iterations of the experiment. The following details the set up for each experiment: 1) Experiment #1: a) Client: Windows Vista Machine b) Server: Ubuntu w/ Apache Web Server 2.2 c) Server: Host OS Ubuntu: VirtualBox w/ Ubuntu running Apache d) Network: Local switch network with no outside traffic 2) Experiment #2: a) Client: Mac b) Server: Windows Vista Machine w/ Apache Web Server 2.2 c) Server: Host OS Windows Vista: Virtual Box w/ Windows Vista running Apache d) Network: Local switch network with no outside traffic 3) Experiment #3: a) Client: Windows Vista Machine b) Server: Windows Vista Machine w/ Apache Web Server 2.2

5 GRADUATE OPERATING SYSTEMS 5 c) Server: Host OS Windows Vista: Virtual Box w/ Windows Vista running Apache d) Network: Both client and server on Notre Dame s CVRL am 4) Experiment #4: a) Client: Windows Vista Machine b) Server: Ubuntu w/ Apache Web Server 2.2 c) Server: Host OS Ubuntu: VirtualBox w/ Ubuntu running Apache d) Network: Server on Notre Dame CVRL subnet, client on outside network. Experiment #4 has a slightly different setup from the rest. Instead of keeping the client and server on the same network, the client is connected to the Internet through the use of a Sprint Mobile 3G/4G Hotspot device while the server is connected to Notre Dame s CVRL subnet just as it was for Experiment #3. This experiment was done in an attempt to move away from the relatively ideal setup of the first three experiments. Unfortunately, the recent establishment of a firewall on the Notre Dame network prevented the observation of actual packet data, so ping timing between the client and server was recorded instead. 5 RESULTS Our results show that there appears to be noticeable differences when examining the packet data that is generated when the server (Apache) resides on the native operating system when compared to the situation where the server is being hosted by the VM. As will be shown in subsequent sections, the differences found in the fractional Hamming metric provided an indication that there were varying bit level differences when examining packets from a VM versus packets from native machine. The following sections provide the results for each of the experiments as described in Section Experiment 1: Client Vista, Host (Native and VM) Linux This experiment was conducted under a constrained environment in which client and host were connected to a switch with no traffic from outside sources. As an initial indicator we can see in Figure 4 that there is a slightly larger fractional Hamming distance examining packet that are passed from the web server hosted by the native machine versus the packet passed from a web server hosted on a virtual machine. Figure 5 provides the round trip time measurements taken for the case in which the client is the Vista machine and the host (VM and Native) is the Linux system. Lastly Figure 6 provides the aggregate bit difference count of the eighth packet from the set of packets captured during experiment 1. Figure 6 is displayed here to highlight the bits differences that occur when comparing similar packets originating for the two different connection types (Native and VM). Fig. 4. Boxplot of fractional Hamming distances calculated for the each of the described server configurations with the Windows machine as acting as the client Fig. 5. Boxplot of timing tests for a SYN ACK sequence with the Windows machine as acting as the client 5.2 Experiment 2: Client Mac, Host (Native and VM) Vista This experiment was conducted under a constrained environment in which client and host were connected to a switch with no traffic from outside sources. Similar to the first experiment we see in Figure 7 a elevated fractional Hamming for the virtual machine to native machine packet comparisons. Figure 8 provides the round trip time measurements taken for the case in which the client is the Mac machine and the host (VM and Native) is the Vista system. As in the first experiment Figure 9 shows bits which differ when examining packet comparison between the web server on the native machine verses the web server on the virtual machine.

6 GRADUATE OPERATING SYSTEMS 6 Fig. 6. Histogram indicating the number of time a specific bit position differed when comparing the ordered packets. Fig. 8. Boxplot of timing tests for a SYN ACK sequence with the Mac machine as acting as the client and the Vista acting as the host Fig. 7. Boxplot of fractional Hamming distances calculated for the each of the described server configurations with the Mac machine as acting as the client and the Vista acting as the host 5.3 Experiment 3: Client Vista, Host (Native and VM) Linux Unlike the previous two experiments the client and host were connect to the subnet of the Computer Vision Research Lab. At the time the experiment was conducted, there was some network traffic but not to an overwhelming amount. Again we see an higher fractional Hamming distance for the compared packets of the virtual machine verses the native machine Figure 10. Figure 11 provides the round trip time measurements taken for the case in which the client is the Mac machine and the host (VM and Native) is the Vista system. Figure 12 shows bits which differ when examining packet comparison between the web server on the native machine verses the web server on the virtual machine. Fig. 9. Histogram indicating the number of time a specific bit position differed when comparing the ordered packets. 5.4 Experiment 4: Round Trip Time Examining the variations in the round trip from the first three experiments we conducted one last experiment to attempt to determine if the round trip time could be used as a virtual machine indicator. Figure 13 6 ANALYSIS The first thing to note is from each experiment is that there is an elevated fractional Hamming distance when comparing similar packets of different origins. With the varying fractional Hamming distance we probe the bit level differences to find that there are bits that differ in the TCP packet information that could be an indicator of a possible virtual machine connection. Both the fractional Hamming distance and bit level differences were found to provide a potential feature in which indicate that a

7 GRADUATE OPERATING SYSTEMS 7 Fig. 10. Boxplot of fractional Hamming distances calculated for the each of the described server configurations with the Windows machine as acting as the client Fig. 12. Histogram indicating the number of time a specific bit position differed when comparing the ordered packets. Fig. 11. Boxplot of timing tests for a SYN ACK sequence with the Windows machine as acting as the client client was connected to a VM. However, the round trip time was inconclusive at best and should not be used to indicate if a client is connected to a virtual machine. 7 CONCLUSIONS & FUTURE WORK Malicious programs need to be aware if they are running in a virtual environment so that they may modify their behavior to avoid detection in the isolated system. Previous research has found that it possible for a program to infer information about its environment through the monitoring of system calls and instructions. The goal of our work was to examine the viability of detecting traces of a client who communicates via a virtual machine to a web server through the examination of the TCP/IP packets sent over a network. We examined packet information from a high level down to specific bit Fig. 13. Ping average round trip times as a function of varying byte size for pings to the VM and Native Machine from outside the subnet of CVRL comparisons and found that timing tests did not provide conclusive evidence of a connection to a virtual machine. Our fractional Hamming distance metric provided the first level of insight to finding these discrepancies and subsequent analysis of the corresponding bit-level data within the individual TCP/IP packets showed that differences could be found in both the TCP and IP portions of the packet. Future directions for this topic would include transitioning from the ideal scenarios with where network traffic is restricted and analyzing packets on multi-hop connections. REFERENCES [1] C. P. W. Y. V. C. W. H. L. J. King, S., Implementing malware with virtual machines, in In IEEE Symposium on Security and Privacy, May 2006, pp

8 GRADUATE OPERATING SYSTEMS 8 [2] G. Popek and R. Goldberg, Formal requirements for virtualizable third generation architectures, in SOSP 73 Proceedings of the 4th ACM Symposium on Operating System Principles, October [3] P. Chen and B. Noble, A fistful of red-pill: How to automatically generate procedures to detect cpu emulators, in HOTOS 01 Proceedings of the, May 2001, pp [4] S. Gueron and J. Seifert, On the impossibility of detecting virtual machine monitors, vol. 297, pp , [5] M. L. R. G. Paleari, R. and D. Brushi, A fistful of red-pill: How to automatically generate procedures to detect cpu emulators, in In Proceedings of the USENIX Workshop on Offensive Technologies, [6] P. A. M. R. Vishnani, K., Detecting and defeating split personality malware, in SECURWARE 2011: The Fifth International Conference on Emerging Security, [7] T. Garfinkel and M. Rosenblum, A virtual machine introspection based architecture for intrusion detection, 2003, pp [8] K. Kourai and S. Chiba, Hypersector: Virtual distributed monitoring environments for secure intrusion detection, in In Proceedings of the 1st ACM/USENIX International Conference on Virtual Execution Environments, 2005, pp [9] D. Zhu and E. Chin, Detection of vm-aware malware, [10] A. J. M. Z. B. M. Chen, X., Towards an understanding of antivirtualization and anti-debugging behavior in modern malware, pp , June [11] Wireshark. [12] O. I. VirtualBox,