How To Identify Different Operating Systems From A Set Of Network Flows

Passive OS detection by monitoring network flows Siebren Mossel University of Twente P.O. Box 217, 7500AE Enschede The Netherlands s.mossel@gmx.net ABSTRACT` Network flow monitoring is a way of monitoring network activity without looking at individual packets or the payload of these packages. This paper proposes a method to detect a specific operating system in a network within a set of network flows. This is desirable because it is not feasible to capture individual packets or to inspect payload of the network traffic of a company or university. An administrator might want to know which operating systems are being used in his/her network. The update procedure of the operating system is different for different operating systems. This could be visible within network flows. The method is demonstrated by a proof of concept and validated using real flow data from the routers of the University of Twente. Keywords Network management, Flow monitoring, Fingerprinting, OS Fingerprinting 1. INTRODUCTION Network and computer management is a complex task in environments such as corporations and universities. One thing an administrator might want to know is what operating systems are being used. This is useful from a security perspective. Another scenario might be that the company or university wants to upgrade all OS licenses. Therefore the company wants to know how many licenses they would need to buy. Sometimes all computers are managed by a computer administrator. In this case, it might be possible to make an inventory. In most cases, however, this will not be possible. When employees or students manage their own computers and install operating systems, the administrator does not know which operating systems are being used. A possible solution for identifying operating systems in a network would be to identify the operating systems by their network signature, using a tool such as p0f[1] and PRADS[2]. Unfortunately, this is not possible for large networks such as campus networks. These tools require full headers of the network traffic. The University of Twente for example has an average bandwidth usage of 5 gb/s. It is not feasible to capture all this traffic. Another approach could be an analysis based on network flows. A network flow is a sequence of Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. 16 th Twente Student Conference on IT, June 25 st, 2012, Enschede, The Netherlands. Copyright 2010, University of Twente, Faculty of Electrical Engineering, Mathematics and Computer Science. packets from a source computer to a destination. The advantage of capturing network flows instead of all packets is it takes fewer resources. Another advantage is that flows are less privacy sensitive since only source and destination are captured and not the headers or payloads. Based on these flows, we aim at identifying different operating systems. The goal of our research is to find out how we can identify different operating systems from a set of network flows. Our assumption is that the update procedure of the OS is different for different operating systems. This would be visible when we look into the network flows. This leads to our main research question: How can we identify different operating systems within a set of network flows by looking at contacted update hosts? To answer this question the following sub questions will be answered: 1) What are the differences between the update procedures of operating systems at flow level? 2) Are the versions of an operating system different enough to be distinguished at flow level? In order to answer these research questions the following three steps were taken: The first step was to learn the flowbased signatures. The way we learned these signatures are described in Section 3, while the characteristics of the update procedures of Operating Systems, obtained from these signatures, are discussed in Section 4. With the information acquired we created a prototype. The prototype is discussed in Section 5. The last step was testing this prototype using a real set of traces. Section 6 gives the results of our validation of the prototype. We show the prototype classifies up to 93% of the Operating Systems correctly. 2. RELATED WORK Several works have been published about OS Fingerprinting. Taleck[3] describes how an Intrusion Detection System can gain more detailed information about the end host in a passive environment. He maps the captured TCP headers with a prebuilt table containing fingerprints of operating systems. Since we are not capturing the TCP headers, we cannot use this approach. Tools such as p0f[1] and PRADS[2] need TCP headers as well, therefore they are not useful either. Another way of detecting the OS running on a remote host is by active fingerprinting. Tools such as nmap[4] can help to do this. Active fingerprinting is done by sending faulty packets to a host. The values returned by the host can be used to determine the OS. The disadvantage of this approach is it generates a lot of network traffic. Our approach differs from this approach; we used a passive approach. Perelman et al.[5] use a flow based approach to identify applications in a flow trace. They were able to identify applications such as Skype, Opera and Chrome within user

flow traces. However they did not try to identify operating systems, which we do in this paper. 3. FLOW-BASED SIGNATURES The first thing we needed to know was how to identify the Operating Systems from a set of network flows. In order to learn this we created a lab setup. With this lab setup we captured the network traffic of Operating Systems both while doing manual and automatic updates. The next step was to convert this network traffic into flow records since we want to detect an operating system just from a set of network flows. The following sections explain what a network flow is and how we used these flows to learn the flow-based signatures. 3.1 Network flows A traffic flow or network flow is defined by RFC3917[6] as a set of IP packets passing an observation point in the network during a certain time interval. It can also be seen as a summary of the network traffic. A flow record typically consists of source and destination IP addresses, source and destination ports and the protocol used such as TCP/UDP/ICMP. The duration of the flow depends on the settings of the flow collector. One setting is the idle-timeout setting. This setting specifies the maximum arrival time between packets to still belong to the same flow. Another setting is the active-timeout. This setting specifies the maximum duration of a flow. 3.2 Lab setup and data capturing We made a selection of operating systems such that we could create a proof of concept. According to NetMarketShare[7] 92.5% of the Desktop users use a Windows Operating System, 6.4% use a Mac OS Operating System and 1% use a Linux distribution. For this reason we installed a Windows Operating System, a Mac OS Operating System and a Linux distribution. We chose Windows XP because the license was easier to obtain. With the lab setup we were able to answer sub question 1. Around 2% of the Operating Systems in total is Android; for the sake of comparison we selected Android as well. We installed a machine with Virtual Box and 4 different guest operating systems: Windows XP, Ubuntu 11.04, OS X 10.6 and Android 2.2. Installing Windows XP and Ubuntu within Virtual Box was no problem. Installing OSX was possible with the OSX86 project. Unfortunately we were not sure if the update procedure would be the same as with regular OSX installations. Therefore we used another machine with OSX 10.7.2 installed. It was possible to install Android 2.2 with the Android-x86 project[9]. Unfortunately with this installation it was impossible to perform updates. Therefore we left Android of the results. Figure 1 shows the setup. Figure 1. The lab setup. Our first idea was to capture the flows for the guest Operating Systems at the host machine as shown in Figure 1. The only interface to capture was the Ethernet interface of the host machine. If we would capture this interface we would capture the traffic of all virtual machines. This would have made isolating typical Operating System update behavior more difficult. Instead, we installed Wireshark on the Windows XP machine and used tcpdump on the Ubuntu machine. We saved the dumps to the host machine. We only captured the first 128 bytes of every packet both with Wireshark and tcpdump. The first thing we tried to find out was how the update procedures look like. In order to identify the update procedure of the operating systems we wanted to compare the network traffic of operating systems performing no tasks with network traffic of operating systems which were updating. We performed two captures. First we captured the traffic of the Operating Systems for one week while not performing any tasks on these Operating Systems. This way we captured the background traffic with the automatic updates included. Then we captured the network traffic while manually updating every Operating System. The network traffic was dumped in the pcap format because that is the format used by Wireshark and tcpdump. In the next section we describe how we used this captured data to create flow data. 3.3 Converting captures to flow data Since we wanted to analyze flow data, we converted the dumped pcap files to IPFIX[10] data using yaf[11]. Yaf is an IPFIX exporter able to convert pcap files into flow data. IPFIX is one of the formats used for flow data. We used yaf with the following parameters: --idle-timeout 30 --active-timeout 120 uniflow The first parameter is the idle timeout. After 30 seconds a flow is considered to be complete. If more traffic follows after this time a new flow record is created. After 120 seconds of active traffic a new flow record is created. The uniflow parameter specifies that flow records will be created for both directions. We chose those parameters because these are the closest to the flows we retrieve from the core routers of the University of Twente. To be able to read this IPFIX data we converted this data with yafscii[11] to a human readable form. Yafscii is YAF Flow Printer. After this step we were able to analyze the captured data by hand. The results of this analysis are discussed in Section 4, which will give an overview of the update procedures at the flow level. 4. OS UPDATE PROCEDURES This section gives a high level description of the update procedures of Windows XP, Ubuntu 11.04 and OSX 10.7.2. The similarities and differences between the update procedures are discussed as well. 4.1 Windows XP When the update service of Windows XP is triggered, either manually or by the Automatic Update feature of Windows XP, the Operating System first sends a DNS request. After the response for this query, Windows starts a TCP session on a port number in the ephemeral port range suggested by IANA[12]. The address in this response is in the Microsoft IP range (65.52.0.0/14 at the time our lab setup captured the

flows). In this TCP session Windows communicates with a Microsoft update server. When Windows establishes there are new updates available it sends out another DNS request, this time for the Microsoft download server for Windows updates. The address for the download server (download.windowsupdate.com) resolves to a server hosted by Akamai Technologies. Akamai Technologies is one of the world s largest content delivery networks. After receiving the DNS query response Windows sets up another TCP session to a server of Akamai Technologies. The port number used for this session normally is the next one in the ephemeral port range. Figure 2 shows this communication. Figure 2. Updating Windows XP. As depicted in Figure 2, the port number used for the TCP session with www.update.microsoft.com was x, the port number for acquiring the update from Akamai is performed on port x + 1. When a hosts contacts an Akamai server there is not enough information to make a classification since both Windows as Mac OS contact this host. When the information is combined with the port numbers there is enough information to make a classification. 4.2 Ubuntu 11.04 The Ubuntu update manager is triggered manually or automatically. The procedure starts with a DNS query to archive.ubuntu.com. The DNS requests resolve to servers of Canonical Ltd. Canonical Ltd. is a software company which markets commercial support for the Ubuntu family of Linux distributions. At the time of the lab setup the Canonical IP range was 91.189.88.0/21. Ubuntu then establishes a TCP connection using a random port from the ephemeral port range. After a list with new updates is downloaded by Ubuntu, the Operating System sends out another DNS request for a download server located in the same country as the host running Ubuntu. The new updates are then downloaded. The port number used for the TCP sessions for these downloads are randomly chosen again from the Ubuntu ephemeral port range. Figure 3 shows the communication for the update procedure of Ubuntu 11.04. Figure 3. Updating Ubuntu 11.04. 4.3 OSX 10.7.2 The Mac OS X Software Update can be triggered manually or by schedule. First, Mac OS X puts out a DNS request on a random ephemeral port for the host swscan.apple.com. This request resolves to a server of Apple Inc. A list with new software is retrieved in an http session over a TCP connection on port x, where x is in the ephemeral range. After Mac OS X establishes new software updates should be downloaded, it sends out a DNS request for swcdn.apple.com. This resolves to a server of Akamai Technologies. This is shown in Figure 4. Figure 4. Updating Mac OS X.

When a hosts contacts an Akamai server there is not enough information to make a classification since both Windows as Mac OS contact this host. When the information is combined with the port numbers there is enough information to make a classification. 4.4 Distinguishing between versions This section explains some ideas to distinguish between versions of an Operating System. All operating systems have an ephemeral port range. This information can be exploited to classify versions of Operating Systems. We did not test, however, these ideas; this is only a theoretical description. 4.4.1 Windows Our assumption is that it would be possible to distinguish between Windows XP and Windows 7. There is no difference between the hosts contacted. However, there is a difference in the ephemeral port range. IANA suggests the range 49152 to 65535 for dynamic and private ports. Windows 7 follows this suggestion, while Windows XP and earlier versions use the traditional BSD range of 1024 through 4999 for its ephemeral port range. The information about the hosts contacted and the port number information is included in the flow data. With this information it should be possible to classify a flow either as Windows XP and older or Windows Vista and newer. 4.4.2 Ubuntu Ubuntu is a Linux distribution. A Linux distribution uses a kernel. Ubuntu 11.04 for example uses kernel version 2.6. Kernel version 2.4 and higher use the ephemeral port range 32768 through 61000 while kernel version 2.2 uses the BSD range of 1024 through 4999. Our assumption is one can distinguish different Linux kernels. With this information one can make deductions about the Linux version as well. 4.4.3 OSX All versions of OS X use the suggested port range of 49152 through 65535. Based on the information of port numbers one cannot distinguish between different versions of Mac OS. 4.5 Discussion What all update procedures of the Operating Systems have in common is they generally put out a DNS request for the main update server of the Operating System. Then they download a file with a list of new software. When new software is available the Operating System retrieves it from the download server of the Operating System. Another DNS request is send out. While Windows and Mac OS get their updates from a server hosted by Akamai Technologies Ubuntu retrieves the updates from one of its own servers, hosted by Canonical Ltd. Using the information from the contacted hosts together with port range unique signatures for major Operating Systems can be built. Different versions of a single OS can sometimes be distinguished from the used port range. This provides, however, only very coarse classifications. 5. BUILDING A PROTOTYPE The goal of the prototype is to classify flow records. The prototype can identify Operating Systems in a set of network flows. We used Java to implement the prototype. The layout of the prototype can be divided into three steps. First the prototype parses the data from the set of flows. The next step is analyzing every flow record. At the end a list of identified hosts is exported. Figure 5 shows a diagram of the prototype. Figure 5. Prototype. 5.1 Parsing the NFSEN data The prototype uses a set of network flows provided by NFSEN[13] as the input. NFSEN is the data collector used by the University of Twente. The captures we made for learning the signatures were captured by YAF. NFSEN is comparable to YAF, but since the University of Twente uses NFSEN we used this data. A flow record is exported in the following pipe format: 2 1318333975 335 1318333975 335 6 0 0 0 3537069421 6548 0 0 0 2186929017 49487 0 0 5 0 18 0 1 48 The integer notation is used for IP addresses. We needed to know the IP addresses to be able to confirm the source address was in the university network and the destination to confirm the destination server. We converted these IP integer notated IP addresses to Host Addresses with Java. Java has standard methods to determine if a Host Address is in a certain IP range. 5.2 Analyzing every flow record Figure 5 shows how every flow record is processed. We only take flows into account which originated from the University of Twente. Then we check if the destination has a destination in the range of Microsoft s update servers. If this is the case the corresponding source port number is stored together with the source IP address. The same happens for Apple s range. Then if a flow record has a destination to one of the servers of Akamai Technologies the source port is compared to the saved port for this source IP address. If they are preceding each other the prototype classifies this host as Windows or Mac OS. When the destination IP address is in the range of a Canonical update server the prototype classifies the host as Ubuntu. 5.3 Exporting the results After the whole set has been classified the list with identified hosts is exported. An example of an export is presented in Table 1. Note: All IP addresses in this document are anonymized. Any coincidence with real IP addresses is by pure chance.

Table 1. Anonymized results of classification of prototype When a computer runs a virtual machine with NAT the result looks like the result for the address 130.89.10.4. 6. RESULTS AND VALIDATION To validate the algorithm we identified the operating systems in a subnet of the University of Twente network, using the flow data exported by the cores routers of the university. A real set of traces was obtained with NFSEN. A subset of the flows of the University of Twente was captured for one week. This data was used as an input for the prototype. 6.1 Detected Operating Systems We detected 228 Operating Systems on 217 different hosts. Of those 228 machines 167 were classified as Windows, 32 as Ubuntu and 29 as Mac OS. Figure 6 shows this division. Because checking a host is quite time consuming we only checked the first 50 results of the prototype. We can see here the prototype classified 22 machines as Windows. The 20 IP addresses classified by Nmap as Windows were classified as Windows by the prototype as well. The two Mac OS and 7 Ubuntu classifications were classified the same by the prototype. Nmap classified one host as Ubuntu while the prototype classified it as Windows. Another host was classified as Solaris 8 by Nmap and as Windows by the prototype. The findings of the prototype are the same for 94% of classifications by Nmap. 6.3 Validation by hand The second validation was done by hand. We checked 12 computers. The users were asked their IP address and the Operating Systems they were using on that machine. Two users were using a virtual Operating System together with their main Operating System. Table 3 shows the results of the validation. Table 3. Confusion matrix by hand vs. prototype Figure 6. Division of Operating Systems The subset of the flows contained a maximum of 362 IP addresses. With the prototype 60% was identified. The other 40% consists of Operating Systems other than Windows, Ubuntu or Mac OS; the hosts down during the capturing of flows; or the Operating Systems which did not perform any updates during the capture. In order to be able to classify all hosts the time frame could be extended for the captures. Another way to improve the results is creating more signatures for other Linux distributions. 6.2 Validation with NMAP Nmap is a utility for network discovery and security auditing. It can identify the Operating System running on a given IP address with the following command: Nmap O 130.89.x.x Nmap uses raw IP packets in novel ways to determine the Operating System using an active approach. We checked 50 of the 217 hosts. The results are shown in Table 2. Table 2. Confusion matrix nmap vs. prototype The prototype classified 5 IP addresses running Windows and 3 addresses running Ubuntu. All of these classifications were correct. The prototype classified 6 IP addresses as running Mac OS. Three of these IP addresses hosted a virtual machine according to the prototype. In two cases this was correct; in the other instance our prototype was incorrect. When we checked this host by hand we only found Mac OS. We looked into the flow records and discovered that this host started a MSN session. This resulted in a flow to one of the Microsoft servers. At the same time an Akamai server was contacted. The prototype concluded by this information a Windows Operating System was running. These false classifications could be avoided by classifying an Operating System only after recognizing the signature a few times. 7. CONCLUSIONS This paper discussed a method to detect Operating Systems using only flow data. This can be useful for a system administrator, for instance for security or inventory reasons. The advantage of using only flow data is the amount of data to be captured, which is only a fraction of the total data traffic. The main research question of this paper was whether it is possible to identify different operating systems within a set of network flows by looking at the contacted hosts for the updates. We conclude that it is possible to identify the main Operating Systems within a set of network flows. In order to achieve this we looked at the contacted update servers. When this information is combined with information about flows accessing a cloud provider such as Akamai we can classify Operating Systems. Section 5 covered the prototype which allowed us to do this. The first sub question addressed the differences of update procedures are at flow level. Section 4 shows the differences between update procedures. The main difference of update

procedures is the host contacted first by the Operating System. The last sub question addressed the question whether versions of an operating system are different enough to be distinguished at flow level. In certain situations this might be possible. Windows 7 for example could be distinguished from Windows XP because the ephemeral range differs. This, however, is not always possible since some Operating System versions do not differ enough at the flow level. There are a few ways to improve our prototype. The first way is to fully integrate the ephemeral range of different operating systems. Another possible improvement could be to include more data in the flow records. When IPFIX flow records are used extra information can be added. The following information can be useful to further improve our results: - DNS requests - TCP header - Content of first http packet of a flow All these possibilities are, however, out of the scope of this work and are left for future work. REFERENCES [1] M. Zalewski, "The new p0f." [2] E. Fjellskål, "PRADS - Passive Real-time Asset Detection System." [3] G. Taleck, "Ambiguity Resolution via Passive OS Fingerprinting," Lecture Notes in Computer Science G. Vigna, C. Kruegel and E. Jonsson, eds., pp. 192-206: Springer Berlin / Heidelberg, 2003. [4] G. F. Lyon, Nmap Network Scanning: The Official Nmap Project Guide to Network Discovery and Security Scanning: Insecure, 2009. [5] V. Perelman, N. Melnikov, and J. Schonwalder, "Flow signatures of popular applications." pp. 9-16. [6] J. a. Z. Quittek, T. and Claise, B. and Zander, S., "RFC 3917: Requirements for IPFlow Information Export (IPFIX)," http://www.ietf.org/rfc/rfc3917, 2004]. [7] NetMarketShare. "Operating System market share," June, 2012; http://www.netmarketshare.com/operating-systemmarket-share.aspx?qprid=8&qpcustomd=0. [8] OSx86. "OSx86," October 1 2011; http://wiki.osx86project.org/. [9] Android-x86. "Porting Android to x86," http://www.android-x86.org/. [10] B. Claise, M. Fullmer, P. Calato et al., IPFIX protocol specifications, draftietf-ipfix-protocol-03. txt, 2004. [11] C. M. Inacio, and B. Trammell, YAF: yet another flowmeter, in Proceedings of the 24th international conference on Large installation system administration, San Jose, CA, 2010, pp. 1-16. [12] IANA. "IANA - Internet Assigned Numbers Authority," http://www.iana.org. [13] P. Haag, "Nfsen: Netflow sensor. nfsen. sourceforge. net," April, 2008.