A Framework for Stateful Inspection

Transcription

1 A Framework for Stateful Inspection - Applied to TCP and Linux Netfilter Group D505c Mikkel Refsgaard Bech Torben Vinther Schmidt Carsten Stiborg Department of Computer Science Aalborg University Jan 14th, 2002

2

3 Project group: D505c Group members: Mikkel Refsgaard Bech Torben Vinther Schmidt Carsten Stiborg Supervisors: Mikkel Christiansen Emmanuel Fleury Numbers of Copies: 7 Number of Pages: 66 Synopsis This project examines the concept of stateful inspection and how it is applied in firewalls. First a framework containing a modeling language for networking protocols is derived. This is used to model the Transmission Control Protocol (TCP) which is used as case study. Then the concept of stateful inspection is applied to TCP in order to get a model for use in firewalls. Afterwards, the Linux Netfilter code is reverse engineered in order to create a model for comparison with the model for stateful inspection on TCP. The result of the comparison is a proposal for improved stateful inspection within Linux Netfilter. The report concludes that the framework was applicable in improving stateful inspection within Linux Netfilter. Aalborg University - Fredrik Bajers Vej 7E - DK-9220 Aalborg - Phone Telefax

4

5 This project was prepared on the DAT5-semester at the Department of Computer Science at Aalborg University, Distributed Systems and Semantics division, in the fall of 2001 and January It has been made with help from the open source community and their resources (homepages, FAQs, HOWTOs, and mailing lists). The goal of the project was to provide a framework for improving implementations of stateful inspection firewalls. We propose a framework for modeling network protocols and use this to model stateful inspection on TCP. We have reverse engineered an implementation of stateful inspection in order to model it. Based on a comparison of the models we propose improvements for the implementation. Our approach is not deeply theoretical but emphasizes application of our findings to practical use. Throughout the report, there are examples, tables, and figures. These are all enumerated within each chapter, e.g. Figure 3-1 means the first figure in chapter 3. In the appendix A and B are important parts of the source code of the implementation. Aalborg, January 14th, 2002 Mikkel Refsgaard Bech Torben Vinther Schmidt Carsten Stiborg iii

6

7 1 Introduction Firewalls The Generic Firewall Open Systems Interconnection Packet, Stateful Inspection, and Application Firewalls Packet Firewalls Stateful Inspection Firewalls Application Layer Firewalls Alignment and comparison Comparison of Firewalls Improving Stateful Inspection Firewalls Project Objective The Structure of the Report Framework The Formalism Graphical representation The Transmission Control Protocol The Protocol The TCP Header Connection Phases Connection Establishment Established Connection Connection Termination TCA for a TCP Connection Summary v

8 4.2.3 Resetting the Connection The Stateful Inspection Model Stateful Inspection Issues Model Completeness Passive or Active Other Protocols Summary The Netfilter Implementation Linux and Netfilter Netfilter Architecture Netfilter Modules Implementation of TCP State Tracking Handshake Check Transitions The State Table The State Entry Summary Applying the Stateful Inspection TCP Model to Netfilter Similarities Differences Improvement Proposals Conclusion 53 A Excerpt of ip_conntrack_core.c 55 B ip_conntrack_proto_tcp.c 61

9 1-1 A typical proxy relaying FTP requests for a HTTP browser The SOCKS proxy relays the client requests through the SOCKS layer A TCA containing three states The header format for TCP Connection establishment for the client Connection establishment for the server Active connection termination, shown as a TCA Passive connection termination, shown as a TCA This TCA shows how a TCP connection acts as specified The retransmission workaround The ACK is blocked even though it is legal, because the state has changed back The final state diagram for stateful inspection on TCP for the client The final state diagram for stateful inspection on TCP for the server The Netfilter architecture Schematic representation of the minimal Netfilter struct used at each state entry State changes, client side State changes, server side vii

10

11 1-1 The TCP/IP model aligned to the OSI model Where firewalls operate in accordance to the ISO and TCP/IP models Timeout periods for TCP as defined by ip_conntrack ix

12

13 During the last 10 years the use of computer networks has exploded. A network is a number of computers that are connected by some medium that allows them to communicate. Networks are established internally in almost all companies and professional organizations. Some networks provide connectivity through them to other networks in order to share resources. This creates a larger network, which is called the Internet. It may not be the case that all the resources that a network provide is willing to share them with others. Some resources are meant only for our private network. Each of the computers in a network has an address so it can be contacted. This also implies that if someone want to access this computer from any other network, that person is able to do that. This also means that this person could attack this computer and gain access to confidential information. That someone could access confidential information is of course unwanted. Therefore policies are created for what is allowed on a network. This policy can be expressed in many ways including, but not limited to, something as simple as piece of written paper or a list in the network administrators head. ½º½ Ö Û ÐÐ However, since the policy can be ignored by someone on a different network, a method for enforcing these policies are needed. A way to enforce the policies against other networks is by using a firewall. In this section we will define the generic firewall and introduce our perspective of different types ½º½º½Ì Ò Ö Ö Û ÐÐ of firewalls. In order to understand in which environment firewalls works, they will be aligned to the Open Systems Interconnection (OSI) reference model. The general types of firewalls that we will introduce are packet firewalls, stateful inspection firewalls and application firewalls. First, the generic firewall will be defined. As mentioned earlier, networks are composed of connected computers. Networks are used to send messages from one machine to another. The passing of messages between computers is called communication. We define that the private network is a network over which we have administrative control and want to protect from public networks, which is a network that we can not control, e.g. the Internet. The firewall is a machine which is on the border between networks and inspects communication that enters or leaves the network for messages that we do not allow in our policy. To define what 1

14 of formal rules for how communication between computers must proceed. These protocols exist on different layers. ½º½º¾ÇÔ ÒËÝ Ø Ñ ÁÒØ ÖÓÒÒ Ø ÓÒ In order to find out what layers protocols and firewalls work on we look at the Open Systems Interconnection (OSI) reference model. This is a standard for network protocols on different layers proposed by the International Standards Organization (ISO) [Tan96]. Each layer has a welldefined behavior and does not overlap. Following are the different layers, starting from the bottom layer ascending to the top. Physical The physical layer is the actual hardware, it is this layer that handles the actual communication, on this layer the data type considered are bits. Data Link The data link layer handles the errors originating in the physical layer, thereby assuring that the data sent and received are alike. This layer works with frames. Network The network layer is the first layer to handle end-to-end communication, the previous layers only consider the next step in the route. This layer considers the data sent and received as messages or rather packets. Transport The transport layer takes care of sending the data received from the session layer, splitting it up into smaller pieces if needed and concatenating the data it receives from the network layer, if it has been split. The main goal of the transport layer is to separate the network layer from the session layer, the network layer changing with the hardware it is built upon, and the session layer only being dependent on the operating system it is implemented in. [RAD01] Session The session layer enlarge the service provided by the transport layer, while the transport layer only handles data going in one direction, the session layer applies the possibility of traffic going in both directions. The session layer also implements the concept of checkpoints in the received stream. If either the sender or the receiver brakes down, the checkpoint is used to restart the transmission from its position. Presentation The presentation layer takes care of presenting datatypes like integers, strings, characters, floats, etc. correct on different types of machines. Application The application layer contains many different protocols, like , directory lookup, remote job execution, etc. 2

15 application. Because the OSI model is build in layers, an implementation on one layer can be replaced by another implementation without affecting the other layers. A more thorough description of the different layers can be found in [Tan96]. The OSI model is very general and this may be hindering when designing network protocols. Therefore, designers tend to cut corners. This is the case with the most commonly used protocols on the Internet. These are the Transmission Control Protocol (TCP) and Internet Protocol (IP) [TMW97]. These are almost always used together and therefore also known as TCP/IP. In Table 1-1 the TCP/IP model is aligned to the OSI model. As shown, the physical and data link layer is combined into one, since the software needed to control the physical layer always follows the network interface. The presentation and the session layer from the OSI model does not exist in TCP/IP. Some session procedures are, however, present in the TCP protocol, e.g. two-way connections. Also presentation is commonly handled on the application layer to the extend needed for that protocol. Layer OSI TCP/IP 7 Application Application 6 Presentation 5 Session 4 Transport Transport (TCP) 3 Network Internet (IP) 2 Data Link Host-to-network 1 Physical ½º¾ È Ø ËØ Ø ÙÐÁÒ Ô Ø ÓÒ Ò ÔÔÐ Ø ÓÒ Ö Û ÐÐ Table 1-1: The TCP/IP model aligned to the OSI model. ½º¾º½È Ø Ö Û ÐÐ We will take a closer look at three of most common types of firewalls, namely packet, stateful inspection, and application firewalls. In Section 1.3 we will place each of these types of firewalls in alignment to the models in Table 1-1. These are firewalls that operate on the Internet and Transport layer of the TCP/IP model and focuses on packets. This means that decisions are made based on single packets, i.e. not considering the context. 3

16 Example 1 Allow ½º¾º¾ËØ Ø ÙÐÁÒ Ô Ø ÓÒ Ö Û ÐÐ protocol TCP source IP = 192.*.*.* Deny ALL This says that TCP packets with source IP starting with the number 192 is allowed. If the packet does not apply to this rule, the next rule denies it. Stateful inspection firewalls operate on the network, transport, and application layer. They focus on protocols rather than single packets or data. We have found no definition of what stateful inspection is and firewalls that are claimed to be stateful differ a lot in functionality. In order to clarify what we mean by stateful inspection we will define it. In stateful inspection the protocol and the states of the protocol are known. Likewise, all messages of a communication is known and the states of each of the communications are known. On the basis of this we define it as: Determining whether the state of a communication conforms with the state of the corresponding protocol. This implies that the events dictated in a protocol are known and the sequence of these events are also known. We say that when a certain event has happened the protocol is in a certain state. Likewise, communication have the same events and therefore states. However, it can at some point differ from the protocol and when this happens it is known. While stateful inspection is only to determine the state of communication, stateful inspection firewalling is the act of filtering on the basis of the state of communication. In the rest of the report when referenced to stateful inspection, the act of filtering using stateful inspection is meant. What is done is ensuring that a communication behaves the way it should. If we discover that a communication changes state, we check if it is a legal state change and if not, react by discarding or rejecting the message, otherwise we allow it to pass and note that the state has changed for that connection. In addition to this, rule filtering can be done, i.e. determining by rules which kind of message is allowed based on the state of communication. This could be a rule which says that the communication is not allowed to change to certain state, e.g. a rule that says that a certain type of communication is not allowed to be initiated. 4

17 A proxy firewall is an application that relays packets for a client. The main idea is to deny all, except that which is allowed to pass through the firewall. What is allowed through the firewall is determined by rules for who is allowed to use it and what the firewall supports. Proxies are used for various reasons in addition to firewalling, e.g. a proxy can be used to mask the real IP-address of the clients using it. Also, it can be used to translate messages between protocols, e.g. letting a client use a Hyper Text Transfer Protocol (HTTP) browser to browse a File Transfer Protocol (FTP) site, communicating through HTTP to the proxy server, while the proxy server communicates with the FTP server through FTP. This is illustrated in Figure 1-1. Private Network Public Network HTTP Browser HTTP request HTTP Proxy FTP request FTP Server HTTP answer FTP answer ËÇ ÃËÈÖÓÜ Figure 1-1: A typical proxy relaying FTP requests for a HTTP browser. Similar to the proxies previously described, the SOCKS works as illustrated in Figure 1-2 on the next page. A client runs a program that works as a layer between the operating system and the application when the application accesses the network. Basically, it replaces the send and recv primitives used with sockets, which makes it operating system independent. The SOCKS client then connects to the SOCKS server which then in return relays the communication to the application server which the client applications is trying to connect to. The SOCKS client works like a wrapper for network communication and the SOCKS server then works as a gateway for the application, denying all other communication. Some applications have their own SOCKS client and therefore have no need for an external one. The SOCKS proxy technology has been developed by NEC [NEC01] and is free for non-commercial use. 5

18 SOCKS Client SOCKS Server Application Server OS OS OS Client SOCKS Server Server ÅÁÅ Û Ô Ö ÌÓÓÐ Figure 1-2: The SOCKS proxy relays the client requests through the SOCKS layer. Some firewalls look specifically at the content of the data, e.g. is it , news, etc. A typical example of such is the products of MIMEsweeper, like Mailsweeper and Websweeper. These are not placed on the border of networks, but instead with the applicable service, i.e. Mailsweeper is placed on the same machine that runs as a mail server. MIMEsweeper protects by providing a plug-in interface for handling different application protocols. This is e.g. done by providing an anti-virus tool for the Mailsweeper interface. Mailsweeper then uses this anti-virus tool with each mail going to or from the mail server. The ½º MIMEsweeper Ð ÒÑ ÒØ Ò ÓÑÔ Ö ÓÒ tools contain a very large interface for policy declaration, this is needed since data is separated into many distinct types on this layer. When defining the policies one should consider each possible data type that can be transferred through the network, e.g. movies, pictures, and documents which can be a quite demanding task. [MIM01] In Table 1-2 on the facing page each of the firewalls are placed corresponding to the layer they operate on using the layers from Table 1-1 on page 3. ½º º½ ÓÑÔ Ö ÓÒÓ Ö Û ÐÐ We can see that stateful inspection firewalls include filtering on the application, transport, and network layers. All that is really needed for stateful inspection firewalls is a protocol for which states exists. This means that the concept of stateful inspection could in theory be used on any layer of the OSI model. When considering which type of firewall to use there are different points which speaks for each of them and we will discuss them here. 6

19 2 Data Link Host-to-network 1 Physical Table 1-2: Where firewalls operate in accordance to the ISO and TCP/IP models. What kind of firewall one chooses is dependent on what kind of security is needed. It is possible to combine different kinds of firewalls for more expressiveness and more security. However, combining them could lead to a difficult configuration of the firewalls, as is often the case with using several programs for the same purpose, due to overlaps and conflicts. In other words it is preferable to only have one which cover all. Where, being kernel space or user space, the different firewall operates is essential for the efficiency of the filter. In kernel space interaction with the Transport and Network layer is done much faster and thus allowing for a higher load of communication. However, kernel space is very limited and very little can be stored. Filters that work in user space access the network through kernel space and thus are slower than filters in kernel space because they have to go through this layer. However, user space have much more storage space. As we want the firewall to be efficient we of course tend towards those that run in kernel space. Packet filters does not need to store anything and can therefore easily run in kernel space. Stateful inspection firewalls need to store information about the communication but not the content of messages and can also run in kernel space. Application firewalls need to store the content of messages, which can be quite a lot and must therefore run in user space. We also want a firewall to be able to express our policies fully and we therefore tend towards those which have a lot of expressiveness. When it comes to expressiveness, packet filters are limited because they can not express relations in communication. What is meant by this is that if we allow communication from a computer inside the network to a computer outside the network, we also allow for that computer to communicate in the other direction. We can not express that we only allow replies and not requests from that computer. Stateful inspection surpasses this by considering the communication instead of packets. Stateful inspection firewalls can then express that only if the computer from within the network starts to communicate to a computer outside, the computer outside is allowed to reply. When considering application firewalls we have even more information to base our rules on. These are able to express that a certain content is not allowed within a message. Where we want both expressiveness and efficiency of a firewall, stateful inspection offers the best tradeoff between these. Along with this and its ability to operate on all layers makes it a desirable type of firewall. 7

20 done. This should both be in terms of a protocol and a current implementation of stateful inspection firewall. The purpose is for the framework to pinpoint problems within the implementation and/or vice versa. In order to apply the framework, a model of the protocol from a firewall s point of view has to be ½º ÈÖÓ ØÇ Ø Ú made in addition to a model of the implementation. Comparing these will show differences and similarities. On the basis of this comparison, an improvement proposals for stateful inspection within the implementation must be made. When networks are joined together, security becomes an issue. Network administrators have a need to protect their networks from malicious use, especially from outsiders. Firewalls play an essential role in protecting networks from outsiders. Filtering communication on different network layers using only one tool is a step towards a better firewall, since information on all layers can be inspected, and decisions whether to accept or drop packets can be made from more information. One such type is stateful inspection firewalls which gathers information about the state of communication going to and from the network. It is on the basis of this information that it decides what kind of communication is allowed. Because this information can be gathered on different layers it has an advantage over packet filters and application firewalls. Because it runs in kernel space it is reasonably effective. Alas, this advantage does not come without a cost. Information has to be stored in the limited kernel space. The goal of this project is: To examine the concept of stateful inspection for use in firewalls and reach a framework for stateful inspection firewalls, the behavior of which is well-defined and unambiguous. This framework is proved by example by improving an implementation through the use of the framework. Through this project we contribute to the development of stateful inspection in firewalls by: 1. A framework for modeling network protocols for the purpose of stateful inspection. 2. A model of the transmission control protocol for use with stateful inspection. 3. A reverse engineered model of a implementation. 4. Propose an improved design for the implementation based on a comparison of the models. 8

21 using requirement specifications for TCP. Once this model has been created we will consider the model and the role of a firewall to create a model for stateful inspection on TCP. We then reverse-engineer a current implementation of stateful inspection to model this using the ½º Ì ËØÖÙØÙÖ Ó Ø Ê ÔÓÖØ language and compare it to the stateful inspection on TCP model. Based on a comparison of these two models we will propose a new design for stateful inspection in the implementation. The new design must be an improvement over the old and must be implementable. In Chapter 2 we will derive a modeling language for networking protocols. We will examine the TCP protocol and proposes a model of a correctly behaving connection in Chapter 3. In Chapter 4 we will discuss stateful inspection for use in firewalls and proposes a model of stateful inspection on TCP. In Chapter 5 we will reverse engineer an implementation of stateful inspection and a model of the code will be made. In Chapter 6 the model for stateful inspection on TCP and the implementation will be compared and the final design will be proposed. We will conclude in Chapter 7. 9

22

23 In the introduction we stated that one of the problems in stateful inspection is the lack of a formalized model used to describe the procedures of stateful inspection. In this chapter we will present such a model using automata to depict state and state changes within protocols. However, as regular deterministic automata does not suffice for modeling protocols we need to expand the concept of regular deterministic automata. ¾º½ Ì ÓÖÑ Ð Ñ We will first present an overview of the contents of a generic network protocol, then describe the formalism and finally an alternate graphical form of the formalism will be presented. Through this section a formal language will be defined to describe generic protocols. Before designing such a formal language, it must be clear what a generic network protocol consists of. ÙØÓÑ Ø Å Ì Ñ A protocol consist of a set of rules that dictates what sort of action should be carried out in a given situation. An automaton is a well defined modeling language [Sip96] that we can use to describe a protocol, since it changes state on the basis of the current state. We can say that protocols change their state because they act on a given situation, that is, they remember their situation, they are in a state. In a network protocol packets are transmitted and received, through these packets data are exchanged. In the framework it should be possible to model this, since data could be of meaning to the protocol, forcing a change of state. All protocols must somehow apply rules for ending their data exchange, but doing nothing is a quite legitimate way of managing it. However, it must also be handled by the protocol that nothing happens in the end. Timeouts is a way to handle the closing of a protocol. Even if the data exchange is nicely closed with special purpose packets, it may be a problem if the packets did not arrive, so time is still an important part of a protocol. 11

24 The models for network communication will from now on be described as Timed Counter Automata (TCA) as it includes a subset of both timed automata, introduced by [AD90], and counter andì Ñ ÓÙØ¾Æ ½ automata as described by [Min67]. A TCA is a 6-tuple (É Õ¼ Î ), where: 1.Éis a finite set of states. Each state contains a timeout described by: Ø Ì Ñ ÓÙØ, whereøis a global clock Timeout describes the maximum amount of time in which it is allowed to stay in the same state. 2.Õ¼¾Éis the initial state. When the automaton is started, the global clockøis equal to¼. 3. is a finite set of messages. Messages describes contents of a packet which the automaton models. This could be a predefined bit-sequence in the data header of a packet. The set of is defined by all variable names of the content of possible transmittable and receivable packets, including. Before creating ¾ Ú Îand Ú Î a TCA it must be clear which messages exist, and their content. 4.Îis a finite set of bounded variables in.úis the set of global variables and Úis the set of variables contained within the packet. Thus: holds: Ú Ú Ú Ú ¾ ½ Ò Where the following 12 5.ƾ É Å Í É, is the set of transitions where: ¾ ØÖÙ Ü Ü Ý Ø Ø Ý µ and ¾ withü Ý¾Ú Ú,¾Æ, is the set of all possible guards. A guard is a boolean expression which must be evaluated to true in order for a transition to be taken. Variables must be equal to either constants, or other variables. Every entry is defined by an expression based on the following grammar:

25 Ù¾Í Ù Ü Ü Ý Ù Ù withü Ý¾Ú Úand¾ On each transition the global clockøis reset to¼. 6. Éis the set of accepted states. This is a state which is equivalent to the termination of ¾º¾ Ö Ô ÐÖ ÔÖ ÒØ Ø ÓÒ a protocol. Now we have defined the TCA as a 6-tuple we will define a graphical representation of it. Here we will present a graphical representation of the TCA. The graphical representation is not as strict as the formal description. The general rules which should be followed are listed here: States should be drawn as ellipses. With their names in the upper line inside the ellipse, and the timeout period below. The timeout period should be without theø as it is always the same, moreover, whenì Ñ ÓÙØ ½no timeout value should be written. Neither the corresponding transition be shown, since it would never be taken. The initial state should be drawn as a double lined ellipse. Transitions are drawn as arrows between the states. Conditions on the transitions (Guards, Messages, and Assignments) should be written above or right of the transitions they affect, where each line is ANDed together. Accept states should be drawn as rectangle. These are only general guidelines for how to depict TCAs in general depiction should be done as seen most fitting, using the given syntax, see 2. Example 2 Figure 2-1 on the next page illustrates a simple automaton with three states. One is the initial state, A, and one is an accept state, end. From the initial state a transition can be taken when a message of the type packet is transmitted, this message will, before being transmitted, have its local variable named Answer assigned the value of the unknown global variable The_Answer. This model do not care where the variable is set, it could be on another level of the protocol or simply a part not described. The state A do indeed have a timeout 13

26 transition will be taken, which leads to the accepting state end. A packet! packet!.answer := The_Answer packet? packet?.answer = 42 B t=11000 end Figure 2-1: A TCA containing three states. 14

27 In this chapter we look at how a TCP connection behaves and focus on what is relevant for stateful inspection from a firewall point of view. We use the previously defined Timed Counter Automaton to provide a model for TCP. º½ Ì ÈÖÓØÓÓÐ In this section we will focus on a specific protocol, namely TCP. We have chosen to use this protocol as our case study because, as shown in [TMW97], most of the traffic on the Internet use this protocol. TCP provides a connection-oriented reliable channel of communication between two º½º½Ì Ì ÈÀ Ö peers. The basic TCP is described in RFC 793 [Pos81]. To provide a reliable connection-oriented channel of communication the protocol specification describes a header, connection establishment, connection termination and data communication. These elements of the protocol will be described in the following with focus on what is needed for stateful inspection. 0 Bit 16 Bit 32 Bit Source Port Destination Port Sequence Number Acknowledgment Number Data Offset Reserved URG ACK PSH RST SYN FIN Window Checksum Urgent Pointer Options Padding data Figure 3-1: The header format for TCP. The header is a chunk of data prepended to each data packet. The format of the header is shown in Figure 3-1. It contains information about which port of the sender of the packet it is send from, and to which port on the receiver it is intended. The port numbers, together with the IP-addresses from 15

28 received. In TCP it is not required that every packet received is acknowledged, but instead every byte transfered has been acknowledged. The field named data offset is used to describe how many 32 bit words are currently present in the TCP header. The reserved field is not used and must be set to 0. The following 6 bits are used to determine what type of data is attached, and which fields in the packet are used. The first bit is the Urgent flag (URG), it signals that urgent mode is activated. The Acknowledge flag (ACK) indicates that this packet acknowledges some received data. The Push flag (PSH) is a reminiscence from early implementations of TCP. Its meaning is to push the data sent forward to the receiving application without waiting for further data. The Reset flag (RST) is a control flag that tell the peer that the connection has to be dropped, e.g. as a reply to a faulty synchronization. The Synchronize flag (SYN) is used to start the connection by synchronizing the two parts acknowledgment numbers. The Final flag (FIN) is used to signal that there is no more data from the sender. A packet with one of the flags set will be noted as being of that type, i.e. a SYN/ACK packet is a packet with the SYN and ACK flag set. The Window field contains the size of the senders current data buffer, implying that no more data than that specified in this field can be sent to the sender. If the receiving TCP implementation has not yet delivered the data to the appropriate application, the window size will become smaller. In other words, this is the size of the data buffer subtracted the size of the data it contains. The Checksum field is used to minimize the risk of badly transferred data. The Urgent Pointer points to the last byte of urgent data. The Options field is a variable field. It can contain multiple options selected for the current packet. Each options may vary in size and Padding is applied in order to make the header be composed of 32-bit words. The fields that are interesting for doing stateful inspection are those that change the state of the connection, identify the connection or somehow can be used to confirm the correctness of the connection. These fields are: 16 Source port and destination port, because they, together with the source and destination address from the IP layer, uniquely identify the connection. Sequence number and acknowledgment number, because they show the progress of the connection. The data length of a packet and window size are also needed for flow control which determines how a connection can progress. The flags ACK, RST, SYN, FIN, because they determine and change the state of the connection.

29 Ì ÑÓ Ð A TCP connection session can be divided into three phases, namely establishing the connection, transmission of data and connection termination. Ô Ø Ô Ø ÝÒ Ð ØÖÙ µ Ô Ø Ð In this section we will describe these phases. First we will consider some modeling issues and then move on to connection establishment. For each part we will make a TCA. In order to simplify the figures a special notation will be used. A transition with a Ë Æ is equal to Ð µ Ô Ø Ò Ð Ð µ Ô Ø Ö Ø Ð Ð µ. As this is quite long and there is no ambiguity with using the shorter version, the short version will be used. Likewise a ÁÆ Ã is a packet with both the FIN and the ACK flag set. Also, to avoid misunderstanding Ë Æ will be used instead of Ô Ø to denote the acknowledgment field in the packet. There are a number of different variables that are used. TYPE is the type of role the machine takes. This is commonly determined by the implementation and can only be client or server. the ack, seq, win, and dl in the packet scope are respectively the acknowledgment field, the º¾º½ ÓÒÒ Ø ÓÒ Ø Ð Ñ ÒØ sequence number field, the window size field, and the data length of the packet. The data length is not a field in itself, however, it is calculated as the total length from the IP header subtracted the IP header length field from the IP header and data offset field from the TCP header of the current packet. Before data can be transferred between a client and a server, a connection must be established. In TCP this is done by synchronizing sequence numbers. The client sends a packet with the SYN flag set, and include the ISN in the packet. The server must reply to this packet with an ACK packet, acknowledging the ISN plus one, since a packet with the SYN flag set demands that the sequence number is incremented. This is shown in Figure 3-2 on the next page. However, until now the establishment have only been one way. A similar process is needed for the server, which is shown in Figure 3-3 on page 19. This implies that the server must send a packet to the client with the SYN flag set containing the servers ISN and the client to reply with an ACK. The established state is dashed because this is not the complete figure and more will follow. The normal course is for the client to send its ISN and the server to respond with a SYN/ACK packet, which is an acknowledgment of the clients ISN and its own ISN. The client then responds with an acknowledgment of the servers ISN. However, TCP also allows both peers to initiate a connection at the same time. This means that they simultaneously sends their ISN and acknowledges the other s ISN. These two scenarios would both result in the client considering its connection 17

30 arise. To compensate for any packet loss retransmissions are used. E.g. if the SYN packet sent by the client does not reach the server, the client will simply time out and retransmit its packet. If the packet lost was the ACK packet sent by the server, the result would be somewhat the same, the client would still time out and re-send its SYN packet and the server sends an ACK in response assuming that its previous ACK packet was lost. The IP layer can also cause what may seem as errors by e.g. delivering packets out of order. start SYN! synsent SYN? ack := SYN?.seq + max(syn?.dl,1) ACK! SYN/ACK? seq = SYN/ACK?.ack ack := SYN/ACK?.seq + max(syn/ack?.dl,1) ACK! synrcvd ACK? seq = ACK?.ack ACK?.dl<=win ack := ACK?.seq + ACK?.dl º¾º¾ Ø Ð ÓÒÒ Ø ÓÒ established Figure 3-2: Connection establishment for the client. In this section we will not denote the two peers of the connection as client and server, since they both transmit and receive data, making neither one of them different from the other. Therefore we instead call them sender and receiver. In this phase of the connection data flows from the sender to the receiver. To ensure that the data arrives at the other end of the connection, it can be ordered, and that data is not duplicated we use 18

31 synrcvd ACK? seq = ACK?.ack ACK?.dl <= win ack := ACK?.seq + ACK?.dl established Figure 3-3: Connection establishment for the server. the sequence number and acknowledgment number. The window size of the receiver is used to ensure that the sender does not send more that the receiver can receiver. If this is done anyway, the excess packets are not acknowledged. If a packet is missing, data following it will not be acknowledged until the missing packet has been received. Also acknowledgments themselves will not be acknowledged, only when new sequence number is presented, this will be acknowledged. If a packet is lost it implies that the current information about the state of the sender is also lost. The sender, whose packet is lost, will, if the packet contained data, retransmit it as a result of the receiver not acknowledging the data. The unacknowledged data will first be acknowledged when the receiver retransmits the lost acknowledgment, is ready to transmit something else, or if additional data are sent from the sender. Until then, the retransmissions will be continued and they will be dropped as they are received. Acknowledgments are not required to be sent right away when a packet is received, as TCP acknowledges the individual bytes continuously. If packets are transmitted from a client that has not yet acknowledged data, the ACK flag will be set and the acknowledgment number will contain the next sequence number to be received. In most implementations the peer, that has received data and not yet sent an acknowledgment, waits 200 ms before transmitting the acknowledgment. This is done in order to avoid transmitting additional data onto the network, since all the packets received within the 200 ms can be acknowledged in one packet and acknowledgments are piggy-backed onto other transmitted packets if possible. [Ste94] 19

32 are different from each other. The only exception is that the connection is defined as half-way closed after one has terminated the connection to the other and data can still be transferred in the direction that has not been closed. The connection termination is a two-way handshake. First, one of the sides, in our example the client, sends a packet with the FIN flag set. The server must reply to the FIN packet with an ACK packet containing the acknowledgment number incremented one from the sequence number, or if the received FIN packet contains data, the acknowledgment number must be increased by the value of the data length. Moreover, the server must consider if data has been lost between the last receive packet and the FIN packet received. As usual packets can vanish or be delayed and in that coincidence the packets must be retransmitted. The connection will be considered closed from the client to the server when the ACK packet is received by the client. The server closes its connection in the same manner. If the very last acknowledgment is lost, it will generate a retransmission of the FIN packet from the server to the client (as the server closes its connection last in our current example), however, the connection could already be thought of as closed. But in order to avoid that unnecessary packets are sent, the connection will be in a Closewait state. In theory it can be there forever, since no timeout period has been specified [Pos81]. In the coincidence of simultaneous closing, the only difference from the described is that both connections can not be sure that their last packet arrived at the other peer. Figure 3-4 on the next º Ì ÓÖ Ì È ÓÒÒ Ø ÓÒ page and Figure 3-5 on the facing page depicts two TCA respectively describing the closing phase of the connection for both the client and the server, the client being the peer which actively closes the connection and the server which passively closes the connection. Here we present a TCA describing the entire TCP connection. The TCA is a concatenation of the previous depicted TCA. The model is depicted in Figure 3-6 on page 22 and we call this the TCP peer model. There are a few remarks to the figure. Some transitions have been omitted in order to minimize the figure. Retransmissions are not modeled in the figure. Retransmissions Ã Ã Õ Ã Ð Ã ÊËÌ are basically a loop on the same state, however, since two transitions can lead to the same state to avoid wrong modeling a workaround can be made by splitting the state in two, this is exemplified in Figure 3-7 on page 23. But as this would make the figure very large this have been omitted. Also on every state succeeding the established state there are transitions with to closed and loops with Õ Ã ÊËÌ going Û Ò Ã Õ, Ã Õ Ã Ã Õ Ã Ð 20

33 ack := FIN/ACK?.seq + max(fin/ack?.dl, 1) ACK! ACK?.ack=seq ack := ACK?.seq + ACK?.dl finwait2 ACK? ACK?.ack < seq ack := ACK?.seq + ACK?.dl closing ACK? ACK?.ack < seq ack := ACK?.seq + ACK?.dl FIN/ACK? FIN/ACK?.ack = seq ack := FIN/ACK?.seq + max(fin/ack?.dl, 1) ACK! ACK? ACK?.dl=0 ACK?.ack=seq FIN/ACK? FIN/ACK?.ack = seq ack := FIN/ACK?.seq + max(fin/ack?.dl, 1) ACK! timewait timeout t = timeout Closed Figure 3-4: Active connection termination, shown as a TCA. established FIN/ACK? seq <= FINACK?.ack ack := FIN/ACK?.seq + max(fin/ack?.dl,1) FIN/ACK? seq <= FIN/ACK?.ack ack := FIN/ACK?.seq + max(finack?.dl,1) FIN/ACK! closewait ACK! ACK? ACK?.dl = 0 seq <= ACK?.ack FIN/ACK! lastack ACK? ACK?.dl = 0 seq <= ACK?.ack ACK? ACK?.dl = 0 seq = ACK?.ack closed Figure 3-5: Passive connection termination, shown as a TCA. 21

34 22 start RST! TYPE = Server Figure 3-6: This TCA shows how a TCP connection acts as specified. ACK? seq < ACK?.ack ACK/RST! ACK/RST? seq = ACK/RST?.ack closing ACK? ACK?.ack=seq ack := ACK?.seq + ACK?.dl finwait1 FIN/ACK! FIN/ACK? FIN/ACK?.ack = seq ack := FIN/ACK?.seq + max(fin/ack?.dl, 1) ACK! synsent established TYPE = Client SYN! SYN! SYN? ack := SYN?.seq + max(syn?.dl,1) ACK! listen SYN/ACK? seq = SYN/ACK?.ack ack = SYN/ACK?.seq + max(syn/ack?.dl,1) ACK! ACK? seq > ACK?.ack ack := ACK?.seq + ACK?.dl FIN/ACK? seq <= FIN/ACK?.ack ack := FIN/ACK?.seq + max(fin/ack?.dl,1) FIN/ACK! FIN/ACK? FIN/ACK?.ack < seq ack := FIN/ACK?.seq + max(fin/ack?.dl, 1) ACK! FIN/ACK? FIN/ACK?.ack = seq ack := FIN/ACK?.seq + max(fin/ack?.dl, 1) ACK! timewait ACK? ACK?.dl=0 ACK?.ack=seq t = timeout SYN? ack := SYN?.seq + max(syn?.dl,1) SYN/ACK! synrecv ACK? seq = ACK?.ack ACK?.dl <= win ack := ACK?.seq + ACK?.dl finwait2 FIN/ACK? seq <= FIN/ACK?.ack ack := FIN/ACK?.seq + max(fin/ack?.dl,1) closewait lastack ACK? ACK?.dl = 0 seq = ACK?.ack FIN/ACK! ACK? ACK?.dl = 0 seq <= ACK?.ack ACK? ACK?.dl = 0 seq <= ACK?.ack ACK/RST? TYPE = server seq = ACK/RST?.ack ACK? seq < ACK?.ack ACK/RST! closed

35 end end º ËÙÑÑ ÖÝ and à à Õ. This state. Figure 3-7: The retransmission workaround. is of course not on the closed state as this is an accepting In this chapter we have studied the TCP protocol with focus on the details that are important for stateful inspection. We have studied the connection phases and we have made a model for the behavior of each of the peers in a connection. This model represents a single peer and is the same for both peers. 23

36

37 In the previous chapter we looked at the behavior of TCP. From this behavior we derived a model of how a peer should react. In this chapter the behavior of TCP is examined from a stateful inspection firewall s point of view and a new model is derived. Afterwards we address other º½ Ì Å ÒÁÒÌ Å Ð problems concerning modeling stateful inspection. More specific, we focus on whether a firewall should be active or passive, and behavior of protocols that significantly differs from TCP. We will start by examining where the stateful inspection firewall is placed. Communication is between two points and it is somewhere between these points that a firewall exists. Where it is actually placed, be it closer to point A than point B or vice versa, does not matter since we only need to filter at some point between peers. Considering an arbitrary network, literally millions of different routes could exist for a single message to travel in order to reach its destination. Virtually every message could travel its own route. This problem is eliminated by forcing packets to travel through that point, e.g. by making it physically impossible to go another way. This is necessary because the intention of a firewall is to ensure that unwanted traffic is filtered away and therefore all traffic must travel through the firewall. In other words, to get traffic in or out of the protected network everything have to pass the firewall and thereby we have to ensured that a firewall is always in between peers. Considering communication on a network, a peer can only tell what it have received and send of messages. It is unknown to a single peer if messages send have actually been received by the º¾ ËØ Ø ÙÐÁÒ Ô Ø ÓÒÓÒÌ È other peer. Likewise a firewall only knows what messages it has seen, it does not know if the messages are actually received by a peer once it has seen them. To make matters even worse it is not guaranteed that a message sent before another will be received before the other. As shown, TCP can be described as a TCA, which any valid TCP connection will conform to. So, in order to examine if any TCP connection is valid, it is a question of changing the state of the connection according to every received packet. We will now look at how we can use this information in order to create a TCA for how a stateful inspection firewall should react to a TCP connection. As said previously, a connection viewed from a point in the network, be it the peers or in between them, it is only possible to say what has come through that point. The firewall, which is in the middle, does not know whether a packet it has seen is received by the receiver or if a response has 25

38 positives and positive negatives, in other words, the number of wrongly blocked packets and the number of accepted packets which does not belong to the connection. All this will be taken into consideration as we look at how to design a stateful inspection firewall for TCP. º¾º½À Ò The initializing handshake is a three-way handshake or two two-way handshakes. First there is no connection in which the state is equal to the TCP state Listening and Start depending on whether it is the server or client. Then a SYN packet is sent from the client to the server and the connection then changes state equal to the Synsent. We will always consider the client to be the one which first sent a SYN packet, even if it is a simultaneous connection establishment. The server then receives the packet and responds with an ACK to the SYN and sends a SYN itself, commonly done by a combined SYN/ACK packet. Again the connection changes state accordingly to what has been send. Similar in the case of simultaneous connection establishment both peers change state accordingly. Similar the handshakes used to close the connection changes the state of the connection. Again, as it is not specified who sends the first FIN packet, both scenarios have to be taken into account, and also simultaneous FINs. As with TCP there are transitions that allow state changes. By filtering out packets which are not a legal state change packet, it is possible to securely make rules for opening and closing connections. In both cases we check that the acknowledgments acknowledges the correct sequence number. This is done by storing the sequence number from the SYN packets and waiting for the corresponding acknowledgments. After synchronization this is done by checking that the data length plus sequence number is equal to the acknowledgment number. Retransmission should also be allowed through the firewall. To do this a looping state change is used to allow the last seen packet through the firewall. How retransmissions are handled in an established connection is a more difficult task and we need more information to handle this, therefore we will come back to this later. However, due to retransmissions it is not always easy to inspect a connection between peers. Consider the following example: Example 3 A client initiates a connection by sending a SYN. This is registered by the firewall. The server replies with SYN/ACK, which the firewall also registers and now waits for an ACK. However, the client times out before receiving the SYN/ACK and retransmits the SYN. There are two possible actions by the firewall when it sees the SYN, either it can change back to its previous state or remain in the same state. 26