Network packet capture in Linux kernelspace

Network packet capture in Linux kernelspace An overview of the network stack in the Linux kernel Beraldo Leal beraldo@ime.usp.br http://www.ime.usp.br/~beraldo/ Institute of Mathematics and Statistics - IME 25th October 2011 Beraldo Leal 25th October 2011 Network packet capture in Linux kernelspace 1 / 25

Outline Introduction Introduction Beraldo Leal 25th October 2011 Network packet capture in Linux kernelspace 2 / 25

Introduction Sniffers; Improvements in packet reception; Linux kernel network subsystem; Beraldo Leal 25th October 2011 Network packet capture in Linux kernelspace 3 / 25

Sniffers Introduction tcpdump, wireshark, snort, etc; Using the well-known library libpcap; Not suitable for > 10 Gbps; Packet loss; Beraldo Leal 25th October 2011 Network packet capture in Linux kernelspace 4 / 25

Improvements in packet reception Commodity hardware for packet capture; 3COM Intel endace,... Many Interruptions NEW API or NAPI (interruption coalescence) zero-copy Direct Memory Access - DMA mmap() Beraldo Leal 25th October 2011 Network packet capture in Linux kernelspace 5 / 25

Linux kernel network subsystem Kernel number of files: 36.680 1 2 net/ number of files: 1.293 ( 3.5% ) drivers/net/ number of files: 1.935 ( 5.27% ) Kernel SLOC: 9.723.525 net/ SLOC: 480.928 ( 5% ) drivers/net/ SLOC: 1.155.317 ( 12% ) 1 kernel 3.0.0 2 source: wc, find, cat, etc.. Beraldo Leal 25th October 2011 Network packet capture in Linux kernelspace 6 / 25

Introduction L5: Application http, ftp, ssh, telnet,... (message) L4: Transport tcp, udp,... (segment) L3: Network ipv4, ipv6,... (datagram/packet) L1/2: Link / host-to-network ethernet, token ring,... (frame) Beraldo Leal 25th October 2011 Network packet capture in Linux kernelspace 8 / 25

Important data structs: net device include/linux/netdevice.h sk buff include/linux/skbuff.h Beraldo Leal 25th October 2011 Network packet capture in Linux kernelspace 9 / 25

Important data structs: net device (include/linux/netdevice.h) unsigned int mtu unsigned int flags unsigned char dev addr[max ADDR LEN] int promiscuity Beraldo Leal 25th October 2011 Network packet capture in Linux kernelspace 10 / 25

Important data structs: sk buff (include/linux/skbuff.h) struct sk buff *next; struct sk buff *prev; ktime t tstamp; struct net device *dev; unsigned int len; unsigned int data len; u16 mac len; u8 pkt type; be16 protocol; sk buff data t transport header; (old h) sk buff data t network header; (old nh) sk buff data t mac header; (old mac) Beraldo Leal 25th October 2011 Network packet capture in Linux kernelspace 11 / 25

Important sk buff routines alloc skb(); dev alloc skb(); kfree skb(); dev kfree skb(); skb clone(); skb network header(skb); skb transport header(skb); skb mac header(skb); Beraldo Leal 25th October 2011 Network packet capture in Linux kernelspace 12 / 25

Introduction When working in interrupt driven model, the nic registers an interrupt handler; This interrupt handler will be called when a frame is received; Typically in the handler, we allocate sk buff by calling dev alloc skb(); Copies data from nic s buffer to this struct just created; nic call generic reception routine netif rx(); netif rx() put frame in per cpu queue; if queue is full, drop! net rx action() decision based on skb->protocol; This function basically dequeues the frame and delivery a copy for every protocol handler; ptype all and ptype base queues Beraldo Leal 25th October 2011 Network packet capture in Linux kernelspace 13 / 25

Introduction ip v4 rcv() will receive the ip datagram (if is a ipv4 packet); ip checksum, check ip headers,... ip rcv finish() makes route decision (ip forward() or ip local delivery()) ip local delivery() defrag fragmented packets, and call ip local deliver finish() ip local deliver finish() find protocol handler again; tcp v4 rcv(), udp rcv(), or other L4 protocol handler... Beraldo Leal 25th October 2011 Network packet capture in Linux kernelspace 14 / 25

ip_local_deliver_finish() (net/ipv4/ip_input.c) find protocol handler or send icmp_dst_unreach NF_IP_LOCAL_IN NF_IP_PRE_ROUTING ip_rcv() (net/ipv4/ip_input.c) verify skb, IP headers and IP checksum NF_IP_FORWARD Layer 3 Network ip_local_deliver() (net/ipv4/ip_input.c) defrag fragmented packets ip_rcv_finish() (net/ipv4/ip_input.c) find route and handle IP options arp_rcv() (handle arp requests and replies) ip_forward() (net/ipv4/ip_forward.c) handle route alert; send redirect if necessary; decrease TTL; verify if frag is possible (mtu) packet_rcv() <tcpdump_process> <dhcpd process> <...> <continue> ip_error() (net/ipv4/route.c) routing error, send icmp pkt <...> netif_rx() (net/core/dev.c) input_queue [cpu] Network Drivers (drivers/net/*) net_rx_action() (net/core/dev.c) decision based on skb->protocol field Layer 1/2 Physical/Link

Application Socket Layer (net/core/sock.c) userspace kernelspace tcp_v4_do_rcv() (net/ipv4/tcp_ipv4.c) check for socket state tcp_v4_lookup() (net/ipv4/tcp_ipv4.c) check for socket in LISTEN, with dst_port generate ICMP error tcp_v4_rcv() (net/ipv4/tcp_ipv4.c) check for tcp headers udp_rcv() (net/ipv4/udp.c) check for udp headers <...> Layer 4 Transport ip_local_deliver_finish() (net/ipv4/ip_input.c) find protocol handler or send icmp_dst_unreach NF_IP_LOCAL_IN NF_IP_PRE_ROUTING NF_IP_FORWARD Layer 3 Network ip_local_deliver() (net/ipv4/ip_input.c) defrag fragmented packets ip_rcv_finish() (net/ipv4/ip_input.c) find route and handle IP options <continue> ip_forward() (net/ipv4/ip_forward.c) handle route alert; send redirect if necessary; decrease TTL; verify if frag is possible (mtu) ip_error() (net/ipv4/route.c) routing error, send icmp pkt

protocol handler register a function to handler packets with dev add pack() netfilter hooks userspace tools; socket AF PACKET, libpcap,... Beraldo Leal 25th October 2011 Network packet capture in Linux kernelspace 17 / 25

1 struct packet type my proto; 2 3 int my packet rcv(struct sk buff skb, struct net device dev, 4 struct packet type pt, struct net device orig dev) { 5 6 printk(kern ERR + 1!\n ); 7 8 kfree skb(skb); 9 return 0; 10 } 11 12 static int hello init(void) { 13 printk( <1> Hello world!\n ); 14 15 my proto.type = htons(eth P ALL); 16 my proto.dev = NULL; 17 my proto.func = my packet rcv; 18 19 dev add pack(&my proto); 20 return 0; 21 } 22 23 static void hello exit(void) { 24 dev remove pack(&my proto); 25 printk( <1> Bye, cruel world\n ); 26 } 27 module init(hello init); 28 module exit(hello exit); Beraldo Leal 25th October 2011 Network packet capture in Linux kernelspace 18 / 25

1 int my packet rcv(struct sk buff skb, struct net device dev, struct packet type pt, struct net device orig dev) 2 { 3 switch (skb >pkt type) { 4 case PACKET HOST: 5 printk( PACKET HOST ); 6 break; 7 case PACKET BROADCAST: 8 printk( PACKET BROADCAST ); 9 break; 10 case PACKET MULTICAST: 11 printk( PACKET MULTICAST ); 12 break; 13 case PACKET OTHERHOST: 14 printk( PACKET OTHERHOST ); 15 break; 16 case PACKET OUTGOING: 17 printk( PACKET OUTGOING ); 18 break; 19 case PACKET LOOPBACK: 20 printk( PACKET LOOPBACK ); 21 break; 22 case PACKET FASTROUTE: 23 printk( PACKET FASTROUTE ); 24 break; 25 } 26 printk( %s 0x%.4X 0x%.4X \n, skb >dev >name, ntohs(skb >protocol), ip hdr(skb) >protocol) 27 28 kfree skb(skb); 29 return 0; 30 } Beraldo Leal 25th October 2011 Network packet capture in Linux kernelspace 19 / 25

Netfilter hooks Introduction iptables = userspace; netfilter = kernelspace; Netfilter is merely a series of hooks in various points in a protocol stack; packet filtering, network address [and port] translation (NA[P]T) and other packet mangling; www.netfilter.org Beraldo Leal 25th October 2011 Network packet capture in Linux kernelspace 20 / 25

References Introduction br.kernelnewbies.org/node/150 has many links Beraldo Leal 25th October 2011 Network packet capture in Linux kernelspace 23 / 25

Thankyou! Question? Beraldo Leal 25th October 2011 Network packet capture in Linux kernelspace 24 / 25

Network packet capture in Linux kernelspace An overview of the network stack in the Linux kernel Beraldo Leal beraldo@ime.usp.br http://www.ime.usp.br/~beraldo/ Institute of Mathematics and Statistics - IME 25th October 2011 Beraldo Leal 25th October 2011 Network packet capture in Linux kernelspace 25 / 25