Download Network packet capture in Linux kernelspace

Introduction
Network stack
Packet ingress flow
Methods to capture packets
University of Sao Paulo - USP
Network packet capture in Linux kernelspace
An overview of the network stack in the Linux kernel
Beraldo Leal
[email protected]
http://www.ime.usp.br/~beraldo/
Institute of Mathematics and Statistics - IME
University of Sao Paulo - USP
25th October 2011
Beraldo Leal
25th October 2011
Network packet capture in Linux kernelspace
1 / 25
Introduction
Network stack
Packet ingress flow
Methods to capture packets
University of Sao Paulo - USP
Outline
Introduction
Network stack
Packet ingress flow
Methods to capture packets
Beraldo Leal
25th October 2011
Network packet capture in Linux kernelspace
2 / 25
Introduction
Network stack
Packet ingress flow
Methods to capture packets
University of Sao Paulo - USP
Introduction
• Sniffers;
• Improvements in packet reception;
• Linux kernel network subsystem;
Beraldo Leal
25th October 2011
Network packet capture in Linux kernelspace
3 / 25
Introduction
Network stack
Packet ingress flow
Methods to capture packets
University of Sao Paulo - USP
Sniffers
• tcpdump, wireshark, snort, etc;
• Using the well-known library libpcap;
• Not suitable for > 10 Gbps;
• Packet loss;
Beraldo Leal
25th October 2011
Network packet capture in Linux kernelspace
4 / 25
Introduction
Network stack
Packet ingress flow
Methods to capture packets
University of Sao Paulo - USP
Improvements in packet reception
• Commodity hardware for packet capture;
• 3COM
• Intel
• endace, ...
• Many Interruptions
• NEW API or NAPI (interruption coalescence)
• zero-copy
• Direct Memory Access - DMA
• mmap()
Beraldo Leal
25th October 2011
Network packet capture in Linux kernelspace
5 / 25
Introduction
Network stack
Packet ingress flow
Methods to capture packets
University of Sao Paulo - USP
Linux kernel network subsystem
• Kernel number of files: 36.680
1 2
• net/ number of files: 1.293 ( 3.5% )
• drivers/net/ number of files: 1.935 ( 5.27% )
• Kernel SLOC: 9.723.525
• net/ SLOC: 480.928 ( 5% )
• drivers/net/ SLOC: 1.155.317 ( 12% )
1
2
kernel 3.0.0
source: wc, find, cat, etc..
Beraldo Leal
25th October 2011
Network packet capture in Linux kernelspace
6 / 25
Introduction
Network stack
Packet ingress flow
Methods to capture packets
University of Sao Paulo - USP
Network stack
L5: Application
http, ftp, ssh, telnet, ... (message)
L4: Transport
tcp, udp, ... (segment)
L3: Network
ipv4, ipv6, ... (datagram/packet)
L1/2: Link / host-to-network
ethernet, token ring, ... (frame)
Beraldo Leal
25th October 2011
Network packet capture in Linux kernelspace
8 / 25
Introduction
Network stack
Packet ingress flow
Methods to capture packets
University of Sao Paulo - USP
Important data structs:
• net device
• include/linux/netdevice.h
• sk buff
• include/linux/skbuff.h
Beraldo Leal
25th October 2011
Network packet capture in Linux kernelspace
9 / 25
Introduction
Network stack
Packet ingress flow
Methods to capture packets
University of Sao Paulo - USP
Important data structs:
• net device (include/linux/netdevice.h)
• unsigned int mtu
• unsigned int flags
• unsigned char dev addr[MAX ADDR LEN]
• int promiscuity
Beraldo Leal
25th October 2011
Network packet capture in Linux kernelspace
10 / 25
Introduction
Network stack
Packet ingress flow
Methods to capture packets
University of Sao Paulo - USP
Important data structs:
• sk buff (include/linux/skbuff.h)
• struct sk buff *next;
• struct sk buff *prev;
• ktime t tstamp;
• struct net device *dev;
• unsigned int len;
• unsigned int data len;
• u16 mac len;
• u8 pkt type;
• be16 protocol;
• sk buff data t transport header; (old h)
• sk buff data t network header; (old nh)
• sk buff data t mac header; (old mac)
Beraldo Leal
25th October 2011
Network packet capture in Linux kernelspace
11 / 25
Introduction
Network stack
Packet ingress flow
Methods to capture packets
University of Sao Paulo - USP
Important sk buff routines
• alloc skb();
• dev alloc skb();
• kfree skb();
• dev kfree skb();
• skb clone();
• skb network header(skb);
• skb transport header(skb);
• skb mac header(skb);
Beraldo Leal
25th October 2011
Network packet capture in Linux kernelspace
12 / 25
Introduction
Network stack
Packet ingress flow
Methods to capture packets
University of Sao Paulo - USP
Packet ingress flow
• When working in interrupt driven model, the nic registers an
•
•
•
•
•
•
•
•
interrupt handler;
This interrupt handler will be called when a frame is received;
Typically in the handler, we allocate sk buff by calling
dev alloc skb();
Copies data from nic’s buffer to this struct just created;
nic call generic reception routine netif rx();
netif rx() put frame in per cpu queue;
if queue is full, drop!
net rx action() decision based on skb->protocol;
This function basically dequeues the frame and delivery a copy
for every protocol handler;
• ptype all and ptype base queues
Beraldo Leal
25th October 2011
Network packet capture in Linux kernelspace
13 / 25
Introduction
Network stack
Packet ingress flow
Methods to capture packets
University of Sao Paulo - USP
Packet ingress flow
• ip v4 rcv() will receive the ip datagram (if is a ipv4 packet);
• ip checksum, check ip headers, ....
• ip rcv finish() makes route decision (ip forward() or
ip local delivery())
• ip local delivery() defrag fragmented packets, and call
ip local deliver finish()
• ip local deliver finish() find protocol handler again;
• tcp v4 rcv(), udp rcv(), or other L4 protocol handler
• ...
Beraldo Leal
25th October 2011
Network packet capture in Linux kernelspace
14 / 25
ip_local_deliver_finish()
(net/ipv4/ip_input.c)
find protocol handler or
send icmp_dst_unreach
NF_IP_LOCAL_IN
NF_IP_PRE_ROUTING
<continue>
NF_IP_FORWARD
Layer 3
Network
ip_local_deliver()
(net/ipv4/ip_input.c)
defrag fragmented
packets
ip_rcv_finish()
(net/ipv4/ip_input.c)
find route and handle
IP options
ip_forward()
(net/ipv4/ip_forward.c)
handle route alert;
send redirect if
necessary;
decrease TTL;
verify if frag is
possible (mtu)
ip_error()
(net/ipv4/route.c)
routing error, send
icmp pkt
ip_rcv()
packet_rcv()
arp_rcv()
(net/ipv4/ip_input.c)
<tcpdump_process>
(handle arp requests
verify skb, IP headers
<dhcpd process>
and replies)
and IP checksum
<...>
netif_rx()
(net/core/dev.c)
input_queue
[cpu]
Network Drivers
(drivers/net/*)
<...>
net_rx_action()
(net/core/dev.c)
decision based on
skb->protocol field
Layer 1/2
Physical/Link
Application
userspace
kernelspace
Socket Layer
(net/core/sock.c)
__tcp_v4_lookup()
(net/ipv4/tcp_ipv4.c)
check for socket in
LISTEN, with dst_port
tcp_v4_do_rcv()
(net/ipv4/tcp_ipv4.c)
check for socket state
tcp_v4_rcv()
(net/ipv4/tcp_ipv4.c)
check for tcp headers
udp_rcv()
(net/ipv4/udp.c)
check for udp headers
ip_local_deliver_finish()
(net/ipv4/ip_input.c)
find protocol handler or
send icmp_dst_unreach
NF_IP_LOCAL_IN
NF_IP_PRE_ROUTING
generate ICMP
error
<...>
Layer 4
Transport
<continue>
NF_IP_FORWARD
Layer 3
Network
ip_local_deliver()
(net/ipv4/ip_input.c)
defrag fragmented
packets
ip_rcv_finish()
(net/ipv4/ip_input.c)
find route and handle
IP options
ip_forward()
(net/ipv4/ip_forward.c)
handle route alert;
send redirect if
necessary;
decrease TTL;
verify if frag is
possible (mtu)
ip_error()
(net/ipv4/route.c)
routing error, send
icmp pkt
Introduction
Network stack
Packet ingress flow
Methods to capture packets
University of Sao Paulo - USP
Methods to capture packets
• protocol handler
• register a function to handler packets with dev add pack()
• netfilter hooks
• userspace tools;
• socket AF PACKET, libpcap, ...
Beraldo Leal
25th October 2011
Network packet capture in Linux kernelspace
17 / 25
Introduction
Network stack
Packet ingress flow
Methods to capture packets
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
University of Sao Paulo - USP
struct packet type my proto;
int my packet rcv(struct sk buff ∗skb, struct net device ∗dev,
struct packet type ∗pt, struct net device ∗orig dev) {
printk(KERN ERR ”+ 1!\n”);
kfree skb(skb);
return 0;
}
static int hello init(void) {
printk(”<1> Hello world!\n”);
my proto.type = htons(ETH P ALL);
my proto.dev = NULL;
my proto.func = my packet rcv;
dev add pack(&my proto);
return 0;
}
static void hello exit(void) {
dev remove pack(&my proto);
printk(”<1> Bye, cruel world\n”);
}
module init(hello init);
module exit(hello exit);
Beraldo Leal
25th October 2011
Network packet capture in Linux kernelspace
18 / 25
Introduction
Network stack
Packet ingress flow
Methods to capture packets
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
University of Sao Paulo - USP
int my packet rcv(struct sk buff ∗skb, struct net device ∗dev, struct packet type ∗pt, struct net device
∗orig dev)
{
switch (skb−>pkt type) {
case PACKET HOST:
printk(”PACKET HOST − ”);
break;
case PACKET BROADCAST:
printk(”PACKET BROADCAST − ”);
break;
case PACKET MULTICAST:
printk(”PACKET MULTICAST − ”);
break;
case PACKET OTHERHOST:
printk(”PACKET OTHERHOST − ”);
break;
case PACKET OUTGOING:
printk(”PACKET OUTGOING − ”);
break;
case PACKET LOOPBACK:
printk(”PACKET LOOPBACK − ”);
break;
case PACKET FASTROUTE:
printk(”PACKET FASTROUTE − ”);
break;
}
printk(”%s 0x%.4X 0x%.4X \n”, skb−>dev−>name, ntohs(skb−>protocol), ip hdr(skb)−>protocol)
kfree skb(skb);
return 0;
}
Beraldo Leal
25th October 2011
Network packet capture in Linux kernelspace
19 / 25
Introduction
Network stack
Packet ingress flow
Methods to capture packets
University of Sao Paulo - USP
Netfilter hooks
• iptables = userspace;
• netfilter = kernelspace;
• Netfilter is merely a series of hooks in various points in a
protocol stack;
• packet filtering, network address [and port] translation
(NA[P]T) and other packet mangling;
• www.netfilter.org
Beraldo Leal
25th October 2011
Network packet capture in Linux kernelspace
20 / 25
Introduction
Network stack
Packet ingress flow
Methods to capture packets
University of Sao Paulo - USP
References
• br.kernelnewbies.org/node/150 has many links
Beraldo Leal
25th October 2011
Network packet capture in Linux kernelspace
23 / 25
Introduction
Network stack
Packet ingress flow
Methods to capture packets
University of Sao Paulo - USP
Thankyou! Question?
Beraldo Leal
25th October 2011
Network packet capture in Linux kernelspace
24 / 25
Introduction
Network stack
Packet ingress flow
Methods to capture packets
University of Sao Paulo - USP
Network packet capture in Linux kernelspace
An overview of the network stack in the Linux kernel
Beraldo Leal
[email protected]
http://www.ime.usp.br/~beraldo/
Institute of Mathematics and Statistics - IME
University of Sao Paulo - USP
25th October 2011
Beraldo Leal
25th October 2011
Network packet capture in Linux kernelspace
25 / 25