Latency investigation of a bridging linux box

Latency investigation of a bridging linux box

This document was written by Eicke Friedrich. Last change 26/08/2003.

0. Content
0. Content
1. Introduction
2. The setup
2.1. System
2.2. Testing modes
2.3. How to read the plots
3. The results
3.1. UDP
3.2. TCP
4. Problems
A. Links

1. Introduction
This report describes an investigation of a linux box carried out at the University of Leipzig. Obtaining knowledge about the delay caused by a linux bridge under particular circumstances was the purpose of this investigation. The main focus layed on the increasment of delaytimes if a packet has to went through more modules. Very important was the influence of netfilter on a packet traveling time. The research was restricted to UDP and TCP traffic.

2. The setup
This chapter describes the setup for all tests done. First of all we generated the traffic and send it through a recording box. This system was equipped with two DAG Network Measurement cards (see [1] and [2] for more details) that are able to record very exact timestamps with an accuracy below 100 ns. A timestamp for each packet was recorded. Then the traffic went through the bridge and back into the recording box. A second timestamp was recorded. The journey of the traffic ended at the traffic receiver. As result we have two logfiles with the timestamps for all packets before entering and after leaving the bridge. Comparing the timestamps for a single packet gave us the delay caused by the bridge.

				|-------------------------------|
				|	  recording box		|
				|________		________|
				|| dag0	|		| dag1 ||
				|-------------------------------|
				  A    V		 A    V
				  |    |		 |    |
|-----------------|		  |    |		 |    |			       |-----------------|
|traffic generator|>--------------/    |		 |    \----------------------->|traffic receiver |
|-----------------|		       |		 |			       |-----------------|
				       V		 A
				|-------------------------------|
				||| eth0 |	       | eth1 |||
				||--------     br0     --------||
				||_____________________________||
				|				|
				|	  linux bridge		|
				|-------------------------------|

For all tests we used the following four boxes:
-traffic generator (PIII 500MHz)
-traffic reciever (PII 450MHz)
-recording box (PIII 900MHz)
-linux bridge (Dual Athlon MP 1900+)

2.1. System
The bridge was running linux kernel 2.4.21 from kernel.org (see [3]). Bridging was enabled and the netfilter patch for bridges was applied. QoS and HTB was also enabled in kernel config, two HTB patches were applied (see [4] for more details) resulting in HTB version 3.12. For firewalling we used netfilter ([5]) 1.2.8 and applied the following patches from patch-o-matic:

submitted/01_2.4.19
submitted/02_2.4.20
submitted/03_2.4.21
submitted/06_early_drop-backwards
submitted/07_unused_var
submitted/24_conntrack-nosysctl
submitted/25_natcore-nohelper
submitted/26_morethan32cpufix
submitted/27_include-nfc-order
submitted/42_conntrack-bigendian-fixes
submitted/44_nat-bigendian-fixes
submitted/45_ip-conntrack-outdated-comment
submitted/46_ip-conntrack-unneeded-check
submitted/47_ip-conntrack-unconfirmed-expect
submitted/48_unneeded-init
submitted/50_ip-conntrack-remove-expectations-list_for_each_safe
submitted/51_ipt_recent
pending/28_mirror-fixes
pending/29_netfilter-policy-routing
pending/30_irc-parser-fix
pending/39_ip_conntrack-proc
pending/43_admin-prohib
pending/49_ip-conntrack-expect-drop-refcnt
pending/50_ip-conntrack-expect-drop-refcnt-fix2
base/NETLINK
base/TTL
base/iprange
base/mport
base/quota
extra/CLASSIFY
extra/CONNMARK
extra/IPMARK
extra/string
extra/CLASSIFY

There was also a new netfilter match module called "ipp2p" installed. Identifying Peer-to-Peer traffic to be able to shape this traffic is the intention of ipp2p. You can download the current version at [6]. This module should be testet as well during thi s investigation.

2.2. Testing modes
In this section we describe the different modes of the bridge used for the investigation.

plain mode
running kernel modules: bridge
description: This mode was used to find out the basic delay caused by bridging. The results are the base for any comparison.
introducing:

brctl addbr br0
brctl stp br0 off

brctl addif br0 eth0
brctl addif br0 eth1

ifconfig br0 0.0.0.0 up

QoS mode
running kernel modules: bridge, sch_htb
description: Here we see the influence of HTB on the latency. Two qdiscs containing three classes each were created. Every packet gets sorted into the default class.
introducing:

tc qdisc add dev eth0 root handle 1:0 htb default 10
tc qdisc add dev eth1 root handle 2:0 htb default 10

tc class add dev eth0 parent 1:0 classid 1:1 htb rate 102400kbit ceil 102400kbit quantum 20000
tc class add dev eth0 parent 1:1 classid 1:10 htb rate 102380kbit ceil 102400kbit prio 3 quantum 20000
tc class add dev eth0 parent 1:1 classid 1:12 htb rate 20kbit ceil 25kbit prio 5 quantum 1500 cburst 1501b

tc class add dev eth1 parent 2:0 classid 2:2 htb rate 102400kbit ceil 102400kbit quantum 20000
tc class add dev eth1 parent 2:2 classid 2:10 htb rate 102380kbit ceil 102400kbit prio 3 quantum 20000
tc class add dev eth1 parent 2:2 classid 2:12 htb rate 20kbit ceil 25kbit prio 5 quantum 1500 cburst 1501b

ipp2p mode
running kernel modules: bridge, sch_htb, ip_tables, ip_conntrack, iptable_mangle, ipt_CONNMARK, ipt_mark, ipt_MARK, ipt_CLASSIFY, ip_p2p
description: Now firewalling comes into play. Packets could be marked and classified by their mark. No shaping was done because of traffic beeing random and not P2P.
introducing:

iptables -A PREROUTING -t mangle -p tcp -j CONNMARK --restore-mark
iptables -A PREROUTING -t mangle -p tcp -m mark ! --mark 0 -j ACCEPT
iptables -A PREROUTING -t mangle -p tcp -m ipp2p --ipp2p -j MARK --set-mark 22
iptables -A PREROUTING -t mangle -p tcp -m mark --mark 22 -j CONNMARK --save-mark

iptables -A POSTROUTING -t mangle -o eth0 -m mark --mark 22 -j CLASSIFY --set-class 1:12
iptables -A POSTROUTING -t mangle -o eth1 -m mark --mark 22 -j CLASSIFY --set-class 2:12

2.3. How to read the plots
Chapter 3 presents the results of this investigation. This section explains how to read the plots to relieve understanding the results. When analysing the logfiles we got two different values: 1. the delay for a single packet and 2. the current throughput rate. Suming up for all packets belonging to the same throughput rate gave us the distribution density. These three values are shown in 3D plots. Note that delay axis is focused on the main delay but there are a couple of packets delayed much longer but showing them in these plots too would give us a very small view of the main interest.
For the average plots all values for a throughput rate were sumed up and divided by their total number giving us the average delay for the current testing mode and throughput rate.

3. The results
We decided to do testing with two different packet sizes per protocol. All test have been done using a constant packet rate. First 64 byte payload was choosen for both protocols and to encounter differences we took a packet size equal to the MTU (1500) re sulting in 1472 byte payload for UDP and 1460 byte for TCP. For every available result the serialization time is already deducted that we are able to compare delays for different packet sizes.

3.1. UDP
The traffic generator mgen (see [7]) was used to create UDP packets. For 64 byte it was configured to create 1, 11, 21 and 31 MBit per second of traffic. As the generating box hits it's packet rate limit no further tests could be done. The steps for 1472 byte were 1, 10, 20, 30, 40, 50, 60, 70, 80 and 90 MBit per second. Only specifying the packet rates and sizes these throughput rates are not 100% exact line speed. In the beginning we present the results for the 64 byte tests.

Figure 3.01: delay for 64 byte UDP packets in plain mode First of all comes the plain mode. As you can see the minimum delay is somewhere around 0.027ms. Most packets were delayed 0.035 to 0.045ms. As the throughput increases the delay times increase a little bit as well.

Figure 3.02: delay for 64 byte UDP packets in QoS mode The next figure shows the results for QoS mode. There is a slight increase of the overall delay time. The performance for 1MBit/sec is worse than for 11MBit/sec as figure 3.04 also clarifies. The reason for this is a little unclear. The first assumption w as that it has something to do with the token bucket architecture. Martin Devera, linux kernel QoS/HTB maintainer, disagreed to this speculation and guessed some resonance between QoS/HTB delays and the NIC or the NIC's driver. More tests to come. Thanks Martin for the help.

Figure 3.03: delay for 64 byte UDP packets in ipp2p mode For the ipp2p mode test the packet delay times went up again. Also the minimum delay increased. Overall the plot looks very similar to the QoS mode one. Very interessting to observe is the delay for the 31MBit/sec stream with delay times slightly below th e others. We've got no explaination yet.

Figure 3.04: average delay times for 64 byte UDP packets Figure 3.04 shows the calculation for the average delay times of all 64 byte tests. You can see very clearly the latency influences that the different modes had on the passing packets. Also you can spot the already mentioned observations.

Figure 3.05: delay for 1472 byte UDP packets in plain mode The plain mode plot does not show any conspicuity. With increasing throughput the delay times increase as well. The minimum delay is around 0.1ms what is more then triple the 64 byte minimum values.

Figure 3.06: delay for 1472 byte UDP packets in QoS mode In QoS mode the delays increase again. Also for the 60MBit/sec class you can spot a sgnificant difference in the plot compared to all the others. As there is no obvious reason or explaination we have to redo this test.

Figure 3.07: delay for 1472 byte UDP packets in ipp2p mode Again a test without any conspicuity. The observation for the 60MBit/sec class in the QoS mode test could not be affirmed during this test. With rising throughput rates you can see a fairly strong increase in delay times.

Figure 3.08: average delay times for 1472 byte UDP packets If you compare these values with the 64 byte results you see an overall increase of delaytimes by factor 2 or even more. Remember that serialization time is already deducted. This does NOT cover with other tests done on a FreeBSD router where delay times keept fairly stable independent of packet sizes. More tests have to be done here to eliminate possible NIC resonances.

3.2. TCP
No tests done yet. If you know a good tcp traffic generator feel free to let me know e<dot>friedrich<at>uni<dot>de .

4. Problems
-different network interfaces were used (eth0=3com; eth1=RealTek)
-no TCP traffic generator found yet that allows a specified packet rate per second and disables the Nagle Algorithm

A. Links
[1] Endace Measurement Systems homepage
[2] The DAG project at the University of Waikato
[3] The Linux Kernel Archives
[4] HTB home
[5] netfilter and iptables homepage
[6] IPP2P Homepage
[7] MGEN traffic generator hompage