TCP Tuning parameters for different OS:es

NOTE! The information on this page is for informational purposes only. SUNET is not responsible for problems occuring when using this information.

The operating systems of today has good support for auto configuration of TCP. Good performance can be acheived just by changing the TCP window size. All operating systems we tested had much too low TCP window size as default to utilize a gigabit etehernet card, even on fairly short distances. We have also found out that only very late releases of the operating systems have good TCP stacks and drivers for the network cards, so upgrading to a fairly rescent OS is vital.

Below is a short description of what we did on the different operating systems we tested. In the table are some values shown with dark grey background - those values are usually the default, and thus not nesseceary to change.

We have used the following conventions:

[wmax] - maximum size (in bytes) of TCP window.
[wstd] - standard size (in bytes) of TCP window.
[wmin] - minimum size (in bytes) of TCP window.
[sbmax] - maximum allowed size of the system socket buffer

The values of those parameters varies according to transmission speed and round-trip-time (RTT). Don't blindly set theese values too large, as they consume kernel memory, and too large values may cause degradation in performance.

The [sbmax] parameter should be set to a size larger than the maximum TCP window size.

The theoretical value (in bytes) for [wstd] is bps / 8 * rtt, where bps is bandwidth in bits/second and rtt=round-trip-time. You can use this Perl program that will determine the RTT and do the calculations for You..

NetBSD

Only the NetBSD 2.0 (and later) release support really fast connections, as there has been much work done on the "zero-copy" TCP features and removal of TCP stack linear searches.

Install NetBSD 2.0
Enter the line "options NMBCLUSTERS=16384" in your kernel config file
Recompile and install your new kernel.
Do not try to set the values below too high until you have increased NMBCLUSTERS - risk of "hanging" the network part of the OS!

Command	Description
sysctl -w net.inet.tcp.rfc1323=1	Activate window scaling and timestamp options according to RFC 1323.
sysctl -w kern.somaxkva=[sbmax]	Set maximum size for all socket buffers together in the system
sysctl -w kern.sbmax=[sbmax]	Set maximum size of socket buffer for one TCP flow
sysctl -w net.inet.tcp.recvspace=[wstd]	Set default size of TCP receive window.
sysctl -w net.inet.tcp.sendspace=[wstd]	Set default size of TCP transmit window.
sysctl kern.mbuf.nmbclusters	View maximum number of mbuf clusters. Used for storage of data packets to/from the network interface. Can only be set by recompiling Your kernel - see above!

You can enter the values to /etc/sysctl.conf to have them set at boot-time, as in this example.

FreeBSD

We haven't been able to get as good results with FreeBSD as with NetBSD, which is mostly due to linear searching in buffer chains in the transmit/receive path. FreeBSD 5.2 though gives significantly better performance that the FreeBSD 4 releases.

Install FreeBSD 4.10-RELEASE or FreeBSD 5.2.1-RELEASE
Apply this "netlion" patch.
Enter the following line to /boot/loader.conf: kern.ipc.nmbclusters="16384" # Set the number of mbuf clusters
Reboot your system. Do not try to set the values below too high until you have increased nmbclusters - risk of "hanging" the network part of the OS!

Command	Description
sysctl net.inet.tcp.rfc1323=1	Activate window scaling and timestamp options according to RFC 1323.
sysctl ipc.maxsockbuf=[sbmax]	Set maximum size of TCP window.
sysctl net.inet.tcp.recvspace=[wstd]	Set default size of TCP receive window.
sysctl net.inet.tcp.sendspace=[wstd]	Set default size of TCP transmit window.
sysctl kern.ipc.nmbclusters	View maximum number of mbuf clusters. Used for storage of data packets to/from the network interface. Can only be set att boot time - see above.
sysctl net.inet.tcp.liondmask=7	Used to enable the "netlion" patch above.

You can enter the values to /etc/sysctl.conf to have them set at boot-time, as in this example.

Linux

The table below shows which parameters that may need to be changed. These are true for both 2.4 and 2.6 kernels. With theese changes, it is possible to get results in the same order as our NetBSD tests, with the exception that Linux will do a data copy in the transmit path so the transmitting machine will be more loaded.

Command	Description
echo "1" > /proc/sys/net/ipv4/tcp_window_scaling	Activate window scaling according to RFC 1323
echo "1" > /proc/sys/net/ipv4/tcp_timestamps	Activate timestamps according to RFC 1323
echo [wmax] > /proc/sys/net/core/rmem_max	Set maximum size of TCP receive window.
echo [wmax] > /proc/sys/net/core/wmem_max	Set maximum size of TCP transmit window.
echo [wmax] > /proc/sys/net/core/rmem_default	Set default size of TCP receive window.
echo [wmax] > /proc/sys/net/core/wmem_default	Set default size of TCP transmit window.
echo "[wmin] [wstd] [wmax]" > /proc/sys/net/ipv4/tcp_rmem	Set min, default, max receive window. Used by the autotuning function.
echo "[wmin] [wstd] [wmax]" > /proc/sys/net/ipv4/tcp_wmem	Set min, default, max transmit window. Used by the autotuning function.
echo "bmin bdef bmax" > /proc/sys/net/ipv4/tcp_mem	Set maximum total TCP buffer-space allocatable. Used by the autotuning function.
ifconfig eth? txqueuelen 1000	Define length of transmit queue. Replace "?" with actual interface number.

Windows 2000/XP/2003

Tests performed with Windows 2003 give results similar to the NetBSD results. To achieve this we had to change some variables in the registry.

Registry entry	Description
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\ Services\Tcpip\Parameters\Tcp1323Opts=1	Turn on window scaling option
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet \Services\Tcpip\Parameters\TcpWindowSize=[wmax]	Set maximum size of TCP window