troubleshooting BSD TCP network performance

by z411 @ 2026-03-04 [ tech ]

Recently I've been trying the BSD family of operating systems and I've been pleasantly surprised. I am no UNIX evangelist, but what started as mere curiosity (due to ocassionally using Solaris and HP-UX at work) has turned into full blown love for the feeling of simplicity and smoothness that comes with learning and administering the BSDs.

That's why I've decided to move my servers to one (or more) of them. In my local network they work perfectly, but when I installed them abroad I noticed some severe disparity in their handling of high latency TCP and HTTP traffic:

iperf3 TCP Internet

I am especially concerned about NetBSD, so I decided to take a closer look, and hopefully, ask for help.

Overview

First, let's start with what I tried to test the network speeds:

iperf3: server->client, TCP and UDP, over the Internet (US->Chile) and over my local network
HTTP download: server->client, over the Internet (US->Chile) and over my local network

Tested operating systems:

Alpine Linux v3.23 (Linux 6.18)
FreeBSD 15.0-RELEASE
OpenBSD 7.8
NetBSD 10.1 and 11.0 RC1

Things to consider:

All remote servers are installed on the same Proxmox (KVM+virtio) node in the US
RTT latency to remote servers is ~209ms
Local servers run in Proxmox as well

LAN: iperf3

Locally, all tested OSes are very capable of saturating the local network link, both through TCP and UDP:

iperf3 TCP/UDP over Local network

We see no problems here.

LAN: HTTP

We can also observe flawless and quick HTTP downloads over the network:

HTTP over Local network HTTP over Local network (detail)

Internet/WAN: iperf3

Over the Internet we can also observe in the iperf3 UDP tests that all OSes can easily saturate the uplink of the remote node at around 600 Mbit/s. That said, we can start seeing problems when it comes to TCP:

iperf3 TCP/UDP over Internet

We can see the expected unstability that comes with a high-latency congested network, but they don't all seem to react equally to it. HTTP downloads show a clearer picture.

Internet/WAN: HTTP

HTTP over Internet HTTP over Internet (detail)

[ tcpdump : alpine freebsd openbsd netbsd ]

Stock, I can observe the following:

Alpine Linux seems to have the least trouble recovering from speed dips caused by network congestion.
FreeBSD occassionally reaches higher speeds than Linux but speed dips are much more severe when they happen.
OpenBSD seems stable enough but it seems to be capping at around 8-10 MiB/s. Dips are pronounced as well.
NetBSD has the most trouble here, displaying several speed dips and seemingly a hard cap at around 986 KiB/s.

I generally don't need high speeds, but the hard wall that NetBSD seems to be hitting is concerning. You can see the tcpdump output here.

Tuning attempts

Since the speed stability issues seem to stem from congestion, I tried some different congestion control algorithms on Linux and FreeBSD. I enabled BBR on Linux:

/etc/sysctl.conf (Linux)

net.core.default_qdisc=fq net.ipv4.tcp_congestion_control=bbr

And did the same on FreeBSD:

/boot/loader.conf (FreeBSD)

tcp_bbr_load="YES"

/etc/sysctl.conf (FreeBSD)

net.inet.tcp.functions_default=bbr

I also tried to increase the buffer sizes on NetBSD, as I heard they're tiny by default:

/etc/sysctl.conf (NetBSD)

net.inet.tcp.recvbuf_auto=1 net.inet.tcp.sendbuf_auto=1 net.bpf.jit=1 net.bpf.maxbufsize=4194304 net.inet.tcp.sendbuf_max=16777216 net.inet.tcp.recvbuf_max=16777216 net.inet.ip.ifq.maxlen=4096 net.inet.tcp.mssdflt=1072 kern.sbmax=16777216 net.inet.tcp.congctl.selected = cubic

While NetBSD didn't improve, the speed boost achieved by BBR on FreeBSD was pretty incredible:

HTTP downloads post tunables HTTP downloads post tunables (detail)

Linux stayed mostly the same.
FreeBSD was so fast it finished the download in 30 seconds. Congestion still causes dips but it also recovers much faster from them.
OpenBSD didn't have tunables I could find (I may be wrong) since window sizing seems to be done fully automatically, but I'm happy with the stock result.
NetBSD's tunables around send and recv buffers didn't help much with speed, they only increased packet loss instead.

Final thoughts and NetBSD

Overall I'm pretty happy with the result - congestion-triggered speed dips aren't too bad in practice, and for my usage (mostly HTTP web serving) OpenBSD's speeds are more than enough.

I'd really want to get a better result out of NetBSD, though. I'm really enjoying using it so far and it'd be a waste to replace it with something else if it's something that can be fixed. Browsing the mailing lists I saw someone getting 17 MB/s with the default settings so I'm not sure what's wrong in my case.

To rule out routing I tried installing it on different datacenters (Germany and Japan) and the results were similar.

So there you go - I'm ultimately looking forward to moving to the BSDs. But I'd love to have NetBSD as an option for me as well. I wish I had enough networking knowledge to pinpoint the issue. If you have any idea of what could be wrong, please contact me (preferably e-mail).

Tweet | Permalink | 18:30