Sunday, 20 February 2011

Using the Linux tbf qdisc for rate limiting on local or loopback interfaces

If you have ever played with the Linux tbf (Token Bucket Filter) on either some local interfaces, or on the loopback interface (lo) then you may have run into problems - like the attained rate is only a few hundred kilobits/s or less (zero)....?
tc qdisc add dev lo root tbf rate 10Mbit burst 10kb latency 5ms 

Basically if your interface has TSO/GSO enabled (check using ethtool -k ethX), or you're using the loopback interface - then you'll probably hit a problem. It turns out that the loopback interface has GSO/TSO enabled as default, plus since it is a software interface its default mtu is 16384 (as compared to 1500 for normal Ethernet interface). This matters as the tbf queue checks the size of the incoming 'packets' - which in the case of GSO/TSO are much larger than a normal on-the-wire packet - instead they're up to 9 x iface's mtu. So for normal interfaces it's about 12K, but for loopback it is about 100k!

In which case you'll need to add the 'mtu' argument to the tc command and then it all works.
tc q a dev eth0 root tbf rate 10Mbit burst 10kb latency 5ms mtu 100000