Symptoms

Since early august, we have encountered sporadic local network performance issues on Exoscale Open Cloud instances. These issues could happen on instances doing large data transfers with other instances on Exoscale Open Cloud. Typical throughput would drop to a ridiculous 50kbps data transfer rate for data exceeding several MB.

The problem

While offloading technologies are great, they are not always mature. We have discovered that our stack does not respect rightfully the TCP segmentation offloading parameters.

What is TCP segmentation offloading?

TCP segmentation offloading or TSO is a network interface driver feature used to instruct the OS Kernel not to take care of the TCP packet fragmentation and let the network interface do the work. Fragmentation is seen on a network each time there is a need to transmit data larger than the maximum allowed packet size (MTU).

Corrective images

We have deployed new OS template images for all our supported OS that are correctly configured. On a longer term we are working in order to re-enable this feature in our insfrastructure stack with 100% support and maximum performance. In the meantime, users we need performance for large data transfers between Exoscale VM should either deploy instances via new images or apply the above recommendations.

The immediate fix for existing VMs

Ubuntu

In order to apply without rebooting the parameters issue the following commands as root:

ethtool -K eth0 tso off
ethtool -K eth0 gso off

These settings will be applied immediately. To make those persistent through reboots, create a file named ethtool-tso in the /etc/network/if-pre-up.d folder. To do so, issue the following commands as root:

echo '#!/bin/sh' >> /etc/network/if-pre-up.d/ethtool-tso
echo "ETHTOOL=/sbin/ethtool" >> /etc/network/if-pre-up.d/ethtool-tso
echo '$ETHTOOL -K eth0 tso off gso off' >> /etc/network/if-pre-up.d/ethtool-tso
chmod 755 /etc/network/if-pre-up.d/ethtool-tso

CentOS

In order to apply without rebooting the parameters issue the following commands:

 ethtool -K eth0 tso off
 ethtool -K eth0 gso off

Just like for Ubuntu, these settings will be applied immediately. To make these persistent through reboots, create a file called /sbin/ifup-local with the following content: (use vi or another editor as root)

 #!/bin/bash
 case "$1" in
 eth0)
 /sbin/ethtool -K $1 tso off gso off
 ;;
 esac
 exit 0

Then issue the last 2 commands to make it executable and run at each boot:

chmod +x /sbin/ifup-local
chcon --reference /sbin/ifup /sbin/ifup-local

Windows (all versions)

To apply permanently the settings in all Windows versions, launch a Command Terminal as administrator.

Issue then the following command:

netsh int tcp set global chimney=disabled

And reboot your instance.