Sauf qu'avant, apparemment, le kernel ne se comportait pas TSO-style pour le découpage et l'envoi des packets. Là, pour les drivers qui le supporte, tu émules un TSO logiciel, ce qui ne te fait rien gagner en terme de consommation CPU (il indique dans son commit 30% d'usage avec ou sans), mais permet de gagner en débit, car comme pour le TSO, tu envoies plus de packets d'un coup.

C'est comme ça que je le comprends à la lecture des commits et de la LKML, n'hésitez pas à me corriger...

EDIT:

Add SG and software TSO support for FEC. This feature allows to improve outbound throughput performance. Tested on imx6dl sabresd board, running iperf tcp tests shows: * 82% improvement comparing with NO SG & TSO patch

$ ethtool -K eth0 sg on $ ethtool -K eth0 tso on [ 3] local 10.192.242.108 port 35388 connected with 10.192.242.167 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0- 3.0 sec 181 MBytes 506 Mbits/sec * cpu loading is 30%

$ ethtool -K eth0 sg off $ ethtool -K eth0 tso off [ 3] local 10.192.242.108 port 52618 connected with 10.192.242.167 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0- 3.0 sec 99.5 MBytes 278 Mbits/sec

FEC HW support IP header and TCP/UDP hw checksum, support multi buffer descriptor transfer one frame, but don't support HW TSO. And imx6q/dl SOC FEC Gbps speed has HW bus Bandwidth limitation (400Mbps ~ 700Mbps), imx6sx SOC FEC Gbps speed has no HW bandwidth limitation.

The patch set just enable TSO feature, which is done following the mv643xx_eth driver.

Test result analyze: imx6dl sabresd board: there have 82% improvement, since imx6dl FEC HW has bandwidth limitation, the performance with SW TSO is a milestone.

Addition test: imx6sx sdb board: upstream still don't support imx6sx due to some patches being upstream... they use same FEC IP. Use the SW TSO patches test imx6sx sdb board in internal kernel tree: No SW TSO patch: tx bandwidth 840Mbps, cpu loading is 100%. SW TSO patch: tx bandwidth 942Mbps, cpu loading is 65%. It means the patch set have great improvement for imx6sx FEC performance.

https://gitorious.org/linux-can/linux-can-next/commit/fba0e1a3cfcc1d61e593f97650e18931a2aa1fc8

A final ça réduit quand même bien les cycles bouffés

Non, avant il respectait la MTU pour séparer les paquets, ce qui génère pas mal d'overhead sur des gros fichiers: http://www.linuxjournal.com/content/queueing-linux-network-stack

Huge Packets from the Stack

Most NICs have a fixed maximum transmission unit (MTU), which is the biggest frame that can be transmitted by the physical media. For Ethernet, the default MTU is 1,500 bytes, but some Ethernet networks support Jumbo Frames (https://en.wikipedia.org/wiki/Jumbo_frame) of up to 9,000 bytes. Inside the IP network stack, the MTU can manifest as a limit on the size of the packets that are sent to the device for transmission. For example, if an application writes 2,000 bytes to a TCP socket, the IP stack needs to create two IP packets to keep the packet size less than or equal to a 1,500 MTU. For large data transfers, the comparably small MTU causes a large number of small packets to be created and transferred through the driver queue.

In order to avoid the overhead associated with a large number of packets on the transmit path, the Linux kernel implements several optimizations: TCP segmentation offload (TSO), UDP fragmentation offload (UFO) and generic segmentation offload (GSO). All of these optimizations allow the IP stack to create packets that are larger than the MTU of the outgoing NIC. For IPv4, packets as large as the IPv4 maximum of 65,536 bytes can be created and queued to the driver queue. In the case of TSO and UFO, the NIC hardware takes responsibility for breaking the single large packet into packets small enough to be transmitted on the physical interface. For NICs without hardware support, GSO performs the same operation in software immediately before queueing to the driver queue.

Ici, l'émulation est bien logicielle et se fait bien au niveau du driver (qui doivent donc être adaptés, d'où ma remarque d'au dessus sur les deux drivers implèmentants la chose), ce qui du coup explique aussi le gain CPU observé (moins de paquets à découper pour respecter la MTU).

Je sais pas si je suis très clair en fait...

C'est un gain de CPU, de débit, de latence ou aucun des 3 ?

A la vue des micro-benchs postés dans les commits, CPU et débits (il y en a un dans la partie en gras dans mon édit deux messages au dessus).

En cherchant des détails sur la chose, on tombe sur un post de 2008 sur la mailing list d'OpenWall (distrib orientée serveurs) (!) ici: http://lists.openwall.net/netdev/2008/07/31/14 qui explique un peu plus clairement la chose:

If a netdevice does not support hardware GSO, allowing the stack to
use GSO anyway and then splitting the GSO skb into MSS-sized pieces
as it is handed to the netdevice for transmitting is likely still
a win at least as far as CPU usage is concerned, since it reduces
the number of trips through the output path.

This patch enables the use of GSO on any netdevice that supports SG
and hardware checksumming.  If a GSO skb is then sent to a netdevice
that supports SG and checksumming but does not support hardware GSO,
net/core/dev.c:dev_hard_start_xmit() will take care of doing the
necessary GSO segmentation in software.

Donc ce que j'en comprends, c'est qu'avant, on ne profitais pas du SG et du checksumming matériel sans TSO, ce qui est normal. La en émulant le TSO de façon logicielle, on peut en tirer parti, d'où les gains constatés que ce soit en terme de débit ou de conso CPU. Si quelqu'un à une meilleure explication...

Le driver permet de paramétrer le TOE :


(copie d'écran du driver de mai 2008 pour Windows Server 2003)

De nouvelles options sont disponibles avec Ubuntu 13.10 et le noyau 3.11 : ce sont les 6 lignes en gras

# ethtool -k eth0
Features for eth0:
rx-checksumming: on
tx-checksumming: on
   tx-checksum-ipv4: on
   tx-checksum-ip-generic: off [fixed]
   tx-checksum-ipv6: on
   tx-checksum-fcoe-crc: off [fixed]
   tx-checksum-sctp: off [fixed]
scatter-gather: on
   tx-scatter-gather: on
   tx-scatter-gather-fraglist: off [fixed]
tcp-segmentation-offload: on
   tx-tcp-segmentation: on
   tx-tcp-ecn-segmentation: on
   tx-tcp6-segmentation: on
udp-fragmentation-offload: off [fixed]
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off [fixed]
rx-vlan-offload: on
tx-vlan-offload: on
ntuple-filters: off [fixed]
receive-hashing: on
highdma: on [fixed]
rx-vlan-filter: off [fixed]
vlan-challenged: off [fixed]
tx-lockless: off [fixed]
netns-local: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off [fixed]
tx-gre-segmentation: off [fixed]
tx-udp_tnl-segmentation: off [fixed]
tx-mpls-segmentation: off [fixed]
fcoe-mtu: off [fixed]
tx-nocache-copy: on
loopback: off [fixed]
rx-fcs: off [fixed]
rx-all: off [fixed]
tx-vlan-stag-hw-insert: off [fixed]
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: off [fixed]

C'est Ubuntu server 13.10 64bits sur un serveur Dell PowerEdge R210 (Xeon X3450 @2.67GHz)

Par curiosité je reviens quelques années après, et je vois que de nouvelles options sont encore disponibles, ci-dessous avec Ubuntu 19.10 et le noyau 5.3 : ce sont les 18 lignes en gras

# ethtool -k enp3s0
Features for enp3s0:
rx-checksumming: on
tx-checksumming: off
   tx-checksum-ipv4: off
   tx-checksum-ip-generic: off [fixed]
   tx-checksum-ipv6: off
   tx-checksum-fcoe-crc: off [fixed]
   tx-checksum-sctp: off [fixed]
scatter-gather: off
   tx-scatter-gather: off
   tx-scatter-gather-fraglist: off [fixed]
tcp-segmentation-offload: off
   tx-tcp-segmentation: off
   tx-tcp-ecn-segmentation: off [fixed]
   tx-tcp-mangleid-segmentation: off
   tx-tcp6-segmentation: off
udp-fragmentation-offload: off
generic-segmentation-offload: off [requested on]
generic-receive-offload: on
large-receive-offload: off [fixed]
rx-vlan-offload: on
tx-vlan-offload: on
ntuple-filters: off [fixed]
receive-hashing: off [fixed]
highdma: on [fixed]
rx-vlan-filter: off [fixed]
vlan-challenged: off [fixed]
tx-lockless: off [fixed]
netns-local: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off [fixed]
tx-gre-segmentation: off [fixed]
tx-gre-csum-segmentation: off [fixed]
tx-ipxip4-segmentation: off [fixed]
tx-ipxip6-segmentation: off [fixed]
tx-udp_tnl-segmentation: off [fixed]
tx-udp_tnl-csum-segmentation: off [fixed]
tx-gso-partial: off [fixed]
tx-sctp-segmentation: off [fixed]
tx-esp-segmentation: off [fixed]
tx-udp-segmentation: off [fixed]
fcoe-mtu: off [fixed]
tx-nocache-copy: off
loopback: off [fixed]
rx-fcs: off
rx-all: off
tx-vlan-stag-hw-insert: off [fixed]
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: off [fixed]
l2-fwd-offload: off [fixed]
hw-tc-offload: off [fixed]
esp-hw-offload: off [fixed]
esp-tx-csum-hw-offload: off [fixed]
rx-udp_tunnel-port-offload: off [fixed]
tls-hw-tx-offload: off [fixed]
tls-hw-rx-offload: off [fixed]
rx-gro-hw: off [fixed]
tls-hw-record: off [fixed]

Le test est fait sur une carte PC entrée de gamme : Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 0c) qui a déja quelques années (PC Dell Core i3-4150)

A noter qu'une ligne a été supprimé :
tx-mpls-segmentation: off [fixed]