Axi Ethernet Linux driver for Microblaze and Zynq and Zynq Ultrascale+ MPSoC


Introduction
This page gives an overview of Axi Ethernet Linux driver which is available as part of the Linux distribution. Paths, files, links and documentation on this page are given relative to the Linux kernel source tree.

HW IP features

  • Support for MII, GMII, RGMII, SGMII, and 1000BASE-X PHY interfaces.
  • Support for 1000BASE-X and SGMII over Select Input/Output (I/O) Low Voltage Differential Signaling (LVDS) in faster devices.
  • Support for pause frames for flow control
  • Media Independent Interface Management (also called as MDIO), is used for accessing The PHY registers.
  • Ethernet Audio Video Bridging (AVB) support
  • AXI4-Stream transmit/receive interface
  • Support for 2.5G Ethernet. This feature is enabled for the following devices
    • Kintex®-7, Virtex®-7 with GTH and GTX transceivers
    • Artix®-7 devices with GTP and speed grade -2 and -3
    • UltraScale™, UltraScale+™ devices with GTH and GTY transceivers.
  • IEEE Standard 1588 Support.
  • AXI4-Lite register interface
  • Full/Partial/None Check Sum Offloading
  • Optional support for jumbo frames up to 16 KB
  • Optional TX and RX Transmission Control Protocol/ User Datagram Protocol (TCP/UDP) partial checksum offload
  • Optional IPv4 TX and RX TCP/UDP full checksum offload
  • Support for VLAN frames
  • Optional TX and RX VLAN tagging, stripping, and translation
  • Independent 4K, 8K, 16K, or 32KB TX and RX frame buffer memory
  • Optionally extended filtering for multicast frames
  • Optional TX and RX statistics gathering
  • Auto PAD and frame check sequence (FCS) field insertion or pass through on transmit
  • Auto PAD and FCS field stripping or pass through on receive
  • Ethernet Audio Video Bridging (AVB) at 100/1000 Mb/s (Additional license required)
  • Filtering of bad receive frames
  • Option to include or exclude shareable logic resources in the subsystem
  • Optional support for board-based I/O constraints generation

Features supported in the driver

  • Support for ethtool queries
  • NAPI support.
  • IEEE 1588 Support for 1G and legacy 10G MAC (PG157) and 10G/25G MAC (PG210)
  • Full/Partial Checksum offload support
  • Speed support for 10/100/1000 Mbps for 1G MAC
  • 10G Base-R support for Legacy 10G MAC(PG157) and 10G/25G MAC (PG210)
  • Support for GMII/RGMII/SGMII/1000Base-X Phy Configurations
  • Supports Independent 4K, 8K, 16K, or 32KB TX and RX frame buffer memory
  • Support for Jumbo Frames
  • Supports Axi Ethernet with AXI DMA Configuration.
  • Supports Axi Ethernet with AXI MCDMA Configuration
  • Multi-queue support
  • USXGMII support (PG251)

Missing Features and Known Issues/Limitations in Driver

  • The current driver assumes that Axi Ethernet IP is connected to the DMA at the hardware level.
  • The current driver doesn't use dma engine framework and contains DMA programming sequence i.e doesn't use separate DMA driver. Hence compatibility string of axidma node (DTS) is set to a dummy device-tree property compatible = "xlnx,eth-dma";
  • The driver doesn't support software timestamping. It supports only hardware timestamping.
  • No support for fixed-link
  • For 1588 testing the Current driver assumes that AXI Stream FIFO is connected to the MAC TX Time stamp Stream interface at the design level.
  • 10G/25G and USXGMII configurations do not support dynamic link status/change in the background if there is no external PHY using PHY framework.

Kernel Configuration

The following config options should be enabled in order to build the Axi Ethernet driver
CONFIG_ETHERNET
CONFIG_NET_VENDOR_XILINX
CONFIG_XILINX_AXI_EMAC
CONFIG_AXIENET_HAS_MCDMA (Select this option In the design if Axi Ethernet is configured with Axi MCDMA)
CONFIG_XILINX_PHY (For testing SGMII/1000Base-x Configuration with PCS/PMA Core)


axi_ethmenuconfig.PNG

Device-tree

For more details on phy bindings please refer "Documentation/devicetree/bindings/net/phy.txt"
axi_ethernet_eth_buf: ethernet@40c00000 {
axistream-connected = <&axi_dma_1>;
axistream-control-connected = <&axi_dma_1>;
clock-frequency = <100000000>;
clocks = <&clk_bus_0>;
compatible = "xlnx,axi-ethernet-1.00.a";
device_type = "network";
interrupt-parent = <&microblaze_1_axi_intc>;
interrupts = <4 2>;
reg = <0x40c00000 0x40000>;
xlnx,phy-type = <0x4>;
xlnx,phyaddr = <0x1>;
xlnx,rxcsum = <0x0>;
xlnx,rxmem = <0x8000>;
xlnx,txcsum = <0x0>;
phy-handle = <&phy0>;
mdio {
#address-cells = <1>;
#size-cells = <0>;
phy0: phy@7 {
device_type = "ethernet-phy";
reg = <7>;
};
};
};

Soft Ethernet MAC(1G, legacy 10G or 10G/25G MAC) Configured with MCDMA

When Axi Ethernet Configured with MCDMA device-tree node will be like below
xxv_ethernet_0: ethernet@80020000 {
            axistream-connected = <&axi_dma_hier_axi_mcdma_0>;
            axistream-control-connected = <&axi_dma_hier_axi_mcdma_0>;
            clock-frequency = <100000000>;
            compatible = "xlnx,xxv-ethernet-1.0";
            device_type = "network";
            phy-mode = "base-r";
            reg = <0x0 0x80020000 0x0 0x10000>;
            xlnx = <0x0>;
            xlnx,add-gt-cntrl-sts-ports = <0x0>;
            xlnx,anlt-clk-in-mhz = <0x64>;
            xlnx,axis-tdata-width = <0x40>;
            xlnx,axis-tkeep-width = <0x7>;
            xlnx,base-r-kr = "BASE-R";
            xlnx,channel-ids = "1","2","3","4","5","6","7","8","9","a","b","c","d","e","f","10";
            xlnx,clocking = "Asynchronous";
            xlnx,core = "Ethernet MAC+PCS/PMA 64-bit";
            xlnx,data-path-interface = "AXI Stream";
            xlnx,enable-datapath-parity = <0x0>;
            xlnx,enable-pipeline-reg = <0x0>;
            xlnx,enable-preemption = <0x0>;
            xlnx,enable-preemption-fifo = <0x0>;
            xlnx,enable-rx-flow-control-logic = <0x0>;
            xlnx,enable-time-stamping = <0x0>;
            xlnx,enable-tx-flow-control-logic = <0x0>;
            xlnx,enable-vlane-adjust-mode = <0x0>;
            xlnx,family-chk = "zynquplus";
            xlnx,fast-sim-mode = <0x0>;
            xlnx,gt-diffctrl-width = <0x4>;
            xlnx,gt-drp-clk = "100.00";
            xlnx,gt-group-select = "Quad X0Y0";
            xlnx,gt-location = <0x1>;
            xlnx,gt-ref-clk-freq = "156.25";
            xlnx,gt-type = "GTH";
            xlnx,include-auto-neg-lt-logic = "None";
            xlnx,include-axi4-interface = <0x1>;
            xlnx,include-dre ;
            xlnx,include-fec-logic = <0x0>;
            xlnx,include-rsfec-logic = <0x0>;
            xlnx,include-shared-logic = <0x1>;
            xlnx,include-user-fifo = <0x1>;
            xlnx,lane1-gt-loc = "X0Y4";
            xlnx,lane2-gt-loc = "NA";
            xlnx,lane3-gt-loc = "NA";
            xlnx,lane4-gt-loc = "NA";
            xlnx,line-rate = <0xa>;
            xlnx,mii-ctrl-width = <0x8>;
            xlnx,mii-data-width = <0x40>;
            xlnx,num-of-cores = <0x1>;
            xlnx,num-queues = /bits/ 16 <0x10>;
            xlnx,ptp-clocking-mode = <0x0>;
            xlnx,ptp-operation-mode = <0x2>;
            xlnx,runtime-switch = <0x0>;
            xlnx,rxmem = <0x40000>;
            xlnx,switch-1-10-25g = <0x0>;
            xlnx,tx-latency-adjust = <0x0>;
            xlnx,tx-total-bytes-width = <0x4>;
            xlnx,xgmii-interface = <0x1>;
            xxv_ethernet_0_mdio: mdio {
                #address-cells = <1>;
                #size-cells = <0>;
            };
        };

When Soft Ethernet MAC configured with MCDMA, The driver supports several features of the MCDMA
  • The driver supports random Queue/Channel selection.
Ex: At the h/w level let's assume MCDMA is configured for 16 channels and user don't want Linux driver to use all the 16 channels

Then need to do below changes in the mac device-tree node
For Example, User would like to use Channels 2, 5 and 10 with Linux driver and rest of the channel for his own use cases.

Need to modify below properties in the device-tree for MAC node
xlnx,num-queues = <0x3>;
xlnx,channel-ids = "2","5","10";
  • Driver supports channel observer feature through sysfs entries.
  • The driver supports Per Channel weight configuration through sysfs entries.

To support USXGMII + MCDMA, use above devicetree as reference and select:
xlnx,phy-type = <0x7>;

Related device tree information

For PHY related DT information, refer to
https://github.com/Xilinx/linux-xlnx/blob/master/Documentation/devicetree/bindings/net/phy.txt

When selecting phy specific settings, make sure to mention interface type, speed (if limited/fixed) and phy address properties.
PHY/Converter devices that may be used with this MAC:
-> Xilinx GMII2RGMII converter (https://github.com/Xilinx/linux-xlnx/blob/master/Documentation/devicetree/bindings/net/xilinx_gmii2rgmii.txt)
-> Xilinx PCS PMA PHY

IEEE 1588 Support

The Existing Axi Ethernet driver in the Xilinx git hub supports 1588 for 1G MAC and legacy 10G MAC and 10G/25G MAC
It does timestamping at the MAC level. 1588 is supported in 7-series and Zynq.
The Current driver assumes that AXI Stream FIFO is connected to the MAC TX Time stamp Stream interface at the design level.

Note:
In order to test the hardware platform, a 1588 timestamp capable timer is required at the h/w level.

Pre-requisite for 1588 testing:
Need to enable XILINX_AXI_EMAC_HWTSTAMP in the kernel config which does the timestamping at the MAC level (Tx/Rx)
Users should write their own timer driver for time stamping which does time adjustment/ freq adjustment.

For testing 1588 please refer to Testing tools section below.

1588 Device-tree Example Node
     axi_ethernet_eth_buf: ethernet@40c00000 {
            axistream-connected = <&axi_dma_1>;
            axififo-connected = <&axi_fifo_0>;
            clock-frequency = <100000000>;
            clocks = <&clk_bus_0>;
            compatible = "xlnx,axi-ethernet-1.00.a";
            device_type = "network";
            interrupt-parent = <&microblaze_1_axi_intc>;
            interrupts = <4 2>;
            reg = <0x40c00000 0x40000>;
            xlnx,phy-type = <0x4>;
            xlnx,phyaddr = <0x1>;
            xlnx,rxcsum = <0x0>;
            xlnx,rxmem = <0x8000>;
            xlnx,txcsum = <0x0>;
            xlnx,txmem = <0x8000>;
            phy-handle = <&phy0>;
            mdio {
              #address-cells = <1>;
              #size-cells = <0>;
                  phy0: phy@7 {
                      compatible = "marvell,88e1111";
                     device_type = "ethernet-phy";
                     reg = <7>;
                };
            };
        };

Performance

These benchmark performance numbers were obtained by connecting Xilinx boards to Linux PCs/server machines (Ubuntu/Red Hat Enterprise).
The tools used are netperf or iperf (Refer to tool information below).

1G Ethernet with AXIDMA

Kernel version: 4.14

ZynqMP
Board: ZCU102 board (production silicon) + SFP Module


TCP (Mbps)
UDP (Mbps)
MTU
TX
CPU(%)
RX
CPU(%)
TX
CPU(%)
RX
CPU(%)
1500
941
9.5
941
53.8
961
9.7
892
38.0
8192
981
7.1
959
20.5
949
16.5
961
12.8

Zynq
Board: ZC706 board + SFP Module

TCP (Mbps)
UDP (Mbps)
MTU
TX
CPU(%)
RX
CPU(%)
TX
CPU(%)
RX
CPU(%)
1500
912
34.6

622
50.0
831
50.2
617
91.0
8192
984
19.0
851
74.3
735
30.2
906
71.7

10G Ethernet with AXIMCDMA

Kernel version: 4.14

ZynqMP
Board: ZCU102 board (production silicon) + SFP Module


TCP (Gbps)
UDP (Gbps)
MTU
TX
CPU(%)
RX
CPU(%)
TX
CPU(%)
RX
CPU(%)
1500
3.01
82.4
1.96
58.8
2.77
99.9
1.71
86.7
8192
5.00
59.2
4.37
52.4
6.69
77.4
5.33
64.7
NOTE: In this design 1588 is not enabled.

Setup Details
Host setup: Dell System Precision Tower 7910 (0619)
Iperf: iperf 3-CURRENT (cJSON 1.5.2)
OS : Linux xhdpunnaia40 3.13.0-147-generic #196-Ubuntu SMP Wed May 2 15:51:34 UTC 2018 x86_64
NIC (10G Solarflare's SFN6322F Dual-Port 10GbE SFP+ Adapter) : Default

Performance benchmarking

Pre-requisites:
  • Set Ethernet MCDMA TX interrupt affinity to core-1
root@10g-mcdma-no1588-build:~# echo 2 > /proc/irq/xx/smp_affinity
  • Run iperf servers on ZynqMP (core2 and core3)
root@10g-mcdma-no1588-build:~# taskset -c 2 iperf3 -s -p 5101
root@10g-mcdma-no1588-build:~# taskset -c 3 iperf3 -s -p 5102
  • CPU Utilization reporting
root@10g-mcdma-no1588-build:~# ./mpstat -P ALL 1 50
  • Run iperf servers on the remote host
server:~# iperf3 -s -p 5101 & ; iperf3 -s -p 5102 & ; iperf3 -s -p 5103 & ; iperf3 -s -p 5104 &

Testcases:




Test Procedure

Diagnostic and Protocol Tests

PING
This utility used to test the reachability of a host on an Internet Protocol(IP) network and to measure the round trip time for messages sent from the originating host to a destination computer

How to Run
ping <Remote IP Address>


Telnet
telnet <Server IP Address>


Pkt Generator
Please refer the below link for how to run and various options
https://www.kernel.org/doc/Documentation/networking/pktgen.txt

Stress Test

Iperf with option -d
Run iperf in dual testing mode. This will cause the server to connect back to the client on the port specified in the -L option (or defaults to the port the client connected to the server on). This is done immediately therefore running the tests simultaneously.

./iperf -c <Server IP> -d

Ping flood test
Users can send hundred or more packets per second using -f option. It prints a ‘.’ when a packet is sent, and a backspace is printed when a packet is received
ping -f localhost

Performance Test

Netperf
More information please refer to the below link
http://www.netperf.org/netperf/

How to Run
Server:
netserver

Client:
taskset 2 ./netperf -H <Server IP> -t TCP_STREAM
taskset 2 ./netperf -H <Server IP> -t UDP_STREAM

Iperf
More information please refer to the below link
http://en.wikipedia.org/wiki/Iperf

How to Run
Server:
./iperf -s -u
./iperf -s

Client:
./iperf -c <Server IP> -u -b <banwidth>
./iperf -c <Server IP>

1588 Testing

Build Insturctions for PTPd
--> Get the source from
git clone git://git.code.sf.net/p/linuxptp/code linuxptp
--> Set the CROSS_COMPILE environment variable arm tool chain
--> Install the kernel headers
--> Include the headers path in makefile
INC = -I/proj/epdsw/punnaiah/git/test/ethernet/1588/header/include
CFLAGS = -Wall $(VER) $(incdefs) $(DEBUG) $(INC) $(EXTRA_CFLAGS)
--> run make

Installing kernel headers:
https://www.kernel.org/doc/Documentation/kbuild/headers_install.txt

Before synchronization
phc2sys -s <devicename> -w &

can be run to synchronize system and PTP clock if intended.

In order to perform master-slave sync, run the following:

Master (Linux server for example):
ptp4l -i < interface name> -m
Slave:
ptp4l -i <interface name> -m -s

Mainline status

The current Axi Ethernet driver is currently in sync with mainline except for the following
  • SGMII/1000Base-x support
  • NAPI Support
  • Support for 1588
  • Support for legacy 10G MAC
  • Support for 1G MAC Non-processor Mode
  • Support for 10G/25G MAC with 1588
  • Support for 2.5G MAC with 1588
  • Support for Axi Ethernet with AXI MCDMA Configuration
  • Support for USXGMII IP.

Change Log


2018.2
  • None

2018.1
  • Fix xxv mac padding issue.
  • Add support for USXGMII IP.
  • Update interrupt-names property.
  • Code cleanup of tx with no DRE.
  • Fix remove() bug for XXV no MDIO.
Commit IDs:
10681b8 net: ethernet: xilinx: update interrupt-names property with ip interupt naming convention
fe44c16 net: ethernet: xilinx: Fix xxv mac padding issue - only pad last element.
cea5c97 net: ethernet: xilinx: axienet cleanup of tx with no dre
f5de0cd net: ethernet: xilinx: only teardown mdio when available
9733c76 net: xilinx: axiethernet: Add USXGMII support

2017.4
  • None

2017.3
  • Added Support for Ethernet MCDMA Configuration
  • Added Support for 1588 in buffered mode configuration.
  • Fixed race condition in the transmit path
  • Fixed race condition in the random queue selection for Ethernet MCDMA configuration
Commit Id's:
2108c9fnet: ethernet: xilinx: Add support for mcdma
2108c9fnet: ethernet: xilinx: Add support for 1588 in buffered mode
2108c9fnet: ethernet: xilinx: Fix race condition in the tx path
2108c9fnet: ethernet: xilinx: Fix race in the random queue selection

2017.2
  • None

2017.1
  • Added Support for 10G/25G MAC (PG210)
  • Added Support for 2.5G MAC
  • Fixed issues with the without DRE DMA based design on zynqMP SOC
  • Added Clock Support
  • Fixed bug in the rx reject interrupt handling.
Commit Id's:
22fa015 net: ethernet : xilinx Add config structure to differnt axienet macs
e3c3b23 net: ethernet: Add quirk checks in the driver
453042c net: ethernet: Add Support for 10G/25G MAC
4027f19 net: ethernet: Add 1588 support for 10G/25G MAC
f475798 net: ethernet: Fix race condition in the driver for 10G/25G MAC
486d636 net: ethernet: Add support for 2.5G MAC
2857aee net: ethernet: Fix issues in the driver when DRE is not enabled in the h/w
a15cd73 net: ethernet: Add Clock support
9b904af net: ethernet: Fix Bug in rx reject interrupt handling.

2016.4
  • None

2016.3
  • Fix kernel crash on the 64-bit platform reported while testing 1588 on Zynq UltraScale+ MPSOC.
Commit Id's:
ea9f81c net: ethernet: xilinx: Fix kernel crash on 64-bit platform

Related Links

---> http://www.wiki.xilinx.com/Linux+Drivers