Introduction

USB 3.0 is the second major revision of the Universal Serial Bus (USB) standard for computer connectivity. USB 3.0 implements a 5.0 Gbit/s raw transfer rate using 8b/10b encoding. USB 2.0 implements the Hi-Speed mode (HS – 480 Mbit/s), while USB 1.1 implements Low Speed (LS) – 1.5 Mbit/s and Full Speed (FS) – 12 Mbit/s). The USB 3.0 Controller used shall provide one 5.0Gbit/s USB channel using the PS internal GT as PHY.

HW/IP Features

The ZynqMP USB 3.0 Controller shall provide one 5.0Gbit/s USB channel using the PS internal GT as PHY.
  • Two USB 2.0/3.0 controllers
  • Compliant with USB 3.0 specs
  • Compliant with xHCI standard
  • Supports 5.0 Gbps data rate
  • Support host, device and OTG modes
  • Support On The Go (OTG) 2.0 host/device selection
  • Provide simultaneous operation of the USB2.0 and USB3.0 interfaces where applicable
    • In host mode, as required by the standard for speed negotiation and switching
    • In device mode, statically configured as USB2.0 or USB3.0
    • In OTG mode, dynamically switch between host role and device role
      • HNP, SRP supported at USB 2.0 speeds in OTG
      • DRD supported in USB 3.0 speeds
  • 64-bit AXI master port with built-in DMA
  • Register programming via AXI and/or APB slave ports
  • Power management features: hibernation mode
  • Support 44-bit address space
  • Supports Link Power Management (LPM) transfers to save power when bus is idle

Features supported in driver

  • All the HW/IP features are supported by driver

Missing Features, Known Issues and Limitations

  • ZynqMP USB 3.0 controller doesn't support USB 3.0 OTG (On The Go) host/device selection
  • For silicon 1.0 & 2.0 USB ip has HW bugs , we see intermidate resets during transfers. To get USB host/device to work properly we need to make sure below mentioned workarounds are added in fsbl
  1. HSRX disable bypass
    For this workaround please update L<lane number>_TM_ANA_BYP_12 to 0x40 (This should be added before lane PLL lock code sequence in fsbl)
  2. USB2 phy suspend disable
    For this workaround please update GUSB2PHYCFG to 0x2417

Host Mode

ZCU102 board jumper settings for Host mode
J7 - ON
J113 - 1-2
J110 - 2-3

Please refer the below image for jumper settings on ZCU102 board
image2016-6-8 14 39 20.png

USB 3.0 Host mode setup snapshot
image2016-6-3 19-3-25.png


Kernel Configuration

Device Drivers------>
    USB support
        <*> xHCI HCD (USB 3.0) support
        <*> USB Mass Storage support
        <*> DesignWare USB3 DRD Core Support
        DWC3 Mode Selection (Dual Role mode)  --->
            (X) Dual Role mode

By enabling the above we need to see the below mentioned Kconfig parameter enabled
CONFIG_USB_DWC3 = y
CONFIG_USB_DWC3_DUAL_ROLE = y
CONFIG_USB_SUPPORT=y
CONFIG_USB_COMMON=y
CONFIG_USB_ARCH_HAS_HCD=y
CONFIG_USB=y
CONFIG_USB_XHCI_HCD=y
CONFIG_USB_XHCI_PLATFORM=y

Devicetree

usb0: usb@fe200000 {
    #address-cells = <2>;
    #size-cells = <2>;
    status = "disabled";
    compatible = "xlnx,zynqmp-dwc3";
    clock-names = "bus_clk", "ref_clk";
    clocks = <&clk125>, <&clk125>;
 
    dwc3_0: dwc3@fe200000 {
        compatible = "snps,dwc3";
        status = "disabled";
        reg = <0x0 0xfe200000 0x0 0x40000>;
        interrupt-parent = <&gic>;
        interrupts = <0 65 4>;
    };
};
 
&usb0 {
    status = "okay";
};
 
&dwc3_0 {
    status = "okay";
    dr_mode = "host";
};

Test Procedure

Connect the USB 3.0/2.0 pendrive to USB 3.0 capable board and see it getting detected in /dev/sd<x>.

Expected Output

Once Linux boots the below highlighted prints should be visible when we connected the mass storage device
84943.023896] usb 2-1: new SuperSpeed USB device number 4 using xhci-hcd
 
[84943.044949] usb 2-1: New USB device found, idVendor=0781, idProduct=5581
 
[84943.051579] usb 2-1: New USB device strings: Mfr=1, Product=2, SerialNumber=3
 
[84943.058695] usb 2-1: Product: SanDisk Ultra
 
[84943.062858] usb 2-1: Manufacturer: SanDisk
 
[84943.066939] usb 2-1: SerialNumber: A20037A3AD063355
 
[84943.072653] usb-storage 2-1:1.0: USB Mass Storage device detected
 
[84943.078836] scsi host1: usb-storage 2-1:1.0
 
[84944.080804] scsi 1:0:0:0: Direct-Access SanDisk SanDisk Ultra PMAP PQ: 0 ANSI: 6
 
[84944.089531] sd 1:0:0:0: [sda] 30283008 512-byte logical blocks: (15.5 GB/14.4 GiB)
 
[84944.097708] sd 1:0:0:0: [sda] Write Protect is off
 
[84944.103904] sd 1:0:0:0: [sda] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
 
[84944.309725] sda:
 
[84944.312888] sd 1:0:0:0: [sda] Attached SCSI removable disk
 

Performance

When Mass Storage device is connected
Device used :
USB 3.0 : HP USB 3.0 16 GB pendrive (idVendor=03f0, idProduct=4840)
USB 3.0 - UASP capable drive - Transcend Storjet 128GB
USB 2.0 : Sandisk Cruzer Blade USB 2.0 8 GB Pendrive (idVendor=0781, idProduct=5567)

Using HDPARM tool
Mode
Speed
USB 3.0 (Super Speed)
128 MB/sec
USB 3.0 (UASP capable)
327.10 MB/sec
USB 2.0 (High Speed)
25.48 MB/sec

Note : Above results may vary from device to device

When Ethernet device is connected
Device used :
USB 3.0/2.0 : Xilinx ZynqMP RNDIS/Ethernet Gadget

Using Iperf tool (TCP)
Mode
Speed
USB 3.0 (Super Speed)
761.6 Mbits/sec
USB 2.0 (High Speed)
467.2 Mbits/sec

Using Iperf tool (UDP)
Mode
Speed
USB 3.0 (Super Speed)
761.6 Mbits/sec
USB 2.0 (High Speed)
467.2 Mbits/sec

Note : The above results are taken with LPM mode disabled

Peripheral Mode

ZCU102 jumper settings for Peripheral mode
J7 - OFF
J113 - 1-2
J110 - 1-2

Please refer the below image for jumper settings required for peripheral mode on ZCU102 board
image2016-6-8 14-40-39.png

USB 3.0 peripheral mode setup snapshot
image2016-6-3 19-2-9.png

This document only explains the USB 3.0 peripheral mode configurations for MASS STORAGE gadget or ETHERNET Gadget profiles.

Mass Storage

Kernel Configuration

Device Drivers-------->
    USB support
        <*> USB Gadget Support
            <M>   USB Gadget Drivers
            <M>   Gadget Filesystem
            <M>   Mass Storage Gadget
File systems  --->
    Pseudo filesystems  --->
        {M} Userspace-driven configuration filesystem

By enabling the above we need to see the below mentioned Kconfig parameter enabled
CONFIG_USB_DWC3 = y
CONFIG_USB_DWC3_DUAL_ROLE = y
CONFIG_USB_SUPPORT=y
CONFIG_USB_COMMON=y
CONFIG_USB_ARCH_HAS_HCD=y
CONFIG_USB=y
CONFIG_USB_XHCI_HCD=y
CONFIG_USB_XHCI_PLATFORM=y

CONFIG_CONFIGFS_FS=m
CONFIG_USB_GADGET=y
CONFIG_USB_LIBCOMPOSITE=m
CONFIG_USB_F_MASS_STORAGE=m
CONFIG_USB_CONFIGFS=m
CONFIG_USB_CONFIGFS_MASS_STORAGE=y

Please find the required .ko files in the below mentioned paths:

linux-xlnx/fs/configfs/configfs.ko
linux-xlnx/drivers/usb/gadget/libcomposite.ko
linux-xlnx/drivers/usb/gadget/legacy/gadgetfs.ko
linux-xlnx/drivers/usb/gadget/function/usb_f_mass_storage.ko
linux-xlnx/drivers/usb/gadget/legacy/g_mass_storage.ko

Devicetree

usb0: usb@fe200000 {
    #address-cells = <2>;
    #size-cells = <2>;
    status = "disabled";
    compatible = "xlnx,zynqmp-dwc3";
    clock-names = "bus_clk", "ref_clk";
    clocks = <&clk125>, <&clk125>;
 
    dwc3_0: dwc3@fe200000 {
        compatible = "snps,dwc3";
        status = "disabled";
        reg = <0x0 0xfe200000 0x0 0x40000>;
        interrupt-parent = <&gic>;
        interrupts = <0 65 4>;
    };
};
 
&usb0 {
    status = "okay";
};
 
&dwc3_0 {
    status = "okay";
    dr_mode = "peripheral";
};

Performance

Using HDPARM tool (Linux Host)
Link Power Management (LPM) mode Enable
Mode
Speed
USB 3.0 (Super Speed)
127.18 MB/sec
USB 2.0 (High Speed)
38.82 MB/sec

Link Power Management (LPM) mode Disable
Mode
Speed
USB 3.0 (Super Speed)
237 MB/sec
USB 2.0 (High Speed)
38.47 MB/sec

Testing Procedure

Please use the below settings for configuring USB as MASS STORAGE profile in device mode:
insmod configfs.ko
 
insmod libcomposite.ko
 
insmod usb_f_mass_storage.ko
 
insmod gadgetfs.ko
 
dd if=/dev/zero of=/tmp/mydev count=256 bs=1M
 
insmod g_mass_storage.ko file=/tmp/mydev removable=1 stall=1 iSerialNumber=7ABC7ABC7ABC7ABC7ABC7ABC
 

Testing the mass storage functionality by connecting the board to host using the below steps:
  1. Connect the cable from board to windows host machine
  2. Format the mass storage device that got detected in windows
  3. The windows screen shot when we the device gets detected should be as below
    image2016-6-1 15-58-57.png
  4. The board will be detected as mass storage drive with size 256 MB.
  5. Copy some files into the mass storage drive , remove the cable and connect it again. We should be able to see the files that we copied into the drive.

Ethernet Gadget

Kernel Configuration

For the xilinx-v2016.2 and prior tags, apply the following patch.


USB support----->
    <*> USB Gadget Support
        <M>   USB Gadget Drivers
        <M>   Gadget Filesystem
        <M>   Ethernet Gadget (with CDC Ethernet support)
            [*]       RNDIS support
File systems  --->
    Pseudo filesystems  --->
        {M} Userspace-driven configuration filesystem
By enabling the above we need to see the below mentioned Kconfig paramter enabled
CONFIG_USB_DWC3 = y
CONFIG_USB_DWC3_DUAL_ROLE = y
CONFIG_USB_SUPPORT=y
CONFIG_USB_COMMON=y
CONFIG_USB_ARCH_HAS_HCD=y
CONFIG_USB=y
CONFIG_USB_XHCI_HCD=y
CONFIG_USB_XHCI_PLATFORM=y

CONFIG_CONFIGFS_FS=m
CONFIG_USB_GADGET=y
CONFIG_USB_LIBCOMPOSITE=m
CONFIG_USB_ETH=m
CONFIG_USB_ETH_RNDIS=y

Please find the required .ko files in the below mentioned paths:
linux-xlnx/fs/configfs/configfs.ko
linux-xlnx/drivers/usb/gadget/libcomposite.ko
linux-xlnx/drivers/usb/gadget/legacy/gadgetfs.ko
linux-xlnx/drivers/usb/gadget/function/u_ether.ko
linux-xlnx/drivers/usb/gadget/function/usb_f_rndis.ko

Devicetree

usb0: usb@fe200000 {
    #address-cells = <2>;
    #size-cells = <2>;
    status = "disabled";
    compatible = "xlnx,zynqmp-dwc3";
    clock-names = "bus_clk", "ref_clk";
    clocks = <&clk125>, <&clk125>;
 
    dwc3_0: dwc3@fe200000 {
        compatible = "snps,dwc3";
        status = "disabled";
        reg = <0x0 0xfe200000 0x0 0x40000>;
        interrupt-parent = <&gic>;
        interrupts = <0 65 4>;
    };
};
 
&usb0 {
    status = "okay";
};
 
&dwc3_0 {
    status = "okay";
    dr_mode = "peripheral";
};

Performance

Iperf tool (TCP)
Mode
Speed
USB 3.0 (Super Speed)
761.6 Mbits/sec
USB 2.0 (High Speed)
467.2 Mbits/sec

Iperf tool (UDP)
Mode
Speed
USB 3.0 (Super Speed)
761.6 Mbits/sec
USB 2.0 (High Speed)
467.2 Mbits/sec

Testing Procedure

Please use the below settings for configuring USB as ETHERNET gadget profile in device mode:
insmod configfs.ko
 
insmod libcomposite.ko
 
insmod u_ether.ko
 
insmod usb_f_rndis.ko
 
mount -t configfs none /sys/kernel/config
 
cd /sys/kernel/config/usb_gadget
 
mkdir g1
 
cd g1
 
echo "64" > bMaxPacketSize0
 
echo "0x200" > bcdUSB
 
echo "0x100" > bcdDevice
 
echo "0x03FD" > idVendor
 
echo "0x0502" > idProduct
 
mkdir configs/c1.1
 
mkdir functions/rndis.rn0
 
ln -s functions/rndis.rn0/ configs/c1.1/
 
echo "fe200000.dwc3" > UDC
 
ifconfig usb0 10.10.70.1
 
ifconfig usb0 up
 

Testing the ethernet gadget by connecting to linux/windows machine:
  1. Connect the cable from board to windows/linux host machine
  2. Set the ipaddress on device side
    systest# ifconfig usb0 192.168.1.200
  3. Set the ipaddress on host side
  4. ping host address from device side and the result should be ass below mentioned
    ping <host address>

Expected Output:

image2016-8-25 18-30-40.png

USB Attached SCSI Protocol(UASP)

Limitation
TCM MODULE USED FOR UASP does not support ATA_12 commands. so user needs to change VID and PID in file drivers/usb/gadget/legacy/tcm_usb_gadget.c as below
#define UAS_VENDOR_ID 0x0bc2
#define UAS_PRODUCT_ID 0xa013

Kernel Configuration

Device Drivers  --->
    <*> Generic Target Core Mod (TCM) and ConfigFS Infrastructure  --->
        --- Generic Target Core Mod (TCM) and ConfigFS Infrastructure
        <*>   TCM/IBLOCK Subsystem Plugin for Linux/BLOCK
        <*>   TCM/FILEIO Subsystem Plugin for Linux/VFS
        <*>   TCM/pSCSI Subsystem Plugin for Linux/SCSI
        <*>   TCM/USER Subsystem Plugin for Linux
        <*>   TCM Virtual SAS target and Linux/SCSI LDD fabric loopback module
        <*>   Linux-iSCSI.org iSCSI Target Mode Stack
    [*] USB support  --->
         <*> USB Gadget Support
            <*>   USB functions configurable through configfs
            [*]     USB Gadget Target Fabric
            <M>     USB Gadget Target Fabric Module

KCONFIG options for UASP profile
CONFIG_USB_DWC3 = y
CONFIG_USB_DWC3_DUAL_ROLE = y
CONFIG_USB_SUPPORT=y
CONFIG_USB_COMMON=y
CONFIG_USB_ARCH_HAS_HCD=y
CONFIG_USB=y
CONFIG_USB_XHCI_HCD=y
CONFIG_USB_XHCI_PLATFORM=y

CONFIG_CONFIGFS_FS=m
CONFIG_USB_GADGET=y
CONFIG_USB_LIBCOMPOSITE=m
CONFIG_USB_CONFIGFS=m
CONFIG_BLK_DEV_INTEGRITY=y
CONFIG_TARGET_CORE=y
CONFIG_TCM_IBLOCK=y
CONFIG_TCM_FILEIO=y
CONFIG_TCM_PSCSI=y
CONFIG_TCM_USER2=y
CONFIG_LOOPBACK_TARGET=y
CONFIG_ISCSI_TARGET=y
CONFIG_USB_F_TCM=y
CONFIG_USB_CONFIGFS_F_TCM=y
CONFIG_USB_GADGET_TARGET=m
CONFIG_CRC_T10DIF=y

Devicetree

Example Device tree node:
&dwc3_0 {
  status = "okay";
  dr_mode = "peripheral";
}
Please find the required .ko files in the below mentioned paths:
linux-xlnx/fs/configfs/configfs.ko
linux-xlnx/drivers/usb/gadget/libcomposite.ko
linux-xlnx/drivers/usb/gadget/legacy/gadgetfs.ko
linux-xlnx/drivers/usb/gadget/legacy/tcm_usb_gadget.ko

Performance

HDPARM Tool (UBUNTU Host)
Link Power Management (LPM) mode
Result
Enabled
367.77 MB/sec
Disabled
381.42 MB/sec

Test procedure

Connect the USB 3.0 cable to linux PC and perform the steps given below on the device side

Install module
modprobe tcm_usb_gadget

Mount the configfs
mount a -t configfs /sys/kernel/config

Create a memory that will be assigned to gadget (here creating memory of 750MB)
mkdir /root
dd if=/dev/zero of=/root/file.bin bs=10M count=75
 

Format the memory that just has been created
mkdosfs -v /root/file.bin -n ramfs

Assign memory to gadget module
mkdir -p /sys/kernel/config/target/core/fileio_0/fileio
echo "fd_dev_name=/root/file.bin,fd_dev_size=734003200" > /sys/kernel/config/target/core/fileio_0/fileio/control
echo 1 > /sys/kernel/config/target/core/fileio_0/fileio/enable
mkdir -p /sys/kernel/config/target/usb_gadget/naa.6001405c3214b06a/tpgt_1
mkdir /sys/kernel/config/target/usb_gadget/naa.6001405c3214b06a/tpgt_1/lun/lun_0
echo naa.6001405c3214b06a > /sys/kernel/config/target/usb_gadget/naa.6001405c3214b06a/tpgt_1/nexus
ln -s /sys/kernel/config/target/core/fileio_0/fileio /sys/kernel/config/target/usb_gadget/naa.6001405c3214b06a/tpgt_1/lun/lun_0/virtual_scsi_port
 

Change max busrt to get maximum performance
echo 15 > /sys/kernel/config/target/usb_gadget/naa.6001405c3214b06a/tpgt_1/maxburst

Connect device to Host
echo 1 > /sys/kernel/config/target/usb_gadget/naa.6001405c3214b06a/tpgt_1/enable

Ubuntu
On performing the steps given above, below prints should be shown on host side on typing "dmesg"
img4.PNG


Do data transfer on HOST(UBUNTU) side
img5.PNG


Windows
Note: UASP is supported from windows 8

On windows need to follow below steps in order to mount device

1. open manage by right click on Desktop (screen shot as below)

img6.PNG



2. open disk management and format the device (screen shot as below)

img7.PNG


3. It will mount the device as drive
4. Do data transfer


OTG Mode

Using the correct cables is the key to OTG operation. Testing was done using two cables joined together to create an OTG cable. An OTG cable has a micro A connector on one end and a micro B connector on the other end. The micro A connector is the host side of the cable and the micro B connector is the device side by default.

Hardware Setup
Boards: ZynqMP ZCU102 (2 boards back to back connected)
Host Machine: Linux Machine with USB 3.0 ports

Kernel Configuration

Linux Kernel image with xHCI Host enabled (static) and Mass Storage Gadget created as dynamic module
  • Boot image for initial FSBL sequence
  • Device tree with DWC USB OTG mode enabled
Steps to enable OTG in Linux Kernel (add below configurations via menuconfig)
  • CONFIG_PM=y
  • CONFIG_USB_OTG=y
  • CONFIG_USB_OTG_FSM=y
  • CONFIG_USB_DWC3_OTG=y
After this compile the kernel, install modules in it and create kernel Image.

Test Procedure

  • When the Linux kernel boots up, give the below commands:
$ dd if=/dev/zero of=/tmp/mydev count=256 bs=1M
$ modprobe g_mass_storage file=/tmp/mydev removable=1 stall=1 iSerialNumber=7ABC7ABC7ABC7ABC7ABC7ABC

Peripheral Mode
  • Connect board to any Host PC via USB Std-A-Male to Micro-B-Male cable.
  • It should be detected as a Mass Storage Device
  • Do some data transfers

Host Mode
  • Connect any pen-drive with the board via USB Micro-A-Male to Std-A-Female type connector
  • Check that the pen-drive is detected properly
  • Do some data transfers.

HNP & SRP
  • When two boards (OTG enabled) are connected back to back, OTG 2.0 supports HNP and SRP
  • To start HNP from the Device, issue below command
    • $ echo 1 > /sys/devices/platform/amba/ff9d0000.usb0/fe200000.dwc3/hnp
  • Please check the role reversal on both the boards
  • To start SRP from the Device, issue below command
    • $ echo 1 > /sys/devices/platform/amba/ff9d0000.usb0/fe200000.dwc3/srp

Note : Both Host and Peripheral mode should work without resetting/rebooting the board.

Mainline Status

The driver is already in sync with the mainline driver present in 4.9 kernel except the below changes
  • Support for SMMU support
  • Support for UASP streams
  • support for disabling clocks during suspend
  • Fix errors when dwc3 is loaded is module

Phy Settings

This USB controller on Zynq UltraScale+ MPSoC is connected to High Speed GTR through PIPE3 interface. This phy requires reference clock to operate with USB 3.0, which can be
configured to any one among 26MHz, 52 MHz and 100MHz. On ZCU102 board this is factory programmed to 26MHz.

Change log

2016.3
Summary:
  • Open source changes 4.6 kernel
  • Fixed kernel hang during unbind
  • Fixed warning during unbind
Commits:
556131
f69e4ea

2016.4
Summary:
  • Added device tree nodes for LPM transfers support on zcu102 board
  • Added device tree nodes for LPM support for DC1 board
Commits:
  • None

2017.1
Summary:
  1. DWC3 DRD driver
    • Assigned dwc3 archdata to xhci dev to avaiod empty dma ops
    • Added U3 suspend quirk for silicon revisions > 3.0 for fixing ep configuration error
    • Programme GFADJ register based on the value from dts
    • Added device hibernation changes
    • Added CCI support in USB if "dma-coherent" flag is set in dts
    • Changed dwc3-of-simple.c file for configuring PIPE3 signals as part of phy_init
    • Added CCF support for USB
    • Corrected f_hid gadget
  2. XHCI host driver
    • Added support for Light reset instead of HARD reset if OTG is enabled
    • Correct the logic for ep ring caching in xhci
    • Fixed the logic for enabling host bulk streaming support
    • Open source changes 4.9 kernel
  3. DWC3 OTG driver
    • Added OTG driver to DWC3
Commits:
f80c29
8fac50
a22ddb
6e3067
554c94
68e686
2cdd9b
76bec5
5bfa74

2017.2
Summary:
  • Removed unnecessary ep_queue calling from UVC class driver
Commits:
17360fb

2017.3
Summary:
  • Added SMMU support for HOST and DEVICE mode
  • Added workaround for BULK IN streams in HOST UAS mode
  • Add support for disabling clock during suspend
  • Add supportf or remvoing VBUS when suspended
  • Corrected errors when dwc3 loaded as module
  • Add device mode UASP (USB Attached SCSI PROTOCOL) support
Commits:
2fcec8
95349
80ac0
a4769
244dd
43a81

2017.4
Summary:
  • Corrected logic for finding the parent node when multiple instances of USB are enabled
Commits:
6294b

2018.1
Summary:
  • Added code for supporting entering into D3 state during suspend
  • Put usb core in reset after entering D3 state
  • Updated driver to 4.14 kernel
  • Add hibernation support when operating in gadget mode
  • Add support for enabling/disabling hibernation dynamically
  • Don't return error when phy is not found
  • Don't wait for end transfer to finish for isoc endpoints
  • Fix the broken suspend/resume functionality in dwc3
  • In OTG mode don't clear event buffers when changing to Host mode

Commits:
4e93d
18f04
818f1
e061e
276fd
c0c94
144e1
90f0c
e6680

2018.2
Summary:
  • Fix ISOC transfers for UVC class
Commits:
7e8d

Related Links