History
VIVADO DS 2014.3 Zynq PCIe Targeted Reference Design
ISE DS 14.7 Zynq PCIe Targeted Reference Design
ISE DS 14.6 Zynq PCIe Targeted Reference Design
ISE DS 14.5 Zynq PCIe Targeted Reference Design
ISE DS 14.4 Zynq PCIe Targeted Reference Design
ISE DS 14.3 Zynq PCIe Targeted Reference Design

1 Introduction

This page provides instructions on how to build various components of the Zynq PCIe Targeted Reference Design (TRD) and how to setup the hardware platform and run the design on the ZC706 Evaluation Kit. The ZC706 Evaluation kit is based on a XC7Z045 FFG900-2 Zynq-7000 SoC device. For additional information, refer to UG961.

1.1 About the Zynq PCIe TRD

The Zynq PCIe Targeted reference design expands the Base Targeted Reference Design (UG925) by adding PCI Express communication with a host system at PCIe x4 GEN2 speed. In the Base Targeted Reference design, the input of the video processing pipeline is generated by a test pattern generator in the FPGA fabric. In this design, the input of the video processing pipeline is generated by an application on the host computer at 1080p60 resolution and transmitted to the ZC706 board via PCIe. The data is processed by video pipeline and passed back to the host system via PCIe. As full 1080p60 video stream only take up around 4Gbps, an additional data generator and a checker are implemented and connected to channel 1 of PCIe DMA showcasing the maximum PCIe x4 GEN2 bandwidth achieved by the hardware. For additional information, please refer to UG963
z7_pcie_trd_block_diagram.JPG

1.2 Zynq PCIe TRD Package Contents

The Zynq PCIe TRD package is released with the source code, Xilinx PlanAhead and SDK projects, and an SD card image that enables the user to run the video demonstration and software
application. It also includes the binaries necessary to configure and boot the Zynq-7000 SoC board. The package also contains the software driver source files required to run application
software in the PCIe host machine. This wiki page assumes the user has already downloaded the PCIe TRD package and extracted its contents to the PCIe TRD home directory referred to as
ZYNQ_TRD_HOME in this wiki and to the PCIe host machine in a folder of choice.

zynq_pcie_dir_struct_2014_4.png
2 Prerequisites
  • The ZC706 Evaluation Kit ships with the version 14.x Device-locked to the Zynq-7000 XC7Z045 FFG900-2 device and all required licenses to build the TRD. For additional information, refer to UG973 Vivado Design Suite User Guide: Release Notes, Installation, and Licensing .
  • PC with PCIe v2.0 slot. Recommended PCI Express Gen2 PC system motherboards are ASUS P5E (Intel X38), ASUS Rampage II Gene (Intel X58) and Intel DX58SO (Intel X58).
  • Fedora 16 LiveCD for booting Linux on PCIe host machine.
  • A Linux development PC with the ARM GNU tools installed. The ARM GNU tools are included with the Xilinx ISE Design Suite Embedded Edition or can be downloaded separately.
  • A Linux development PC with the distributed version control system Git installed. For more information, refer to Using Git and to UG821: Xilinx Zynq-7000 EPP Software Developers Guide.
  • Petalinux 2014.2 SDK
  • Open JDK
  • Other system utilities like make (3.82 or higher) and corkscrew if accessing git behind a firewall.
  • A Linux development PC with QT and QWT libraries cross-compiled for Zynq platform. Set ZYNQ_QT_INSTALL environment variable by referring to Xilinx Zynq Qt/Qwt Libraries - Build Instructions

3 Building the FPGA Hardware Bitstream

This section explains how to generate the FPGA hardware bitsream using the Xilinx PlanAhead tool and how to export the hardware platform to Xilinx Software Development Kit (SDK) for software application development. Inside the PlanAhead project, a Xilinx Platform Studio (XPS) project is referenced that contains the embedded hardware design. The design top level file instantiates the embedded top level file along with the system with PCIe IP wrapper, PCIe DMA, PCIe performance monitor and hardware generator and checker blocks.

3.1 Building the Bitstream

Note: The TRD uses Tandem PROM flow to generate the bitstream.
Tandem PROM flow generates a two staged bitstream.
The first stage bitstream is smaller sized bitstream and is used to
meet 100 ms boot up time requirement in PCIe based End Points.
For more information, please refer to PG054,
7 Series FPGAs Integrated Block for PCI Express Product Guide
Steps for building the FPGA hardware bitstream

Browse to $ZYNQ_TRD_HOME/hardware/vivado/scripts directory:
  • On Windows ,Open Vivado 2014.4 Tcl Shell by clicking Start > All Programs > Xilinx Design Tools > Vivado 2014.4 > Vivado 2014.4 Tcl Shell
    • In the command prompt navigate to $ZYNQ_TRD_HOME/hardware/vivado/scripts and enter " vivado -source z7_pcie_trd_project.tcl " command
  • On Linux, enter " vivado -source z7_pcie_trd_project.tcl " at the command prompt.
This opens the Vivado® Integrated Design Environment (IDE), loads the block diagram, and adds the required top file and XDC file to the project.

zynq_pcie_bd_2014_3.png


In the Flow Navigator pane on the left-hand side under Program and Debug, click Generate Bitstream.
  • Note: Click on Yes if a window appears as No Implementation Results are available.

The bit stream will be generated at $ZYNQ_TRD_HOME/hardware/vivado/runs/z7_pcie_trd_2014.4.runs/impl_1/z7_pcie_trd_top.bit

zynq_pcie_bit_gen_2014_3.png

3.2 Exporting the Hardware Platform to SDK


Steps for exporting the hardware platform to SDK

From the VIVADO project flow navigator click on Open Implemented Design.
From the VIVADO menu bar, select File > Export > Export Hardware.

zynq_pcie_expo_hw_2014_3.png

In the Export Hardware window press OK. The hardware platform will be exported to $ZYNQ_TRD_HOME/hardware/vivado/runs/z7_pcie_trd_2014.4.sdk

zynq_pcie_expo_hw_2_2014_3.png


4 Installation of Petalinux SDK

4.1 Prerequisites

  • 2GB RAM (recommended minimum for Xilinx tools)
  • Pentium 4 2GHz CPU clock or equivalent.
  • 5 GB free HDD space.
  • Supported OS:
    • RHEL 5 (32-bit or 64-bit)
    • RHEL 6 (32-bit or 64-bit)
    • SUSE Enterprise 11 (32-bit or 64-bit)
  • Petalinux release package downloaded.
  • Valid Petalinux license.
  • Common system packages and libraries are installed on your workstation. The installation process will check for these. See the section Required Tools and Libraries for more details. For detailed information refer petalinux installation guide UG1144 .
  • Download Petalinux 2014.4 SDKsoftware from Xilinx website download section

4.2 Extract the Petalinux Package

Assuming all the prerequisites described in the last subsection are satisfied, Petalinux installation is very straight forward.Without any options, the installer will installl as a subdirectory of the current directory. Alternatively, an installation path may be specific . Run the downloaded petalinux installer.
bash> ./petalinux-v2014.4-final-installer.run
Petalinux will be installed in the petalinux-v2014.4-final directory, directly underneath the working directory of this command.
So, if you install the installer into your home directory /home/user, Petalinux will be installed in /home/user/petalinux-v2014.4-final.
You may move the resulting petalinux-v2014.4-final directory to a preferred location before continuing.

4.3 Install License

PetaLinux licenses are managed using the same system as all other Xilinx Design Tools. For more details on licensing and setup of license please refer to the
Vivado Design Suite User Guide (UG973)" section "Obtaining and Managing a License".

4.4 Setup Petalinux Working Environment

After extracting the package, the remainder of the setup is completed automatically.
1. Go to the Petalinux root directory by running this command on the command console:
cd <path-to-installed-PetaLinux>
e.g.:
bash> cd /home/user/petalinux-v2014.4-final
2. Source the appropriate PetaLinux setup script by running this command on the command console:
For Bash:
bash> source settings.sh
The first time the setup script is sourced, it will perform some post installation tasks to check system dependencies and initialise the Linux kernel source tree.
Below is an example of the output from sourcing the setup script for the first time:

Petalinux environment set to ’/home/user/petalinux-v2014.4-final
INFO: Finalising Petalinux installation
INFO: Checking free disk space
INFO: Checking installed tools
INFO: Checking installed development libraries
INFO: Checking network and other services
The post-install step only occurs once. Subsequent runs of the settings script should be much quicker, and simply output a confirmation message such as that shown below:
PetaLinux environment set to ’/home/user/petalinux-v2014.4-final'

4.5 Verify Petalinux Installation

Verify that the Petalinux working environment has been set:
bash> echo $PETALINUX
/home/user/petalinux-v2014.4-final
Environment variable "$PETALINUX" should point to the path to the installed Petalinux. Your echo output may be different from this example, depending upon where you installed Petalinux.

5 Add Linux kernel 3.15.0 support

Petalinux configuration help us to select custom kernel required for Zynq PCIe TRD project.
bash> cd $ZYNQ_TRD_HOME/software/zynq_ps/petalinux
bash> petalinux-config
This brings up a GUI. DO NOT MODIFY ANY OPTIONS.
Simply EXIT from the GUI.

Now, Petalinux starts fetching all required components, that include Linux source tree (based on xilinx-v2014.4 tag).
Wait till the command completes.

6 Modify Linux kernel for TRD support
Zynq PCIe TRD uses Xilinx Linux 3.17.0 kernel version.Petalinux provides option to add individual project
specific kernel/u-boot version. Below steps demonstrates how to add/config/build the Linux kernel.
bash> cd $ZYNQ_TRD_HOME/software/zynq_ps/petalinux/build/linux/kernel/download/linux-xlnx
Apply the kernel patch that contain changes specific to Zynq PCIe TRD version 2014.4 release.
bash> git apply --stat $ZYNQ_TRD_HOME/software/zynq_ps/patch/zynq_pcie_trd_2014_4.patch
bash> git apply --check $ZYNQ_TRD_HOME/software/zynq_ps/patch/zynq_pcie_trd_2014_4.patch
bash> git am $ZYNQ_TRD_HOME/software/zynq_ps/patch/zynq_pcie_trd_2014_4.patch
bash> cd $ZYNQ_TRD_HOME/software/zynq_ps/petalinux
bash> cp subsystems/linux/configs/kernel/xilinx_zynq_pcie_trd_defconfig
         subsystems/linux/configs/kernel/config
This steps completes the setup of Linux source tree for Zynq PCIe TRD.

7 Build Petalinux images

Finally, it’s time to build your petalinux image. TRD installable BSP auto-configures required software settings.
Below steps are categorized into two sections:
a) Petalinux images for SD BOOT
b) Petalinux images for QSPI boot

For the purpose of keeping the images generated in step 8 and 9 , it is recommended to create $ZYNQ_TRD_HOME/build/ready_to_test/prog_qspi directory and also copy the init.sh script to build directory.
bash> mkdir -p $ZYNQ_TRD_HOME/build/ready_to_test/prog_qspi
bash> cp $ZYNQ_TRD_HOME/ready_to_test/prog_qspi/init.sh  $ZYNQ_TRD_HOME/build/ready_to_test/prog_qspi

7.1 Petalinux SD BOOT images

Run ’petalinux-build’ in the petalinux SDK project directory to build the Petalinux system image:
bash> cd $ZYNQ_TRD_HOME/software/zynq_ps/petalinux
bash> petalinux-build
bash> petalinux-package --image -c kernel --format uImage
 
Note: For more verbose build message use petalinux-build -v.
The console shows the compilation progress. e.g.:
INFO: Checking component...
INFO: Generating make files and build Linux
INFO: Generating make files for the subcomponents of linux
INFO: Building Linux
<snip>

NOTE:
Compilation log are stored in build.log in the $(PETALINUX)/zynq_pcie_trd_2014_4/build directory.
Build image.ub is generated in $(PETALINUX)/zynq_pcie_trd_2014_4/images/linux directory.
Copy built image.ub and BOOT.BIN to $ZYNQ_TRD_HOME build directory
bash> cd $ZYNQ_TRD_HOME/software/zynq_ps/petalinux/images/linux
bash> cp uImage $ZYNQ_TRD_HOME/build/ready_to_test/prog_qspi
bash> cp system.dtb $ZYNQ_TRD_HOME/build/ready_to_test/prog_qspi/devicetree.dtb
bash> cp urootfs.cpio.gz $ZYNQ_TRD_HOME/build/ready_to_test/prog_qspi/uramdisk.image.gz
bash> petalinux-package --boot --fsbl $ZYNQ_TRD_HOME/software/zynq_ps/petalinux/images/linux/zynq_fsbl.elf --uboot
bash> cp BOOT.BIN $ZYNQ_TRD_HOME/build/ready_to_test/prog_qspi

7.2 Petalinux QSPI BOOT images

Since petalinux BSP default configures to SD mode , we need to update petalinux configuration
for QSPI boot mode and TRD specific device tree.

a) Update default device tree source (system.dts) with TRD device tree.
bash> cd $ZYNQ_TRD_HOME/software/zynq_ps/petalinux
bash> cp $ZYNQ_TRD_HOME/software/zynq_ps/petalinux/subsystems/linux/configs/device-tree/zynq-zc706-pcie-trd-qspi.dts
         $ZYNQ_TRD_HOME/software/zynq_ps/petalinux/subsystems/linux/configs/device-tree/system-top.dts
b) Change boot mode to QSPI.
bash> petalinux-build -c device-tree -x distclean
bash> petalinux-build -c device-tree
bash> cd $ZYNQ_TRD_HOME/software/zynq_ps/petalinux/images/linux
bash> cp system.dtb $ZYNQ_TRD_HOME/build/ready_to_test/prog_qspi/devicetree_qspi.dtb
bash> petalinux-package --boot --fsbl $ZYNQ_TRD_HOME/software/zynq_ps/petalinux/images/linux/zynq_fsbl.elf --fpga $ZYNQ_TRD_HOME/software/zynq_ps/petalinux/subsystems/linux/hw-description/z7_pcie_trd_top.bit --uboot -o zc706_pcie_trd.bin
bash> cp zc706_pcie_trd.bin $ZYNQ_TRD_HOME/build/ready_to_test/prog_qspi
8 Building the Zynq SW application (XSDK)

This section demonstrates building of TRD applications using Xilinx software development kit (XSDK) .
NOTE: It is an optional step and recommended for advance users who are not using petalinux SDK and may wish to modify application/build and deploy it on target .

Launch Xilinx SDK:
  • On Linux, enter xsdk at the command prompt.

In the Workspace Launcher window, click Browse and navigate to $ZYNQ_TRD_HOME/software/zynq_ps/xsdk, then click OK.
Close the welcome screen.

8.1 Import project

To import the performance library (perfmon_lib), video library (video_lib) and QT application (zynq_pcie_qt) into the SDK workspace.

  • Select File > Import.
  • In the Import wizard, expand the General folder, select Existing Projects into Workspace, and click Next. All projects are located at the top-level inside your SDK workspace.
  • Click Browse and navigate to $ZYNQ_TRD_HOME/software/zynq_ps/xsdk. Press OK.
  • Make sure the filter_lib, perfmon_lib, video_lib and zynq_pcie_qt are checked . Press Finish.
screen_2014_4_a.png


8.2 Build project

  • Right click on filter_lib project ,and select Clean Project followed by Build Project.
  • Right click on perfmon_lib project, and select Clean Project followed by Build Project.
  • Right click on video_lib project, and select Clean Project followed by Build Project.

INFO: Please ensure $ZYNQ_QT_INSTALL environment variable is set , before building QT application.
  • Right click on zynq_pcie_qt project ,and select Clean Project followed by Build Project. Since video QT project adds "video_lib" , "filter_lib" and "perfmon_lib" as a project references , it is automatically selected and build by XSDK.

8.3 Deploy applications


Copy the newly build application zynq_pcie_qt\zynq_pcie_qt on SD card and execute.
Refer section 11, and use the newly build application inplace of sd_image\zynq_pcie_qt.


9 Building the PCIe host SW application

The software application compilation procedure is provided here.
IMPORTANT: The traffic generator needs the C++ compiler which is not shipped with the Fedora 16 live
CD. Likewise, Java compilation tools are not shipped as part of the Fedora 16 live CD. Hence, GUI compilation will need additional installations.
The source code is provided for an end user to build upon this design; for TRD testing recompilation of application or GUI is not recommended.

Steps for building the PCIe host software application

The source code (threads.cpp) for the Zynq-7000 PCIe TRD is available under the directory:
$ZYNQ_TRD_HOME/software/host/gui/jnilib/src

User can add debug messages or enable log verbose to aid in debug.
IMPORTANT: Changes in data structure will also lead to changes in the GUI compilation, which is not recommended.
To compile the traffic generator application:
1. Open a terminal window.
2. Navigate to the $ZYNQ_TRD_HOME/software/host/gui/jnilib/src directory and execute ./genlib.sh
Shared object (.so) files are generated and copied into 32 or 64 directories present in the jnilib directory, according to the type of Operating System being used.
TIP: Enable verbose messages by adding the -DDEBUG_VERBOSE flag to genlib.sh.

10 Running Demo Applications

This section explains through step by step instructions how to bring up the ZC706 board for video demonstration part of the TRD and running different video demonstrations out of the box.

The ZC706 Evaluation Kit comes with an SD-MMC card pre-loaded with binaries that enable the user to run the video demonstration and software applications. It also includes the binaries necessary to configure and boot the Zynq-7000 SoC based ZC706 board.

Note:
If the evaluation kit design files were downloaded online, copy the entire folder ZYNQ_TRD_HOME/ready_to_test/prog_qspi from the package onto the primary partition of the SD-MMC card (which is formatted as FAT32) using a SD-MMC card reader for loading the QSPI device with boot image and Linux kernel image. Once QSPI programming is over, load ZYNQ_TRD_HOME/ready_to_test/sd_image content onto the primary partition of the SD-MMC card (which is formatted as FAT32) using a SD-MMC card reader.
Petalinux console Login Details:-
User : root
Password : root

10.1 NOTE ON QSPI PROGRAMMING

Please note below precaution while programming QSPI.
During prog_qspi images boot, the init.sh script programs the zc706_pcie_trd_bin file in mtd3 (QSPI partition 3).
The mtd3 partition is located at non-zero location (multiple of 32KB). So that BOOTROM can locate the bin image.
 
However during QSPI boot, if BOOTROM finds any old images at any 32KB boundary before the mtd3 partition, then zc706_pcie_trd.bin will not be executed.
This is an issue!!
 
To avoid this, the QSPI locations should be erased till the begin of mtd3 partition.
This should be done once, on the given board (so that QSPI on this board is clean).
Steps to erase the QSPI locations.
 
1. Boot the board
2. Stop at uboot (press any key at the count-down of uboot)
3. Enter below commands at the uboot prompt
   zynq-uboot> sf probe 0 0 0
   zynq-uboot> sf erase 0 0x200000     /* assume mtd3 partition starts at 0x200000 */
4. Now, boot normally.
5. Copy the images (init.sh script does it all).
 

10.2 Hardware Setup Requirements

The ZC706 board setup to run & test the video demonstration applications require the following items:
Requirements for TRD Linux application demo setup
  • The ZC706 evaluation board with the XC7Z045 FFG900-2 part
  • Reference Design zip file containing BOOT.BIN and Petalinux FIT image. Host system driver and GUI files
  • A control PC with ISE design Suite Logic Edition Tools v2014.3
  • A control PC with teraTerm pro (or similar) terminal program
  • A control PC with USB-UART driver from silicon labs installed
  • Mini USB cable
  • USB mouse (not included with the kit)
  • Class 4 equivalent SD card
  • Monitor supporting 1080p
  • HDMI cable
  • 4-pin to 6-pin PCIe adapter cable
  • Fedora 16 LiveCD
  • PC with PCIe v2.0 slot. Recommended PCI Express Gen2 PC system motherboards are ASUS P5E (Intel X38), ASUS Rampage II Gene (Intel X58) and Intel DX58SO (Intel X58). Note the Intel X58 chipsets tend to show higher performance. This PC could also have Fedora Core 16 Linux OS installed on it.
Please note that two PCs are required for running this TRD. A control PC which is connected to the zc706 board via mini-USB running teraTerm and a host PC running Fedora Core 16 Linux connected to zc706 via PCIe slot on the motherboard.
Note: The example mentioned in this package has been tested with Ivy Bridge and Sandy Bridge PCIe Host machine and a Dell model #P2412H display monitor. However, the example should work well with any HDMI-compatible output device.

10.3 Running the Host GUI and Qt-based GUI Application on Zynq PS

For running the Host GUI and QT-based application on Zynq PS, please refer to Zynq-7000 SoC ZC706 Evaluation Kit UG961.

11 References