Design Overview

This module shows how to add two image-processing filters between the capture and the display. The 2D filter and dense optical flow algorithm are implemented in hardware using the hardware-optimized xfopencv libaries.


Design Components

This module requires the following components:
  • zcu102_base_trd (SDSoC)
  • filter2d_optflow (accelerators)
  • sdxfilter2d (gstreamer plugin)
  • sdxopticalflow (gstreamer plugin)
  • video_qt2
    • video_lib
    • gst_lib

Build Flow Tutorials

2D Filter and Optical Flow Combined Sample and GStreamer Plugins

This tutorial shows how to build the hls_video version of filter2d combined with the xfopencv dense optical flow algorithm sample based on the Base TRD SDSoC platform.

  • Set the build environment variables. This requires that you have previously completed the PetaLinux SDK installation step from DM5.
    Note 1: Make sure you set the env variables in the same shell that is used to launch SDx. Also make sure the env variables are set before starting SDx, otherwise close and re-start SDx.
    Note 2: The below command might ask you execute unset LD_LIBRARY_PATH and then re-execute the command. Go ahead and follow those steps.
    % source $TRD_HOME/petalinux/sdk/environment-setup-aarch64-xilinx-linux
  • Open the existing SDx workspace from design module 8 using the SDx tool.
    % cd $TRD_HOME/workspaces/ws_f2dof
    % sdx -workspace . &
  • Click 'Import Project' from the welcome screen, select "Eclipse workspace or zip file" as Import type, browse to the current working directory and make sure all projects are selected as shown in the figure. Click finish.
  • Click 'File -> New SDx Project'. Select 'Application Project' as project type and click 'Next'. Enter filter2d_optflow as project name and click 'Next'.
  • Click 'Add Custom Platform', browse to the $TRD_HOME/zcu102_base_trd folder and click OK. Select the "zcu102_base_trd [custom]" platform from the list and click 'Next'.
  • Select 'Shared Library' as output type and click 'Next'.
  • Select 'Filter 2D and Optical Flow' template and click 'Finish'.
  • In the "Application Project Settings" panel, set the active build configuration to "Release". Make sure the 'Generate SD card image' box is checked. Make sure the pre-selected Hardware Functions are marked for acceleration as shown in the figure.
  • Right-click the filter2d_optflow project in the explorer pane and select 'Build Project'. This step can take several hours. Copy the generated SD card image.
    % cp -rf filter2d_optflow/Release/sd_card $TRD_HOME/sd_card/dm9
  • When the build is finished, right-click the gstsdxfilter2d project and select "Build Project". Then right-click the gstsdxopticalflow project and select "Build Project". Copy the generated libraries and plugins.
    Note: These projects have a dependency on filter2d_optflow and the builds will fail if filter2d_optflow was not built previously.
    % mkdir $TRD_HOME/sd_card/dm9/lib $TRD_HOME/sd_card/dm9/gstreamer-1.0
    % cp gst/allocators/Debug/ gst/base/Debug/ $TRD_HOME/sd_card/dm9/lib
    % cp gst/plugins/filter2d/Debug/ gst/plugins/optical_flow/Debug/ $TRD_HOME/sd_card/dm9/gstreamer-1.0

Video Qt Application

This tutorial shows how to build the video library and the video Qt application

  • Make sure the build environment is set (see DM5 tutorial). Generate a Makefile for the Qt project for DM9.
    % cd $TRD_HOME/workspaces/ws_video/video_qt2
    % qmake -r -spec linux-oe-g++
  • If not already open from a previous module, browse to the Qt workspace and open it.
    % cd ..
    % sdx -workspace . &
  • Right-click the video_qt2 project and click 'Build Project'.
  • Copy the generated video_qt2 executable to the dm9 SD card directory.
    % cp $TRD_HOME/workspaces/ws_video/video_qt2/video_qt2 $TRD_HOME/sd_card/dm9

Run Flow Tutorial

  • See here for board setup instructions.
  • Copy all the files from the $TRD_HOME/sd_card/dm9 SD card directory to a FAT formatted SD card.
  • Power on the board to boot the images; make sure INIT_B, done and all power rail LEDs are lit green.
  • After ~30 seconds, the display will turn on and the application will start automatically, targeting the max supported resolution of the monitor (one of 3840x2160 or 1920x1080 or 1280x720). The application will detect whether DP Tx or HDMI Tx is connected and output on the corresponding display device.
  • To re-start the TRD application with the max supported resolution, run
  • To re-start the TRD application with a specific supported resolution use the -r switch e.g. for 1920x1080, run
    % -r 1920x1080
  • The user can now control the application from the GUI's control bar (bottom) displayed on the monitor.
  • The user can select from the following video source options:
    • TPG (SW): virtual video device that emulates a USB webcam purely in software
    • USB: USB Webcam using the universal video class (UVC) driver
    • TPG (PL): Test Pattern Generator implemented in the PL
    • HDMI: HDMI input implemented in the PL
    • CSI: MIPI CSI image sensor pipeline in the PL
    • File: Raw video file source
  • The user can select from the following accelerator options:
    • Passthrough (no accelerator)
    • 2D convolution filter with configurable coefficients
    • Dense optical flow algorithm (shown in figure)
  • The supported accelerator modes depend on the selected filter:
    • SW - accelerator is run on A53 (filter2d only)
    • HW - accelerator is run on PL
  • The video info panel (top left) shows essential settings/statistics.
  • The CPU utilization graph (top right) shows CPU load for each of the four A53 cores.

Continue with Design Module 10.
Return to the Design Tutorials Overview.