Zynq UltraScale+ MPSoC - Power States and Transition Times

This tutorial explains procedure to measure transition times and respected power values when either PS or PL suspends or wake up. By following below procedure
Suspend and wake up time of APU, RPU,PL can be measured.

These procedures are for 2017.1 and later releases.

1. Create petalinux project

Run below commands from bash terminal to create project petalinux.
source <petalinux-install-dir>/settings.sh
petalinux-create -t project -s /proj/petalinux/petalinux-v2017.3_bsps_daily_latest/xilinx-zcu102-v2017.3-final.bsp

2. Apply Linux kernel patch to signal wakeup

  • Create directory <plnx-proj-root>/project-spec/meta-user/recipes-kernel/linux/files/ (if not present)
  • Copy patch file to <plnx-proj-root>/project-spec/meta-user/recipes-kernel/linux/files/ (Patch attached here: pm_wakeup_latency.patch)

  • Open <plnx-proj-root>/project-spec/meta-user/recipes-kernel/linux/linux-xlnx_%.bbappend (create if not present)
  • Add below line in file:
FILESEXTRAPATHS_prepend := "${THISDIR}/files:"
SRC_URI_append = "file://pm_wakeup_latency.patch"

3. Minimum kernel configuration

follow below steps if latency is to be measured with minimum kernel configuration:
  • Copy kernel min config file (defconfig) to project-spec/meta-user/recipes-kernel/linux/files/defconfig

  • Add below line in project-spec/meta-user/recipes-kernel/linux/linux-xlnx_%.bbappend file:
SRC_URI_append = " file://defconfig"

4. Build petalinux

petalinux-build -c kernel

5. Run Linux and RPU app

5. 1. Boot Linux using petalinux-boot


Boot Linux on APU using SD Card by coping BOOT.BIN and image.ub on the SD Card.

Once boot start U-boot prompt will appear on serial. Press Enter again and again to interrupt u-boot till ZynqMP prompt appear. Run below command from u-boot to disable cpuidle.
ZynqMP > setenv bootargs ${bootargs} cpuidle.off=1

5.2. Run RPU app

If XSDB method is used to start linux, this step should be skipped.
  • Load RPU application using xsdb (RPU application (rpu_deep_sleep.elf) when Linux is up)


Perform following command on xsdb
xsdb% connect
xsdb% target -set -filter {name =~ "Cortex-R5 #0"}
xsdb% rst -c
xsdb% exec sleep 2
xsdb% dow ./rpu_deep_sleep.elf
xsdb% mask_write 0xff9a0000 0x8 0x8
xsdb% con

6. Measure Power values

Use Power Advantage tool for measuring power values.

7. All on (PS + PL)

Run below command to load all APU from Linux terminal on board
root@plnx_aarch64:~# fulload() { dd if=/dev/zero of=/dev/null | dd if=/dev/zero of=/dev/null | dd if=/dev/zero of=/dev/null | dd if=/dev/zero of=/dev/null & };fulload; read;
Use Power Advantage tool for measuring power values. That will give power values for All on (PS+PL)

8. PS all ON

8.1 Turn off PL

Run below command from Linux terminal on board to power off PL (force power down unused PU).
root@plnx_aarch64:~# echo request_wakeup 8 1 0 1 > /sys/kernel/debug/zynqmp_pm/power
root@plnx_aarch64:~# echo force_powerdown 8 >  /sys/kernel/debug/zynqmp_pm/power
root@plnx_aarch64:~# time echo release_node 69 > /sys/kernel/debug/zynqmp_pm/power

8.2 Turn ON PL

Run below command from Linux terminal on board to power off PL (force power down unused PU).
root@plnx_aarch64:~# echo request_wakeup 8 1 0 1 > /sys/kernel/debug/zynqmp_pm/power
root@plnx_aarch64:~# echo force_powerdown 8 >  /sys/kernel/debug/zynqmp_pm/power
root@plnx_aarch64:~# time echo request_node 69 > /sys/kernel/debug/zynqmp_pm/power

9. 3 APUs on/off

9.1 Power OFF 3 APUs

Run below command from Linux terminal on board to power off 3 APUs.
root@plnx_aarch64:~# time echo 0 > /sys/devices/system/cpu/cpu1/online
root@plnx_aarch64:~# time echo 0 > /sys/devices/system/cpu/cpu2/online
root@plnx_aarch64:~# time echo 0 > /sys/devices/system/cpu/cpu3/online

9.2 Power ON 3 APUs

Run below command from Linux terminal on board to power off 3 APUs.
root@plnx_aarch64:~# time echo 1 > /sys/devices/system/cpu/cpu1/online
root@plnx_aarch64:~# time echo 1 > /sys/devices/system/cpu/cpu2/online
root@plnx_aarch64:~# time echo 1 > /sys/devices/system/cpu/cpu3/online

10. Frequency scaling (min freuquency)

Run following command to get available frequencies.
root@plnx_aarch64:~# cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_available_frequencies

10.1 Set minimum frequency

Run following command to set minimum frequency
root@plnx_aarch64:~# time echo <min freq. from above command> > /sys/devices/system/cpu/cpu0/cpufreq/scaling_setspeed

10.2 Set maximum frequency

Run following command to set maximum frequency

root@plnx_aarch64:~# time echo <max freq. from above command> > /sys/devices/system/cpu/cpu0/cpufreq/scaling_setspeed

11. Suspend to RAM and FPD off

Run following command from Linux terminal to suspend Linux(with FPD on):
root@plnx_aarch64:~# echo 0 > /sys/module/printk/parameters/console_suspend
root@plnx_aarch64:~# echo MMIO_WRITE 0xFFD80064 1 1 > /sys/kernel/debug/zynqmp_pm/power
Now, Linux will be suspended and it will be resumed by RPU after suspend. RPU has requested NODE_SATA from FPD, so Linux will be suspended with FPD on.

After 10 second of Linux suspend, RPU application will release NODE_SATA. Once NODE_SATA is released, FPD will be off. Use Power Advantage Tool in these 10 seconds to get power value for Suspend to RAM.

Following are two print messages to identify these 10 second window.
RPU0: Suspended to RAM, FPD is on
(10 seconds delay)
RPU0: Release NODE_SATA, FPD will be off after releasing NODE_SATA

Once FPD is off, RPU will resume APU after 10 second of FPD off. To measure power with FPD off, Use Power Advantage Tool in these 10 seconds.
Following are two print messages to identify these 10 second window.

RPU0: Suspended.
(10 seconds delay)
RPU0: Resuming APU.

On console, you should able to see suspend/wakeup latency of APU from RPU print message.
Note: Above loop will run for 5 times and print minimum, maximum and average values for latency.

12. R5 Sleep and Deep-Sleep mode


Once above measurements are done, RPU puts FPD in off state and goes to idle mode (wfi state) for 10 seconds.

To measure power with R5 sleep mode, Use Power Advantage Tool in these 10 seconds.

Following are two print messages to identify these 10 second window.
RPU0: Go to RPU Sleep mode
(10 seconds delay)
RPU0: Go to deep sleep

After 10 seconds RPU gets timer interrupt after that it suspend itself and goes to deep sleep mode for 10 seconds.
After RPU goes to deep sleep, Use Power Advantage tool for measuring power values within these 10 seconds.
Following are two print messages to identify these 10 second window.
RPU0: Go to deep sleep
(10 seconds delay)
RPU0: Running in Split mode

After 10 seconds timer will generate interrupt and Resumes RPU. Then RPU resumes APU. This state is same as 3 APU off with minimum frequency.

To measure Transition times for R5 Sleep to R5 Deep Sleep mode download baremetal.rar and follow the steps which are given in steps.txt

By following steps you can able to put R5 in sleep and deep sleep mode and you can measure transition times of R5 going to Deep sleep mode from Sleep mode.





Note: "time" command used in above steps, will give time to execute that command. "time" gives real, user and sys time values. "real" time is considered from time command.

13. Steps to Measure latency/power from left to right of dimmer mode table

  1. Perform steps 1 to 5 to start Linux and RPU application
  2. Perform step 7 to load APU
  3. Use Power Advantage Tool to measure power for ALL (PS+PL) on mode.
  4. Perform step 8.1 to turn off PL.
  5. Use Power Advantage Tool to measure power for PS all ON.
  6. Perform step 9.1 to turn of 3 APUs.
  7. Use Power Advantage Tool to measure power for APU(3 off) mode.
  8. Perform step 10.1 to frequency scaling(min frequency)
  9. Use Power Advantage Tool to measure power for APU(3 off) minimum frequency.
  10. Perform step 11 and 12 to measure power/latency for suspend to RAM, FPD off, RPU sleep, Deep sleep mode.

14. Steps to Measure latency/power from right to left of dimmer mode table

  1. Step 11 and 12 performed for above section will also measure values for right to left of dimmer table.
  2. Perform 10.2 to frequency scaling (max frequency) .
  3. Perform 9.2 to turn on all APUs.
  4. Perform 8.2 to turn on PL.

15. Example log

RPU0: Request Suspend Latency in useconds of Node NODE_APU_0:  Min: 4456, Max: 5301, Avg: 4774
RPU0: FPD off Latency in useconds of Node NODE_APU_0:  Min: 74, Max: 75, Avg: 74
RPU0: FPD on Latency in useconds of Node NODE_APU_0:  Min: 79444, Max: 79525, Avg: 79489
RPU0: APU0 Wakeup Latency in useconds of Node NODE_APU_0:  Min: 9083, Max: 9092, Avg: 9088
RPU0: Wakeup Latency in useconds of Node NODE_APU_0:  Min: 15163, Max: 121539, Avg: 48056

Here,
RPU0: Request Suspend Latency in useconds of Node NODE_APU_0: Latency for power state transition from "3 APU off (min frequency)" to "Suspend to RAM".

RPU0: FPD off Latency in useconds of Node NODE_APU_0: Latency for power state transition from "Suspend to RAM" to "FPD off".

RPU0: FPD on Latency in useconds of Node NODE_APU_0: Latency for power state transition from FPD off to "Suspend to RAM".

RPU0: APU0 Wakeup Latency in useconds of Node NODE_APU_0: Time require to wakeup APU_0 after RequestWakeup call.

RPU0: Wakeup Latency in useconds of Node NODE_APU_0: Latency for power state transition from "Suspend to RAM" to "3 APU off (min frequency).


16. Configuration Object

Configuration Object file Generated by PetaLinux tool chain and Vivado is attached below.


Subsystems -
1) Linux Subsystem
Master -
  1. APU
Slaves -
  1. DDR
  2. L2 Cache
  3. OCM Bank 0, 1, 2 and 3
  4. I2C0
  5. I2C1
  6. SD1
  7. QSPI
  8. PL

2) R5-0 Subsystem
Master -
  1. RPU0
Slaves -
  1. TCM Bank 0 - A
  2. TCM Bank 0 - B

3) R5-1 Subsystem
Master -
  1. RPU1
Slaves -
  1. TCM Bank 1 - A
  2. TCM Bank 1 - B



Related Links

1. Create petalinux projectConfiguration ObjectConfiguration Object file Generated by PetaLinux tool chain and Vivado is attached below.