ZCU104 board¶

The ZCU104 board enables testing DDR4 SO-DIMM modules. It features a Zynq UltraScale+ MPSoC device consisting of PS (Processing System with quad-core ARM Cortex-A53) and PL (programmable logic).

On the ZCU104 board the Ethernet PHY is connected to PS instead of PL. For this reason it is necessary to route the Ethernet/EtherBone traffic through PC <-> PS <-> PL. To do this, a simple EtherBone server has been implemented (the source code can be found in the firmware/zcu104/etherbone/ directory).

The following instructions show how to set up the board for the first time.

Board configuration¶

To make the ZCU104 boot from the SD card it is necessary to ensure proper switch configuration. The mode switch (SW6) consisting of 4 switches is located near the FMC LPC Connector (J5) (the same side of the board as USB, HDMI, Ethernet). For details, see “ZCU104 Evaluation Board User Guide (UG1267)”. To use an SD card, configure the switches as follows:

Preparing SD card¶

For easiest setup, get the pre-built SD card image zcu104.img from github releases. It has to be loaded to the microSD card. To load it to the SD card, insert the card into your PC card slot and find the device name. lsblk can be used to check available devices. Example output can look like:

$ lsblk
NAME        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda           8:0    0 931.5G  0 disk
└─sda1        8:1    0 931.5G  0 part /data
sdb           8:16   1  14.8G  0 disk
├─sdb1        8:17   1   128M  0 part /run/media/BOOT
└─sdb2        8:18   1   128M  0 part /run/media/rootfs
nvme0n1     259:0    0 476.9G  0 disk
├─nvme0n1p1 259:1    0   512M  0 part /boot
└─nvme0n1p2 259:2    0 476.4G  0 part /

In the example output the SD card is sdb with two partitions sdb1 and sdb2.

Warning

Make sure that you have selected the proper device name or you may damage the hard drive on your system! Check the device SIZE to match the size of your microSD card and compare the outputs of the lsblk command with and without the SD card inserted.

Make sure to unmount all partitions on the card before loading the image. For example, assuming the SD card is /dev/sdb (device is without a number), use sudo umount /dev/sdb1 /dev/sdb2 to unmount its partitions.

To load the image, use the following command, replacing <DEVICE> according to the output of lsblk (in the example above it would be /dev/sdb):

sudo dd status=progress oflag=sync bs=4M if=zcu104.img of=<DEVICE>

Now, the microSD card should be ready to use. If it has been loaded successfully, you should be able to mount the two partitions (BOOT and rootfs) on your PC and browse the files. First, check if your system hasn’t auto-mounted the partitions. If not, then you can for example use:

sudo mkdir -p /mnt/boot /mnt/rootfs
sudo mount /dev/sdb1 /mnt/boot
sudo mount /dev/sdb2 /mnt/rootfs

Loading bitstream¶

Instead of loading the bitstream through the JTAG interface, it must be copied to the microSD card BOOT partition (FAT32). The bitstream will be then loaded by the bootloader during system startup.

The prebuilt card image comes with a sample bitstream, but in order to use the provided rowhammer Python scripts, a fresh bitstream must be created. Build the bitstream as you would do for other boards:

export TARGET=zcu104
export IP_ADDRESS=...  # optional
make build

Note

You only need to export IP_ADDRESS if you want to use a different address than the default one (see Network setup).

The resulting bitstream file will be located in build/zcu104/gateware/zcu104.bit. Copy it to the BOOT partition (FAT32) of the microSD card. Make sure it is named zcu104.bit.

When the SD card is ready, insert it into the microSD card slot on your ZCU104 board and power it on.

Verifying if bitstream is loaded¶

The first indication that the bitstream has been loaded successfully are the onboard LEDs. When the board is powered up, the LED will be red, which then turns green if the bitstream is loaded successfully. The ZCU104 bitstream will also make the four LEDs near the user buttons turn on and off in a circular pattern.

_images/zcu104_loaded.jpg — Fig. 2 The state when the bitstream has been loaded successfully.¶

The serial console over USB can be used to further check if the board is working correctly.

ZCU104 microUSB¶

ZCU104 has a microUSB port connected to the FTDI chip. It provides 4 channels, which are connected as follows:

Channel A is configured to support the JTAG chain.
Channel B implements UART0 MIO18/19 connections.
Channel C implements UART1 MIO20/21 connections.
Channel D implements UART2 PL-side bank 28 4-wire interface.

In general they should show up as subsequent /dev/ttyUSBx devices (0-3 if no others were present). Channel B is connected to the console in the PS Linux system.

To login to the board connect the microUSB cable to the PC and find Channel B among the ttyUSB devices in your system. If there are only ttyUSB0 to ttyUSB3 then Channel B will be ttyUSB1.

Then use e.g. picocom or minicom to open the serial console. Install one of them if not already installed on your system. With picocom use the following command (may require sudo):

picocom -b 115200 /dev/ttyUSB1

Then press enter, and when you see the following prompt,

Welcome to Buildroot
buildroot login:

use login root and empty password. You can set up a password if needed.

Network setup¶

Connect the ZCU104 board to your local network (or directly to a PC) using an Ethernet cable.

The board uses a static IP address. By default it will be 192.168.100.50. If it does not conflict with your local network configuration you can skip this section. (the default configuration can be found here).

To verify connectivity, use ping 192.168.100.50. You should see data being transmitted, e.g.

$ ping 192.168.100.50
PING 192.168.100.50 (192.168.100.50) 56(84) bytes of data.
64 bytes from 192.168.100.50: icmp_seq=1 ttl=64 time=0.332 ms
64 bytes from 192.168.100.50: icmp_seq=2 ttl=64 time=0.072 ms
64 bytes from 192.168.100.50: icmp_seq=3 ttl=64 time=0.081 ms

Modifying network configuration¶

If you need to modify the configuration, edit file /etc/network/interfaces. The Linux rootfs on the SD card is fairly minimal, so there is only the vi editor available. You could also just mount the card on your PC and edit the file.

After changing the configuration, reboot the board (type reboot in the serial console) and test if you can ping it with ping <NEW_IP_ADDRESS>.

SSH access¶

These instructions are optional but can be useful for more convenient updates of the bitstream (no need to remove the SD card from the slot on ZCU104).

Note

SSH on the board is configured to allow passwordless root access for simplicity but if that poses a security risk, modify /etc/ssh/sshd_config according to your needs and add a password for root.

You can login over SSH using (replace the IP address if you modified board network configuration):

ssh root@192.168.100.50

To access the boot partition, first mount it with:

mount /dev/mmcblk0p1 /boot

This can be automated by adding the following entry in /etc/fstab:

/dev/mmcblk0p1 /boot            vfat    rw              0       2

When the boot partition is mounted, you can use scp to load the new bitstream, e.g.

scp build/zcu104/gateware/zcu104.bit root@192.168.100.50:/boot/zcu104.bit

Then use the reboot command to restart the board.

Controlling the board¶

When the setup has been finished the board can be controlled as any other board. Just make sure to use export TARGET=zcu104 before using the scripts (and export IP_ADDRESS=... if you modified the network configuration).

ZCU104 SD card image¶

The easiest way is to use the prebuilt SD card image, as described in Preparing SD card section. However, it is possible to build the image from source, if needed.

The SD card image consists of boot partition and rootfs. Currently, only rootfs is built using buildroot. The boot partition content has to be built manually.

In the future, the buildroot configuration should be revised to build all the required software. Initial configuration has been included but it is still a work in progress and does not boot.

Bootloaders & kernel¶

Currently, we are using Xilinx FSBL, but it should be possible to use U-Boot SPL (link1, link2).

FSBL and PMU firmware can be built with following the steps:

Create a project from the Vivado example project “Base Zynq UltraScale+ MPSoC” for ZCU104 eval board. Open the PS IP configurator and add the following: * PS-PL Interfaces -> AXI HPM0 FPD (32-bit), AXI HPM1 FPD (32-bit) * disable Carrier Detect in Memory Interfaces -> SD -> SD 0

The following script can be used to generate FSBL, PMU firmware and Device Tree:

#!/usr/bin/tclsh

set hwdsgn  [open_hw_design PATH/TO/Base_Zynq_MPSoC_wrapper.hdf]

generate_app -hw $hwdsgn -os standalone -proc psu_cortexa53_0 -app zynqmp_fsbl -compile -sw fsbl -dir ./fsbl/
generate_app -hw $hwdsgn -os standalone -proc psu_pmu_0 -app zynqmp_pmufw -compile -sw pmufw -dir ./pmufw

set_repo_path PATH/TO/device-tree-xlnx
create_sw_design device-tree -os device_tree -proc psu_cortexa53_0
generate_target -dir dts

close_hw_design [current_hw_design]

The Device Tree generated by Vivado is missing the ethernet-phy node. Modify `pcw.dtsi` as follows:

&gem3 {
    phy-mode = "rgmii-id";
    status = "okay";
    xlnx,ptp-enet-clock = <0x0>;
    phy0: phy@c {
        reg = <0xc>;
        ti,rx-internal-delay = <0x8>;
        ti,tx-internal-delay = <0xa>;
        ti,fifo-depth = <0x1>;
        ti,rxctrl-strap-worka;
    };
};

Then generate the Device Tree Blob in the dts directory:

gcc -I include -I . -E -nostdinc -undef -D__DTS__ -x assembler-with-cpp -o system.dts system-top.dts
dtc -I dts -O dtb -o system.dtb system.dts

Build the rest of the required components:

ARM trusted firmware: (https://github.com/Xilinx/arm-trusted-firmware.git) e6eea88b14aaf456c49f9c7e6747584224648cb9 (tag: xlnx_rebase_v2.2)
U-Boot: (https://github.com/Xilinx/u-boot-xlnx.git) d8fc4b3b70bccf1577dab69f6ddfd4ada9a93bac (tag: xilinx-v2018.3)
Linux kernel: (https://github.com/Xilinx/linux-xlnx.git) 22b71b41620dac13c69267d2b7898ebfb14c954e (tag: xlnx_rebase_v5.4_2020.1)

Note

It may be necessary to apply the patches from firmware/zcu104/buildroot/board/zynqmp/patches when building U-Boot/Linux.

When building U-Boot make sure to update its configuration (u-boot-xlnx/.config) with the following options:

CONFIG_USE_BOOTARGS=y
CONFIG_BOOTARGS="earlycon clk_ignore_unused console=ttyPS0,115200 root=/dev/mmcblk0p2 rootwait rw earlyprintk rootfstype=ext4"
CONFIG_USE_BOOTCOMMAND=y
CONFIG_BOOTCOMMAND="load mmc 0:1 0x2000000 zcu104.bit; fpga load 0 0x2000000 $filesize; load mmc 0:1 0x2000000 system.dtb; load mmc 0:1 0x3000000 Image; booti 0x3000000 - 0x2000000"

These configure U-Boot to load the bitstream from the SD card and then start the system. Unfolding CONFIG_BOOTCOMMAND we can see:

load mmc 0:1 0x2000000 zcu104.bit
fpga load 0 0x2000000 $filesize
load mmc 0:1 0x2000000 system.dtb
load mmc 0:1 0x3000000 Image
booti 0x3000000 - 0x2000000

Example of building ARM Trusted firmware:

make distclean
make -j`nproc` PLAT=zynqmp RESET_TO_BL31=1

Example of building U-Boot:

make -j`nproc` distclean
make xilinx_zynqmp_zcu104_revC_defconfig
# now modify .config directly or using `make menuconfig` as described earlier
make -j`nproc`

Example of building Linux:

make -j`nproc` ARCH=arm64 distclean
make ARCH=arm64 xilinx_zynqmp_defconfig
# optional `make menuconfig`
make -j`nproc` ARCH=arm64 dtbs
make -j`nproc` ARCH=arm64

Then download zynq-mkbootimage and prepare the following boot.bif file:

image:
{
    [fsbl_config] a53_x64
    [bootloader] fsbl.elf
    [pmufw_image] pmufw.elf
    [, destination_cpu=a53-0, exception_level=el-2] bl31.elf
    [, destination_cpu=a53-0, exception_level=el-2] u-boot.elf
}

Open a terminal and make sure that the filepaths specified in boot.bif are correct. Then use``mkbootimage –zynqmp boot.bif boot.bin`` to create the boot.bin file.

Root filesystem¶

Download buildroot

git clone git://git.buildroot.net/buildroot
git checkout 2020.08.2

Note

As of time of writing git checkout f45925a951318e9e53bead80b363e004301adc6f was required to avoid fakeroot errors when building.

Then prepare configuration using external sources and build everything:

make BR2_EXTERNAL=/PATH/TO/REPO/rowhammer-tester/firmware/zcu104/buildroot zynqmp_zcu104_defconfig
make -j`nproc`

Flashing SD card¶

One can use fdisk to directly partition the SD card /dev/xxx or use the provided genimage configuration to create an SD card image that can be then directly copied to the SD card. The second method is usually more convenient.

Formatting SD card manually¶

Use fdisk or other tool to partition the SD card. The recommended partitioning scheme is as follows:

Partition 1, FAT32, 128M
Partition 2, ext4, 128M

Then create the filesystems:

sudo mkfs.fat -F 32 -n BOOT /dev/OUR_SD_CARD_PARTITION_1
sudo mkfs.ext4 -L rootfs /dev/OUR_SD_CARD_PARTITION_2

Write the rootfs:

sudo dd status=progress oflag=sync bs=4M if=/PATH/TO/BUILDROOT/output/images/rootfs.ext4 of=/dev/OUR_SD_CARD_PARTITION_2

Mount the boot partition and copy the boot files and kernel image created earlier and the ZCU104 bitstream:

cp boot.bin /MOUNT/POINT/BOOT/
cp /PATH/TO/rowhammer-tester/build/zcu104/gateware/zcu104.bit /MOUNT/POINT/BOOT/
cp /PATH/TO/linux-xlnx/arch/arm64/boot/Image /MOUNT/POINT/BOOT/
cp /PATH/TO/linux-xlnx/arch/arm64/boot/dts/xilinx/zynqmp-zcu104-revA.dtb /MOUNT/POINT/BOOT/system.dtb

Note: make sure to name the device tree blob system.dtb for the U-Boot to be able to find it (as shown in above commands).

Using genimage¶

By default ZCU104 buildroot configuration will also build the genimage tool for the host system. Image configuration is described in the firmware/zcu104/image.cfg file. There is also a script firmware/zcu104/genimage.sh for convenience. Run it without arguments to get help. Then run it providing correct paths to all the required files to generate the zcu104.img file.

The image can be then copied to the SD card device (not partition! so e.g. /dev/sdb, not /dev/sdb1) using dd:

sudo dd status=progress oflag=sync bs=4M if=/PATH/TO/zcu104.img of=/dev/OUR_SD_CARD

Last update: 2023-04-07