Booting from NAND

Instead of booting into FEL mode and downloading U-Boot, Linux and the root file system every time, we want to load everything from the NAND and boot into Linux automatically when we turn on C.H.I.P.

When powered on, the Allwinner R8 SOC on C.H.I.P first executes code in the built-in Boot-ROM (BROM). At this time the 512 MB DRAM on C.H.I.P are not initialized and the R8 SOC can only work with the internal 48 KiB SRAM. The BROM code checks if the FEL pin is low (i.e. connected to GND) and if so goes to the FEL USB boot mode. With the R8 in FEL USB mode, we can interact with the SOC via the sunxi-fel tool as described various times in the previous chapters. If the FEL pin is high, the BROM code starts checking the various storage options (e.g. C.H.I.P's NAND) for a valid boot signature at the right location. If no boot signature is found the BROM code switches to FEL mode. If such a boot signature is found the code is loaded from the storage into the SRAM and executed. The maximum size of this early stage boot code is 0x7e00 bytes (31.5 KiB) on Allwinner R8/A13 SOCs.

The limited space in the SRAM is why we need the U-Boot secondary program loader (SPL). We've already used U-Boot SPL in the first chapters without explaining anything about it. Without going into detail, the following paragraph tries to gives a very high level (and probably too simplified) overview.

We used the sunxi-fel uboot command that expects a binary containing U-Boot including the SPL code. The SPL code is downloaded to the SRAM of the SOC and executed to initialize the DRAM. C.H.I.P's DRAM configuration is hard-coded in the SPL. After initializing the DRAM, the SOC goes back to FEL mode. Now U-Boot, Linux, the device tree and the root file filesystem can be downloaded to DRAM by the sunxi-fel tool. Next, U-Boot is executed and unpacks and starts the Linux kernel.

In this chapter we want to write the SPL code to the first blocks of the NAND where it can be picked up and executed by the BROM code. Once the DRAM is initialized, we want the SPL code to read the full U-Boot code from the NAND into DRAM and then hand over control to U-Boot. For that to work, not only the DRAM configuration but also the NAND configuration must be hardcoded into the SPL code. The DRAM and NAND parameters are specified in the the U-Boot configuration.

Different NAND types

NAND memory device technology is used inside of SD card, SSD's and EMMC chips which behave like block devices. These block devices abstract away many of the things necessary to operate the underlying raw NAND. C.H.I.P does not use SD cards or EMMC chips, but comes with a raw NAND chip. This requires special treatment in software to implement error correction and wear-leveling.

There are two different variants of C.H.I.P out in the wild - those with SK hynix H27UCG8T2ETR NAND modules and those with Toshiba TC58TEG5DCLTA00 NAND modules:

Both chips are Multi-level cell "MLC" NAND chips. Compared to Single-level cell "SLC" NAND, they store multiple bits per cell resulting in a higher data density. Unfortunately, that comes at a price - they are more sensitive to errors and have a lower endurance. MLC NAND is not supported by the UBI/UBIFS layers in mainline Linux. However, the original NextThingCo Linux 4.4 branches supported MLC NAND. Both NAND chips can also be operated in an SLC mode, where only one bit per cell is stored. Only half their capacity is available for data storage as a result. Here we focus on mainline Linux and thus use the NAND in SLC mode.

NAND consists of so called "erase-blocks", which are 4 MB in size for the SK hynix and the Toshiba NAND. Whenever data is written to the NAND, a whole erase-block is erased and re-written. Even when only a single bit has changed, it is necessary to write a full erase-block. Erase-blocks wear out with a growing number of write-cycles and can become unusable, so-called "bad blocks". Wear-leveling algorithms to miminimize wear-out and bad-block handling needs to be implemented in software.

Data is read in pages of 16 KB size in our case - same for both NAND types. Each erase block consists of 256 pages (4 MB / 16 KB = 256). Pages can also be divided in sub-pages, but that is not the case for the NAND used on C.H.I.P (page size = subpage size). For every page there is a so-called out-of-band (OOB) area where error-correction-code (ECC) data is stored. So, in reality the SK hynix NAND has a page size of 16384 + 1664 = 18048 bytes of which 16384 are available for data storage. Consequently, the erase blocks are really 4620288 bytes in size with 4194304 usable bytes. For Toshiba its the real page size is 16384 + 1280 = 17664 bytes and the real erase block size is 4521984 bytes. This becomes important when we want to read from raw NAND and prepare raw NAND images for flashing.

As shown above, the SK hynix and the Toshiba NAND used on C.H.I.P differ in their capacity and their OOB size. In the CHIP_defconfig for U-Boot in the previous chapter we have defined OOB size matching the SK hynix NAND module CONFIG_SYS_NAND_OOBSIZE=0x680 - as 0x680 converted to decimal is 1664. So far this setting did not matter, as we always loaded U-Boot and it's SPL part to C.H.I.P's RAM via the sunxi-fel tool. But as we want the U-Boot SPL to load the rest of U-Boot from the NAND we now have to define the correct OOB size.

So, when building the U-Boot SPL for a the Toshiba NAND, we need update the U-Boot nand.cfg configuration fragment:

sed -i -e 's/\(CONFIG_SYS_NAND_OOBSIZE\)=.*/\1=0x500/' \
${BR2_EXTERNAL}/board/nextthingco/CHIP/uboot/nand.cfg
cd ${BR_DIR}
make uboot-reconfigure

And when we want to switch back to SK hynix NAND do this:

sed -i -e 's/\(CONFIG_SYS_NAND_OOBSIZE\)=.*/\1=0x680/' \
${BR2_EXTERNAL}/board/nextthingco/CHIP/uboot/nand.cfg
make uboot-reconfigure

SPL NAND image

The BROM code only has a very basic NAND driver implementation that does not know about the parameters of the actual NAND chip. It tries to read the SPL code from page 0 on block 0 of the NAND. If it does not find valid boot code there it continues at page 64, page 128, and page 192 on erase block 0. Then it continues with pages 256, 320, 384 and 448 on erase block 1. For maximum robustness, in total eight copies of the SPL code can be written to the NAND.

As the BROM code does not know about the size of the NAND, for each page various formats are probed - more details are explained in the linux-sunxi.org wiki.

The sunxi-nand-image-builder tool takes the sunxi-spl.bin as input and writes an output file including scrambling and error correction codes that can be flashed onto the raw NAND and is recognized by the BROM code. Various formats can be selected - the most robust one only using 1024 bytes per page plus 64 bit for error correction.

Let's add the sunxi-tools (including the sunxi-nand-image-builder) to the Buildroot host tools and build them:

cat <<EOF >>"${BR2_EXTERNAL}"/configs/nextthingco_chip_defconfig
BR2_PACKAGE_HOST_SUNXI_TOOLS=y
EOF

cd ${BR_DIR}
make nextthingco_chip_defconfig
make host-sunxi-tools

Now we can create a BROM image for sk Hynix NAND:

cd ${BR_DIR}/output/images
${BR_DIR}/output/host/bin/sunxi-nand-image-builder -p 16384 -o 1280 -e 0x400000 -s -b -u 1024 -c 64/1024 sunxi-spl.bin sunxi-spl.bin.nand

This blows up the 16384 bytes sunxi-spl.bin into a 282624 bytes sunxi-spl.bin.nand file. That corresponds to 282624 / (16384 + 1280) = 16 raw nand pages.

We need to prepare an image for a full erase block, so we need to add 48 pages padding and then repeat that three times:

| Page |  0  | 16  | 32  | 48  |
| ---- | --- | --- | --- | --- |
|   0  | SPL | PAD | PAD | PAD |
|  64  | SPL | PAD | PAD | PAD |
| 128  | SPL | PAD | PAD | PAD |
| 192  | SPL | PAD | PAD | PAD |

Buildroot supports a so-called post-image script. The post-image script is executed after U-Boot, Linux and the rootfs have been built. Let's add a post-image.sh script to our Buildroot configuration:

cat <<EOF >>"${BR2_EXTERNAL}"/configs/nextthingco_chip_defconfig
BR2_ROOTFS_POST_IMAGE_SCRIPT="\${BR2_EXTERNAL_CHIP_PATH}/board/nextthingco/CHIP/post-image.sh"
EOF

cd ${BR_DIR}
make nextthingco_chip_defconfig

The following post-image.sh script takes the U-Boot sunxi-spl.bin as input a produces a sunxi-spl.bin.nand image with all the error correction, repetition of the SPL and padding:

cat <<EOF >${BR2_EXTERNAL}/board/nextthingco/CHIP/post-image.sh
#!/bin/bash

# Environment variables passed in from buildroot:
# BR2_CONFIG, HOST_DIR, STAGING_DIR, TARGET_DIR, BUILD_DIR, BINARIES_DIR and BASE_DIR.

echo "##############################################################################"
echo "## \$0 "
echo "##############################################################################"

echo "# \\\$1 = \$1"
echo "# \\\$2 = \$2"

IFS=", " read -r -a EXTRA_ARGS <<< "\$2"

echo "# BR2_CONFIG=\$BR2_CONFIG"
echo "# BR2_EXTERNAL=\$BR2_EXTERNAL"
echo "# HOST_DIR=\$HOST_DIR"
echo "# STAGING_DIR=\$STAGING_DIR"
echo "# TARGET_DIR=\$TARGET_DIR"
echo "# BUILD_DIR=\$BUILD_DIR"
echo "# BINARIES_DIR=\$BINARIES_DIR"
echo "# BASE_DIR=\$BASE_DIR"

ROOT_DIR="\${BR2_EXTERNAL_CHIP_PATH}"

# Read U-Boot version and config file from the Buildroot configuration
UBOOT_VER="\$(sed -n -e 's/BR2_TARGET_UBOOT_VERSION="\([^"]*\)"/\1/p' \$BR2_CONFIG)"
UBOOT_CFG="\${BUILD_DIR}/uboot-${UBOOT_VER}/.config"
[ ! -f "\${UBOOT_CFG}" ] && echo "ERROR: cannot find U-Boot config $UBOOT_CFG" && exit 1

# Read NAND parameters from the U-Boot configuration
BLOCK_SIZE="\$(sed -n -e 's/CONFIG_SYS_NAND_BLOCK_SIZE=\(.*\)/\1/p' \$UBOOT_CFG)"
PAGE_SIZE="\$(sed -n -e 's/CONFIG_SYS_NAND_PAGE_SIZE=\(.*\)/\1/p' \$UBOOT_CFG)"

OOB_SIZE_TOSHIBA=1280
OOB_SIZE_HYNIX=1664

INPUT_SPL="\${BINARIES_DIR}/sunxi-spl.bin"
OUTPUT_SPL="\${BINARIES_DIR}/sunxi-spl.bin.ecc"

OUTPUT_IMAGE_HYNIX="\${BINARIES_DIR}/sunxi-spl.bin.hynix.nand"
OUTPUT_IMAGE_TOSHIBA="\${BINARIES_DIR}/sunxi-spl.bin.toshiba.nand"

function create_image()
{
    OUTPUT_IMAGE="\${1}"
    OOB_SIZE="\${2}"
    rm -rf "\${OUTPUT_IMAGE}"
    for i in \$(seq 1 16)
    do
            \${HOST_DIR}/bin/sunxi-nand-image-builder -s -b -c 64/1024 -u 1024 -e \${BLOCK_SIZE} -p \${PAGE_SIZE} -o \${OOB_SIZE} \${INPUT_SPL} \${OUTPUT_SPL}
            cat "\${OUTPUT_SPL}" >> "\${OUTPUT_IMAGE}"
    done
}

create_image "\${OUTPUT_IMAGE_HYNIX}" "\${OOB_SIZE_HYNIX}"
create_image "\${OUTPUT_IMAGE_TOSHIBA}" "\${OOB_SIZE_TOSHIBA}"

EOF
chmod a+x ${BR2_EXTERNAL}/board/nextthingco/CHIP/post-image.sh

NOTE: instead of padding random data we simply generate BROM image 16 times. Each time the output is different, as the sunxi-nand-image-builder mixes in random data for robustnes on the NAND. So effectively the post image script creates a full erase block with the following content:

| Page |  0  | 16  | 32  | 48  |
| ---- | --- | --- | --- | --- |
|   0  | SPL | SPL | SPL | SPL |
|  64  | SPL | SPL | SPL | SPL |
| 128  | SPL | SPL | SPL | SPL |
| 192  | SPL | SPL | SPL | SPL |

The resulting ${BR_DIR}/output/images/sunxi-spl.bin.hynix.nand (or ${BR_DIR}/output/images/sunxi-spl.bin.toshiba.nand) can now be flashed to the first two erase blocks of the NAND.

Full U-Boot image

The SPL image is created using the sunxi-nand-image-builder which implements randomization and ECC necessary to achieve the greates possible reliablity. This is not necessary for the full U-Boot image as the NAND controller of the R8 / A13 SOC is taking care of that when writing the image. We can simply use the dd command to create an image:

dd if=${BR_DIR}/output/image/u-boot.bin of=${BR_DIR}/output/image/u-boot.bin.nand bs=${BLOCK_SIZE} conv=sync

Note, the resulting u-boot-bin.nand image is excactly 4 MB - the size of the NAND erase blocks.

For future use, also add this to our post-image.sh script:

# Read NAND parameters from the U-Boot configuration
cat <<EOF >>${BR2_EXTERNAL}/board/nextthingco/CHIP/post-image.sh
INPUT_IMAGE="\${BINARIES_DIR}/u-boot.bin"
OUTPUT_IMAGE="\${BINARIES_DIR}/u-boot.bin.nand"
dd if="\${INPUT_IMAGE}" of="\${OUTPUT_IMAGE}" bs=\$(printf "%d" \${BLOCK_SIZE}) conv=sync
EOF

Write SPL, U-Boot and Linux rootfs to the NAND

Type the following to download SPL, U-Boot, the Linux kernel, our DTB, the NAND images and the rootfs to C.H.I.P's DRAM and boot:

D=${BR_DIR}/output/images

sunxi-fel -v -p uboot ${D}/u-boot-sunxi-with-spl.bin \
                write 0x42000000 ${D}/zImage \
                write 0x43000000 ${D}/sun5i-r8-chip.dtb \
                write 0x43400000 ${D}/sunxi-spl.bin.hynix.nand \
                write 0x43800000 ${D}/u-boot.bin.nand \
                write 0x50000000 ${D}/rootfs.cpio.uboot

Via UART, interrupt U-Boot's auto-boot mechanism, and type the following commands in order to write the SPL and U-Boot images to the NAND and then boot into to the Linux Ramdisk:

nand erase.chip
nand write.raw.noverify 0x43400000 0x0 0x100
nand write.raw.noverify 0x43400000 0x400000 0x100
nand write 0x43800000 0x800000 0x400000
bootz 0x42000000 0x50000000 0x43000000

In Linux, run these commands to create the UBIFS on the NAND rootfs partition and then copy the rootfs from the ramdisk to the NAND.

mtdinfo
mtdinfo /dev/mtd0
flash_erase /dev/mtd4 0 2035
ubiformat -y /dev/mtd4
ubiattach -m 4                           # --> generates /dev/ubi0, also displays number of LEBs = e.g. 1952
ubimkvol /dev/ubi0 --name rootfs -S 1952 # --> creates /dev/ubi0_0
mkfs.ubifs /dev/ubi0_0                   # --> doesn't really create ubifs
mount -t ubifs /dev/ubi0_0 /mnt          # --> ubifs is created as part of mounting
cp -va /bin /boot /crond.reboot /dev /etc /init /lib /lib32 /linuxrc /media /opt /root /sbin /usr /var /mnt # --> copy stuff from ramdisk to nand
cd /mnt
mkdir mnt run proc sys tmp
cd /
umount /mnt
poweroff

Now, remove FEL pin and power on C.H.I.P. Then interrupt U-Boot auto-boot, and type the following to boot from NAND:

ubi part rootfs
ubifsmount ubi0:rootfs
ubifsload 0x42000000 /boot/zImage
ubifsload 0x43000000 /boot/sun5i-r8-chip.dtb
setenv bootargs root=ubi0_0 rootfstype=ubifs ubi.mtd=4 rw earlyprintk waitroot
bootz 0x42000000 - 0x43000000

This still requires manual intervention on the U-Boot command line.

But: we've booted without using the sunxi-fel tool!

You have probably still connected C.H.I.P to your work station via USB to power it. Try using a power supply or a battery instead - your C.H.I.P is flashed now. The next chapter shows how to automate the flashing and how to boot C.H.I.P without typing commands manually in U-Boot via UART.