Running jailhouse on a NVIDIA Jetson TK1 with Gentoo from scratch

TL;DR: You can find latest kernel configs, pre-built kernel images, u-boot configs and pre-built u-boot binaries as well as a pre-built gentoo linux image here. I try to keep everything up to date:

  • Kernel: 4.9 (stable)
  • U-Boot: 2016.05-rc1
  • Gentoo RootFS: 2018-06-04 (with systemd)

Contents


Compile everything from scratch

Prerequisites

  • Gentoo Host
  • ARMv7 Compiler
  • Jetson TK1 board
  • Serial cable (for jailhouse output and general purpose debugging)
  • Micro USB cable

On your local gentoo box, you can install the cross-compiler with crossdev.

$ emerge crossdev
$ crossdev -S -v -t armv7a-hardfloat-linux-gnueabi

Prepare Gentoo Base System

On your gentoo box, create a workspace

$ mkdir /tmp/jetson
$ cd /tmp/jetson

Inside this directory, download and unpack the latest Gentoo Stage 3. You may also want to download the latest portage snapshot to usr/.

$ curl http://gentoo.oregonstate.edu/releases/arm/autobuilds/current-stage3-armv7a_hardfp/stage3-armv7a_hardfp-20150721.tar.bz2 | sudo tar -xvjp -C .
$ curl http://gentoo.oregonstate.edu/snapshots/portage-latest.tar.bz2 | sudo tar -xvjp -C usr

Adjust your Gentoo make.conf. You probably want to add something like

$ sudo vim etc/portage/make.conf
>> MAKEOPTS="-j5"

Get a salted hash of your root password using openssl and write it to /etc/shadow

$ openssl passwd -1 gentoo
$1$xQl51XUf$.eUKt5EynZu4OWQnnIdZf.
$ sudo vim etc/shadow
root:$1$xQl51XUf$.eUKt5EynZu4OWQnnIdZf.:10770:0:::::

Adjust the etc/fstab. The following line should remain the only one.

/dev/mmcblk0p1 / ext4 noatime 0 1

Change the speed of the serial console /dev/ttyS0 to 115200 Baud (later we have to comment out this line for using jailhouse)

$ sudo vim etc/inittab
s0:12345:respawn:/sbin/agetty -L 115200 ttyS0 vt100

Never forget to think about a nice hostname and to set the timezone

$ echo hostname=\"iridium\" > etc/conf.d/hostname
$ ln -sf usr/share/zoneinfo/Europe/Berlin etc/localtime

We will use auto-DHCP for our networking interface

$ ln -sf net.lo etc/init.d/net.enp1s0
$ echo config_enp1s0=\"dhcp\" > etc/conf.d/net

Time to send out RootFS to the Jetson TK1. A pretty nice feature of u-boot allows us to mount the eMMC of the Jetson as USB mass storage device. Connect the Mini-USB to your machine, push the reset button your jetson, interrupt the bootloader and type:

Hit any key to stop autoboot: 0
Tegra124 (Jetson TK1) # ums 0 mmc 0
UMS: LUN 0, dev 0, hwpart 0, sector 0x0, count 0x1d5a000

Now the whole eMMC of your Jetson is available as a mass storage device. I recommend to create one single huge ext4 partition using gparted. After creating the partition, mount it and copy the root file system:

mount /dev/sdX1 /mnt/temp/
cp -av /tmp/jetson/* /mnt/temp/
sync

Take a cup of coffee or tea.

Compile your own kernel

As your Jetson is unable to boot yet, you have to cross-compile the kernel on your local machine.

$ emerge gentoo-sources
$ cd /usr/src/linux
$ make ARCH=arm menuconfig # Adjust everything you need
$ make ARCH=arm CROSS_COMPILE=armv7a-hardfloat-linux-gnueabi- -j 5
# Copy the kernel
$ cp arch/arm/boot/zImage /mnt/temp/boot/
# Copy the device tree block
$ cp arch/arm/boot/dts/tegra124-jetson-tk1.dtb /boot
# Install all modules
$ make modules_install INSTALL_MOD_PATH=/mnt/temp/
# Sync file system and unmount the Jetson
$ sync
$ umount /mnt/temp

Next step is to place a file called /boot/boot.scr. It’s pretty similar to GRUB’s menu.cfg. U-Boot reads this file when booting and executes its commands. Unfortunately, U-Boot uses a binary format and it must be compiled first. But first of all, install the u-boot-tools:

$ emerge u-boot-tools

Create the file boot.script that contains the following content:

# Optionally add 'init=/usr/lib/systemd/systemd', if you want to use systemd insteas of OpenRC
setenv bootargs 'console=ttyS0,115200 root=/dev/mmcblk0p1 rw rootwait'

#Comment the line above and uncomment these lines for running jailhouse
#setenv bootargs 'root=/dev/mmcblk0p1 rw rootwait mem=1984M vmalloc=512M'
#setenv bootm_boot_mode nonsec

load ${devtype} ${devnum}:1 ${kernel_addr_r} /boot/zImage
load ${devtype} ${devnum}:1 ${fdt_addr_r} /boot/tegra124-jetson-tk1.dtb
bootz ${kernel_addr_r} ${ramdisk_addr_r}:${filesize} ${fdt_addr_r}

Use the mkimage tool of the u-boot-tools to compile it to a .scr file:

$ cd /tmp/jetson/boot
$ wget https://ramses-pyramidenbau.de/~ralf/jetson-tk1/boot.script
$ mkimage -A arm -O linux -T script -C none -a 0 -e 0 -n Boot-Script -d boot.script boot.scr

After rebooting, make sure that all CPUs are started in HYP mode:

$ root@iridium:~# dmesg | grep HYP
[ 0.101989] CPU: All CPU(s) started in HYP mode.

Here you can also download a prebuilt boot.scr. Copy the boot.scr file to the /boot directory of your jetson.

Compile U-Boot

Jailhouse requires the processor to run in non-secure HYP mode. The processor needs to be staged to HYP mode very early by the bootloader. HYP is not enabled in NVIDIA’s stock bootloader, so we need to compile and flash it on our own.

It took me quite some time to find out how to build the bootloader correctly.
Here’s the summary 🙂 Run the following commands:

$ mkdir /tmp/uboot
$ cd /tmp/uboot
# Get all the required stuff
$ git clone https://github.com/NVIDIA/tegra-uboot-flasher-scripts.git
$ git clone https://github.com/NVIDIA/tegrarcm.git
$ git clone https://github.com/NVIDIA/cbootimage.git
$ git clone https://github.com/NVIDIA/cbootimage-configs.git
$ git clone git://git.denx.de/u-boot-tegra.git u-boot

# build tegrarcm
$ cd tegrarcm
$ ./autogen.sh
$ make
$ cd ..

# build cbootimage
$ cd cbootimage
$ ./autogen.sh
$ make
$ cd ..

# build u-boot
$ export PATH=$PATH:/tmp/uboot/cbootimage/src:/tmp/uboot/tegrarcm/src
$ cd tegra-uboot-flasher-scripts
$ export CROSS_COMPILE=armv7a-hardfloat-linux-gnueabi-
$ ./build --socs tegra124 --boards jetson-tk1 build

Et voilĂ , you just built your own U-Boot bootloader. Let’s flash it! Before sending the next command, bring your device in recovery mode by pressing the “RESET” + “FORCE RECOVERY” buttons.


$ cd ../tegra-uboot-flasher-scripts
$ ./tegra-uboot-flasher --data-dir ../_out flash jetson-tk1

With a bit of luck and some magic unicorn dust, your jetson will directly boot your custom bootloader that boots your custom kernel. After rebooting, you can check if HYP is enabled with

$ root@iridium:~# dmesg | grep HYP
[ 0.101989] CPU: All CPU(s) started in HYP mode.

First Boot

After your device was successfully flashed it will automatically reboot. You can now login to it by using a serial connection.

$ putty -serial /dev/ttyUSB0 115200 -sercfg 115200

As we don’t have NTP yet, our first step will be to set the current time.

On your TK1 login and type:

$ date --set="20150723 18:37"
$ ####
$ # Your network should be autoconfigured via DHCP, if not try
$ ifconfig enp1s0 a.b.c.d
$ route add default gw a.b.c.e
$ echo nameserver 8.8.8.8 > /etc/resolv.conf
$ ####
$ emerge --sync
$ emerge -av ntp
$ rc-update add ntp-client default
$ rc-update add sshd default
$ # Perform a full system upgrade
$ emerge -uDNav world


Compile Jailhouse

This will guide you how to compile Jailhouse on your Jetson TK1. Jailhouse comes with some dependencies and requires the mako python package in order to compile. Install mako as well as git before cloning and compiling jailhouse.

$ emerge git mako
$ git clone https://github.com/siemens/jailhouse.git
$ cd jailhouse
$ cat > hypervisor/include/jailhouse/config.h # Create this file with the following content
#define CONFIG_ARM_GIC_V2 1
#define CONFIG_MACH_TEGRA124 1
$ make
$ make install
$ depmod -a
$ modprobe jailhouse
$ lsmod
# and check if the module was successfully loaded


Playing around with jailhouse

This tutorial will give you a brief introduction on how to run a simple uart-demo inmate in jailhouse on the TK1. It is important not to use the UART in the root cell as it will be assigned to the non-root cell after creating it. Any further UART access from the root cell after creating the non-root cell will lead to a panic of the hypervisor. Additionally, jailhouse needs some memory for the hypervisor itself and its inmates that is not used by the kernel of the root cell. This results in a boot.script as follows:

# Reserve 64MiB for jailhouse
setenv bootargs 'root=/dev/mmcblk0p1 rw rootwait mem=1984M vmalloc=512M'
# This enables HYP mode (disabled by default)
setenv bootm_boot_mode nonsec
load ${devtype} ${devnum}:1 ${kernel_addr_r} /boot/zImage
load ${devtype} ${devnum}:1 ${fdt_addr_r} /boot/tegra124-jetson-tk1.dtb
bootz ${kernel_addr_r} ${ramdisk_addr_r}:${filesize} ${fdt_addr_r}

Compile the boot.script using mkimage to boot.scr and copy it to ‘/boot/’.

$ mkimage -A arm -O linux -T script -C none -a 0 -e 0 -n Boot-Script -d boot.script /boot/boot.scr

Also, comment out the serial console from ‘/etc/inittab’. Use ssh to access your device. After rebooting your device, you can load the jailhouse module and activate the hypervisor:

$ modprobe jailhouse
$ jailhouse enable ~/jailhouse/configs/jetson-tk1.cell

Dmesg should now tell you, that the jailhouse has opened:

$ dmesg | tail
[...]
[ 183.608569] The Jailhouse is opening.

On your serial console, you should see something like:

Initializing Jailhouse hypervisor v0.5 (132-g997802f) on CPU 0
Code location: 0xf0000020
Page pool usage after early setup: mem 22/16112, remap 64/32768
Initializing processors:
CPU 0... OK
CPU 3... OK
CPU 2... OK
CPU 1... OK
Page pool usage after late setup: mem 34/16112, remap 64/32768
Activating hypervisor

Now you can create a new non-root cell using the jetson-tk1-demo cell configuration:

$ jailhouse cell create ~/jailhouse/configs/jetson-tk1-demo.cell

Now that your cell is created, you can load a binary to it and start the cell:

$ jailhouse cell load jetson-tk1-demo ~/jailhouse/inmates/demos/arm/uart-demo.bin -a 0
$ jailhouse cell start jetson-tk1-demo

On your serial console, you can now see the output of the demo cell.

To stop and destroy the cell, just type:

$ jailhouse cell shutdown jetson-tk1-demo
$ jailhouse cell destroy jetson-tk1-demo

Use my precompiled RootFS + Bootloader

If you don’t want to compile everything on your own – which i truly can understand – then you can use my pre-compiled images.

I try to keep those images up to date. Look at the top of the page for the current versions. SSH server is enabled, root password is ‘gentoo’, dhcp client on wired ethernet. So just a few steps for you to start.

Reset your Jetson, interrupt the bootloader and attach it as USB mass storage device (explained above) by typing

Tegra124 (Jetson TK1) # ums 0 mmc 0

to your serial console. Mount and partition your Jetson and extract the root file system:

$ mount /dev/sdX1 /mnt/temp/
# Extract Root FS
$ curl https://ramses-pyramidenbau.de/~ralf/jetson-tk1/rootfs/RootFS-latest.tar.gz | tar x -C /mnt/temp/
# Extract pre-compiled kernel
$ curl https://ramses-pyramidenbau.de/~ralf/jetson-tk1/kernel/precompiled/latest.tar.bz2 | tar x -C /mnt/temp/
# Place boot.scr (non-jailhouse variant)
$ curl https://ramses-pyramidenbau.de/~ralf/jetson-tk1/u-boot/boot.scr > /mnt/temp/boot/boot.scr
# Place boot.scr (jailhouse variant)
$ curl https://ramses-pyramidenbau.de/~ralf/jetson-tk1/u-boot/boot.scr.jailhouse > /mnt/temp/boot/boot.scr

Next step is to flash the correct bootloader. On your local machine, type

$ git clone https://github.com/NVIDIA/tegra-uboot-flasher-scripts.git
$ git clone https://github.com/NVIDIA/tegrarcm.git
$ cd tegrarcm
$ ./autogen.sh
$ make
$ export PATH="$PATH:`realpath src`"
$ cd ../tegra-uboot-flasher-scripts
$ curl https://ramses-pyramidenbau.de/~ralf/jetson-tk1/u-boot/jetson-tk1-uboot-2016.05-rc1.tar.bz2 | tar -xzv -C .
$ ./tegra-uboot-flasher --data-dir . flash jetson-tk1

Et voilĂ . Reboot your device and check if HYP is enabled if you chose the jailhouse variant:

root@iridium:/# dmesg|grep HYP
[ 0.101987] CPU: All CPU(s) started in HYP mode.

13 thoughts on “Running jailhouse on a NVIDIA Jetson TK1 with Gentoo from scratch”

  1. Hi and thanks for the guide,

    I’ve managed to boot my jetson to /dev/sda1 using the t4l kernel.
    But my gentoo-sources kernel is not loading.
    uboot – options
    1 : default T4L — loading
    2 : gentoo using t4l kernel — loading
    3 : gentoo using gentoo-sources — not loading .. blank screen

    Do i need to take care of special kernel options besides the tegra124 stuff?

  2. Hi Ralf,

    thanks for the reply.
    It just stops here:
    MMC: no card present
    switch to partitions #0, OK
    mmc0(part 0) is current device
    Scanning mmc 0…
    Found /boot/extlinux/extlinux.conf
    Retrieving file: /boot/extlinux/extlinux.conf
    2351 bytes read in 71 ms (32.2 KiB/s)
    Jetson-TK1 eMMC boot options
    1: primary kernel
    2: T4L gentoo
    3: gentoo kernel 4.2
    Enter choice: 3
    3: gentoo kernel 4.2
    Retrieving file: /boot/zImage_gentoo
    4310480 bytes read in 417 ms (9.9 MiB/s)
    append: console=ttyS0,115200n8 console=tty1 no_console_suspend=1 lp0_vec=2064@0xf46ff000 mem=2015M@2048M memtype=255 ddr_die=2048M@2048M section=256M pmuboard=0x0177:0x0000:0x02:0x43:0x00 tsec=32M@3913M otf_key=c75e5bb91eb3bd947560357b64422f85 usbcore.old_scheme_first=1 core_edp_mv=1150 core_edp_ma=4000 tegraid=40.1.1.0.0 debug_uartport=lsport,3 power_supply=Adapter audio_codec=rt5640 modem_id=0 android.kerneltype=normal fbcon=map:1 commchip_id=0 usb_port_owner_info=0 lane_owner_info=6 emc_max_dvfs=0 touch_id=0@0 board_info=0x0177:0x0000:0x02:0x43:0x00 root=/dev/sda1 rw rootwait tegraboot=sdmmc gpt init=/usr/lib/systemd/systemd
    Retrieving file: /boot/tegra124-jetson-tk1_gentoo.dtb
    62999 bytes read in 324 ms (189.5 KiB/s)
    Kernel image @ 0x81000000 [ 0x000000 – 0x41c5d0 ]
    ## Flattened Device Tree blob at 82000000
    Booting using the fdt blob at 0x82000000
    Using Device Tree in place at 82000000, end 82012616

    Starting kernel …

    1. Hi,

      first of all, you seem to have pretty much kernel arguments. In my case, for booting my own kernel
      console=ttyS0,115200 root=/dev/mmcblk0p1 rw rootwait
      was absolutely sufficient. Second, you specify root=/dev/sda1. Is this really correct?

      But however, this should actually not prevent your kernel from showing its early output to your serial console…

      So my guess is the .dtb file: Where does it come from? Did you ensure that /boot/tegra124-jetson-tk1_gentoo.dtb corresponds to your kernel? You have to copy arch/arm/zImage as well as the .dtb file to /boot.
      cd /usr/src/linux
      cp arch/arm/boot/dts/tegra124-jetson-tk1.dtb /boot/tegra124-jetson-tk1-gentoo.dtb

      Ralf

  3. Ralf,
    If we transfer the UART from the root cell to inmate and it manages to print messages successfully, why does it trap and hangs, when we attempted to type something on the putty window?

    thanks

    1. Hi Dan,

      what’s the exact trap message? After Jailhouse takes away the UART from the root cell, no further reads/writes from/to MMIO are allowed. Any further access will cause an access violation.

      This is why ‘echo asd > /dev/ttyS0’ from the root cell will hang your root cell as well.

      In your case, I think that the interrupt still reaches the root cell, and the UART driver of the root cell tries to handle that interrupt and probably ends up in a forbidden MMIO access. You’re welcome to ask your question on the mailing list as well.

      Ralf

  4. Hello Ralf,
    We pass the UART to inmate, and the inmate is capable of printing messages out.
    I can’t figure out, why once we type something in, the system hangs, shouldn’t the input just go symmetrically to the inmate?

    Thanks
    Dan

  5. Hi,
    I don’t see where you made the switch to systemd as far as I know, the default ARM stage3s are using OpenRC and other than for AMD64 there are no systemd stage3 builds to my knowledge.
    BG, Thomas

    1. Hi Thomas,

      right, thanks for mentioning. Will add some remarks. I decided to switch to systemd at some point.
      As far as I remember, the images have the desktop/systemd profile enabled.

      Cheers
      Ralf

  6. Great stuff here!

    Is this approach compatible with TK2 system. If not, are you going to test it in near future?

    1. Hi,

      unfortunatelly not, don’t have the hardware. But I have a TX1 running with Gentoo + Jailhouse. Should basically be compatible to the TX2.

      Cheers

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.