Category Archives: Kernel

Running jailhouse on a NVIDIA Jetson TK1 with Gentoo from scratch

TL;DR: You can find latest kernel configs, pre-built kernel images, u-boot configs and pre-built u-boot binaries as well as a pre-built gentoo linux image here. I try to keep everything up to date:

  • Kernel: 4.9 (stable)
  • U-Boot: 2016.05-rc1
  • Gentoo RootFS: 2018-06-04 (with systemd)


Compile everything from scratch


  • Gentoo Host
  • ARMv7 Compiler
  • Jetson TK1 board
  • Serial cable (for jailhouse output and general purpose debugging)
  • Micro USB cable

On your local gentoo box, you can install the cross-compiler with crossdev.

$ emerge crossdev
$ crossdev -S -v -t armv7a-hardfloat-linux-gnueabi

Prepare Gentoo Base System

On your gentoo box, create a workspace

$ mkdir /tmp/jetson
$ cd /tmp/jetson

Inside this directory, download and unpack the latest Gentoo Stage 3. You may also want to download the latest portage snapshot to usr/.

$ curl | sudo tar -xvjp -C .
$ curl | sudo tar -xvjp -C usr

Adjust your Gentoo make.conf. You probably want to add something like

$ sudo vim etc/portage/make.conf
>> MAKEOPTS="-j5"

Get a salted hash of your root password using openssl and write it to /etc/shadow

$ openssl passwd -1 gentoo
$ sudo vim etc/shadow

Adjust the etc/fstab. The following line should remain the only one.

/dev/mmcblk0p1 / ext4 noatime 0 1

Change the speed of the serial console /dev/ttyS0 to 115200 Baud (later we have to comment out this line for using jailhouse)

$ sudo vim etc/inittab
s0:12345:respawn:/sbin/agetty -L 115200 ttyS0 vt100

Never forget to think about a nice hostname and to set the timezone

$ echo hostname=\"iridium\" > etc/conf.d/hostname
$ ln -sf usr/share/zoneinfo/Europe/Berlin etc/localtime

We will use auto-DHCP for our networking interface

$ ln -sf net.lo etc/init.d/net.enp1s0
$ echo config_enp1s0=\"dhcp\" > etc/conf.d/net

Time to send out RootFS to the Jetson TK1. A pretty nice feature of u-boot allows us to mount the eMMC of the Jetson as USB mass storage device. Connect the Mini-USB to your machine, push the reset button your jetson, interrupt the bootloader and type:

Hit any key to stop autoboot: 0
Tegra124 (Jetson TK1) # ums 0 mmc 0
UMS: LUN 0, dev 0, hwpart 0, sector 0x0, count 0x1d5a000

Now the whole eMMC of your Jetson is available as a mass storage device. I recommend to create one single huge ext4 partition using gparted. After creating the partition, mount it and copy the root file system:

mount /dev/sdX1 /mnt/temp/
cp -av /tmp/jetson/* /mnt/temp/

Take a cup of coffee or tea.

Compile your own kernel

As your Jetson is unable to boot yet, you have to cross-compile the kernel on your local machine.

$ emerge gentoo-sources
$ cd /usr/src/linux
$ make ARCH=arm menuconfig # Adjust everything you need
$ make ARCH=arm CROSS_COMPILE=armv7a-hardfloat-linux-gnueabi- -j 5
# Copy the kernel
$ cp arch/arm/boot/zImage /mnt/temp/boot/
# Copy the device tree block
$ cp arch/arm/boot/dts/tegra124-jetson-tk1.dtb /boot
# Install all modules
$ make modules_install INSTALL_MOD_PATH=/mnt/temp/
# Sync file system and unmount the Jetson
$ sync
$ umount /mnt/temp

Next step is to place a file called /boot/boot.scr. It’s pretty similar to GRUB’s menu.cfg. U-Boot reads this file when booting and executes its commands. Unfortunately, U-Boot uses a binary format and it must be compiled first. But first of all, install the u-boot-tools:

$ emerge u-boot-tools

Create the file boot.script that contains the following content:

# Optionally add 'init=/usr/lib/systemd/systemd', if you want to use systemd insteas of OpenRC
setenv bootargs 'console=ttyS0,115200 root=/dev/mmcblk0p1 rw rootwait'

#Comment the line above and uncomment these lines for running jailhouse
#setenv bootargs 'root=/dev/mmcblk0p1 rw rootwait mem=1984M vmalloc=512M'
#setenv bootm_boot_mode nonsec

load ${devtype} ${devnum}:1 ${kernel_addr_r} /boot/zImage
load ${devtype} ${devnum}:1 ${fdt_addr_r} /boot/tegra124-jetson-tk1.dtb
bootz ${kernel_addr_r} ${ramdisk_addr_r}:${filesize} ${fdt_addr_r}

Use the mkimage tool of the u-boot-tools to compile it to a .scr file:

$ cd /tmp/jetson/boot
$ wget
$ mkimage -A arm -O linux -T script -C none -a 0 -e 0 -n Boot-Script -d boot.script boot.scr

After rebooting, make sure that all CPUs are started in HYP mode:

$ root@iridium:~# dmesg | grep HYP
[ 0.101989] CPU: All CPU(s) started in HYP mode.

Here you can also download a prebuilt boot.scr. Copy the boot.scr file to the /boot directory of your jetson.

Compile U-Boot

Jailhouse requires the processor to run in non-secure HYP mode. The processor needs to be staged to HYP mode very early by the bootloader. HYP is not enabled in NVIDIA’s stock bootloader, so we need to compile and flash it on our own.

It took me quite some time to find out how to build the bootloader correctly.
Here’s the summary 🙂 Run the following commands:

$ mkdir /tmp/uboot
$ cd /tmp/uboot
# Get all the required stuff
$ git clone
$ git clone
$ git clone
$ git clone
$ git clone git:// u-boot

# build tegrarcm
$ cd tegrarcm
$ ./
$ make
$ cd ..

# build cbootimage
$ cd cbootimage
$ ./
$ make
$ cd ..

# build u-boot
$ export PATH=$PATH:/tmp/uboot/cbootimage/src:/tmp/uboot/tegrarcm/src
$ cd tegra-uboot-flasher-scripts
$ export CROSS_COMPILE=armv7a-hardfloat-linux-gnueabi-
$ ./build --socs tegra124 --boards jetson-tk1 build

Et voilà, you just built your own U-Boot bootloader. Let’s flash it! Before sending the next command, bring your device in recovery mode by pressing the “RESET” + “FORCE RECOVERY” buttons.

$ cd ../tegra-uboot-flasher-scripts
$ ./tegra-uboot-flasher --data-dir ../_out flash jetson-tk1

With a bit of luck and some magic unicorn dust, your jetson will directly boot your custom bootloader that boots your custom kernel. After rebooting, you can check if HYP is enabled with

$ root@iridium:~# dmesg | grep HYP
[ 0.101989] CPU: All CPU(s) started in HYP mode.

First Boot

After your device was successfully flashed it will automatically reboot. You can now login to it by using a serial connection.

$ putty -serial /dev/ttyUSB0 115200 -sercfg 115200

As we don’t have NTP yet, our first step will be to set the current time.

On your TK1 login and type:

$ date --set="20150723 18:37"
$ ####
$ # Your network should be autoconfigured via DHCP, if not try
$ ifconfig enp1s0 a.b.c.d
$ route add default gw a.b.c.e
$ echo nameserver > /etc/resolv.conf
$ ####
$ emerge --sync
$ emerge -av ntp
$ rc-update add ntp-client default
$ rc-update add sshd default
$ # Perform a full system upgrade
$ emerge -uDNav world

Compile Jailhouse

This will guide you how to compile Jailhouse on your Jetson TK1. Jailhouse comes with some dependencies and requires the mako python package in order to compile. Install mako as well as git before cloning and compiling jailhouse.

$ emerge git mako
$ git clone
$ cd jailhouse
$ cat > hypervisor/include/jailhouse/config.h # Create this file with the following content
#define CONFIG_ARM_GIC_V2 1
#define CONFIG_MACH_TEGRA124 1
$ make
$ make install
$ depmod -a
$ modprobe jailhouse
$ lsmod
# and check if the module was successfully loaded

Playing around with jailhouse

This tutorial will give you a brief introduction on how to run a simple uart-demo inmate in jailhouse on the TK1. It is important not to use the UART in the root cell as it will be assigned to the non-root cell after creating it. Any further UART access from the root cell after creating the non-root cell will lead to a panic of the hypervisor. Additionally, jailhouse needs some memory for the hypervisor itself and its inmates that is not used by the kernel of the root cell. This results in a boot.script as follows:

# Reserve 64MiB for jailhouse
setenv bootargs 'root=/dev/mmcblk0p1 rw rootwait mem=1984M vmalloc=512M'
# This enables HYP mode (disabled by default)
setenv bootm_boot_mode nonsec
load ${devtype} ${devnum}:1 ${kernel_addr_r} /boot/zImage
load ${devtype} ${devnum}:1 ${fdt_addr_r} /boot/tegra124-jetson-tk1.dtb
bootz ${kernel_addr_r} ${ramdisk_addr_r}:${filesize} ${fdt_addr_r}

Compile the boot.script using mkimage to boot.scr and copy it to ‘/boot/’.

$ mkimage -A arm -O linux -T script -C none -a 0 -e 0 -n Boot-Script -d boot.script /boot/boot.scr

Also, comment out the serial console from ‘/etc/inittab’. Use ssh to access your device. After rebooting your device, you can load the jailhouse module and activate the hypervisor:

$ modprobe jailhouse
$ jailhouse enable ~/jailhouse/configs/jetson-tk1.cell

Dmesg should now tell you, that the jailhouse has opened:

$ dmesg | tail
[ 183.608569] The Jailhouse is opening.

On your serial console, you should see something like:

Initializing Jailhouse hypervisor v0.5 (132-g997802f) on CPU 0
Code location: 0xf0000020
Page pool usage after early setup: mem 22/16112, remap 64/32768
Initializing processors:
CPU 0... OK
CPU 3... OK
CPU 2... OK
CPU 1... OK
Page pool usage after late setup: mem 34/16112, remap 64/32768
Activating hypervisor

Now you can create a new non-root cell using the jetson-tk1-demo cell configuration:

$ jailhouse cell create ~/jailhouse/configs/jetson-tk1-demo.cell

Now that your cell is created, you can load a binary to it and start the cell:

$ jailhouse cell load jetson-tk1-demo ~/jailhouse/inmates/demos/arm/uart-demo.bin -a 0
$ jailhouse cell start jetson-tk1-demo

On your serial console, you can now see the output of the demo cell.

To stop and destroy the cell, just type:

$ jailhouse cell shutdown jetson-tk1-demo
$ jailhouse cell destroy jetson-tk1-demo

Use my precompiled RootFS + Bootloader

If you don’t want to compile everything on your own – which i truly can understand – then you can use my pre-compiled images.

I try to keep those images up to date. Look at the top of the page for the current versions. SSH server is enabled, root password is ‘gentoo’, dhcp client on wired ethernet. So just a few steps for you to start.

Reset your Jetson, interrupt the bootloader and attach it as USB mass storage device (explained above) by typing

Tegra124 (Jetson TK1) # ums 0 mmc 0

to your serial console. Mount and partition your Jetson and extract the root file system:

$ mount /dev/sdX1 /mnt/temp/
# Extract Root FS
$ curl | tar x -C /mnt/temp/
# Extract pre-compiled kernel
$ curl | tar x -C /mnt/temp/
# Place boot.scr (non-jailhouse variant)
$ curl > /mnt/temp/boot/boot.scr
# Place boot.scr (jailhouse variant)
$ curl > /mnt/temp/boot/boot.scr

Next step is to flash the correct bootloader. On your local machine, type

$ git clone
$ git clone
$ cd tegrarcm
$ ./
$ make
$ export PATH="$PATH:`realpath src`"
$ cd ../tegra-uboot-flasher-scripts
$ curl | tar -xzv -C .
$ ./tegra-uboot-flasher --data-dir . flash jetson-tk1

Et voilà. Reboot your device and check if HYP is enabled if you chose the jailhouse variant:

root@iridium:/# dmesg|grep HYP
[ 0.101987] CPU: All CPU(s) started in HYP mode.

Installing Gentoo on a Raspberry PI

Install ARM Toolchain

$ emerge -av crossdev
$ crossdev -S -v -t armv6j-hardfloat-linux-gnueabi

Get Raspberry Kernel Sources

$ git clone --depth=1
$ wget
$ patch -p1 -d linux/ < linux-arm.patch

Configure the Kernel for QEMU

$ cd linux
$ make ARCH=arm menuconfig
$ make ARCH=arm
$ make ARCH=arm INSTALL_MOD_PATH=../modules modules_install
$ cp arch/arm/boot/zImage ../raspi-kernel

This is my .config
and the corresponding zImage and Modules: kernel_raspi_qemu.tar

Prepare “SD Card”

Create a sparse Image (this will be the SD Card)
$ dd if=/dev/null of=gentoo-raspi.img bs=1G seek=8 # 8GiB Image Partition Image
$ cfdisk gentoo-raspi.img
#first partition: boot, vfat ~80MiB (Partition Type: 0x0C)
#second: Gentoo Root (Partition Type: 0x83)

Mount Image as NBD

$ modprobe nbd max_part=4
$ qemu-nbd --connect=/dev/nbd0 -n /path/to/gentoo-raspi.img

$ emerge -av dosfstools
$ mkfs.vfat -n boot /dev/nbd0p1
$ mkfs.ext4 -j -L Gentoo -E lazy_itable_init=0,lazy_journal_init=0 /dev/nbd0p2
$ mount /dev/nbd0p2 /mnt/raspi
$ cd /mnt/raspi
$ curl | tar -xvjp

Prepare Base System

$ dd if=/dev/zero of=/mnt/raspi/var/swap bs=1M count=512
$ chmod 0600 /mnt/raspi/var/swap
$ mkswap /mnt/raspi/var/swap

$ cd /mnt/raspi/etc
$ vi fstab

Insert the following lines:
LABEL=Gentoo / ext4 noatime 0 1
LABEL=BOOT /boot vfat noatime,defaults 0 0
/var/swap swap swap defaults 0 0

Set Root Password
$ openssl passwd -1 raspi
$ vi shadow

first line: root:$1$pDGeoXQj$0zq3sa5b9o9rJ8CpcIpPR/:10770:0:::::

Set Hostname & Keymap & timezone
$ vi conf.d/hostname
$ vi conf.d/keymaps
$ ln -sf ../usr/share/zoneinfo/Europe/Berlin localtime

Configure locale.gen
$ vi /etc/locale.gen

Copy Kernel Modules
$ cp -av /path/to/modules/lib /mnt/raspi

Install Portage
$ curl | tar -xvjp -C /mnt/raspi/usr

Set correct make.profile
$ cd /mnt/raspi/etc/portage rm make.profile
$ ln -sf ../../usr/portage/profiles/default/linux/arm/13.0 make.profile

Disconnect NBD
$ umount /mnt/raspi
$ qemu-nbd --disconnect /dev/nbd0

Boot your Raspberry on QEMU

$ qemu-system-arm -kernel zImage -cpu arm1176 -m 256 -M versatilepb -no-reboot -serial stdio -append "root=/dev/sda2 panic=1 console=ttyAMA0" -hda gentoo-raspi.img
Login: root:raspi
Set Date (Raspi has no hardware Clock 🙁 )
$ date --set="20130808 14:00"
$ rc-update del hwclock default

Configure Network
$ ifconfig eth0 #USE THIS IP!
$ route add default gw
$ echo nameserver > /etc/resolv.conf #just use google's dns

Get dhcp Client and configure network
$ emerge dhcpcd
$ dhcpcd

Configure eth0 persistently
$ cd /etc/init.d
$ ln -s net.lo net.eth0
$ rc-update add net.eth0 boot

$ cd /etc/conf.d/
$ nano net

add the line: config_eth0="dhcp"

You WANT distcc.
emerge -av distcc #Run on both, host and guest

Edit make.conf
$ echo FEATURES="distcc" >> /etc/portage/make.conf
$ echo MAKEOPTS="-j6" >> /etc/portage/make.conf

Set DistCC Hosts
$ cd /etc/distcc
$ nano hosts

Now compile everything you want

Emerge boot stuff and kernel
$ mkdir /etc/portage/package.keywords
$ cd /etc/portage/package.keywords
$ echo sys-boot/raspberrypi-firmware ** ~arm >> raspi
$ echo sys-kernel/raspberrypi-sources ** ~arm >> raspi
$ echo media-libs/raspberrypi-userland ** ~arm >> raspi

$ mount /boot
$ emerge -av raspberrypi-firmware raspberrypi-sources raspberrypi-userland

Compile the kernel
$ cd /usr/src/linux
$ make menuconfig

$ make CC="distcc armv6j-hardfloat-linux-gnueabi-gcc" modules -j5
$ make modules_install
$ make CC="distcc armv6j-hardfloat-linux-gnueabi-gcc" -j5
$ cp arch/arm/boot/zImage /boot/kernel.img

This is my precompiled Kernel and Modules / Firmware:

You also want to use SSH:
$ rc-update add sshd default

And as we do have no RTC:
$ rc-update add ntp-client default

And a syslogger:
$ emerge -av syslog-ng
$ rc-update add syslog-ng boot

Here's my image:
It needs at least 2.8GiB of space (2.3GiB without /var/swap)

As .tar.gz archive:
VFAT Boot Partition: Raspberry-Boot-20130816.tar.gz
Gentoo Installation: Raspberry-Stage4-20130816.tar.gz

As full 6GiB raw image which is bootable out of the box:

You can use dd to flash it on a SD Card
$ gunzip Raspberry-Gentoo-6GiB-Image-20130816.img.gz
$ dd if=Raspberry-Gentoo-6GiB-Image-20130816.img of=/dev/yourraspisdcard

Have fun with Gentoo on your Raspi 🙂

Update 2014-06-03:

Here's another nice tutorial without any cross-compilation

Linux’s strange memory allocation strategy

As I learned today, Linux has a real strange memory allocation strategy. Have a look at this piece of code:

#include <stdio.h>
#include <stdlib.h>

int main(void) {
  while(1) {
    char *foo = (char*)malloc(1024);
    if(foo == NULL) {
      printf("Couldn't alloc\n");
      return 0;
  return 0;

According to the malloc reference it should return NULL if it is not able to allocate memory:

“If the function failed to allocate the requested block of memory, a null pointer is returned.”

So I thought my process would write Couldn’t alloc to stdout and exit in a proper way. But it didn’t act that way. When my system ran out of memory the process got killed by the kernel with signal SIGKILL. So why does it act like this? Wikipedia writes about Out of memory:

“A process which exceeds its per-process limit and then attempts to allocate further memory – with malloc(), for example – will return failure. A well-behaved application should handle this situation gracefully; however, many do not. An attempt to allocate memory without checking the result is known as an “unchecked malloc”.”

Well… Yes… Of course, you always should check malloc() if it returned NULL, but under normal conditions it never will return NULL because of Linux’s memory overcommitting. By default Linux has an optimistic memory allocation strategy. When allocating memory, malloc() will return a pointer. The space on the heap which is needed for that pointer gets allocated at the first read/write operation to that address. In my opinion, this is a really strange behaviour, because when memory gets allocated, it will actually be used. Here’s a short example to see the effects of that strange behaviour:

#include <stdio.h>
#include <stdlib.h>
#include <string.h>

#define MALLOC_SIZE (1024*1024*10)

int main(void) {
  int i = 0;
  while(1) {
    char *foo = (char*)malloc(MALLOC_SIZE);
    //memset(foo, 0xaa, MALLOC_SIZE);
    printf("Allocation No. %d\n", i++);
    if(foo == NULL) {
      printf("Couldn't alloc\n");
      return 0;
  return 0;

With the memset line commented out it results to:

ralf@Pegasus:C$ ./test|tail
Allocation No. 1841101
Allocation No. 1841102
Allocation No. 1841103
Allocation No. 1841104
Allocation No. 1841105
Allocation No. 1841106
Allocation No. 1841107
Allocation No. 1841108
Allocation No. 1841109
Allocation Nzsh: killed     ./test |
zsh: done       tail

And with the memset commented in it results to:

ralf@Pegasus:C$ ./test|tail
Allocation No. 1275
Allocation No. 1276
Allocation No. 1277
Allocation No. 1278
Allocation No. 1279
Allocation No. 1280
Allocation No. 1281
Allocation No. 1282
Allocation No. 1283
Allocazsh: killed     ./test |
zsh: done       tail

This is really funny. You can allocate tons of space you don’t have. That’s creepy. But this behaviour can be switched by

echo 2 > /proc/sys/vm/overcommit_memory

“Since 2.5.30 the values are: 0 (default): as before: guess about how much overcommitment is reasonable, 1: never refuse any malloc(), 2: be precise about the overcommit – never commit a virtual address space larger than swap space plus a fraction overcommit_ratio of the physical memory. Here /proc/sys/vm/overcommit_ratio (by default 50) is another user-settable parameter. It is possible to set overcommit_ratio to values larger than 100. (See also Documentation/vm/overcommit-accounting.)” Source

Then, both results (with and without memset) lead to:

ralf@Pegasus:C$ ./test|tail
Allocation No. 433
Allocation No. 434
Allocation No. 435
Allocation No. 436
Allocation No. 437
Allocation No. 438
Allocation No. 439
Allocation No. 440
Allocation No. 441
Couldn't alloc

And this is exactly what I originally expected to be the normal way. Without this bypass, malloc will never return NULL and a process will not be able to shut down properly if the system runs out of memory.

Here‘s a nice anecdote on that topic.