Category Archives: Linux

AVR-Toolchain for Gentoo Linux using crossdev

Since quite a long time, Gentoo Linux has problems building a GCC Toolchain for the AVR microcontroller architecture. This is just a short summary, how to get your toolchain running under Gentoo:

1. Emerge Toolchain

USE="-openmp -sanitize -vtv" crossdev -t avr -s4 -S --without-headers

2. Somehow, ldscripts go to /usr/x86_64-pc-linux-gnu/avr/lib/ldscripts but are actually expected to be located at /usr/avr/lib/ldscripts. A symlink quick fixes this issue

ln -s /usr/x86_64-pc-linux-gnu/avr/lib/ldscripts /usr/avr/lib/ldscripts

3. Have fun with your working toolchain

Running jailhouse on a NVIDIA Jetson TK1 with Gentoo from scratch

TL;DR: You can find latest kernel configs, pre-built kernel images, u-boot configs and pre-built u-boot binaries as well as a pre-built gentoo linux image here. I try to keep everything up to date:

  • Kernel: 4.9 (stable)
  • U-Boot: 2016.05-rc1
  • Gentoo RootFS: 2018-06-04 (with systemd)


Compile everything from scratch


  • Gentoo Host
  • ARMv7 Compiler
  • Jetson TK1 board
  • Serial cable (for jailhouse output and general purpose debugging)
  • Micro USB cable

On your local gentoo box, you can install the cross-compiler with crossdev.

$ emerge crossdev
$ crossdev -S -v -t armv7a-hardfloat-linux-gnueabi

Prepare Gentoo Base System

On your gentoo box, create a workspace

$ mkdir /tmp/jetson
$ cd /tmp/jetson

Inside this directory, download and unpack the latest Gentoo Stage 3. You may also want to download the latest portage snapshot to usr/.

$ curl | sudo tar -xvjp -C .
$ curl | sudo tar -xvjp -C usr

Adjust your Gentoo make.conf. You probably want to add something like

$ sudo vim etc/portage/make.conf
>> MAKEOPTS="-j5"

Get a salted hash of your root password using openssl and write it to /etc/shadow

$ openssl passwd -1 gentoo
$ sudo vim etc/shadow

Adjust the etc/fstab. The following line should remain the only one.

/dev/mmcblk0p1 / ext4 noatime 0 1

Change the speed of the serial console /dev/ttyS0 to 115200 Baud (later we have to comment out this line for using jailhouse)

$ sudo vim etc/inittab
s0:12345:respawn:/sbin/agetty -L 115200 ttyS0 vt100

Never forget to think about a nice hostname and to set the timezone

$ echo hostname=\"iridium\" > etc/conf.d/hostname
$ ln -sf usr/share/zoneinfo/Europe/Berlin etc/localtime

We will use auto-DHCP for our networking interface

$ ln -sf net.lo etc/init.d/net.enp1s0
$ echo config_enp1s0=\"dhcp\" > etc/conf.d/net

Time to send out RootFS to the Jetson TK1. A pretty nice feature of u-boot allows us to mount the eMMC of the Jetson as USB mass storage device. Connect the Mini-USB to your machine, push the reset button your jetson, interrupt the bootloader and type:

Hit any key to stop autoboot: 0
Tegra124 (Jetson TK1) # ums 0 mmc 0
UMS: LUN 0, dev 0, hwpart 0, sector 0x0, count 0x1d5a000

Now the whole eMMC of your Jetson is available as a mass storage device. I recommend to create one single huge ext4 partition using gparted. After creating the partition, mount it and copy the root file system:

mount /dev/sdX1 /mnt/temp/
cp -av /tmp/jetson/* /mnt/temp/

Take a cup of coffee or tea.

Compile your own kernel

As your Jetson is unable to boot yet, you have to cross-compile the kernel on your local machine.

$ emerge gentoo-sources
$ cd /usr/src/linux
$ make ARCH=arm menuconfig # Adjust everything you need
$ make ARCH=arm CROSS_COMPILE=armv7a-hardfloat-linux-gnueabi- -j 5
# Copy the kernel
$ cp arch/arm/boot/zImage /mnt/temp/boot/
# Copy the device tree block
$ cp arch/arm/boot/dts/tegra124-jetson-tk1.dtb /boot
# Install all modules
$ make modules_install INSTALL_MOD_PATH=/mnt/temp/
# Sync file system and unmount the Jetson
$ sync
$ umount /mnt/temp

Next step is to place a file called /boot/boot.scr. It’s pretty similar to GRUB’s menu.cfg. U-Boot reads this file when booting and executes its commands. Unfortunately, U-Boot uses a binary format and it must be compiled first. But first of all, install the u-boot-tools:

$ emerge u-boot-tools

Create the file boot.script that contains the following content:

# Optionally add 'init=/usr/lib/systemd/systemd', if you want to use systemd insteas of OpenRC
setenv bootargs 'console=ttyS0,115200 root=/dev/mmcblk0p1 rw rootwait'

#Comment the line above and uncomment these lines for running jailhouse
#setenv bootargs 'root=/dev/mmcblk0p1 rw rootwait mem=1984M vmalloc=512M'
#setenv bootm_boot_mode nonsec

load ${devtype} ${devnum}:1 ${kernel_addr_r} /boot/zImage
load ${devtype} ${devnum}:1 ${fdt_addr_r} /boot/tegra124-jetson-tk1.dtb
bootz ${kernel_addr_r} ${ramdisk_addr_r}:${filesize} ${fdt_addr_r}

Use the mkimage tool of the u-boot-tools to compile it to a .scr file:

$ cd /tmp/jetson/boot
$ wget
$ mkimage -A arm -O linux -T script -C none -a 0 -e 0 -n Boot-Script -d boot.script boot.scr

After rebooting, make sure that all CPUs are started in HYP mode:

$ root@iridium:~# dmesg | grep HYP
[ 0.101989] CPU: All CPU(s) started in HYP mode.

Here you can also download a prebuilt boot.scr. Copy the boot.scr file to the /boot directory of your jetson.

Compile U-Boot

Jailhouse requires the processor to run in non-secure HYP mode. The processor needs to be staged to HYP mode very early by the bootloader. HYP is not enabled in NVIDIA’s stock bootloader, so we need to compile and flash it on our own.

It took me quite some time to find out how to build the bootloader correctly.
Here’s the summary 🙂 Run the following commands:

$ mkdir /tmp/uboot
$ cd /tmp/uboot
# Get all the required stuff
$ git clone
$ git clone
$ git clone
$ git clone
$ git clone git:// u-boot

# build tegrarcm
$ cd tegrarcm
$ ./
$ make
$ cd ..

# build cbootimage
$ cd cbootimage
$ ./
$ make
$ cd ..

# build u-boot
$ export PATH=$PATH:/tmp/uboot/cbootimage/src:/tmp/uboot/tegrarcm/src
$ cd tegra-uboot-flasher-scripts
$ export CROSS_COMPILE=armv7a-hardfloat-linux-gnueabi-
$ ./build --socs tegra124 --boards jetson-tk1 build

Et voilĂ , you just built your own U-Boot bootloader. Let’s flash it! Before sending the next command, bring your device in recovery mode by pressing the “RESET” + “FORCE RECOVERY” buttons.

$ cd ../tegra-uboot-flasher-scripts
$ ./tegra-uboot-flasher --data-dir ../_out flash jetson-tk1

With a bit of luck and some magic unicorn dust, your jetson will directly boot your custom bootloader that boots your custom kernel. After rebooting, you can check if HYP is enabled with

$ root@iridium:~# dmesg | grep HYP
[ 0.101989] CPU: All CPU(s) started in HYP mode.

First Boot

After your device was successfully flashed it will automatically reboot. You can now login to it by using a serial connection.

$ putty -serial /dev/ttyUSB0 115200 -sercfg 115200

As we don’t have NTP yet, our first step will be to set the current time.

On your TK1 login and type:

$ date --set="20150723 18:37"
$ ####
$ # Your network should be autoconfigured via DHCP, if not try
$ ifconfig enp1s0 a.b.c.d
$ route add default gw a.b.c.e
$ echo nameserver > /etc/resolv.conf
$ ####
$ emerge --sync
$ emerge -av ntp
$ rc-update add ntp-client default
$ rc-update add sshd default
$ # Perform a full system upgrade
$ emerge -uDNav world

Compile Jailhouse

This will guide you how to compile Jailhouse on your Jetson TK1. Jailhouse comes with some dependencies and requires the mako python package in order to compile. Install mako as well as git before cloning and compiling jailhouse.

$ emerge git mako
$ git clone
$ cd jailhouse
$ cat > hypervisor/include/jailhouse/config.h # Create this file with the following content
#define CONFIG_ARM_GIC_V2 1
#define CONFIG_MACH_TEGRA124 1
$ make
$ make install
$ depmod -a
$ modprobe jailhouse
$ lsmod
# and check if the module was successfully loaded

Playing around with jailhouse

This tutorial will give you a brief introduction on how to run a simple uart-demo inmate in jailhouse on the TK1. It is important not to use the UART in the root cell as it will be assigned to the non-root cell after creating it. Any further UART access from the root cell after creating the non-root cell will lead to a panic of the hypervisor. Additionally, jailhouse needs some memory for the hypervisor itself and its inmates that is not used by the kernel of the root cell. This results in a boot.script as follows:

# Reserve 64MiB for jailhouse
setenv bootargs 'root=/dev/mmcblk0p1 rw rootwait mem=1984M vmalloc=512M'
# This enables HYP mode (disabled by default)
setenv bootm_boot_mode nonsec
load ${devtype} ${devnum}:1 ${kernel_addr_r} /boot/zImage
load ${devtype} ${devnum}:1 ${fdt_addr_r} /boot/tegra124-jetson-tk1.dtb
bootz ${kernel_addr_r} ${ramdisk_addr_r}:${filesize} ${fdt_addr_r}

Compile the boot.script using mkimage to boot.scr and copy it to ‘/boot/’.

$ mkimage -A arm -O linux -T script -C none -a 0 -e 0 -n Boot-Script -d boot.script /boot/boot.scr

Also, comment out the serial console from ‘/etc/inittab’. Use ssh to access your device. After rebooting your device, you can load the jailhouse module and activate the hypervisor:

$ modprobe jailhouse
$ jailhouse enable ~/jailhouse/configs/jetson-tk1.cell

Dmesg should now tell you, that the jailhouse has opened:

$ dmesg | tail
[ 183.608569] The Jailhouse is opening.

On your serial console, you should see something like:

Initializing Jailhouse hypervisor v0.5 (132-g997802f) on CPU 0
Code location: 0xf0000020
Page pool usage after early setup: mem 22/16112, remap 64/32768
Initializing processors:
CPU 0... OK
CPU 3... OK
CPU 2... OK
CPU 1... OK
Page pool usage after late setup: mem 34/16112, remap 64/32768
Activating hypervisor

Now you can create a new non-root cell using the jetson-tk1-demo cell configuration:

$ jailhouse cell create ~/jailhouse/configs/jetson-tk1-demo.cell

Now that your cell is created, you can load a binary to it and start the cell:

$ jailhouse cell load jetson-tk1-demo ~/jailhouse/inmates/demos/arm/uart-demo.bin -a 0
$ jailhouse cell start jetson-tk1-demo

On your serial console, you can now see the output of the demo cell.

To stop and destroy the cell, just type:

$ jailhouse cell shutdown jetson-tk1-demo
$ jailhouse cell destroy jetson-tk1-demo

Use my precompiled RootFS + Bootloader

If you don’t want to compile everything on your own – which i truly can understand – then you can use my pre-compiled images.

I try to keep those images up to date. Look at the top of the page for the current versions. SSH server is enabled, root password is ‘gentoo’, dhcp client on wired ethernet. So just a few steps for you to start.

Reset your Jetson, interrupt the bootloader and attach it as USB mass storage device (explained above) by typing

Tegra124 (Jetson TK1) # ums 0 mmc 0

to your serial console. Mount and partition your Jetson and extract the root file system:

$ mount /dev/sdX1 /mnt/temp/
# Extract Root FS
$ curl | tar x -C /mnt/temp/
# Extract pre-compiled kernel
$ curl | tar x -C /mnt/temp/
# Place boot.scr (non-jailhouse variant)
$ curl > /mnt/temp/boot/boot.scr
# Place boot.scr (jailhouse variant)
$ curl > /mnt/temp/boot/boot.scr

Next step is to flash the correct bootloader. On your local machine, type

$ git clone
$ git clone
$ cd tegrarcm
$ ./
$ make
$ export PATH="$PATH:`realpath src`"
$ cd ../tegra-uboot-flasher-scripts
$ curl | tar -xzv -C .
$ ./tegra-uboot-flasher --data-dir . flash jetson-tk1

Et voilĂ . Reboot your device and check if HYP is enabled if you chose the jailhouse variant:

root@iridium:/# dmesg|grep HYP
[ 0.101987] CPU: All CPU(s) started in HYP mode.

Pigz – Parallel gzip

This nice tool really is amazing:

Pigz is a parallel implementation of gzip and can be used like gzip.
As it is able to spawn mutiple threads it is much faster than the single-threaded gzip.


pigz -p 8 file # compresses file to file.gz using 8 threads
pigz -p 8 -d file.gz # decompresses file.gz to file using 8 threads

pv file.gz | pigz -d -p 8 > file # decompresses file.gz to file using 8 threads and a nice progress bar using pipe view :-)

dd over ssh / netcat

Nice trick to run dd over ethernet:

dd bs=16M if=/dev/sdX | gzip | nc server 1000

nc -l -p 1000 | gunzip | pv | dd bs=16M of=/dev/sdX

or via ssh:

dd bs=16M if=/dev/sdX | gzip | ssh user@server "gunzip | dd bs=16M of=/dev/sdb"

Gentoo: Using splitdebug for specific packages

It is often favourable to keep debugging information for certain packages.

Gentoo allows to keep those symbols by adding

CFLAGS="${CFLAGS} -ggdb"
FEATURES="${FEATURES} splitdebug"

to your make.conf

But this enables debugging information for all packages. In order to keep debugging symbols only for certain packages, you can define a splitdebug environment in /etc/portage/env/


CFLAGS="${CFLAGS} -ggdb"
FEATURES="${FEATURES} splitdebug"

Now you can add your packages to /etc/portage/package.env:

sys-libs/glibc splitdebug.conf

So when compiling glibc, portage will use the splitdebug environment and keep debugging symbols.