Raspberry Pi Compile Farm
=========================
[TOC]

## operating system
- Ubuntu Core 64bit Server preinstalled for Raspberry Pi -- https://ubuntu.com/download/raspberry-pi
- Replug the FTDI adapter if Serial Communication seems buggy
- the image allows serial connection by default

## hardware
- Pi4B 4GB 60.-
- pw supply 2A 10.-
- 5W -> 76kW a 17Rp/kWh -> per Pi per year ca. 15.-
- Samsung SSD T5 80.-/ 30 bis 40.-
- MicroSD card: 10.- für 32GB from Ultra SanDisk
- Totaling Cost for 3 years: 195.- -> 65.- per year and pi
- with network boot: total of 105 resulting in 35.- per pi and year
- server in france: 8\*4\*12 EURO per year

# overlay file system
## documentation, concepts and helpfull links
> For documentation of the **overlayfs** (built into kernel) **see man page of mount** (search for overlay). The **overlayroot** packet commes preeinstalled with ubuntu with the cloud-initramfs-tools bundle <https://packages.ubuntu.com/source/bionic/cloud-initramfs-tools>. Overlayroot is a high end wrapper for overlayfs and consists of initramfs scripts to automatically mount the rootfs.
- use `dpkg --listfiles overlayroot`. Avoid writing a custom initramfs script.
> Concider using a **memory based overlay file system** with an swap partition - might be beneficial as kenel is probably smarter
in deciding what to outsource and what to keep in memory (as files get chached by memory anywas)
- aditional links
- **basics of overlay** <https://windsock.io/the-overlay-filesystem/>
- <https://openwrt.org/docs/guide-user/additional-software/extroot_configuration>
- <https://spin.atomicobject.com/2015/03/10/protecting-ubuntu-root-filesystem/>

## Setup
- In **/media/root-ro/** is the underlying filesystem mounted.
- In **/media/root-rw/overlay/** the overlay changes are stored.

root@raspi4-1:~ lsblk
NAME        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda           8:0    0 223.6G  0 disk
├─sda1        8:1    0   110G  0 part /media/root-ro
├─sda2        8:2    0    20G  0 part /media/root-rw
├─sda3        8:3    0   3.6G  0 part
└─sda4        8:4    0    90G  0 part /root/bamboo-agent-home

- the home dir of the bamboo agent is persistent.

## Step-by-step
1) `e2label /dev/sda2 overlay` mark overlay partition for later use
2) `e2label /dev/sda4 persistent` mark partition for bambo-agent-home that is not included in the overlay
3) make the bamboo-agent homedir persistent across reboots `echo 'LABEL=persistent /root/bamboo-agent-home/ ext4 defaults,rw 0 0' >> /etc/fstab`
4) ` echo 'overlayroot=device:dev=LABEL=overlay,timeout=180,debug=1,recursive=0' > /etc/overlayroot.local.conf` make the the overlaydevice, recursive=0 excludes on rootfs mounted file systems from the overlay
5) `reboot`
6) chekc motd on logging: output should be (due to debug=1 in config):

root@raspi4-1:~ bash /etc/update-motd.d/97-overlayroot
overlayroot / overlay rw,relatime,lowerdir=/media/root-ro,upperdir=/media/root-rw/overlay,workdir=/media/root-rw/overlay-workdir/_ 0 0
/dev/sda1 /media/root-ro ext4 ro,relatime 0 0
/dev/disk/by-label/overlay /media/root-rw ext4 rw,relatime 0 0

7) Make Script to controll the overlay.

cat > /usr/local/bin/overlay-control < EOF

#!/usr/bin/env bash
# This script controls the behavier of the overlay file system.

# Edits
# Licence GNU GLPv3 (or whatever you want if you ask)
# 04.04.2020: Andrino Meli
#

# TODO: make absolute paths


DEBUG=
[[ ! -z DEBUG ]] && set +x 

USSAGE="$0 [ help | status | chroot | overlay | no-overlay | reset ]"

reboot_prompt () {
	LOOP="loop"
	while [ $LOOP == "loop" ]
	do
		read -e -p "You should reboot, reboot now? [Y,n]: " RES
		case $RES in 
			"y"|"Y"|"")
				reboot
				;;
			"n")
				LOOP="quit"
				;;
			*)
				;;
		esac
	done
}

function isoverlay {
	[[ -d /media/root-ro/ ]] && return 0
	return 1
}

# Print Help
[[ -z $1 ]] && echo "$USSAGE" && exit 1
[[ $1 == "help" ]] && echo "$USSAGE" && exit 0

# print status
if [[ $1 == "status" ]]
then
	if isoverlay; then echo "Running in overlay mode."
	else echo "No overlay active."
	fi
	exit 0
fi


# check permissions
[[ $USER != "root" ]] && echo "run as root! " && exit 5


# reboot with overlay
if [[ $1 == "overlay" ]]
then
	# only chroot if overlay is enabled
	if isoverlay; then overlayroot-chroot sed 's/^# overlayroot=/overlayroot=/' -i /etc/overlayroot.local.conf
	else sed 's/^# overlayroot=/overlayroot=/' -i /etc/overlayroot.local.conf
	fi
	[[ ! -z $DEBUG ]] && cat /etc/overlayroot.local.conf
	reboot_prompt
	exit 0
fi


# reboot with no overlay
if [[ $1 == "no-overlay" ]]
then
	if isoverlay; then overlayroot-chroot sed 's/^overlayroot=/# overlayroot=/' -i /etc/overlayroot.local.conf
	else sed 's/^overlayroot=/# overlayroot=/' -i /etc/overlayroot.local.conf
	fi
	[[ ! -z $DEBUG ]] && cat /etc/overlayroot.local.conf
	reboot_prompt
	exit 0
fi


# reset overlay (delete changes) - consider rebooting
if [[ $1 == "reset" ]]
then
	if isoverlay; then overlayroot-chroot rm -rf /media/root-rw/overlay/*
	else echo "TODO: implement this feature - reformat the overlay partition"
	fi
	reboot_prompt
	exit 0
fi


# reset overlay (delete changes)
if [[ $1 == "chroot" ]]
then
	if isoverlay; then overlayroot-chroot && reboot_prompt
	else echo "Chroot failed: no overlay active"
	fi
	exit 0
fi

# none of the arguments match - print help
echo "$USSAGE" && exit 1

EOF

9) Make the script for shell autocompletion.

cat > /usr/local/bin/overlay-control-completion.bash < EOF
#/usr/bin/env bash
complete -W "help status chroot overlay no-overlay reset " overlay-control

EOF

10) `echo 'source /usr/local/bin/overlay-control-completion.bash' >> /root/.bashrc`
11) `chmod +x /usr/local/bin/overlay-control-completion.bash`
11) `chmod 744 /usr/local/bin/overlay-control`

## use cases
- simplest is to just use the script
- `rm -rf /media/root-rw/overlay/` resets the file system
- use **overlayroot=disabled** as a kernel boot argument to bypass the overlay or edit the config file
- use the ``overlayroot-chroot`` command to make changes (updates) on the underlying fs. As the kernel does not like to remount an in-use rootfs read-only other proagramms could potentially use /media/root-ro/ to write to the rootfs directly. If one is really concered about that, consider rebooting.

# Network Boot
## documentation and links
- https://kiljan.org/2019/11/16/arch-linux-arm-network-boot-on-raspberry-pi-4/
1) the raspberry boots from a a SoC which runs some closed source binary blob in the GPU. The GPU then loads the CPU. This process is controlled by bootconf.txt
2) the EEPROM Boot Code executed in the GPU loads start4.elf (for 4B Model i think not anymore)
3) There is basically no such thing as a BIOS and the raspi is only bootable from SD-Card.
4) There is a beta EEPROM boot code since 23 September 2019. It allows to boot the Pi4B to load the necessary files from an TFTP server using the embedded Ethernet controller. -> EEPROM reprogramming <https://github.com/raspberrypi/rpi-eeprom>
- Boot Configuration in EEPROM (bootconf.txt) or altern. via DHCP
- **DHCP Server** or **SLAAC**
- The Mac Adress and Serial Number determine which Directory from the **TFTP Server** the Pi Boots from, if start.elf not found: the Pi boots from /
- start4.elf proprietary - provided by rootfs
- Loaded Files: start.elf, config.txt, kernel.img, **initramfs-linux.img**, cmdline.txt (nobtcmd.txt)
- initramfs must be prepared to boot with network rootfs
- The kernel can boot with the initramfs but now requires the rootfs iSCSI/NBD or **NFS**/SMB

## server setup
1) **install**, sshd, ufw, hostname, network(netplan), update, upgrade, gen. sshkey to copy to nodes
2) create partition table on ssd(rootfs 50G, vmdisks) and mkfs.ext4
3) move / to the ssd
- create partition table on /dev/sda and make file systems, `mount /dev/sda1 /mnt`
- **rsync** rootfs to ssd `sudo rsync / -aAXv --exclude={"/dev/*","/proc/*","/sys/*","/tmp/*","/run/*","/mnt/*","/media/*","/lost+found"} /mnt`
- Empty the folder /mnt/boot/firmware. The Ubuntu Core OS uses U-Boot.
- U-Boot mounts the boot partition /dev/mmcblk0p1 on /boot/firmware
- U-Boot:<https://github.com/umiddelb/armhf/wiki/Get-more-out-of-%22Das-U-Boot%22>
- official doc: <http://www.denx.de/wiki/view/DULG/Manual>
- For U-Boot config file structure consult /boot/firmware/README.
- change device label of the mounted partition: `gdisk /dev/sda1` and label sda1 *rootfs* `e2label /dev/sda1 rootfs`
- add/replace rootfs label in /etc/fstab ("/" is mount point of the ssd)
- change the LABEL to rootfs in /boot/firmware/nobtcmd.txt (the no bluetooth cmdlne kernel arguments (Alternativelly do the same with UUID)
- reboot check if `lsblk` that /dev/sda1 is mounted on /
- on problems work with u-boot cli interface (press any key to interrupt kernel startup). Use *u-boot-tools* to change environment variables directly from linux
4) Create a Base Image
- create /nfs/base
- get fresh "Ubuntu Core" base image in */nfs/*
- `partprobe $(losetup --find --show /vm/baseimage.img)`
- mount the two partitions of the image on /mnt and /mnt/boot/firmware
- `sudo rsync /mnt/ -aAXv /nfs/base/`
- configure files if necessary (network, ssh, ...)
5) Setup TFTP
- create /tftp
- `echo "/nfs/base/boot/firmware /tftp/ none defaults,bind 0 0" | sudo tee -a /etc/fstab`
- `sudo mount /tftp; sudo chown -R tftp:tftp /tftp`
- install tftp server package *tftpd-ha*
- change config in /etc/default/tftpd-hpa to:
- TFTP_USERNAME="tftp"
- TFTP_DIRECTORY="/tftp"
- TFTP_ADDRESS="192.168.1.2" #nfs server address
- TFTP_OPTIONS="--map-file /etc/tftpd.mapfile --secure -v -R 50000:50000" # use port 50000 (for firewall)
- TODO: create mapfile in */etc/tftpd.mapfile*
- restart *tftp-hpa.service*
- power raspberry, REQUEST CYCLE should be (for future reference) `journalctl _SYTEMD_UNIT=tftp-hpa.service -f`:
RRQ dde3fc98/start4.elf
RRQ dde3fc98/config.txt
RRQ dde3fc98/start4.elf
RRQ dde3fc98/fixup4.dat
RRQ dde3fc98/recovery.elf
RRQ dde3fc98/config.txt
RRQ dde3fc98/syscfg.txt RRQ dde3fc98/nobtcfg.txt
RRQ dde3fc98/usercfg.txt
RRQ dde3fc98/dt-blob.bin
RRQ dde3fc98/config.txt
RRQ dde3fc98/syscfg.txt
RRQ dde3fc98/nobtcfg.txt
RRQ dde3fc98/usercfg.txt
RRQ dde3fc98/bcm2711-rpi-4-b.dtb
RRQ dde3fc98/config.txt
RRQ dde3fc98/syscfg.txt
RRQ dde3fc98/nobtcfg.txt
RRQ dde3fc98/usercfg.txt
RRQ dde3fc98/nobtcmd.txt
RRQ dde3fc98/uboot_rpi_4.bin
RRQ dde3fc98/armstub8-gic.bin
RRQ dde3fc98/uboot_rpi_4.bin
6) Setup NFS
- bind mount /nfs/base to /nfs/node0 (add entry in file system table)
- install *nfs-kernel-server* package
- add "/nfs/node0 192.168.1.3(crossmnt,insecure,rw,fsid=0,no_root_squash,no_subtree_check)" int /etc/exports
or use 192.168.1.0/24 for the entire subnet instead of a single host, alternativelly use a dns hostname
- test mounting with mount 192.168.1.2:/ /mnt/ -v

#### whiteboard
Basic Setup, sshd, ufw, hostname, network(netplan), update, upgrade, setup
Plan:
- setup bootable disk
- reprogramm firmware
- tftp service
- nfs service
- prepare initramfs and kernel cmd to boot via nfs
x. Setup cloud init on vm disk a Server with: rootfs, TFTP, NFS, DHCP (preferably in its own network)

## client setup
### firmware updae
- some help: <https://github.com/raspberrypi/rpi-eeprom/blob/master/firmware/raspberry_pi4_network_boot_beta.md>
1) read all the step by step instructions (ESPECIALLY 8.5 as this might cut down the work to two commands)
2) dedicate a microSD Card to update the EEPROM: reusable of course, so only one is needed
3) Flash Raspian Buster Lite onto the SSD Card and mount both partitions. Put an empty file */ssh* on the root partition or add *enable_uart=1* to /config.txt in the boot partition.
4) Install Raspian Buster Lite, upgrade, update.
5) `apt install rpi-eeprom` Download latest firmware `rpi-update` - not sure if this is needed. For Network Boot you need a firmware newer than 10.Sept 2019. Check with `rpi-eeprom-update`. If the firware is still to old you might need to switch to the Beta-Channel. `sed -i /etc/default/rpi-eeprom-update -e 's/^FIRMWARE_RELEASE_STATUS.*$/FIRMWARE_RELEASE_STATUS="beta"'`
6) Update Firmware with `rpi-eepromt-update -a` and confirm, yeah its that simple. Reboot to install firmware. TODO: Note this allways auto updates to the newest Beta Firmware when booting from SD-Card. This might not be what we want as the Version from 10.9.2019 is sufficient and might be more stable. Beta firmware (and recovery firmware) resides in /lib/firmware/raspberrypi/bootloader/beta/. Check if Firmware is Bricked; unplugg all the devices an power on the board. If the Green LED flashes and eventually starts to cycle (per cycle 4 flashes) Firmware is ok. If the Green LED is off create a recovery SD.
7) If further updates are not wanted `systemctl mask rpi-eeprom-update`
8) Enable Network Boot. (FIRMWARE)
1) Jump to point 5 and see if you can skip the Firmware Building Step
2) `cp /lib/firmware/raspberrypi/bootloader/beta/pieeprom-2020-01-09.bin pieeprom.bin` (use required version)
3) Change Config:
4) BOOT_ORDER=0x21 (first try to boot from SD-Card, then from Network)
5) BOOT_UART=1
6) NET_BOOT_MAX_RETRIES=-1
7) TFTP_IP=192.168.x.x -> this setting can also be retrived with dhcp
8) Rebuild Firware file `rpi-eeprom-config --out newpieeprom.bin --config bootconf.txt pieeprom.bin`
9) Create new binary. `rpi-eeprom-update -d -f ./newpieeprom.bin` **This file resides in the home dir of the flash SSD and can be reused (Firmware 2020-01-09)**
10} EDIT EEPROM file newpieeprom.bin now called **custom-netboot-firmware2020-01-09.bin**
11) Flash: `sudo rpi-eeprom-update -d -f ./newpieeprom.bin` Reboot.
12) Check with `rpi-eeprom-update`.
9) Enable Network Boot. (SOFTWARE - requires U-Boot and a permament SD-Card)
- TODO